As described in an earlier post, I run the Engineering at an early stage #traveltech #startup called Itilite<\/a>. So, one of my responsibility is to architect, build and manage the cloud infrastructure for the company. Even though I have had designed\/built and maintained the cloud infrastructure in my previous roles, this one was really challenging and interesting. Due in part to the fact, that the organisation is a high growth #traveltech startup and hence,<\/p>\n And several others. Somewhere down the lane, after the infrastructure, code-pipeline and CI is set-up, you reach a point where managing (read: trigger intervention, analysis, storage, archival, retention)<\/em> logs across several set of infrastructure clusters like development\/testing, staging and production becomes a bit of an overkill.<\/p>\n Having worked up from a simple tail\/multitail to Graylog-aggregation of 18 server logs, including App-servers, Database servers, API-endpoints and everything in between. But, as my honoured colleague (former) Mr.Naveen Venkat<\/a> (CPO of Zarget) used to mention in my days with Zarget, There are no \u201cGo-To\u201d persons in a start-up. You \u201cGo-Figure\u201d yourself!<\/p>\n There is definitely no \u201cOne size fits all\u201d solution and especially, in a Start-up environment, you are always running behind Features, Timelines or Customers (scope, timeline, or cost in conventional PMI model).<\/p>\n So, After some due research to account for the recent advances in Logstash and Beats. I narrowed down on the possible contenders that can power our little log management system. They are,<\/p>\n (I did not consider anything exotic or involves us paying (in future) anything more than what we pay for it in first year. So, some great tools like splunk, nagios<\/a>, logpacker, logrythm were not considered)<\/p>\n I wrote an Ansible script to create a replica environment and pull in the necessary configurations. And used previously written load-test job to simulate a typical work hour. This configuration was used for each of the frameworks\/tools considered.<\/p>\n I started experimenting with Graylog<\/a>, due to familiarity with the tool. Configured it the best way, I felt appropriate at that point in time.<\/p>\n However, the collector I had used (Sidecar<\/a> with Filebeat) had a major problem in sending files over 255KB and the interval was less than 5 secs. And the packets that are to be sent to the Elasticsearch never made it. And the pile-up caused a major issue for application stability.<\/p>\n One of the main use-case for us is to ingest XML\/JSON data from multiple sources. (We run a <\/em>polynomial regression <\/em><\/a>across multiple sources, and use the <\/em>nth<\/em><\/strong> derivatives to do further business operations). Our architecture had accounted for several things, but by design, we used to hit momentary peaks in CPU utilisation for the \u201cMerges\u201d. And all of these were \u201cNICE\u201d loads.<\/em><\/p><\/blockquote>\n When the daily logs you need to export is in upwards of 5GB for an app (JSON logs), add multiple APIs and some micro-services application logs, web-server, load-balancers, CI (Jenkins), database-query-log, bin-log, redis and\u00a0\u2026 yes, you get the point?<\/em><\/p><\/blockquote>\n (())Upon further investigation, The sidecar collector was actually not the culprit. Our architecture had accounted for several things, but by design, we used to hit momentary peaks in CPU utilisation for the \u201cMerges\u201d. And all of these were \u201cNICE\u201d loads! (in our defence)\u00a0<\/em><\/p><\/blockquote>\n So, once the CPU hit 100% mark, sidecar started behaving very differently. But, ultimately fixed it with a patched version of <\/em>sidecar and actually shifting to NXLog.<\/em><\/strong><\/p><\/blockquote>\n\n
Enter Log Management & Analytics<\/h3>\n
\n
Evaluation Process:<\/h3>\n
Slight setback:<\/h3>\n