Automate server log analysis: infrastructure and syslog
Handling syslog at volume, correlating events across hosts, building baselines that account for daily and weekly rhythm, and catching authentication and resource problems early.
Osman Ramadan10 min read
On this page
Server logs differ from application logs in ways that change how you analyse them. The volume is higher and mostly routine, the interesting events are spread across hosts rather than concentrated in one place, and the failures that matter are frequently gradual: a disk filling over nine days, a certificate approaching expiry, authentication failures climbing slowly from one address.
This page covers that work. For application exceptions, stack traces, and correlating errors with releases, see automating log analysis.
What we'll cover
What server logs are telling you
Syslog carries several distinct conversations in one stream, and separating them is the first useful step.
The kernel and hardware report disk errors, memory pressure, network interface problems, and thermal events. Low volume, high signal: a kernel message about I/O errors is almost always worth attention.
System services report their own starts, stops, failures, and restarts. A service restarting repeatedly is one of the clearest indicators available and is easily lost in volume.
Authentication records every login, sudo, key acceptance, and failure. The highest-value stream for security purposes and usually the noisiest.
Scheduled jobs report what ran and what it produced, which is where you learn that a backup has been failing since a fortnight ago.
Network services report connections, requests, and rejections, at volumes that dwarf everything else.
Routing these to separate handling, rather than analysing one undifferentiated stream, makes everything downstream simpler. The thresholds, retention, and urgency differ by category, and treating them alike means either drowning in web access logs or missing kernel messages.
Getting the volume under control
Infrastructure logging produces far more data than application logging, and processing all of it identically is expensive and unnecessary.
Aggregate at the edge. Summarizing on each host before shipping, so that ten thousand identical entries become one record with a count, reduces volume by orders of magnitude while losing almost nothing that matters.
Sample the routine, keep all of the unusual. Successful requests can be sampled heavily. Errors, warnings, and authentication failures should be kept in full. Sampling uniformly is the mistake, because it thins exactly the rare events you built the system to catch.
Tier your retention. Recent logs in fast searchable storage, older logs compressed in cheap object storage. Most investigation touches the last few days; compliance needs the last year; paying fast-storage prices for the whole year is a common and avoidable expense.
Filter at ingestion, deliberately. Dropping known-noisy sources is reasonable and should be recorded somewhere, because a filter added during an incident two years ago and forgotten is how a blind spot forms.
Parse into fields early. Host, facility, severity, service, message. Once structured, aggregation and correlation become straightforward; left as strings, every question requires text matching.
Correlating across hosts
The distinctive difficulty of infrastructure analysis is that a single incident appears as unremarkable entries on eight different machines.
Synchronize clocks and normalize to UTC. Correlation depends entirely on comparable timestamps. Hosts drifting by thirty seconds make ordering meaningless, and this is the foundation everything else needs.
Look for events clustering in time across hosts. Five machines reporting network problems in the same ninety seconds is a network event, not five host events. Grouping by time window across hosts is what turns a pile of individual alerts into one diagnosis.
Follow the dependency order. A database host reporting slow queries, then application hosts reporting timeouts, then the load balancer reporting failed checks is one causal chain. Knowing which services depend on which lets the analysis point at the cause rather than listing the symptoms.
Compare against fleet peers. When one host in a group of twenty behaves differently, that host is the story. This is frequently easier and more reliable than absolute thresholds, because the peers supply the baseline.
Preserve the whole cluster in the alert. An alert naming one host when eight are affected sends someone to investigate the wrong thing. The alert should carry the shape of the event.
Baselines that respect rhythm
Static thresholds generate false alarms on Monday mornings and miss real problems at three on Sunday, because infrastructure has a rhythm.
Compare like with like. Measure against the same hour of the same weekday over recent weeks, not against a flat average. Traffic at nine on Tuesday should be compared with nine on previous Tuesdays.
Use robust measures. A median and interquartile range are far less distorted by one previous incident than a mean and standard deviation, and your history certainly contains previous incidents.
Exclude known incidents from the baseline. Otherwise last month's outage becomes part of normal, and the system gradually learns to tolerate the thing you want it to catch.
Account for growth. A baseline over a growing service should track the trend rather than flagging every week's normal increase. Comparing against a recent rolling window rather than a fixed period handles this.
Alert on shape as well as level. A metric that is within range but has stopped varying at all is usually a collection failure rather than good news. Flat lines deserve suspicion.
Allow for scheduled work. Backups, batch jobs, and maintenance windows produce large legitimate deviations. Either exclude those windows or teach the baseline to expect them.
The slow failures
The failures that cause real outages rarely announce themselves. They accumulate.
Disk filling. The useful signal is the trend rather than the level. A partition at 71 percent and rising two points a day is a more urgent problem than one sitting at 88 percent unchanged for a year. Project forward and alert on the projection.
Certificates approaching expiry. Entirely predictable and still a common cause of outages. A weekly check on every certificate with its days remaining costs almost nothing and removes the category.
Memory creeping. Gradual growth over days with periodic restarts masking it. Comparing usage at the same point after each restart exposes a leak that a simple threshold never catches.
Backups silently failing. The classic, because the failure is invisible until the restore. Checking that the backup ran, that it completed, and that the resulting file is a plausible size catches nearly all of it.
Queues growing slowly. A consumer keeping up at ninety-five percent of the arrival rate looks healthy on any instantaneous check and is falling behind continuously. The derivative is the signal, not the depth.
Log volume changing. Sudden increases mean something started failing. Sudden decreases mean something stopped reporting, which is usually worse and almost never alerted on.
Describing these checks is considerably easier than assembling them from monitoring components. On CodeWords you describe what should be watched in plain language, what counts as abnormal, and who should hear about it. Cody, the automation builder, builds it, connects it to your logging and alerting tools, and deploys it. Automations connect to more than 3,000 integrations. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.
Authentication and access patterns
Authentication logs are noisy and worth the effort, because they carry the events with the highest consequence.
Failed logins mean little individually. People mistype passwords constantly. Rate and pattern are what matter: many failures against one account, one password tried against many accounts, or failures from an address that has never appeared before.
Success after repeated failure deserves attention. The sequence of many failures followed by a success is worth surfacing whether it was a forgetful colleague or something else, and it costs nothing to look.
Watch privilege escalation. Sudo and administrative actions should be a reviewable stream. Volume is low, and the value when something goes wrong is high.
Flag geographic and temporal impossibility. Successful authentication from two distant locations within an implausible interval is a strong signal, and one of few that works without knowing anything about your particular environment.
Track new sources. An address, key, or user agent authenticating for the first time is worth a note. Most are benign and the exceptions matter a great deal.
Reconcile with your people. Accounts belonging to departed staff, keys nobody claims, and service accounts nobody can name are found by comparing the logs against a list of who should exist. This is an unglamorous automation with a habit of finding things.
A sensible order to build this in
Attempting all of the above at once produces something half-finished that nobody trusts. A staged order gets value early and builds the history the later stages need.
Start with collection and parsing. Logs from every host reaching one place, parsed into fields, with clocks synchronized. This is unglamorous and everything else depends on it, and it is worth confirming that every host is genuinely reporting before moving on.
Add the certain checks next. Certificate expiry, disk projection, backup completion, service restart counts. These need no baseline, produce almost no false alarms, and each removes a known cause of outages. They also demonstrate value quickly, which matters for keeping the work funded.
Then build the baselines. Now that you have a few weeks of history, the same-hour-same-weekday comparisons become meaningful. Run them in report-only mode first and read the output for a fortnight before letting anything page a person.
Add correlation once alerts are trusted. Grouping events across hosts and following dependency order is the most sophisticated part and the least useful on its own. It earns its place when you already have reliable signals to correlate.
Review and prune continuously. Every month, look at what fired and what anyone did about it. Rules nobody acted on should be removed or downgraded to a digest, and the discipline of actually removing them is what keeps the system worth reading.
Frequently asked questions
How much log retention do I actually need?
Long enough to investigate an incident found late, which realistically means several weeks in searchable storage, plus whatever compliance requires in cheap storage. Tiering keeps the cost sensible while preserving the history.
Is this a replacement for a monitoring platform?
No, and the two complement each other. Monitoring platforms handle collection, storage, dashboards, and querying. An automated workflow adds the judgement on top: correlating across sources, applying your own context, and deciding what deserves an interruption. Most teams use both.
How do I stop alert fatigue?
Alert on what someone would act on immediately, and digest the rest. Suppress acknowledged issues, group related alerts into one message, and review what fired every month, removing anything nobody acted on. An alert nobody acts on is training people to ignore the channel.
What about logs from managed and cloud services?
Most export to a central destination, at which point they join the same pipeline. The formats differ and the analysis is the same, and the cross-host correlation above becomes cross-service correlation.
Should I analyse logs in real time or in batches?
Both, for different things. Authentication anomalies and service failures want near real time. Trends, baselines, and slow failures are better in a scheduled pass, which is cheaper and gives more reliable results because it sees a complete window.
How do I handle logs from hosts that come and go?
Identify by role and group rather than by hostname, since autoscaled instances have names that mean nothing. Comparing against peers in the same group handles ephemeral infrastructure far better than per-host thresholds, which never settle.
Do I need a dedicated log platform to start?
No. Useful checks can run against whatever you already have, including logs collected into cloud storage or a modest database. A dedicated platform helps at volume and with interactive investigation, and it is worth starting with the checks rather than waiting for the platform.
What is the single highest-value check to build first?
Backup verification, in most environments. It is simple, it needs no baseline, and the failure it catches is the one with the worst consequences and the longest history of going unnoticed. Certificate expiry is a close second for the same reasons.