How to connect Elasticsearch to Slack with queries as alert definitions

An alert built on Elasticsearch is three things: a query that selects the interesting documents, a threshold that decides when the count is abnormal, and a schedule that decides how often to ask. Most of the difficulty is in the first, because a query is a statement about your data's shape and your data's shape changes.

The queries that survive are narrower and more explicit than the ones people write first, and the alerts that get read are the ones whose query was written to answer a specific question rather than to catch anything unusual.

What we'll cover

• An alert is a query, a threshold, and a schedule
• Queries that stay true
• Building it with CodeWords
• Aggregations, which do the work
• Alerting on absence
• Keeping the query cheap
• What the message should contain
• Reviewing the alert suite
• Making it survive
• Limits worth knowing about
• What to set up first
• Frequently asked questions

An alert is a query, a threshold, and a schedule

Being explicit about the three parts makes each decision clearer.

The query selects. Errors from one service, failed logins, requests above a latency, orders that failed to process — the narrower the better.

The threshold decides. A count, a rate, or a comparison against the same window last week. Absolute counts are the easiest and the most brittle.

The schedule asks. Every minute is rarely necessary and is expensive; most useful alerts run every five to fifteen minutes.

The window matters too. A count over five minutes and over an hour behave very differently, and the window should match how quickly the thing you are watching actually develops.

Write each part down. An alert whose definition lives only in a config file nobody reads is one that nobody can evaluate when it fires at three in the morning.

Queries that stay true

The maintenance problem, and it is the reason log alerts decay.

Match on structured fields, not on message text. A query looking for a phrase in a log line breaks the day somebody rewords the message, and nothing tells you — the alert simply stops firing.

Require the fields you filter on to exist. A filter on a field that stopped being emitted matches nothing, which looks exactly like a healthy system.

Be explicit about the index pattern. A pattern that silently stops matching after an index naming change produces zero results and no error.

Avoid wildcards at the start of a term, which are expensive and often not what you meant.

Test the query against a period you know had the condition. An alert that has never matched anything is usually broken rather than lucky, and this is the check that finds it.

Review quarterly. Logging changes, services get renamed, and an alert suite that nobody audits degrades into decoration.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which query, what threshold, how often, and who should hear about it.
3. Authorize the connection with credentials for a role that can read only the relevant indices, and to your Slack workspace.
4. Describe the exceptions: a query returning nothing when it should return something, a cluster that is slow to respond, a threshold crossed during a known deployment.
5. Test against a historical window where you know the condition occurred.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Aggregations, which do the work

Returning documents to count them in your automation is the common approach and usually the wrong one.

Aggregate in the cluster. A count, a terms aggregation, or a date histogram returns a small result instead of thousands of documents.

Set the document size to zero when you only want the aggregation, which avoids transferring hits you will discard.

A terms aggregation tells you the shape, not just the count — which endpoints, which customers, which hosts are producing the errors. That is the difference between an alert somebody can act on and one that says a number went up.

Date histograms show whether it is a spike or a level, which changes the response entirely.

Watch cardinality. A terms aggregation over a high-cardinality field is expensive, and the answer is usually a top-ten rather than everything.

Alerting on absence

The condition a search cannot express, and the one that catches real outages.

A query for documents that are not there returns zero, which looks identical to a healthy system with no errors.

So alert on a count below a threshold, not only above. A service that normally logs continuously and has stopped is either idle or dead.

Include a minimum expected volume per index or per service, derived from its own history.

Account for the schedule. A batch job that logs once an hour should not alert for fifty-nine minutes of silence.

This is the alert that catches an ingestion pipeline failure, where logs stop arriving entirely and every other alert quietly goes green.

Keeping the query cheap

Alerts run constantly, and an expensive one competes with everything else using the cluster.

Filter by time first and narrowly. Searching the last five minutes is dramatically cheaper than searching the last day.

Use filter context rather than query context where you do not need scoring, since filters are cacheable and scoring is wasted work for an alert.

Restrict to the indices that matter, rather than searching every index a pattern matches.

Do not run every alert every minute. Stagger schedules so twenty alerts do not all fire their queries simultaneously.

Watch the cost of the alerting itself. A suite of alerts querying a busy cluster every minute is real load, and it is load that grows silently as people add alerts.

What the message should contain

The count and the comparison. What the threshold was and what the value is, so the recipient can judge severity immediately.

The shape from the aggregation — the top few contributors — which is what turns the alert into a starting point rather than a prompt to go and look.

A representative example. One document, trimmed, because an error message is worth more than a count.

A link to the query in your search interface, pre-filtered to the same window, so investigation is one click.

Whether it is rising or steady, since an alert that has been firing for an hour and one that just started need different responses.

Not the raw documents. A message containing fifty log lines is unreadable, and the link covers it.

Reviewing the alert suite

Alert suites decay in one direction, and only a deliberate review reverses it.

Count what fired and what was acted on. An alert that fires weekly and has never changed anybody's behaviour is costing attention for nothing.

Find the ones that have never fired. Some are genuinely waiting for a rare condition; most are broken queries nobody noticed. Test each against a historical window that should have matched.

Find the ones that fire constantly. These are either a threshold problem or a real problem nobody is fixing, and both deserve a decision rather than continued muting.

Check ownership. An alert whose owner has left is an alert that will be ignored, because nobody feels responsible for it.

Delete rather than mute. A muted alert still runs, still costs cluster time, and still looks like coverage on a list.

Do it quarterly. It takes an hour, it is uncomfortable, and it is the only thing that stops an alerting suite becoming decoration.

Making it survive

Deduplicate. An alert firing every five minutes for the same ongoing condition should produce one message with updates, not twelve messages.

Resolve explicitly. Say when the condition clears, or people assume it is still broken.

Handle the cluster being unavailable as its own alert, sent somewhere else, since a monitoring query that cannot run is a failure that looks like silence.

Use a read-only role scoped to the relevant indices.

Report the outcome. Alerts evaluated, fired, and suppressed as duplicates.

Limits worth knowing about

Result sets are capped by default for deep pagination, which matters when an alert tries to return everything it matched.

Aggregations have memory limits, and a high-cardinality terms aggregation can be refused rather than slow.

Index patterns resolve at query time, so a naming change silently changes what is searched.

Field mappings matter. A field mapped as text behaves differently in aggregations from one mapped as keyword, and getting this wrong produces confusing results rather than errors.

Watcher and alerting features are licence-dependent in some deployments, which is worth checking before deciding to build externally or not.

What to set up first

An absence alert on your most important log stream: a count below the minimum you would expect for that service at that time of day. It is the alert that catches an ingestion failure, and an ingestion failure turns every other alert you have into a source of false reassurance.

Two habits make the difference. Derive the minimum from the service's own history per hour, since a flat number fires every night. And route it somewhere different from your other alerts, because when logging breaks, the alerting that depends on it is exactly what you cannot trust.

Frequently asked questions

Why did an alert stop firing?

Usually the query stopped matching — a reworded log message, a renamed field, or an index pattern that no longer resolves. All three produce zero results and no error, which is indistinguishable from a healthy system.

How do I alert on something not happening?

Alert on a count below a minimum, derived from the service's own history and aware of the time of day. A search for absent documents returns zero, which looks exactly like everything being fine.

Should the alert return documents or an aggregation?

An aggregation, with size zero. It transfers a fraction of the data and it tells you the shape — which endpoints or customers are involved — which is what makes the alert actionable rather than a number going up.

How often should alerts run?

Every five to fifteen minutes for most things. Every minute is rarely necessary, it multiplies the load on the cluster, and staggering schedules matters once you have more than a handful.

Why is our alerting slowing the cluster?

Because the alerts are load, and it grows as people add them. Filter narrowly by time, use filter context rather than scoring, restrict to relevant indices, and stagger schedules so they do not all run at once.

What should the Slack message contain?

The count against the threshold, the top few contributors from the aggregation, one trimmed example document, a link to the same query in your search interface, and whether it is rising or steady. Not the raw documents.

Should I use built-in alerting instead?

Where it is available in your deployment and covers the case, yes. Build externally when you need to join with data from other systems, route by business logic, or send somewhere the built-in connectors do not reach.

How do I tell which alerts are worth keeping?

Count how often each fired and how often anybody acted. Alerts that never fire are usually broken queries rather than rare conditions — test each against a window that should have matched. Alerts that fire constantly need a threshold change or a fix.

Is muting an alert enough?

No. A muted alert still runs, still costs cluster time, and still appears as coverage on a list while catching nothing. If it is not worth acting on, delete it.

Related reading

• Elasticsearch and CodeWords
• Automating log analysis
• CodeWords integrations and templates