CircleCI and CodeWords: measuring the pipeline honestly

CircleCI, like any CI platform, already runs whatever you put in the config file, so most automation people describe as a CircleCI integration belongs in a job rather than outside one. The useful outside work is narrower and it is almost all measurement.

Specifically: what the pipeline costs, what is slow, and what fails for reasons unrelated to the code. None of those questions is answerable from inside a single job, because each is about the pattern across thousands of runs.

What we'll cover

• What belongs in the config file
• What credits actually buy
• What the CircleCI API reaches
• Connecting it to CodeWords
• Seven automations worth building
• Flaky tests, the expensive kind
• Caching, which quietly stops working
• Secrets and contexts, which deserve an audit
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

What belongs in the config file

Put it in the pipeline if it is any of these, and resist moving it out.

Triggered by a commit. The trigger already exists.

Needs the repository. A job has a checkout.

Should block a merge. Only a pipeline status can gate.

Should be visible to developers when it fails, in the place they are already looking.

Versioned with the code, which config files are and external automation is not.

What belongs outside is reporting across runs, anything connecting to systems CircleCI cannot see, and anything triggered by something other than a commit. That is a short list and it contains most of what is genuinely worth building.

What credits actually buy

CI cost is opaque in a specific way: the config file decides it and almost nobody reads the config file with cost in mind.

Resource class is the main lever. A job on a larger machine consumes credits faster, and jobs are routinely left on a size chosen when the test suite was different.

Parallelism multiplies. Splitting tests across containers reduces wall-clock time and increases total consumption, which is a trade worth making knowingly.

Re-runs double everything. A flaky test failing one run in five adds that share to the bill, plus the developer time.

Workflows that run on every branch cost more than those that run selectively, and filters are easy to add and easy to forget.

Idle waiting on a self-hosted runner is a different cost, and capacity sitting unused is still paid for.

Credit consumption by project, workflow, and job — reported weekly — usually identifies one or two jobs responsible for a disproportionate share. That report is the highest-value automation here.

What the CircleCI API reaches

Pipelines can be triggered, listed, and read with their parameters.

Workflows and jobs can be listed, inspected, cancelled, and re-run, with their timings and status.

Insights endpoints report success rates, durations, and credit consumption at workflow and job level, which is the basis of most reporting.

Test metadata is available where your jobs upload it, which is what makes flaky test detection possible.

Artifacts can be listed and downloaded.

Environment variables and contexts can be managed, which suits auditing rather than routine use.

Webhooks fire on workflow and job completion.

Project settings can be read, which supports configuration drift reporting across many repositories.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which projects, what to measure, and who should hear about it.
3. Authorize the connection with a personal or project API token with read access, since most of this reports rather than acts.
4. Describe the exceptions: a project with no recent runs, a job with no test metadata, a pipeline that never completed.
5. Run it against one project before pointing it at an organisation.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Credit consumption reporting. By project, workflow, and job, with the change week on week. The clearest saving available and read-only.

Flaky test reporting. Tests that pass on re-run without a code change, ranked by frequency. Covered below.

Duration reporting. Which jobs take longest and which are on the critical path, since the slowest job is not always the one holding everything up.

Cache effectiveness reporting. Hit rates over time, because a cache that stopped working produces slower, more expensive builds and no error.

Configuration drift across projects. Which repositories lack required workflows, use outdated orbs, or run on oversized resource classes.

Deployment records. When a pipeline deploys, record it wherever your team tracks changes, so incident investigation has a change history.

Notifying the right person on failure. Main branch failures routed to whoever broke it, with the relevant log excerpt, rather than to a channel everybody has muted.

Flaky tests, the expensive kind

Flakiness is the most expensive thing in most pipelines and the least measured.

Measure it directly. A test that failed and then passed on re-run with no code change is flaky. No instrumentation is needed beyond test metadata and run history.

Rank by frequency and by cost. A flaky test in a job that takes twenty minutes costs far more than one in a job that takes two.

Count the developer time too. Each occurrence costs a context switch and a re-run, and the total is usually larger than the credit cost.

Quarantine deliberately, not by habit. Moving a flaky test out of the blocking path is reasonable as a temporary measure with an owner and a date. Without those, quarantine is deletion with extra steps.

Report the trend. A flakiness rate going up is a sign the suite is decaying, and it is far easier to address early.

Caching, which quietly stops working

Cache keys that always miss produce a pipeline that works correctly and costs several times what it should. Nothing errors.

Keys based on a file that changes every commit never hit. This is the usual cause.

Keys that are too broad hit and restore something stale, which produces mysterious failures rather than slow builds.

Cache size and retention have limits, and a cache that grew past a threshold stops being useful.

Measure the hit rate, since it is the only way to know. A dropped hit rate after a configuration change is a fast, cheap thing to catch and an expensive thing to miss for a quarter.

Secrets and contexts, which deserve an audit

CI holds the credentials that reach everything else, and nobody reviews them after the first configuration.

Contexts are shared across projects, which is convenient and means a context restricted to nobody is available to every project in the organisation.

Restrict contexts to security groups where the platform supports it, particularly anything holding production credentials.

Environment variables set at project level are visible to every job in that project, including ones added later by somebody with different intentions.

Forked pull request builds are the classic exposure. Check whether your projects pass secrets to them, because the default behaviour is the thing to verify rather than assume.

Nothing tells you a credential is old. An inventory of contexts and variables with their age is a short piece of work and the only thing that surfaces a token created three years ago for a system that no longer exists.

Rotate and confirm. A rotation that silently breaks a nightly job is discovered days later, so check the next run rather than assuming.

Building it so it survives

Prefer a job for anything repository-shaped. The best version of many of these is a scheduled workflow rather than an external automation.

Use read-only tokens for reporting, which is what nearly all of this needs.

Handle pagination. Job, workflow, and insights listings all paginate.

Expect missing test metadata. Jobs that do not upload it produce no flakiness data, and reporting which projects those are is itself useful.

Report the outcome. Projects checked, what was found, and what was skipped.

Limits worth knowing about

Rate limits apply, and organisation-wide reporting across many projects needs pacing.

Insights data is aggregated over defined windows, so it answers trend questions rather than individual-run questions.

Test metadata requires configuration. Without it, there is no per-test data to analyse, which is the commonest reason flaky test reporting returns nothing.

Artifact retention is limited, so anything depending on old artifacts will find them gone.

Log output is large, so fetch selectively rather than across an organisation.

What to build first

Credit consumption reporting by project, workflow, and job, with the week-on-week change. It reads only, it takes little building, and it almost always identifies one job — an oversized resource class, an unfiltered workflow running on every branch — responsible for a share of the bill nobody would have approved deliberately.

Two habits make the difference. Report the change, not just the total, since a new expensive job is the thing worth acting on. And send it to the team that owns each project, because a central report of organisation-wide spend is a document and a per-team one is a decision.

Frequently asked questions

Should this be a CircleCI job or an external automation?

A job if a commit triggers it, it needs the repository, or it should block a merge. External if it reports across many runs or projects, or connects to systems CircleCI cannot see. Most CI automation belongs in the config file.

How do I find flaky tests?

Look for tests that failed and then passed on re-run with no code change, using test metadata and run history. It is a direct measure requiring no instrumentation, and it produces a ranking that makes the fixing conversation concrete.

Why is our CI bill higher than expected?

Usually resource classes larger than the jobs need, parallelism added and never revisited, workflows running on every branch, and re-runs caused by flakiness. A per-job consumption report separates these in one pass.

Why did builds get slower with no config change?

Check the cache hit rate. A cache key based on something that changes every commit never hits, and the build runs correctly while doing several minutes of avoidable work every time. Nothing reports it.

Should flaky tests be quarantined?

Temporarily, with an owner and a date. Quarantine without both is deletion with extra steps, and the coverage gap it creates is invisible because the suite still passes.

Why does flaky test reporting return nothing?

Because the jobs are not uploading test metadata. Without it there is no per-test data to analyse. Reporting which projects lack it is a useful first step in its own right.

How should build failures be notified?

Main branch failures to whoever broke it, with the relevant log excerpt, not to a shared channel. A channel receiving every failure is muted within a fortnight, at which point the notification has negative value.

Who can use our CI secrets?

More people than you think, unless contexts are restricted. An unrestricted context is available to every project in the organisation, and project-level variables are visible to every job in that project including ones added later.

What should a CI secrets audit cover?

Which contexts exist and who can use them, which variables are set per project, how old each credential is, and whether forked pull request builds receive secrets. None of it changes on its own and none of it is reported unless you ask.

Is it worth recording deployments somewhere else?

Yes. A pipeline that deploys should write a record wherever your team investigates incidents, because "what changed in the last two hours" is the first question during an outage and reconstructing it from CI history at that moment is slow.

Related reading

• GitLab and CodeWords
• Jenkins and CodeWords
• CodeWords integrations and templates