GitLab and CodeWords: what belongs in CI and what does not

GitLab is unusual among the tools worth automating because it already contains a general-purpose automation engine. Anything that can run in a job can run in CI, next to the code, versioned with it, visible to everyone who works on the repository.

So the useful question is not what you can automate. It is what genuinely belongs outside the pipeline — and the answer is narrower than most integration guides suggest, which makes the things that do belong outside worth getting right.

What we'll cover

• What belongs in CI
• What belongs outside
• What the GitLab API reaches
• Connecting it to CodeWords
• Seven automations worth building
• Self-managed and SaaS are not the same
• Merge request metrics that are not vanity
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

What belongs in CI

Use the pipeline for all of this, and resist the urge to move it out.

Anything triggered by a commit or a merge request. The trigger already exists and the context is already there.

Anything needing the repository contents. A job has a checkout; an external automation has to fetch one.

Anything that should block a merge. Only a pipeline can gate, because the merge request checks are what the branch rules look at.

Anything developers should see fail. A pipeline failure is visible in the merge request. An external automation failing is visible in a log nobody opens.

Scheduled repository work — dependency checks, scans, cleanup — because scheduled pipelines exist and keep the logic beside the code.

If the work is about the code, it goes in CI. That covers most of what teams try to build externally.

What belongs outside

Three categories, and they share a property: they involve something GitLab cannot see.

Reporting across projects. GitLab's own views are per project or per group, and the question is usually across both, joined with data from elsewhere.

Connecting to systems GitLab does not know about — the CRM, the support desk, the customer record, the incident tool. A pipeline job can call out, and once it is doing so on behalf of five systems, it belongs somewhere it can be maintained.

Reacting to things that are not commits. A support ticket, a customer request, a schedule that belongs to the business rather than the repository.

That is the test. If the automation needs context from outside GitLab, it belongs outside. Otherwise it belongs in CI.

What the GitLab API reaches

Projects and groups can be listed and configured, including their settings and members.

Merge requests can be created, read, updated, approved, and merged, with their discussions and diffs.

Issues, epics, and milestones cover planning, with epics depending on tier.

Pipelines and jobs can be triggered, read, retried, and cancelled, including artifacts and logs.

Commits, branches, and tags can be read and manipulated, including protected branch configuration.

Webhooks fire on a wide range of events at project and group level, and system hooks exist on self-managed instances.

Releases can be created with assets and notes attached.

Container and package registries can be queried and cleaned, which matters because registries grow without limit.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which projects, what triggers it, and what should result.
3. Authorize the connection with a project or group access token scoped to the minimum needed, rather than a personal token belonging to an individual.
4. Describe the exceptions: a merge request already merged, a pipeline still running, a project archived since the automation was written.
5. Test against one project before pointing anything at a group.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Cross-project reporting. Merge request throughput, review latency, and pipeline reliability across every project a team owns, which no single GitLab view gives you.

Review latency reporting by reviewer. Where merge requests wait and for how long, reported as a distribution rather than an average, since the tail is the problem.

Connecting issues to customer signal. A support theme or a named account attached to the issue, so prioritisation has information support holds and engineering does not.

Release notes assembled from merge requests. What shipped, readable, drafted before somebody has to write it.

Registry and artifact cleanup reporting. Container images and packages accumulating cost, reported by project with the largest offenders named.

Pipeline reliability reporting. Which jobs fail intermittently, which are slowest, and which are the actual constraint on how fast anything ships. Flaky jobs are the most expensive and least tracked thing in most pipelines.

Configuration drift reporting. Which projects lack protected branches, required approvals, or a pipeline at all, which is how a repository ends up with no checks and nobody realising.

Self-managed and SaaS are not the same

Worth establishing early, since it affects what you can build.

Feature availability differs by tier, and some of what you may want — certain approval rules, epics, some security features — depends on the licence.

API version and behaviour vary on self-managed instances, which may be several releases behind and may lag your documentation.

System hooks exist on self-managed only, covering instance-level events that project webhooks do not.

Rate limits are configurable on self-managed and are set by your administrator rather than published, so ask rather than assume.

Network reachability differs. A self-managed instance behind a corporate network needs a route before anything else matters, and that conversation is best had first.

Merge request metrics that are not vanity

Most engineering dashboards measure things that are easy to count rather than things that matter, and GitLab's data supports better.

Time waiting for a first review is the metric teams can actually act on, and it is usually the largest share of the time between opening and merging.

Report distributions, not averages. The median review waits two hours and the worst decile waits three days, and it is the second number people experience.

Size matters more than anything. Large merge requests are reviewed slowly and reviewed badly, and reporting the size distribution starts a more productive conversation than reporting velocity.

Count re-reviews. A merge request going round several times indicates unclear requirements more often than poor code.

Attribute to the team, not the individual. These metrics describe a process. Pointed at people, they change behaviour in ways that make the numbers better and the work worse — smaller commits for their own sake, reviews rubber-stamped to clear a queue.

Report alongside what shipped, so the numbers stay connected to outcomes rather than becoming their own objective.

Building it so it survives

Prefer CI for anything repository-shaped. The best version of many of these automations is a scheduled pipeline, and reaching for an external tool first is the common mistake.

Use project or group access tokens, not a personal token. Personal tokens break when the person leaves, and they grant everything that person can do.

Handle pagination. Project, merge request, and pipeline listings all paginate, and a report built from the first page is wrong in a way that looks plausible.

Make webhook handling idempotent. Events can repeat and arrive out of order.

Report the outcome. What was checked, what changed, and what was skipped because a project was archived or inaccessible.

Limits worth knowing about

Rate limits differ between SaaS and self-managed, with the latter configured locally, so the same automation can behave differently on two instances.

Some endpoints are tier-dependent, and a call that works on one instance returns a permission error on another for licensing rather than access reasons.

Large diffs are truncated in API responses, so anything analysing changes needs to handle a partial diff rather than assume completeness.

Archived projects are read-only, and group-wide automation will meet them.

Job logs can be very large, so fetch selectively rather than retrieving them wholesale across a group.

What to build first

Pipeline reliability reporting: which jobs fail intermittently, which are slowest, and how often a re-run succeeds without any change. It is read-only, it needs one token, and flaky jobs are simultaneously the most expensive thing in most pipelines and the least measured.

Two habits make the difference. Count re-runs that pass without a code change, since that number is the clearest possible measure of flakiness. And report per job rather than per pipeline, because the aggregate hides the two jobs responsible for most of the pain.

Frequently asked questions

Should this be a CI job or an external automation?

CI if it is triggered by a commit, needs the repository, should block a merge, or should be visible to developers when it fails. External if it needs data GitLab cannot see, spans projects, or is triggered by something other than a repository event.

What token should the automation use?

A project or group access token scoped to what it needs. Personal access tokens carry an individual's full permissions and stop working when that person leaves, which is a failure that always happens at an inconvenient moment.

Why does an endpoint work on one instance and not another?

Tier or version. Self-managed instances may be several releases behind and licence tiers gate some features, so the same call can return a permission error for licensing rather than access reasons.

How do I measure flaky tests?

Count jobs that failed and then passed on re-run with no change to the code. It is a direct measure, it needs no instrumentation, and it produces a per-job ranking that makes the conversation about what to fix straightforward.

Can automation approve or merge merge requests?

Technically yes, and mostly it should not. Automated merges on a green pipeline are reasonable for narrow, well-defined cases such as dependency updates. Approval is a human judgement and automating it removes the only check the process has.

What about the container registry?

It grows without limit unless something cleans it. GitLab has cleanup policies; the useful automation is reporting what is accumulating by project so the policies get configured where they matter rather than uniformly.

Do I need webhooks or is polling enough?

Webhooks for anything reacting promptly, polling for reporting. Most of the valuable automation here is reporting on a schedule, which needs no webhook at all and is considerably simpler to operate.

Which merge request metric is worth reporting?

Time waiting for a first review, as a distribution, alongside merge request size. Both are actionable and both describe the process rather than the people. Cycle time on its own tells you something is slow without telling you where.

Should these metrics be attributed to individuals?

No. They describe a process, and pointed at people they get gamed — smaller merge requests for the sake of the number, reviews cleared without being read. Report at the team level and use the numbers to find friction rather than to rank anybody.

How do I find projects with no checks configured?

List every project in the group and report which lack protected branches, required approvals, or any pipeline at all. Repositories created in a hurry never get configured afterwards, and nothing surfaces them until something goes wrong in one.

Is there value in automating release notes?

Yes, as a draft. Assemble what merged since the last release, grouped sensibly, and let a person edit it. The assembling is the tedious part and the editing is where the judgement is.

Related reading

• GitHub and CodeWords
• CircleCI and CodeWords
• CodeWords integrations and templates