Together AI and CodeWords: running open models in production

Together AI runs open-weight models as a service, which puts it in a specific position: you get the choice of model that open weights allow, without operating the infrastructure that usually implies. The interface is familiar enough that switching a client library is often most of the integration work.

That ease is also the trap. The hard parts of putting a language model into production are not the API call — they are choosing a model on your own data, controlling cost as volume grows, and handling output that is confidently wrong. This page is about those.

What we'll cover

• Where the work runs
• What Together AI reaches
• Connecting it to CodeWords
• Seven automations worth building
• Choosing a model on your own data
• Cost, which grows quietly
• Output you can depend on
• Retrieval, where most of the quality comes from
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Where the work runs

Serverless inference is per-token against shared capacity, with no idle cost. It is the right default and it suits variable volume.

Dedicated endpoints reserve capacity, giving predictable latency and throughput at a cost that runs whether or not you use it. Worth it at sustained high volume or where latency variance is unacceptable.

Batch processing suits work with no latency requirement — classifying a backlog, generating embeddings over a corpus — and is usually the cheapest route for it.

Fine-tuning is available where a model needs to learn your specific task, and it is worth trying prompt changes and retrieval first, since both are faster to iterate on and easier to reverse.

The deciding questions are the same as for any inference platform: how variable is your volume, how sensitive are you to latency, and is the work interactive or can it wait.

What Together AI reaches

Chat and completion endpoints across a broad set of open models, with an interface close enough to the common standard that client libraries generally work.

Embeddings for retrieval and similarity work.

Image models where your use case needs them.

Structured output support, which is what makes a model's response parseable rather than hopeful.

Batch endpoints for high-volume asynchronous work.

Fine-tuning with your own datasets, producing a model you then serve.

Dedicated endpoints that can be created, scaled, and removed through the API, which is what makes scheduled cost control possible.

Usage reporting, which is the basis of the cost automation below.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: what the model should do, with what input, and what happens to the output.
3. Authorize the connection with an API key.
4. Describe the exceptions: an output that fails validation, a response that was truncated, a rate limit, an input longer than the model accepts.
5. Run it against a set of inputs where you already know what a good answer looks like.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Classification and routing. Incoming requests, documents, or messages categorised and sent where they belong, with anything uncertain going to a person.

Extraction into structured records. Documents and emails turned into fields another system can use, validated before it is trusted.

Batch enrichment. A backlog processed asynchronously, which is where batch endpoints are considerably cheaper than doing it one at a time.

Embedding generation and refresh. For retrieval, kept current as content changes rather than generated once and left.

Evaluation runs. A fixed set of inputs through the current model and a candidate, reported side by side, so a change is measured.

Endpoint cost control. Dedicated endpoints paused when nothing uses them, which for anything internal is most of the week.

Usage and cost reporting. Token consumption by workflow, so the economics stay visible as volume grows rather than arriving as a surprise.

Choosing a model on your own data

Benchmarks are a starting point, not an answer. They measure a distribution that is not yours.

Build an evaluation set from real inputs, with the outputs you consider correct. Fifty well-chosen examples decide more reliably than any leaderboard.

Measure cost per completed task, not per request. A cheaper model needing two attempts and a validation failure is not cheaper.

Measure latency at your input lengths, since quoted figures assume short inputs.

Check how it fails. Plausible-sounding wrong answers are far more dangerous than obvious refusals, and models differ markedly in which they produce.

Re-run the evaluation when anything changes — a model version, a prompt, a retrieval change — because all three move quality and none announces it.

Cost, which grows quietly

Input tokens usually dominate, particularly with retrieval, where the context is far longer than the question.

Retrieved context is the lever. Sending ten documents when three would do triples the input cost of every request.

Cap output length deliberately, since an unbounded response is an unbounded cost.

Retries multiply. A validation failure causing a retry doubles the cost of that request, and a flaky prompt does it constantly.

Batch what can wait. Moving non-interactive work to batch endpoints is usually the largest single saving available.

Report by workflow, not in aggregate. A total tells you the bill went up; a breakdown tells you which automation did it.

Output you can depend on

Ask for structured output and validate it against a schema. An unvalidated response is a string you are hoping about.

Validate the content, not only the shape. A correctly formatted response with an invented value passes a schema check.

Decide what happens when validation fails. Retry once, then escalate to a person — an automation that retries indefinitely on a malformed response burns money and still fails.

Detect truncation. A response cut off at the token limit can be valid-looking and incomplete, which is the quietest failure available.

Route uncertainty to people. Where the model is not confident, a person is cheaper than the consequences of a wrong answer, and the threshold is worth tuning against your evaluation set.

Log inputs and outputs for anything consequential, because when somebody questions a decision, reconstructing it afterwards is not possible without them.

Retrieval, where most of the quality comes from

For anything answering questions over your own content, the retrieval step determines the outcome far more than the model does.

Bad retrieval cannot be fixed by a better model. If the relevant passage was not fetched, no model will produce the right answer, and the wrong answer will be fluent.

Chunk with the structure of the content. Splitting by a fixed character count cuts through the middle of the thing somebody is asking about; splitting by section or paragraph usually does not.

Keep the source with the chunk, so the answer can cite where it came from, which is both useful and the fastest way to spot bad retrieval.

Refresh embeddings when content changes. An index generated once drifts away from the content it describes, and nothing reports it.

Evaluate retrieval separately from generation. Did it fetch the right passage, and did the model use it correctly, are two different questions with two different fixes.

Fewer, better passages beat more. Sending ten chunks because the budget allows costs more and often produces a worse answer than three relevant ones.

Building it so it survives

Pin the model version, so a model update does not silently change your outputs.

Retry with backoff on rate limits, which are normal at volume rather than exceptional.

Handle long inputs deliberately. Truncation at the context limit is often silent and produces answers based on part of the input.

Keep prompts in version control, so a change to behaviour has a diff and an author.

Report the outcome. Items processed, validation failures, escalations, tokens consumed, and estimated cost.

Limits worth knowing about

Rate limits apply and vary by model and account tier.

Context windows differ per model, and exceeding one truncates rather than errors in many client configurations.

Model availability changes. Open models are added and retired, so an unpinned dependency can disappear.

Licences apply to the underlying models, and serving through a provider does not change what a model's licence permits.

Batch endpoints have their own turnaround expectations, which is the trade for the lower cost.

What to build first

An evaluation run: a fixed set of real inputs with known-good outputs, executed against your current model and reported. It costs very little, it takes an afternoon, and it converts every subsequent decision — model choice, prompt change, retrieval change — from an argument into a measurement.

Two habits make the difference. Build the evaluation set from real inputs including the awkward ones, since a set of easy examples tells you nothing useful. And run it on every prompt or model change, because the whole point is catching the change that made things quietly worse.

Frequently asked questions

How do I choose between models?

Run your own inputs through the candidates and compare against outputs you consider correct. Benchmarks measure a different distribution from yours, and the model that wins on a leaderboard frequently loses on a specific task.

Where does the cost actually go?

Input tokens, usually, and especially retrieved context. Sending more documents than the answer needs multiplies the cost of every request. Capping output length and moving non-interactive work to batch are the next two levers.

Should I fine-tune?

Try prompt changes and retrieval first — both iterate in minutes and reverse cleanly. Fine-tuning is worth it when the task needs behaviour that prompting cannot reach, and it commits you to maintaining a model.

How do I stop the model returning unusable output?

Request structured output, validate against a schema, and validate the content as well as the shape. Then decide explicitly what happens on failure: one retry, then a person. Indefinite retries cost money and still fail.

Why did quality drop without any change on our side?

Check whether the model version moved. Pin it. Unpinned model identifiers give you whatever is current, and an update changes outputs with nothing in your logs to explain it.

Serverless or a dedicated endpoint?

Serverless for variable volume, which is most cases. Dedicated when volume is sustained and high or when latency variance is unacceptable — and then pause it on a schedule if it is internal, since it bills whether or not anybody uses it.

Does the provider change what a model's licence allows?

No. The licence attaches to the model, and serving it through a provider does not alter what it permits. Check it before building around a model, particularly for anything commercial.

Our answers are wrong — is it the model or the retrieval?

Evaluate them separately. Check whether the correct passage was retrieved at all; if it was not, no model change will help. Retrieval accounts for most quality problems in question-answering over your own content.

How many passages should we send?

Fewer than the context budget allows. Three relevant passages usually beat ten mixed ones, and they cost a third as much. Sending more because there is room is the commonest source of both higher cost and worse answers.

Should prompts live in version control?

Yes. A prompt is behaviour, and a change to it deserves a diff, an author, and a review like any other change. Prompts edited in a console are changes nobody can attribute when quality moves.

Related reading

• Hugging Face and CodeWords
• Replicate and CodeWords
• CodeWords integrations and templates