Hugging Face and CodeWords: picking where the model runs

Hugging Face is two things that get discussed as one: a hub holding an enormous number of models and datasets, and a set of ways to run inference against them. Confusing the two is how a project ends up with a model chosen on benchmark scores and no plan for what serving it will cost.

The decision that matters is where the model runs, and there are three genuinely different answers with different cost, latency, and operational profiles.

What we'll cover

• Three places a model can run
• Licences, which actually restrict you
• What Hugging Face reaches
• Connecting it to CodeWords
• Seven automations worth building
• Choosing a model without benchmarks misleading you
• Where your data goes
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Three places a model can run

Serverless inference runs shared infrastructure with no setup and no idle cost. It is the right way to try something, and it has variable latency and cold starts, which makes it a poor fit for anything user-facing with a latency expectation.

Dedicated inference endpoints give you an instance to yourself, with predictable latency and a bill that runs whether or not you send requests. This is the usual answer for production, and the cost calculation is about utilisation rather than request volume.

Running it yourself — your own infrastructure, your own serving stack — is the cheapest at sustained high volume and the most work. It is the right answer less often than engineers think and more often than product teams expect.

The deciding questions are utilisation and latency tolerance. Intermittent, latency-tolerant work belongs on serverless. Steady, latency-sensitive work belongs on a dedicated endpoint. Very high sustained volume justifies running it yourself, and nothing else does.

Licences, which actually restrict you

Open weights do not mean unrestricted use, and this catches teams late.

Licences vary widely across the Hub, from genuinely permissive to ones with real commercial restrictions.

Some prohibit specific uses, and others restrict commercial use above a threshold or require particular attribution.

Derived models inherit constraints. Fine-tuning does not reset the licence of what you fine-tuned.

Dataset licences are separate from model licences and are frequently more restrictive.

Gated models require accepting terms and approval, which is an account action and not something an automation should work around.

Check the licence before the benchmark. Discovering a restriction after building around a model is an expensive correction, and an automation that records which models are in use with their licences is a short piece of work that prevents it.

What Hugging Face reaches

The Inference API runs models without any deployment, across text, image, audio, and other modalities.

Inference Endpoints can be created, scaled, paused, and deleted through an API, which is what makes cost automation possible.

The Hub API searches models and datasets, reads model cards, and retrieves metadata including licences and download counts.

Repositories for models, datasets, and spaces can be created and managed, including private ones.

Datasets can be uploaded, versioned, and retrieved, which is a reasonable home for evaluation sets.

Spaces host demonstration applications, useful for internal review of a model before committing to it.

Webhooks notify on repository changes, which suits tracking a model you depend on.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which model, what input, and what to do with the output.
3. Authorize the connection with a token scoped to the access the workflow needs.
4. Describe the exceptions: a cold start timing out, a rate limit, an output that fails validation.
5. Run it against a set of known inputs where you already know what good looks like.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Endpoint cost control. Pause dedicated endpoints outside the hours they are used, which for anything internal is most of the week. The clearest saving available and it is a scheduled call.

Batch processing on a schedule. Classification, extraction, embedding, or transcription over a queue of work, run when it suits rather than on demand.

Evaluation against a held-out set. Run a fixed set of inputs through the model and report results, so a model change is measured rather than assumed.

Model update tracking. Notification when a model you depend on publishes a new revision, since models are updated and your pinned version stops being the default.

Licence and usage inventory. Which models are used where, with licence and version, which is the record nobody keeps and everybody eventually needs.

Output validation. Check the shape and plausibility of what comes back before it reaches anything downstream, because a confidently wrong output propagates further than an error.

Comparison runs. The same inputs through two candidate models, reported side by side, so selection uses your data rather than a public leaderboard.

Choosing a model without benchmarks misleading you

Benchmarks measure benchmark performance. They correlate imperfectly with performance on your inputs, which is what you care about.

Build a small evaluation set from real examples, with the answers you consider correct. Fifty examples chosen well is worth more than any leaderboard position.

Measure latency at your input sizes. Latency figures are usually quoted at short inputs, and yours may not be short.

Measure cost per unit of work, not per request. A cheaper model needing two attempts is not cheaper.

Check the failure mode. How a model fails matters as much as how often — silently plausible output is far worse than an obvious refusal.

Pin the revision. A model identifier without a revision means you get whatever is current, and it changes.

Where your data goes

Worth settling early, because it is the question that stops a project after the build rather than before it.

Serverless inference sends your input to shared infrastructure, which may be acceptable and needs to be a decision rather than an oversight.

Dedicated endpoints run in a region you choose, which is usually what makes them acceptable where serverless is not.

Running it yourself keeps everything inside your boundary, which is the reason organisations with strict requirements end up there despite the operational cost.

Check what is retained and for how long against your own obligations, and record the answer alongside the model in your inventory.

Do not send what you do not need to send. Redacting identifiers before inference is frequently possible and removes the question entirely for a large class of tasks.

Say so in your processing records. An inference provider handling customer data is a processor, and the paperwork is easier to complete before deployment than afterwards.

Building it so it survives

Pin revisions. Always, for anything in production, or your results change without any deployment on your side.

Handle cold starts. Serverless inference can take a while to respond first time, and an automation with a short timeout will treat a normal cold start as a failure.

Validate outputs before using them. Structure, length, and plausibility. Model output is not guaranteed to have the shape you asked for.

Retry with backoff, since rate limits and transient loading states are normal rather than exceptional.

Report the outcome. Items processed, failures, latency, and estimated cost, so the economics stay visible as volume grows.

Limits worth knowing about

Serverless inference is rate limited and shared, with variable latency and cold starts that make it unsuitable for latency-sensitive work.

Dedicated endpoints cost while idle, which is the trade for predictability and the reason scheduled pausing matters.

Model size constrains where it can run, and a large model needs hardware that is not cheap on any of the three options.

Input length limits apply per model and truncation is often silent, which produces quietly wrong results rather than errors.

Gated models require approval, which is an account-level action and blocks an automation that assumes access.

What to build first

Endpoint cost control: pause dedicated endpoints on a schedule when nothing is using them, and resume before the working day. For an internal endpoint, that is the majority of the week, and the saving is immediate and unambiguous.

Two habits make the difference. Report what was paused and for how long, so the saving is visible and the automation is not quietly switched off by somebody who does not know what it is for. And handle the resume carefully — an endpoint starting up has its own delay, so resume before people arrive rather than when the first request fails.

Frequently asked questions

Serverless inference or a dedicated endpoint?

Serverless for trying things and for intermittent, latency-tolerant work. Dedicated for production with a latency expectation. The deciding factor is utilisation: a dedicated endpoint bills whether or not you use it, so low utilisation makes it expensive per useful request.

Can I use any model on the Hub commercially?

No. Licences vary and some carry real restrictions on commercial or specific uses, and fine-tuning inherits them. Check the licence before you evaluate the model, not after you have built around it.

Why is the first request slow?

Cold start. Serverless inference loads the model on demand, and an automation with a short timeout treats that as a failure. Allow for it, retry, and use a dedicated endpoint if predictable latency matters.

Should I pin the model revision?

Yes, for anything in production. An unpinned identifier gives you whatever is current, and models are updated, so your outputs change without any deployment on your side and with nothing in your logs to explain it.

How do I choose between two models?

Run your own inputs through both and compare against answers you consider correct. Fifty real examples beat any public benchmark, because the benchmark measures a different distribution from yours.

What is the biggest hidden cost?

An idle dedicated endpoint. It bills continuously, and an internal one is genuinely used for a small fraction of the week. Scheduled pausing is a short automation with an immediate and visible return.

Why are results worse than expected on long inputs?

Often silent truncation at the model's input limit. Nothing errors; the model simply never saw the end of your input. Check the limit and handle longer inputs deliberately rather than assuming they fit.

Is it acceptable to send customer data to serverless inference?

It depends on your obligations, and it should be a decision rather than something discovered later. A dedicated endpoint in a chosen region resolves it for many teams; redacting identifiers before inference removes the question entirely for many tasks.

How do I keep a record of which models we use?

Inventory them: model, revision, licence, where it runs, and what it processes. It is a short automation and it answers both the licensing question and the data question the first time either is asked seriously.

How do I know when a model I depend on has changed?

A webhook on the repository, or a scheduled check of the current revision against your pinned one. Models are updated regularly, and a pinned deployment drifting further from the default is worth knowing about before you next upgrade.

Is running the model ourselves cheaper?

At sustained high volume, yes, and it is a different job rather than a cheaper version of the same one. Below that threshold the operational cost outweighs the saving, and a dedicated endpoint with scheduled pausing gets most of the benefit for a fraction of the work.

Related reading

• MLflow and CodeWords
• Replicate and CodeWords
• CodeWords integrations and templates