The blogTake back control · Episode 5/5
Published on
Scaling up: the guide for those whose agent succeeded
From the first agent that works to the infrastructure that holds, the users, the models, the machines and the euros.
"Take back control" series, Episode 5/5 · 1. Sovereign AI in Europe · 2. The AI Act explained · 3. GDPR and AI · 4. Building your first sovereign AI agent
The problem no one promised you
There's a precise moment when the AI projects that survived enter their second crisis. It isn't failure, Episode 4 protected you from that. It's success. Your document agent is running, colleagues come back the next day, and one morning three things happen at once: the neighbouring department wants its own, management wants to plug it into a business flow that processes a thousand documents a night, and someone asks, innocently, whether it could read customer files. Or patient records.
Your small endpoint that served twelve polite users suddenly faces a question it never had to ask: how many, on what, and where?
This episode answers all three. How many users before you change your infrastructure. Which model for which load. And local or API, with the numbers every time, because that's the question you'll be asked in the budget committee.
The core idea: it isn't humans who tip the calculation
Here's the false intuition this episode wants to dismantle: "the more users you have, the more self-hosting pays off." It's true at the margin, and false in practice, because humans consume very little.
Do the maths. On a sovereign API, the model is billed by the token, a Mistral Medium rate lands at roughly €1.50 per million tokens of input and €7 per million of output. At those prices, a human who queries an assistant a few times a day costs the change from your pocket. Scale that to a hundred colleagues and you're still in coffee-budget territory: no GPU server on earth amortises against that.
What actually tips the calculation is two things, and neither is a human:
-
Machines. A pipeline that summarises ten thousand documents every night, an agent looping autonomously, a continuous production flow. Here you no longer count in users but in hundreds of millions of tokens, and the curve eventually inverts: at very high machine volume, running your own inference becomes cheaper than paying the API per token.
-
The law. The moment identifying health data, defence secrets or professional confidentiality (legal privilege, medical confidentiality) enter the corpus, the cost question becomes secondary: hosting is prescribed (more on this below, and for some of you it's the most important passage in this episode).
Carve the rule in stone: you don't size an AI infrastructure on the number of registered users, but on the volume of tokens and the nature of the data. Everything else follows.
First question: how many concurrent users?
The second false intuition to dismantle: confusing total users with concurrent users. Two hundred colleagues each asking a few questions a day rarely adds up to more than five to ten requests at the same instant. It's that number, concurrency, that sizes the machines.
And this is where a technical distinction from Episode 4 takes on its full weight. Ollama, your sandbox, handles only a limited number of requests in parallel by default, then queues the rest: beyond a handful of simultaneous users, the wait grows and the experience dies. vLLM (or its cousins SGLang, TensorRT-LLM) does something else: continuous batching, it serves far more conversations in parallel on the same GPU, interleaving the computations. It's the difference between a filter coffee machine and a commercial espresso machine. Same water, same coffee, not the same service.
The practical grid:
| Concurrent users | What you need | Order of magnitude |
|---|---|---|
| 1 to 5 | Ollama is enough (proto, demo, pilot team) | A laptop, or a mini PC |
| 5 to 50 | vLLM on one serious GPU | A ~24B model, quantised or not |
| 50 to 200+ | vLLM on several GPUs, or a bigger model well served | The infrastructure becomes a profession |
A detail that isn't a detail: GPU memory doesn't only hold the model. Each open conversation maintains its context cache (the KV cache) there, which grows with the length of the exchanges. A document RAG with generous contexts multiplied by thirty concurrent sessions is tens of gigabytes on top of the weights. Size with headroom, or your users will discover the "out of memory" message at peak hour.
Second question: which model for which load?
Episode 4 gave you the grid for your first agent. At scale it sharpens, and a new concept enters the stage: the difference between a dense model and an MoE model (Mixture of Experts).
A dense model activates all its parameters for every word generated. An MoE only wakes a fraction, "experts" selected on the fly. Direct consequence: at equal advertised size, the MoE is much faster, because in inference the bottleneck is almost never compute but memory bandwidth, the speed at which you move the weights around. That's why a 235-billion-parameter MoE can run faster than a 128-billion dense model: it only moves around twenty billion parameters per token.
The 2026 grid, on the Mistral side (but the logic holds elsewhere):
- Mistral Small 4 (119B, MoE, 6.5B active, Apache 2.0): still the workhorse. Thanks to its MoE design (6.5B active per token), a single high-end GPU serves it with vLLM to many concurrent users. For 80% of internal agents, reaching for something bigger is a whim.
- Mistral Medium 3.5 (128B, dense, open weights): the self-hostable frontier, cut out for heavy agentic work: long reasoning, bursts of tool calls, code. But 128 billion dense parameters means a full removal of memory at every token: count on four datacentre-class GPUs to serve it properly for multiple users, or two very large GPUs in a quantised version.
- Mistral Large (MoE, hundreds of billions): datacentre or API. At the scale of an SME or mid-cap, you don't self-host it, you rent it.
The architect's reflex: only move up in model size if the evaluation demands it. Your test set from the previous episode is there for exactly that. If Small answers correctly, Small goes to production, and the GPU budget is divided by four.
The hardware aside: the siren song of "unified memory" mini PCs
You've seen them go by: those compact boxes, Mac-mini style, boasting 128 GB of "unified memory" to run big models at home, the machines built on AMD Strix Halo chips, around €1,500 to €2,500, or their equivalents. Let's say it plainly, because the question comes up in every project: these are excellent single-user machines, and poor servers.
Their memory is vast but slow (~256 GB/s, against 2,000 to 3,350 GB/s for a datacentre GPU like the H100). A large dense model generates its tokens there drop by drop, and above all these machines don't do the batching that makes multi-user serving possible. Their place is real, the developer's workstation, the lab, the sovereign POC on a corner of a desk, but it stops at the door of shared production. If someone offers to serve fifty colleagues on a mini PC, that's the demo mistaken for prod, pitfall no. 2 from Episode 4, in a pretty box.
Third question: local or API: the numbers, every time
Three routes, three cost profiles. The figures are orders of magnitude for mid-2026, to be re-checked before you commit, the memory market is in shortage and prices move.
Route 1, The sovereign API. Zero capex, zero ops, a data non-retention clause (Zero Data Retention, ZDR) to demand. Cost: proportional to usage. For a human-facing assistant, a few hundred euros a month even at a hundred users. This is the right default answer, and it stays right far longer than you'd think.
Route 2, The GPU rented in the EU/UK. A mid-range professional GPU (such as the L40S, 48 GB of VRAM) rents at around €1.50/h, an H100 (80 GB of VRAM) around €2 to €3/h with sovereign providers. Running 24/7, count on ~€1,500 to €2,000/month to serve a Small comfortably on one H100, on the order of €9,000 to €11,000/month for a Medium in full precision on four NVLink-connected H100s. Reversible, no capex, hostable on certified infrastructure. This is the route for machine-scale volumes and sensitive data.
Route 3, The purchase. You buy the GPUs outright, a serious capex, doubling again for H100-class cards. Amortised over three years, Route 3 beats Route 2 provided it runs continuously, and provided you add the line item everyone forgets: operations. Monitoring, updates, security, on-call: running GPUs in production is a real ops job that demands real ops skills, and it doesn't come for free. A GPU server is a product, not a project, the orphan agent from Episode 4 has a cousin, the orphan server, and it costs more.
The tipping table, in one line: API by default; sovereign rental when the machines pile up the tokens, or the moment the data demands it; purchase when the load is continuous, predictable, and the operations team is real.
The case that flips everything: health data (and its cousins)
Now, the passage some readers were waiting for, those whose agent has to read medical reports, patient files, data covered by confidentiality. Here, the economic logic of the three routes takes a back seat: the law chooses the architecture.
The regime differs by country, but the principle is universal across the EU and the UK. Under the GDPR (and the UK GDPR), health data is a special category demanding reinforced safeguards, and the requirement runs across the whole chain: the model, the vector database, the orchestration. Several member states add sectoral rules, in France, hosting identifying health data requires an HDS-certified ("Health Data Host") provider; the UK adds NHS Data Security and Protection Toolkit expectations; other member states have their own. Remember the core idea from Episode 4: an agent is only as sovereign as its least sovereign brick. Health version: an agent is only as compliant as its least certified brick. And the pooled APIs of model providers, including European ones, are, to date, generally not certified for these regimes. Check on the date of your project, but never presume it.
The architectures that hold up in 2026:
- Self-hosting on certified infrastructure (OVHcloud, Scaleway, Outscale, a sovereign UK-region provider, or your own certified infra): an open-weight model served by vLLM, vector database and orchestration in the same place. The royal road, the one that aligns the five bricks of Episode 4 in the same place.
- A hybrid setup with upstream pseudonymisation: a filter removes or replaces the identifiers before the text leaves for a more powerful model, and the answer is re-identified on the client side. Powerful, but the filter becomes your critical component: and your responsibility.
- API under a reinforced contract + certified host, a pattern that European medical-AI players have publicly validated: a European model under Zero Data Retention, data hosted with a certified provider. Feasible, but it's a precise contractual arrangement, not a box ticked on a form.
And don't forget Episodes 2 and 3: AI in health is classified as high-risk by the AI Act, a Data Protection Impact Assessment (DPIA) is mandatory before deployment, POC included, and the penalties stack (GDPR/UK GDPR on one side, sector-specific health law on the other). One last word of caution, the same as in every episode: this is a map, not legal advice. Qualifying your processing belongs to your DPO and your counsel.
The budget, no spin
Four typical scenarios, to fix the ideas:
- 100 employees, internal document assistant, non-sensitive data → sovereign API + ZDR: €200 to €500/month. Buy nothing.
- The same, plus a nightly pipeline for massive document processing → one professional GPU rented in the EU/UK, Small served by vLLM: ~€1,500/month, moving to a bought machine only if the load proves durable.
- Frontier agent for heavy agentic work (code, long reasoning), non-sensitive data → Medium via API to start; move to a 4-GPU rental (~€9,000–11,000/month) only if the volume proves it.
- Identifying health data, whatever the volume → self-hosting on certified infrastructure: €2,000 to €8,000/month depending on the model, plus the DPIA, plus operations. It isn't the cheapest scenario. It's the only legal one.
And always the invisible line item: continuous evaluation, corpus updates, supervision. At scale it doesn't shrink, it becomes the heart of the job.
The four pitfalls of scaling up
1. Sizing on the registered, not the concurrent. You'll buy four times too many GPUs, or you'll saturate at the first peak hour. Measure real concurrency for a month before any commitment.
2. Forgetting that context eats memory. The model fits on the GPU, the demo passes, and production falls over: thirty long-context RAG sessions have devoured the KV cache. Plan the headroom, test under load.
3. Buying the big server before measuring. Hardware evolves fast, sharply higher memory bandwidth announced for coming generations, volatile memory prices, models that shrink at equal quality. Rental is your friend until the load is proven over six months.
4. The hidden non-certified brick, the regulated version. The health cousin of pitfall no. 3 from Episode 4: everything is certified except the vector database, or except the embeddings service, or except the logs. Audit the complete chain with the same rigour as the itinerary of your bytes. In a regulated sector, a forgotten brick isn't technical debt: it's a breach.
The clip-out box: three measures, three decisions
| 📏 The measure | ⚖️ The tipping threshold | ✅ The decision it dictates |
|---|---|---|
| Token volume, machines included, not only humans | When machines, not humans, drive the load | When to leave the API. Human-scale: sovereign API + ZDR, buy nothing. Machine-scale volume: rented GPU, then bought if the load is continuous. |
| Concurrent users, not the registered | ~ 5 (out of the sandbox) · ~ 50 (out of the single GPU) | How many machines, which engine. Ollama → vLLM on one GPU → multi-GPU. And headroom for the context cache. |
| The legal sensitivity of the data, not the volume | Identifying health data, professional confidentiality | Where to host. The full chain on certified infrastructure (GDPR/UK GDPR), DPIA before deployment. Here, the law chooses, not the cost. |
The last word
Episode 4 ended on a promise: building well is already being compliant. Scale adds its corollary: measuring well is already spending well. Token volume tells you when to leave the API. Real concurrency tells you how many GPUs. The nature of the data tells you where. Three measures, three decisions, and none is taken by guesswork in a board meeting.
Notice, one last time, what happened along the way: frugality, compliance and performance turned out, once again, to be the same gesture. The smallest model that passes the evaluation is at once the cheapest, the fastest and the simplest to host sovereignly. At scale as with the first agent: sobriety isn't a constraint, it's the architecture.
So, this scale-up: measure first, the API while it's enough, a rented GPU when the machines get involved, certified infrastructure when the data demands it. You have the map, the same as always, one scale further.
All that's left is to climb.
📖 In the same series, Episode 1: Sovereign AI in Europe · Episode 2: The AI Act explained · Episode 3: GDPR and AI · Episode 4: Building your first sovereign AI agent.
Main sources: Mistral AI documentation (open weights Small/Medium), pricing grids for sovereign APIs and GPUs (Scaleway, OVHcloud), vLLM 2026 inference benchmarks, field reports on unified-memory platforms (AMD Strix Halo, 128 GB mini PCs), health-data hosting frameworks (national certification schemes such as HDS, NHS DSPT), GDPR and UK GDPR, EU Regulation 2024/1689. Prices are orders of magnitude for mid-2026, during a period of tension on the memory market: check the up-to-date grids before any commitment.