Let’s talk
FREN

The blogTake back control · Episode 4/5

Published on

Building your first sovereign AI agent: the guide for people who take action

From a blank page to an agent in production, in seven steps, with the building blocks, the budgets and the pitfalls.

Series "Take back control", Episode 4/5 · 1. Sovereign AI in Europe · 2. The AI Act explained · 3. GDPR and AI · 5. Scaling up

The moment everyone checks out

There is a precise moment when AI projects die. It isn't in the boardroom, there, everyone is enthusiastic. It isn't at the demo, there, everyone applauds. It's just after, when someone asks: "Right. So concretely, where do we start?"

Silence.

The first three episodes of this series gave you the where (sovereignty), the law (the AI Act) and the duty (GDPR). This one gives you the how. By the end of this article, you'll know exactly what to install, in what order, for what budget, and the pitfalls to avoid.

A point of vocabulary first, because it changes the architecture. A chatbot answers a question. An agent acts: it chains steps, calls tools (search, database, business API), reads your documents, and loops until it reaches a result. It's that loop, model + tools + memory, that we're going to build. Sovereignly.

The core idea: an agent is five building blocks: and sovereignty is decided on each of them

Forget the magic. An AI agent is an assembly of five building blocks, always the same:

  1. The model: the brain that reasons.
  2. The inference engine: the server that runs the brain.
  3. The knowledge: your documents, injected via RAG (Retrieval-Augmented Generation: the agent fetches the relevant passages from your documentation before answering, instead of making things up).
  4. The orchestration: the loop that chains reasoning, tool calls and actions.
  5. The interface: what your users actually see.

Here is the point this episode wants to carve in stone: your agent is only as sovereign as its least sovereign building block. A European model called through US infrastructure, or a RAG that ships your internal documents to a non-EU API to vectorise them: the chain is broken. Sovereignty isn't a building block, it's a property of the assembly.

Good news: in 2026, every building block exists in a sovereign, mature and often open-source version. Let's go.

The seven steps

Step 1: Choose a simple use case (1 week)

The temptation: start with the most impressive case. Experience says: start with the most containable one.

A good first agent ticks three boxes: internal (your staff, not your customers, a human stays in the loop), documented (it draws on a corpus you own: procedures, standard contracts, product documentation), measurable (you can count the time saved). The classics that work: an internal support assistant over your knowledge base, drafting first-pass answers to tenders, summarising case files.

Reflex from episodes 2 and 3: qualify the AI Act risk level (an internal documentation assistant = minimal risk; an agent that touches HR = high risk, save it for later) and check whether personal data enters the corpus (if so: legal basis, transparency, and probably a DPIA). This applies across the EU and the UK, whose own GDPR carries the same duties.

Step 2: Prepare the corpus (1-2 weeks)

This is the step everyone rushes and that determines 80% of the final quality. Gather the documents, remove obsolete versions (an agent that cites the 2019 procedure destroys trust in a single demo), purge unnecessary personal data (minimisation, episode 3), and structure everything: clean, dated, titled documents.

Golden rule: garbage in, garbage out, but with an agent, it's garbage in, confident and well-phrased garbage out. That's worse.

Step 3: Choose the model (1 day)

For a first agent, there's no need to reach for the biggest. The simple grid:

  • Mistral Small 4 (119B, MoE, 6.5B active, Apache 2.0): the default choice if you have a high-end GPU: multimodal, 256k context, fast thanks to its MoE (mixture of experts: only a fraction of the parameters activate per token) despite its size. For a lighter first agent (a consumer GPU, edge), a Ministral 3 (14B) does the job nicely. Enough for 90% of documentation agents.
  • Mistral Medium 3.5 (~128B): when the agent has to chain complex reasoning and many tool calls: it's purpose-built for agentic architectures (structured outputs, robustness on long instructions).
  • Ministral 3 (8B/14B): for sub-tasks (classification, routing, extraction) supporting the main model.

The Apache 2.0 licence guarantees your right to self-host, modify and exploit commercially, with no trap clause.

Step 4: Choose the infrastructure (1 week)

Reuse the logic of episode 1, applied to your agent:

  • Quick start, low-sensitivity data → sovereign API (Mistral, or European providers such as Scaleway Managed Inference / OVHcloud AI Endpoints, which serve open-source models from EU-based GPUs, at around €0.09-0.15/M input tokens). Zero infra to manage, legal sovereignty assured.
  • Sensitive data or high volumes → GPU rented in the EU (Scaleway H100 ~€2-3/h, OVHcloud, or an equivalent European provider) with the model served by vLLM, the production engine: thanks to its continuous batching, it serves far more requests in parallel on one GPU, where Ollama handles only a limited number at a time by default before queueing the rest.
  • Highest assurance requirement (health, public sector, defence) → an EU-qualified sovereign offer (for example SecNumCloud-qualified providers in France, or the equivalent national scheme in your country).

And for prototyping on your own machine: Ollama, one command and it runs. It's the ideal sandbox before you commit a single euro of cloud.

An architecture tip worth its weight in gold: all these options expose an OpenAI-compatible API. Code your agent against that standard, and you'll be able to switch inference building blocks, local, sovereign cloud, API, by changing one line (base_url). That's your anti-lock-in insurance.

Step 5: Build the RAG and orchestration (2-3 weeks)

The core of the reactor. The typical sovereign stack:

  • Vectorisation and search: an open-source embeddings model + a self-hosted vector database (Qdrant, pgvector on your PostgreSQL...). Absolute point of vigilance: the embedding must run on your side too: it's the component that reads every single one of your documents.
  • Orchestration: a framework that handles the agentic loop: LangChain, CrewAI or equivalent. Start simple: one agent, two or three tools (documentation search, possibly a read-only query against a business system).
  • Interface: self-hosted Open WebUI makes an excellent starting front end: authentication, history, direct connection to your endpoint.

Resist the temptation of the mega-agent that does everything. One agent, one scope, read-only tools to begin with. You'll widen its rights once it has proven its reliability.

Step 6: Put the guardrails in place (1 week, in parallel)

This is where episodes 2 and 3 become concrete again:

  • Logging: log every request, every response, every tool call. It's a traceability requirement (AI Act) and your best debugging tool.
  • Transparency: the interface states clearly that it's an AI, and the agent cites its sources (RAG allows this natively: demand it in the system prompt).
  • Human oversight: for anything that leaves the internal perimeter, the agent proposes, the human decides.
  • Filters: block, upstream, the sending of sensitive data to the agent if the corpus is not meant to contain any.
  • The system prompt: write down in black and white the scope, the tone, the obligation to cite, and the instruction to say "I don't know" when off-corpus. It's your first line of defence against hallucinations.

Step 7: Measure, then only then expand (ongoing)

Before opening: build a test set of 30 to 50 real questions with their correct answers, and evaluate. After opening: track three metrics, rate of correct answers (hand-sampled), real usage rate, self-reported time saved.

The success criterion of a first agent isn't "it's stunning". It's: "colleagues come back the next day."

The budget, without spin

To set the orders of magnitude for a first internal agent (adjust to your volumes):

  • Sovereign API route: a few tens to a few hundred euros a month depending on usage: the cost is dominated by the human time of setting it up (reckon on 15 to 30 person-days across all seven steps).
  • Dedicated GPU route: an on-demand H100 GPU at ~€2-3/h, i.e. ~€400-600/month for business-hours use, or €1,500-2,000/month running 24/7: worthwhile beyond a few hundred thousand requests a month (the tipping point quantified in episode 1, refined in episode 5).
  • The invisible line item: corpus preparation and evaluation. That's where the real budget goes, and it's the best-invested.

The four traps of a first agent

1. The greedy scope. "While we're at it, let's plug it into the CRM, the emails and the ERP too." No. Each added tool multiplies the risks and the obligations. A narrow scope that works beats ten ambitions that disappoint.

2. The demo mistaken for production. Ollama on the intern's laptop is not an architecture. Moving to vLLM, authentication, logs and backups is 60% of the work, plan for it from the start.

3. The hidden non-sovereign building block. Embedding via a US API, the telemetry of a framework that "phones home", a SaaS interface hosted outside the EU/UK. Audit the full chain: every byte of your documents must have a known itinerary.

4. The orphan agent. No designated owner, no maintenance budget, no corpus updates. An agent is a product, not a project: it lives, or it dies in three months.

The closing word

Four episodes to lay the foundations, one single idea: take back control. Knowing where your AI runs (episode 1), under what conditions you're allowed to use it (episode 2), what you owe to the people whose data it processes (episode 3), and now, being able to build one, block by block, without depending on anyone.

Notice what happened along the way: sovereignty, compliance and technical quality, always presented as three competing constraints, turned out to be the same move. A clean corpus is both GDPR-compliant and better for RAG. Complete logs serve the AI Act and debugging alike. An OpenAI-compatible architecture is both sovereign and reversible.

Building well is already being compliant. Being compliant is already being sovereign.

So, this first agent: internal use case, clean corpus, Mistral Small, sovereign API to start, seven steps, one quarter. You have the map.

All that's left is to walk it.

And when your agent has succeeded, the day three departments want their own and someone innocently asks whether it can read customer files, episode 5 is waiting for you: scaling up.

📖 In the same series, Episode 1: Sovereign AI in Europe (state of play, roadmap, self-hosting vs API) · Episode 2: The AI Act explained (risk pyramid, obligations, pitfalls) · Episode 3: GDPR and AI (legal bases, DPIA, data-subject rights) · Episode 5: Scaling up (users, models, machines and euros).

Main sources: Mistral AI documentation, Scaleway Managed Inference / OVHcloud AI Endpoints offers, vLLM/Ollama inference benchmarks 2026, EU Regulation 2024/1689 (AI Act), and EU/UK GDPR guidance. Prices are mid-2026 orders of magnitude: check the current pricing grids before any commitment.

← Back to the blog