The blogBrick by brick · Episode 5/5
Published on
The model is the last question
Four questions to ask before opening a single piece of technical documentation. And what happens when you don't ask them.
"Brick by brick" series, Episode 5/5 · 1. The code Auditor · 2. Documented search · 3. The FHIR bridge · 4. ITFacto
Half past nine, meeting room
You're shown to a chair. Lukewarm coffee. On the wall, a six-slide deck: management has decided the company "will do AI this year". Around the table, a chief executive who believes in it, a business lead waiting to be convinced, and someone from IT who already tried this last year - the prototype is asleep somewhere in a Git repository, and everyone avoids mentioning it.
Then the question arrives. It always arrives. It arrives within the first five minutes.
"Which model would you go with?"
It took me a while to find the right answer. These days I say: I don't know yet. Not out of commercial caution, not to buy time. Because that question arrives four questions too early.
The model is a consequence. It gets chosen in an afternoon once everything else is clear, and it never gets chosen well when it isn't. What actually decides a project's fate is four questions you can ask at the very first meeting, with a whiteboard and nobody technical in the room.
Here they are, in order.
Question 1 - "How will we know it works?"
Ask it and watch what happens. First a silence, then three different answers.
The chief executive talks about time saved. The business lead says: don't let it cost me more time checking than doing it myself. IT answers: don't let it fall over on a Friday night. Three people, three definitions of success, and not one of them written down anywhere.
As long as nobody around the table can describe what a good result looks like, there is no project yet. There's an intention. That isn't the same thing, and it doesn't get funded the same way.
The remedy is almost disappointing in its simplicity: before writing a line of code, you gather thirty to fifty real cases - real questions, real records, real documents - and you write next to each one the answer you expect. An afternoon's work, often tedious, sometimes contentious, because that's where you discover that two departments don't share a definition of a good record.
It's the most useful document of the project. It will settle every disagreement of the following six months, and it turns debates of opinion into observable facts.
I learned that building my first tool, which produced nothing at all: it analysed existing code and proposed what to improve first. Impossible to do without first writing down, in black and white, what I called well-built code - and then building a corpus of 2,784 repositories so that the score meant something. Describing quality is a job in itself. It's the job that precedes all the others.
Question 2 - "What should the system do when it doesn't know?"
That one has never been asked of me spontaneously. Not once. And yet it decides whether teams will still be using the tool in three weeks.
A system that always answers is a system you must always check. Do the arithmetic on what that costs: if a user has to verify every answer against the source, they do the work twice. The tool doesn't save time, it costs time, with a smile and a nice interface.
Conversely, a system that can say "I find nothing that answers your question" makes each of its other answers credible. Refusal isn't a failure. It's what gives everything else its value.
To put that idea to the test, I placed my first complete system in the most demanding field I know: documentary search in healthcare. Ground where a plausible but wrong answer does more damage than no answer at all. The rule there was non-negotiable: every statement cites the extract that grounds it, or the system says nothing.
What holds under those conditions holds everywhere else. The reverse has never been true.
One methodological detail, because it says everything about the rest: the quality measure for that system is published as a range, from 0.322 to 0.693 depending on what you accept as a success. The upper bound was the flattering one. They travel together.
Keep the formulation for your next meeting: trust isn't earned by being right often, it's earned by being predictable.
Question 3 - "What does it need to be plugged into, and what state is that data in?"
Here is where the project's real timeline lives. Not in the technical choice - in the state of your data.
What you find on opening the files, without fail: three versions of the same procedure with no dates, a product reference list maintained in a spreadsheet by someone who left in June, two applications describing the same client under two different identifiers, exports that don't reconcile.
None of that is an artificial intelligence problem. These are problems of interoperability, shared vocabulary, plumbing. And they are the leading cause of delay in AI projects, far ahead of model performance.
I wanted to confront that in the sector that has worked hardest on the question: healthcare, with its exchange formats and its national terminologies. A field that is equipped, standardised and funded for exactly this.
The most instructive part wasn't an absence of standards: they exist and they're good. Nor was it the connecting itself, which is no great feat once the bricks are identified. It's the gap between the standard as announced and the standard as applied. Across three servers plugged into the same client, each behaved differently, and two departed from their own documentation. That gap is written down nowhere. It gets measured. And if it turns up in the best-equipped sector of all, don't bet on its absence in yours.
The good news, and it's real: that work is worth doing even if the AI project stops. Data that's clean, dated and identified is an asset. You won't have done it for nothing.
Question 4 - "Who will use it, and what does it change in their work?"
This question maps onto no technical brick. It cuts across all of them.
I learned it on the only one of my projects that had users, an executive team and adoption to win: twenty months on an apprenticeship contract with a media and B2B marketing company, from first prototype to five applications that teams open in the morning. That's where I understood something demonstrators cannot teach.
A system handed to a team that nobody opens costs twice. First the work already done. Then the appetite to try again, and that second cost is paid by projects that haven't started yet.
People almost never refuse a tool. They refuse having someone else decide how they work. The difference between the two is settled at kickoff: did the people concerned take part in defining the good result from question 1, or is it presented to them at delivery?
So you spend time in the offices before spending it in front of a screen. You listen to how the work really gets done, including what appears in no procedure. You spot what actually slows people down. And you choose the first use case there, not in a steering committee.
You answer this question at kickoff, or you discover it at delivery. The price is not the same.
So, what about the model?
It comes now, and it comes fast.
Once the four answers are on the table, the technical decisions follow almost mechanically. Data sensitivity says where to host. Volume says whether you need a simple interface or an infrastructure. How demanding the evaluation is says what size of model is needed - and the answer is "the small one is enough" more often than people expect.
The reverse doesn't work. No technical feat rescues a project whose four answers are missing. You get an impressive demonstration, congratulations in committee, and a Git repository that sleeps.
Last word
People sometimes ask what I like about this job. It isn't the technical part - that part is fascinating, but it's the most documented, the best equipped, the one where you're least alone.
What I like is the translation. In both directions.
On one side, turning a vague need, expressed in the words of a trade I don't yet know, into something precise enough for a machine to honour - and precise enough that we can say whether it did. On the other, bringing the system's real limits back to the teams, honestly, so they know when to trust it and when to open the file themselves.
A useful AI system is almost always that: a faithful translation, resting on clean data, whose accuracy has been measured and whose silences have been accepted.
So the next time someone asks you, five minutes in, which model you'd go with, you have your answer.
Good question. Let's keep it for the end.
📖 In the same series - Episode 1: the code Auditor (describe quality before producing it) · Episode 2: documented search (an answer that cites its sources, or says nothing) · Episode 3: the FHIR bridge (plugging knowledge into the real world) · Episode 4: ITFacto (twenty months, five applications, real users).
The three systems described in this series are online and public. I design traceable, verifiable AI systems, from use case to deployment - more about my background, or write to contact@lamoulinette.ai.