Walk ten steps and check the map before walking a kilometre in the wrong direction.
When I started building our AI insurance concierge, I made a mistake that felt responsible at the time. I tried to make it useful across too much of the business before I showed it to anyone. Insurance support has an enormous surface area. Different universities have different plans, different enrollment processes, different benefits, different administrators, different edge cases, and different questions that appear frequently enough to matter. A student might ask about a copay, a waiver, a prescription, a claim, a provider, an ID card, a deductible, a reimbursement, or something obscure enough that nobody had thought to put it in a FAQ. If I was building an AI system to help answer those questions, it seemed reasonable that the system should understand all of that. So I kept adding things to the mental definition of “ready.” It needed more universities. More plan documents. More knowledge. Better handling of exceptions. Better retrieval. Better confidence around ambiguous questions. Better fallbacks. Every time I got closer to something I could demo, I found another reason it was not ready yet.
The work itself was moving. That was the strange part. I was building things, testing responses, organizing knowledge, thinking through retrieval, figuring out what context should be passed to the model, and considering how the concierge should interact with our support workflow. But because I was comparing what I had against the eventual system rather than the question I was trying to answer, it felt unfinished all the time. “Not ready yet” had quietly become a place for scope to hide. I was treating the prototype as though it were already a production product and judging it by production standards before anyone had even seen whether the basic experience was useful. I wanted to expose something complete, but by doing that I was delaying the exact thing that would have helped me make it more complete intelligently: feedback.
In hindsight, the obvious question should have been much smaller. Not: can we build an AI concierge that understands our entire insurance business? The question should have been: if we give the system enough context about one meaningful part of the business, can it answer support questions well enough that this direction is worth pursuing? Those are very different questions. The first one invites months of architecture, data organization, exception handling, and abstraction. The second one can be answered with a narrow slice of reality. It took me longer than it should have to realize that I did not need to prove the whole system. I needed to prove the next important assumption.
So I cut the scope down aggressively. I picked our highest-grossing university. Instead of trying to build a knowledge system that understood every plan and every university, I created a bare-minimum knowledge base for the support questions that mattered most there. I stopped worrying about how the same structure would eventually work across every possible variation. I stopped trying to account for every strange question a student might ask. I built just enough of the concierge to demonstrate the path end to end: a real question comes in, the system gets the relevant context, produces an answer, and fits into a workflow that somebody could actually use. There were still gaps. There were things I knew would need to change. Some of the implementation was intentionally narrow. But for the first time, the scope was small enough that I could finish the thing I was actually trying to evaluate.
That changed the project immediately. I could show it to leadership instead of explaining what it would eventually become. We could look at real outputs and decide whether they were useful. We could see whether the tone was right, whether the answers contained the right details, whether the workflow felt natural, and whether this was solving a problem worth continuing to solve. Questions that had been abstract suddenly became concrete. What context does the model actually need to answer this correctly? Which information belongs in the knowledge base, and which information should be retrieved dynamically? When does conversation history matter? Which answers are safe to draft automatically, and which ones need human review? Where does the model fail because the knowledge is missing, and where does it fail because the question itself is ambiguous? Which edge cases are happening in real support conversations, and which ones exist only because I can imagine them happening?
Those questions were much more useful than the ones I had been asking myself earlier. Before the prototype, I was thinking about how to represent universities, plan variations, documents, rules, exceptions, customer context, and support workflows in a generalized system. Those are legitimate architecture questions, but I was trying to answer them without enough evidence. I could have spent much longer designing a neat architecture that supported every university, every plan type, every possible knowledge source, and every category of exception. I probably could have produced convincing diagrams for it too. But premature architecture is often just speculation with cleaner diagrams. Until I saw the concierge working in one real environment, I did not actually know which differences between universities mattered structurally, which ones were just content differences, which knowledge needed to be normalized, which information needed to stay close to the original source, or which abstraction boundaries would survive contact with real usage.
That is the part of prototyping I think I had misunderstood. A prototype is not simply a smaller version of the finished product. Its job is different. A finished product exists to solve the user’s problem reliably. A prototype exists to help you discover what the product needs to become. That means the prototype should be designed around uncertainty rather than completeness. If the biggest uncertainty is whether the model can answer insurance questions accurately enough with the context you can provide, build the smallest system that lets you test that. If the uncertainty is whether support agents will trust AI-generated drafts, put rough drafts in front of them. If the uncertainty is how university-specific knowledge should be represented, implement one university deeply enough to observe the actual patterns before designing a universal schema. Building one narrow path all the way through often teaches you more than partially building ten paths, because the end-to-end version exposes the interactions between all the parts.
The narrower version also changed the speed at which I could think. When everything was in scope, every decision felt architectural. If I changed the structure of the knowledge base, I wondered whether it would work for other universities. If I changed the prompt, I wondered how it would behave across different plan types. If I added a data source, I immediately started thinking about the generalized ingestion layer that might eventually be required. Once I constrained the environment, I could make decisions locally and observe what happened. Some decisions turned out to matter. Others did not. Some assumptions survived. Others disappeared within a few test conversations. The prototype reduced the number of unknowns, and once there were fewer unknowns, the architecture became easier to reason about.
This does not mean architecture is unimportant, or that prototypes should be piles of code nobody understands. The point is about when to commit. Architecture becomes much more valuable when it is organizing things you have actually observed rather than things you are guessing might exist. After the first concierge prototype, I had a much clearer idea of the real shapes in the problem. I could see what information the model repeatedly needed, where retrieval mattered, which context was university-specific, which patterns might be shared, and where human review needed to sit in the workflow. At that point, building something more general stopped being an exercise in predicting the future and became a response to evidence.
It also became easier to accept that some prototype code might be temporary. I used to see throwing code away as evidence that I had designed badly. I do not think that is always true anymore. If a rough implementation lets you discover that an assumption is wrong in two days instead of encoding that assumption into a generalized system over two weeks, deleting the code is not necessarily waste. The code bought information. Of course, there is a difference between intentionally temporary code and careless code. I still want a prototype to be understandable enough that I can trust what I am learning from it. But I no longer expect every early implementation decision to deserve a permanent place in the eventual system.
There are also places where “prototype first” is not permission to move casually. Anything involving security, privacy, money, compliance, production customer data, or irreversible actions deserves more thought upfront. An insurance product makes that especially obvious. You do not get to treat sensitive health or personal information recklessly because something is “just a prototype.” You do not let a model make consequential actions because you want to learn faster. You do not skip access controls, data handling boundaries, or human review where they are necessary. The goal is not to remove caution. It is to avoid committing early to structures and abstractions that you do not yet have enough information to justify.
What I was really doing wrong at the beginning was confusing uncertainty with incompleteness. The system felt incomplete because it could not answer everything, so I assumed the solution was to keep expanding it. But the most important unknown had nothing to do with coverage. I still did not know whether the core experience worked. Every extra university and every extra document I added before answering that question increased the amount of work resting on an unvalidated assumption. Narrowing the scope did not make the idea less ambitious. It gave me a way to test the ambition without having to build all of it first.
There is a version of this project where I could have spent a long time designing the “right” architecture before anyone interacted with the concierge. It probably would have felt productive because architecture produces tangible things: schemas, service boundaries, pipelines, abstractions, diagrams, interfaces. But those things can create a false sense that the important decisions are being made, when the important decision might still be whether the product is useful at all. Walking ten steps and checking the map is less satisfying than planning the entire kilometre, but it is much cheaper than discovering later that you have been walking in the wrong direction.
The first useful version of the concierge did not prove that the system could cover every university or every support question. It did not prove that the initial knowledge structure would survive scale. It did not settle every decision about retrieval, memory, automation, or human review. It gave us something better than that at the time: a real system we could react to instead of an imagined one we could debate.
The prototype did not answer every question. It answered the question that mattered first: are we moving in the right direction?
And once that answer was clear, the rest became much easier.
Thanks for reading.
More about Animesh