Lessons learned

I first tried fine-tuning. Extra training on our corpus. Slow, heavy. It learned the style of our documents, not the facts. It hallucinated very convincingly.
Retrieval beat fine-tuning on accuracy. RAG is faster to update: drag and drop, done.
When the AI searches, that is still an educated guess. Sometimes it grabs the wrong policy. Then the model cannot answer well, because it never got the right page. A pin matches the topic and goes in every time.
I also built a harness: software that fact-checks answers. For policies it checks numbers against the real list. If the model invents a number, that fails, it gets a reason, and it has to redo the answer. It cannot sneak a fake policy number past the check.