How RAG Proxy Works
The setup: proxy in the middle, search, pins, harness, brain UI, Faculty hardware. Same idea as the talks, written out.
Two layers. The Perl files on the code page are the May 2026 snapshot: proxy, brain, ingest. Production on Faculty hardware also has pins, the harness, a brain web UI, and two NVIDIA DGX Sparks. Those are not in the download.
Request flow (production)
User → chat client → RAG Proxy → local model on Faculty Sparks
|
1. Search the dental brain
2. Pin a cheat sheet if the topic matches
3. Inject the retrieved policy chunks
4. Model writes an answer
5. Harness checks policy numbers
6. Invented number? retry. Real list? pass
- A person types in a chat app (Open WebUI, or anything that already talks to a standard chat API).
- The question hits RAG Proxy, the middleman. The chat app does not do the RAG. The proxy does.
- Brain.pm searches the policy library by meaning, not exact words. “Needlestick” finds percutaneous injury.
- If the question matches a pinned topic, a cheat sheet goes in even if search grabbed the wrong file.
- The enriched prompt goes to a local model on Faculty hardware (two NVIDIA DGX Sparks). Not a cloud chat API.
- After the model talks, the harness checks cited policy numbers against the real list. A fluent fake number fails. The model has to redo the answer.
- The person sees a chatbot that knows the policies. They never wire RAG into the chat app.
The stack
Key pieces
RAG Proxy
A server that sits between the chat interface and the language model. It speaks the usual chat APIs, so you change the URL, not the app.
Every question is searched, optionally pinned, then forwarded. Streaming and non-streaming both work.
Model-name routing picks the brain: dental/… means search the dental library, then generate with the named model. Several knowledge domains can share one proxy.
Brain.pm
Ingest, chunk, embed, search. Each brain is one knowledge domain: SQLite for text, a PDL matrix for vectors, plus a system prompt.
Chunks are about two paragraphs. Each becomes a 1,024-number fingerprint via BGE-M3. A question gets the same treatment. Cosine similarity ranks the library. Top chunks come back in milliseconds.
The dental brain holds the Faculty clinic policies (94 documents in the May snapshot). Knowledge stays in that folder, so dental never answers HR.
Semantic search, not keywords
Ask “I pricked my finger on a needle.” Keyword search misses Policy H.03 because the word percutaneous is not in the question. Vector search finds it. Same meaning, different words.
That is still an educated guess. Search can grab the wrong file. That is why pins exist.
Cheat sheets the proxy injects
A pin is not a search ranking score. When a question matches a topic, the proxy pastes a short, known-good file into the prompt. Search still runs on the rest of the library.
If search missed, the model still sees the right sheet. Reranking cannot invent a document that was never retrieved. The pin is there every time.
Came after the first public code snapshot. More on pins.
Software after the model talks
The model cannot sweet-talk it. For clinic policies we keep the real list of policy numbers. If the answer cites a number that is not on that list, the check fails. The model gets a short reason and has to try again.
Same idea as perl -c: you do not trust the author because they sound sure. You run the checker.
Also: if the question names a policy id the brain does not have, the proxy tells the model not to invent one. Came after the first public code snapshot. More on the harness.
Drag-and-drop, not a rebuild
Our own web UI for the brains. Drop a markdown file, it indexes immediately. You are not rebuilding a chat app to update a policy. You can stand up another brain (clinic, IT help desk, whatever) in the same proxy.
This is in production on Faculty hardware. It is not in the three Perl files on the code page.
Transparent proxy
The chat client thinks it is talking to a normal model server. The model thinks it received a normal prompt. RAG, pins, and the harness sit in the middle.
Swap the chat app, the model, or the proxy without rewriting the others. No vendor plugin. No cloud API for clinic content.
Faculty Sparks, not a laptop demo
The May snapshot ran generation through Ollama on a Mac Studio. Production is two NVIDIA DGX Sparks in the building: proxy and embeddings on the front box, generation on the other, private interconnect.
PHIPA / FIPPA is why it stays on premises. See the changelog for the dates.
Brain folder
Each brain is a small folder. The May dental library (94 policy documents) fits in under 1 MB.
How indexing works
Sample query
View the code
First public snapshot. Pins, harness, and the brain UI are not in these files.