← TKF talk · Slide 8 of 18
Run 5: RAFT
Run 5 of 6.
💪 RAFT = Retrieval-Augmented Fine-Tuning. Train the model on examples where the source policy is included in the prompt. The model learns to read the doc, cite it, answer from it. 580 pairs, 411 iterations, healthiest training curve of any run. Then I asked it the same question two ways.
WITH the source policy attached:
Staff
What is the policy for releasing patient records?
RAFT + docs
Policy T.02 - Release and Transfer of Patient Records. "Patients are advised that it will take approximately five (5) business days for the duplication of clinical notations and digital films to a CD. Duplication of analog radiographs may take up to ten (10) business days."
⭐ Cited correctly.
WITHOUT the source policy attached:
Staff
What is the policy for releasing patient records?
🤖 RAFT alone I don't have the relevant documents available to answer this question. Please consult the appropriate policy documentation or contact your supervisor for guidance.
Useless without scaffolding.
THE REALIZATION: if reading the docs at runtime is what made it work, then reading is the trick. Not the training.
→ FIX FOR RUN 6: drop the document scaffolding. Push more training pairs. See if the model can recall on its own.