Databricks
Generative AI

Building a Source-Grounded Support Answer Workflow

Every answer comes from one current support note and shows that note beside it. When no note fits, the answer says so instead of guessing.

Role: Practice project
Year: 2026
Stack: Databricks · Unity Catalog · Delta Lake · Databricks AI Functions · Databricks Model Serving · Databricks Secrets · MLflow · Gemini · Python · Spark SQL

Orion Robotics is a fictional home-robot maker. Asked how long the battery lasts, three support agents gave three answers: 8 hours, 6 hours from an archived spec sheet, and “it depends”. I built a Databricks workflow that answers from current notes only, shows the source, and says “I cannot tell” when no note fits.

The problem

The 214-document library keeps old versions beside current ones, and no reply showed its source. The sample: 14 notes (3 archived) and 12 questions. Three things make it hard:

Owners and notes use different words.
Archived notes match almost as well as current ones.
A search always returns some note, even when none answers.

The approach

Support notes and owner questions go in. Archived notes are filtered out, each question is matched to its closest current note by meaning, Gemini answers from that note alone, and every answer is saved with its source and checked.

Orion Robotics architecture in seven numbered steps. 1 Support notes and owner questions: 14 notes and 12 questions. 2 Document preparation: notes structured with metadata such as status. 3 Current-note filtering: 11 current notes used, 3 archived notes filtered out. 4 Question-to-note matching: the most relevant current note found by semantic search, for example Spec sheet v3 (D01) for the battery question, with archived notes never searched. 5 LLM answer generation from the question, the retrieved note and the system instructions. 6 Answer with source: Orion runs for 8 hours on a full charge, source D01 Spec sheet v3. 7 Validation and saved output: question, answer, source and status. Below, three key behaviours: answers based on current documentation, each answer shows its source, and Cannot tell when the notes don't support an answer. — From owner question to a sourced answer: archived notes are removed before the search ever runs.

Set up the data, the key and the prompt

The setup follows the same pattern as my complaint reply generator: the CSVs uploaded into managed Delta tables under Unity Catalog, the Gemini key kept in the google_ai secret scope, and the model reached through the gemini_free serving endpoint, this time running gemini-3.5-flash-lite.

Databricks Catalog Explorer showing the orion schema in the databricksgenaibootcamp catalog with three tables: ai_answers, orion_docs and orion_questions. — Notes and questions in, answers out, all three as governed Delta tables.

Python cell importing get_deploy_client from mlflow.deployments, sleep and json, creating a Databricks deploy client for the gemini_free endpoint, and defining an ask function that sends a system message and a user message to the endpoint and returns the reply text. — One small helper, so every call to the model goes through the same path.

The system prompt is short on purpose. It tells the model to use only the notes in the message, answer in one sentence, and give one fixed reply when the notes don’t answer the question.

System message for the Orion support assistant: You answer questions from Orion robot owners. Use only the support notes in the message. Answer in one short sentence. If the notes do not answer the question, reply only with: I cannot tell from Orion's notes. — A fixed refusal sentence is easy for code to detect, which the next step relies on.

Find the note, answer from it, keep the source

Challenge 01

Never let an archived note answer a question

The archived spec sheet says 6 hours and matches the battery question almost as well as the current one. If it is ever allowed into the search, it can win.

The status filter runs inside the search itself. Each question is matched against the notes with Databricks’ ai_similarity function, which compares meaning rather than shared words, and the query only looks at rows WHERE status = 'current'. Archived notes are never ranked at all, however well they match, so they can never be the note an answer is built from.

Challenge 02

Show a source the model cannot invent

If the model is asked to cite its source, it can name a note that sounds right but isn't the one it read.

The model is never asked to name its source. The source comes straight from the search result: the doc_id of the note that was passed in. If the reply contains “cannot tell”, the source is recorded as none instead, so an answer that admits it doesn’t know never carries a note that didn’t support it.

Python loop over every question: a Spark SQL query selects doc_id, text and ai_similarity between the note text and the question from orion_docs where status is current, ordered by score descending, limit 1. The top note and the question are sent to the ask function. If the answer contains cannot tell, the source is none, otherwise the note's doc_id. Each result is appended with the question id, question, expected note, source and answer, then the loop sleeps 65 seconds. The results are written to databricksgenaibootcamp.orion.ai_answers. — Search current notes only, answer from the top match, and take the source from the search, not from the model.

Check the answers

Each question in the sample has an expected note, so every source can be checked against it.

The ai_answers table for 12 questions. Matching rows: Q02 restart, D03, hold the power button for 10 seconds; Q03 Spanish, D05; Q04 repair after 18 months, D06, yes the repair is covered; Q05 charging, D08, 3 hours on the dock; Q06 stairs, D09, Orion cannot use stairs; Q07 mopping, D10, about 60 square metres; Q08 5G router, D11, 2.4 GHz Wi-Fi only; Q09 brushes, D12. Q11 play music, expected none, source none, I cannot tell. Three rows answer I cannot tell with source none where a note was expected: Q01 battery (D01), Q10 old iPhone (D13) and Q12 French (D05). — Every answered row carries its source. Four rows say 'I cannot tell', and only one of them should have.

Python cell counting answers with no source, and comparing each answer's source with its expected note: matches are counted, and the question id and expected note of each mismatch are stored and printed. — The source on each row is compared with the note that actually answers the question.

Output: None 4, Matched 9, Not Matched 3, Not Matched IDs Q01 expected D01, Q10 expected D13, Q12 expected D05. — 9 of 12 matched. The 3 misses all answered 'I cannot tell' rather than giving a wrong answer.

Challenge 03

Be honest about what the run got wrong

The battery question, the one this whole case is about, came back 'I cannot tell from Orion's notes'. So did the old iPhone question and 'Does Orion speak French?', where the voice guide's 'English and Spanish only' answers it.

All three misses failed safe. Every answer that was given had the right source, the one question no note covers (playing music) was correctly refused, and no answer used an archived note or made something up. The cost is three owners who would be told “I cannot tell” when the notes did hold the answer.

What I can’t yet say is why each one missed. When the model refuses, the workflow records the source as none and drops the note it was shown, so the table can’t tell me whether the search picked the wrong note or the model refused a note that did answer. The French question looks like the second case: the answer needs one small step of reasoning from “English and Spanish only” to “no”. Keeping the retrieved note and its similarity score on every row, refusals included, is the next change, and it is also what the team asked for: a match score on every answer, weakest first.

Limits and trade-offs

One note per answer. Each question is answered from its single closest note. Questions that need two notes are still to do.
Nearest is not always a match. The top result is always some note. The fixed refusal sentence is what stops a weak match from becoming a confident answer.
Refusals hide their reason. Until the retrieved note is saved on every row, a miss can’t be traced to the search or to the model.
The run is slow on purpose. A 65-second pause between calls keeps it inside the free-tier limit, so 12 questions take about 13 minutes.

What this shows

A support answer is only as trustworthy as the evidence behind it. The workflow searches current notes only, takes each answer’s source from the search instead of the model, and says “I cannot tell” when the note doesn’t support an answer. On this run that meant 9 of 12 right and 3 cautious refusals, with no wrong answers, and the misses point to exactly what to change next.