Databricks
Generative AI

Triaging Customer Support Tickets

Gemini labels each support ticket, the team's own rules decide where it goes, and anything unclear goes to a person.

Role: Practice project
Year: 2026
Stack: Databricks · Unity Catalog · Delta Lake · Databricks Model Serving · Databricks Secrets · MLflow · Gemini · Python · pandas

Pantry Lane is a fictional grocery and meal-kit delivery company. Every support ticket lands in one shared inbox and is read oldest first. A customer charged twice asked for a refund the same day, and her ticket waited 52 hours behind older ones. By the time billing read it, she had already disputed the charge with her bank. I built a pipeline on Databricks where Gemini reads each ticket and describes it in fixed terms, and the team’s own rules decide where it goes.

The problem

Nothing on a ticket says what it is about or how urgent it is. So the team read them in order, one at a time, at about three minutes each: more than two hours of reading for this week’s 44 tickets before anyone had been answered. Eleven of those 44 needed a reply the same day, scattered through the queue like everything else.

Two things made sorting hard:

Urgent is not loud. “Two payments came out of my account… Rent is due Friday” is calm and urgent. “WHERE IS MY BOX. It was supposed to come at 6 and it’s now 7:15” is loud and can wait. Urgency comes from what is at stake, not from capital letters.
Labels have to match exactly. When the team tried labelling by hand, one person wrote “money problem”, another “billing issue”, a third “payment”. All three were right, and a routing rule could only act on one of them. A label in free words is a note for a person. A label in fixed terms is something a rule can act on.

The approach

The model only describes the ticket. It never decides where the ticket goes. Every ticket comes out with the same four facts, each picked from a fixed list, and the team’s routing rules do the rest.

Fact	Allowed values
Category	billing, delivery, food_quality, account, other
Urgency	high, low
Sentiment	positive, neutral, negative
Summary	one short sentence

Solution architecture in six numbered stages. 01 Ingest: tickets.csv from the support inbox uploaded to a Delta table. 02 Govern: the tickets Delta table in Unity Catalog, with a secret scope keeping the Gemini API key out of code. 03 Label: a notebook that assembles a prompt from the allowed terms and each ticket, calls Gemini, parses the reply into four facts and collects one row per ticket. 04 Validate: each fact checked against its allowed terms. 05 Route: five rules checked in order, first match wins: check by hand, escalate, billing team, delivery team, normal queue. 06 Save and check: the ticket_triage Delta table and a 12-ticket check of output against expected. — AI describes each ticket; the team's rules decide where it goes.

Set up the data, the key and the prompt

The setup follows the same pattern as my complaint reply generator: the CSV uploaded into a managed Delta table under Unity Catalog, the Gemini key kept in the google_ai secret scope, and the model reached through the gemini_free serving endpoint, this time running gemini-3.5-flash-lite.

The pantry_lane_tickets table with ticket_id and ticket_text columns, rows T-1041 to T-1051, including Charged twice this week, I need a refund today; My box arrived a day late, but the food was lovely; My account password stopped working; and WHERE IS MY BOX. It was supposed to come at 6 and it's now 7:15. — 44 tickets, each just an id and the customer's own words.

Challenge 01

Teach the model what urgent means here

Left to itself, a model is likely to treat shouting as urgency, the same mistake that makes an inbox unfair. And a free-text reply like 'billing issue, quite urgent' can't be routed by any rule.

The system prompt does two jobs. It asks for raw JSON only, with no code block and no other text, using four keys with fixed values, so the reply can be read by code. And it defines urgency by what is at stake: high when the customer was charged twice or without permission, the food is unsafe, or a plan for today is ruined; low otherwise, however loudly they write.

Notebook cells: read the pantry_lane.pantry_lane_tickets table ordered by ticket_id; create an MLflow deploy client for the gemini_free endpoint; and a system prompt telling the model to reply with only raw JSON using the keys category (billing, delivery, food_quality, account or other), urgency (high when charged twice or without permission, the food is unsafe, or a plan for today is ruined, low otherwise however loudly they write), sentiment (positive, negative or neutral) and summary (one short sentence). — The definition of urgent, written once, so every ticket is judged by the same test.

Read, validate and route every ticket

One pass over every ticket: send it to Gemini, read the four facts out of the JSON reply, check them, and route.

Python loop over every ticket: client.predict sends the system prompt and the ticket text to the gemini_free endpoint, then the reply is parsed with json.loads and category, urgency, sentiment and summary are read out of it. — The model's reply is parsed as JSON, so each fact lands in its own field.

Challenge 02

Never let a label the rules don't know slip through

A model can return 'very high' or 'billing issue' instead of the allowed terms. A rule that only knows 'high' would quietly treat 'very high' as not urgent, and the ticket would sit in the normal queue.

Before any routing rule runs, the category and urgency are checked against the allowed lists. Anything off the list goes to Check by hand, so the pipeline never guesses. Then the team’s rules run in order, and the first match wins: high urgency is escalated, then billing goes to the billing team, delivery to the delivery team, and everything else to the normal queue. Because urgency is checked first, an urgent double charge is escalated before the billing rule ever sees it.

Python routing code: if the category is not in allowed_categories or the urgency is not in allowed_urgency, route is Check by hand; elif urgency is high, Escalate; elif category is billing, Billing Team; elif category is delivery, Delivery Team; else Normal queue. The ticket id, text, four facts and route are packed into one record, appended to the results, then the loop sleeps for 65 seconds. — Validation first, then the team's five rules in order. The 65-second pause keeps the run inside Gemini's free-tier limit.

Python cell creating a DataFrame from the results with ticket_id, ticket_text, category, urgency, sentiment, route and summary, and writing it in overwrite mode to pantry_lane.tickets_triage. — One triage table, one row per ticket: its words, its four facts and its route.

The triage table, rows T-1041 to T-1051. T-1041 billing, high, negative, Escalate. T-1042 delivery, low, positive, Delivery Team. T-1043 account, low, negative, Normal queue. T-1044 billing, high, negative, Escalate. T-1045 food_quality, low, positive, Normal queue. T-1046 billing, high, negative, Escalate. T-1047 delivery, high, negative, Escalate. T-1048 food_quality, high, negative, Escalate. T-1049 account, low, neutral, Normal queue. T-1050 billing, high, negative, Escalate. T-1051 delivery, low, negative, Delivery Team. Each row has a one-line summary. — The double charge (T-1041) now goes straight to Escalate. The thank-you note waits in the normal queue.

Check the results

Python cell counting how many tickets went to each route with value_counts and printing the result. — A count per route, one of the things the team asked for.

Route counts: Escalate 18, Normal queue 13, Delivery Team 10, Billing Team 3. — All 44 tickets were routed and none needed checking by hand.

Every label came back inside the allowed terms. Escalate was the largest route because the urgency rule runs before the category rules: only 3 tickets reached the billing team, because most billing tickets were marked urgent and escalated first.

But the model marked 18 tickets urgent, and the team expected 11. That gap was worth testing.

Challenge 03

Find out whether the AI's urgency can be trusted

Routing every ticket is only useful if the urgency is right. Too few urgent and a double charge waits again. Too many and the escalation queue stops meaning anything.

The team had already judged 12 tickets by hand. I compared the AI’s urgency with theirs, ticket by ticket. They agreed on 10 of the 12 (83%). Both disagreements went the same way: the AI said high where the team said low. One was T-1047, “WHERE IS MY BOX”, the loud-but-can-wait ticket the prompt was written to handle. Even with “however loudly they write” in the instructions, it was still marked urgent. The other was T-1044, “You charged the wrong card”, which the prompt’s “charged without permission” wording may have pulled in.

Neither mistake missed an urgent ticket. Both over-flagged, which matches the 18 against 11. Over-flagging is the safer way to be wrong here, but it is the next thing to tune.

Python dictionary of expected urgency for 12 tickets: T-1041 high, T-1042 low, T-1043 low, T-1044 low, T-1045 low, T-1046 high, T-1047 low, T-1049 low, T-1052 high, T-1058 high, T-1060 high, T-1071 high, followed by counters for agree, disagree and missing. — The team's own judgement on 12 tickets, used as the answer key.

Python loop over the triage results: for each ticket in the expected list, count agree if the AI's urgency matches and disagree if not, printing each disagreement and recording the expected urgency, AI urgency and result; tickets not in the list are counted as missing. Then a summary table and the three counts are printed. — Ticket by ticket, so every disagreement is named rather than buried in an average.

Output: Disagree Ticket T-1044 high low, Disagree Ticket T-1047 high low, then a table of the 12 tickets with expected urgency, AI urgency and result: agree on T-1041, T-1042, T-1043, T-1045, T-1046, T-1049, T-1052, T-1058, T-1060 and T-1071; disagree on T-1044 and T-1047, where the team said low and the AI said high. — Ten agreements, two over-flags, no urgent ticket missed.

Agree ticket count 10, disagree ticket count 2, missing ticket count 32. — The 32 'missing' are simply the tickets the team had not labelled.

Make sure allergy tickets are always escalated

A calm note about a nut allergy reads like a quiet question, but it can’t wait. The team wanted allergy tickets escalated every time, so I added a separate check that doesn’t depend on the model: a keyword search of the customer’s own words for allergy, allergic, allergen or reaction.

Python cell converting the triage table to pandas and adding an allergy_flag column that is Yes when the lowercased ticket text contains allergy, allergic, allergen or reaction, then counting the flag values. — A plain keyword check, independent of the model.

Allergy flag counts: No 43, Yes 1. — One ticket in the file mentions an allergy.

Then I cross-checked the allergy flag against the route, to confirm that ticket had actually been escalated.

Python function check_two_flag returning Both Allergy and Escalate when the route is Escalate and the allergy flag is Yes, Only Escalate, Only Allergy, or Neither, applied to every row and counted. — If an allergy ticket ever lands outside Escalate, it shows up as 'Only Allergy'.

Counts: Neither 26, Only Escalate 17, Both Allergy and Escalate 1. — The one allergy ticket was escalated. 'Only Allergy' is zero, so none slipped through.

Limits and trade-offs

One category per ticket. “Box arrived late and I was charged twice for it” is about two things, but only one category can set the route. Flagging two-topic tickets for review is still to do.
The allergy check is a keyword list. It catches “allergy” and “allergic”, but a ticket that only says “nut-free” or names a symptom could get past it. It is a safety net under the model’s urgency, not a replacement for it.
Urgency over-flags. 18 urgent against the team’s 11. The 12-ticket check is the tool for tuning the prompt and proving the change helped.
The run is slow on purpose. A 65-second pause between calls keeps it inside the free-tier limit, so 44 tickets take about 48 minutes.

What this shows

The routing rules were never the hard part. Describing every ticket the same way was. The AI is useful here because its output is held to fixed terms and checked before any rule acts on it: it describes, the team’s rules decide, and anything the rules can’t read goes to a person.