Portfolio project · Youssf Hazem
Support Inbox X-ray
Support tickets in, a one-page audit out. An LLM tags every ticket (what the customer wants, how urgent it is, whether a bot could handle it), and the tags are checked against the dataset's real labels.
Is the tagging right?
Each dot is a model's accuracy on the same 2,000 tickets; the line is its 95% range. The ranges don't overlap, so the gap between the models is real, not luck. The axis starts at 85%.
| Model | Intent accuracy | 95% range | Right category | Per 1,000 tickets | Time per 1,000 |
|---|---|---|---|---|---|
| Gemini 3.8 Flash (thinking: low) | 95.3% | 94.3%–96.2% | 99.1% | $0.14 | 87 s |
| Gemini 3.5 Flash-Lite | 91.3% | 90.0%–92.5% | 97.8% | $0.12 | 38 s |
What a bot could take
For each intent, the share of its tickets Gemini 3.8 Flash judged a bot could resolve, start, or leave to a person. No answer key exists for this part: it's the call a person reviews in a real audit. In a real inbox, weigh each row by how many tickets that intent gets.
Show as a table
| Intent | Bot can resolve | Bot starts, person finishes | Needs a person |
|---|---|---|---|
| edit_account | 100% | 0% | 0% |
| check_cancellation_fee | 100% | 0% | 0% |
| delivery_options | 100% | 0% | 0% |
| delivery_period | 100% | 0% | 0% |
| track_order | 100% | 0% | 0% |
| check_payment_methods | 100% | 0% | 0% |
| check_refund_policy | 100% | 0% | 0% |
| recover_password | 99% | 1% | 0% |
| newsletter_subscription | 99% | 1% | 0% |
| get_invoice | 96% | 4% | 0% |
| check_invoice | 95% | 5% | 0% |
| review | 95% | 0% | 5% |
| create_account | 91% | 9% | 0% |
| contact_customer_service | 91% | 0% | 9% |
| switch_account | 88% | 11% | 1% |
| delete_account | 81% | 19% | 0% |
| set_up_shipping_address | 72% | 28% | 0% |
| track_refund | 68% | 31% | 1% |
| cancel_order | 41% | 59% | 0% |
| change_shipping_address | 38% | 62% | 0% |
| place_order | 38% | 55% | 7% |
| contact_human_agent | 26% | 0% | 74% |
| get_refund | 14% | 85% | 1% |
| registration_problems | 8% | 92% | 0% |
| payment_issue | 5% | 92% | 3% |
| complaint | 3% | 0% | 97% |
| change_order | 1% | 99% | 0% |
Where the tags go wrong
Gemini 3.8 Flash's most common mix-ups (real intent → tagged as, tickets). Most are near-twin labels, not wild guesses.
track_refundtagged asget_refund21check_invoicetagged asget_invoice20create_accounttagged asregistration_problems7delivery_periodtagged astrack_order7set_up_shipping_addresstagged aschange_shipping_address6
How stable are the judgement calls?
Urgency and "could a bot do it" have no answer key, so two models judged the same 2,000 tickets. Agreement shows how stable the call is, not whether it is right.
How it works
- Sample. 2,000 tickets from a public dataset, the same count for each of its 27 intents. Seeded, so anyone can redraw the same sample.
- Tag. 25 tickets per API call. The model must answer in a fixed JSON shape with one of the 27 labels; a half-answered batch is asked again for the missing tickets.
- Score. Every tag is compared with the dataset's own label. Unanswered tickets count as wrong.
- Price. The API's own token counts × Google's published paid-tier price, failed calls included.