Sanad — Arabic Support Assistant

Problem

An online marketplace's customers ask about returns, orders and listings in Arabic (often Egyptian dialect) and English. A support bot that invents a policy is worse than no bot: every answer has to come from the real help docs, and anything it isn't sure about has to reach a person.

Approach

Sanad answers from the help docs and policies with retrieval-augmented generation, runs fully locally on Ollama (qwen2.5 for chat, bge-m3 for embeddings), and calls the marketplace backend for live order and listing status.

• Arabic-aware retrieval: text is normalized (alef, yaa and taa-marbuta forms, diacritics, tatweel, Arabic-Indic digits) so dialect questions like ازاي أرجّع منتج؟ match the policy docs. Dense embeddings and an Arabic-aware BM25 are fused with Reciprocal Rank Fusion.
• Grounded answers: the model may use only the numbered passages and must cite them; the API returns structured sources (document, section, URL, score).
• Handoff instead of hallucination: low retrieval confidence, a HANDOFF reply, an answer with no valid citation, a failing model or backend, or a customer asking for a person all open a ticket and tell the customer its number.
• Backend tools: order and listing IDs (ORD-10023, الطلب رقم ١٠٠٢٥) are detected deterministically and answered from bilingual templates, so statuses and dates are never paraphrased by the model.

Result

A FastAPI service with an RTL-aware web chat, admin endpoints and a Docker Compose setup. All 80 tests run offline (a hash embedder and an extractive fallback replace Ollama), and sanad eval scores a 37-question golden set on retrieval hit@k, answer rate, handoff precision/recall and tool-routing accuracy.