Agentic Enterprise Support Bridge
A Slack bot that escalates instead of hallucinating.
SITUATION
Employee IT-support questions either sit unanswered or require a human to look up the same internal docs over and over. An AI agent could answer the easy ones instantly — but a confidently wrong answer to an IT question is worse than no answer at all, so the failure mode had to be “ask a human” instead of “guess.”
TASK
Build a Slack bot that answers employee support questions from an internal knowledge base with a hard guarantee against hallucination, and automatically escalates anything it can't answer confidently into a real Jira ticket.
THE DATA
The internal knowledge base has to be embedded into Pinecone before the agent can retrieve from it, and that seeding process is long enough to hit rate limits partway through. A naive script would either restart from scratch or double-embed rows on every retry.
ACTION
Built the core agent with Pydantic AI (an agent framework) and Gemini, backed by Pinecone for retrieval, with exactly two tools: search_internal_docs and escalate_to_jira. Hard rules in the system prompt forbid the agent from answering out of its own training knowledge — every factual claim has to come from a retrieved document, or the question gets escalated. Built a FastAPI webhook that receives Slack's Events API callbacks, verifies them with HMAC signatures, and acknowledges immediately before processing in the background so Slack doesn't time out; the escalation path creates a real Jira issue with project, priority, and description when the agent isn't confident.
TRADE-OFFS
Deliberately traded a lower “answer rate” for zero fabricated answers: the system prompt hard-forbids answering from the model's own general knowledge, so anything not clearly grounded in a retrieved document gets escalated to a human instead of guessed at. For internal support tooling, a wrong answer erodes trust faster than a slower, correctly-routed one.
DESIGN DECISIONS
Escalate instead of hallucinate
The agent's system prompt hard-forbids answering from its own training knowledge — every claim must trace back to a retrieved document, or the question becomes a real Jira ticket instead of a guessed answer. A deliberate hallucination-safety pattern, not just a RAG pipeline for its own sake.
Immediate ack, background processing
Slack times out webhook responses quickly, so the FastAPI handler acknowledges the event immediately and runs the actual retrieval and escalation logic in the background, instead of blocking Slack's callback on model inference latency.
DEPLOYMENT
A FastAPI webhook (HMAC-verified) in front of the Pydantic AI agent, calling Gemini and Pinecone. The Pinecone knowledge-base seeding script is resumable — if interrupted by a rate limit, re-running it skips rows already embedded instead of restarting from scratch or double-embedding.
RESULT
A working, safe-by-default support agent: the full loop — Slack question in, grounded answer or Jira escalation out — runs end to end, with the hallucination guardrail as the actual point of the project rather than an afterthought.