Most AI support bots fail the same way: they can only answer questions that happen to match words in a help article, and everything else gets a shrug.
For ai-track.app, my Laravel SaaS product, I built something different: a support system that understands the product from its own codebase and understands each user from their actual account data, then answers abstract support questions instantly with no help-article library at all. This is a walkthrough of how it works, the safeguards that make it trustworthy, and why the approach generalizes to almost any web application.
Why Most AI Support Bots Are Bad
The standard architecture is retrieval over documentation: embed your help articles, search them when a ticket arrives, and have an LLM paraphrase whatever matched. That design inherits two fatal problems:
- The docs are the ceiling. If nobody wrote an article about it, the bot has nothing. Users don’t ask questions in documentation vocabulary; they describe their situation.
- The docs are stale. Products change weekly, help centers don’t. The bot confidently describes last quarter’s UI.
The product itself is the only source of truth that’s always complete and always current. Documentation is a lossy, lagging copy of it. So answer from the product.
The Architecture
When a ticket comes in, three things combine before the LLM ever sees the question:
- Product knowledge, derived from the codebase. A distilled product brief lives in the system prompt: what every feature does, the exact rules and thresholds in the code (a chart that needs 3 days of data, prompts running once daily, tickets auto-closing after 5 idle days, which billing provider, where every setting lives). It’s maintained from the application code using AI coding tools that read the codebase, so when the product changes, regenerating it is cheap and it never drifts the way help articles do.
- The user’s full account context. At ticket time the system queries everything about this user: their organization, what they’ve configured, every resource they’ve created, how long each has been active, and whether anything on their account errored recently. The bot isn’t answering “how do charts work”, it’s answering “why is your chart empty”, with their actual data in hand.
- Ticket metadata captured automatically. The support widget records the page they were on, browser, OS and viewport, so “it looks broken” arrives with the where already attached.
All of that goes to an LLM API with the ticket text, and the model is asked for a structured response: a reply plus its own confidence rating.
Why It Can Answer Abstract Questions
Because the model reasons over the product’s actual rules and the user’s actual state, it handles exactly the tickets that keyword-matching bots can’t:
| Ticket | Doc-search bot | This system |
|---|---|---|
| “My performance chart is just empty?” | Returns the generic charts article, if one exists | Sees the account is 2 days old, knows the chart requires 3 days of data, says so and tells them when it unlocks |
| “Something failed overnight I think” | No matching keywords, escalates or hallucinates | Checks for recent errors on their resources, finds the failed run, explains it retries on the next daily cycle |
| “Why don’t I see my competitor in the list?” | Finds nothing, apologizes | Knows how competitor data accumulates from their tracked prompts, sees how many days they’ve tracked, explains the gap |
None of those answers exist in any help article. They’re derived, the way a senior support engineer with database access would derive them.
The Safeguards
Giving an LLM your product rules and a user’s data earns trust only if it’s allowed to stay silent. The safeguards matter as much as the intelligence:
- Confidence gating. The model must return a structured verdict (high, medium, or low confidence) alongside its reply, and the reply is only posted on high. Medium and low answers are discarded and a human takes the ticket. No guessed answer ever reaches a customer.
- Hard topic exclusions. Billing specifics, anything ambiguous, and anything requiring investigation are explicitly instructed to come back low-confidence, regardless of how good the answer looks.
- No invented features, no promised timelines. Written directly into the prompt contract, and the structured output makes violations easy to audit.
- Human-visible by design. Auto-replies post from a real admin account into the normal ticket thread, flip the ticket to a feedback state rather than closed, and the user can simply reply to reach a person. The bot shortens the queue; it never ends the conversation.
- Scoped context. The model sees one user’s own data for one ticket, assembled server-side at request time. Nothing cross-tenant, nothing persisted into training, nothing the user couldn’t already see in their own account.
The design question isn’t “can the AI answer?” It’s “does the AI know when it shouldn’t?” Confidence gating is the whole product.
What It Took to Build
Surprisingly little, which is the part worth internalizing. In a Laravel application this is one service class, one LLM API integration, and discipline about context:
- A context builder that gathers the user’s account state with a handful of queries
- A system prompt holding the codebase-derived product brief plus that context
- A structured-output call to a fast, cheap model (per-ticket cost is fractions of a cent)
- The confidence gate, the bot reply posting, and the status transition
No vector database, no embedding pipeline, no help-center migration, no retraining cycle when features ship. The hard part isn’t infrastructure, it’s product judgment: knowing what belongs in the product brief, which account data actually explains tickets, and where the gate must refuse.
Results in Practice
- Tickets that map to product rules or account state get a specific, personal answer in seconds, at any hour.
- Everything else lands in the human queue untouched, with no wrong answer ever posted ahead of it.
- When the product changes, the brief is regenerated from the codebase; there’s no help center to rewrite.
Applying This to Your Product
This pattern fits any application with real users and real support volume: SaaS, membership platforms, eCommerce. The ingredients are a codebase an AI can be made to understand, account data that explains most tickets, and a gate that keeps the model honest.
It’s also a good example of what AI-era product engineering looks like: the leverage wasn’t a bigger team or a new platform, it was one person who knew the codebase, the data model, and the failure modes designing the right 300 lines. That’s the work I do as a Laravel consultant building custom application features like this, and as a fractional CTO and product lead for funded startups deciding which of these capabilities are worth building at all.