Our AI and software practice treats LLM applications like any other production surface: they need SLOs, ownership, and evaluation—not only prompt craft. The same portfolio themes you would expect from serious MLOps apply: data boundaries, reproducibility, and monitoring that explains why a bad answer happened.
Value-driven scope. Start from workflows with measurable lift—support deflection, internal search, document drafting with human review—not from “chat with our entire knowledge base” as a science project.
Secure data handling. PII redaction, residency constraints, and tenant isolation are non-negotiable design inputs. Retrieval layers must respect authorization from source systems, not only from the UI.
Evaluation that engineers run. Golden datasets, automatic regression on prompts, and online checks for toxicity or leakage should live in CI alongside unit tests. If only researchers can run evals, they will not run at release time.
Agent-in-the-loop patterns. For high-stakes domains, pair automation with explicit human checkpoints and telemetry on override rates—those signals tell you when the model or the process is drifting.
RAG, tool use, and fine-tuning each have a place; the portfolio discipline is picking the smallest combination that meets the bar, then hardening. Fancy models rarely compensate for messy ingestion or missing observability.
Related: AI strategy & MLOps, DevOps consulting, and resources.
A worked example: gating a support-deflection feature
Take a customer support assistant that drafts responses to inbound tickets for a human agent to approve. Before this ships past a pilot, we build a golden dataset of two hundred to five hundred real tickets, hand-labeled with the response an experienced agent would have given, and split across the categories that matter most to the business—refunds, account access, and product questions tend to dominate. Every prompt or retrieval change runs against this set before merge, and a drop in match quality on any single category blocks the release, not just an aggregate score sliding a point or two.
The harder gate is the one nobody wants to build: an online check that samples a small percentage of live traffic, scores it against the same rubric asynchronously, and pages a human reviewer when the score drops below a threshold for a sustained window. This catches the failure mode offline evals cannot: a knowledge base update three weeks after launch that silently degrades the retrieval layer for one product line the golden set does not happen to cover well. We have seen this exact scenario cost a client two weeks of quietly wrong answers before a support lead noticed complaints trending up in the ticket queue.
The override telemetry matters as much as the automated score. If human reviewers are rejecting or heavily editing more than roughly fifteen percent of drafts in a category, that is a signal the model is not ready for that category regardless of what the offline eval says, and the honest move is to route that category back to fully manual handling until the gap closes.
Common pitfalls we keep seeing
The first is treating the initial golden dataset as permanent. Teams build two hundred examples at launch, gate every release against them for a year, and slowly overfit the prompt to that exact set while real user traffic drifts into phrasing and edge cases the set never covered. We refresh golden sets quarterly with a sample of real production failures, retired examples get archived rather than deleted so regressions are still catchable.
The second is conflating a helpful-sounding answer with a correct one. LLM-as-judge scoring is useful for catching tone and format regressions, but it is a poor substitute for domain-expert review on factual correctness, especially in regulated categories like billing or eligibility. We pair automated judges with a rotating sample sent to a subject-matter expert every week, specifically to catch cases where a fluent answer is confidently wrong.
The third is skipping the boring infrastructure work because the model feels like the interesting part. Prompt versioning that is not tied to a git commit, evaluation runs that are not reproducible because temperature or retrieval index version was not pinned, and dashboards that show an aggregate pass rate with no breakdown by category all show up later as the reason nobody can explain why quality quietly dropped last Tuesday.
Deciding when a gate is strict enough to ship behind
Clients often ask for a single quality threshold number, as if 95 percent accuracy were a universal bar. We push back on that framing. The right threshold depends on the cost of the two error types, and they are rarely symmetric. A support draft that is too cautious and defers to a human costs a few seconds of review time. A support draft that confidently states an incorrect refund policy costs a chargeback, a complaint, or in regulated industries a compliance finding. We set separate thresholds for false confidence versus false caution, and we set the confidence threshold considerably tighter, because the downside is asymmetric even when the aggregate accuracy numbers look similar.
We also insist on a staged rollout tied to the eval results rather than a single go/no-go gate. A feature that clears the golden set moves to five percent of live traffic with the online check active, then to twenty five percent after a week of clean online scores, then to full traffic. Each stage has its own rollback trigger, and rolling back one stage is a config change, not an incident. Teams that skip staged rollout tend to find out about a regression from a spike in support escalations rather than from their own monitoring, which is a worse way to learn the same lesson.
Finally, we write the eval gate's pass criteria into the same document that defines the feature's launch criteria, reviewed by product and engineering together before a line of prompt gets written. Retrofitting an evaluation framework onto a feature that already shipped to real users is possible, but it costs more than building the gate first, and in the meantime the feature runs without the safety net it was always going to need. Build the gate before the demo, not after the first customer complaint forces the conversation, and treat the first month of real traffic as an extension of the pilot rather than proof the feature is finished—the golden set you built from historical tickets will always miss something that only shows up once actual users start typing.