Forward Deployed Engineer, how I deliver it, requirement 1 of 12
LLM applications in production
On Luca, model calls normally go through a gateway that holds provider keys, records each chat call and sets a model per task.
The requirementShip LLM features that are safe, measurable and cost-aware across more than one model and more than one provider.
How I deliver it at a customer
- One gateway for model callsOn Luca, the gateway holds the keys for OpenAI, Anthropic, Gemini, Mistral and Azure and also reaches local models, so apps call one interface. The general chat route can fall back to another model or provider when one fails. Pinned routes, such as the judges and bulk generation, fail rather than switch providers, and each denial is logged. Some paths still call a provider directly.
- Right-size the model per taskIn the gateway, named task routes set their own default model. The high-volume scoring judge is set to Claude Haiku, bulk catalog generation to a local model, and the mastery judge to Claude Opus. The demo measures that trade-off for a small and a large model: latency, tokens and Workers AI neurons.
- Where guardrails run todayOn Luca's public Ask service, pattern filters check both the question and the reply. A Llama Guard classifier is built into those checks, but it passes all text when no endpoint is configured or when it fails. In Luca's agent dispatcher, a failed classifier call marks the request medium risk and flags it for human approval. This demo screens the prompt and both replies with Llama Guard 3.
- Evaluate with a rubric judgeFor a tutoring product on Luca, I built a judge that scores up to four rubric metrics against versioned anchors. When a judge call fails, the score stays empty instead of being guessed. The harness computes agreement with human expert labels. The golden set it ships with is synthetic and still waits for those labels. At a customer, I would build that set from their own examples first.
- Observe in productionFor each chat call, the gateway records route, provider, model and status, plus latency and tokens when the call succeeds. Logs are shipped to Loki and read in Grafana. Alert rules fire when the service that hosts the gateway has no available pods, and when ingress 5xx errors pass 5 percent. Latency and error SLO alerts for model calls are planned, but nothing emits those metrics yet.
- Humans approve risky jobsIn the Luca job engine, build, infrastructure, data-migration and custom jobs wait for approval from a user with job write permission. That user can be the same person who submitted the job. Tests, scans, reports, docs and code-repair jobs run without approval. A rejection is saved with its reason, but the approver's name is not recorded yet. These approval gates are in early access.
Tools I use
| Tool | What it does here | Status |
|---|---|---|
| Luca gatewayAvailable | Routes model calls to OpenAI, Anthropic, Gemini, Mistral and local models, with per-task routes and a record of each chat call. | Available |
| Claude (Anthropic API)Available | Set as the model for the gateway's judge, tutoring dialogue, document intake, email drafting and terminal routes. | Available |
| Luca rubric judgeEarly access | Built for a tutoring product: up to four scored metrics, a draft rubric and a synthetic golden set, not yet checked against human labels. | Early access |
| Spend caps, approval gatesEarly access | Per-user spend caps and a cost ledger, built but off by default, plus approval on build, infrastructure and data-migration jobs. | Early access |
| Cloudflare Workers AIDemo only | Runs this public demo: Llama Guard 3, Llama 3.2 3B and Llama 3.3 70B. | Demo only |
Available means shipped and in use. Early access means built but not generally available. Roadmap means designed, not built. Demo only means it powers the public demo, not customer deployments.
Send one prompt to a small model (Llama 3.2 3B) and a large one (Llama 3.3 70B) side by side. Your input and both replies are screened by Llama Guard 3, and each column shows latency, tokens and Workers AI neurons.
Open the live demoThe other answers
Prompt engineeringI write the output shape first, measure the prompt against each model's context window, and check replies against that shape.Retrieval-augmented generationChat answers over a customer's own documents, returned with the matched document names, after a check that the caller may use that persona.AI agents and orchestrationI build agents that call your systems through registered tools, still in early access, and workflows that start on business events.Request classification and routingI build automation that classifies each inbound request, plans bounded tasks, and holds high-risk tasks for approval; the dispatcher is in early access.AI workflow orchestrationI map the steps, API contracts, and data flows between a customer's systems and the models before I build the workflow.API and integration designI publish the API contract first, gate it with keys and scopes, then build integrations against it.Hybrid enterprise integrationI connect AI services to your identity, applications and data, and run the platform on Kubernetes on Azure Local.Prototype to productionI take a prototype through generated tests and a two-approval merge gate to a production deploy that rolls back if the new pod keeps crashing or cannot start.Approvals and exception handlingIn chat, I hold each Gatekeeper tool marked destructive until the signed-in user approves it, with expiry and a record. This path is in early access.Evals and output validationI built a versioned rubric judge for a tutoring app, and I trace failed or slow model calls to their provider.Client-facing deliveryI turn engineering work into updates that client architects, security leads and business owners can each act on.
Built by Gus Assuncao of Gus IT LLC for a Forward Deployed Engineer application. Luca is the AI platform Gus IT builds and operates. Contact gus@gusit.de. Claude is a product of Anthropic PBC. Cloudflare and Workers AI are trademarks of Cloudflare, Inc. Llama is a trademark of Meta Platforms, Inc. Other names are trademarks of their respective owners. Gus IT LLC is an independent engineering firm and is not affiliated with or endorsed by these companies.