Forward Deployed Engineer, requirement 11 of 12
Evals and output validation
I score answers against a versioned rubric, trace failing calls to the integration that broke, and tune latency per call.
The requirementKeep integrations working, prove that AI outputs are correct, and hold latency and failure rates under control.
How I deliver it at a customer
- Version the rubricI keep every metric and its 0.0, 0.5 and 1.0 anchors in versioned config, outside the running code. Each scored result stores the rubric id and version that produced it, so a scoring change reads as a diff.
- Version the golden setThe held-out items and their per-metric expert labels live in versioned config beside the rubric. The set shipping today is labeled synthetic, and a synthetic set can never clear a readiness gate.
- Run the judgeI pass the answer and the rubric text verbatim through the model gateway and take one number per judged metric. If the judge call fails I store nulls instead of a guess, and the user request still returns.
- Debias and measureOffline, I run the same judge prompt with position swap, blind scoring and length normalization against the expert labels. I report Krippendorff alpha and Cohen kappa per metric, so a weak metric cannot hide inside an average.
- Gate the model routeI re-run the golden set whenever the judge model route changes, and intake for that track pauses first. It resumes only when agreement clears the threshold in config, and a pooled pass hiding a failing subgroup does not count.
- Trace and tuneEvery model gateway call records route, provider, model, latency and status, so I can name the integration that failed. I read pod logs in Loki through Grafana when a call errors or runs slow, then tune the route.
Tools I use
| Tool | What it does here | Status |
|---|---|---|
| Rubric scoring judge | Scores one answer against the versioned rubric anchors and returns a number per judged metric, plus a short reason. | Available |
| Model gateway call telemetry | Records route, provider, model, latency and status for every model call, so I can find the slow or failing integration. | Available |
| Golden set and fairness harness | Runs the judge offline with position swap and blind scoring, then reports agreement with the expert labels per metric. The shipped set is synthetic. | Early access |
| Eval regression gate | Re-runs the golden set when the judge model route changes and holds intake for that track until agreement clears the threshold. | Early access |
| Workers AI judge demo | Runs this page's scoring demo on a Llama model at Cloudflare, separate from the judge that runs on the platform. | Demo only |
Available means shipped and in use. Early access means built but not generally available. Roadmap means designed, not built. Demo only means it powers this public demo, not customer deployments.
I send one answer to a judge model with fixed rubric anchors, and it returns a score per metric with a short reason. This shows the shape of the check; the real run scores a versioned golden set offline, not in a browser.
Open the live demoThe other answers
LLM applications in productionEvery model call goes through one gateway, guardrails run on both sides, and each task gets a measured model choice.Prompt engineeringI write the output shape first, version the prompt, budget its tokens, and compare model replies before shipping.Retrieval-augmented generationGrounded, cited answers over the customer's own documents, with access rules applied before retrieval.AI agents and orchestrationI build agents that call your systems as registered tools, and the workflows that trigger and sequence them.Request classification and routingI build automation that classifies each inbound request, routes it by role, and holds risky changes for human approval.AI workflow orchestrationI design the steps, contracts, and data flows, then ship them as running workflows on customer systems.API and integration designI publish the API contract first, gate it with keys and scopes, then build integrations against it.Hybrid enterprise integrationI connect AI services to your identity, applications and data, then run them where your policy allows.Prototype to productionI take a prototype through generated test suites and merge review gates to a zero-downtime deploy that rolls back on failure.Approvals and exception handlingI put a person between the agent and the change, with gated tools, a pending queue, expiry, and a record.Client-facing deliveryI turn engineering work into updates that client architects, security leads and business owners can each act on.
Gus IT LLC. Contact gus@gusit.de. Claude is a product of Anthropic PBC. Cloudflare and Workers AI are trademarks of Cloudflare, Inc. Llama is a trademark of Meta Platforms, Inc. Other names are trademarks of their respective owners. Gus IT LLC is an independent engineering firm and is not affiliated with or endorsed by these companies.