Forward Deployed Engineer, how I deliver it, requirement 11 of 12
Evals and output validation
I built a versioned rubric judge for a tutoring app, and I trace failed or slow model calls to their provider.
The requirementKeep integrations working, prove that AI outputs are correct, and hold latency and failure rates under control.
How I deliver it at a customer
- Version the rubricI built this judge for a tutoring app on Luca, with tracks for reading, writing and arithmetic. Its metrics, such as reasoning and productive struggle, and their 0.0, 0.5 and 1.0 anchors live in versioned config. Each judged score is stored with its rubric id and version, and the rubric is still a draft.
- Version the golden setThe test items and a label for each metric live in versioned config beside the rubric. Today those labels are synthetic placeholders, not expert labels, so the harness never reports a track ready on this set.
- Run the judgeThe judge sends the learner's answer and the rubric anchors through the Luca model gateway and reads back one number per metric. If that call fails, the scores stay empty instead of guessed, and the scoring request still returns.
- Measure agreementApart from learner scoring, a harness runs the judge over the golden set with blind scoring and an instruction not to reward length. It computes Krippendorff alpha and Cohen kappa for each metric, and each metric must clear the target on its own. No run against expert labels exists yet.
- Gate the model routeWhen an admin reports that the judge's model route changed, the harness re-runs, including a reasoning check for each learner subgroup. Its verdict holds intake if any subgroup fails, even when the pooled result passes. Opening intake on any track is a manual config change, and no track is open today.
- Trace and tuneCalls through the model gateway record route, provider, model and status, plus latency and tokens when the call succeeds. When a call errors or runs slow, I read the pod logs in Loki through Grafana and adjust that route's model settings.
Tools I use
| Tool | What it does here | Status |
|---|---|---|
| Tutoring rubric judgeEarly access | Scores a learner's answer against the draft versioned rubric and returns a number per metric, with a short reason when the model gives one. | Early access |
| Model gateway call telemetryAvailable | Records route, provider, model and status for each call through the gateway, plus latency on success, so I can find the failing or slow provider. | Available |
| Golden set and agreement harnessEarly access | Runs the judge on the golden set, apart from learner scoring, with blind scoring and computes agreement with the labels per metric. The shipped labels are synthetic, so no real agreement has been measured. | Early access |
| Eval regression gateEarly access | Re-runs the harness, per subgroup too, when the judge's model route is reported changed, and pauses intake for tracks that fail. No track is open today. | Early access |
| Workers AI judge demoDemo only | Runs this page's demo on Llama 3.3 70B at Cloudflare, scoring six metrics from 1 to 5. It is separate from the tutoring judge. | Demo only |
Available means shipped and in use. Early access means built but not generally available. Roadmap means designed, not built. Demo only means it powers the public demo, not customer deployments.
Paste a question and an answer. The judge scores six metrics from 1 to 5 with evidence, caps grounding when no source is supplied, and makes a ship, fix first or hold call by a fixed rule.
Open the live demoThe other answers
LLM applications in productionOn Luca, model calls normally go through a gateway that holds provider keys, records each chat call and sets a model per task.Prompt engineeringI write the output shape first, measure the prompt against each model's context window, and check replies against that shape.Retrieval-augmented generationChat answers over a customer's own documents, returned with the matched document names, after a check that the caller may use that persona.AI agents and orchestrationI build agents that call your systems through registered tools, still in early access, and workflows that start on business events.Request classification and routingI build automation that classifies each inbound request, plans bounded tasks, and holds high-risk tasks for approval; the dispatcher is in early access.AI workflow orchestrationI map the steps, API contracts, and data flows between a customer's systems and the models before I build the workflow.API and integration designI publish the API contract first, gate it with keys and scopes, then build integrations against it.Hybrid enterprise integrationI connect AI services to your identity, applications and data, and run the platform on Kubernetes on Azure Local.Prototype to productionI take a prototype through generated tests and a two-approval merge gate to a production deploy that rolls back if the new pod keeps crashing or cannot start.Approvals and exception handlingIn chat, I hold each Gatekeeper tool marked destructive until the signed-in user approves it, with expiry and a record. This path is in early access.Client-facing deliveryI turn engineering work into updates that client architects, security leads and business owners can each act on.
Built by Gus Assuncao of Gus IT LLC for a Forward Deployed Engineer application. Luca is the AI platform Gus IT builds and operates. Contact gus@gusit.de. Claude is a product of Anthropic PBC. Cloudflare and Workers AI are trademarks of Cloudflare, Inc. Llama is a trademark of Meta Platforms, Inc. Other names are trademarks of their respective owners. Gus IT LLC is an independent engineering firm and is not affiliated with or endorsed by these companies.