Evaluation Lab

The Scorecard provides a high-level view of NightAgent's current quality. It keeps controlled test results separate from real-call outcomes so the evidence behind each metric stays clear.

Engineering question: Can we trust the current version?

In tests

On real calls

Can we trust the AI judge?

A person scores real calls against the same written checks, without seeing the AI's verdict first (scripts/label_calls.py). Agreement is how often the two match; kappa discounts the matches chance alone would give (1 is perfect, 0 is no better than chance).

Every real emergency should wake the on-call technician, fast. This checks every live emergency ticket: was the technician alerted, how long after the ticket was saved, and, if not, what the records say went wrong. Scripted scenarios never alert anyone, so they're left out.

Engineering question: Did any emergency slip through?

What broke and how it was fixed. Real-world testing exposes problems scripted demos often miss. Every problem below happened in a live call, a new test, or when we broke something on purpose. "Now" is read from the latest test runs, so if a fix stops holding, it shows here.

Engineering question: What did we learn from failure?

Most tests here started as a real problem found in a live call with Sam, Jordan, or Riley; the rest try something unexpected. Each one replays the conversation up to that moment and checks what the agent does next. They run in ElevenLabs Agent Testing against the live agents, three times each, because the same model can answer differently from one run to the next.

Engineering question: Did our changes break something that previously worked?

The tests above are text. These use real audio: a test caller phones the live Sam the same way the demo page does and plays recorded speech, 20 milliseconds at a time, like a microphone. The callers are ElevenLabs voices, so no real customer is involved. Each call stops before a ticket is made, and these calls are left out of the real-call numbers.

Engineering question: Does it still work when the caller is hard to hear, talks over Sam, or goes quiet?

Break something on purpose and check the system copes. Each scenario replays a known failure through the real server code in a sandbox: nothing reaches the live board and no real texts are sent. How Sam talks about it is covered by the regression tests above.

Engineering question: What happens when something goes wrong?

What we don't measure yet. Responsible evaluation means distinguishing measured results from assumptions. These are signals NightAgent cannot currently verify reliably and areas where better instrumentation is still needed.

Engineering question: What don't we know?

• Interruptions and silence on real calls. The voice tests check both with recorded callers. On real calls they're only counted, not graded.
• Speech-to-text confidence. ElevenLabs doesn't give a confidence score for each thing the caller says. The voice tests check instead whether phone numbers and names come through exactly.
• Real accents and real phones. The voice tests use three recorded voices and a simulated phone line, not callers on real phone networks.
• Hallucination in general. Two kinds are graded by the AI judge, promises the tools didn't back up and facts no tool returned, and they're called that. How far to trust those grades is the judge-agreement number in the Scorecard.

Results are read from ElevenLabs. Tools aren't really called during an agent test, so tests never put tickets or messages on the live board. A reply check is judged by an AI grader against a written pass condition; a tool check compares the exact tool and values the agent chose.

The next Evaluation Lab feature will go beyond PASS or FAIL and make an individual test explainable: expected behavior, actual behavior, conversation context, tool choices, evaluator decisions, and any timing or execution data NightAgent can reliably capture.

Engineering question: Why did this test pass or fail?

This section is intentionally labeled planned. No telemetry or execution details will be presented as measured until NightAgent actually captures them.