Multi-Agent LLM Evaluation Platform

At the heart of SIRIUS is an orchestration graph built with LangGraph. Rather than a single linear script, the evaluation is expressed as a state graph: a shared, typed state object flows through a set of nodes, each node reads from it and contributes its part, and the framework merges those contributions safely.

When a response comes in for evaluation, the graph fans out to three judge nodes that run concurrently — one per judge model. Because they have no dependencies on one another, they execute in parallel. Each judge independently produces a schema-constrained, validated verdict. Their results then fan back in to a consensus node, which combines them.

A subtle but important property lives here. Because the three judges run concurrently, their results can arrive in any order. SIRIUS assigns each judge a fixed position and sorts results by that position before any aggregation, so the outcome is identical no matter which judge happens to answer first. Consensus is deterministic with respect to arrival order — a prerequisite for repeatable scoring.

Every judge is asked for structured output — scores within fixed numeric ranges and a confidence value — and that output is validated against a strict schema before it is ever used. Requests are issued at temperature 0 to make sampling as deterministic as the provider allows.

Crucially, there are no silent fallbacks. If a judge returns something that can't be parsed or validated, SIRIUS fails loudly rather than quietly substituting a default. A score you see is a score a judge actually produced and the system actually validated.

Three judges will not always agree, and that disagreement is information, not noise. SIRIUS combines the three verdicts by majority vote and, separately, quantifies how much they agreed.

Scoring answers is one thing; catching fabricated facts is another. SIRIUS handles hallucination detection as a two-stage process.

Stage one — canonical decomposition. A response is broken down into a list of atomic factual claims. This decomposition is grounding-constrained: the system is explicitly instructed to extract only assertions actually present in the response, and never to introduce entities, names, or facts the response didn't contain. The number of claims per response is capped so verification cost stays bounded.

Stage two — panel verification. All three judges then verify the same canonical claim list. Because every judge is verifying an identical set of claims, per-claim consensus becomes well-defined: each claim gets three verdicts, and it is flagged only when at least two of the three judges agree it is false. This is genuine claim-level consensus — not one model's opinion, and not a response-level guess.

This two-stage design runs alongside the original response-level detection rather than replacing it, so the platform gains claim-by-claim attribution without disturbing its established response-level signals.

Flagging a claim as probably false is useful; proving it is better. For claims the panel flags, SIRIUS runs a ReAct agent — an agent that interleaves reasoning, action, and observation in a loop.

Unlike a single-shot prompt, the agent's next move depends on what it just observed. It can look a fact up in an external source (the Wikipedia REST API), and it can re-question the chatbot under test to check whether it contradicts itself.

A typical loop: the agent reasons about what it needs, calls a tool, reads the result, and — if that result was a dead end — reformulates its approach and tries a different angle rather than repeating itself. It continues until it can conclude, bounded by a hard step limit so it always terminates. When it concludes, it returns TRUE, FALSE, or UNSURE together with the evidence it gathered.

SIRIUS is engineered so that its own behaviour is verifiable, not just its subjects'.

The core is built around small abstract interfaces for its three main extension points — the evaluator, the chatbot connector, and the consensus strategy. Because the model layer speaks the OpenAI-compatible protocol, swapping the underlying model provider is a minimal change rather than a rewrite.

New judges, new chatbot targets, or new ways of combining verdicts can be added without rewriting the orchestration that ties them together, across 12 REST endpoints and the React frontend.