ZynoviqZynoviq✕
Build an Evaluation Mission
How it works

Six layers. No single error ships.

A case becomes a certified record only after clearing all six layers. Fail any one, and it goes back or is discarded. This page covers the chain, how professionals are calibrated before they join it, how disagreements are settled, and how confident we are in each answer.

A magnifying glass over a printed document
  1. 01Authoring by qualified professionals
  2. 02Rule-based machine checks
  3. 03Independent solving
  4. 04Review
  5. 05Adjudication
  6. 06Evidence and integrity

Layer 1: Authoring by qualified professionals

Cases are written by calibrated professionals in the domain, from a fixed template: facts, documents, question, expected answer fields. Every author's identity and credential are checked against their profession's register, and every author has passed calibration in the capability they write for. How credentials are checked.

Layer 2: Rule-based machine checks

Our platform runs fixed checks, with no AI involved: are all fields complete, are formats valid (GSTINs, state codes, amounts), does anything look like real data (real names, real registration numbers), is the case too similar to an existing one in either pool? Failures go straight back to the author.

Layer 3: Independent solving

At least two professionals solve each case without seeing each other's answers: the treatment, the posting, the legal basis and their confidence. Agreement on the answer and the basis moves the case on; anything else triggers a third solver.

Layer 4: Review

A different, higher-tier professional checks the agreed answer and its reasoning, and flags any ambiguity or missing exception.

Layer 5: Adjudication

Where solvers disagree, or a reviewer raises a problem, a senior professional decides and writes the reasoning. Genuinely contested cases are marked ambiguous and kept out of pass/fail scoring. The full routing rule is below.

Layer 6: Evidence and integrity

The legal desk ties the final answer to the governing section, rule, notification or circular, with a valid-as-of date: no citation, no certification. Integrity checks confirm the work was done by the people credited, within normal time patterns, with no signs of AI-written answers. Evidence and currency · Keeping human-certified answers human.

What each control prevents

ControlWhat it prevents
Qualified authorsUnrealistic or badly framed cases; cases written outside the author's demonstrated capability
Independent reviewOne person's error or blind spot becoming the answer
Evidence verification with valid-as-of datesAnswers not supported by the law, or supported by a rule that has since changed
Duplicate checks across poolsThe same facts appearing in a training dataset and a held-out benchmark
Rubric validationScoring rules that mark a correct alternative wrong, or let a serious error pass
AdjudicationGenuine disagreement being averaged away or decided by whoever answered last
Version controlA result that cannot be traced to the exact case and answer it was marked against
Test-set access restrictionsHeld-out answer keys reaching contributors, customers or the open web
Customer acceptance testsA delivery that meets our standard but not the specification you approved

Two reviews do not make an item 100% correct. Independent solving and review catch most individual error. High-risk items, where a wrong answer would be costly or the law is intricate, get an additional authoritative-source check, specialist adjudication or an independent calculation before release.

Calibration: credentials get you in, calibration gets you work

A qualification shows someone studied the law. It does not show they apply it accurately, consistently and to our standard. So before anyone touches a customer's case, they prove it.

  1. Calibration test.

    A timed set of 25 cases with certified answers, in the domain and at the level the professional will work at. Pass marks are set per domain and per role.

  2. Probation.

    The first 30 pieces of work are all checked by a senior reviewer. Feedback is given on every correction.

  3. Continuous measurement.

    After probation, every contribution keeps being measured: agreement with other solvers, agreement with adjudicators, reviewer corrections, and planted control cases with known answers.

Calibration decides which roles a professional can take (solver, reviewer, case writer, adjudicator), which topics they are assigned (calibration is by topic, not just by domain) and which tier they are paid at. Accuracy that falls below the standard triggers re-calibration and closer review; repeated problems remove a professional from the network and their recent work is re-checked. Before every new mission, the selected team completes a short calibration round on that mission's topics, so differences in interpretation are settled at the start. How we qualify professionals · Reliability profiles.

What we measure for each professional

MeasureWhat it shows
Calibration scoreAccuracy on known cases before starting
Agreement with final certified answersHow often their answer survives review and adjudication
Control-case accuracyAccuracy on hidden cases with known answers
Acceptance rateShare of work accepted first time
ConsistencySame answer when the same point is asked differently

These scores route work. They are never published or sold.

Adjudication: two opinions first, a senior decides when they differ

If one professional says something, it is an opinion. If two work independently and reach the same answer for the same legal reason, that is evidence. If they differ, that is a signal worth investigating, not something to average away.

  1. Two solvers answer each case independently.
  2. Same answer, same legal basis: a reviewer checks it, then on to the legal desk.
  3. Different answers, or the same answer for different reasons: a third solver answers independently.
  4. Still split: a senior adjudicator reviews all answers, decides, and writes the reasoning.
  5. Genuinely contested law, where qualified professionals would reasonably differ: the case is marked ambiguous.

Adjudicators are senior professionals with 12 or more years of practice in the domain and a strong record of agreement with final certified answers. They never settle cases they solved themselves.

What an adjudication records

Illustrative example, drawn from the synthetic canteen case on the evidence page.
ItemExample
All solver answers and reasonsBlocked (80%) · Allowed (90%) · Allowed (85%)
The decisionCredit allowed
Written reasoningThe proviso applies to legally required facilities
Edge notesRecovery from employees needs separate treatment
The tempting wrong answerTreating it as a blocked credit

The answer a capable professional nearly gave is often exactly what an AI gets wrong. We keep it in the record, so test reports can explain failures and training data can teach the difference. And consensus is not the end: even unanimous professionals can be wrong, so every final answer still goes to the legal desk. Evidence and currency.

Confidence: how sure are we, as a number with its reasons

Every certified record carries a confidence score from 0 to 100. It tells you how strongly the answer is supported, not just what the answer is. AI teams use it to weight training data; test reports use it to separate firm failures from borderline ones.

FactorWhat raises itWhat lowers it
AgreementIndependent solvers agree on answer and reasonSplit answers, or agreement for different reasons
Who agreedSolvers with strong track recordsNewer or lower-tier solvers only
Review and adjudicationClean review; clear adjudication with reasoningReviewer corrections; close adjudication calls
Legal evidenceDirect statutory text or clear circularReliance on interpretation or limited authority
CurrencyRecently verified against current lawNear a pending or recent change

The exact weighting is documented in our method note and is the same for every record.

How to read the score

  1. 0190–100Firmly settled
  2. 0275–89Well supported, with some judgment
  3. 0360–74Supported, but open to argument
  4. 04Below 60Not certified
ScoreMeaningHow we use it
90–100Firmly settledScored normally in tests; safe for training
75–89Well supported, with some judgmentScored normally; flagged in reports where relevant
60–74Supported, but open to argumentUsed with care; explained in reports
Below 60Not certifiedNot used, or moved to the ambiguous set

Bands may be refined as we gather more data. Where qualified professionals legitimately disagree and the law does not settle it, the case goes into a separate ambiguous set: valuable for testing whether an AI recognises uncertainty, but never scored as simply right or wrong. You see the score in each data record, in test reports next to each failed case, and in the quality report delivered with every dataset.

Variants from certified templates

Planned, subject to founder decision. Software may generate scenario variants (amounts, counts, dates) from a certified template by fixed rules, each spot-checked by a professional. People still write and certify every answer; a variant changes the numbers around a legal point that has already been certified, never the point itself.

Measured, not assumed

We track first-pass agreement, adjudication rates, reviewer corrections and errors found after certification, and use them to improve case writing and calibration. We plan to publish headline quality figures every quarter from mid-2027.

Quality chain: questions people ask

What are the six layers of the quality chain?+

Authoring by qualified professionals, rule-based machine checks, independent solving, review, adjudication, and evidence with integrity checks. Only cases that clear all six are certified.

Why not ask ten professionals on every case?+

Most cases do not need it, and it would add cost without adding certainty. Extra solvers are added only when a case is hard. For flagship or unusually difficult sets, larger panels are used deliberately.

Who can be an adjudicator?+

Senior professionals with 12 or more years of practice in the domain and a strong record of agreement with final certified answers. They never settle cases they solved themselves.

What score is needed to pass calibration?+

Pass marks are set per domain and per role. For solvers it is currently 80%, and 85% for reviewers. A professional's first 30 pieces of work are all checked by a senior reviewer.

Does two professionals agreeing make an answer certain?+

No. Two reviews do not make an item 100% correct. Every answer is still tied to the law by the legal desk, and high-risk items get an authoritative-source check, specialist adjudication or an independent calculation on top.

Are low-confidence answers used?+

Answers below the certification threshold are not used, or move to the ambiguous set, where they test whether an AI recognises uncertainty rather than being scored right or wrong.