ZynoviqYour training data will never be our test. Our test will never be your training data.
Benchmarks only mean something if the AI has never seen the answers. We run training data and benchmarks from the same production system but from separate, controlled pools.
Why this matters
A system that has been trained on the questions can pass a test without being any good, like a student who saw the paper in advance. If a supplier sells you training data and then tests you on overlapping cases, the score tells you nothing. If a benchmark's answer keys reach the open web, they end up in the next generation of training data and the benchmark stops measuring anything, for you and for everyone else.
So the two products share one source of truth, the qualified professionals and the quality chain behind every case, and nothing else.
Two pools, one production system
- 01Qualified professionalsOne network, one quality chain.
- 02Certified casesVersioned, dated, tied to the law.
- 03Training poolDelivered to builders under licence.
- 04Held-out poolRun against systems; keys never delivered.
The controls
| Control | What it prevents |
|---|---|
| Separate case pools | A case written for a training dataset ever becoming a benchmark item, or the reverse |
| Separate access rights | Training contributors seeing held-out answer keys; a professional working on both sides of the same topic |
| Duplicate and near-duplicate detection across pools | The same facts, or a thin variant of them, appearing in a dataset we sold and a benchmark we run |
| Answer keys in an isolated store, readable only by the marking service | Keys leaving our systems, or staff copying them into a delivery |
| Benchmark version recorded against every result | A score that cannot be traced to the exact set of cases it was run on |
| No publication of answer keys for paid or confidential benchmarks | Keys entering the public web, and from there the next generation of training data |
| Checks that a benchmark measures the skill | A system scoring well because it memorised phrasing or formatting rather than applying the rule |
The operating rule
We never sell improvement data for a system whose benchmark score we publish. Data goes to builders. Benchmarks go to buyers. A customer may buy both a dataset and a private benchmark run on the same topics, and the held-out set is never part of the training set, but a published score and a data sale for the same system do not happen together.
Where a customer's AI is re-tested after using our data, we report a before-and-after measurement. We do not call it proof that the data worked. What we do not do.
How to check it
- Every result names the benchmark version it was run on. Evidence.
- Every dataset ships with a provenance file listing the pool each record came from. Formats, delivery and licences.
- Licences for benchmark packs forbid use in training, including through outside AI vendors. Benchmark licences.
Training and test separation: questions people ask
Do training cases and benchmark cases come from the same professionals?+
They come from the same qualified network and the same production system, but never from the same items. Access rights stop a contributor who writes training cases on a topic from seeing the held-out keys for that topic.
If we buy a dataset from you, can you still benchmark our model fairly?+
Yes. The benchmark is drawn from a held-out pool that the dataset never touched, and duplicate detection runs across both pools before every delivery and every run. The benchmark version is recorded against your result so the separation can be checked later.
Why will you not sell improvement data for a system whose score you publish?+
Because a supplier that both tunes a system and publishes its score is not a credible source for either. Data goes to builders, benchmarks go to buyers, and we keep the two on separate sides of the line. Improvement data.
How do you know a benchmark measures the skill and not memorisation?+
The same legal point is tested in different forms, with different amounts, dates and states. A system that applies the rule gets them all right; a system that recognised a phrasing does not. Where a pattern suggests memorisation, the affected cases are retired and replaced.