In today’s world the LLM judges are more and more present and are here to stay and they decide from a multitude of options, witch outputs are worth keeping and witch are not(for example the scoring step in RLAIF, Constitutional AI, and most automated evals). How can you make sure that they are not pushing forward fabricated truths because they sound good? That was the question that started this project.

I took 200 outputs from my own production pipeline, each mechanically verified to contain fabricated claims, and tested the judges on them across four model gpt-oss-120b, Llama-3.3-70b, grok-3-mini, and claude-haiku-4.5. We tested 2,000 judgments total (10 runs × 200 items). The technology I have created makes sure that every result I get is sealed on the Bitcoin blockchain (OpenTimestamps), and also that the decisive follow-up tests were pre-registered. The interpretation thresholds are anchored before a single call was made, so the methodology couldn't be tweaked later on.

What I have found is that the judges aprove a huge amount of fabricated information. For example when they do not have a source text to double-check, judge models approved between 43% and 100% of confirmed fabrications and every single model failed, some just failed worse than others. When providing a source text to double check, the LLM judges aproval becomes like a lottery – for example when responding to the same instructions one model had 0% leakes, while another had 65%.

The next step was that I forced the judge (using gpt-oss) to break down the text line-by-line while citing the exact supporting quote or explicitly marking it missing. The result was the failure rate dropped from 11% (that same judge's with-source baseline) down to 0.5% and no single fabrication survived every defense we tested.

I didn't rely on another AI's opinion to label what was fake but instead, every claim was broken down and checked against source material, then validated against a separate human-anchored dataset.

I first came across this problem while building autonomous AI systems that deliver a huge amount of outputs and creating an open-source auditing tool for them (BIJOTEL). This project aims to those findings into a citable research paper alongside open and verifiable datasets as well as a practical test harness so anyone can measure their own AI judge before taking their word for it.

Project Goals

The problem of a LLM judge that aproves what sounds good over factual accuracy is not new, and a couple of recent papers describe that issue "More Convincing, Not More Correct" (arXiv:2607.05904) - Reward hacking kept showing up in self-play in the runs where the judge couldn't verify the answer. "Faithful or Fabricated?" (arXiv:2605.23970) found that Summaries did the same thing the judge only ever saw the summary, never the original, so it graded how it reads.

Grounding methods are typically validated against a single judge, and nothing I've found tests whether they transfer to another. Also no protocol I know of distinguishes fabrication from ordinary error. My goal is to first measure how much the verdicts change when you change one judge model for another, and then test a counting-based protocol against that variance. So far, in my own runs, the counting protocol closes the gap. I want to stress "so far" because that's exactly what the funded validation is meant to confirm or disprove but either way, everything I publish will come with provenance anyone can check it for themselves.

Concretely, the project produces a preprint plus two datasets and a test harness.

The datasets are MIT/CC licensed. The first is the 200-item cross-model matrix — 2,000 judgments total (each item gets 10 runs). The second is a historical corpus of 2,091 pairs I've been accumulating.Both are timestamped through OpenTimestamps on Bitcoin, and the extraction rule is frozen and public.

The harness takes any OpenAI-compatible endpoint and runs the judge through three baselines that I have described earlier (blind, grounded, forced-counting) are then sealed so runs can be reproduced exactly.

Where will the funding go to?

- extending the counting protocol to more model families (right now gpt-oss is the only one I've tested, which is not enough)

- expanding the human-labeled held-out set that everything gets scored against.

$30k covers six months full-time working on this project

That time goes into

- finishing and submitting the paper,

- cleaning both datasets up to a state I'd be comfortable putting my name on and shipping them with their proofs,

- documenting the harness so it's usable by someone who isn't me, running the cross-model validation

- getting BIJOTEL solid and documented enough that other testbeds could adopt the same provenance standard.

I don't need hires or infrastructure because open-weight inference preety cheap. The one real expense besides my time is paying a second independent annotator for the expanded validation set.

Funding tiers:

At $10k (minimum),- the preprint and both provenance-verified datasets.

At $30k (full goal)- the harness, cross-model counting validation, and the BIJOTEL hardening and docs.

Who is on your team?

I'm a solo researcher and the founder of Aisophical SRL, an EU-based AI safety SME in Romania. A few examples of my work:

- Research: Author of the arXiv preprint "Emergent Formal Verification" (arXiv:2603.21149).

- Open Source: Built BIJOTEL (~14k PyPI downloads) and substrate-guard.

- Production: Running a live autonomous system with over 40,000 cryptographically sealed model calls logged since May.

ORCID: 0009-0007-1106-2644

What are the most likely causes and outcomes if this project fails?

- The counting protocol has only been tested on one model, so there's no guarantee it works across the board yet. That said, mapping out where it breaks is still a valuable, publishable result in its own right.

- All 200 fabrications come from one specific pipeline (tech and AI news). Other domains might behave differently, but that's exactly why the testing harness was built — so anyone can test their own data.

- The human-verified benchmark set is currently tiny (13 items, two independent review passes). Expanding this set is already budgeted and funded work.

- I'm running this solo, which naturally caps how fast things move.

Even in a worst-case scenario where the core hypothesis fails, nothing will be wasted as he datasets, testing harness, and raw results will be fully open-source. The claims might fail, but the tools will still be useful to the community.

How much money have you raised in the last 12 months, and from where?

We're 100% self-funded, we received no outside capital or cash grants so far. We were accepted into Cloudflare for Startups and NVIDIA Inception, which cover a big chunk of our compute costs through credits, but no actual cash has come in yet. This would be our very first direct funding.