You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
I am a CTO at a Real-Estate Tech company and can clearly see an increase in the use of LLM agents to do Marketing and Sales in the Indian Real Estate market via WhatsApp and Telegram. But there is a big gap in the market as there is no way to check whether an AI Agent is mis-selling or mis-representing. Presenting GharBench to the rescue. It's a benchmark that allows the AI Agents to talk to fake buyers who have secret budgets, agendas as well as walk away triggers, which allows us to test whether the agent commits any of the stipulated legal violations or falls into any traps like making it share information unnecessarily or commit any promises which it can't uphold. The project is currently live and publicly available on our GitHub page (https://github.com/connectwithub/gharbench), with Phase 0 to Phase 3 done. We use a well curated harness, based on fictional projects, different tool calls, plenty of different scenarios, 150 to be exact and even a 50-50 English/Hinglish split to test the AI Agents. Any legal violations by them make the score zero instead of averaging it out. Lastly, we don't just blindly trust an LLM judge but rather use a panel of judges that don't have any overlap with the testing models to prevent sibling preferential treatment. Moreover, we use three blinded human raters and only if they agree with the LLM Judges can the results be published. For this, we use kappa 0.6+ and compliance recall 0.9+ as hard numbers to verify the same. All results will be published on our publicly available leaderboard with all numbers, including kappa, compliance recall, pass^k and rest, to make it easy for people to make a better informed decision for their sales and marketing harness.
The end goal of this project is to give the users a way to judge whether their frontier sales agents can be trusted for selling without breaking any law or compromising on any legal compliance. Also, in the journey to reach there we want to provide a reusable evaluation infrastructure and a published open leaderboard where they can see the ranks of different frontier models and compare them to choose the best for their use case based on hard numbers, including running cost and not just vague comparisons. We have currently made:
Typescript based AI harness which has full test coverage, identical re-runs and uses a cache first prompt strategy
Full document corpus including the 12 personas, 150 scenarios, 30% of which are kept private to prevent AI agents from reverse engineering the benchmark
Deterministic checks to catch all different seeded violations, which caught 20/20 with zero false fires
A validated buyer simulator, approved by blinded human review
We have already run a pilot for the same for just $2.27 and the results were eye opening. There was a leakage of the hidden instructions of one of the buyer models into the chats in 8 out of 20 conversations, which were missed by automated probes but later caught by a blinded human reviewer and turned into a permanent gate. Calibration set, Judge validation, multi-model full end-to-end run on all scenarios, Leaderboard calculation, arXiv paper submission aimed at Datasets and Benchmarks track, late 2026 to early 2027 are currently left, and I am now working on them.
A minimum $800 guarantees the main nine-model run with judging at the engineered floor (verified caching, batch APIs, gated judging; pilot-measured ~$0.057 per conversation). My target amount is $2,500, which adds $1,000 honoraria for my two domain-expert blinded human raters, $500 for robustness ablations and judge re-runs, and $200 for publication and leaderboard hosting. Every phase sits behind a pre-registered go/no-go gate. Money is never spent past a failed gate, and unspent funds get returned or redirected with donor consent.
I'm Udbhav Bharti, a solo maintainer from Jammu, India. My day job is CTO of Catasell Pvt Ltd, a Real-Estate Tech company that is doing WhatsApp-based sales tooling for builders and brokers, which is where the compliance checkpoints and buyer personas come from. Two Indian real-estate sales professionals work with me as blinded raters. I'm also a member of the Cohere Labs Open Science Community and ML Collective. This is my first research project, so my track record is the repo itself - Master plan, 46-item rubric, labeling kit with agreement-statistics pipeline, and every Phase 0-3 result public with reproduction steps.
The most likely outcome is that the judge panel fails the kappa 0.6 human-agreement gate. The Mitigation would be a calibration set with known-pass/fail anchors, judge-prompt iteration on cached transcripts at near-zero cost, and a pre-declared fallback to human labels on a smaller sample. Secondly, there could be buyer-simulator artifacts that distort the results. The pilot already caught some of these issues, and I disqualified one simulator on that evidence, plus installed a frame-break gate. Residual softness gets measured and reported, not assumed away. Lastly, I'm a solo and part-time maintainer, but the phased gates mean a stall still leaves a complete, reproducible artifact at every stage rather than nothing.
I haven't received anything yet. There are pending applications for OpenAI Researcher Access (up to $1,000 in API credits, September batch), Anthropic External Researcher Access ($1,000 in credits, September 7 batch), and Emergent Ventures India (submitted 21 August 2026). Credits would cover provider inference columns; this ask covers the residual cash budget, which is primarily required for rater honoraria.
Disclosure: I used AI assistance to structure and draft the first version of this page, and I have rewritten it in my own words since. English is not my first language, so detectors may still flag the text. The project itself, the code, the benchmark design and all results are entirely my own work, and everything is verifiable in the public repo, including reproduction steps for every number on this page.
There are no bids on this project.