You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$20,000 left to reach the funding goal.
Project summary
Current frontier-level VLM benchmarking is vulnerable to "cheating": many models beat static tests via visual heuristics and break down under perturbations. Benchmarking to fixed lengths further leads to large numbers of unnecessary API calls.
As mentioned above, I developed VLM-TRACE and made this publicly available. In the following 6 months, I will extend this by integrating the anytime-valid sequence-based hypothesis testing described in my IEEE ICICAIEECE 2027 publication. The output will be a pip package and Inspect AI plugin. This combination will allow benchmarking for safety erosion and cheating with dramatically reduced API usage.
What are this project's goals? How will you achieve them?
VLM - TRACE REPO: REPO LINK
I created a public repo implementing the E-Audit algorithm from my publication. Next, I will apply VLM-TRACE to 4 different VLMs (all supporting English, Hindi, Tamil, Malayalam) and aggregate using a robust scheme. The corresponding solver and benchmark will then be distributed as an Inspect AI plugin. As a final step, I will run some lightweight spot-checks against state-of-the-art models via their respective platforms, terminating once sufficient data has been collected, and report back.
How will you know it worked?
Achieve ~40% reduction in required API calls compared to current practice of benchmarking to fixed length.
An open-source MIT-licensed repo containing the E-Audit and VLM-TRACE code will successfully complete automated tests. My contribution will be merged to [inspect_ai].
Corresponding evaluation outputs and summary statistics will be uploaded to the HuggingFace Hub.
Associated preprint will be posted to arXiv and submitted for consideration at NeurIPS 2027.
How will this funding be used?
Total: $20,000 across 6 months.
Stipend to self: $10,200 ($1,700 per month for 6 months, based in Trivandrum, India). This will also provide back-dated recognition for VLM-TRACE.
Costs associated with accessing frontier-scale VLMs and LLMs and related infrastructure: $3,300 ($550 per month for 6 months, billed directly).
IIT Bombay supervision: $2,000, contingent on receipt of formal documentation from them. Any remaining funds would be re-allocated to the line item above.
Further red-tearing, cloud storage, conference attendance: $1,900.
GPU Computational Access: $1,800
Bank transfer and currency conversion charges: $800.
Who is on your team? What's your track record on similar projects?
My name is Johan Shiju Varughese and I am completing my final year B.Tech in Computer Science (Artificial Intelligence) at MBCET, Trivandrum. I am pursuing this project independently. I am responsible for developing VLM-TRACE, and previously contributed PR#3 to address support for BF16 to the UBC TEA Lab's vlm-probe repository. My paper (described above) lists me as the lead author, and I additionally carried out an unsuccessful adversarial test case which I have archived on Zenodo. I currently serve as Chairperson of the IET Student Chapter, MBCET, and was invited to give a talk at the IET's Festival of Volunteering in London on Role of Artificial Intelligence in developing soft skills in Sept 2026.
What are the most likely causes and outcomes if this project fails?
Should the degree of long-tail behavior exhibited by frontier VLMs result in less than a 20% saving in benchmarking samples, I will communicate this finding publicly and dedicate Months 4-6 to developing the VLM "cheating" benchmark. Should I be unable to secure an advisor from IIT Bombay, I will proceed without one and the relevant amount of funding will go unallocated. Should there be insufficient interest in the proposed solution, I will implement a dedicated command-line interface around the Inspect AI framework to facilitate uptake.
How much money have you raised in the last 12 months, and from where?
$0.
There are no bids on this project.