Model genealogy hackathon proposal

We propose running a hackathon at Constellation where participants identify the provenance and genealogy of distilled models.

Motivation

1. Model distillation is a major concern; it may enable open-weight models to catch up to the closed-weight frontier more quickly than otherwise, contributing to race dynamics and the rapid diffusion of dual-use capabilities outside of trusted access programs.

2. Therefore, it would be useful to measure how pivotal distillation is to current open-weight model performance (vs. other from other research efforts or data). This requires measuring the extent of distillation present in these open-weight models.

3. We propose developing methods for assessing the provenance of a distilled model, that is, which teachers have contributed in which amounts to which capabilities.

Details

• Proposed schedule: December 5, 2026 from 8:30am to 8pm in Constellation 6E.

  • 08:30 Breakfast + onboarding

  • 09:00 Introductory talk (speaker TBD, tentatively Joachim Schaeffer)

  • 09:30 Introduction of the task – Hackathon begins

  • 12:30 Lunch break (1hr)

  • 13:30 Hackathon continues

  • 18:30 Dinner break (1hr) – Hackathon finishes and judging begins

  • 19:30 Concluding remarks and awards

• Participants: ~30, with a goal of ⅔ in-network and ⅓ external who are invited by in-network participants/organizers, or selected by application. Teams of 3-5.

• Task details

  • Given a set of models distilled from various teacher models, predict the teachers that each model was distilled from (if any) and, for bonus points, the type of data that was used in distillation. Participants also submit a brief writeup of their methods.

    • We will also provide a training/validation set of models with publicly known distillation recipes to guide method development

  • Awards for best performance, a creativity award, and a judge’s choice award.

• Post-hackathon

  • We will work with hackathon participants to promptly produce a paper and blog post documenting our findings.

Proposed funding request (item: details, estimated $)

GPUs: Distilling models and participant usage, $10,000

OpenRouter credits: Distilling models and participant API usage, $18,000

OpenAI / Anthropic tokens: Participant coding agents and API usage, $100,000

Food: Three meals for 35 people at $150/person, $5,250

Prizes: $3k for the top team, $1k for other tracks, $4,000

Total: $137,250

Additional funding details

• GPUs are for us (the organizers) to set up the challenge by creating our family of distilled models and for participants to use during the challenge. We expect to spend roughly ~150 node-hours for the distillation. For the participants, we plan to rent eight H100 nodes for 12 hours, adding 96 node-hours. At $40/hr, this gives a total of $9,840 for 246 node-hours.

• We plan to generate about ~10B tokens from teacher models for distillation. Estimating the OpenRouter cost from some of our candidate models including Muse Spark Open, Mistral Medium 3.5, Mistral Large 4, Reflection Beam 1.0, and Llama 4 Maverick, we arrive at roughly $10,000 in tokens including fees. Note that datasets will be mixed and reused across student models, so they will see substantially more tokens in aggregate than 10B. We also plan to give $1,000 per team in OpenRouter credits for the day of the challenge, totaling $18,000.

• $100k is about $3k in credits per participant over 10 hours of challenge time. We roughly estimate this to be sufficient for ample Opus/Sol and reasonable Fable/Astra usage (i.e. not /fast maxxing).

• We will provide breakfast, lunch, and dinner for the participants and on-site organizers during the hackathon.

• The prize split is tentative, and we may add to it if we have leftover funds.

• We are in talks with OpenAI and Anthropic for token sponsorship and have raised $10k for the other expenses, we are requesting for the rest of the funding required for this hackathon.

• The current organizers are OpenAI Safety Fellows and Anthropic Fellows: myself (Henry Castillo), Ida Caspary, Johnathan Sun, Shane Arora, Xavier Roberts-Gaal, and Eric Modesitt.