You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
Large language models acquire almost all of their knowledge and behavioural dispositions during pretraining, yet safety alignment is typically added only during post-training, and growing evidence shows this to be superficial and removable with minimal further fine-tuning. Recent work on safety pretraining and midtraining has been mostly data-centric interventions that add alignment-related documents for training under the standard next-token prediction objective (Maini et al., 2025; Tice et al., 2026; Minder et al., 2026; Li et al., 2026), yielding safer models at minimal capability cost. However, these alignment priors are sensitive to intervention timing (Sam et al., 2026) and can fade under subsequent post-training (Korbak et al., 2026).
This project shifts the intervention from the data to the training objective itself: we adapt post-training methods for safety alignment during pretraining to identify the most effective methods, intervention timing, and data curricula. If successful, this work will deliver training recipes for open-weight models whose safety is entangled with their capabilities rather than layered on top, making safety resistant to removal and robust across downstream fine-tuning and deployment.
Question. Do post-training methods (SFT, DPO, KTO, RLVR) applied during pretraining instil a narrow model value (e.g., pro-affordability) that generalises out-of-distribution better than the synthetic-document, next-token intervention of Model Spec Midtraining (MSM; Li et al., 2026)? And does intervening during pretraining outperform the same intervention applied after pretraining? RQ1 provides a relatively fast and inexpensive signal on whether applying post-training methods for alignment during pretraining is promising before committing to RQ2's larger runs.
Setup. We pretrain a Qwen3-1.7B-architecture model from scratch on 100B FineWeb tokens with no intervention, checkpointing at 20 / 40 / 60 / 80 / 100% of the token budget. From each checkpoint we branch: we apply one of four intervention methods — SFT, DPO, KTO, or RLVR — using Li et al.'s MSM corpus (≈8M tokens) converted via LLM API calls into each method's native format (KTO reuses the DPO pairs), then resume standard pretraining to the full 100B tokens. Interventions at the 100% checkpoint need no continued pretraining and reproduce the standard "intervene after pretraining" setting. All final models then receive identical alignment fine-tuning (AFT) using Li et al.'s data (165k tokens of value-preference alignment fine-tuning data + 2M tokens of instruction-tuning data). We also replicate two baselines from Li et al.: (i) vanilla pretraining + AFT, and (ii) vanilla pretraining + MSM + AFT, to isolate the effect of the training objective from that of the data. MSM, each of the four intervention methods, and AFT are repeated with 4 random seeds to measure run-to-run variance (Olmo 3, 2025).
Evaluation. Capabilities: MMLU, ARC-Easy, ARC-Challenge. Alignment: Li et al.'s evaluation on about 400 held-out preference pairs across unseen domains, sampling 4 responses per pair and scoring the value-aligned preference rate with an LLM judge. We compare our intervention methods and timings against the MSM baselines.
Question. Does the best alignment intervention from RQ1, applied during pretraining, produce agentic alignment that survives full post-training (including RL) and adversarial fine-tuning?
Setup. We pretrain a SmolLM3-3B-architecture model on 500B Dolma 3 tokens (matching Minder et al.), checkpointing at 20 / 40 / 60 / 80 / 100%. We intervene with each of the four intervention methods — SFT, DPO, KTO, or RLVR, at each checkpoint, using Sun et al.'s Alignment Shaping data (16k mixed-domain synthetic preference samples; converted via LLM API calls to RLVR format), then resume pretraining to 500B tokens. Each intervened model — plus a vanilla pretraining baseline — undergoes identical post-training adapted from the Tulu 3 pipeline exactly as in Sun et al.: SFT on 234,835 samples (25% of the Tülu 3 SFT mix, 2 epochs), DPO on 272,898 preference pairs (the full Tulu 3 DPO set), and RLVR via GRPO on 16,900 difficulty-filtered prompts (16 rollouts per prompt). Each of the intervention methods, as well as post-training, are repeated with 4 seeds to measure variance. Finally, each post-trained model receives Alignment Repair (DPO on the same 16k mixed-domain samples, 1 epoch, full fine-tuning), yielding the final models.
Comparisons and ablation. Comparing against Sun et al. (identical data, Alignment Shaping applied after pretraining) isolates the effect of intervention timing; comparing against Minder et al. (identical pretraining setup, data-centric intervention under the next-token objective) isolates the effect of the training objective. Ablation: for the Sun et al. baseline and our best-performing intervention (2 post-training seeds each), we re-run post-training with the safety data removed and token-matched with additional general data, to study whether safety post-training remains necessary on top of safety pretraining and how the two interact.
Evaluation (following Sun et al.). Agentic misalignment: Leaking and Murder (Lynch et al., 2025; 6,000 rollouts/model) and Sun et al.'s 25-scenario Reward Hacking suite (45,000 rollouts/model across a 36-condition pressure matrix). Chat alignment: EM Questions (44 questions × 50 rollouts; Betley et al., 2025) and SYCON-Bench multi-turn sycophancy (22 scenarios × 20 rollouts × 6 turns). Capabilities: GSM8K, MMLU, IFEval, MATH. Over-refusal: OR-Bench.
Adversarial fine-tuning robustness. We subject the Sun et al. baseline and our best-performing intervention (1 post-training seed) to three standard fine-tuning attacks and re-evaluate agentic and chat alignment:
Benign fine-tuning on GSM8K (1 epoch, 7.5k examples, 4M tokens) — the canonical benign attack shown to erode alignment (used by Maini et al., 2025 and Sam et al., 2026).
Harmful-example fine-tuning on 100 explicitly harmful instruction-response pairs for 5 epochs (≈0.1M tokens), following Qi et al. (2023), which subverts safety alignment at negligible cost.
Emergent-misalignment fine-tuning (Betley et al., 2025) on bad-medical-advice and risky-financial-advice data (≈1k samples; 60 full-fine-tuning steps, checkpoints every 10), matching Sun et al.'s stress test.
Costing assumptions
Seeds. All totals include 2 pretraining seeds throughout; and for each pretraining run, we include 4 seeds for safety interventions and post-training.
Contingency. A 15% contingency is applied to each RQ total for restarts, hyperparameter tuning, and failed runs.
Vanilla pretraining — 2 runs — 1,700 H200-hours Qwen3-1.7B from scratch, 100B tokens, no intervention; checkpoints at 20/40/60/80/100%; 2 seeds.
Alignment interventions — 200 runs — 800 H200-hours {SFT, DPO, KTO, RLVR} × {20, 40, 60, 80, 100%} × 4 seeds, on ≈8M tokens of format-converted MSM data; plus MSM (next-token) baseline × {20, 40, 60, 80, 100%} × 4 seeds.
Continued pretraining — 40 runs — 17,000 H200-hours Resume each intervened checkpoint to 100B tokens: 5 methods × {80, 60, 40, 20}B × 2 pretraining seeds = 2,000B tokens total.
AFT + baselines — 208 runs — 360 H200-hours Li et al. alignment fine-tuning (165k + 2M tokens, LoRA) on all 50 intervened models ({SFT, DPO, KTO, RLVR, MSM-baseline} × {20, 40, 60, 80, 100%} × 2 pretraining seeds) + 2 vanilla pretraining runs, 4 seeds.
Subtotal: 19,860 H200-hours
Capabilities — 208 models — 295 H200-hours MMLU, ARC-Easy, ARC-Challenge.
Value alignment (OOD) — 208 models — 295 H200-hours Li et al.'s 400 held-out preference pairs × 4 samples per pair; responses classified by LLM judge.
Subtotal: 590 H200-hours
RQ1 total (incl. 15% contingency): (19,860 + 590) × 1.15 ≈ 23,500 H200-hours. Reducing to just 1 seeded run would result in 11,750 H200-hours instead.
Vanilla pretraining — 2 runs — 20,000 H200-hours SmolLM3-3B from scratch, 500B tokens, no intervention; checkpoints at 20/40/60/80/100%.
Alignment interventions — 160 runs — 1,600 H200-hours {SFT, DPO, KTO, RLVR} × {20, 40, 60, 80, 100%} × 4 seeds × 2 pretraining seeds, on Sun et al.'s 16k Alignment Shaping samples.
Continued pretraining — 32 runs — 160,000 H200-hours Resume each intervened checkpoint to 500B tokens: 4 methods × {400, 300, 200, 100}B × 2 pretraining seeds = 8,000B tokens total.
Post-training — 168 runs — 9,240 H200-hours Tulu-3-style SFT + DPO + RLVR (Sun et al.); (20 intervention variants + vanilla pretraining baseline) × 2 pretraining seeds × 4 post-training seeds.
Post-training ablation — 8 runs — 440 H200-hours Safety data removed, token-matched; vanilla + best-performing configs × 2 pretraining seeds × 2 post-training seeds.
Alignment Repair — 176 runs — 530 H200-hours DPO on 16k mixed-domain samples, 1 epoch, full fine-tuning (Sun et al.).
Subtotal — training: 191,810 H200-hours
Agentic misalignment — 352 models — 2,000 H200-hours Leaking and Murder (≈6,000 rollouts/model) + Reward Hacking (45,000 rollouts/model).
Chat alignment — 352 models — 355 H200-hours EM Questions (2,200 responses/model) + SYCON-Bench sycophancy (2,640 turn classifications/model).
Capabilities & over-refusal — 352 models — 495 H200-hours GSM8K, MMLU, IFEval, MATH (lm-evaluation-harness) + OR-Bench.
Adversarial fine-tuning — 4 models × 4 settings — 100 H200-hours Benign (GSM8K), harmful-100, and EM fine-tuning × 2 datasets on 4 final models; attack training + per-checkpoint re-evaluation.
Subtotal — evaluations: 2,950 H200-hours
RQ2 total (incl. 15% contingency): (191,810 + 2,950) × 1.15 ≈ 224,000 H200-hours. Since pretraining takes up the most compute, reducing to one pretraining seed cuts down to ≈112,000 H200-hours.
Project total: 23,500 + 224,000 = 247,500 H200-hours (models evaluated: 208 in RQ1; 352 in RQ2, i.e. 176 post-Alignment-Repair finals + 176 pre-repair checkpoints to decompose Shaping vs. Repair effects). Alternatively, 123,750 H200-hours for one pretraining seed.
We budget at $3.00 per H200-hour.
247,500 H200-hours × $3.00/h = $742,500
123,750 H200-hours × $3.00/h = $371,250
We need to store tokenized corpora, model checkpoints, and evaluation artifacts. Checkpoint sizes: ≈3.5 GB (1.7B, bf16) and ≈6.5 GB (3B, bf16); a small number of optimizer-state checkpoints (×6 size) are retained per active run for resumption.
Tokenized corpora — 1.45 TB (FineWeb 100B-token subset ≈0.25 TB; Dolma 3 500B-token subset ≈1.2 TB)
RQ1 checkpoints — 1.85 TB (trunk + resume states, 200 intervened models, 40 continue-pretraining finals, 208 post-AFT finals)
RQ2 checkpoints — 4.30 TB (trunk + resume states, 160 intervened models, 32 continue-pretraining finals, 176 post-trained + 176 post-repair finals, attack checkpoints)
Evaluation transcripts, logs, experiment tracking exports — 0.40 TB
Total (provisioned: 8 TB for 12 months) — ≈8.0 TB
Cost. On AWS S3 Standard ($0.023/GB-month), 8 TB × 12 months ≈ $2,200. We budget $2,500 to allow for any archival storage that might be needed.
LLM API calls are needed for (a) converting existing alignment datasets into each intervention method's native format, and (b) LLM-as-judge evaluation. Li et al. use judges for data filtering and QA scoring; Sun et al. use an LLM judge to classify Leaking-and-Murder and EM-Questions responses and for Reward Hacking detection.
RQ1 — dataset conversion — ≈52,000 calls — 71M in / 26M out — $402 Convert 8M-token MSM corpus (≈8k docs) to SFT / DPO / RLVR formats, QC pass, RLVR verifier construction.
RQ1 — preference-eval judging — ≈333,000 calls — 150M in / 20M out — $500 ≈1,600 calls per model × 208 models.
RQ2 — dataset conversion — ≈20,000 calls — 22M in / 10.4M out — $148 Convert 16k Alignment Shaping samples to RLVR format + verifiers.
RQ2 — eval judging — ≈7,430,000 calls — 5,940M in / 445M out — $16,340 ≈21,100 calls per model (L&M 6,000; Reward Hacking fallback 9,000; EM 2,200; SYCON 2,640; OR-Bench 1,300) × 352 models.
RQ2 — adversarial-FT judging — ≈342,000 calls — 240M in / 27M out — $750 ≈21,400 calls per attack experiment × 16 experiments.
Total (standard rates): ≈8.17M calls, ≈6.42B in / 529M out — $18,140. Budgeted (incl. ≈25% contingency): $23,000.
Per Anthropic's published API pricing (platform.claude.com, July 2026), Claude Sonnet 5 is $2 / $10 per million input / output tokens; the Batch API applies a 50% discount to both. We budget everything at standard Sonnet 5 rates as an upper bound, but in practice, this workload is almost entirely batch-eligible.
The estimated cost at standard rates would be $2 × 6.42B + $10 × 529M ≈ $18,140. We budget 25% contingency which totals at ≈$23,000.
We plan to submit this work to NeurIPS 2027 (hosted in Europe). Two co-first authors, both UK-based, will attend: one full-time student and one registrant at the academic rate.
Registration — early-bird (student ≈$550 + academic ≈$1,000) — $1,550
Return flights, incl. checked bag (2 × $250) — $500
Accommodation (6 nights × $150 × 2 people) — $1,800
Meals and incidentals (7 days × $50 × 2 people) — $700
Local transport, poster printing, miscellaneous — $400
Total — $4,950
GPU compute — 247,500 H200-hours @ $3.00/h — $742,500
Storage — 8 TB × 12 months — $2,500
LLM API — Claude Sonnet 5, standard rates + contingency — $23,000
Conference travel — NeurIPS 2027, 2 authors — $4,950
Total — $772,950
The total cost is dominated by GPU compute and in particular, the continued pretraining cost. For both RQ1 and RQ2, we can do a one-seed pretraining run instead of two, which would reduce the total project cost to about $390,000.
Minder, J., Moskvoretskii, V., Singhal, R., Jiao, D., Bali, K., Borjigin, Y., Cui, S., Krsteski, S., Anderson, A., Aydin, R. and West, R. (2026). Synthetic Persona Pretraining: Alignment from Token Zero. LessWrong. https://www.lesswrong.com/posts/3xQQK9i8mhJDE2uMg/synthetic-persona-pretraining-alignment-from-token-zero
Li, C., Wichers, N., Price, S., Marks, S. and Kutasov, J. (2026). Model Spec Midtraining: Improving How Alignment Training Generalizes. arXiv. https://arxiv.org/abs/2605.02087
Maini, P., Goyal, S., Sam, D., Robey, A., Yash, S., Jiang, Y., Zou, A., Fredrikson, M., Lipton, Z. and Kolter, Z. (2026). Safety pretraining: Toward the next generation of safe AI. Advances in Neural Information Processing Systems, 38, pp.43971–44008. https://proceedings.neurips.cc/paper_files/paper/2025/hash/3e84c4e0acee2be072571fedc70700a9-Abstract-Conference.html
Sam, D., Goyal, S., Maini, P., Robey, A. and Kolter, J.Z. (2026). When Should We Introduce Safety Interventions During Pretraining? arXiv. https://arxiv.org/abs/2601.07087
Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D. and O'Brien, K. (2026). Alignment pretraining: AI discourse causes self-fulfilling (mis)alignment. arXiv. https://arxiv.org/abs/2601.10160
Korbak, T., Raymond, C., Carroll, M., Williams, M., Balesni, M., Guo, A., Wolfe, J., Jagadeesh, A. and Kivlichan, I. (2026). How Far Does Alignment Midtraining Generalize? OpenAI. https://alignment.openai.com/how-far-does-alignment-midtraining-generalize/
Sun, J., et al. (2026). Training Against Misalignment: Offline Preference Training with Simple Synthetic Safety Data (draft paper). https://drive.google.com/file/d/1tNFB4GaC_caDrbUUJaXutQ4tXACC96zF/view?usp=sharing
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P. and Henderson, P. (2023). Fine-Tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! arXiv. https://arxiv.org/abs/2310.03693
Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N. and Evans, O. (2025). Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs. OpenReview. https://openreview.net/forum?id=aOIJ2gVRWW
Lynch, A., Wright, B., Larson, C., Ritchie, S.J., Mindermann, S., Hubinger, E., Perez, E. and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. https://arxiv.org/abs/2510.05179
Olmo, T., et al. (2025). Olmo 3. arXiv. https://arxiv.org/abs/2512.13961