Formal Disco: Growing Large, Diverse Datasets of Certified Programs from Scratch
November 19, 2026Large Language Models for code, especially open-weight models, still struggle with programming languages that support formal verification, largely because there are relatively few human-written formally verified programs to learn from. We introduce Formal Disco, a system in which LLM workers collaborate to write, repair and extend formally verified programs in an open-ended fashion. Using it, we generate datasets of over 100,000 verified programs across three verification-aware languages (Dafny, Verus and Frama-C), and show that using a simple principle of maximum entropy lets an open model improve itself training on its own verified programs while its generations become more diverse over time, avoiding the typical collapse that happens when training models on their own outputs.
AI coding agents are quickly changing how software gets written. Frontier agents can now produce entire code bases with tens of thousands of lines of code: Anthropic recently described a team of parallel Claude agents building a working C compiler, for instance. The cost of writing code is collapsing, but the cost of checking it is not. Tests can only cover finitely many cases, and humans can’t review code at the speed that coding agents (or swarms of them) write it. But bugs are still expensive, perhaps more than ever as our lives rely increasingly on automated systems.
Can we guarantee that a program is correct without reading the code? Formal software verification offers the strongest answer we know of. In a verification-aware language such as Dafny, Verus (for Rust) or Frama-C (for C), a program can have a logical specification of what it should do, as well as a mathematical proof that the implementation does what it is supposed to do, which a symbolic verifier checks mechanically. If AI agents wrote verified programs, a human would only need to read (or write) the specification, and could trust the verifier on the implementation.
Unfortunately, current LLMs, especially open-weight models, are still weak at writing verified code. That isn’t surprising once we look at the available training data. For popular languages like Python, there are tens of millions of public repositories on GitHub. In contrast, the largest public dataset of self-contained Dafny programs, DafnyBench, has fewer than 800 files after deduplication. Verification-aware languages are about as low-resource as programming languages get.
In earlier work on dafny-annotator, we saw a promising signal: a small synthetic dataset of Dafny programs generated by prompting GPT-4o, for about $10 in API credits, gave a large boost to a fine-tuned model’s ability to annotate Dafny programs on DafnyBench. While this initial work hit several limitations (* for instance, we would struggle to get diverse programs after the initial dataset, since the model would frequently propose many small variations of textbook algorithms), we saw this as an encouraging sign that synthetic program generation could be used to overcome the data scarcity issue of verification-aware languages. This post is about scaling that idea up to three languages, generating hundreds of thousands of programs, and using open models that improve themselves rather than fully relying on APIs. We also propose a simple, principled fix to the diversity issue, providing one conceptual answer to the question of what makes a good dataset of programs, which gives us an optimization objective for the system.
Example: a synthetic verified program
Before we explain our system, Formal Disco, let’s just look at one concrete example of the types of programs it will generate. The full system is language-agnostic: different languages can be used as backends in Formal Disco. Below is an excerpt of a simple Dafny program generated by Formal Disco, for illustration. The model that wrote this program was initially (at random) shown the README of the repository at nomadium/kochbuch, a GitHub repository whose name is German for “cookbook” and which has nothing to do with verification. Still, Formal Disco will use sampled REAMES from Github to diversify the themes of programs that it generates.

Programs in Dafny are like regular programs in other languages, but they can have logical specifications in them. Here, the ensures clauses are the specification: the returned index from FindRecipe is either -1 or within bounds, and if it is a valid index, the recipe there has the requested name. The method body is the implementation. The two invariant lines are logical annotations. They aren’t needed to run the code, but the verifier needs them to prove that the loop respects the specification. Writing these annotations is often a tedious and time-consuming part of verified programming in Dafny, and it is exactly the kind of skill we would like models to learn.
The full program also defines recipe scaling, validity predicates for recipes and cookbooks, and a lemma about scaling. The whole file is checked by Dafny with no errors. After generating many such programs, we can then use them for training models on verification-related tasks, such as adding logical annotations until a program verifies.
Formal Disco: agenda and LLM workers

Formal Disco’s architecture is inspired by classical discovery systems such as AM and Eurisko (Lenat & Brown, 1984), which tried to automate mathematical discovery. Like them, Formal Disco is organized around an agenda: a shared store of tasks (e.g., “repair program X”) and objects (programs, along with their verifier feedback). Unlike them, the agents that work on the agenda are LLMs rather than hand-coded heuristics. The agenda is also distributed, so any number of workers on any number of machines can join and claim tasks at any time.
We use three kinds of workers:
- Initiator: samples a seed context and asks the LLM for a small verified program inspired by it, plus ideas for how the program could be extended later. The program is then compiled and verified. If it passes, an “extend” task is created; if not, a “repair” task.
- Fixer: claims a repair task, reads the program and the compiler/verifier errors, reasons about them, and proposes a diff. If the patched program verifies, it moves on to being extended. Otherwise the task is retried, up to 3 attempts.
- Extender: claims a working program and proposes a diff that adds something to it, such as a new lemma, a new method, or a new feature that builds on what is already there. The result goes back to either an extend or a repair task.
Why bother with three workers instead of asking one model to write a complete verified program and repair it if it fails? We tested this directly. With Claude Opus 4.5 on Dafny, a “monolithic agent” (given verifier feedback and up to 3 repair attempts) produced programs averaging 85 lines, and never exceeding 128. The three-worker setup above, with the same number of calls to Claude Opus 4.5, produced programs averaging 268 lines, with the top 10% above 598 lines and the largest above 3,000, and it also had a higher yield per LLM call (29.8% vs. 23.0%). Splitting the work lets the Initiator be modest: the Extender adds complexity step by step, and the Fixer can handle more local errors that happen after each of these steps. This mirrors how large programs tend to evolve from long series of small edits.
What makes a good dataset of programs?
Before building a system to generate programs, we need to know what we’re aiming for. Prior work on synthetic data for Verus, such as SAFE and VeruSyn, was guided by several sensible intuitions:
- Bigger datasets are better, but not always. One “upgrade” from SAFE to VeruSyn was simply volume: VeruSyn had over 1M Verus programs, compared to around 20K in SAFE. More data usually helps, but clearly not in a trivial sense: we could create unboundedly many small variations of a single program, for instance, and that would not give us a useful dataset.
- Cover the language’s features. For programs, a good dataset should likely show usage of a broad sweep of features of the language. That again was an explicit motivation in VeruSyn, observing, for instance, that SAFE contained no lemmas, which are usually necessary to prove deeper properties. VeruSyn fixed this and similar gaps by seeding generation with material from the Verus documentation.
- Include complex programs. We want varied loop structures, long proofs and nontrivial implementations, not just toy functions. SAFE addressed this by having their programs all come from existing, real-world Rust programs, which they added specifications and proofs to using LLMs. VeruSyn started from SAFE as its seed.
Is there one principle behind all of these desiderata? We propose that there is a simple way to unify these and other intuitions: a good synthetic dataset of programs maximizes its entropy over a set of relevant program features.
A program feature is any function $f$ that maps a program to a discrete value. Examples include “the nesting structure of the loops in this method”, “the number of lines in this lemma’s proof”, “does this program use bitvectors?”, or “the content words in this program’s identifiers”. Given a dataset $\mathcal{D}$, each feature induces a distribution $P^{\mathcal{D}}_f$ over its values, and we measure diversity with its Shannon entropy:
$$H(f, \mathcal{D}) = – \sum_{v} P^{\mathcal{D}}_f(v) \log P^{\mathcal{D}}_f(v)$$
This single objective recovers each of the intuitions above:
- Dataset size gives a ceiling. Entropy is bounded by $\log |\mathcal{D}|$, so larger datasets have a higher potential for diversity.
- Dataset size alone isn’t enough. A trillion copies of one program have zero entropy over every feature.
- Complexity can be casted as entropy. If no program in the dataset has nested loops, a feature that measures loop usage in functions (e.g., the maximum nesting depth) would collapse most programs into few (or the same) values, yielding low entropy.
- Coverage can be casted as entropy. If no program uses lemmas, the binary feature “uses lemmas” has entropy zero. Raising it requires some programs that do and some that don’t.
This idea of using feature entropy gives us a quantity we can measure, use to compare existing datasets of very different sizes, and, as we’ll see below, directly optimize.
Getting entropy from external sources: READMEs and documentation
How do we maximize the entropy of our generated dataset? One knob we can use is the seed context that the initiator will receive when asked to sketch a program. We use this step to explicitly inject entropy from two external sources.
The first is a random README from a set of 100,000 GitHub repositories. These almost never have anything to do with verification. They can be web apps, configuration tools, game projects, and so on. The Initiator is told it can draw inspiration from the README, not that it must faithfully implement it. That is how the “cookbook” repository above turned into a verified recipe manager, and how we end up with verified cookie parsers, certificate chain validators, keyboard layout checkers, and taxonomic name databases of plant species.
Without READMEs, we saw a diversity collapse in the themes of the generated programs: the most frequent themes in the generated Dafny programs in a control without READMEs were stack, reverse, insert, pop, push, peek and bst. The model kept falling back on the same few textbook data structures.
The second part of the seed context is up to two documentation snippets from the language reference: a short description of a language feature with one usage example. Without them, some features never show up at all. Of the 30 Dafny features we track, corpora generated with documentation snippets cover all 30, while corpora without them plateau at 26. Bitvectors, extreme predicates, failure-compatible types and iterators/coinduction never appear without documentation. READMEs and documentation snippets are therefore complementary entropy sources: the first diversifies what programs are about, the second diversifies how they are written.
Self-improvement without diversity collapse
Every LLM call in Formal Disco is recorded in the agenda along with its prompt, its output, and whether it ended in a program that compiles and verifies. This gives us a natural loop for training open models:
- Distill. We first run Formal Disco with frontier models (Claude Sonnet 4.5 and Opus 4.5). Even these fail often. For example, the Initiator produces a verified program only 11% (Verus) to 23% (Dafny) of the time, so generating hundreds of thousands of programs this way would be expensive. But a run of around 10,000 attempts is enough to fine-tune an open model, Qwen 2.5-Coder 32B, into a usable worker.
- Self-improve. We then run Formal Disco with the fine-tuned Qwen, collect its traces, fine-tune again on everything gathered so far that verifies, and repeat.
The key question is which examples to train on. Training only on successful ones is the classic recipe (ReST, RAFT), but it has an obvious failure mode here: the easiest way for a worker to succeed more often is to write simpler programs. More generally, it is well known that models trained repeatedly on their own outputs tend to lose diversity.
Our entropy principle suggests a fix. The entropy of a feature can be written as the average surprisal of each program’s feature value, $-\log P^{\mathcal{D}}_f(f(p))$. This tells us how much each individual program contributes to the dataset’s diversity. We rank successful programs by surprisal on each of four target features (number of annotations per method, lemma proof size, loop skeletons, and method body size), give each program its best rank across the four features, and train only on the top third. In short, we train the model on its most unusual successes.






Figures 3 and 4 show the two parts of the story. First, workers get better. In Verus, for example, the Initiator’s success rate goes from about 11% with Claude to 43% with Qwen at iteration 5. At the same time, the programs they generate get more diverse. Loop-skeleton entropy roughly doubles relative to the Claude corpus in all three languages. By the last iteration, the Qwen corpus beats the Claude one on all four targeted features in every language, with one exception (lemma proof sizes in Verus). And because Qwen is open and cheap to run, this combination makes the pipeline practical at scale.
How do these datasets compare to existing ones?
Comparing datasets of very different sizes is tricky. VerusBench has fewer than 200 programs, while VeruSyn has over a million. For this we borrowed a conceptual tool that is very often used in the field of ecology: the idea of rarefaction curves, which ecologists use to compare species diversity across samples of different sizes (Raup, 1975). A rarefaction curve plots a diversity measure (here, feature entropy) against the number of programs sampled. This shows how quickly each dataset reveals new variety, and lets us compare them at matched sizes.
Figure 6: Rarefaction curves over four features for Dafny (left) and Verus (right). Three of these (annotation templates, language features, subject words) are not targeted by entropy maximization, while loop skeletons are. Formal Disco corpora are shown for both the initial Claude run and the fifth Qwen iteration.
A few things stand out:
- In Dafny, human-written code still wins in diversity of annotations. DafnyBench has the most diverse annotation templates (the shapes of assertions, invariants, and pre/postconditions), even though it is much smaller. On the other three features, Formal Disco programs are more diverse than DafnyBench at matched sample sizes.
- In Verus, Formal Disco leads on three of four features. On annotation templates, language features and subject words, Formal Disco’s curves sit on top across the range they cover. VeruSyn keeps gaining annotation-template entropy as it grows and eventually overtakes us there, but it saturates early on the other features. On loop skeletons, the most diverse dataset is VeruSAGE, which comes from real-world Verus code.
- What we don’t optimize can drift. On features we don’t target, such as subject words, the self-improved Qwen corpus is somewhat less diverse than the original Claude corpus. On the feature we do target, loop skeletons, it is far ahead. Entropy maximization does what it says, but only on the features we point it at.
No single dataset wins everywhere, which is exactly why a tool like this is useful: “our dataset has N programs” says very little, while rarefaction curves over meaningful features say much more.
Do synthetic programs help on human-written code?
Diversity is a proxy for dataset quality, but we also want our data to be useful for models to improve at verification tasks. We tested this on two tasks, training Qwen 2.5-Coder 32B with the simplest recipe possible of plain supervised fine-tuning (SFT) on examples extracted deterministically from our programs, with no chain-of-thought.
Task 1: Annotation. Given a program with its specification but with all assertions and loop invariants removed, put back annotations so that it verifies. We created training examples by stripping annotations from Formal Disco programs, and evaluated on DafnyBench (555 programs) and VerusBench (149 programs) stripped the same way.
Figure 5: Success rate on the annotation task as a function of the number of independent attempts k. The dashed line marks Claude Opus 4.5’s single-attempt success rate. On VerusBench, Qwen fine-tuned on Formal Disco data matches Opus at Pass@1 and keeps improving with more attempts, outperforming the same model fine-tuned on SAFE.
On VerusBench, fine-tuning lifts Qwen from 8.7% to 43.0% at Pass@1, exactly matching Claude Opus 4.5. We also ran our exact same pipeline on SAFE, a dataset derived from real-world Rust code. The SAFE-trained model is comparable at Pass@1 (40.9%) but plateaus at 51% with more attempts, while the Formal Disco model reaches 59.1% at Pass@32. This suggests our programs teach a broader range of strategies. On DafnyBench, fine-tuning roughly doubles Qwen’s success at 16 attempts (19.5% → 40.4%), reaching Opus’s single-attempt performance (41.4%).
Task 2: Lemma proving. Given a Dafny program in which a lemma’s proof has been removed, the task is to write a proof. On 357 DafnyBench lemmas, fine-tuning raises Pass@1 from 4.5% to 25.5%. A single sample from the fine-tuned model beats 32 samples from the base model (20.7%). Claude Opus 4.5 remains much stronger here (61.6% at Pass@1). That gap is consistent with proofs benefiting from reasoning that our fine-tuned model was never trained to produce. Interestingly, Opus barely improves with more attempts (64.7% at Pass@4), while our model keeps climbing up to Pass@32 (39.8%).
Since this is about the simplest way to use the data, we read these numbers as lower bounds on what the datasets can do.
Why this matters
We see three main takeaways.
First, the data barrier in formal verification can be overcome. Verifiers provide exactly what synthetic data generation needs: a cheap, scalable and trustworthy signal of correctness. With an open model doing the heavy lifting, generating hundreds of thousands of verified programs becomes affordable.
Second, synthetic data diversity can be measured and optimized directly. Framing dataset quality as entropy over program features unifies our intuitions about dataset size, coverage and complexity, gives us tools such as rarefaction curves for comparing different datasets, and provides a concrete objective function. Optimizing that objective lets a model train on its own outputs and become more diverse over time.
Third, open-ended generation still needs a source of novelty. Left to themselves, models drift back to proposing standard ideas, writing programs about stacks and binary search trees. In our case, random READMEs and documentation snippets are simple, effective ways to keep them exploring, and removing either one causes a measurable collapse on the features they were introduced to influence.
There is plenty left to do. Distilling reasoning traces or doing reinforcement learning against the verifier should extract much more from these corpora than plain SFT. We also pick the features to maximize entropy over by hand. Learning which features matter, with downstream usefulness as the meta-objective, is an exciting open problem. Finally, nothing in Formal Disco is specific to these three languages: the same framework should apply to interactive theorem provers and to open-ended exploration of formal mathematics.
More broadly, we think our approach can also overcome another puzzling issue of AI coding models. Programmers today learn to increasingly rely on AI, but traditionally a programming language takes decades to accumulate the volume of code that makes AI assistance possible. As coding agents become more prevalent, the quality of AI assistance for a given language might start being a key deciding factor for whether a language is adopted at all. This is an issue since we would ideally keep evolving our tools, including programming languages — not freeze them with what we have today because future tools won’t be able to compete in training data. Systems like Formal Disco suggest that a new verification-aware language could instead potentially launch with a large synthetic dataset on day one, so that even its earliest users have capable AI support. Strong correctness guarantees could then become a practical default for AI-written software as well, ultimately alleviating the verification burden that human developers still carry.
Code and data
Formal Disco is open source at github.com/metareflection/formal-disco. Our datasets (dafny-disco, verus-disco and framac-disco, with over 100,000 verified programs in total) are available on Hugging Face.
Acknowledgments
This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Cluster experiments were partially made possible through a Kempner Institute Accelerator Award. This research was partially supported by a Coefficient Giving grant, funding and credits from an Amazon Research Award, and credits from Google Cloud. Simon Henniger was supported by the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) Prize Fellowship and the German Academic Fellowship Organization, funded by the German Federal Ministry for Economic Affairs and Energy.
- Baksys, M., Zetzsche, S., Bouissou, O., Delmas, R., Kong, S., & Holden, S. B. (2026). ATLAS: Automated Toolkit for Large-Scale Verified Code Synthesis. arXiv:2512.10173.
- Carlini, N. (2026). Building a C Compiler with a Team of Parallel Claudes. Anthropic Engineering.
- Chen, T., et al. (2026). Automated Proof Generation for Rust Code via Self-Evolution. arXiv:2410.15756.
- Cuoq, P., Kirchner, F., Kosmatov, N., Prevosto, V., Signoles, J., & Yakobowski, B. (2012). Frama-C: A software analysis perspective. International Conference on Software Engineering and Formal Methods (SEFM).
- Di, N., et al. (2026). Reducing the Costs of Proof Synthesis on Rust Systems by Scaling Up a Seed Training Set. arXiv:2602.04910.
- Dong, H., et al. (2023). RAFT: Reward Ranked Finetuning for Generative Foundation Model Alignment. arXiv:2304.06767.
- Gulcehre, C., et al. (2023). Reinforced Self-Training (ReST) for Language Modeling. arXiv:2308.08998.
- Hui, B., et al. (2024). Qwen2.5-Coder Technical Report. arXiv:2409.12186.
- Lattuada, A., et al. (2023). Verus: Verifying Rust Programs using Linear Ghost Types. Proceedings of the ACM on Programming Languages (OOPSLA).
- Leino, K. R. M. (2010). Dafny: An Automatic Program Verifier for Functional Correctness. International Conference on Logic for Programming, Artificial Intelligence and Reasoning (LPAR).
- Lenat, D. B., & Brown, J. S. (1984). Why AM and EURISKO appear to work. Artificial Intelligence, 23(3).
- Loughridge, C., et al. (2024). DafnyBench: A Benchmark for Formal Software Verification. arXiv:2406.08467.
- Poesia, G., Loughridge, C., & Amin, N. (2024). dafny-annotator: AI-Assisted Verification of Dafny Programs. arXiv:2411.15143.
- Raup, D. M. (1975). Taxonomic diversity estimation using rarefaction. Paleobiology, 1(4).
- Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631.
- Yang, C., et al. (2025). AutoVerus: Automated Proof Generation for Rust Code (VerusBench). Proceedings of the ACM on Programming Languages (OOPSLA).