Your training set is the model

There is a Java class in the ClassSum dataset that is eighty lines long. After the dataset’s preprocessing step runs, it is forty-five. Method bodies gone. Control flow gone. Inline logic gone.

The ground-truth summary attached to it reads: class representing a connection to the telegram api.

That is a fine description of a stub. It is a useless description of an eighty-line class. And a model trained on thousands of pairs like this learns exactly the wrong lesson, which is that a class summary is a one-line restatement of the class name.

The reduction script is public. You can read it. This is not a subtle bug, and it is not unique to one dataset.

The actual bottleneck

For class-level code summarization, performance is limited by data quality more than by parameter count. That claim is not controversial anymore. What is still unresolved is what to do about it.

Method-level summarization is comparatively forgiving. A method is mostly self-contained, so a noisy training pair damages one local example. A class is not self-contained. Its intent lives across fields, constructors, method interactions, and inheritance. The model has to aggregate distributed evidence into an architectural narrative. Every dimension of noise compounds when the unit of summarization gets bigger.

Then there is the budget. You mine a corpus of a few million pairs. You can afford to train on a fraction of it. Which fraction is now a design decision with real consequences, and most pipelines treat it as a preprocessing chore.

Why one signal is not enough

The standard move is coherence filtering. Embed the code, embed the comment, compute similarity, drop the low scores.

Vitale et al. showed this is already insufficient at the method level. A single alignment signal captures one dimension of quality. It says nothing about whether the example is structurally representative, whether it is a near-duplicate of four hundred others, whether the comment is boilerplate, or whether the class is complex enough to be worth learning from.

At the class level the gap widens. A perfectly aligned code-comment pair can still be a terrible training example, because the comment is shallow, or the class is a two-field DTO, or you already have five hundred identical Spring controllers in your subset.

Coherence filtering also has a quiet failure mode: it is monotonic. It ranks examples and takes the top K. That systematically strips out rare, architecturally interesting classes, because unusual code tends to score lower on similarity to its documentation. You optimize your dataset toward the mean and then wonder why the model cannot handle anything unusual.

Dataset selection is combinatorial optimization

The framing I argue for in this paper is that picking a training subset is not a filtering problem. It is a constrained subset selection problem over a space of size N choose K, which for any real corpus is not searchable exactly.

That reframing changes what you optimize. Instead of scoring examples independently and taking the best ones, you score subsets. A subset can be good in ways no individual example can be: it can cover the complexity distribution, it can avoid redundancy, it can include rare patterns.

The approach, EvoQD, has three parts.

A multi-attribute quality score. Six normalized signals per example instead of one. Structural completeness, whether the class has the elements a real class has. Complexity appropriateness, rewarding coverage across complexity bins rather than a single band. Documentation richness, which penalizes summaries that just echo the class name. Alignment, the classic embedding similarity. Redundancy penalty, which zeroes out near-duplicates above a similarity threshold. And an LLM judge score.

The judge is class-aware, not borrowed from method-level work. It keeps content adequacy and adds two dimensions that only make sense at this level: architectural insight, whether the summary captures the class’s design role, and internal consistency, which penalizes a summary that contradicts itself or drifts in terminology halfway through.

A genetic algorithm over subsets. Fitness combines mean quality, diversity measured as cluster entropy over the embedding space, and a valuation term. Crossover takes the union of two parent subsets and downsamples to budget, which preserves what both parents agreed on. Mutation is a guided swap rather than a random flip: drop low-quality or redundant members, add high-quality members from underrepresented clusters.

Distributional valuation. Quality and diversity together still do not guarantee that the selected subset resembles the distribution you actually care about. We use SAVA, an optimal-transport method, to score how much each candidate contributes to matching a small curated reference set. It is learning-agnostic, so it costs nothing in retraining.

The honest part

This paper argues the framework. It does not yet report results.

I am saying that plainly because the alternative is worse. This is a vision paper accepted at the JAWS workshop at ICSE 2026, and the implementation and evaluation are the next cycle. What I have is the formalization, the attribute schema, the operators, and an argument for why single-signal filtering cannot get there.

The thing that would falsify it is straightforward, and I would rather state it than have someone else point it out: if a subset chosen by plain coherence filtering trains a summarizer that matches one chosen by EvoQD, at equal budget, then the extra machinery buys nothing and the field should stick with the cheap heuristic. That is the experiment. It runs next.

What I am confident about is the framing. Under a fixed compute budget, what you train on is a first-order decision, not a preprocessing step. Right now most of us are making it with a similarity threshold and a vibe.

Paper: Search-Based Evolutionary Data Pruning for Class-Level Code Summarization, JAWS at ICSE 2026.