How Scaling a Diffusion-LM made it worse

Diffusion models are best known for generating images. Start with noise, remove it step by step, and eventually recover a coherent sample.

I wanted to see how the same idea behaved with text, so I directed DeepSeek V4 Flash, accessed free through OpenRouter, to build a Diffusion-LM-style model from scratch and train it to solve short riddles.

The interesting part was not just that scaling failed. In an oracle forward-corruption test, the gold answer remained perfectly recoverable at low noise, but the learned denoiser degraded that recoverable signal.

What I directed the agent to build

The model has three main stages:

1. A Transformer encoder turns the riddle into a contextual representation.
2. A bidirectional Transformer decoder iteratively denoises Gaussian noise into answer embeddings.
3. Each denoised embedding is rounded to its nearest vocabulary word by Euclidean distance.

The dataset started with 1,000 synthetic riddles. After normalization and duplicate removal, 298 unique examples remained: 232 for training and 66 held out (66/298) through a deterministic SHA-256 split.

The study evaluated three checkpoints:

The training column is descriptive only: evaluation dropped the final incomplete batch, so it covers 192 of 232 examples. I didn't use that partial metric as the primary evidence for memorization; the oracle diagnostic below is stronger.

This is an empirical result for these runs, not a universal scaling law. The phases also changed depth, hidden size, diffusion timesteps, and optimization history, so parameter count is not the only variable. What the comparison establishes is narrower: the larger configuration performed much worse under the tested setup.

The oracle diagnostic that isolated the failure

In this oracle diagnostic, x_t is created by adding noise to the gold answer embedding x_0 during the forward process. At t=2, the cumulative signal coefficient is approximately 0.999, so x_t is almost clean.

This is not a state normally available when generation begins from pure noise. It does not mean the generated reverse chain always contains the correct answer; it tests whether the model preserves answer information when that information is present.

The diagnostic decoded that same near-clean input with and without the learned model:

The test does not prove why the denoiser learned this behavior. Capacity, conditioning dominance, objective weighting, training duration, and optimization dynamics all remain possible causes.

The larger model mostly ignored the noisy answer state

The Phase 4 original model produced essentially the same held-out token accuracy across the full tested noise range: 68.3% at every sampled timestep.

That flat response suggests a conditioning-dominant denoiser: its predictions depended much more on the encoded riddle than on changes in x_t. This remains an inference. A shuffled-conditioning or zeroed-x_t ablation would be needed to establish the stronger causal claim.

A smoother learning-rate schedule was not sufficient

The original large model used a constant learning rate and showed loss spikes. DeepSeek trained the same 46.5M architecture again with linear warmup and cosine decay.

Optimization became much smoother. Final loss fell from 0.029 to 0.0033, and the spikes disappeared.

Held-out exact match still declined from 15.2% to 13.6%.

Learning-rate instability was therefore not sufficient to explain the failure. This is a useful warning: a cleaner loss curve does not guarantee a better generative model. The scheduled run optimized its objective more neatly without learning a better decoding behavior, but it tested only one alternative recipe.

More samples did not help either

The study also tested ten diffusion samples per riddle with majority voting. Exact match did not improve because the samples were nearly identical: ten draws produced only 1.08–1.26 unique answers on average, with mean pairwise token Jaccard above 0.97.

Voting mostly counted the same answer ten times.

Companion study: Qwen2.5 with LoRA

I then directed DeepSeek to test a different question: how would a pretrained 1.5B-parameter language model behave on exactly the same 66-riddle held-out set?

Zero-shot Qwen performed worse than the 8M diffusion model. Five examples recovered some ground, and LoRA fine-tuning moved beyond the diffusion baseline. The best normalized mixed-data run answered 42/66 riddles correctly.

Adapter rank was not the bottleneck: ranks 8 and 16 both reached 57.6% (38/66). Label-schema alignment mattered much more.

The external datasets followed a different answer-label convention: about 2% of their answers included articles such as "a" or "the," versus 77% in the original data. An unnormalized mixed-data run scored 21.2%; normalized runs reached 56.1–63.6%. This was not a strict single-variable ablation, so I read it as evidence that label-schema alignment mattered, not as a causal estimate of normalization alone.

Across three seeds, the mixed-data LoRA runs averaged 60.1% with a standard deviation of 3.8 percentage points. Reporting only the best seed would therefore overstate expected performance.

This is not a clean architectural contest. Qwen arrives with extensive pretraining, while the diffusion model learned from 232 task examples. The useful comparison is about adaptation: the small purpose-built model was surprisingly competitive, but the pretrained model benefited more when the fine-tuning data became broader and consistent.

The adapter combines the 177 original riddles with examples from RiddleSense, a Kaggle synthetic riddle dataset, and Crawsome riddles; full provenance and licensing details are on the model card.

Try the diffusion model

The public repository includes a CLI and downloads the Phase 3 model weights from Hugging Face on first use:

git clone https://github.com/beme08/riddle-diffusion-lm.git
cd riddle-diffusion-lm
pip install -e .

Solve a riddle:

riddle-solve "what can you catch but not throw"
# → a cold

Or inspect the sampler's candidates:

riddle-solve --show-candidates "what gets wetter as it dries"

Interactive mode is simply:

riddle-solve

Limits of the result

The experiment has several boundaries:

• The dataset is tiny and synthetic: 232 training riddles and 66 held out.
• Phase 3 was reconstructed after its original checkpoint was overwritten, although it reproduced the same 31/66 held-out score.
• Evaluation is strict: exact match penalizes semantically equivalent answers, answers have a fixed maximum length, and unseen held-out vocabulary maps to an unknown token.
• The scaling comparison changes architecture and diffusion length as well as parameter count, and these results concern one continuous-embedding diffusion formulation, not every diffusion language model.

What I took from it

When a diffusion language model underperforms, it is tempting to start with the noise schedule, sampler, or learning rate.

The oracle raw-x_t test offers a faster diagnostic question: when gold-answer information is present, does the denoiser preserve it?

If direct rounding beats the learned path in that controlled test, investigate the denoiser's behavior before assuming the corruption process is the bottleneck.

• GitHub repository
• Phase 3 diffusion checkpoint
• Qwen2.5 LoRA adapter and data provenance
• Full experimental report