Where can
two blocks
become one?

Too few output directions. Then, a learning bottleneck.

The original replacements removed 99.52% of the four selected blocks’ parameter payload: 850.15 MB became 4.08 MB of compiled meshes. That was an extreme capacity test, not merely two blocks becoming one. A new diagnostic separates missing output directions from errors in predicting the coordinates that are available.

82.18% of pair 11–12’s squared endpoint error and 89.28% of pair 15–16’s lies outside the current fixed decoder subspace. These percentages use the last eight positions of each of 64 previously tested 64-token passages: 512 positions. This is post-hoc diagnostic reuse, not a new independent held-out result. They are fractions of squared hidden-state error, not token-error rates.

The exact decomposition

Let Δ = F(H) − H be the native pair update, and let the mesh predict Δ̂ = b + Dg(H). With orthonormal columns Q spanning D, write P = QQᵀ. Token by token:

‖Δ − b − Dg‖² = ‖(I − P)(Δ − b)‖² + ‖P(Δ − b) − Dg‖²

The first term is error outside the current output subspace. The second is coordinate-prediction error inside it. The identity is checked numerically in float64, including rank-deficient synthetic cases. It constrains the current fixed decoder, not every trainable basis of that rank.

Wider output spaces recover more of the native update

We fitted nested output bases on 256 training passages × 32 tokens only. The table shows the teacher-assisted projection residual ‖Δ − b − P(Δ − b)‖/‖Δ‖ on the reused diagnostic passages. Each numerator and denominator is aggregated before division. Lower is better.

These projections receive the true target update at every token. They are teacher-assisted oracles, not deployable predictors. They minimize squared endpoint error in a fixed affine subspace; they are not upper bounds on next-token agreement.

What actual learned predictors achieved

We fitted 18 floating predictors: three ranks per pair (the original 128/64, then 512 and 1024), each with a smaller token-only linear control, the original causal features, or same-size multiscale features. The “linear” control retains clipping at the encoded inputs and output coefficients; it is not globally linear. The multiscale alternative replaces the original context summary with exponential memories at decay 0.5, 0.875 and 0.96875 and a binding term. Both nonlinear feature sets have 6r+1 components; the zero internal stage is omitted algebraically. No gradient updates were used: these are train-only SVD/ridge fits.

Wider nonlinear fits at ridge 10 often overfit. A bounded follow-up fitted 24 additional models using ridge 80, 800 and 8000; each rank/feature family selected among those and the ridge-10 baseline by validation endpoint error. All 42 artifacts replay exactly. Native-suffix evaluation covered 40 distinct cases in 50 executions, including 10 exactly repeated controls. Native controls reproduce Qwen’s final states exactly.

The rank-1024 linear predictor for 15–16 reduces validation KL by 33.89% versus the original MLX mesh, with the same 122/128 agreement. Its paired article-bootstrap KL difference is −0.01088, descriptive 95% interval [−0.01689, −0.00569]. It is a 46.17 MB floating surrogate file, not a measured compiled-HDC payload. A complex operator mesh has not established an advantage over this linear control.

For 11–12, the lowest-KL learned candidate is the regularized rank-1024 mesh: 22.45% lower KL, but agreement falls from 123/128 to 121/128. This is a quality trade-off, not an unqualified improvement. The teacher-assisted 15–16 oracle reaches 126/128, but its coordinates are supplied from the teacher rather than learned.

These are post-hoc candidate comparisons on a repeatedly used validation set. The intervals do not adjust for model selection or multiple comparisons. The diagnostic’s FP64 original-mesh control differs slightly from the historical MLX float32 path; separate original-MLX controls reproduce the earlier validation scores, and all percentage improvements above use those MLX controls.

What this changes in the test program

Both representation and prediction matter. The tiny original bases leave most endpoint error outside their output space; widening creates headroom, but weakly regularized nonlinear predictors fail to generalize. The next target is a predictor that beats the linear control under a frozen quality and payload budget, followed by fresh article-disjoint confirmation, integer compilation and genuine joint composition.

No new held-out, joint or integer quality result is claimed here. Each new output test replaces one pair, with native-reference prefix states, 62 other native blocks, and native final normalization/head. The earlier joint integer result remains 451/512 (88.09%). No all-64 experiment was changed; no new compiled payload, speedup or autoregressive capability was measured. Cloud spend remains $0/$25.

Full diagnostic evidence: decomposition, capacity ladder, all fitted candidates, output controls, descriptive intervals and hashes.

Two separated pairs. One actual trajectory.

The frozen selected meshes for blocks 11–12 and 15–16, used together, preserve 451/512 teacher choices (88.09%) with mean output KL 0.05747 on 64 fresh passages from 48 articles excluded from all listed prior corpus manifests. This is an integer-replacement composition test with 60 native transformer blocks remaining, not whole-network HDC inference.

The joint integer mesh misses the predeclared descriptive point-estimate gate (agreement≥95%, KL≤0.05). Its agreement 95% article-bootstrap interval is [85.07%, 90.91%]. This does not establish a broad capability guarantee.

The selected mesh combination has lower KL than the trained/selected bias combination (0.06196). The paired 95% intervals cross zero for both KL and agreement versus the bias control and versus skipping. This test does not establish a reliable mesh advantage over those simpler controls. Agreement is not task accuracy; no autoregressive-generation or long-context benchmark was run.

How much did training help?

The starting meshes were already fitted by SVD/ridge, and the starting biases by averaging training updates. They were not random or untrained networks. This pilot tests whether additional output-guided gradient updates improve them.

Every arm received 16 output-guided updates, one 32-token passage per update, from the same 256-passage pool and with the same sampled order. Mesh learning rate was 1e-6, bias rate 1e-3; objective was native-target KL through the differentiable suffix plus 0.1 times endpoint update-normalized MSE. Native validation KL on 128 positions selected among checkpoints 0,8,16. Selected checkpoints: pair 11 mesh: step 0, pair 11 bias: step 0, pair 15 mesh: step 0, pair 15 bias: step 0. Step 0 means training did not improve the selection score and the original parameters were retained.

These are short training pilots, not full convergence studies. Bias initialization uses the expanded 256-passage mean; mesh initialization reuses the prior 32-passage SVD/ridge fit. Test contexts are 64 tokens, twice the training/validation length, and come from a different SQuAD split. This is a stricter transfer test, not a direct continuation of the earlier 128-position validation percentage.

Composition was not a teacher-state reset.

Native prefix → replacement 11–12 → native blocks 13–14 → replacement 15–16 → native suffix. The second replacement received the actual altered trajectory. The evaluator verifies that isolated pair 15 inputs match the teacher exactly, while all joint pair 15 inputs differ. A fully native control reproduces the collected final states exactly.

Integer lowering: checked through final predictions.

The selected meshes and biases were compiled into pair-specific Q24 artifacts with signed power-of-two input normalization, bounded population/count operations and exact output rescaling. Reloaded fast and literal implementations agree exactly on one actual token per artifact; eight-token probes pass causality and floating-to-integer drift checks. All nested payloads and runtime dependencies are hash-bound. The integer cases below run these compiled artifacts on every reserved passage, including genuine joint composition. Floating cases above retain the training surrogate for comparison. The other 60/62 blocks remain native; composition inside the fully HDC chain remains a separate next gate.

The two mesh artifacts total 4,079,836 bytes versus 850,148,224 bytes for the four native blocks being replaced. They replace four blocks' parameter payload, not the remaining 60 blocks or the entire checkpoint. This is not a measured speedup or a packaged compressed-model release.

The first gradient check halted before training because the smallest perturbations disagreed. A wider perturbation sweep established adjacent-scale agreement without relaxing the 15% derivative-error threshold; failed checks remain recorded. Cloud spend remains $0/$25.

Verified continuation: all 21 cases, selected checkpoints, paired intervals, integer probes and hashes.

Historical: initial validation screen.

The lowest validation output KL is 0.02836, for blocks 11–12 at rank 64. Its teacher agreement is 120/128 (93.75%). This is selection evidence from 16 short passages, not held-out accuracy or a deployment certificate.

Capacity warning: rank 128 fits the training endpoints better on 63/63 pairs, but improves validation endpoint error on only 2/63. This is evidence of a generalization problem in this small-data fitting regime, not proof that the pairs cannot be fused. More capacity alone is not the next answer; training data and regularization need attention.

Why output tests matter: pair 0–1 at rank 64 has the lowest validation endpoint error (18.97%), yet preserves only 23.44% of teacher token choices. A low relative hidden-state error can hide behaviorally important damage; endpoint error alone would select the wrong region.

Does the learned mesh beat simpler controls?

After ranking the meshes, we tested the three lowest-KL distinct pairs (11–12, 14–15, 15–16), plus pair 0–1 as a negative control, on the same validation positions. “Skip” passes x unchanged. “Mean” adds the training-only average pair update to x; it learns only a 5,120-component bias. This is an exploratory selection-set comparison, not independent confirmation.

Every candidate leaves 62 transformer blocks native. Prefix states come from the native reference; candidate execution runs only the unchanged suffix after the injected pair endpoint. No candidate is yet compiled to integer HDC arithmetic, and no two replacements have been tested together.

Same fitting budget. Every pair.

Training uses 32 passages × 32 tokens. Validation uses 16 separate passages × 32 tokens; the last eight positions per passage are scored at the output head. Both ranks use the same seed, train-only activation SVD and ridge coefficient 10, with zero gradient updates. These tiny capacities prioritize follow-up work; failure does not prove a pair cannot be fused.

For pair i, the teacher endpoint is y = B_i+1(B_i(x)). A training-only power-of-two scale s normalizes the input and target: u = x/s. The learned replacement predicts ŷ = s·M(u).

z = clip(Eu − c, −16, 16); z′ = clip(z + Aφ(z), −16, 16); M(u) = u + D·clip(Cφ(z′), −16, 16) + b. Here φ concatenates z, ReLU(z), causal prefix mean, one- and four-token lags, clipped z×prefix-mean binding, and a constant. A starts at zero. This screen fits the other weights by SVD/ridge; it does not yet train the full mesh with gradients.

Endpoint error = ‖ŷ − y‖_F / ‖y − x‖_F. Output KL = mean Σ_v p_teacher(v) log[p_teacher(v)/p_candidate(v)]. Agreement counts matching argmax token choices, not task correctness.

The floating surrogate files are 2,840,774 bytes at rank 64 and 6,052,806 bytes at rank 128. Native pair parameter payload is approximately 425–431 MB. These are different representations: no compiled-HDC size, speedup, or matched-quality compression claim follows from this comparison.

Screen, train, compile, compose.

1. Extend the completed skip/mean controls to more pairs and matched-capacity alternatives; give shortlisted meshes equal output-guided training budgets.
2. Freeze selections before a larger, article-disjoint held-out evaluation and longer-context/generation checks.
3. Compile replacements with explicit deep-state scaling and arithmetic bounds; check reloaded integer outputs independently.
4. Re-evaluate inside the HDC chain, then combine nonoverlapping windows and measure accumulated errors. Isolated success does not imply joint success.

126 comparisons on one small validation set create selection bias. Pair rankings are provisional and depend on capacity, data, numerical representation, and quality tolerance. No pair is labeled intrinsically “unfusable.”

Research log

2026-09-08 · Representation and prediction errors separated

Measured the fixed-subspace error split, fitted 42 floating predictors and ran 40 distinct native-suffix validation cases. Wider linear prediction lowers pair15 KL while preserving its validation agreement; new integer or held-out acceptance remains unmeasured. Full evidence and limits appear above.

2026-09-07 · Frozen training and held-out composition completed

Four arms received 16 updates each. The selected mesh combination agrees on 451/512 held-out choices; mean KL0.05747. Simple controls, paired uncertainty and limited integer probes are reported in the new continuation above. Earlier scan results below remain historical validation evidence.

2026-09-07 · Uniform native-reference scan completed

Collected all 65 boundaries, fitted 126 meshes, and scored every candidate through the native suffix. An unchanged h2 control reproduces the collected native h64 exactly. All 130 trajectory-file hashes verify; all 126 candidate predictions replay exactly after reload and pass a finite-sample future-token perturbation check. Cloud spend remains $0 of the existing $25 cap. The separate frozen all-64 experiment was not changed.

Numbers with their boundaries.

All 126 results, protocol scope, audit and artifact hashes. Previous study: four-block endpoint fusion. These are floating native-suffix screening results, not a no-matmul inference demonstration.