Endpoint prediction · SVD initialization · Compiled causal mesh
Four blocks.
One small mesh.
Does behavior survive?
Instead of reproducing each transformer block, predict the representation at the end of four blocks. This experiment removes the inherited projection base entirely and tests how much intelligence the smaller replacement preserves.
AND ITS LIMITS
Small enough is not enough.
New: output-guided training transfers, modestly.
The compiled rank-1,024 mesh now reduces final-output KL by 4.62% versus its unchanged starting weights on 64 fresh passages from 29 articles absent from the three prior experiment corpus manifests. The replacement still contains no inherited teacher projection base. Its payload remains approximately 36.15 MB.
| Compiled candidate | Teacher agreement | Mean KL | Payload |
|---|---|---|---|
| Starting cosine-trained mesh | 255 / 512 · 49.8046875% | 1.562038 | 36,146,756 bytes |
| After downstream-guided training | 259 / 512 · 50.5859375% | 1.489807 | 36,146,755 bytes |
Resampling whole articles 2,000 times gives a 95% bootstrap interval of [−0.10686, −0.03690] for the trained-minus-starting KL change: the measured KL gain survives that uncertainty check. The agreement gain is only 0.78125 percentage points, with interval [−1.0246, +2.7344] points, so a reliable agreement gain is not established. This is one seed and one corpus, not broad capability validation.
This is not a regression from the earlier 57.03% score: that score used a different test set. The valid fresh-test comparison is 49.80% → 50.59%. Both fresh candidates still fail the original absolute quality gates; their endpoint errors are 25.8462% and 26.0000%. We did not run the unfused HDC control on this fresh set, and we do not claim matched-quality compression. These are teacher-forced predictions, not autoregressive generation trials.
The predeclared validation-KL rule selected update 64 after 64 updates, each using one 64-token passage sampled from a 256-passage pool. Validation KL fell 1.53094 → 1.40103 while agreement fell 32/64 → 30/64. Selection was frozen before the fresh test. Integer compilation, literal/count agreement and causality checks pass; trained float-to-compiled drift is 0.08165% of the teacher update on a 512-token probe. No parameters were added.
Interpretation: learned operator fusion can be improved through the frozen model’s output signal without enlarging the replacement. The gain is modest and far from usable quality. This continuation combines KL, an endpoint anchor, additional sampled contexts and a smaller learning rate; it does not isolate the contribution of KL alone. See the verified fresh-test record.
Earlier endpoint-only pilots
The first candidates use approximately 5.44 MB of compiled matrices, scales, biases and metadata. The unfused four-block HDC control uses 3.178 GB, and the original native four-block tensors use 856.4 MB. These are parameter-payload comparisons, not total application memory or measured speedups.
The compiled cosine-trained mesh agrees with the teacher on244 of512 held-out token choices (47.66%). The paired unfused HDC control achieves496 of512 (96.88%). The parameter reduction is real, but this is not useful matched-quality compression.
| Candidate | Teacher agreement | Mean KL | Endpoint error |
|---|---|---|---|
| Unfused HDC control | 496 /512 ·96.875% | 0.004807 | 6.9923% |
| Endpoint MSE mesh | 240 /512 ·46.875% | 2.145500 | 23.2183% |
| MSE + cosine mesh | 244 /512 ·47.65625% | 2.109007 | 23.0632% |
Cosine alignment adds four matching choices and slightly lowers KL in this one-seed comparison. Both meshes miss all three absolute quality gates. The control is rescored in the same batch as the candidates; its96.875% result does not replace the earlier independently batched96.29% result.
| Training objective | Selected step | Endpoint update-relative error |
|---|---|---|
| Endpoint MSE | 100 / 1,600 | 24.0377% |
| MSE + cosine alignment | 300 / 1,600 | 23.8870% |
Cosine alignment also produces a small validation improvement. That does not establish a general advantage. Both arms started from the same SVD/ridge initialization and used the same minibatches, optimizer, capacity and update count.
Larger capacity helps output agreement, but not enough.
The completed rank1,024 follow-up was declared before the rank256 output scores were inspected, based on training-only SVD residuals. It uses the same paired objectives and selection rule. The cosine mesh reaches 292 / 512 choices (57.03125%), with 36,146,756 bytes of compiled payload. This is 98.863% smaller than the unfused HDC payload and 95.779% smaller than the original native four-block tensors. It still fails every absolute quality gate.
| Candidate | Teacher agreement | Mean KL | Endpoint error |
|---|---|---|---|
| Paired unfused HDC control | 496 /512 ·96.875% | 0.004719 | 6.9923% |
| Endpoint MSE mesh | 282 /512 ·55.078125% | 1.479433 | 25.8648% |
| MSE + cosine mesh | 292 /512 ·57.03125% | 1.454096 | 24.8097% |
| Training-only mean update | 33 /512 ·6.4453125% | 7.851096 | 31.8931% |
The mean-update control adds a fixed training-only mean(H₄−H₀) to each embedding; its Q24 bias uses40,960 bytes. The learned mesh substantially beats this constant shift. Larger capacity improves final agreement even though endpoint error worsens. Thus endpoint L2 is not a sufficient stand-in for behavioral fidelity. Cross-rank runs use the same saved benchmark but separate native evaluation batches; each retains its own paired control. These are single-seed results, not an established general scaling law.
The subsequent downstream-guided result appears above. Its short-prefix precursor lowered validation KL by2.9% at step4, then regressed; agreement did not improve. The full64-token follow-up used a ten-times smaller learning rate. The current results do not rule out more moderate compression under the original50% payload ceiling.
This is separate from Experiment 01’s all-64 replacement chain. Here, only blocks 0–3 are replaced. The embedding, remaining 60 transformer blocks, final normalization and vocabulary head stay native.
AND THE MATH
Learn the destination,
not every intermediate stop.
LLM-JEPA adds latent prediction between related views to language-model training. Our adaptation uses the input and four-block exit as the prediction pair. It is predictive distillation with a frozen teacher, not a replication of the paper’s two-view training procedure.
The substituted computation
Written per token in column-vector convention, E maps 5,120 coordinates to256 or1,024 latent coordinates, depending on the run. D restores the native width. All four matrices and both bias vectors are trainable. SVD is fitted only on training activations and output updates; these directions are not established semantic eigenvectors.
Missing lag positions are zero; clip₁₆ bounds values to [−16,16]. The prefix mean and elementwise binding provide context, but they are a restricted alternative to attention—not an assertion that attention is unnecessary.
In the completed endpoint-only pilots, both checkpoints were selected by the same validation endpoint error. Output KL was an evaluation metric in those pilots, not a training term. The new downstream-guided follow-up changes that objective explicitly.
The teacher uses native BF16 execution; gradients use an operations-based floating-point recurrence, checked against the teacher and a numerical directional derivative. Layerwise reverse-mode propagation bounds training memory. Only mesh parameters change. Selection uses native validation KL, and acceptance still requires the reloaded integer artifact.
Training uses floating-point linear algebra. The reloaded deployment artifact uses packed population/count-plane projections, bounded integer features and Q24 interfaces. The optimized CPU count path is checked against literal AND/popcount. This is CPU simulation, not FPGA execution.
AND THE BUDGET
Measure what survives the compression.
- Data: 128 training and 32 validation passages, up to 64 tokens. The held-out comparison reuses the earlier article-disjoint 64-passage four-block benchmark; it is not a newly created benchmark for the research program.
- Payload gate: at most 50% of the unfused HDC four-block payload. All deployed matrix and metadata files count. The retained native suffix is outside this four-block comparison.
- Quality gates: update-relative L2 ≤10%, mean teacher-to-student output KL ≤0.05, and teacher next-token agreement ≥95%. Matched quality also limits KL increase to0.005 and agreement loss to one percentage point against the paired unfused control.
- Freeze: select on validation, freeze weights and compiled payload hashes, then score held-out outputs. Failed gates are reported without changing their thresholds.
- Budget: $25 maximum cloud spending for this experiment. Current cloud spend is $0: training and evaluation use the Mac; no paid instance has been launched.
Teacher agreement is not task accuracy. Endpoint error is not a token-error rate. Payload reduction alone is not evidence of useful compression.
UPDATES
The record, including failed checks.
Same-size mesh, 4.62% lower output KL
Update 64 was selected before fresh scoring. On 64 new passages across 29 articles, compiled KL improves 1.56204 → 1.48981 at unchanged 36.15 MB payload. Agreement improves 255/512 → 259/512, but its article-level confidence interval includes zero. The KL interval excludes zero. Quality remains far below the original target. All checkpoint file hashes, integer audits and 512 position metrics verify; cloud spend remains $0.
Verified gradient; full-context training started
The16-token training-only probe gives native/proxy KL0.000432 and directional-derivative error0.246%. The8step pilot selects step4: validation KL1.90059→1.84638, but agreement6/16→5/16; step8 regresses to2.28655. This is not quality success. At64tokens, a large finite-difference step initially fails the gradient check; two smaller steps converge within0.2%, passing the unchanged15% gate. The64step run starts from the original rank1,024 cosine checkpoint with lr10⁻⁶. Its fresh test remains unscored. See the downstream evidence snapshot. Earlier entries below describe the completed endpoint-only phase.
Paired endpoint predictors trained and frozen
1,600 updates per arm, identical initialization and data ordering within each rank. Cosine alignment improves validation endpoint error slightly; every candidate misses the10% target.
A zero-group scale-reduction bug was caught
Initial compiled outputs diverged sharply from the float surrogate. Zero lag groups exposed an invalid large shift in a shared native reducer. Literal and optimized projections shared that bug, so their mutual agreement was not sufficient. Those initial outputs are excluded from quality claims.
New bounded reducer; unchanged learned weights
The experiment-local safe-v2 path adds explicit sum headroom and defined large-shift behavior. A separate scalar oracle covers zero groups and extreme accumulators. On a512-token validation probe, compiled-versus-float drift is0.0605% for rank256 MSE and0.0594% for cosine, relative to the teacher update. Earlier frozen model artifacts are not modified.
47.66% agreement: small, but not faithful
The corrected cosine mesh matches244 of512 teacher choices, versus496 for the paired unfused control. A roughly99.83% four-block HDC payload reduction does not compensate for this quality loss.
57.03% agreement: capacity helps behavior, not enough
The larger cosine mesh matches292 of512 choices, versus33 for a fixed mean update and496 for the paired unfused control. Its initial training-only output-subspace residual was9.85%, versus17.1% at rank256; this diagnostic did not guarantee generalization. Selected validation endpoint errors are26.5244% for MSE and25.6327% for cosine. Compiled-versus-float drift on the512-token probe is0.0774% and0.0800%, respectively. All four learned candidates fail the unchanged quality gates.
AND SCOPE
A dated, reproducible snapshot.
The source experiment preserves training configuration, frozen weight digests, compiled-file inventories, integer audit results and per-position output scores. Publication-safe summaries appear in the rank256 evidence record and rank1,024 evidence record.
No checkpoint payloads, raw prompts or infrastructure credentials are published here. This page is a dated research record, not a live monitor. The all-64 evaluation and its evidence remain separate.