Verace V1: A Multimodal LLM

A standard transformer's core subsystems were designed for machine translation in 2017 and have since been patched, rather than redesigned, to handle long context, retrieval-scale memory, and sparse expert computation. Softmax attention recomputes an O(seq_len^2) score matrix. The KV cache stores one key/value pair per past token, so its memory footprint grows linearly with context length. Discrete routed mixture-of-experts requires shipping tokens across devices to reach their assigned expert, an all-to-all communication step that becomes a distributed-training bottleneck at scale. And every token, regardless of how easy or hard it is to process, runs through the same fixed number of layers.

Verace V1 was designed top-down: start from a target property a subsystem must satisfy (memory cost must not grow with context length; a recurrent state must not drift in norm over long sequences; expert computation must not require cross-device dispatch), then construct the mechanism that provably satisfies it, rather than incrementally regularizing an existing transformer component toward the same property.

Each decoder layer applies three residual sub-layers in sequence, each with its own pre-norm: SSSD attention, then CHAM associative memory, then M-CMoE. Layer iteration is controlled by the ACD Engine, which gathers only the still-active tokens for each layer call and halts tokens independently once their accumulated halting probability crosses a threshold. Every 12th layer also snapshots its input into a cross-layer residual buffer that subsequent layers can attend back into, giving later layers direct access to earlier-layer representations beyond what the residual stream alone provides.

The final hidden state is normed and projected to logits through a tied embedding/LM-head matrix. A Latent Energy Critic separately scores candidate continuations against this hidden state, used both for branch selection during generation and as a consistency term during training. An optional vision encoder patchifies and merges image tokens before splicing them into the leading positions of the token sequence, so images and text share the same downstream decoder stack with no separate vision pathway past that point.

Every parameter update, for every 2D weight matrix in the model, is projected onto the Stiefel manifold by the Unitary Muon optimizer before being applied.

SSSD replaces softmax self-attention with a per-head recurrent state Psi, updated at every step by an exactly orthogonal rotation. Given a skew-symmetric matrix A_t = k_t v_t^T - v_t k_t^T constructed from the current key and value projections, the Cayley transform R_t = (I - 0.5 delta_t A_t)^-1 (I + 0.5 delta_t A_t) is guaranteed orthogonal, because the Cayley transform maps the Lie algebra of skew-symmetric matrices onto the rotation group by a standard, well-known result, not an architecture-specific assumption. The state updates as Psi_t = R_t Psi_(t-1), and because multiplying by an orthogonal matrix cannot change Frobenius norm, ||Psi_t||_F = ||Psi_0||_F holds for every t by construction. delta_t controls how strongly each token's key/value pair rotates the state; when a token is halted by the ACD Engine, delta_t is zeroed, which makes R_t the identity matrix, so a halted token's state stops changing exactly rather than being masked after the fact.

We verified this by direct measurement rather than trusting the derivation: running a forward pass and comparing the final state's Frobenius norm against its identity-initialized starting value (expected sqrt(head_dim) = 8.0 for the tested configuration) gave a final measured norm of 8.0000, a deviation below 1e-4.

CHAM replaces the KV cache with a single fixed-size complex matrix H, held on the unitary manifold (H^H H = I) after every update instead of being appended to. Each step folds the current token's key/value outer product into H through an infinitesimal unitary transformation gated by gamma_t, then re-projects the result back onto the unitary manifold by Newton-Schulz iteration (H <- 0.5 H (3I - H^H H), repeated a small fixed number of times), which converges quadratically to the nearest unitary matrix. Reading queries the current hologram directly: Re(H q_t).

Because H has a fixed shape regardless of how many tokens have been processed, both the memory footprint and the per-token compute cost are independent of sequence length, unlike a KV cache whose read cost grows with the number of cached tokens. As with SSSD, halted tokens have gamma_t zeroed, which stops their contribution to H exactly.

We measured the deviation from the unitary constraint directly after a forward pass: ||H^H H - I|| = 0.000000 at the precision reported, comfortably inside the 1e-3 tolerance the property is checked against.

Standard sparse MoE routes each token to a small number of discrete expert weight matrices, which at scale live on different devices and require an all-to-all communication step to dispatch tokens to their assigned expert and ship results back. M-CMoE removes this step by replacing discrete experts with a shared bank of low-rank basis generators and assembling each token's expert contribution locally: Delta W(x) is a top-K-weighted sum over basis components, where each component's contribution is a low-rank matrix scaled per-token by singular values produced from a small hypernetwork conditioned on x. Because this composition happens as local matmuls against shared parameters already resident on the device, there is no token to dispatch anywhere.

Unlike SSSD, CHAM, and Unitary Muon, M-CMoE does not maintain an algebraic manifold constraint at inference time, so it is checked with a numerical-stability smoke test rather than a conservation measurement: output shape correctness and the absence of NaN values across a forward pass, both of which held in our test run.

ACDE implements per-token early exit in the style of Adaptive Computation Time. At each layer, a halting probability is computed for every still-active token from its current hidden state and accumulated across layers; once a token's accumulated halting probability crosses a threshold, it stops being updated, subject to a minimum and maximum depth. The final representation for each token is a probability-weighted mixture of its per-layer outputs.

The naive implementation of this idea would run every layer over the full batch tensor and mask out halted positions afterward, which still pays the full compute cost and risks leaking information across batch items through shared-tensor operations. ACDE instead gathers only the active token indices for each batch item independently before calling each layer, so a layer's matmuls only ever see the tokens still active for that specific item.

This gives two properties we verified directly rather than assumed. First, batch independence: running the same input item in two different batches produced logits differing by a maximum of 0.00000000 at the precision reported, well under the 1e-5 tolerance, meaning which other items happen to share a batch cannot influence a token's output. Second, correct per-example gathering under early halting: forcing an aggressive halting threshold produced a mean cognitive depth of 2.00 layers (against a maximum depth of 4 in the tested configuration) with correctly shaped, non-degenerate output.

Sampling a single continuation token-by-token commits early to choices that may be locally plausible but globally inconsistent. The Latent Energy Critic scores a candidate latent representation against the prompt with a quadratic energy function, E(x, y) = ||h_cand - W_energy x||^2, where W_energy is a learned linear projection. Lower energy means the candidate is closer to what the projection predicts from the prompt. During generation, this is used to pick the lowest-energy candidate among several sampled continuations at each step (at the cost of one extra forward pass per candidate branch); during training, the same projection scores consecutive hidden states against each other as an additional consistency term in the loss.

Images are patchified by a strided convolution, refined by a stack of pre-norm transformer blocks, then merged in 2x2 groups before a final projection maps them to the language model's hidden dimension. This token merge reduces the number of visual tokens by 4x before they enter the decoder, and the resulting visual tokens are spliced directly into the leading positions of the token embedding sequence, so images and text are processed by the same decoder stack with no separate vision-specific pathway downstream of the encoder.

Every 2D parameter's momentum buffer is projected onto the Stiefel manifold before being applied as the update, via exact SVD-based polar decomposition: given the momentum buffer G with economy SVD G = U diag(S) V^T, the update direction is Q = U V^T, which satisfies Q^T Q = I (or Q Q^T = I for wide matrices) to floating-point precision in a single non-iterative pass, regardless of G's singular value spectrum.

This replaced an earlier iterative Newton-Schulz-based orthogonalization, and the reason for the change is worth reporting as a negative result rather than omitting: traced through the iteration, matrices with small singular values (the common case for real gradient and momentum matrices) do not converge toward an orthogonal matrix. They lock into a permanent two-step oscillation, observed at roughly 0.68 and 1.13 alternating indefinitely, and additional iterations do not resolve it. This affected every non-square parameter, which is most linear layers in this architecture. SVD-based polar decomposition has no iteration and therefore no convergence condition to fail, at the cost of an SVD instead of a few matrix multiplications per step.

We verified orthogonality directly across square, tall, and wide matrix shapes, and separately for a matrix with a deliberately pathological near-zero singular value. Measured deviation from orthogonality was 0.00000902 (64x64), 0.00000557 (128x32), 0.00000517 (32x128), and 0.00000573 for the pathological small-singular-value case, all several orders of magnitude inside the 1e-3 tolerance. A full optimizer step on a rectangular parameter completed without producing NaN values.

Every claim above that concerns a manifold or conservation constraint is backed by an executable test in the repository, and we ran the full suite rather than citing it secondhand: 8 tests, 8 passed. The measured values reported in each section above are taken directly from that run, not from the derivations alone. A separate end-to-end integration test constructs a small model, runs a forward pass, takes one training step with the Unitary Muon optimizer, generates a completion, and runs it through the evaluation harness, confirming the full pipeline is wired together correctly rather than only its individual components.

This is a reference implementation, not a trained model. The repository ships module code, configuration, and the test suite described above; it does not include pretrained weights, and the training script is a minimal, single-step reference for wiring the model, loss, and optimizer together correctly, not a distributed, production-scale pipeline.

Nothing in this paper should be read as a claim about downstream task performance. All test configurations use small hidden dimensions and layer counts (64 to 128 hidden dim, 4 layers) chosen to make the manifold-constraint checks fast to run, not to approximate a deployable model. The evaluation harness's accuracy_score is a case-insensitive substring match against a ground-truth string, explicitly documented as a placeholder correctness metric suitable for exercising the pipeline, not a rigorous evaluation methodology; a real benchmark harness with a legitimate scoring method would need to replace it before any accuracy number from it means anything. Similarly, flop_reduction_factor is the ratio of maximum to actually-used layer depth, a directional proxy for compute saved by early exit, not a measured wall-clock or FLOP-counter result.

The spectral_dim configuration parameter is accepted by SSSD's constructor but not currently consumed by the default Cayley-transform code path; a separate Triton GPU kernel path uses a mathematically equivalent complex-phase formulation driven by a different parameter. Several other constructor parameters (M-CMoE's top_k_components and SiTU-GLU shape parameters, the Unitary Muon optimizer's learning rate and momentum) are hardcoded at their call sites rather than read automatically from the shared configuration object, which we note here because it is exactly the kind of gap between what a config file implies and what code actually does that is easy to miss and worth stating plainly rather than leaving implicit.