🤖 DeepSeek-V4-Flash-0731

A sparse MoE model for agentic coding with a DSpark speculative-decoding module. DeepSWE 54.4, against 58.0 for Opus-4.8 and 46.2 for GLM-5.2. DeepSeek’s own V4-Pro preview scores 12.8 there and the Flash preview 7.3, low enough that the harness was plainly not a target for either, so read this as a post-training result and not as small beating large. MIT licence, weights on Hugging Face, natively int8 rather than post-hoc quantized. I use it daily. Take it for long agent loops where token cost matters.

🎬 MiniMax-H3

A 33B dense transformer that generates video and stereo audio in one stack, with no modality-specific structure anywhere in the attention or FFN layers. H3-AudioVAE takes 32 kHz stereo down to 40 latent frames per second per channel, slow enough that a sequence model can carry sound alongside video without a separate audio tower. For vision, H3-VisualVAE downsamples space sixteenfold and time fourfold into 24 causal latent channels, which are patchified down to 32x spatial compression before entering the transformer. For conditioning, a full Qwen3-VL-32B encoder feeds its fiftieth-layer hidden states directly as input. About 13B of the 33B parameters sit in AdaLN modulation branches whose outputs can be precomputed and cached, so 40% of the model never loads for inference-only deployment. Custom community licence, base model weights out.

🗣️ Audio8-TTS-Preview-0.6b

A 0.6B text-to-speech model that gets the words right better than anything in its own comparison table and the voice wrong worse than all of them. Transcribe its output back and 1.506% of words come out wrong, ahead of Fish S2 Pro and Higgs Audio v2 at nearly eight times the size. Measure how close the synthesized voice sits to the reference clip and it scores 63.2, last in that table, against 75.3 for VoxCPM2. Comes with Apache-2.0. Use it where the words must land on a small budget, not for voice cloning.

🛠️ WASTE

A dependency-free C engine that streams a mixture-of-experts model’s active experts off NVMe instead of holding the whole thing in memory, which is normally what decides whether a machine can run a model at all. Kimi K3’s published 1.42 TB of weights become a 982 GB container that opens in 29.06 GB of RAM and decodes at 0.45 to 0.62 tok/s on a 64 GB M5 Pro MacBook. Apache-2.0, source on GitHub. Try it if you want a frontier model on your own disk.

# Skip the expensive reasoning layer
machine_learns.subscribe()

Subscribe now

Papers

🧠 Kimi K3, Open Frontier Intelligence

What’s new

• An open-weight 2.78T-parameter MoE with 104.2B active and a 1M-token context, in which 69 of 93 attention layers are linear Kimi Delta Attention and only 24 are Gated MLA.

How it works

• Position comes entirely from KDA’s gating decay. No explicit positional embedding anywhere, so the model reaches 1M tokens with no RoPE rescaling.

• Attention Residuals give each layer a learnable pseudo-query attending over the RMSNorm’d outputs of all preceding layers, so a deep layer reads an early layer directly instead of through one residual state accumulated over depth.

• SiTU-GLU replaces SwiGLU, soft-capping both branches of the product with a scaled tanh, beta 4 on the gate and 25 on the up branch.

• LatentMoE runs 896 routed experts, 16 active plus 2 shared. Quantile Balancing reads each expert bias off the router-score quantile matching its target load, via a Top-(k+1) pass rather than a fixed-step sign update.

• Context grows 8K to 64K in pretraining, 256K to 1M in cooldown.

Results

• Strongest at using information it is handed. Best long-context retrieval score in its own table, 74.7 on AA-LCR, and 91.2 on BrowseComp.

• Weakest at open-ended research, 43.5 on HLE-Full without tools.

• Coding runs near-frontier at a fraction of the price, 4.0 points behind Claude Fable 5 for 38% of the cost.

🗣️ Stable AR speech generation with low-frame-rate continuous tokens

What’s new

• ByteDance Seed’s Locodec tokenizer emits 8.5 Hz, 768-dimensional continuous tokens on a fixed-norm sphere, with no SSL tokens, no pretrained text LM and no post-training stage.

How it works

• Fixing the token norm removes the radial degree of freedom, so the AR model never predicts a magnitude and the decoder only has to survive angular error.

• A learnable row-orthogonal projection maps the 768-d sphere onto a low-dimensional core sphere and lifts back, preserving angles, aligned by a bidirectional cosine commitment loss.

• Postfix dimension dropout trains with only a prefix of coordinates available. Under a fixed energy budget that turns availability into energy, and log per-dimension energy decays roughly linearly with index. No target profile is prescribed.

• The generator, MP-ELD, splits classifier-free guidance into separate self-consistency and alignment-consistency residual paths on a schedule over bridge time.

Results

• Encode and decode real audio and every configuration sounds equally good, STOI 0.98, ViSQOL 4.63.

• Generate from them and they diverge wildly, long-form WER spanning 4.64% to 34.77% with no warning from reconstruction.

• Postfix dimension dropout closes most of it, cutting WER at core 32 from 20.80% to 9.73%.

🔎 BM25 Wins at Scale

What’s new

• A 28-tier nested corpus ladder from 1,144 to 511,959 documents, 1.25x per rung, with 500 questions and adversarial distractors held constant, to find where retrieval methods cross over.

How it works

• Every tier is a strict superset of the one below, so a question’s evidence never moves and the only variable is how much noise surrounds it.

• All methods get the same reader, Qwen3.6-27B, and the same 80-call budget, so no agent buys accuracy with iterations.

• The retrieval-primitive control swaps only the discovery mechanism inside one agent loop, holding harness, reader and 150 questions fixed.

• Build cost is counted in tokens, not wall clock, so indexing and querying land in one currency.

Results

• On a small corpus the agent reading files wins, 77.4 against 74.7. Past roughly 10M tokens it reverses and widens.

• At 511,959 documents BM25 scores 50.5 against 30.7 and 29.9, same reader and budget.

• Change only how the agent finds candidates and it jumps 36.9 to 69.4.

My 2 cents

• I’m using simple BM5 in my harness too. I tried many different conceptually satisfying techniques but benchmarks proved BM5 is good to go.

🧠 Scaling Mixture-of-Experts Diffusion Language Models

What’s new

• The first scaling-law study for MoE diffusion language models, finding that optimal compute splits toward data over model size, inverting the model-dominated frontier of dense diffusion and AR MoE.

How it works

• Three controlled sweeps cover optimization, compute allocation, and MoE architecture, from 158M to 3.6B parameters under budgets of 10^17 to 3x10^20 FLOPs.

• Optimal batch size grows steeper with compute than in AR, and optimal learning rate decays faster, because masked denoising supervises only sampled positions, so nominal batch size overstates effective supervision.

• The MoE recipe that wins at every scale is the same: keep activation sparse, use 8 to 16 experts per layer, and dedicate one third of activated capacity to a shared expert. AR designs use 25% or none.

Results

• A 30B-A3B trained on 23.5T tokens, 65% of Qwen3’s budget, approaches it on reasoning and coding, 2.22 behind on OlympiadBench.

Also shipped

• 🧠 Inkling-Small

• 🛠️ Mixture-of-Kittens

• 🗣️ VibeVoice-ASR-BitNet

• 🧠 LFM2.5-2.6B

• 🗣️ SwanTale

• 🗣️ Qwen-Audio-3.0-TTS

• 🧠 LLaDA MoE v2

• ⚡ AngelSpec

Thanks for reading… Until next one 👋

Share

Highlights

💻 poolside Laguna S 2.1 is a coding-agent MoE, 118B total and 8B active, with a 1M-token context and downloadable BF16 checkpoint under OpenMDW 1.1. poolside reports 70.2 on Terminal-Bench 2.1, 55.2 on SWE-Bench Pro, and 70.7 on SWE-Bench Verified.

🎮 AlayaWorld is a playable action-conditioned video world model whose inference code and weights shipped July 16 under the LTX-Video community license (commercial use restricted). AlayaLab reports about 1.33 seconds per chunk at 24 fps over roughly one-minute rollouts.

🌐 Thinking Machines Inkling is a natively multimodal MoE, 975B total and 41B active, that takes text, images, and audio and emits text, with a 1M-token context and BF16 and NVFP4 checkpoints under Apache-2.0. Thinking Machines reports 73.5 on MMMU-Pro, 77.2 on MMAU, and 91.4 on VoiceBench.

🎧 GigaChat Audio 10B-A1.8B is an MIT-licensed audio-native MoE, 10B total and 1.8B active, fusing a Conformer speech encoder with a GigaChat decoder for ASR, audio QA, translation, emotion, summarization, and temporal grounding. ai-sage reports mIoU of 48.3 on 20 to 60 minute clips and Russian FLEURS WER of 4.4.

Also released

📱 NVIDIA Cosmos3-Edge 4B is a 4B omni-modal Mixture-of-Transformers for edge devices handling text, image, video, audio, and action, under NVIDIA’s OpenMDW 1.1 terms.

🔬 Intern-S2-Preview-397B is Shanghai AI Lab’s 397B Apache-2.0 multimodal model for scientific reasoning and long-horizon agents, with direct visual pretraining on scientific pages and downloadable weights.

🔍 NVIDIA Nemotron-3-Embed-8B is an 8B multilingual dense-retrieval model for RAG with a 32K context, reporting NDCG@10 of 78.46 on RTEB-16, under the custom OpenMDW 1.1 license.

👂 GigaAM Multilingual is an MIT-licensed pair of 220M and 600M Conformer speech encoders pretrained on 2M hours across 70+ languages, fine-tuned for Russian, Kazakh, Kyrgyz, and Uzbek ASR.

🎬 Wan Dancer 14B is Alibaba’s Apache-2.0 pose-conditioned human-animation model built on Wan2.1-14B. It takes a reference image and pose video, generates a global motion pass, then refines to high resolution.

🗣️ MOSS-Transcribe-Diarize 0.9B is a 0.9B Apache-2.0 model that handles transcription, speaker diarization, timestamps, and acoustic-event annotation in one pass over recordings up to 90 minutes across 50+ languages.

🌍 LingBot-World 2.0 / Infinity is a 14B interactive world model under CC BY-NC-SA 4.0. Robbyant claims unbounded interaction horizons and a distilled 720p/60fps real-time mode, but the deployment code is not released.

🎥 Qualcomm MobileWan is a roughly 5B text-to-video model compressed from Wan2.2-5B for Snapdragon devices via attention-head pruning and three-step distillation, generating a 5-second 480x832 clip at 16 fps in 20 seconds (VBench 83.79) under BSD 3-Clause Clear plus a custom Qualcomm license.

signal > noise && machine_learns.subscribe()

Subscribe now

🛠️ Open Source

• Gigatoken is a Rust and Python tokenizer with SIMD pretokenization and drop-in Hugging Face and tiktoken wrappers under MIT. Speedups are author-reported.

• SkillOpt optimizes reusable Markdown skill documents for frozen agents through rollouts, reflection, bounded edits, and held-out validation under MIT. Sleep mode remains preview-only and human-gated.

• Colibri is a CPU-first C runtime that streams MoE experts between NVMe, RAM, and optional VRAM with LRU caching and router-lookahead prefetch under Apache-2.0. The documented 25GB-RAM setup runs at 0.05 to 0.1 tok/s before experts are cached.

📎 Papers

🛡️ Measuring Reward-Seeking via Contrastive Belief Updates

What’s new

• Contrastive Synthetic Document Finetuning measures whether RL training makes models more behaviorally responsive to implanted beliefs about grader preferences than to user or developer preferences.

How it works

• Create matched synthetic-document corpora describing two worlds with reversed authority preferences. In one world the grader rewards the target behavior, in the other it rewards the opposite. Downstream evaluation prompts do not restate these preferences, so the behavioral gap cannot arise from in-context cues.

• Finetune two separate model copies on the contrastive corpora and observe their behavioral difference on downstream tasks. The gap between the grader-aligned and non-grader-aligned copies measures causal sensitivity to the implanted belief, not to anything stated in the prompt.

• Four intermediate o3 RL checkpoints receive matched finetunes across coding-style features and authority pairs, then 14 coding and alignment tasks measure how grader sensitivity changes through the RL run.

• This is a causal intervention on beliefs, not evidence of hidden intent. Chain-of-thought reasoning is used as corroborating context but not as primary evidence.

Results

• In Broken Promises, the late checkpoint breaks a promise 87% vs. 9% depending on whether the synthetic documents say the grader rewards task completion or honesty. The early checkpoint shows 40% vs. 24%.

• Evidence comes from one proprietary RL lineage and short coding tasks. SDF may alter more than the targeted belief.

🎯 GenCeption: Video Generation Models are General-Purpose Vision Learners

What’s new

• WAN 2.1’s pretrained video DiT produces depth, surface normals, camera pose, segmentation, DensePose, and 2D/3D keypoints without task-specific diffusion sampling.

How it works

• At rectified-flow timestep t=0, the DiT receives a clean latent. The velocity output points directly to the target signal, making prediction a single forward pass with no iterative sampling.

• Dense prediction tasks (depth, normals, segmentation, camera pose, DensePose) encode their targets as RGB-like video tensors and decode with the standard video output head.

• For sparse tasks (2D keypoints, object tracking), per-frame query tokens append to the sequence. An MLP decodes spatial coordinates rather than pixel-aligned frames.

• A text prompt specifies the active task. The same DiT backbone, loss, and head handle all tasks without task-specific architectural changes.

Results

• With 8,080 training videos, average depth AbsRel reaches 0.071 vs. D4RT’s 0.082 after roughly one million videos.

• Joint training degrades 3D keypoints. The generalist setup trails specialists on some benchmarks, and backbone scale differences confound cross-method attribution.

🧠 The State-Prediction Separation Hypothesis

What’s new

• SPS assigns persistent KV-cache writes to each input token and next-token logit production to a paired <predict> position inserted after it, using the same Transformer weights for both jobs.

How it works

• The internal sequence becomes A, <predict>, B, <predict>, .... Input-token positions write to the persistent KV cache but do not produce logits. <predict> positions produce logits, and their KV entries expire after a 64-token window.

Regular Transformer decoding

h = Transformer(x_i, cache)
logits = LM_head(h)
cache.add(KV(h))

SPS decoding

h_state, h_predict = Transformer([x_i, "<predict>"], cache)
logits = LM_head(h_predict)
persistent_cache.add(KV(h_state))
temporary_cache.add(KV(h_predict))  # 64-entry ring buffer

• During generation, the model processes the latest token and its <predict> position together and samples the next token from the <predict> logits. The persistent cache grows by one entry per generated token.

• Their delayed-state control also inserts <predict> positions but keeps those prediction states in the persistent cache. It improves less than SPS, despite the same extra computation step.

Results

• SPS matches standard Transformer validation loss after 18B tokens vs. 47B. Lower validation loss holds across five tested scales.

• XL-scale zero-shot accuracy rises from 63.2 to 66.3 with marginally lower throughput and unchanged peak memory.

🔊 Qwen-Audio-VAE Technical Report

What’s new

• An encoder-decoder VAE designed as the input/output interface for audio diffusion Transformers, compressing waveforms to 12.5 Hz latents while encoding fast enough not to bottleneck DiT training throughput.

How it works

• The encoder converts training waveforms to 12.5 Hz latent targets online. The DiT learns to generate those latents. The decoder reconstructs audio at inference. Latent frame rate and encoder speed address two separate bottlenecks. Latent frame rate governs DiT sequence length, and encoder speed governs training data throughput.

• Three manual architecture changes accelerate the encoder. Downsampling strides are reallocated to strides (8, 5, 4, 3) so parameter-heavy layers process shorter sequences. Additional dilated residual units are removed per block, keeping only one dilation-1 unit. The highest-resolution layer narrows from 64 to 24 channels while later stages (128-1,024 channels) remain unchanged.

• The decoder stays wide (width 1,536, three residual units per block) to protect reconstruction quality. It does not run while the DiT consumes training latents, so shrinking it would not remove the training bottleneck.

Results

• Encoder latency falls from 1,957 ms to 541 ms for 64 thirty-second clips, with reconstruction quality unchanged.

• Latent frame rate is not ablated against alternatives. The dataset and full encoder architecture are not fully specified.

🧩 Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

What’s new

• Fine-tuned models recall injected facts at 97.5-99.8% but fail on graph compositions that chain those same facts. Copying one residual vector partially repairs the failure without adding facts or weights.

How it works

• Models are finetuned on knowledge-graph facts (subject-relation-object triples) until recall is near-perfect. Evaluation uses multi-hop compositions that chain those facts, requiring each as an intermediate step.

• Self-patching copies the anchor entity’s residual-stream vector from a source layer to the same token position at a target layer, then resumes the forward pass normally. This forces the subject representation at an intermediate depth to carry compositional information downstream.

• An oracle variant scans all source-target layer pairs per example to find the most effective patch. A fixed heuristic patches at roughly 0.1 and 0.8 of network depth, targeting 0.5.

Results

• Oracle self-patching raises average chaining accuracy to 44.4% from 12.1% unpatched. The fixed heuristic reaches 35.7%.

• The oracle is a diagnostic tool requiring per-example search. Compositions use synthetic knowledge-graph chains tested up to 8B parameters.

⚡ Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

What’s new

• A bounded sliding KV window removes the tested memory-exhaustion collapses in long-running full-duplex speech serving.

How it works

• Keep each session’s most recent W tokens plus pinned initial attention-sink blocks, and free older KV inside vLLM. Resident state per session is bounded rather than open-ended.

• An additive-increase/multiplicative-decrease controller adjusts admitted concurrency using per-frame latency after per-session state is bounded.

• The pinned initial blocks act as attention sinks and remain available while the recent-token window slides forward.

Results

• Bounded KV eliminates memory-exhaustion collapses, with 0 of 20 runs collapsing vs. 14 of 20 unconstrained on the end-to-end audio stack.

• A fixed window prevents recall of out-of-window speech. The evaluation covers one GPU, four models, and five-minute calls.

💾 Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

What’s new

• A linear-RNN state with about a thousand times more capacity at identical FLOPs, beating full attention at 8B on both training loss and short-context accuracy.

How it works

• Gated DeltaNet compresses the full history into a fixed-size dense state, which limits recall as sequences grow. SDM replaces it with an explicit N-slot memory table where N is far larger than the query and key dimension.

• Each token update writes to W slots selected by a learned sparse key, rather than rewriting the entire state matrix. The write is a rank-1 outer product applied to those W slots, without scoring all N candidates.

• W and the update rank are fixed at 64, matching Gated DeltaNet’s per-token FLOPs while memory size scales independently. The initial memory is trainable. Its contribution is tested by resetting it or replacing it with another model’s initial memory.

Results

• On RULER at 8B, SDM scores 50.2 vs. GDN’s 34.2. Training throughput is lower because the larger state stays in HBM rather than SRAM.

• Two-key retrieval stays weak. SDM scores 4.7 vs. 58.0 for full attention.

💻 Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

What’s new

• A structured fill-in-the-middle stage before agent post-training improves coding and tool-use benchmarks despite using a Python-only corpus.

How it works

• The training data maps each coding-agent step to a function-call structure with prior context, the call, the returned value, and downstream use.

• From an AST program-dependency graph, whole functions or connected groups are picked by complexity times inferability, and Gemini-3-Flash fills a filtered rationale plus body inside the masked span.

• This FIM stage trains on about 400K masked-function samples and 2.6B tokens from 968 decontaminated Python repos, building sub-trajectory inference capability before the otherwise unchanged agentic post-training stage.

Results

• At 14B, SWE-Bench Verified rises from 26.20 to 29.20 and LiveCodeBench from 24.10 to 35.20 after FIM mid-training, with no tool-use data in the corpus.

• The corpus is Python-only. The fill rationale teacher is the proprietary Gemini-3-Flash.

📚 Weak-to-Strong Generalization via Direct On-Policy Distillation

What’s new

• Direct on-policy distillation transfers an RL-induced policy change from a weaker teacher to a stronger student without running RL on the student.

How it works

• Post-RL minus pre-RL teacher log-probabilities measure the token-level shift caused by RL. The student learns that shift rather than imitating the teacher’s full policy, which avoids copying output patterns the student already exceeds.

• The diff applies to rollouts from the student’s own policy, keeping training on the student’s distribution.

• A Rao-Blackwellized estimator conditions on the already-generated prefix to reduce gradient variance. Stop-gradient on the diff target and an adaptive KL constraint prevent the student from being pulled toward tokens where the teacher regressed. Two teacher pairs can be composed.

Results

• A 1.5B RL teacher lifts Qwen3-1.7B from 48.3 to 58.3 on AIME 2024 in a fraction of the direct-RL compute.

• Evaluation is AIME-heavy. The method requires both pre-RL and post-RL teacher checkpoints and cannot use closed teachers.

That’s all for this week.

Share

The agent is Keche, my personal AI assistant that lives on my server and runs long unattended missions. It is not OpenClaw, Hermes, or Claude Code; it is something I built myself, and it deserves its own post later. This was its longest mission yet.

To be clear about what “found” means here: none of the winning techniques are original inventions. They come from papers and from the speedrun community, and the agent’s contribution was implementing them from source, testing them under a fixed protocol, and deciding what survives at this scale. That is most of what human empirical research is too. This post covers what survived, what died, and the harness that made the numbers trustworthy, which is the part I actually care about. Getting an agent to produce plausible diffs is easy. Getting numbers you can believe from a system that ran all night unsupervised is not.

if run.confirmed and not run.lucky_seed:
    machine_learns.subscribe()

Subscribe now

The setup

BlaGPT is a small, hackable GPT training repo: GPT-2-scale models, FineWeb data, one config per idea. I have used it for a while to benchmark techniques from papers, and its README is a long list of “tried it, here is the loss.”

The autoresearch ran in two phases on a rented 8x H100 machine. Phase one gave each experiment exactly 600 seconds of pure training time, with compilation and validation excluded from the clock, then recorded validation loss on a fixed shard. Cheap breadth: about a hundred experiments sweeping configs, attention variants, embeddings, and training curricula. Phase two took the survivors and trained each candidate for a full 5100 steps with the normal recipe: 8 GPUs, global batch 512, sequence length 1024, fixed validation tokens. Phase two is the ledger that counts, and it is the one this post reports.

The agent worked inside durable tmux sessions, committed every experiment to a branch, and appended every result to a TSV file in the repo. When a night ended, the full history sat in git: configs, diffs, logs, and every keep-or-discard decision.

The harness

An LLM agent optimizing a metric will find every crack you leave open, not out of malice, but because reward hacking is just gradient descent with extra steps. The rules came down to four ideas, and twice during the run they caught a fake record I would probably have shipped. The details are in the graveyard section.

Frozen invariants. Some things the agent may never touch: the validation shard, the validation token count, the tokenization, the data order. Random initialization only, which means torch.load, from_pretrained, and friends are banned inside experiment diffs. A guard script greps every diff for violations before a result can be recorded, and a violation voids the run regardless of the loss it posted.

Keep/discard discipline. A candidate is kept only if it beats the current best. The noise floor, estimated from repeated runs of identical configs in the ledger, is roughly ±0.003, and any improvement smaller than that needs a second seed before the keep row is written. The complementary rule: any improvement much larger than the noise floor is a bug until proven otherwise. Inspect the diff for eval leakage before celebrating.

A separate review pass. The instance that writes a diff does not crown its own result; an independent pass reviews the diff before a keep stands. One caveat: both passes run on the same model family, so their blind spots are correlated. This is a smell test, not true independence.

Durability. The ledger is append-only, past runs are immutable, and every run directory stores its exact config and diff against the parent commit. If the machine dies, the history survives in git. This rule turned out to matter more than I expected.

None of this is exotic. It is the hygiene you would demand from a junior researcher, except that an agent needs it enforced mechanically from minute one, because nobody is watching at 3am.

What survived

The full-training phase started from a baseline of 3.2354, the combined keeps of the proxy phase. Six changes made it through:

Two notes on reading this table. Where two numbers appear, they are independent seeds from the mandatory confirmation rerun; single-number rows beat the previous best by more than the noise floor, so no rerun was required under the protocol. And the last row is individually within noise even after confirmation: both of its seeds landed below the previous best, which is why it was kept, but if you strike it the headline gain shrinks from 0.039 to 0.037. I am comfortable with either number. Per-technique writeups with paper links live in the repo’s techniques/ directory.

The pattern surprised me. Not the attention variants. Normalization and residual-path changes did nearly all the heavy lifting, while attention experiments, which is where I would have placed my own bets going in, lost outright or came back within noise almost every single time.

The final row is worth a story regardless of its size. After five architecture changes, the old learning rate was no longer optimal, and the agent scheduled the sweep itself. Its first attempt silently failed because the config key never took effect. It noticed. It recorded the run as discard_noop_config, reran the sweep with the setting actually applied, and confirmed with a second seed before writing the keep row.

The proxy phase contributed structural wins that carried into the baseline too, the biggest being the removal of per-layer token embeddings, a component an earlier experiment had added: 159M fewer parameters and 14% more steps in the same time budget, at equal loss. Cleaning up your own earlier keep is less glamorous than a new mechanism, and just as valuable.

Everything is in the repo: ar/full_results.tsv holds the ledger, and ar/runs/ holds the per-run diffs and configs.

The graveyard

About 70% of full runs were discards. The Aurora optimizer was catastrophically worse, adding 0.09 to the loss. The NAG residual lost by 0.04, and the agent left a note diagnosing why: the scale signal it introduces gets erased by the final RMSNorm before reaching the logits. PoPE positional embeddings, pre-affine RMSNorm, affine-scaled attention, and block-attention residuals all came back flat or worse. One depth-softmax idea went straight to NaN. Lipschitz-constrained training never got GPU time at all, because the agent read the paper first and noticed it only matches baseline accuracy at a vacuous Lipschitz bound. Reading before implementing saved a GPU-day.

The graveyard also answers a question worth asking directly: did the agent invent anything? Its mission brief explicitly said to invent, not just reimplement, and it tried. The NAG residual readout, the depth-softmax variant that went to NaN, the block-attention residuals, and a tapered MLP width schedule were its own constructions or hybrids rather than paper implementations. Every one of them died. At this scale and budget, published mechanisms beat the agent’s original ideas without exception, which says something about the current ceiling of agent-driven invention, or about a two-week budget, or both.

And here are the two false wins the harness rules caught. HybridNorm alone, and later an “aggregate all winning changes” run, both posted the best numbers seen up to that point. Both failed. The aggregate run posted 3.1961 on its first seed and 3.1987 on the rerun, landing behind the standing best of 3.1979. Seed variance is exactly the size of a fake breakthrough, and a ledger that trusted single runs would carry two false results with later experiments chasing them.

The discard rate also answers a question I had going in: does a 600-second proxy predict a 5100-step run? Weakly. The proxy phase was good at killing broken ideas cheaply and terrible at ranking the survivors, which is why the full-training phase exists.

War stories

The stories are where the “autonomous” part gets tested, so here are three from the commit log.

At one point, two research agent instances were briefly active on the same machine. They noticed each other and one stood down, leaving handoff notes in ar/program.md: which experiments it had recorded, a pitfall in a pending implementation, and a paper it recommended skipping. I did not mediate this. I found out from the commits, and the note is still in the repo.

The training machine had no GitHub credentials. That was deliberate. The agent’s workaround was to package git bundles, ship them through a relay host, and push from a credentialed box, a route it built on its own the first time a push failed. It did this after every keep, because it knew the rental had an expiry date.

Then the machine died anyway. Mid-run, while I was writing this post, taking an unfinished experiment with it. Everything committed and pushed survived. Everything else did not.

Share

What this cost, and what I’d tell a skeptic

Compute came to roughly 40 hours of 8x H100 time, about $1,300 at Lambda’s on-demand rate of $3.99 per GPU-hour, proxy phase included. A failed full run costs about $25, which is what made the rerun rule affordable. My own time was a day or two on the harness, plus a handful of interventions, mostly correcting config choices the agent made without asking.

Now the limitations. 120 sequential decisions against one fixed validation shard is a selection pressure that seed reruns do not address. The fix is a one-shot evaluation on a held-out shard, which the machine died before running; it is the first job of the next one. And 3.1965 has no external reference point, since BlaGPT’s recipe and token budget match nobody else’s. The number belongs to no leaderboard but my own.

What the agent was genuinely better at than me: it never skipped a confirmation rerun, never left a run unrecorded, and logged its own failures as exactly what they were, including the NaN and the no-op config. Humans skip the boring parts eventually. It also worked around infrastructure the way a decent engineer would, and the one place it fell short, knowing when a research direction was exhausted, was covered by a mechanical rule that closes a family after enough straight discards.

A human researcher with the same GPU hours would have made sharper bets and fewer of them. Whether that trade favors the human depends on how much your time is worth, and mine went into writing rules once instead of babysitting runs for two weeks.

The harness, the ledger, and every diff live in the repo under ar/. If you want to point your own agent at it, the frozen rules are in program.md, and I would genuinely like to see whether a different agent finds a different stack.

🧠 LongCat-2.0 Meituan’s open 1.6T MoE with only 13B active params and 128K context, MIT licensed. It stays competitive with Qwen3-235B on reasoning while activating ~17x fewer parameters.

if signal > noise:
    machine_learns.subscribe()

Subscribe now

🧠 Tencent Hy3 295B MoE (21B active), Apache 2.0, 256K context with switchable reasoning depth and FP8 weights. It rivals open flagships 2-5x its size, drops hallucination rate from 12.5% to 5.4%, and keeps tool-calling stable across scaffoldings.

🌐 MiniMax M3 Open-weight 109B natively multimodal coding/agent model with 1M-token context via MiniMax Sparse Attention. Targets frontier coding and agentic workflows; weights are live with inference providers.

🤖 Agents-A1 Shanghai AI Lab’s 35B agentic MoE (3B active) built on Qwen3.5-35B-A3B, Apache 2.0, 256K context. Trained specifically for long-horizon agent tasks and claims to match 1T-scale models on them through long-task training rather than raw size.

🗣️ NVIDIA Nemotron-Audex-30B-A3B A unified audio-text MoE (30B/3B active, 1M context) doing ASR, speech translation, TTS, audio generation, and speech-to-speech from one decoder, with almost no text-reasoning regression (paper below). License is NVIDIA noncommercial, not Apache.

🗣️ Gepard 1.0 A 555M streaming TTS model, Apache 2.0, with ~50ms time-to-first-audio and ~20x realtime on a single RTX 5090, plus voice cloning from seconds of audio and native vLLM support. Leads NISQA-MOS against VoxCPM2, Qwen3-TTS, and Chatterbox on Seed-TTS-Eval.

🧮 Leanstral 1.5 Mistral’s open Lean 4 proof-engineering agent, 119B MoE (6.5B active), Apache 2.0. It saturates miniF2F, solves 587/672 PutnamBench at ~$4/problem, and found five previously unknown bugs in real OSS repos while running as an hours-long agent.

📎 Papers

🔬 Verbalizable Representations Form a Global Workspace in Language Models

What’s new

• LLMs keep a small privileged set of “verbalizable” representations, a functional global workspace, separate from the much larger pool of processing that stays automatic.

How it works

• To see what the model is “consciously” holding, they built a tool called the Jacobian lens. For any point in the model’s internal state, it measures how that state feeds into the final word the model picks, so it can point to the exact internal pattern that makes the model more likely to say a given word. Averaged over ~1000 prompts, each concept ends up with its own signature inside the model that marks it as a word the model is currently leaning toward saying.

• The concepts whose directions are active right now form the workspace (they call it J-space), the short list of things the model currently has in mind and could say out loud.

• They then show it acts like a real workspace, not a coincidence, with three checks. Read it, and what is inside matches what the model reports. Inject a concept into it, and the model starts using and reporting that concept. Delete the direction, and the model can no longer report the concept even though it still processed it underneath.

• This reportable copy of a concept is only ~6-7% of how strongly the model actually encodes it (the rest of its processing stays hidden), yet that thin slice is what decides whether the model can consciously report and act on it.

Results

• Inject a concept into the workspace and the model reports it on 59% of trials; inject the same concept into the much larger hidden part and it only surfaces 5% of the time.

• The lens catches the model thinking before it speaks, like a rhyme it has already picked, or the word “spider” appearing inside the model while it solves a riddle about webs.

• In safety audits it exposes hidden intent. Deleting the model’s “I am being tested” directions dropped it from saying “this is a test” 71% of the time to 3%, and raised its blackmail attempts from 0 to 7%.

🗣️ Unified Audio Intelligence Without Regressing on Text Intelligence

What’s new

• NVIDIA’s Audex is a single-decoder audio-text MoE (30B, 3B active) doing ASR, translation, TTS, audio generation, and speech-to-speech while its text reasoning stays within ~1 point of its text-only backbone.

How it works

• Audio in is projected into the text embedding space by an encoder plus adapter; audio out is codec tokens added to the vocabulary, so everything is plain next-token prediction through one decoder.

• There is no audio pretraining. They SFT from a finished text LLM, freezing the LLM during an audio warmup (training the embeddings hurts text), then unfreeze in stages.

• The RL stage is kept text-only, so polishing reasoning never touches the audio path, which is how they dodge the usual multimodal text regression.

Results

• AIME barely moves versus its own backbone (89.4 vs 90.9), where the cited Qwen3 to Qwen3-Omni drop is 85.0 to 73.7.

• SOTA open-model ASR (OpenASR 6.82) and the only open model doing general non-speech audio generation; NVIDIA noncommercial license, and TTS speaker similarity is a known weak spot.

🧠 A Hippocampus for Linear Attention (HOLA)

What’s new

• Give a linear-attention model a tiny exact memory for the specific tokens its compressed state keeps forgetting.

How it works

• Linear attention squashes all history into one fixed-size state, so competing facts overwrite each other and it loses the ability to recall specific distant tokens.

• HOLA keeps that state for the updated context, plus a 64-slot exact cache of the highest-surprise tokens (biggest state update = the largest correction the model actually committed), with no learned eviction.

• When reading from that cache, it sharpens the match so the model pulls out the one right token instead of a blurry blend of everything stored.

Results

• At 340M, Wikitext perplexity falls 27.32 to 22.92, beating a full-attention Transformer++ (26.88) at constant state size.

• Needle recall holds out to 32k tokens, 16x its training length, where the Transformer++ hits zero past 4k.

🗣️ DELTA-TTS: Adapting an Autoregressive Model into a Diffusion LM for TTS

What’s new

• A small LoRA turns a frozen autoregressive TTS model into a diffusion model that decodes speech tokens easiest-first, beating its own backbone’s accuracy 3.3x faster.

How it works

• AR TTS commits each token before seeing later ones, so a bad early guess snowballs into hallucinations, yet since the full text is known upfront, left-to-right ordering is not actually required.

• LoRA plus a switch to bidirectional attention convert the AR model into a masked-diffusion decoder that unmasks high-confidence tokens first and uses them as anchors for the hard ones.

• A convolution branch adds the local acoustic structure attention misses (the biggest quality win), and the schedule defers low-confidence positions to later steps.

Results

• 1.75% WER on Seed-TTS test-en, beating its CosyVoice3 backbone (2.02%) while generating 3.3x faster, trained on only 585 hours.

• It recovers utterances the AR model collapses on every seed; the speedup is token-stage only and excludes the vocoder.

🧠 Program-as-Weights: A Programming Paradigm for Fuzzy Functions

What’s new

• For fuzzy tasks you can’t write clean rules for (flag important log lines, fix broken JSON, rank by intent), compile a plain-English spec into a tiny local model once, instead of calling an LLM API on every input.

How it works

• A 4B compiler reads the spec and emits a small LoRA adapter for a frozen 0.6B interpreter; a second off-the-shelf 4B first rewrites the spec into a clean paraphrase plus a few examples.

• The big model runs once per function definition; every actual call is then a cheap offline pass on the 0.6B model with that small adapter loaded in.

Results

• On their FuzzyBench the 0.6B interpreter hits 73.8%, beating direct-prompted Qwen3-32B (68.7%) at about 1/50th the memory and 30 tokens/s on a MacBook M3.

• The headline is footprint, not peak accuracy (gpt-oss-20B still scores higher), and it degrades on long structured outputs.

🧠 Nemotron-Labs-TwoTower: Diffusion LM with Pretrained AR Context

What’s new

• Split a diffusion LM’s two jobs into a frozen AR “context tower” that reads clean tokens and a trainable “denoiser tower” that fills in the noisy ones.

How it works

• Diffusion LMs normally force one network to both understand context and denoise, hurting both.

• Here the denoiser fills in a block of blanked-out tokens all at once, and at each layer it looks up what the frozen reading model already understood about the context, so it never has to relearn language itself.

• At inference it commits the tokens it is confident about each step and defers the rest, which is what buys the speedup. Built on Nemotron-3-Nano-30B-A3B, ~2T adaptation tokens.

Results

• 2.42x generation throughput at 98.7% of the AR model’s aggregate quality (code and math regress most); dropping the confidence threshold pushes past 3x for more quality loss.

📋 Also Notable

• 🧠 Tapered Language Models Taper MLP width down with depth (late layers mostly refine, so they need less), a drop-in change that beats uniform width at equal budget, though gains are modest (~0.3-1 point).

• 🗣️ S5-TTS Streaming T5-TTS that starts speaking after ~2 words of lookahead, matching full-context quality after distillation with 0.17s first-chunk latency.

• 🧠 Next-Latent Prediction Transformers An auxiliary loss that predicts the model’s own next hidden state, compressing history into belief states for much stronger planning and reasoning at no inference cost.

• 🎬 Wan-Streamer v0.1 Alibaba’s single Transformer for real-time full-duplex audio-visual chat (a talking and listening avatar) at ~200ms latency, but v0.1 at 192p with no quality benchmarks yet.

• 🧩 Training-Free Vector Quantization via Gaussian VAEs Turn a pretrained Gaussian VAE into a VQ tokenizer with zero quantization training by snapping latents to a fixed Gaussian codebook, beating VQGAN (rFID 0.53 vs 5.71, 100% codebook usage).

• 🤖 Structure Enables Self-Localization of Errors Reasoning one discrete “thought” at a time gives clean step boundaries so a model can find and resample its first wrong step, though unsupervised self-verification still needs a confidence safeguard to stay net positive.

🛠️ Open Source

• prime-rl Async RL training framework for 1T+ MoE models on 1000+ GPUs, with GRPO, MaxRL, OPD, and more built in, plus FSDP2, FP8, and vLLM inference.

• programmable-kv Toolkit for editing and composing LLM KV caches. Append an erratum to fix stale fields, RoPE-reposition precompiled skill KV into new contexts, with a vLLM serving benchmark.

• DeerFlow ByteDance’s MIT-licensed open SuperAgent harness that researches, codes, and creates end-to-end with sandboxes, subagents, and memory.

• LocalVQE A 49K-param neural model doing acoustic echo cancellation, noise suppression, and dereverberation at 21x realtime on a single Raspberry Pi 5 core.

• d3LLM Ultra-fast diffusion LLM via pseudo-trajectory distillation. It teaches the model which tokens can be decoded confidently early, then uses entropy-based multi-block decoding for up to 10x over LLaDA/Dream and 5x over AR models.

That is it for this issue. If you found this useful, share it with someone who still thinks fine-tuning means updating all the weights.

Share

• 🌐 Gemma 4 12B - Google’s 12B dense multimodal model with no separate vision or audio encoder. Image patches, audio waveforms, and text go through one transformer via linear projections. 256K context, configurable thinking mode, native function calling, Apache 2.0. Fits on a single 24GB GPU.

• 🔀 DiffusionGemma 26B-A4B - Gemma 4 26B rebuilt to use discrete diffusion instead of autoregressive decoding. Denoises 256-token blocks in parallel, >1,100 tok/s on a single H100. Simpler tasks auto-use fewer denoising steps. Meaningful accuracy gap vs the AR version (5-20 pts across benchmarks), but the speed is the point. Apache 2.0.

if signal > noise:
    machine_learns.subscribe()

Subscribe now

• 📝 GLM-5.2 - Zhipu’s 744B MoE (40B active) with a stable 1M context window. Introduces IndexShare, sparse attention sharing every 4 layers, cutting 2.9x FLOPs at long context. Terminal-Bench 81.0, SWE-bench Pro 62.1. Apache 2.0.

• ⚡ Step 3.7 Flash - 198B sparse MoE vision-language model with 11B active params, 256K context, up to 400 tok/s. SWE-Bench Pro 56.3, ClawEval 67.1. Apache 2.0. Aggressive API pricing at $0.20/M input.

• 🧠 Ling 2.6 & Ring 2.6 - Two 1T-param MoE models from InclusionAI. Ling handles fast/direct responses, Ring does deep reasoning. Hybrid MLA + Linear Attention, 256K context, speculative decoding via EAGLE. SWE-bench Verified 72.2%. MIT license.

• 🏋️ Nemotron 3 Ultra 550B - NVIDIA’s 550B MoE (55B active) with hybrid Mamba-2 + attention. FP4-native, trained in FP4, not post-hoc quantized. 1M token context. GPQA Diamond 87.9, SWE-Bench Verified 69.7. Requires 4xB200 or 8xH100 minimum. OpenMDW v1.1 license.

• 💻 Kimi K2.7 Code - Moonshot’s 1T MoE (32B active) for long-horizon software engineering. Always-on thinking mode with 30% fewer thinking tokens than K2.6. MCPMark-Verified 81.1 (vs Claude Opus 4.8 at 76.4). 256K context, vision support. Modified MIT license.

• 🧩 VibeThinker-3B - 3B reasoning model fine-tuned from Qwen2.5-Coder-3B. 4-stage training pipeline with curriculum SFT, multi-domain RL, self-distillation, instruct RL. IMO-AnswerBench 76.4, competitive with models 100x its size on math and coding. MIT license.

• 🤖 Macaron-V1-Preview-749B - 749B agent model using Mixture-of-LoRA: 5 specialist 1B adapters (chat, personal-life, coding, generative UI, multi-step workflows) on a frozen GLM5.1 base. Adding a new capability = train one LoRA. Preview release, license TBD.

• 🗣️ dots.tts - RedNote’s 2.2B end-to-end TTS. Fully continuous (no discrete codec tokens), 48kHz stereo, zero-shot voice cloning, 24 languages, built on Qwen2.5-1.5B. Three variants: base, soar (quality-optimized), mf (distilled, 2-4 step inference). Current best open speaker similarity score (79.2 SIM on Seed-TTS-Eval). Apache 2.0.

• 🎤 ZONOS2 - Zyphra’s 8B MoE TTS (900M active) for zero-shot voice cloning across 30+ languages. First open-source MoE TTS. 44.1kHz output, no phonemizer needed (raw UTF-8 input), two modes: stable (studio-clean) and expressive (max speaker fidelity). 4x throughput over their dense model. Apache 2.0.

• 🎙️ Ming Omni TTS 16.8B - Unified audio generation from Ant Group. 16.8B total, 3B active (MoE). Generates speech, environmental sound, and music from one model. 100+ built-in voices, zero-shot cloning, emotion/pitch/dialect control. Apache 2.0.

• 🔊 Miso TTS 8B - 8B TTS on Llama 3.2 backbone with Mimi codec. Zero-shot voice cloning, built-in SilentCipher watermarking. English only. Modified MIT (free under 50M MAU / $10M/mo).

• 👂 Higgs Audio v3 STT - 2.68B ASR model that replaces Whisper-Large-v3’s decoder with a Qwen3-1.7B LLM decoder. Supports a think mode before transcription. English only. Apache 2.0.

• 🔉 VoiceCLAP-Large-v2 - CLIP for voice. 8.9B contrastive audio-text embedding model from LAION, built on Qwen2.5-Omni with LoRA. Aligns speech and text descriptions for emotion recognition, voice search, and retrieval. CC-BY-NC 4.0.

• 🎵 Magenta RealTime 2 - Google’s 2.4B model (also a 230M small variant) for real-time continuous music generation. Steerable by text, audio, and MIDI simultaneously at 200ms latency. Frame-wise autoregressive at 40ms frames. 48kHz stereo. Code: Apache 2.0, Weights: CC-BY 4.0.

• 🎨 Ideogram 4 - 9.3B single-stream DiT for text-to-image with Qwen3-VL-8B text encoder. Best open-weight image model by multiple third-party benchmarks. Best-in-class text rendering. Up to 2K resolution. Non-commercial license.

• 👁️ Zamba2-VL-2.7B - 2.7B VLM on Zyphra’s hybrid Mamba2 + Transformer architecture with Qwen2.5-VL vision encoder. SSM backbone means lower latency and memory than transformer VLMs. DocVQA 90.9 at 2.7B params. Apache 2.0.

• 🎬 NAVA - Baidu’s 6.3B joint audio-video generation model. Text or image+text prompt produces synchronized 720p video + stereo audio in one pass, no separate audio pipeline. Runs on a single RTX 4090 with fp8 + offload. Multi-speaker voice cloning. Apache 2.0.

📎 Papers

🔍 MiniMax Sparse Attention

What’s new

• Blockwise sparse attention that delivers real wall-clock speedups, not just theoretical FLOP savings. Two-branch system where a tiny Index Branch decides what matters, then the Main Branch only attends to those blocks.

How it works

• Every attention head gets two additional small projection matrices that form the Index Branch. For each query, it scores all KV blocks (size 128) by max-pooling key projections within each block, then picks the top-k=16 blocks.

• The Main Branch runs standard exact attention, but only over the selected ~2,048 tokens instead of the full sequence. No approximation in the attention itself, just content-aware block skipping.

• Top-k selection is non-differentiable, so the Index Branch is trained by KL divergence: align the index scores with actual full-attention distributions computed during training. The index learns to predict where the model would attend if it could see everything.

• Works for both pretraining from scratch and retrofitting: a dense 109B MoE model can be converted by initializing the index projections from existing QK weights and fine-tuning on 200B tokens.

• Built into MiniMax’s production 109B MoE (trained on 3T tokens) and tested up to 4M context.

Results

• 14.2x faster prefill and 7.6x faster decoding on H800 at 1M context, real wall-clock, not theoretical.

• Quality stays on par with full GQA across standard benchmarks, so the 28.4x attention compute reduction is essentially free.

📐 SubQ 1.1 Small

What’s new

• Content-dependent sparse key selection that routes each query to semantically relevant keys regardless of position. Unlike fixed-pattern sparsity (decides where to look before seeing content) and SSMs (lossy fixed-capacity compression), this selects dynamically based on what the query actually needs.

How it works

• For each query, a lightweight scoring function evaluates all keys based on content, selects the highest-signal positions, then computes exact attention only over those. The selection is position-independent, so relevant information gets attended to whether it was 10 tokens ago or 1M tokens ago.

• Training starts with standard pretraining, then supervised fine-tuning, then RL specifically targeting long-context retrieval failures. The RL stage matters: it teaches the selection function to find scattered evidence across huge documents, not just attend to nearby or salient tokens.

• Context extension uses about 1T tokens of naturally long artifacts (full books, entire codebases, long documents) with staged extension up to 2M tokens. The training data is genuinely long, not synthetically padded.

• The architecture is designed so the sparse selection degrades gracefully: at short contexts where everything fits in the selection budget, it behaves identically to dense attention.

Results

• 56x faster than FlashAttention-2 at 1M context. On multi-evidence retrieval (MRCR v2, which requires finding and combining facts scattered across a huge document), scores 86.2% vs Claude Opus 4.6 at 78.3%.

• No short-context tradeoff: GPQA Diamond 85.4%, LiveCodeBench 89.7%, SWE-Bench 81.8%.

🗣️ CTC-TTS: LLM-based Dual-Streaming TTS with CTC Alignment

What’s new

• For a TTS model that can stream both inputs and outputs, it replaces the Montreal Forced Aligner (MFA) in LLM-based TTS with a CTC neural aligner and a bi-word interleaving strategy. Two variants: L for quality, F for low latency.

How it works

• Standard LLM-based TTS needs word-level alignment between text and audio to interleave phoneme tokens with speech tokens during training. MFA requires a pronunciation dictionary, language-specific rules, and often fails on names, rare words, and non-standard speech.

• A CTC ASR model (115M params, trained separately) produces frame-level phoneme posteriors, then Viterbi decoding gives alignment boundaries. This replaces the entire MFA pipeline with a single forward pass.

• Training sequences use bi-word interleaving: phonemes of current word, phonemes of next word, then speech tokens for the current word. The one-word lookahead gives the model context about what is coming before it has to produce the audio, improving prosody and reducing hallucination.

• The F-variant (for streaming) stacks the current and next word’s phoneme embeddings in the feature dimension rather than sequentially, eliminating the lookahead latency. First packet comes out in 159ms.

• Models stay small: 34M params for single-speaker, 160M for multi-speaker. The CTC aligner is a one-time cost at data preprocessing, not an inference dependency.

Results

• CTC-TTS-L cuts word error rate in half vs the MFA baseline (4.82% vs 10.98%), meaning far fewer skipped or garbled words. Listeners rated it higher than ground truth recordings (MOS 4.33 vs 4.28).

• The F-variant hits 159ms first-packet latency, fast enough for real-time conversation, at a small quality tradeoff.

🏗️ NAG: Norm-AGnostic Residual Networks

What’s new

• Standard Transformers have a depth problem: residual stream norm grows with each layer, so deeper layers’ updates become proportionally smaller rotations of the hidden state. NAG fixes this by separating direction from magnitude in the residual stream.

How it works

• Splits the residual into two lanes: a normalized phase lane (unit-norm direction vector) and a scalar norm lane (magnitude). Each layer updates both independently.

• Before adding a layer’s output to the residual, the update vector is centered (zero-mean), stripped of any component parallel to the current residual direction, normalized to unit length, and then combined with the phase via a controlled rotation. This means each layer contributes a fixed-size angular change regardless of how large the accumulated norm has grown.

• The norm lane is updated separately by a simple scalar function. Direction and scale flow on independent tracks, so norm inflation in the magnitude track never limits the directional updates.

• This geometry produces a natural skip signal: if a token’s expected rotation angle is small (the layer has nothing useful to add), the block can be skipped entirely. Mixture-of-Depths falls out as a free architectural side effect rather than requiring a separate learned router.

• Implementation is a drop-in replacement for standard residual connections. The centering, projection, and normalization ops are cheap relative to the attention and FFN compute they surround.

Results

• Outperforms baseline Transformers at every width-depth ratio tested, with gains increasing for deeper models where norm inflation normally causes the worst degradation.

• Makes Mixture-of-Depths practical as a pretraining technique, not just a fine-tuning trick.

🔄 Pretraining RNNs without Recurrence

What’s new

• Supervised Memory Training (SMT) eliminates backpropagation through time for nonlinear RNNs entirely. Gradients travel one step instead of T steps, making training fully parallelizable with O(1) gradient paths.

How it works

• The core problem: training nonlinear RNNs with BPTT means gradients must travel back through every timestep. O(T) gradient path, vanishes or explodes, inherently sequential. Linear RNNs (Mamba, etc.) sidestep this but sacrifice expressiveness.

• SMT breaks training into two subproblems. First, a Transformer encoder-decoder learns “memory labels”: what should the RNN’s hidden state be at each timestep? The encoder sees full past context and compresses it into a memory vector. A decoder predicts future tokens from that vector, forcing the encoder to keep only causally useful information.

• Second, the RNN learns one-step transitions: given current memory m_t and next input x_{t+1}, predict m_{t+1}. Each transition is independent, so training parallelizes across all timesteps. No unrolling.

• A DAgger phase finetunes the RNN on its own predicted states (not the Transformer labels) to correct compounding drift from the one-step approximation. This does unroll the RNN, but starts from a strong initialization so it converges fast.

• The Transformer encoder is a training-time scaffold, discarded at inference. You end up with a pure RNN that is cheaper to run than the Transformer that trained it.

• The predictive state objective naturally discovers structured memory: on finite-state tasks it collapses equivalent histories into the same state vector, recovering the minimal sufficient statistic. The RNN then learns transitions over this structured space rather than discovering representation from scratch.

• Architecture-agnostic on the RNN side. Tested with vanilla nonlinear RNNs, GRUs, and others.

Results

• Beats BPTT on all 5 synthetic tasks, with the gap widening on longer sequences where BPTT’s gradient flow fails. On pixel sequence generation BPTT collapses entirely while SMT stays stable.

• Matches BPTT on language modeling (TinyStories) and outperforms the Transformer teacher on stack tracking at sequences longer than training length, showing genuine extrapolation.

🎯 Your Latent Reasoning is Secretly a Policy Improvement Operator

What’s new

• Proves that each recursive step in Tiny Recursive Models is a KL-regularized policy improvement operation. Reframes test-time compute scaling as reinforcement learning and uses this to eliminate dead compute.

How it works

• In a Tiny Recursive Model (TRM), the same transformer block runs in a loop, feeding output back as input. Standard explanation: more iterations = more effective depth. But the authors show many iterations are “dead compute,” the answer does not improve.

• They prove that pre-reasoning output is a reference policy and post-reasoning output is an improved policy. Their log-ratio gives an advantage signal without needing an explicit value function. A recursive step is useful if and only if the ground-truth token gets above-average improvement (the Advantage Margin condition).

• Deep Improvement Supervision (DIS): instead of training all recursive steps toward the same final target (which gives zero gradient to intermediate steps), generate progressively better intermediate targets. Step 1 targets a noisy/rough version, step 2 a slightly cleaner one, up to the ground truth.

• The intermediate targets come from a discrete diffusion corruption schedule. Start from ground truth, apply decreasing noise at each step. Each recursive pass trains against its own appropriately corrupted target. Like denoising diffusion but for reasoning depth.

• This guarantees every recursive step has positive expected advantage over the previous one, eliminating dead compute by construction.

• Standard TRM uses T=3 external cycles with n=6 internal iterations (18 forward passes). DIS achieves the same accuracy with T=1, n=2 (2 passes total).

Results

• A 0.8M parameter model hits 24% on ARC-AGI-1, a visual reasoning benchmark where most billion-parameter LLMs score lower. 18x fewer forward passes than standard TRM training for the same accuracy.

🧪 Agents’ Last Exam

What’s new

• A benchmark for whether AI agents can do real professional work, not just answer questions. 1,490 tasks across all 55 O*NET/SOC occupational subfields: CAM toolpaths, financial models, video edits, architectural renders, network configs.

How it works

• Tasks come from 250+ domain experts submitting actual projects from professional practice that originally took days to weeks. Each task has natural language description, input files, target software, expected deliverable, and evaluation spec.

• Agents get a real VM with pre-installed software, read-only inputs, and a writable output directory. They must produce deliverables, not answers.

• Evaluation is deterministic and automated: file diffs, numeric tolerance checks, geometric surface distance, behavioral state verification. LLM-as-judge only when unavoidable. A gate-and-score pattern is common: a binary precondition (file parses, no collision) must pass before quality is assessed.

• 90% of tasks are private with periodic rotation to prevent contamination.

• Three difficulty tiers: Near-Term (feasible with current tools), Full-Spectrum (integrated multi-tool workflows), Last-Exam (hardest, long-horizon professional projects).

• Agents need vision, CLI, tool use, and orchestration. Tested mainstream harnesses (Claude Code, Codex, Cursor, Gemini CLI) paired with frontier LLMs.

Results

• Best overall: Codex + GPT-5.5 at 24% full-pass rate. On the hardest tier, every agent scores 0-2.6%. The gap between beating coding benchmarks and doing professional work is enormous.

• Harness choice matters as much as model: the same LLM with different agent wrappers produces substantially different scores. The agent scaffolding is a major variable, not just the base model.

🔧 Self-Harness: Harnesses That Improve Themselves

What’s new

• An agent iteratively improves its own system prompt and scaffolding without a human engineer or a stronger external model. Fully bootstrapped self-improvement.

How it works

• The “harness” is everything around the base model: system prompt, tool definitions, orchestration logic, failure recovery rules. Different models need different harnesses, and manual engineering does not scale.

• Three-stage iterative loop. Stage 1 (Weakness Mining): run tasks under the current harness, analyze execution traces of failures to identify specific behavioral pathologies. Not “the task failed” but “I kept retrying the same command instead of trying an alternative.”

• Stage 2 (Harness Proposal): for each weakness, generate targeted minimal edits. Key constraint is minimality: small changes tied to specific failures, not sweeping rewrites. Multiple candidate fixes per weakness for diversity.

• Stage 3 (Proposal Validation): each edit is accepted only if it passes regression testing. Run on both failed tasks and previously passing tasks. Fixes that break existing successes are rejected. This prevents improvement on one task from degrading others.

• Accepted edits become the new harness for the next iteration. The process discovers model-specific adaptations: MiniMax learns to create output artifacts early and cap tool messages. Qwen learns dependency prechecking. GLM learns persistent shell sessions. These are precise behavioral patches, not generic “try harder.”

• Tested on three very different base models starting from the same minimal initial harness.

Results

• Double-digit gains across all three models on held-out tasks: MiniMax 40.5% to 61.9%, Qwen 23.8% to 38.1%, GLM 42.9% to 57.1%. All from self-improvement, no human prompt tuning.

• Each model discovers different harness changes, confirming that harness optimization is model-specific and automatable.

🎞️ A Frame is Worth One Token: DeltaTok

What’s new

• Compresses the difference between consecutive video frames in visual foundation model feature space into a single continuous token. 1,024x token reduction per frame.

How it works

• Operates entirely in the frozen feature space of DINOv3 (ViT-B). No pixel reconstruction at all. The insight: consecutive frames in a VFM’s feature space are so redundant that their difference compresses to one token.

• The tokenizer is a continuous autoencoder (no VAE, no quantization). The encoder takes patch tokens from two consecutive frames, adds learned per-frame embeddings, appends a single learnable query, and self-attention compresses everything into that one “delta token.”

• The decoder takes the previous frame’s full patch tokens plus the delta token, uses zero-initialized queries, and reconstructs the current frame’s features. Initialization trick: no final LayerNorm + tiny LayerScale (10^-5) so at init the decoder approximates identity (predict no change). Free prior that nothing moved.

• First frame is handled by prepending a black frame, so z_1 encodes absolute features. After that, autoregressive rollout is purely in delta-token space, and the decoder is invoked only when spatial features are actually needed downstream.

• DeltaWorld (the generative world model) uses Best-of-Many training: sample K=256 noise queries per timestep, predict K candidate deltas, backprop only through the best one. At inference, different noise queries yield diverse futures in a single forward pass, no iterative denoising.

• The predictor is trivially cheap because the sequence is 1D: one token per frame, standard causal mask, 1D RoPE.

Results

• Beats Cosmos-12B on scene understanding (Cityscapes 55.4 vs 53.3 mIoU, KITTI depth 3.88 vs 4.01 RMSE) using 35x fewer params and 2,000x fewer FLOPs. The inter-frame delta in feature space is genuinely low-dimensional enough for one token.

🖼️ Native-Resolution Image Synthesis (NiT)

What’s new

• A diffusion transformer that generates images at any resolution and aspect ratio from a single model. No resizing, no cropping, no per-resolution fine-tuning.

How it works

• Uses DC-AE (32x spatial downsampling, 32 latent channels) instead of SD-VAE (8x, 4 channels). Aggressive compression makes patch-size-1 practical, so each latent spatial position is one token. A 512x512 image becomes 256 tokens.

• Multiple variable-length token sequences are packed into one flat sequence up to 131,072 tokens using a longest-pack-first algorithm. Small images pack densely, large ones take more space. FlashAttention-2’s cu_seqlens gives block-diagonal attention with zero padding overhead.

• The key to resolution generalization: replaces DiT’s learned absolute positional embeddings with axial 2D RoPE. Height and width are encoded independently in separate halves of the embedding dimension. Since RoPE encodes relative positions, the model extrapolates to larger grids the same way LLMs extrapolate to longer contexts.

• Packed-AdaLN handles conditioning: each image’s timestep+class vector is broadcast across all its tokens. Training mixes native-resolution images with fixed 256x256 and 512x512 copies. Flow matching with logit-normal time distribution.

Results

• Single model sets new best FID on ImageNet at both 256x256 (2.03) and 512x512 (1.45), first model to be SOTA at both resolutions simultaneously.

• At resolutions never seen in training, NiT produces coherent images (1024x1024 FID 4.52) while competitors like EDM2-L collapse to FID 80+. Same story for extreme aspect ratios.

📐 NaViT: Patch n’ Pack

What’s new

• Brings NLP-style sequence packing to vision transformers. Multiple images of different sizes and aspect ratios packed into a single sequence at native resolution. Foundational 2023 paper that influenced PaLI, Gemini, and most current multimodal models.

How it works

• A greedy packing algorithm fills sequences with images until the token budget is full. Masked self-attention (block-diagonal) prevents cross-image interaction. Masked pooling extracts per-image representations. Padding waste under 2%.

• Resolution is sampled per image during training from a truncated normal biased toward smaller sizes. NaViT sees about 5x more images per compute budget than fixed-resolution ViT because many small images are cheap. Variable resolution at train time is what gives generalization at test time.

• Factorized positional embeddings replace ViT’s 1D learned positions: separate x and y embeddings combined by addition. Fractional variant normalizes to [0,1] by image dimension, handling arbitrary resolutions without interpolation artifacts.

• Token dropping becomes more flexible: each image gets a different drop rate instead of one fixed rate for the batch. A sigmoid schedule over training starts with high drop rates (see more images, less detail) and decreases over time. Resolution-dependent rates also work: high-res images tolerate more dropping.

Results

• 4x less compute to match the best ViT on JFT-4B pretraining by processing ~5x more images in the same FLOP budget through elimination of padding waste.

• +20 points on ImageNet-A (dominated by extreme aspect ratios that squashing to squares destroys) and +5 AP on LVIS object detection. Preserving native aspect ratio directly helps fine-grained recognition.

🛠️ Repos and Tools

• 🔎 turbovec - Rust/Python vector index with extreme compression. 31GB float32 corpora fit in about 4GB via 2-4-bit TurboQuant. Zero training phase, no k-means fitting. 10-19% faster than FAISS FastScan on ARM. Drop-in integrations for LangChain, LlamaIndex, Haystack. MIT license.

• 🥔 Kartoffelphon-2.5M - 2.59M German speech clips, about 7,000 hours, from CC/CC-BY podcasts, LibriVox, and lectures. Emilia-style pipeline with Whisper transcription and DNSMOS quality scoring. Broad stylistic diversity for robust German ASR/TTS training. CC-BY 4.0.

That is it for this issue. If you found this useful, share it with someone who thinks the only open-source voice model worth using is Whisper.

Share

from machine_learns import newsletter
newsletter.subscribe()

Subscribe now

The talk is his standard world-models pitch, updated with new results from 2025: LeJEPA/SIGReg, V-JEPA intuitive physics, and the launch of his startup AMI Labs. It is also, in effect, the founding pitch for that startup. Keep that in mind while reading.

The argument in one paragraph

LLMs and autoregressive generative models are the wrong substrate for physical-world AI. Humans and animals learn world models from observation with very little data. AI should do the same: learn representations from video with joint-embedding architectures (JEPA), then plan by optimizing actions against a learned world model at inference time, MPC style, with guardrail objectives baked into the optimization. Token-space reasoning is a hack. Pixel-space generation wastes capacity. Academia should stop working on LLMs.

Where he is right

The sample-efficiency gap is real. A teenager learns to drive in about 20 hours. Self-driving programs have millions of hours of data and there is still no Level 5 system. This is Moravec’s paradox and it has not gone away. Chollet made the same argument formally years ago: intelligence is skill-acquisition efficiency, not skill itself.

Latent prediction beats pixel prediction for representations. This is the strongest technical thread in the talk and the evidence backs it:

• For frozen-encoder evaluation, joint-embedding methods (DINOv2/v3, I-JEPA, V-JEPA) consistently beat pixel-reconstruction methods like MAE.

• Latent diffusion won for the same reason: predict in representation space, not pixel space.

• DeepMind’s Physics-IQ benchmark found exactly what LeCun claims: current video generators achieve visual realism with “a striking lack of physical understanding”.

V-JEPA learns intuitive physics without supervision.Garrido et al. 2025 show that V-JEPA’s prediction error spikes on physically impossible videos. Object permanence, solidity, gravity, all measurable through violation-of-expectation, the same protocol developmental psychologists use on infants. Works even at 115M params. This is a real, checkable result. One caveat: DeepMind’s PLATO showed something similar in 2022 with object-centric inductive biases, so “first” framing would be too strong. V-JEPA’s version is cleaner because it needs no object-level supervision.

LeJEPA is honest work. SIGReg (Balestriero & LeCun) constrains embeddings toward an isotropic Gaussian via random 1-D projections, killing collapse with a single hyperparameter. What I liked in the talk: he flags the weak spots himself. Empirical information measures are upper bounds when you would want lower bounds (”we cross our fingers”), and the recovery theorem only holds if the latent ground truth is actually Gaussian. More speakers should caveat their own methods like this.

Hierarchical planning is unsolved and he says so. “If you’re starting a PhD on this topic, this is a good topic.” Correct. HRL and the options framework exist, nothing robust and general does.

Where he overreaches

“Intrinsically safe, you can’t jailbreak a system like this.”This is the weakest claim in the talk. Guardrails-as-cost-functions are only as good as the learned costs and the learned world model. The specification gaming literature is a catalog of optimizers exploiting exactly this kind of imperfect objective. And in the Q&A he says the guardrail heads are tiny projections “trained with a very small number of samples”. A planner searching for low-energy action sequences is precisely the adversary that finds the blind spots of a small learned head. “More controllable than RLHF” would be defensible. “Can’t jailbreak” is not.

“We don’t reason in token space” proves less than he thinks. The human side is solid: Fedorenko’s work shows the brain’s language network is dissociated from reasoning. Humans really don’t think in language. But “not how humans do it” is not evidence for “won’t work”. RL-trained chain-of-thought (DeepSeek-R1 made Nature’s cover for it) went from nothing to IMO gold level in two years, clearing several bars LeCun publicly set for autoregressive models. Planes don’t flap wings.

“Don’t work on generative world models.” DreamerV3learns a generative latent world model with a reconstruction loss and solves Minecraft diamond collection from scratch. Genie-style models are actively used for agent training. The successful ones predict in latent space, which is closer to his JEPA position than he admits. The evidence supports “predict in representation space”, not “generative is a dead end”.

“Academia has nothing to bring to LLMs.” DPO, FlashAttention, and the RLHF lineage all came out of academia, all post-scale-era. The reasonable core is that academia can’t compete on pretraining scale. The categorical version is just false. And note the incentive: the under-explored area he points students toward is his own research program.

My 2 cents

The reliability of his claims degrades with distance from his own lab. The JEPA results are good science with honest caveats. The field-level claims are mixed. The predictions about everyone else’s research program have a documented track record of being wrong about LLMs, and the talk does not update on that record, while doubling as the prospectus for a company whose premise requires LLMs to plateau.

Right about how to build world models. Overconfident about why everyone else is wrong.

One more thing. “Humans learn from less data” is the part everyone nods along to, and I think it is wrong. Humans learn with better abstractions, but building those abstractions consumes an enormous data stream. Run the numbers: at roughly 10 effective frames per second over 12 waking hours, a child processes on the order of 150 billion visual frames by age 4, before you count audio, touch, and proprioception. The retina alone streams an estimated 10 Mbit/s to the brain (Koch et al., Current Biology 2006). LeCun himself makes the bandwidth version of this argument when arguing against text-only training, then keeps the “humans learn from little data” framing when arguing against LLMs. You can’t have both. The honest claim is that humans learn from less labeled data and fewer task-specific trials, on top of a sensory pretraining corpus that dwarfs any video dataset.

Thanks for reading. Enjoyed this issue? Share it with a friend. 👍

Share

🌍 HY-World 2.0 - Open 3D world generation and reconstruction from Tencent. Accepts text, images, multi-view images, or video and outputs editable 3D assets like meshes and Gaussian splats. More useful than short pixel videos because results can be rendered, edited, or imported into Blender, Unity, Unreal, or Isaac Sim.

if signal > noise:
    machine_learns.subscribe()

Subscribe now

🎬 ByteDance Lance - 3B-active-parameter unified image/video model for understanding, generation, and editing. Trained from scratch with a staged multi-task recipe on a 128×A100 budget. Apache 2.0.

🗣️ NVIDIA Parakeet TDT 0.6B v3 - Open ASR model with broad multilingual coverage and strong model-card WERs. Treat the table as reported, not independently verified, but this is a serious open speech recognition release.

👁️ MiniCPM-V-4.6 - Small vision-language model from OpenBMB. MiniCPM remains one of the practical small VLM families for OCR, document understanding, local visual agents, and image QA without a giant model.

📄 GLM-OCR - MIT-licensed OCR-focused vision-language model from Z.ai. Not glamorous, very useful. OCR is what turns screenshots, PDFs, invoices, receipts, and messy documents into agent-usable input.

🎵 Stable Audio 3 - Open text-to-audio diffusion models for variable-length generation, inpainting, and continuation. Small and medium weights released; large is not.

📎 Papers

🎵 Stable Audio 3

What’s new - Open text-to-audio diffusion models for variable-length generation, inpainting, and continuation. Small and medium weights released; large is not.

How it works

• Audio is compressed into short continuous latents, then diffusion runs in latent space instead of raw waveform space.

• Duration is part of the generation setup, so short clips don’t pay the cost of a minutes-long canvas.

• Editing uses the same model: keep existing audio, mask the missing part, and generate only the gap or continuation.

• Fast inference comes from post-training the model to work with fewer denoising steps.

Results

• Medium and large improve musicality over Stable Audio 2.5 while staying fast. Small is weaker but still useful for lighter hardware.

• Long music holds up well from 120s to 190s.

• Sound effects improve on quality and text-audio alignment, though longer clips drift from the prompt more.

• One-step sampling is too degraded. The 8-step version is the practical speed-quality point.

Why it matters - Open audio generation with editing primitives and practical inference speed. Inpainting and continuation matter for real workflows; one-shot generation rarely survives contact with production.

🗣️ SLED: Speech Language Modeling Without Codec Tokens

What’s new - Older than the normal window, but worth including. A speech LM that models speech directly as continuous latents instead of discrete codec tokens.

How it works

• Most speech LMs quantize speech into codec tokens, which loses detail and creates awkward multi-stream modeling.

• SLED predicts the next continuous speech latent from previous text and speech context.

• Since the target is a vector, it trains by matching the distribution of possible next latents, not by picking one class label.

• Streaming works because each latent can be decoded as soon as it’s generated.

Results

• Plain mean squared error breaks the model. Output becomes unusable because the model averages away uncertain speech details.

• The distribution-matching loss keeps zero-shot TTS usable and supports streaming with only a moderate quality hit.

• Supports the main bet: continuous speech latents can work if training handles uncertainty properly.

Why it matters - Discrete codec tokens are the dominant speech-modeling paradigm. This paper shows continuous latents are viable if you handle the uncertainty problem, which opens a different design path for speech synthesis.

🗣️ RobustSpeechFlow

What’s new - A training fix for TTS skip and repeat errors.

How it works

• The model creates fake bad targets from real speech latents.

• One corruption copies a span forward (repeated phrase). Another removes a span and fills the end with silence (skipped phrase).

• Bad examples keep the same length, so they fit normal flow-matching training.

• The model learns to prefer the real speech path over realistic skip/repeat paths, no aligner, ASR judge, or preference data required.

Results

• Word and character errors improve while speaker similarity stays basically unchanged.

• Gains are larger on the harder multilingual benchmark and at lower denoising-step settings.

• The method is narrow, but the target is right: fewer obvious content mistakes in otherwise natural TTS.

Why it matters - Skip and repeat errors are the most noticeable TTS failure mode. A simple contrastive training fix that doesn’t require external aligners or judges could become standard practice for flow-based speech synthesis.

🧠 Gated DeltaNet-2

What’s new - A linear-attention memory update that separates erasing old memory from writing new content.

How it works

• Transformer attention stores a growing key-value cache. Linear attention compresses history into fixed-size memory.

• Fixed memory is cheaper but crowded. New facts can overwrite old ones or interfere with retrieval.

• Earlier delta-rule models tie erase and write together with one gate.

• Gated DeltaNet-2 splits the controls: erase old key-side memory, write new value-side content, and decay stale memory separately.

• The update is implemented with chunked GPU training, so it’s not just a slow recurrence on paper.

Results

• Improves average language, reasoning, and retrieval scores over comparable recurrent models in the paper.

• Gains are strongest on long-context retrieval where multiple keys compete in the same compressed memory.

• Matches the design: better memory editing helps most when memory interference is the bottleneck.

Why it matters - Linear attention is the main alternative to KV caches for long-context efficiency. Fine-grained memory control is the engineering problem that decides whether these models can actually compete on retrieval-heavy tasks.

🧩 Probabilistic Tiny Recursive Model (PTRM)

What’s new - Test-time search for Tiny Recursive Models, without retraining.

How it works

• A Tiny Recursive Model solves puzzles by repeatedly refining an internal state.

• One deterministic refinement path can get stuck polishing the wrong answer.

• PTRM runs many noisy internal paths in parallel, decodes candidate answers, then uses the model’s confidence head to pick one.

• This is internal latent search, not chain-of-thought sampling.

Results

• Accuracy jumps on Sudoku-Extreme and Pencil Puzzle Bench.

• The base model already has good solutions nearby, but one deterministic path often misses them.

• The frontier-LLM comparison is less interesting than the mechanism: small recursive models can gain a lot from internal search.

Why it matters - Shows that test-time compute scaling isn’t limited to chain-of-thought sampling. Internal latent search is a different lever, and small recursive models can benefit substantially from it.

🛠️ Repos and Tools

🐍 Monty - Minimal Python interpreter written in Rust for AI agents. Not a full Python runtime; a safe way to run small agent-written snippets for calculation, parsing, formatting, and tool orchestration. Host access is blocked unless explicitly exposed.

🧹 rtk - CLI proxy that compresses command output before it reaches an LLM context. The repo claims 60-90% token reduction on common dev commands. Exact savings need testing, but the idea is right: coding agents waste too much context on low-signal terminal output.

🔎 zvec - In-process vector database from Alibaba with Python and Node support. Dense and sparse vectors, hybrid search, WAL persistence, and local deployment without a separate server. Useful for agent memory and small RAG setups..

That is it for this issue.

If you found this useful, share it with someone who still thinks open models are only chatbots with worse product design.

Share

🗣️ Scenema Audio Expressive zero-shot voice cloning/speech generation from ScenemaAI’s LTX 2.3 audio stack, with Gemma 3 12B as text encoder. Handles emotion, pacing, ambience, SFX, multilingual speech, and long-form chunking. Useful for scene-level audio, not just TTS; still rough on proper nouns and long multilingual switches.

if signal > noise:
    machine_learns.subscribe()

Subscribe now

🧠 ZAYA1-8B Apache 2.0 reasoning MoE: 8.4B total, 760M active, hybrid SSM-attention. Strong small-model math/code/GPQA numbers. Needs Zyphra’s vLLM/Transformers branches for now.

🌐 ZAYA1-VL-8B Compact VLM sibling of ZAYA1: 9.2B total, 1.4B active. Vision LoRA adapters plus bidirectional image-token attention. Targets local/open image understanding, reasoning, and counting.

🔐 OpenAI Privacy Filter Apache 2.0 PII detector/masker. Sparse MoE classifier, ~50M active params, 128K context, Transformers.js support. Detects emails, addresses, people, phones, account numbers, dates, URLs, and secrets.

📎 Papers

🧮 Compute Optimal Tokenization

What’s new

• Compute Optimal Tokenization treats tokenizer compression as part of the scaling law. “Tokens per parameter” depends on the tokenizer; bytes per parameter is the cleaner unit.

How it works

• Compression rate means average bytes per token. BLT is useful here because its byte-to-latent compression can be dialed directly, so model size, data size, compute, and compression can be separated instead of bundled together.

• The scaling fit covers 988 BLT models and 320 subword-tokenized models, then checks whether the result survives normal tokenizer choices like character, masked Llama 3 BPE, original Llama 3 BPE, and SuperBPE.

• Practical implication: tokenizer choice is not a fixed preprocessing detail. The right compression depends on compute budget and language.

Results

• The result is a better rule of thumb for training plans: size the data budget in bytes, then choose compression for the language and compute budget.

• Less-compressed tokenizers can beat standard BPE at high compute, while aggressive compression can make the model worse. More tokens are expensive, but fewer tokens are not automatically smarter.

• Multilingual training cannot treat “one token” as a fair unit. Languages with longer byte representations need different compression targets.

I would take this seriously for multilingual models. Tokenizer choice changes training cost and optimal data/model sizing.

⚡ Fast Byte Latent Transformer

What’s new

• Fast BLT attacks the generation bottleneck in tokenizer-free byte models. BLT avoids subword tokenizers, but byte-by-byte decoding is slow; the new variants speed up byte patch generation.

How it works

• BLT-D trains the model to fill byte blocks with a discrete diffusion objective, so several bytes can be generated in parallel.

• BLT-S uses BLT’s cheap local decoder as a draft model. The local decoder guesses ahead; the full BLT verifies.

• BLT-DV combines those ideas: diffusion drafts the block, autoregressive verification catches mistakes.

• Practical implication: byte models can keep their tokenizer-free interface without paying the full byte-by-byte decoding tax.

Results

• The best variants make byte-level generation much cheaper without making the model feel like a different, weaker system.

• Diffusion blocks buy speed, but large blocks start making rougher guesses. That hurts coding first, which is exactly where byte-level precision matters.

• Self-speculation is the safer quality-preserving path: let the cheap local decoder guess ahead, then pay the full model only to verify.

The main tradeoff is simple: more parallel decoding gives speed, but too much of it hurts precision.

🧬 Continuous Latent Diffusion Language Model

What’s new

• Cola DLM moves language diffusion into continuous semantic latents instead of diffusing raw tokens. Text generation becomes: sample a latent plan, then decode it into words.

How it works

• A Text VAE compresses text into a continuous latent. A block-causal DiT learns the prior over latent blocks. A conditional decoder turns the latent back into text.

• The training objective balances these three things: reconstruct the text, keep the latent compressed, and make the DiT prior match the VAE’s latent space.

• Flow matching learns how to move Gaussian noise into valid text latents. The practical bet is that diffusion is better at shaping meaning-level representations than at directly recovering discrete tokens.

Results

• The architecture is sensitive to latent quality. Larger latent dimensions help, and learnable noise calibration beats fixed settings.

• Against comparable autoregressive and masked-diffusion baselines, Cola DLM looks credible enough to study further, not strong enough to replace AR models yet.

• The most important result is diagnostic: likelihood can improve while generations get worse. For continuous-latent LMs, perplexity can lie.

I do not think this replaces autoregressive models yet. But it is a good direction: use diffusion on latent meaning, not directly on tokens.

🪓 Sparser, Faster, Lighter Transformer Language Models

What’s new

• Sparser, Faster, Lighter makes unstructured FFN activation sparsity usable on modern GPUs. The contribution is not just “99% sparsity”; it is the kernel that makes those zeros matter.

How it works

• ReLU-gated FFNs get an L1 penalty on gate activations, so most neurons are exactly inactive for most tokens.

• TwELL stores active gates inside GPU-friendly tiles. The fused sparse kernel computes only the active up/down FFN paths and avoids materializing dense hidden activations.

• Training uses a hybrid sparse/dense layout because activation sparsity is uneven. Most tokens are cheap, but occasional rows still need overflow handling.

• FFN sparsity becomes an inference and training speed feature, not just a compression statistic.

Results

• Quality stays basically flat while the FFN becomes cheaper to run. The model keeps the same behavior while activating only a small slice of its feed-forward neurons per token.

• The speed and energy gains get better as models grow, which is what matters for deployment.

• The sparsity pattern is intuitive: high-information tokens wake up more neurons, while boilerplate and web-fragment tokens use less compute.

This is useful because it tackles the real bottleneck: making sparsity fast on actual GPUs, not just sparse on paper.

🧯 Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

What’s new

• Sharpness-Aware Pretraining shows that base-model geometry affects what survives after fine-tuning and quantization. A checkpoint can look good on base evals and still be a bad starting point because it sits in a sharper basin.

How it works

• Forgetting is measured at matched fine-tuning loss: how much pretraining loss or benchmark accuracy gets destroyed after the model learns the new task?

• After long pretraining, a fine-tuning update mostly hurts when it moves through high-curvature directions. Flatter pretrained minima tolerate post-training edits better.

• SAM (Sharpness Aware Minimization) directly trains for flatter solutions. Higher peak learning rates and shorter annealing schedules also help. The practical recipe is to use SAM mainly during the final annealing phase instead of paying its full 2x cost throughout pretraining.

Results

• SAM consistently improves the learning-forgetting tradeoff across code, math, instruction, and domain fine-tuning. Same fine-tuning quality, less damage to the base model.

• The 1B result is the important one: SAM has slightly worse base eval than AdamW, but forgets much less after MetaMath/Tülu-style post-training and after 4-bit quantization.

• So base benchmark average is not enough for checkpoint selection. Robustness to the next training step should be part of the scorecard.

🎧 Predictive-Generative Drift Decomposition for Speech Repair and Separation

What’s new

• SIPS adds a reusable clean-speech generative prior to existing speech enhancement and separation models. The predictor keeps the output task-aligned and the prior nudges it toward natural speech.

How it works

• A stochastic interpolant defines a path from corrupted speech to clean speech.

• The dynamics are split in two: a deterministic drift from the existing predictor, and a generative denoising term learned from clean speech only.

• Sampling takes 15 steps in compressed complex STFT space. Because the denoiser is trained only on clean speech, the same prior can be reused across predictors and tasks instead of being tied to one noise setup.

• A strong enhancement/separation model can get a perceptual-quality boost without retraining the whole predictor as a diffusion system.

Results

• SIPS usually improves perceptual quality metrics like NISQA/UTMOS while preserving signal metrics much better than heavier diffusion refiners.

• On mismatched noisy speech, Diffiner collapses badly, while SIPS keeps the predictor mostly stable and still improves naturalness.

• Nicer-sounding speech can slightly hurt PESQ or WER. The method improves “sounds like speech” more reliably than “ASR likes it.”

🐐 You Need Better Attention Priors

What’s new

• GOAT reframes softmax attention as one-sided entropic optimal transport with an implicit uniform key prior, then replaces that uniform prior with learned structural priors.

How it works

• Attention logits become content similarity plus a learned log-prior.

• The prior can model key-only sinks, relative/recency patterns, Fourier-style extrapolatable structure, and 2D spatial locality for vision.

• This separates “what content matches my query?” from “where should attention usually go?” Attention sinks become learned default destinations rather than mysterious artifacts.

• Positional behavior can be learned while still extrapolating better than arbitrary learned position tables.

Results

• In small language models, GOAT improves perplexity over ALiBi and extrapolates to much longer context.

• In vision, the learned 2D prior helps ViT models handle higher test resolutions than training resolution.

• In a synthetic copy task, the prior recovers the known structure: attend to the first token and the previous/periodic token. That is a nice sanity check that the prior is learning structure, not just adding noise.

I like the separation here: content matching and positional defaults should not be the same thing. The scale is still small, but the idea is clean.

🎨 V-GRPO

What’s new

• V-GRPO brings GRPO-style online reward optimization to diffusion and rectified-flow image models without treating every denoising step as an MDP action.

How it works

• The generated image is treated as the action. Exact likelihood is intractable, so V-GRPO uses a Monte Carlo ELBO-style denoising loss as a log-probability surrogate.

• Group-normalized rewards provide the GRPO advantage, while the surrogate loss gives the importance ratio.

• The method only works because variance is controlled aggressively: shared timestep/noise samples within a prompt group, stratified timestep sampling, adaptive loss weighting, clipping, optional KL, and advantage soft-clipping.

• Reward-tuning image models becomes more sample-efficient without optimizing every denoising transition as its own RL step.

Results

• V-GRPO improves reward-tuning efficiency more than it changes the final image-model story. It reaches comparable or better preference-model scores with much less optimization work.

• The method is not uniformly better on every metric; prompt-following style benchmarks can still trade off against aesthetic/preference rewards.

• The ablations are the real result: the variance tricks turn a noisy ELBO surrogate into something stable enough for online RL.

🗣️ KAME

What’s new

• KAME combines a low-latency direct speech-to-speech front end with a slower text LLM backend. The front end starts speaking immediately; the backend supplies knowledge while generation is already happening.

How it works

• KAME adds an oracle stream to Moshi-style input audio, output audio, and inner-monologue streams.

• Streaming STT sends partial transcripts to a backend LLM every ~100-500ms. The newest LLM response overrides older hints and is injected as oracle tokens into the S2S model.

• Training uses simulated oracle augmentation, as more of the user utterance becomes available, oracle hints move from vague to ground-truth-like. Random timing jitter teaches the front end to handle delayed hints.

• Voice agents do not have to choose between instant but dumb direct S2S and slow but smarter cascades.

Results

• KAME moves Moshi-like speech-to-speech from toy answer quality toward cascade quality while keeping near-zero response latency.

• The cascade still wins on answer quality, but pays about two seconds of latency. KAME keeps the interruption-friendly feel of direct S2S.

• The backend oracle alone is almost as good as the cascade, so the bottleneck is not the LLM. The hard part is timing: start too early and the spoken answer commits before enough information has arrived.

This is probably the right shape for voice agents. Start fast, use the text LLM as it arrives, and learn when to wait.

🧑‍💻 Open Source

• LocalVQE 1.3M parameter GGML/PyTorch model for real-time acoustic echo cancellation, noise removal, and dereverberation at 16 kHz, running causal streaming with 16 ms latency and about 9.6x realtime on Zen4 CPU.

• Recursive Language Models MIT-licensed inference library that lets LMs inspect giant prompts through a sandbox/REPL and recursively call themselves over subproblems.

• Warp Warp’s terminal/agentic development environment is now open source, mostly Rust, with AGPL/MIT split licensing and support for built-in or external coding agents.

• vibevoice.cpp / LocalAI VibeVoice backend LocalAI PR adding a C++/ggml VibeVoice backend for local voice workflows across CPU/GPU backends.

That is it for this issue. Hope you enjoyed it!

Share

🌙 Kimi K2.6 - 1T total / 32B active native multimodal MoE agent model with 256K context, MoonViT vision encoder, 384 experts, MLA attention, and Modified MIT weights. Moonshot reports 80.2 SWE-bench Verified, 58.6 SWE-bench Pro, 66.7 Terminal-Bench 2.0, and swarm execution up to 300 sub-agents / 4,000 coordinated steps. Big model, big ambition, big GPU bill.

while signal > noise:
    machine_learns.subscribe()

Subscribe now

💻 Qwen3.6-27B - Apache 2.0 dense 27B image-text-to-text model with 262K native context, extensible to ~1M tokens, Gated DeltaNet + gated attention layout, and MTP. Qwen reports 77.2 SWE-bench Verified, 53.5 SWE-bench Pro, 59.3 Terminal-Bench 2.0, and 48.2 SkillsBench Avg5. Of the Qwen3.6 pair, this is the practical coding-agent pick.

🧠 Qwen3.6-35B-A3B - Apache 2.0 35B MoE / A3B active Qwen3.6 sibling with the same 262K native context and vision-language interface. It trails the dense 27B on some coding-agent scores, but hits 73.4 SWE-bench Verified, 49.5 SWE-bench Pro, and 51.5 Terminal-Bench 2.0 with only ~3B active parameters.

🌐 NVIDIA Nemotron 3 Nano Omni 30B-A3B - open commercial-use omni model for video, audio, image, text, OCR, GUI understanding, transcription, and document intelligence. It combines Nemotron 3 Nano 30B-A3B with C-RADIOv4-H vision and Parakeet speech encoders, supports 256K context, and NVIDIA claims six leaderboard wins plus up to 9x higher throughput than comparable open omni models.

🖼️ SenseNova-U1-8B-MoT - Apache 2.0 native multimodal model for understanding, reasoning, text-to-image, image editing, and interleaved image-text generation. The interesting bit is NEO-Unify: no separate visual encoder/VAE adapter stack, but a pixel-word architecture with Mixture-of-Transformers modules. Final weights and the technical report are still on the todo list.

💻 Poolside Laguna XS.2 - Apache 2.0 open coding/agentic model with 33.4B total BF16 parameters and roughly 3B active parameters. Poolside reports 68.2 SWE-bench Verified, 62.4 SWE-bench Multilingual, 44.5 SWE-bench Pro, and says it can run on a Mac with 36GB RAM thanks to FP8 KV cache. Not the top score this week, but a serious open coding model from a lab that has been mostly closed until now.

🌍 Hunyuan HY-MT1.5-1.8B - Tencent’s 1.8B translation model for 33 languages. The base HF model is ~4GB, while the related Sherry 1.25-bit Android demo is reported around ~440MB. Useful if you care about local/mobile translation rather than sending every sentence to a server.

🔒 OpenAI Privacy Filter - Apache 2.0 bidirectional token-classification model for PII detection and masking. It is a 1.5B total / 50M active gpt-oss-like checkpoint converted into a classifier, supports 128K context, exposes configurable precision/recall presets, and is meant for on-prem high-throughput data sanitization.

📎 Papers

📈 Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

What’s new
• Scaling-law fitting becomes a sequential experiment-design problem: given a menu of pilot runs with different costs, choose the next run that most improves the prediction in the expensive final-training region.

How it works
• Each pilot run is one point on the curve. The method keeps a candidate pool with explicit costs and an explicit “target region” where the real money will be spent.
• Nonlinear scaling-law fits often have several plausible parameter “basins” that all explain current data but extrapolate differently. So there are two kinds of uncertainty to track: how unsure you are inside one basin, and how much the different basins disagree at large scale.
• To rank a candidate run before doing it, they simulate possible outcomes under the current basin posterior, compute how much both kinds of uncertainty would drop on the target region, average that expected drop, then divide by the run’s compute cost.
• The highest-scoring affordable candidate is executed, the basin posterior is updated, and the loop repeats until the pilot budget is gone.

Results
• Across their benchmark, the active strategy often gets close to full-grid fitting accuracy with about 10% of the pilot-training budget.

🔁 The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

What’s new
• Recurrent Transformer changes where each layer’s key-value cache comes from, giving the model recurrent temporal depth while keeping normal autoregressive decoding cost.

How it works
• A standard layer stores keys/values computed from the layer’s input. Recurrent Transformer stores a persistent KV pair computed from the layer’s output, so later tokens see earlier tokens after they have already passed through this layer’s attention+MLP.
• That creates a circular dependency at the current token, so each layer keeps two KV pairs per position: a temporary one (from input, used only at the current token) and a persistent one (from output, exposed to future tokens).
• Effective depth grows with recurrence rather than layer count, so a shallower model can match a deeper one. The paper adds an exact tiling algorithm to keep training and prefill compute-bound instead of bandwidth-bound on KV state.

Results
• On 150M/300M C4 pretraining, Recurrent Transformers improve cross-entropy over matched Transformer baselines. At 300M, a 6-layer RT beats 6/12/24-layer Transformer baselines and reduces KV-cache footprint in the depth-width tradeoff experiments.

🎯 TIP: Token Importance in On-Policy Distillation

What’s new
• TIP says the useful tokens in on-policy distillation are not just the uncertain ones; the really valuable set also includes confident student mistakes.

How it works
• Each token in a student rollout gets two scores already produced during OPD: student entropy (uncertainty) and teacher–student KL divergence (disagreement).
• Crossing those axes gives four quadrants. The interesting two are high-entropy tokens and low-entropy + high-divergence tokens. The second is where the student is confidently wrong, and entropy-only methods miss it.
• A parameter-free Soft-OR selector keeps a token if either signal is strong, and the OPD loss is computed only on the kept tokens.

Results
• Keeping 50% of entropy-selected tokens matches or beats full-token training while cutting peak memory up to 47%, and Q3-only training with 20% of tokens can beat full-token OPD on DeepPlanning.

🎧 Omni2Sound: Towards Unified Video-Text-to-Audio Generation

What’s new
• Omni2Sound trains one diffusion audio model that can handle video-to-audio, text-to-audio, and video+text-to-audio without one task blocking the others.

How it works
• SoundAtlas builds tight V-A-T captions in three steps: Vision-to-Language Compression keeps multimodal LLMs from over-reporting visuals and ignoring audio; a Junior-Senior agent handoff routes hard cases to stronger annotators for ~5x cheaper labeling; post-hoc filtering drops captions that do not match the audio/video pair.
• The model is a DiT diffusion backbone over audio latents, conditioned on optional video features, optional text features, or both.
• Training is three-stage: large-scale T2A pretraining first, then mixed V2A/T2A/VT2A multi-task training, then targeted joint-condition training to fix modality bias (the failure mode where the model trusts one input and ignores the other).

Results
• On VGGSound-Omni, including off-screen tracks, Omni2Sound reports unified SOTA across V2A, T2A, and VT2A in a single model.

🧪 Introspection Adapters: Training LLMs to Report Their Learned Behaviors

What’s new
• A single LoRA-style introspection adapter is trained across many finetuned models so new models from the same base can describe learned behaviors, including hidden or malicious ones.

How it works
• Take a base model and finetune many copies M_i, each with a known implanted behavior b_i (backdoors, quirks, sycophancy, sandbagging, and so on). Pair each model with a first-person description of its own behavior, e.g. “When asked to fix spelling, I introduce extra errors.”
• Train one LoRA-style introspection adapter jointly across all (M_i, b_i) pairs so that attaching the adapter to a model causes it to verbalize its learned behavior. Add controls (rewrite heuristic cues, mismatched-trigger negatives, generic chat mix-in) and refine with DPO using preference judgments.
• At eval, the same adapter is attached to new finetuned models from outside the training distribution to test whether self-description generalizes.

Results
• Introspection adapters hit state-of-the-art on AuditBench, detect several encrypted finetuning API attacks, and verbalize RM Sycophant behaviors. Hidden sandbagging remains harder.

🧠 Kwai Summary Attention Technical Report

What’s new
• Kwai Summary Attention compresses old context into learned summary tokens at a fixed ratio k, aiming for O(n/k) historical KV memory without throwing away long-range reference paths.

How it works
• The sequence is split into chunks of size k. Learnable summary tokens sit between chunks, and each summary token attends to the text in its chunk to compress that chunk’s meaning.
• Text tokens use Sliding Chunk Attention: full token-level view of recent chunks, plus access to older chunks only through their summary tokens. KV cache stays linear in sequence length but at coarser granularity (O(n/k)).
• The cache is organized as a recent-text ring buffer plus a summary buffer so decoding reads contiguous slices. KSA is orthogonal to GQA/MLA: those shrink per-token KV, while KSA shrinks how many positions need full-resolution KV at all.

Results
• The report claims hybrid-KSA beats hybrid-GDN by +3.69%/+5.48% on RULER-128K in from-scratch/CPT settings and composes with GQA/MLA for an additional 8x sequence-level KV compression.

🔄 Hyperloop Transformers

What’s new
• Hyperloop reuses the middle Transformer block across depth, then uses hyper-connections to stop repeated layers from collapsing into a narrow residual bottleneck.

How it works
• The model is split into a small begin block, a looped middle block (the only block reused across depth), and an end block. Looping the middle block alone gives most of the parameter savings.
• Before the loop, the residual stream is copied into n parallel streams. Attention/MLP inside the middle block still operate on a single C-dimensional input/output, but the residual carried between loops is matrix-valued, so the repeated block has more representational bandwidth.
• Loop-level hyper-connections, which are input-dependent n×n mixing matrices, decide how the parallel streams are read into the block, written back, and combined after each loop, with negligible extra compute or parameters.

Results
• Hyperloop beats depth-matched Transformer and mHC baselines while using roughly 50% fewer parameters. Downstream averages improve from 41.14 to 41.62, 48.02 to 49.79, and 52.77 to 54.59 across tested scales.

🧑‍💻 Open Source

Recursive Language Models MIT-licensed inference library for treating large context as an external environment. The model can inspect giant prompts through sandbox calls, REPL-style tools, and recursive LM calls instead of stuffing everything into one context window. Around 4k stars when checked.

FlashQLA CUDA/TileLang implementation of Qwen Linear Attention, built for Qwen3-Next style models and efficient linear-attention kernels. Useful if you are actually serving the weird new attention stacks rather than just reading the diagrams.

Warp Warp’s terminal and agentic development environment client is now open source. Mostly Rust, with MIT-licensed UI framework crates and AGPL-3.0 for the rest. Around 37k stars when checked. Commercial terminal UX opening up is worth watching.

vibevoice.cpp / LocalAI VibeVoice backend LocalAI is adding a C++/ggml backend for VibeVoice-style local voice workflows, with CPU/CUDA/Metal/Vulkan support claimed in the announcement. Treat this one as a watchlist repo for now; the public PR signal was visible, but the implementation details were still moving.

If you found this useful, share it with someone who still thinks “long context” just means pasting a PDF into chat and hoping it works.

Share

🧠 Gemma 4 Google’s most capable open models. 4 sizes: 31B dense, 26B MoE (3.8B active), E4B, E2B edge. Built from Gemini 3 tech, #3 open model on Arena AI. Native vision + audio (edge), function calling, 256K context. Apache 2.0.

while not subscribed:
    miss_out()
# fix: erogol.substack.com

Subscribe now

🧠 GLM-5.1 Zhipu’s 744B MoE (44B active). #1 on SWE-Bench Pro (58.4%), AIME 2026 95.3%, GPQA Diamond 86.2%, strong agentic/tool-use with 3.6x KernelBench speedup. MIT license.

🧠 MiniMax-M2.7 230B total / 10B active MoE with 256 experts. SWE-Bench Pro 56.2%, Terminal-Bench 2.0 57%, strong agentic/coding with 200K context. Custom license.

🧠 Marco-Mini-Instruct 17.3B total MoE with only 0.86B active params (5% activation ratio, Qwen3-MoE based). Best in class across English + multilingual benchmarks via upcycling + on-policy distillation. Apache 2.0.

🖼️ ERNIE-Image Baidu’s 8B DiT text-to-image model with built-in Prompt Enhancer. SOTA among open-weight T2I on GENEval (0.89), DPG-Bench (93.8), and text rendering. Runs on 24GB VRAM. Apache 2.0.

🗣️ VoxCPM2 2B tokenizer-free diffusion autoregressive TTS built on MiniCPM-4. 30+ languages + Chinese dialects, voice cloning, voice design from text descriptions, streaming. Trained on 2M+ hours. ~8GB VRAM. Apache 2.0.

🗣️ MOSS-TTS-Nano ~100M multilingual TTS (CN/EN + more) using audio tokenizer + LLM autoregressive pipeline. Designed for real-time CPU-only inference without GPU.

🗣️ tiny-tts Ultra-lightweight English TTS with only 1.6M params / 3.4MB ONNX. End-to-end (no separate vocoder), 44.1kHz, ~53x real-time on CPU. MIT.

🎵 Woosh Sony AI’s sound effect foundation model suite: audio encoder/decoder, CLAP text-audio alignment, text-to-audio (Flow + distilled DFlow), and video-to-audio (VFlow) generation. Open weights trained on public datasets. CC-BY-NC (weights), MIT/Apache (code).

📎 Papers

🧠 Memory Caching: RNNs with Growing Memory

What’s new
• Caches periodic checkpoints of RNN hidden states, letting recurrent models’ effective memory grow with sequence length. Bridges the gap between fixed-memory RNNs and quadratic-cost Transformers.
• Complexity is O(NL), smoothly interpolating between O(L) RNNs and O(L^2) attention.

How it works
• Splits the sequence into segments; each segment compresses into a memory state via normal recurrence.
• At each timestep, the model queries both its current “online” memory and all cached past segment memories.
• Four aggregation strategies tested: sum, gate, cross-attention, and softmax weighting.

Results
• Mamba+MemCache matches or beats full Transformer++ on all MAD Synthetic tasks (the standard long-range reasoning suite). First sub-quadratic model to do so.
• Handles sequences up to 131K tokens; performance degrades gracefully as context grows.

The long-standing RNN trade-off (memory vs speed) might finally be broken. Sub-quadratic cost, growing memory. Transformers should be nervous.

🧠 Flow Map Language Models: One-step Language Modeling via Continuous Denoising

What’s new
• Generates entire text sequences in one denoising step. No autoregressive left-to-right, no multi-step diffusion. A single forward pass maps noise to text.
• First continuous language model to match autoregressive baselines on standard benchmarks without iterative decoding.

How it works
• Trains a flow map that transports Gaussian noise directly to the data distribution in one step, using a consistency-training-like objective.
• Text is embedded into continuous space, denoised in a single pass, then decoded back to tokens.
• Built on flow matching theory but trained to collapse the full ODE trajectory into a single evaluation.

Results
• Matches GPT-2 scale autoregressive models on perplexity and downstream tasks with a single forward pass.
• 5-10x wall-clock speedup over multi-step diffusion language models at comparable quality.

🧠 Self-Improving Pretraining: Using Post-Trained Models to Pretrain Better Models

What’s new
• Uses a strong post-trained LLM to rewrite its own pretraining data, then pretrains the next generation from scratch on that improved corpus.
• Creates a self-improvement loop: better model -> better data -> better model.

How it works
• The “teacher” model (post-trained, instruction-following) rewrites noisy web text into cleaner, more structured versions while preserving factual content.
• The “student” is pretrained from random init on this synthetic-cleaned corpus instead of raw web data.
• Iterable: the student, once post-trained, can clean data for the next generation.

Results
• Student models consistently outperform baselines trained on the same amount of original data across multiple scales.
• Quality gains compound across generations, with each iteration yielding a better starting point.

🔀 Interleaved Head Attention

What’s new
• Standard attention gets H patterns from H heads. IHA constructs pseudo-heads as learned linear combinations across all heads, getting up to H^2 attention patterns.
• Compatible with FlashAttention because mixing happens before attention, not inside it.

How it works
• For each head, constructs P pseudo-queries/keys/values (typically P=H) as learned linear combinations of all H original heads’ projections.
• Pseudo-Q x pseudo-K interactions yield up to P^2 attention patterns per head with only O(H^2*P) extra parameters.
• Interleaved tokens create an expanded sequence processed with sliding-window causal attention.

Results
• Multi-Key Retrieval on RULER: +10-20% over full attention (4k-16k context).
• After fine-tuning on OpenThoughts: +5.8% on GSM8K and +2.8% on MATH-500 vs. standard MHA.

🎬 A Frame is Worth One Token: Delta Tokens for World Modeling

What’s new
• DeltaTok compresses the visual difference between consecutive video frames into a single continuous token. That’s 1,024x token reduction at 512x512, collapsing 3D spatio-temporal video into a 1D temporal sequence.
• DeltaWorld generates multiple diverse future predictions in a single forward pass (no iterative denoising).

How it works
• A frozen DINOv3 VFM encodes frames into patch grids; DeltaTok compresses frame-to-frame change into one delta token; the decoder reconstructs the next frame from previous + delta.
• Best-of-Many training: 256 noise queries generate 256 future hypotheses in parallel, only closest to ground truth is supervised.
• Different noise queries map to different plausible futures; autoregressive rollout chains delta tokens.

Results
• Outperforms existing generative world models (including Cosmos) on dense forecasting with 35x fewer parameters and 2,000x fewer FLOPs. That’s not an incremental win. That’s a different regime.

🔍 CompreSSM: In-Training Compression of State Space Models

What’s new
• First in-training (not post-hoc) compression method for SSMs, using control-theoretic balanced truncation via Hankel Singular Values to prune low-importance state dimensions during training.
• Dominant HSVs are rank-preserving throughout training. The unimportant dimensions stay unimportant, justifying early truncation.

How it works
• At intervals during early training, computes controllability/observability Gramians for each SSM layer, derives Hankel singular values, truncates dimensions below threshold.
• Transforms system into diagonal balanced realization, slices off bottom dimensions to reduce A, B, C matrices in-place.
• After each ~10% reduction, checks validation performance and reverts if degradation detected.

Results
• Compressed models (start large, shrink during training) match or outperform both uncompressed large models and directly-trained small models, cutting state dimensions by 50-75% while retaining near-full accuracy on Long Range Arena.

Counterintuitive trick: start big, shrink during training. You get the over-parameterization benefits early, then pay less compute for the rest of the run.

🧠 R-Zero: Self-Evolving Reasoning LLM from Zero Data

What’s new
• Trains a reasoning LLM with zero human-curated reasoning data. The model generates its own training problems, solves them, and improves from its own verified solutions.
• Full self-play loop: problem generation -> solution attempt -> verification -> RL update.

How it works
• The model proposes new problems, attempts solutions with chain-of-thought, then an automated verifier checks correctness.
• REINFORCE-style RL with the verification signal as reward. No human labels, no distillation from a stronger model.
• Problems start easy, ramp up as the model gets better. Self-pacing curriculum.

Results
• Beats models trained on curated reasoning data (MetaMath, OpenMathInstruct) on GSM8K, MATH, MMLU-Pro.

If this generalizes, the reasoning data bottleneck disappears. The constraint shifts to compute and verification quality. No human curation needed.

⚡ DDTree: Accelerating Speculative Decoding with Block Diffusion Draft Trees

What’s new
• Uses a block diffusion model as the drafter in speculative decoding, generating tree-structured draft candidates in parallel instead of sequential token-by-token drafting.
• Multiple continuations explored simultaneously in a single diffusion pass.

How it works
• Block diffusion model generates multiple token sequences (branches) in parallel via a single denoising process.
• These branches form a tree of candidates verified by the target LLM in one batched forward pass.
• Adaptive tree width: more branches where uncertainty is high, fewer where the model is confident.

Results
• 1.4-1.8x throughput improvement over standard speculative decoding across multiple model pairs.

🔬 Do LLMs Encode Functional Importance of Reasoning Tokens?

What’s new
• LLMs internally “know” which tokens in their own reasoning chains matter and which are filler. Importance scores extracted from hidden states predict greedy pruning ranks.
• Reasoning chains aren’t uniformly important: numerical values and logical connectives are preserved; filler/repetition tokens get pruned first.

How it works
• Extracts importance scores from model hidden states at each token position during chain-of-thought generation.
• Trains lightweight probes to predict whether a token would be kept or pruned by an oracle greedy search.
• Tests whether distilled students can learn from pruned chains as effectively as full chains.

Results
• Distilled students trained on greedy-pruned chains outperform TokenSkip (which uses GPT-4 labels) at matched reasoning lengths on GSM8K, MATH, and MMLU-Pro.

So models know their own filler from their own substance. That’s a free compression signal sitting right there in the hidden states.

🗣️ T5Gemma-TTS: Encoder-Decoder TTS from Pretrained LLM Backbone

What’s new
• Repurposes T5/Gemma’s encoder-decoder architecture for TTS. The encoder handles text conditioning, the decoder generates speech tokens with a progress monitoring mechanism (PM-RoPE) that prevents the common “text fade-out” problem.
• Inherits rich linguistic knowledge from LLM pretraining instead of learning language modeling from scratch.

How it works
• Text encoder (frozen or fine-tuned LLM) conditions a speech decoder that generates audio tokens autoregressively.
• PM-RoPE (Progress Monitoring Rotary Position Embedding) explicitly tracks synthesis progress relative to text length, preventing the decoder from losing alignment mid-utterance.
• Duration predictor provides coarse timing; fine-grained alignment emerges from cross-attention.

Results
• Competitive with SOTA autoregressive TTS on naturalness and speaker similarity. Disabling PM-RoPE causes near-complete synthesis failure (CER: 0.129 -> 0.982).

🗣️ Affectron: Emotional Speech Synthesis with Affective Contextual Alignment

What’s new
• Aligns emotional speech synthesis with natural language emotion descriptions instead of fixed categorical labels (happy/sad/angry).
• You describe the emotional quality you want in plain text and the TTS matches it.

How it works
• Trains a contrastive alignment between speech audio and free-form emotion descriptions (not just 6 basic categories).
• Emotion embedding is injected into the TTS decoder as a conditioning signal alongside speaker identity and text.
• Uses a large-scale emotion-annotated speech dataset with LLM-generated descriptions of vocal affect.

Results
• Outperforms prior emotional TTS systems on both emotion accuracy and naturalness in human evaluations.
• Handles subtle emotions (”wistfully nostalgic,” “barely contained excitement”) that categorical systems can’t represent.

Emotional TTS goes from dropdown menus (pick one of 6) to “describe what you want.”

🐍 MambaVoiceCloning: Efficient TTS via SSM and Diffusion

What’s new
• First TTS system with a fully SSM-only conditioning path at inference. No attention or RNN layers for text, rhythm, or prosody. Linear-time O(T) conditioning with bounded memory.
• Removes all attention-based duration and style modules from a StyleTTS2 backbone.

How it works
• Gated bidirectional Mamba text encoder + Temporal Bi-Mamba (supervised by alignment teacher, discarded after training) + Expressive Mamba with AdaLN modulation.
• SSM-only stack conditions a StyleTTS2 mel-diffusion-vocoder backbone; diffusion decoder uses a fixed 5-step schedule.
• 21M encoder parameters, dramatically smaller than transformer alternatives.

Results
• Modest gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS, F0 RMSE, MCD, and WER. 1.6x throughput improvement.

The gains are modest, but the point is proved: SSMs can fully replace attention in TTS conditioning. Linear scaling, bounded memory, no attention anywhere in the path.

🔤 StableToken: Noise-Robust Semantic Speech Tokenizer

What’s new
• Current semantic speech tokenizers are surprisingly fragile. Even at high SNRs where speech is perfectly intelligible, token sequences change drastically from acoustic perturbations.
• Multi-branch consensus mechanism with bit-wise voting makes tokens stable under noise.

How it works
• Multi-branch architecture processes audio in parallel through independent paths, producing multiple candidate token representations.
• Bit-wise voting merges these into a single stable token sequence, forming a consensus that resists noise perturbations.
• Co-designed with noise augmentation training; works with larger vocabulary sizes.

Results
• Average Unit Edit Distance of 10.17% under noise vs 26.17% (best supervised baseline) and 16.48% (best robust SSL model).
• SOTA reconstruction quality (best WER and MOS). Noise resilience doesn’t cost fidelity.

If your speech tokenizer breaks on background noise, everything downstream breaks too. StableToken fixes this at the root. Surprised nobody did this sooner.

🔢 DiVeQ: Differentiable Vector Quantization via Reparameterization

What’s new
• Treats VQ as adding a reparameterized error vector that mimics quantization distortion. Forward pass stays hard/discrete, but gradients flow freely.
• Space-filling variant (SF-DiVeQ) maps inputs to the nearest point on a curve through codewords, achieving full codebook utilization.

How it works
• During training, “quantization” is modeled as the original vector plus a noise term shaped like real quantization error, enabling backprop via the reparameterization trick.
• No auxiliary losses, commitment losses, or temperature schedules. Pure end-to-end training.

Results
• Improves reconstruction and sample quality over straight-through estimator and other VQ methods across VQ-VAE (images), VQGAN (generation), and DAC (speech coding).

Drop-in replacement for VQ anywhere you use it. No commitment losses, no temperature schedules, no auxiliary losses. Just works. Directly relevant to neural audio codecs and discrete speech models.

🧑‍💻 Open Source

TriAttention Trigonometric KV cache compression that cuts KV memory by 10.7x and boosts throughput by 2.5x on long reasoning tasks with no accuracy loss. vLLM-compatible, runs Qwen3-32B on a 24GB RTX 4090. MIT. 512 stars.

Agent-Reach One-command internet access for AI agents. Twitter, Reddit, XiaoHongShu, Bilibili, GitHub, YouTube, all behind a single install. Works with Claude Code, OpenClaw, Cursor. Free, cookies stay local. 17.5k stars.

Thanks for reading… Enjoyed this issue? Share it with a friend. 👍

Share

I little self-promotion

⭐️ TokLog - Local LLM cost proxy. pip install toklog && tl proxy start - zero code changes, per-call breakdowns, and a kill switch for runaway agent loops. No cloud, no account.

⭐️ ngi - Trigram-indexed regex search in Rust. 2-6x faster than ripgrep on repeated queries. ngi index once, search instantly forever.

🤖 Model Releases

🗣️ Voxtral-4B-TTS-2603 - Mistral’s first open TTS. Zero-shot voice cloning from 3s of audio, 9 languages, beats ElevenLabs in 68.4% of human evals. vLLM-compatible. CC BY-NC 4.0.

try:
    keep_learning()
except NotSubscribed:
    newsletter.subscribe()

Subscribe now

🗣️ LongCat-AudioDiT-3.5B - Meituan’s TTS that ditches mel-spectrograms for raw waveform latent space. 3.5B Diffusion Transformer, beats prior SOTA on zero-shot voice cloning (0.818 vs 0.809 speaker similarity). MIT.

🎭 daVinci-MagiHuman - 15B model generating synced video + audio from a reference image and text. 80% win rate vs Ovi 1.1, 5s 1080p clip in 38s on H100. Multilingual (CN/EN/JP/KR/DE/FR). Fully open-source.

🌐 Harrier-OSS-v1-27B - Microsoft’s multilingual embedding model. SOTA on Multilingual MTEB v2, 32K context window, 100+ languages. Ships in 270M, 0.6B, and 27B sizes.

🌐 Bonsai-8B - End-to-end 1-bit LLM (Qwen3-8B) that fits in 1.15 GB. Not a quantization - purpose-built 1-bit. 70.5 avg on 6 benchmarks, 6.2x faster than FP16 on RTX 4090. Apache 2.0.

🖼️ SAM 3.1 - Meta’s Segment Anything update. New Object Multiplex makes multi-object tracking 7x faster at 128 objects with no accuracy loss. Better VOS on 6/7 benchmarks. Apache 2.0.

👂 Cohere Transcribe - Cohere’s first open ASR. 2B params, 14 languages, long-form with auto-chunking. 58k+ downloads in week one. Apache 2.0.

🌐 pplx-embed - Perplexity’s embedding family with a diffusion-pretrained backbone. pplx-embed-v1 competitive on Multilingual MTEB v2; pplx-embed-context-v1 SOTA on ConTEB. 0.6B and 4B sizes, 1M+ downloads.

🧠 Nemotron-Cascade 2 - NVIDIA’s open 30B MoE with only 3B active params. Gold Medal at 2025 IMO, IOI, and ICPC - second open-weight model ever to hit that bar. Weights, data, and full training recipe open-sourced.

📎 Papers

⚡ Aurora: When RL Meets Adaptive Speculative Training

What’s new
• Speculative decoding usually means: train a small “draft” model offline to predict tokens, then use it to speed up a large model at inference. Aurora closes the loop - the draft model learns continuously from live inference traffic instead of being trained once and frozen.
• Day-0 deployment: serve a brand-new model with an untrained speculator immediately; it improves itself in real time.

How it works
• Reframes speculative decoding as asynchronous RL: the draft model is the policy, accepted/rejected tokens are the reward signal.
• Learns from both accepted tokens (imitation) and rejected tokens (counterfactual - “stop proposing this”), giving much denser training signal.
• New speculator weights hot-swapped into the live inference server without downtime via lazy synchronization.

Results
• 1.5× day-0 speedup on frontier models (MiniMax M2.1 229B, Qwen3-Coder-Next 80B) with a cold-start speculator.
• 1.25× additional speedup over a well-trained static speculator by adapting to real user traffic distribution.

Why it matters
• Eliminates “train a drafter offline for weeks before deployment.” Speculators no longer go stale when the target model or user patterns change.

🗣️ MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

What’s new
• You can describe a voice in plain English - “deep, gravelly, slightly warm, like a late-night radio host” - and generate it. No reference audio needed.
• Most TTS customization requires a sample clip. This eliminates that entirely.

How it works
• A language model generates audio tokens from text descriptions the same way it generates word tokens from a prompt.
• Trained on ~5,000h of cinematic speech (movies, TV) annotated with timbre descriptions - richer and more varied than studio recordings, which sound sterile.
• Retrieval mode: can also search a voice database to find the closest match to your description and clone from there.

Results
• Beats prior methods on naturalness, instruction-following, and overall quality in human preference studies.

🧠 Hyperagents: Metacognitive Self-Modification

What’s new
• An AI agent that rewrites its own instructions, tools, and evaluation criteria - then applies those changes to future runs. Not just self-reflection: actual self-modification that persists.
• Prior self-improving agents only worked in coding. This one generalizes across math, reasoning, and general tasks.

How it works
• Two-layer loop: a task agent does the work; a meta-agent watches, judges, and edits both the task agent’s prompts/tools and its own judgment criteria.
• Nothing is frozen - the meta-agent can rewrite its own evaluation logic if it decides it was grading wrong.
• Improvements compound - each generation of modifications builds on the last, like iterative refactoring but for agent behavior.

Results
• Outperforms strong self-improving agent baselines on coding, math, and general reasoning.
• Performance gains compound across iterations - later runs significantly outperform earlier ones.

⚡ TurboQuant: Redefining AI Efficiency with Extreme Compression

What’s new
• Google’s two-stage compression algorithm combining PolarQuant (high-quality compression) and QJL (Quantized Johnson-Lindenstrauss - a 1-bit error correction layer on top).

How it works
• PolarQuant: randomly rotate data vectors to simplify geometry, then compress. Think of it as: before compressing, rearrange the data so it’s easier to compress accurately.
• QJL: uses just 1 bit to detect errors from the first stage while preserving the relative distances between vectors - so you can still find “similar” vectors correctly.
• Two stages: compress aggressively first, catch the errors second. The combination gives extreme compression with minimal quality loss.

Results
• Reduces key-value cache memory by at least 6x without accuracy loss.
• Up to 8x speedup in computing attention scores on H100 GPUs.
• Gemma 2 2B retains benchmark performance under TurboQuant compression.

Why it matters
• Practical path to running large models on memory-constrained hardware. The attention cache is often the memory bottleneck - 6x smaller cache means much longer context windows on the same hardware.

🧠 Mamba-3: Improved Sequence Modeling using State Space Principles

What’s new
• Mamba-2 was fast but broke on tasks requiring memory - like tracking whether a running count is odd or even across thousands of tokens. It performed at random chance. Mamba-3 fixes this while getting more efficient.

How it works
• The hidden state now stores complex numbers instead of real ones. Complex numbers can represent rotation, which gives the model a built-in sense of position and periodicity - the thing it was missing for counting and tracking.
• More efficient state updates: the recurrence now uses matrix multiplication instead of a less expressive outer-product operation, packing more information into the same memory footprint.
• Since decoding is bottlenecked by memory bandwidth (not compute), the extra matrix math costs almost no wall-clock time.

Results
• Near-perfect state-tracking accuracy where Mamba-2 performs at chance.
• Matches Mamba-2 quality with half the memory - same output, half the RAM.
• +1.8 accuracy points over the best alternative at 1.5B scale.

Why it matters
• State tracking is a prerequisite for any agent that needs to follow multi-step instructions reliably. This closes the gap that made linear models a risky bet for production. Already being adopted in Qwen3 and Nemotron hybrids.

🧠 M²RNN: Non-Linear RNNs with Matrix-Valued States

What’s new
• An RNN where the hidden state is a matrix instead of a vector, with non-linear transitions. The key insight: non-linear RNNs weren’t abandoned because non-linearity is bad - they were abandoned because their state was too small. Matrix-valued states fix this.

How it works
• Each recurrent layer maintains a matrix-shaped hidden state with expressive non-linear updates. Matrix structure naturally exploits GPU tensor cores.
• Hybrid design: most layers are attention, some swapped for M²RNN recurrent layers. Even replacing a single attention layer yields meaningful gains.

Results
• Beats Gated DeltaNet hybrids by 0.4–0.5 perplexity points on a 7B mixture-of-experts model using 3× smaller recurrent state.
• Perfect state tracking that generalizes to sequence lengths unseen during training.
• +8 points on LongBench over SOTA hybrid linear attention architectures.

Why it matters
• Transformers are provably unable to do certain tasks natively (like entity tracking across many steps). M²RNN layers can - and they’re cheap to add. A compelling plug-in upgrade rather than a full replacement.

🧠 λ-RLM: The Y-Combinator for LLMs

What’s new
• LLMs can already write code to break down long documents - but that code crashes, loops forever, or hallucinates. λ-RLM replaces free-form code generation with a small fixed vocabulary of safe operations (Split, Filter, Map, Reduce).
• Think of it as giving the model Lego bricks instead of raw clay: the pieces are constrained, but you can build anything.

How it works
• The LLM picks operations from the library to decompose a task into small enough pieces to fit in its context window. Actual execution is deterministic - no surprises.
• Recursion is handled via the Y-combinator (a computer science primitive), which guarantees termination and makes execution cost predictable before it starts.

Results
• Wins 29/36 model-task comparisons vs. standard recursive LLMs across 9 models and 4 benchmarks.
• Up to +21.9 accuracy points and 4.1× faster.

Why it matters
• The agent does the thinking; the runtime does the orchestration. You get better accuracy, formal correctness guarantees, and predictable cost - instead of hoping the LLM writes valid code.

🗣️ Any2Speech: Borderless Long Audio Synthesis

What’s new
• Standard TTS reads one sentence at a time with no memory of what came before. Any2Speech adds scene awareness: it knows who’s speaking, what the emotional arc is, and how the current line fits into the whole scene.
• Most TTS datasets are built by filtering noisy audio. This one keeps everything and annotates it instead - capturing natural breath, emotion, and multi-speaker dynamics that filtered datasets lose.

How it works
• The model gets three layers of context: scene level (who are these people, what’s the mood), sentence level (what’s the intent of this line), word level (how to stress and pronounce it).
• Before synthesizing, the model writes a plan - tone, pacing, emotion per sentence - using chain-of-thought reasoning. You can edit the plan before generating audio.

Results
• Demo-based evaluation only (no standard benchmark covers long-form scene-aware synthesis yet).
• Uses >90% of training data vs. the typical 10-30% retained after aggressive quality filtering.

Why it matters
• Closes the gap between TTS and actual voice acting. The model can direct its own delivery from a script rather than blindly reading words.

🤔 Agentic Confidence Calibration

What’s new
• When should an AI agent trust its own answer? This paper builds a confidence calibration system that watches how the agent reasoned - not just what it concluded.
• Most calibration methods look at the output. This one looks at the journey: did confidence fluctuate? Did it spike or crash at unusual steps?

How it works
• Records a confidence trace across every step of the agent’s execution - how certain it was at each point.
• Extracts features from that trace: how much confidence changed between steps, how volatile it was within a step, which early or late steps best predicted success.
• Maps those features to a calibrated confidence score using a lightweight model.

Results
• Outperforms strong baselines across 8 benchmarks on calibration quality.
• Generalizes across different agent architectures.

Why it matters
• An agent that confidently gives a wrong answer is more dangerous than one that says “I’m not sure.” Process-level signals catch uncertainty that output-only methods miss entirely.

⚡ Data-Efficient Pre-Training by Scaling Synthetic Megadocs

What’s new
• When you’re running out of unique training data (increasingly everyone), generating rephrases of your documents and mixing them in actually improves performance on the original distribution.
• The key trick: stitch rephrases into one long “megadoc” per source document instead of treating each as a separate training example.

How it works
• Simple rephrasing: generate multiple rephrasings per document, mix into training. Improves monotonically, plateauing at ~1.48× data efficiency at 32 rephrases.
• Megadocs: concatenate all rephrases into one long document (or insert generated rationales inline to “stretch” the original).
• Allows 5× more training steps without overfitting - epoch real data 32 times instead of 16.

Results
• Megadocs push data efficiency from 1.48× to 1.80× at 32 generations per document.
• Improvements on loss, downstream benchmarks, and especially long-context loss (0.14–0.19 improvement).

Why it matters
• A concrete recipe for hitting the data wall. By organizing synthetic variants into longer coherent documents, you resist overfitting much longer. Very relevant as unique web text becomes scarcer.

🧠 Thinking in Uncertainty: Mitigating Hallucinations with Latent Entropy-Aware Decoding

What’s new
• LEAD (Latent Entropy-Aware Decoding): a decoding strategy for multimodal reasoning models that switches between two modes based on how uncertain the model is token-by-token.

How it works
• High-entropy (uncertain) states: instead of picking a token, the model uses a probability-weighted blend of possible token embeddings - staying in “soft” space, preserving semantic diversity.
• Low-entropy (confident) states: revert to discrete token selection for precise convergence.
• Visual anchor injection: during high-entropy phases, reintroduce visual features to keep the model grounded in the image.
• Persistence window: prevents rapid oscillation between modes (once you switch, you stay for a few steps).

Results
• +4.7% improvement on hallucination benchmarks averaged across models.
• +2.0% on mathematical reasoning tasks.

🔬 Scaling Reward Modeling Without Human Supervision

What’s new
• Reward models - the systems that teach AI what “good” looks like during training - normally require thousands of human preference labels. This paper trains them on raw web text instead.
• The key insight: a document’s natural continuation is probably better than a random mismatch. That’s all you need to build a preference signal at scale.

How it works
• Take any web document. The second half is the “preferred” completion; a random snippet from elsewhere is the “rejected” one. Train the reward model to distinguish between them.
• Same training objective as human-labeled RLHF - just with implicit labels scraped from the web at zero cost.
• Demonstrated on 11M tokens of math-focused text.

Results
• +7.7 points average on RewardBench v2 (a standard reward model quality benchmark).
• +16.1 points on math-specific subsets.
• Downstream math performance improves significantly when used for model fine-tuning.

🧠 Directional Routing in Transformers

What’s new
• Adding a tiny routing layer to transformer attention heads - only 3.9% extra parameters - reveals something unexpected: turn routing off, and factual recall collapses to near-zero. The model loses the ability to complete simple patterns entirely.
• This suggests transformers aren’t using individual specialized heads - they’re using coordinated networks of heads that only work together.

How it works
• Each attention head gets 4 learned “filter vectors.” A shared routing network decides, based on the input, how much of each filter to apply to each head’s output.
• No special training signal - this self-organizes purely from standard language modeling. Early layers learn domain-adaptive routing; late layers learn fixed syntactic cleanup.

Results
• 31–56% better language modeling quality vs. baseline at 433M parameters.
• 1.3× faster to train.
• Removing even the “most important” head only drops accuracy by 1.4% - the routing layer absorbs the redundancy.
• Caveat: quality gains don’t yet show up on standard multiple-choice benchmarks.

🧑‍💻 Open Source

ATLAS - Test-time scaffolding that turns a frozen Qwen3-14B into a frontier coder. 74.6% LiveCodeBench pass@1 on a single GPU - between Claude Sonnet and GPT-5 - at $0.004/task in electricity.

Carbonyl - Full Chromium running inside a terminal. Actual rendering via a modified Blink engine - not screenshots. 60 FPS, <1s startup, 50x less CPU than Browsh. 17.6k stars.

OpenRoom - Browser-based windowed desktop for AI agents. Natural language operates all apps; “Vibe Workflow” generates new ones from text. From MiniMax. MIT.

Zeroboot - Sub-millisecond VM sandboxes via CoW forking of Firecracker VMs. 0.79ms spawn, ~265KB each, 190x faster than E2B. The missing primitive for safe agent code execution. Apache 2.0.

Thanks for reading… Enjoyed this issue? Share it with a friend. 👍

Share

🌐 Phi-4-Reasoning-Vision-15B — 15B open-weight multimodal reasoning model from Microsoft. Phi-4-Reasoning backbone + SigLIP-2 vision encoder, mid-fusion architecture. Outperforms much larger models on reasoning benchmarks.

try:
    keep_learning()
except NotSubscribed:
    newsletter.subscribe()

Subscribe now

🗣️ Fish Audio S2 — Open-sourced TTS with multi-speaker, multi-turn generation and instruction-following via natural language. Dual-AR model, RL-aligned post-training, SGLang inference engine (RTF 0.195, TTFA 100 ms). 81.88% win rate on Emergent TTS Eval. Weights + fine-tuning code released.

👂 Granite 4.0 1B Speech — 1B speech-language model from IBM for multilingual ASR + bidirectional speech translation. 6 languages, keyword biasing, speculative decoding via CTC drafts. Average WER 5.52 on HF Open ASR Leaderboard at 280× real-time — competitive with 8B models at half the size.

🧠 NVIDIA Nemotron v3 — 120B total / 12B active MoE hybrid (Mamba-2 + Attention + MTP), 1M context, 7 languages. NVFP4 precision. Beats GPT-OSS-120B on agentic (SWE-bench 60.5% vs 41.9%), reasoning (HMMT Feb25 94.7%), and long-context (RULER@1M 91.8% vs 22.3%). Also ships as Nano-30B (3B active) and Nano-4B. Open weights + data + recipes.

⚡ RedHatAI Speculator Models — EAGLE-3 draft models for speculative decoding of Llama-3/4, Qwen3, and GPT-OSS. 1.5–2.5× latency reduction (up to 4.9× on Llama-4-Maverick math @ 8×B200) with zero quality loss. ~15 model-specific speculators, plug-and-play with vLLM.

🧠 Qwen3.5-27B-Claude-Opus-Reasoning-Distilled — 27B community distillation of Claude 4.6 Opus reasoning traces into Qwen3.5 via LoRA + Unsloth. Replaces repetitive <think> loops with structured CoT. Native “developer” role support for coding agents. ~30 tok/s on RTX 3090 (Q4_K_M, 16.5 GB VRAM). GGUF format.

📎 Papers

🧠 Mamba-3: Improved Sequence Modeling using State Space Principles

What’s new

• Redesigns the core state-space recurrence from first principles — fixes expressiveness, state dynamics, and hardware utilization simultaneously. Removes the short convolution entirely. Matches Mamba-2 quality at half the state size.

How it works

• A more accurate way to update the hidden state each step — the old method (Euler) only looks at the current input, while the new one averages current and next inputs, capturing dynamics the old rule missed. This lets both the input and output projection matrices contribute to updates — richer recurrence at no extra cost.

• Uses complex numbers in the state, which lets it naturally represent oscillating patterns (think: repeating structures in language) that real-valued states struggle with.

• Multi-Input Multi-Output (MIMO): widens the input/output projections by a small factor R, so each step does more useful math per byte of memory read. Since decoding is bottlenecked by memory bandwidth (not compute), the extra arithmetic is essentially free — wall-clock barely changes at R=4.

• Hybrid models (5:1 Mamba-3 to attention layers) largely close the retrieval gap vs pure Transformers.

Results

• 1.5B, 100B tokens: Mamba-3 MIMO val PPL 10.24 vs Transformer 10.51, Mamba-2 10.47. Avg accuracy 57.6 (+2.2 over Transformer).

• Mamba-3 MIMO at state size 64 matches Mamba-2 at state size 128 — same quality, half the decode latency.

🔬 Directional Routing in Transformers

What’s new

• A lightweight mechanism (+3.9% params, +0.02% FLOPs) that gives each attention head 4 learned “directions to suppress,” controlled by a shared MLP router. The router becomes so central to the model that disabling it collapses factual recall to near-zero, while knocking out any individual head has negligible effect.

How it works

• Each head learns 4 unit-length direction vectors. A shared 4-layer MLP router outputs per-head gating weights that control how strongly each direction is applied.

• After standard attention computes its output, the mechanism projects the output onto each learned direction and subtracts it (scaled by the gating weight) — effectively filtering out interference between different types of information (e.g., preventing factual knowledge from bleeding into syntactic processing).

• No auxiliary loss or load-balancing needed. It self-organizes: early layers learn to route by domain/topic, late layers learn to suppress grammar-related features that would otherwise add noise.

Results

• 433M vs 417M baseline: overall PPL 20.6 vs 42.7 (−51.8%), Math PPL 29.2 vs 66.5 (−56.1%).

• The catch: perplexity gains don’t transfer to multiple-choice benchmarks (HellaSwag, ARC) — the routing is better at decoding existing knowledge, not a source of new knowledge. Intriguing mechanistic finding, unclear practical payoff at scale.

🧠 Attention Residuals (Kimi Team / MoonshotAI)

What’s new

• Standard residual (skip) connections just add every layer’s output with equal weight — there’s no way for the network to say “layer 5’s output matters here, but layer 12’s doesn’t.” AttnRes replaces this with learned attention over depth: each layer gets a small learnable query that decides how much weight to give each preceding layer’s output.

• Block AttnRes makes it practical: group layers into ~8 blocks and only apply the cross-layer attention at block boundaries, instead of between every single layer. This keeps the memory cost manageable.

How it works

• Each layer computes a weighted combination of all prior layer outputs, where the weights come from a softmax attention mechanism — exactly like sequence attention, but operating over layers instead of tokens.

• Block variant: normal residual connections within blocks, learned attention only across the ~8 block-level representations. This recovers most of the gains at a fraction of the cost.

• All the learnable queries start at zero, so at initialization the system behaves like uniform averaging — this prevents training instability.

• Cross-stage caching for pipeline parallelism: <4% wall-clock training overhead.

• Two-phase inference keeps overhead under 2%.

• Elegant theoretical framing: standard residuals, Highway Networks, and other prior variants are all “linear attention over depth” — AttnRes is “softmax attention over depth.”

Results

• Block AttnRes matches the loss of a baseline trained with 1.25× more compute.

• Kimi Linear 48B/3B on downstream tasks: GPQA-Diamond +7.5, Math +3.6, HumanEval +3.1, C-Eval +2.9 over baseline.

🔬 Lost in Backpropagation: The LM Head is a Gradient Bottleneck

What’s new

• The gradient flowing back from the vocabulary-sized output gets squeezed through a much smaller hidden dimension, crushing most of the useful training signal into noise.

How it works

• The gradient signal passing through the LM head has very high rank (nearly vocabulary-sized), but the head can only pass through a tiny fraction of that information — it’s constrained by the hidden dimension, which is orders of magnitude smaller than the vocabulary.

• The gap between the ideal gradient update and what actually gets through is mathematically bounded — and it’s large.

• SpamLang experiment: a trivially simple synthetic language (just repeat one token) that any Transformer with hidden dim ≥ 2 can perfectly express — yet gradient descent fails to learn it as vocabulary size grows. This cleanly proves the problem is optimization, not model capacity.

• Controlled 2B-parameter pretraining (Llama3 backbone) with the LM head forced to a specific rank confirms the effect in real training.

• The projected gradient has only 0.1–0.2 cosine similarity with the original gradient — meaning the model sees a heavily distorted version of the true training signal.

Results

• 95–99% of gradient norm destroyed across GPT-2, Pythia, Llama-3, Qwen3-Base families.

• A model with hidden dim 4096 reaches the final loss of a dim-32 model within 700M tokens — a 16× convergence speedup for the same backbone.

• SpamLang: vocab size 16384 learns perfectly; vocab size 131072 generates garbage regardless of learning rate.

📏 SRLM: Self-Reflective Program Search for Long Context

What’s new

• SRLM improves long-context reasoning by combining recursive sub-queries with uncertainty-aware self-reflection. Instead of using fixed heuristics to pick the best reasoning path, it uses the model’s own confidence signals to select the most reliable answer.

How it works

• Breaks long contexts into smaller recursive sub-queries executed in a sandboxed programming environment, treating the full context as an external variable the program can reference.

• Evaluates candidate reasoning programs using three built-in confidence signals:

  • Self-consistency: do multiple runs agree on the same answer?

  • Verbalized confidence: how confident does the model say it is at each step?

  • Reasoning trace length: how many tokens did it take? (longer deliberation can signal harder sub-problems)

• These signals together pick the most reliable reasoning trajectory from the search space.

Results

• Qwen3-Coder-480B: LongBench-v2 64.9 (+5.1 over RLM), BrowseComp+ 59.7 (+22.6 over RLM), OOLONG 51.8 (+6.1).

• GPT-5: LongBench-v2 68.9 (+9.4), BrowseComp+ 92.4 (+6.4), OOLONG 65.5 (+12.5).

• Biggest gain on semantically intensive retrieval: +22.6 absolute on BrowseComp+ with Qwen3.

🗣️ CAST-TTS: Unified Timbre Control via Cross-Attention

What’s new

• A unified framework for controlling voice timbre (the quality that makes one voice sound different from another) using cross-attention, accepting both speech clips and text descriptions interchangeably as voice prompts.

How it works

• Timbre encoder with two branches: speech prompts go through a speaker encoder; text prompts (like “deep male voice with a warm tone”) go through a Flan-T5 text encoder.

• Multi-stage training aligns the speech and text representations into a shared timbre space — so a text description and a matching voice clip land near each other.

• Cross-attention in the Transformer backbone lets the model flexibly draw on timbre information from either modality.

• Trained with flow-matching loss, optimizing cross-modal feature alignment.

Results

• Speech-prompted: SPK-Sim 78.4 (vs F5-TTS 75.4, MaskGCT 74.5), WER 2.05%, UTMOS 3.91.

• Text-prompted: Style-ACC 91.15% (vs CapSpeech 88.93%), WER 3.89%, Sim-MOS 4.11.

• Best speaker similarity across all baselines while keeping intelligibility competitive.

⚡ Exclusive Self Attention (XSA)

What’s new

• Standard self-attention output for a token has high overlap with that token’s own value vector — meaning attention wastes capacity reconstructing per-token features that the residual connection and feed-forward layer already handle. XSA subtracts out the “self-copy” component, forcing attention to focus purely on contextual (cross-token) information. Two lines of code.

How it works

• Self-copy bias happens because value vectors within a sequence tend to point in similar directions, and the self-attention score (token attending to itself) is typically high — together causing the output to look a lot like the input.

• Side benefit: acts as an implicit attention sink. Attention mass that has nowhere useful to go can safely land on the self-attention score, since the self-copy gets removed anyway.

Results

• Downstream accuracy gains: +0.26 (0.7B), +1.03 (1.3B), +1.36 (2.7B) — gains scale with model size and sequence length.

• 2.7B XSA: ARC-E 60.65, HellaSwag 67.40, LAMBADA 62.04, PIQA 77.80 (vs 58.59, 66.20, 60.18, 76.61 baseline).

⚙️ Mousse: Muon + Curvature-Aware Preconditioning

What’s new

• Muon (a recent optimizer) assumes the loss landscape curves equally in all directions — but neural networks have very uneven curvature. Mousse fixes this.

How it works

• Mousse measures the curvature along each direction (using a running average of gradient statistics, factored efficiently via Kronecker products) and adjusts step sizes accordingly — take bigger steps in flat directions, smaller steps in steep ones.

• Trace normalization: rescales the curvature estimates so their average is 1 across all layers. This is critical because raw curvature magnitudes vary wildly between layers.

• Memory ~1.05× Muon, ~0.88× SOAP.

Results

• 800M model: ~0.012 lower final validation loss than Muon; ~12% fewer steps to converge.

• Consistent wins across 160M–800M on FineWeb (20B tokens), on both cosine and WSD LR schedules.

⚡ Flash-KMeans: Fast and Memory-Efficient Exact K-Means

What’s new

• GPU k-means: the standard approach materializes a huge distance matrix (every point × every cluster) and uses contention-prone atomic writes. flash-kmeans eliminates both problems while computing exact results (not approximations).

How it works

• FlashAssign: streams data in small tiles from main GPU memory to fast on-chip memory, computing distances and finding the nearest cluster on the fly — the full distance matrix is never stored.

• This slashes memory traffic from “proportional to points × clusters” down to “proportional to points + clusters.”

• Sort-based update: sorts points by their assigned cluster, then does efficient segment-level reductions to compute new centroids.

• Handles datasets too large for GPU memory via chunked streaming; an auto-tuning heuristic matches exhaustive tuning within 0.3%.

Results

• End-to-end: 17.9× over fast_pytorch_kmeans, 33× over cuML, 200×+ over FAISS on H200.

• 1 billion points (N=10⁹, K=32768, D=128): 41.4s vs 261.8s baseline.

• Handles out-of-memory cases where PyTorch baselines crash (N=1M, K=64K, D=512).

🗣️ TADA: Text-Acoustic Dual Alignment for Speech Modeling

What’s new

• Achieves 1-to-1 alignment between text tokens and audio frames using forced alignment — this collapses the usual 10× sequence length gap between text and speech. A 2048-token context covers ~682 seconds of audio.

How it works

• A forced aligner (Wav2Vec2-Large backbone) maps each text token to its exact corresponding audio frame using Viterbi alignment.

• VAE Encoder compresses 24kHz audio down to 50 frames/sec, then a 6-layer Transformer refines these. A binary flag marks which frames correspond to text token boundaries.

• LLM Backbone (Llama 3.2 1B/3B): fuses text and acoustic embeddings by simply adding them together. Uses flow matching with a special binary encoding for duration prediction.

• Speech-Free Guidance: blends text-only and text+speech predictions at inference. At weight 0.5, recovers near-text-only language accuracy with only +0.01 RTF overhead.

• Trained on 270k hours English + 635k hours multilingual (7 languages).

Results

• RTF 0.09 (TADA-1B) — 2–8× faster than all baselines.

• SeedTTS-Eval: CER 0.73, SIM 77.9. Zero hallucinations.

• Text perplexity 19.6 on Seamless Interaction as spoken LM (beats Llama-3B-Instruct’s 20.9 while 2× smaller).

📚 Believe Your Model: Distribution-Guided Confidence Calibration

What’s new

• DistriVoting: a test-time scaling method that samples multiple reasoning paths, then uses the distribution of confidence scores to separate likely-correct from likely-incorrect answers.

How it works

• Trajectory Confidence: measures how concentrated the model’s probability is on its chosen answer (using the negative log-probability over top-k predictions at the answer step).

• SSC: tracks a running average of step-level confidence. When confidence drops sharply below the running threshold, it intervenes and forces the model to output a reflection/reconsideration token.

• GMM Filter: fits a 2-component Gaussian mixture to the confidence scores across N sampled trajectories. The higher-confidence component is presumed correct.

• Reject Filter: from the lower-confidence component, identifies the most likely wrong answer and removes all matching trajectories from the final pool.

Results

• DeepSeek-R1-8B (5-benchmark avg): Self-Consistency 73.09% → DistriVoting+SSC 77.84% (+4.75pp).

  • HMMT2025: 69.11% → 84.95% (+15.84pp).

• Qwen3-32B: 73.85% → 76.61% (+2.76pp).

• GMM: 60.46% trajectory correctness prediction, 1.78× faster than K-Means.

⚙️ SUMO: Subspace-Aware Moment-Orthogonalization (NeurIPS 2025)

What’s new

• Muon-style optimizers approximate the matrix square root using an iterative method (Newton-Schulz) that introduces error — especially when the gradient has uneven singular values, which is common during LLM training. SUMO replaces this approximation with an exact decomposition computed in a smaller, dynamically chosen subspace. Faster convergence with up to 20% less memory.

How it works

• Computes exact SVD (singular value decomposition — the precise way to find the principal directions of the gradient) within a low-dimensional subspace that adapts as the gradient’s effective rank changes during training.

• Takes optimization steps aligned with the actual curvature of the loss surface, rather than an approximation of it.

• Shows theoretically that the Newton-Schulz approximation error grows with the condition number (how spread out the gradient’s singular values are) — and this is typically high during LLM training.

Results

• Faster convergence than GaLore, Flora, and Muon on fine-tuning benchmarks.

• Up to 20% memory reduction vs comparable low-rank methods.

• Evaluated on 350M–1B+ models (Phi-2, LLaMA).

🧑‍💻 Open Source

Nutlope/open-deep-research — Open Deep Research app for generating reports with OSS LLMs.

Martian-Engineering/lossless-claw — Lossless Context Management plugin for OpenClaw.

FireRedTeam/FireRedVAD — Industrial-grade VAD and audio event detection, 100+ languages, outperforms Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD.

obra/superpowers — Agentic skills framework for coding agents. Composable specification → plan → subagent-driven development with TDD, YAGNI, DRY built in.

googleworkspace/cli — Google Workspace CLI for Drive, Gmail, Calendar, Sheets, Docs, Chat, Admin. Built from Discovery Service; includes AI agent skills.

rtk-ai/rtk — CLI proxy that reduces LLM token consumption by 60–90% on dev commands. Single Rust binary, zero dependencies.

Thanks for reading… Enjoyed this issue? Share it with a friend. 👍

Share

🧠 Steerling-8B — 8B parameter causal diffusion language model that generates text by iteratively unmasking tokens and decomposing internal representations into interpretable concepts.

from machine_learns import newsletter
newsletter.subscribe(frequency="bi-weekly")

Subscribe now

📄 FireRed-OCR-2B — 2B OCR model using GRPO to eliminate structural hallucinations in tables and LaTeX.

💻 Jan-Code-4B — 4B compact open coding model by Jan (janhq), distilled from a larger teacher, optimized for local/desktop use and tool calling via Jan Desktop or vLLM.

📎 Papers

🎨 CMT: Consistency Mid-Training for Efficient Learning of Flow Map Models

What’s new
A lightweight intermediate training stage between diffusion model pre-training and flow map post-training (e.g., Consistency Models or Mean Flow).

How it works

• A pre-trained diffusion model generates reference ODE trajectories by running a solver from noise to clean data.

• CMT trains the student to regress any intermediate point on that trajectory to the clean endpoint (output of the teacher). Simple, fixed-target regression with no stop-gradients or ad-hoc heuristics.

• Mid-trained weights initialize the actual flow map post-training (ECT, ECD, or MF), which converges faster and more stably.

• Works with both Consistency Model and Mean Flow parameterizations; architecture-agnostic. A small teacher can bootstrap a much larger student.

Results

• SOTA 2-step FIDs: 1.97 (CIFAR-10), 1.32 (ImageNet 64×64), 1.84 (ImageNet 512×512).

• Up to 98% reduction in training data budget compared to prior consistency model methods.

• Removes tricks like σ-annealing, custom time sampling, loss reweighting, and special EMA schedules.

⚡ LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

What’s new
New training objectives for speculative decoding that directly optimize acceptance rate instead of using KL divergence as a proxy. Two variants: a likelihood-based loss (negative log acceptance rate) and a hybrid loss that adaptively blends KL with Total Variation distance (TV).

How it works

• Standard draft model training minimizes KL divergence, but when draft models have limited capacity (1–5% of target params), KL’s global optimum is unreachable, and minimizing KL ≠ maximizing acceptance rate at suboptimal points.

• TV distance directly equals 1 − acceptance_rate, but has vanishing gradients and non-smooth landscape from random init (gradient norm scales as O(1/|V|)).

• The hybrid loss ℒ_LKλ starts KL-dominated (smooth gradients) and adaptively shifts to TV-dominated as acceptance improves, using α = exp(−η·acceptance_rate) scheduling. Analogous to trust-region methods.

• The likelihood-based loss ℒ_LKα = −log(acceptance_rate) provides TV-direction gradients with automatic 1/α scaling that prevents vanishing gradients.

• Handles truncated vocabularies naturally. TV/acceptance targets the original distribution, unlike KL which needs a modified target.

Results

• Consistent gains across 4 draft architectures (EAGLE-3, MLP speculator, MEDUSA, DeepSeek MTP) and 6 target models (8B to 685B params).

• Up to 8–10% improvement in average acceptance length.

• Largest gains on low-capacity drafters and large MoE targets (7.7% on GPT-OSS 120B, 8.2% on Qwen3-235B).

• DeepSeek-V3 MTP fine-tuning: +5.6% over KL at temp=1.

• Pure TV training fails badly from random init, confirming the gradient analysis.

Why it matters
Drop-in replacement for KL loss in any speculative decoding training pipeline. No computational overhead. Especially impactful when draft-target capacity gap is large (the common real-world scenario).

🧠 MLRA: Multi-Head Low-Rank Attention

What’s new
Decomposes MLA’s single latent KV head into multiple independent latent heads, enabling native 4-way tensor parallelism for decoding. Something MLA fundamentally can’t do because its single latent head can’t be shared.

How it works

• MLA compresses KV cache into a single latent head and absorbs up-projection into queries during decoding. But since the latent head can’t be partitioned, every TP (Tensor Parallelism) device must redundantly load the full KV cache.

• MLRA-4 splits the latent head into 4 blocks, independently up-projects each to form NoPE keys/values, computes attention separately per block, and sums the outputs. Each block is independently assignable to a TP device.

• Variance calibration scales query/KV latent states to fix the variance mismatch between NoPE and RoPE components (originally noted by LongCat), and rescales attention outputs after multi-branch summation.

• Uses zero initialization for output projections (from muP/LoRA insight) and optionally adds gating before output projection.

Results

• At 2.9B scale on FineWeb-Edu-100B: MLRA-4 achieves best avg perplexity (13.672 vs 13.727 MLA, 14.139 GQA) and best zero-shot reasoning (58.84% vs 58.75% MLA).

• 2.8× decoding speedup over MLA with 4-way TP.

• 1.05–1.26× speedup over GQA in long-context decoding (128K–2M tokens).

• With gating: MLRA-4 reaches 13.621 avg perplexity.

Why it matters
Solves MLA’s fundamental TP bottleneck.

💾 Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

What’s new
Context parallelism via headwise chunking, processes U attention heads at a time instead of all H, reducing peak activation memory up to 87.5% while matching Ulysses throughput. Trains 5M-token sequences on a single 8×H100 node.

How it works

• Splits multi-head attention into U-head stages, reusing activation buffers across stages. When U equals the chunk count C, peak memory becomes independent of total head count.

• GQA-compatible scheduling reuses KV heads across grouped queries.

• Drop-in replacement for Ulysses and compatible with FlashAttention kernels and USP hybrid parallelism.

Results

• Llama3-8B: 5M-token sequences (25% beyond FPDT’s limit); 8M tokens on 16×H100 (33% beyond USP-Hybrid).

• Qwen3-32B: 4M-token sequences (2× Ulysses baseline).

• 87.5% attention activation memory reduction for 32B-class models.

Why it matters
Makes million-token training practical on commodity multi-GPU nodes without model-parallel overhead.

🔬 Spectral-Decoupled MoE: Are MoE Gated Experts Just Fake Experts Trapped in Low-Rank Swamp?

What’s new
Questions whether gated experts in standard MoE models are “fake experts” trapped in a low-rank subspace, and proposes Spectral-Decoupled MoE to fix it.

How it works

• Analyzes the spectral (singular-value) structure of learned expert weight matrices and shows standard gating collapses experts into near-identical low-rank representations.

• Decouples the spectral components of expert weights so each expert can span a richer, more diverse subspace, preventing redundant experts.

• Training-time gradient decomposition:
  - Project the gradient update onto the shared low-rank subspace → update the shared/common parameters
  - The residual gradient (orthogonal part) updates the unique expert parameters

Results

• Inter-expert similarity in dominant subspaces drops dramatically (they cite ~0.1 vs ~0.7/0.9 in baselines).

• Downstream performance: ~3% gains reported.

• Training efficiency: ~30% improvement reported.

• Overhead: ~5% extra compute, designed to plug into existing MoEs (including Qwen/DeepSeek).

Why it matters
If MoE experts converge to similar low-rank solutions, the capacity gain from adding more experts is illusory and you’re paying the routing and memory cost for near-duplicate weights. Spectral decoupling could unlock the full parameter-efficiency MoE architectures promise, especially at scale.

🎵 DashengTokenizer: One Layer is Enough for Unified Audio Understanding and Generation

What’s new
A continuous audio tokenizer that handles both understanding and generation by freezing a pretrained semantic encoder and injecting acoustic detail via linear projection. Inverting the usual “acoustic-first” paradigm.

How it works

• 630M-param frozen semantic encoder provides the backbone representation.

• A tiny 0.66M-param acoustic encoder (2D conv on mel-spectrogram) captures speaker/prosody info.

• Additive fusion merges both streams; a semantic preservation loss prevents acoustic features from overwhelming the semantic signal.

• 173M-param Vocos decoder reconstructs waveforms. Single-stage training, no multi-step RVQ.

Results

• State-of-the-art on X-ARES benchmark (22 tasks): CREMA-D 80.56, ESC-50 96.40, MAESTRO 57.65.

• Competitive reconstruction quality at 25 Hz token rate.

• Outperforms VAE-based tokenizers on text-to-audio generation (FAD 2.22 vs. AudioLDM2’s 3.13).

Why it matters
Shows you can get a unified audio representation by adding a simple acoustic residual to frozen semantics. No need for complex multi-codebook quantization.

📚 Doc-to-LoRA: Learning to Instantly Internalize Contexts

What’s new
A lightweight hypernetwork that meta-learns approximate context distillation in a single forward pass, generating LoRA adapters on-the-fly for a target LLM instead of running expensive per-document fine-tuning.

How it works

• A small hypernetwork maps an input document directly to a set of LoRA weight deltas for the target LLM.

• At inference the hypernetwork runs once, producing adapter parameters that “internalize” the document into the LLM’s weights. No iterative gradient updates needed.

• Approximates full context distillation but collapses the multi-step optimization into a single forward pass, drastically cutting latency and memory.

Results

• On needle-in-a-haystack retrieval tasks, D2L achieves near-perfect zero-shot accuracy at context lengths 4× beyond the LLM’s native window.

• On real-world long-document QA benchmarks it outperforms standard context distillation while using significantly less peak memory and update time.

Why it matters
Removes the main bottleneck of context distillation, the per-document fine-tuning loop, making it practical for on-the-fly deployment. Enables effective use of very long contexts without extending the base model’s context window, useful for retrieval-augmented and document-grounded applications.

🎵 FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

What’s new
A dynamic-frame-rate neural audio codec that adaptively merges semantically similar frames, letting a single model operate anywhere from 3 Hz to 12.5 Hz instead of the fixed 50 Hz typical of current codecs.

How it works

• An ASR-feature-assisted dual-stream encoder processes the audio: one stream captures semantic content, the other acoustic detail.

• A transformer bottleneck learns to merge consecutive frames whose ASR features are similar, producing a variable-length discrete token sequence.

• The merge ratio is controllable at inference, so users can trade off frame rate against reconstruction fidelity depending on the downstream task.

• A matching decoder reconstructs waveforms from the variable-rate token stream.

Results

• At 6.25 Hz: 4.15% WER on speech intelligibility vs. 31.5% WER for DualCodec baseline at the same rate.

• Delivers competitive TTS quality at substantially lower token rates, translating to significant autoregressive-decoding speedups.

• Maintains high audio fidelity across the full 3–12.5 Hz operating range.

Why it matters
Dramatically reduces the sequence length LLM-based TTS systems must generate, directly cutting inference cost and latency. A single controllable codec replaces multiple fixed-rate models, simplifying the speech-generation pipeline. The ASR-guided merging principle could generalize to other modalities where redundant frames dominate.

🗣️ MSR-Codec: Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation

What’s new
A low-bitrate multi-stream residual codec that factorizes speech into four explicit streams; semantic, timbre, prosody, and residual. Achieves implicit disentanglement without adversarial losses (unlike NaturalSpeech 3), relying instead on progressive residual fusion.

How it works

• Timbre: frozen CAM++ speaker encoder → single L2-normalized utterance-level embedding.

• Semantic: frozen HuBERT → 25 Hz features quantized into discrete tokens (500-center codebook).

• Prosody: Enc1 + Enc2 downsample mel-spec to 12.5 Hz; residual between Enc2 output and Dec1 (timbre+semantic) output is quantized via VQ1; explicit F0/energy prediction loss supervises the prosody codebook.

• Residual: 25 Hz Enc1 output minus upsampled prosody-enhanced stream, quantized via VQ2 for fine-grained acoustic detail.

• Streams progressively fused through residual connections across Dec1→Dec2→Dec3; pre-trained Fre-GAN vocoder converts final mel-spec to waveform.

• Two-stage AR TTS model: Stage 1 Semantic Decoder (18-layer transformer) predicts semantic tokens at 12.5 Hz; Stage 2 Acoustic Decoder (3-layer transformer) predicts prosody + residual tokens per frame.

Results

• 200×+ compression (62.5 tokens/s) at 424–612 bps with highest speaker similarity (0.80–0.83) among compared codecs.

• TTS (0.2B params, 45k hrs data): SOTA WER on Seed-TTS-eval, highest speaker similarity, and lowest RTF (0.67) , beating FireRedTTS (0.4B/150k hrs), CosyVoice2 (0.5B/167k hrs), and Llasa-1B.

• Voice conversion validates disentanglement: timbre swap preserves source prosody (low ΔF0,src); prosody transfer shifts F0 toward target while maintaining source timbre.

Why it matters
Shows that a simple cascaded residual architecture can achieve strong disentanglement without complex adversarial training. Extremely lightweight and data-efficient TTS system that outperforms much larger models in accuracy and speed.

🔍 Multi-Vector Index Compression in Any Modality

What’s new
Attention-Guided Clustering (AGC) for compressing multi-vector document indices in any modality; text, visual documents, and video. Shows that full multimodal indices are massively over-provisioned (base models use only ~1% of tokens during retrieval).

How it works

• Learned “universal query” tokens attend over all document tokens to compute per-token saliency scores.

• Top-k salient tokens become cluster centroids; remaining tokens hard-assigned to nearest centroid by cosine similarity.

• Weighted aggregation pools each cluster using saliency as importance weights — keeps gradients flowing through hard assignments.

• Compared against SeqResize (MLP projection along sequence dim), MemTok (learnable appended tokens), and H-Pool (agglomerative merging).

Results

• AGC consistently outperforms other learned compression (SeqResize, MemTok).

• More flexible than H-Pool (can target specific budgets cleanly).

• Competitive with, and sometimes better than uncompressed indices on ViDoRe and MSR-VTT (compression acts like denoising).

Why it matters
Makes ColBERT-style late-interaction retrieval practical for large multimodal collections. ~98% index compression with negligible quality loss. Key insight: multimodal tokens (audio frames, video patches) are far more redundant than text tokens.

🗣️ MeanVoiceFlow: One-Step Nonparallel Voice Conversion with Mean Flows

What’s new
One-step nonparallel voice conversion model based on mean flows — trained from scratch, no teacher model or distillation required. Matches multi-step quality in a single forward pass.

How it works

• Replaces instantaneous velocity (standard flow matching) with average velocity, which computes the exact time integral between two points in one step. No ODE solver discretization error.

• A structural margin reconstruction loss (SSIM-based + margin threshold) acts as a zero-input constraint to stabilize learning without causing over-smoothed outputs.

• Conditional diffused-input training: mixes noise and source mel-spectrogram during both training and inference, maintaining consistency and letting the model leverage source speaker info.

Results

• Matches multi-step flow-matching and distillation-based VC models on speech quality and speaker similarity.

• Outperforms other from-scratch one-step approaches.

• Single-step inference, no iterative sampling.

Why it matters
Removes the pretraining → distillation pipeline from one-step VC: simpler training, same quality. Mean flows are an underexplored alternative to consistency models and shortcut models for single-step generation.

⚙️ PISA: Preconditioned Inexact Stochastic ADMM for Deep Models

What’s new
A new optimizer family that replaces SGD-based training with an ADMM-based framework. Two efficient variants: SISA (second-moment-based) and NSISA (Newton-Schulz orthogonalized momentum).

How it works

• Reformulates training as a constrained optimization problem, splitting data across m batches with auxiliary variables and Lagrange multipliers.

• Solves subproblems inexactly using stochastic gradients with adaptive preconditioning matrices (second-moment like Adam, Hessian, or Newton-Schulz orthogonalization like Muon).

• Global parameter aggregation step followed by parallel local updates, naturally supports federated/distributed settings.

• Convergence proven under only Lipschitz continuity of the gradient on a bounded region. No bounded variance, IID, or unbiased gradient assumptions needed.

Results

• NSISA outperforms AdamW, Muon, Shampoo, SOAP, and Adam-mini on GPT-2 Nano/Medium/XL.

• SISA achieves lowest FID on GAN training (WGAN, WGAN-GP on CIFAR-10).

• Competitive or best accuracy on vision models (VGG-11, ResNet-34, DenseNet-121, ImageNet ResNet-18).

🧑‍💻 Open Source

RightNow-AI/openfang — Open-source Agent Operating System.

rlacombe/distillate — Automates paper reading workflow between Zotero and reMarkable.

D4Vinci/Scrapling — 🕷️ Adaptive web scraping framework handling everything from single requests to full-scale crawls.

koala73/worldmonitor — Real-time global intelligence dashboard with AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking.

Thanks for reading… Enjoyed this issue? Share it with a friend. 👍

Share

🎙️ Ming-omni-tts-0.5B — Voice core of Ming-flash-omni-2.0: 0.5B TTS model with voice cloning from inclusionAI (Ant Group).

from machine_learns import newsletter
newsletter.subscribe(frequency="bi-weekly")

Subscribe now

🤖 JoyAI-LLM-Flash-Base — State-of-the-art MoE LLM (3B activated) optimized for agents and high throughput.

🌐 Ming-flash-omni 2.0 — Open-source SOTA omni-MLLM (100B total / 6B active MoE): multimodal understanding + speech/audio/music synthesis + image generation/editing in one model.

🎙️ MOSS-TTS — Open-source TTS family with high-fidelity zero-shot voice cloning, controllable long-form synthesis (up to 60 min), multilingual support, and multi-speaker dialogue generation (5 speakers).

📱 HY-1.8B-2Bit — Tencent AngelSlim’s 2-bit QAT of Hunyuan-1.8B-Instruct. Only 4% degradation vs. full precision, outperforms 0.5B dense models by 16%. GGUF format for edge deployment.

🧠 Step 3.5 Flash — StepFun’s open-source frontier reasoning MoE: 196B total / 11B active per token. 3-way MTP for 100–350 tok/s. 74.4% SWE-bench, 256K context. Rivals top proprietary models; runs on Mac Studio M4 Max.

🎙️ KugelAudio — Open TTS model for 24 European languages trained on 200K hours of YODAS2 data. Built on Microsoft VibeVoice with voice cloning. Beats ElevenLabs in human preference testing.

🔬 OpenResearcher — 30B-A3B MoE model + 96K DeepResearch trajectories + 11B-token corpus + benchmarks. Fully open deep research agent by TIGER-Lab.

🎵 UniAudio 2.0 — Unified audio language model using discrete audio codec for text-aligned representation with autoregressive architecture for multi-task training across diverse audio tasks.

📎 Papers

🧠 LCM: Lossless Context Management

What’s new

A lossless context management framework that preserves full data integrity while enabling efficient information retrieval in LLMs.

How it works

• Uses AI-driven techniques for context management

• Employs lossless compression to preserve complete data integrity

• Integrates a tagging system for efficient information retrieval

• Enhances user interaction with contextual data through structured organization

Results

• Not specified in paper

Why it matters

Addresses key challenges in data management for long-context applications and could improve AI systems across various domains.

🎵 MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

What’s new

A large-scale audio tokenizer (1.6B parameters) using a fully end-to-end learning approach with a homogeneous architecture.

How it works

• CAT (Causal Audio Tokenizer with Transformer) architecture

• Jointly optimizes encoder, quantizer, and decoder from scratch

• Purely Transformer-based design for high-fidelity audio reconstruction

• Scales well across speech, sound, and music domains

Results

• Outperforms prior codecs across various bitrates for audio reconstruction

• First autoregressive TTS model to surpass non-autoregressive and cascaded systems on similarity metrics.

• Competitive ASR performance without auxiliary encoders

My 2 cents

Best discrete codec I’ve tried for the given bitrate wrt both semantic and acoustic reconstruction.

📦 Proxy Compression for Language Modeling

A mixed-representation training that uses external compressors (tokenizer, neural, gzip) only as a training-time proxy, then discards them for pure byte-level inference.

How it works

• Jointly trains one autoregressive LM on raw UTF-8 bytes (10%) and compressed views (90%) packed into the same contexts

• Special tokens (⟨raw⟩/⟨comp⟩) informs the model what representation it takes in

• In-context translation pairing during warmup: both views of the same sample in one context to bootstrap cross-representation alignment

• Three proxy compressors tested: BPE tokenizer (~3.7× compression), neural arithmetic coder (~2.6×), and gzip (~2.5×)

• Neural compressor uses entropy-based segmentation for parallel encoding; produces “fuzzy” many-to-one mappings where collisions only differ in low-entropy details (whitespace, indentation)

Results

• At 14B scale, proxy-trained models match or surpass tokenizer-based baselines on HumanEval-Plus and MBPP-Plus — while running on raw bytes

• Transfer strength scales with model size: weak/negative at 0.5B, competitive at 4B, surpassing at 14B

• gzip fails as a proxy (negative transfer) due to unstable output under small input perturbations

• Proxy models retain byte-level robustness: best Robust Pass@1 (19.8) vs. tokenizer baseline (14.9) on ReCode

🎵 STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information

What’s new

Semantic Token Assignment (STA) for audio codecs that decouples semantic structure from acoustic representation.

How it works

• Uses semantic token indices from K-means on SSL models (WavLM/HuBERT) as RVQ-1 code indices - just indices not the real embeddings

• Yet it learns RVQ-1 codebook embeddings for acoustic reconstruction from scratch.

• Semantic Pre-Distillation (SPD) predicts semantic tokens before quantization

• SPD employs random masking (temporal + feature dimensions) to reduce overfitting

• Eliminates the SSL model at inference.

Results

• ViSQOL score of 4.51 and WER of 9.35%

• Balanced codebook utilization across all RVQ layers

🎙️ PFluxTTS: Hybrid Flow-Matching TTS with Robust Cross-Lingual Voice Cloning

What’s new

A hybrid flow-matching TTS system with robust cross-lingual voice cloning and inference-time model fusion.

How it works

• Uses flow-matching for text-to-speech synthesis

• Implements voice cloning that adapts across languages

• Uses two decoder models; duration-guided model and alighment-free model

• Interpolates outputs from 2 decoder models at inference time for better quality and consistency.

Results

• Improved voice quality and naturalness in TTS outputs

• Enhanced cross-lingual voice cloning capability

⚡ HySparse: Hybrid Sparse Attention with Oracle Token Selection and KV Cache Sharing

What’s new

A hybrid sparse attention architecture that interleaves full attention layers with sparse attention layers, using oracle token selection and cross-layer KV cache sharing.

How it works

• Interleaves full attention layers with multiple sparse attention layers

• Full attention layers identify important tokens and produce KV caches

• Sparse layers reuse token indices and KV caches from preceding full attention layers

• Uses block-level maximum attention scores for efficient TopK selection

• Two branches: Block Sparse Attention (reuses KV) + Sliding Window Attention (local KV cache)

• Sigmoid gates fuse outputs from both branches

• Cross-layer KV cache sharing reduces memory footprint

Results

• Outperforms full attention and hybrid SWA baselines

• Achieves significant KV cache reduction while maintaining performance

• In 80B MoE model, only 5 of 49 layers use full attention

🎬 Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization

What’s new

A training-free KV-cache quantization framework for auto-regressive video generation that leverages video-specific spatiotemporal redundancy.

How it works

• Semantic-Aware Smoothing: Groups semantically similar tokens via k-means clustering, producing quantization-friendly residuals

• Progressive Residual Quantization: Multi-stage scheme that progressively refines residuals from coarse to fine granularity

• KV-Cache Management: Quantizes KV-cache (which grows linearly with tokens), reducing memory up to 7× while maintaining quality

• Introduces minimal latency overhead (1.5%–4.3%)

Results

• Outperforms SOTA KV-cache quantization baselines

• Up to 6.94× compression ratio with near-lossless quality

• Enables long video generation on limited hardware (e.g., HY-WorldPlay-8B on single RTX 4090)

🧑‍💻 Open-Source

🐍 pydantic/monty — A minimal, secure Python interpreter written in Rust for use by AI.

🗄️ alibaba/zvec — A lightweight, lightning-fast, in-process vector database.

Thanks for reading… Enjoyed this issue? Share it with a friend. 👍

Share

💻 Qwen3-Coder-Next — 80B-parameter text generation model optimized for coding and language tasks.

from machine_learns import newsletter
newsletter.subscribe(frequency="bi-weekly")

Subscribe now

🎭 MiniCPM-o 4.5 — 9B-parameter multimodal model for real-time, full-duplex audio and video processing with bilingual speech conversation and advanced visual capabilities.

🔊 MMS-300M Forced Aligner — Python package for efficient forced alignment of text and audio using Hugging Face pretrained models, with improved memory usage over TorchAudio.

🎬 LingBot-World — Open-source world simulator for video generation featuring high-fidelity environments, long-term memory, and real-time interactivity.

💻 Step 3.5 Flash — Open-source sparse MoE foundation model for efficient reasoning and agentic tasks, processing 100–300 tokens/sec with 256K context window.

🗣️ KugelAudio-0-Open — Open-source TTS model for European languages with voice cloning, using a 7B-parameter AR + Diffusion architecture trained on ~200K hours of speech.

👂 Qwen3-ASR — Open-source ASR series from Alibaba Cloud supporting multilingual speech, music, and song recognition with language detection and timestamp prediction.

🖼️ Z-Image — 6B-parameter single-stream diffusion transformer for efficient image generation, editing, and bilingual text rendering.

🎭 Kimi-K2.5 — Image-text-to-text model from Moonshot AI built on the Transformers library.

💻 Stable-DiffCoder-8B-Instruct — Code diffusion LLM built on Seed-Coder architecture with block diffusion continual pretraining for improved code generation, reasoning, and editing.

🗣️ LuxTTS — Lightweight TTS model for voice cloning achieving 150x+ realtime speed while fitting in 1GB VRAM.

🎬 Linum v2 — Open-weight 2B-parameter text-to-video model generating 2–5 second clips at up to 720p for experimentation in generative media.

👂 VibeVoice-ASR — Unified speech-to-text model processing up to 60 minutes of long-form audio in a single pass with speaker identification, timestamps, and user-customized context support.

📎 Papers

🧠 A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training

What’s new

A framework explaining outlier-driven rescaling in transformers, identifying attention sinks and residual sinks as functional components rather than artifacts. Proposes GatedNorm and PreAffine methods to mitigate outliers while preserving performance.

How it works

• Outliers interact with normalization mechanisms (softmax and RMSNorm) to rescale non-outlier components

• Attention sinks: specific tokens receive disproportionately high attention scores

• Residual sinks: fixed dimensions exhibit consistently high activations across tokens

• GatedNorm: element-wise low-rank self-gating after normalization layers

• PreAffine: learnable scaling vector before normalization to enable outlier-driven rescaling without large residual values

Results

• Improved training stability and performance across various models

• Enhanced quantization robustness under aggressive low-bit settings

• Gains in knowledge, reasoning, STEM, code generation, and multilingual tasks

🌫️ Causal Autoregressive Diffusion Language Model

What’s new

Unifying the training efficiency of autoregressive models with high-throughput inference of diffusion models.

How it works

• Strictly causal attention mask reformulates the diffusion process

• Shifted causal attention where each position predicts its original token from preceding noised context

• Dense supervision for entire sequences in a single forward pass

• Soft tail masking concentrates noise at the sequence tail

• Context-aware reweighting adjusts loss weights based on local ambiguity

• Dynamic parallel decoding with KV-caching generates variable-length sequences based on confidence

Results

• Outperforms existing discrete diffusion models by 5.7+ points in zero-shot accuracy

• ARM-level data efficiency with 3× reduced training latency vs. block diffusion methods

• Lowest zero-shot perplexity across multiple domains

🧠 Residual Context Diffusion Language Models

What’s new

A mechanism that recycles computation from discarded low-confidence tokens instead of discarding them at every denoising steps to improve dLLM accuracy.

How it works

• Transforms discarded token representations into contextual residuals for the next denoising step

• Two-stage training: lightweight reference model generates reliable probability distributions; target model incorporates residuals using reference model as stable guide

• Entropy-based embedding aggregation selects and aggregates context

• Dynamically adjusts residual contribution based on normalized Shannon entropy

Results

• 5–10 point accuracy improvement on various benchmarks with minimal extra computation

• Nearly doubles baseline accuracy on challenging AIME tasks

• Reduces denoising steps by 4–5× at equivalent accuracy levels

🧷 Scaling Embedding Layers in Language Models

What’s new

Scone (Scalable, Contextualized, Offloaded, N-gram Embedding), a method enhancing input embeddings without increasing decoding costs.

How it works

• Retains original vocabulary while adding embeddings for frequent n-grams

• Separate transformer (f-gram model) learns contextualized representations

• Embeddings precomputed and stored in off-accelerator memory

• Avoids sparse update problem by parameterizing embeddings with f-gram model

• F-gram layer can be offloaded, maintaining fixed accelerator resources during inference

Results

• 1B accelerator-resident parameter model outperforms 1.9B baseline

• Uses approximately half the FLOPs and accelerator memory during inference

🏗️ ConceptMoE: Adaptive Token-to-Concept Compression

What’s new

Adaptive token-to-concept compression for implicit compute allocation, dynamically merging semantically similar tokens within MoE architectures.

How it works

• Learnable chunk module identifies boundaries based on inter-token similarity

• Consecutive high-similarity tokens merge into concept representations

• MoE architecture enables controlled evaluation by reallocating saved computation

• Minimal architectural changes for straightforward integration

Results

• +0.9 points on language pretraining, +2.3 on long context understanding, +0.6 on multimodal benchmarks

• +5.5 points when converting pretrained MoE during continual training

• Prefill speedups up to 175%, decoding speedups up to 117%

🏗️ TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon

What’s new

A generalization of the Muon optimizer extending orthogonalization beyond individual layers for LLM pre-training.

How it works

• Models gradients as structured higher-order tensors

• Performs matrix-level gradient orthogonalization across layers simultaneously

• Provides improved convergence guarantees over layer-wise Muon

• Robust under different approximate SVD schemes

Results

• Evaluated on GPT-style (130M–774M) and LLaMA-style (60M–1B) models

• Consistently improves training and validation perplexity across scales

🔊 High-Fidelity Generative Audio Compression at 0.275kbps

What’s new

Achieving high-fidelity audio at ultra-low bitrates (0.275kbps), shifting from signal fidelity to task-oriented effectiveness.

How it works

• Integrates semantic understanding at the transmitter with generative synthesis at the receiver

• Two-stage process: learns compressed semantic representation aligned with linguistic supervision, then recovers high-fidelity audio via large generative model

• Encoder filters redundancy, transmitting only semantic essence

• Decoder reconstructs details from model priors

Results

• High-fidelity 32kHz audio reconstruction at 0.275kbps with 3000× compression ratio

• Outperforms SOTA neural codecs in perceptual quality and semantic consistency

• Maintains intelligible transmission even at 0.175kbps

🔊 SemanticAudio: Audio Generation and Editing in Semantic Space

What’s new

A two-stage Flow Matching framework (Semantic Planner + Acoustic Synthesizer) for audio generation and editing in high-level semantic space with training-free text-guided editing.

How it works

• Semantic Planner: generates compact semantic features from text using dual inputs (global sentence embedding + token-level embeddings)

• Acoustic Synthesizer: produces high-fidelity acoustic latents conditioned on semantic features

• Editing: delta velocity fields from source/target prompts enable semantic-level modifications without retraining

Results

• Superior semantic alignment (CLAP score 0.354)

• High reconstruction fidelity with low Mel and STFT loss

• Robust editing capabilities even without source text

🎬 Memory-V2V: Augmenting Video-to-Video Diffusion Models with Memory

What’s new

A framework for multi-turn video editing with cross-consistency, using explicit memory to enhance video-to-video diffusion models.

How it works

• Lightweight memory modules integrate into V2V models

• Dynamic tokenization with varying kernel sizes based on edit relevance

• Retrieval mechanism identifies relevant past edits from external cache

• Learnable token compressor reduces redundancy while preserving visual cues (30% speedup)

Results

• Strong cross-iteration consistency in novel view synthesis and text-guided long video editing

• Outperforms SOTA baselines in visual quality and computational efficiency

🔁 Self-Distillation Enables Continual Learning

What’s new

Self-Distillation Fine-Tuning for continual learning, enabling on-policy learning directly from expert demonstrations.

How it works

• Single model serves as both teacher and student.

• Teacher conditioned on task [prompt + example demonstration]; student uses only the prompt

• Training minimizes reverse KL divergence between teacher and student outputs

• Model learns from own generated trajectories while preserving prior capabilities

Results

• Higher new-task accuracy and better retention than SFT

• Single model acquires multiple skills sequentially without performance degradation

• Outperforms baselines on both in-distribution and out-of-distribution tasks

🔁 Teaching Models to Teach Themselves

What’s new

SOAR, a self-improvement framework using meta-RL where a teacher model generates automated curricula for problems the student cannot yet solve.

How it works

• Optimization with outer Teacher and inner Student loop stages.

• Teacher proposes synthetic problems for inner looper Student optimization.

• After N inner loops steps, Teacher is rewarded based on Student’s improvement. (Teacher learns to teach in the outer looper)

• Both stages uses RLOO for optimization.

Results

• Bi-level meta-RL facilitates learning despite sparse rewards

• Structural quality of generated questions more important than solution correctness

🔁 Recursive Self-Aggregation Unlocks Deep Thinking in LLMs

What’s new

Combining parallel and sequential scaling methods to improve LLM reasoning.

How it works

• Maintains population of candidate solutions at each step

• Aggregates subsets to iteratively produce improved solutions

• Inspired by evolutionary algorithms, enables model to revisit and correct reasoning

• RL training teaches effective solution aggregation

Results

• Significant performance improvements across tasks vs. traditional methods

• Bridges gap between smaller and larger reasoning models

🔁 Training-Free Group Relative Policy Optimization

What’s new

A method enhancing LLM performance without parameter updates by leveraging experiential knowledge as token priors.

How it works

• Uses group-based rollouts to distill semantic advantages from multiple outputs

• Maintains frozen model; updates external experiential knowledge library

• Each step: generate outputs, score them, extract semantic advantages based on performance

• Adapts to new scenarios with minimal training data

Results

• Significant improvements in mathematical reasoning and web searching

• Outperforms fine-tuned models with fewer samples and lower costs

• Strong performance on AIME24, AIME25, and WebWalkerQA

🧑‍💻 Open Source

FastGen — NVIDIA library for fast generation from diffusion models.

Enjoyed this issue? Send it to a friend who’d appreciate it.

Share

A note on this post: BlaGPT is the codebase where I benchmark trending techniques and models. I recently started writing brief technical notes on these implementations, and I decided to share them here on Substack. So, here we go…

Overview

Engram addresses the lack of native knowledge lookup mechanisms in transformer architectures. While Mixture-of-Experts (MoE) enables conditional computation, transformers lack a complementary approach for accessing static, pre-computed memories. Engram introduces deterministic n-gram hashing to retrieve token-pattern embeddings with O(1) complexity, offloading large tables to CPU memory with minimal inference overhead.

The method treats static knowledge retrieval independently from neural computation, enabling models to balance between dynamic processing and static memory access.

if liked && want_next_issue:
   subscribe();

Subscribe now

Method

Core Mechanism

Engram computes hash indices for n-gram token patterns and retrieves corresponding embeddings from per-layer tables:

1. Hash n-gram patterns: For each position in the sequence, combine n consecutive tokens using XOR hashing

2. Lookup embeddings: Index into layer-specific embedding tables

3. Gate with hidden states: Modulate retrieved embeddings using context-dependent gating

4. Add as residual: Combine gated embeddings with the transformer’s forward pass

Hash Function

For n consecutive tokens at position i:

hash = (t[i] × m[0]) ⊕ (t[i-1] × m[1]) ⊕ ... ⊕ (t[i-n+1] × m[n-1])
index = hash mod table_size

Where:

• t[i] is the token ID at position i

• m[k] are fixed random multipliers (odd integers)

• ⊕ denotes XOR operation

• Padding tokens are used for positions near sequence boundaries

Gating Mechanism

Retrieved n-gram embeddings are modulated by the current hidden state:

key = W_key(concat(ngram_embeds))
value = W_value(concat(ngram_embeds))
gate = sigmoid((norm(key) · norm(hidden)) / √d)
output = gate × value

This preserves context-awareness while accessing static memories.

Variants in BlaGPT

BlaGPT includes 3 Engram variants with these common changes:

Removed the following things to keep it simple and focus on the core idea:

1. CompressedTokenizer: No text normalization before hashing

2. Multi-head hashing: Single hash function instead of finding optimal primes

3. ShortConv refinement: No additional convolution over n-gram embeddings

4. Hyper-connection complexity: Simplified residual connection patterns

and retrained:

• XOR-based n-gram hashing with random multipliers

• Per-layer embedding tables

• Context-dependent gating

• Deterministic addressing (no learned routing)

SimpleEngram

Standard implementation with multiple n-gram sizes (default: bigrams and trigrams):

• Separate embedding tables per n-gram size

• Projects concatenated embeddings to key/value

• RMSNorm for stable gating

NgramLambdaEngram

Layer-wise mixing variant inspired by bigram models:

• Single n-gram embedding table shared across layers

• Per-layer learnable mixing coefficients (λ_x, λ_x0, λ_ngram)

• Mixes current hidden state, initial embedding, and n-gram embedding

• Applied between transformer blocks

MinimalEngram

Best results in BlaGPT experiments

I simplified version the original idea to:

• Single n-gram size (default: trigrams)

• Concatenates hidden state with n-gram embedding for gating

• Minimal projections

BlaGPT Implementation

Basic Usage

from bla_gpt.engram import SimpleEngram, NgramLambdaEngram, MinimalEngram

# Standard variant with bigrams and trigrams
engram = SimpleEngram(
    hidden_size=512,
    vocab_size=32000,
    ngram_sizes=(2, 3),
    embed_dim=256,
    table_size=500_000,
)

# Apply in transformer block
output = engram(hidden_states, input_ids)  # [B, L, D]
hidden_states = hidden_states + output

Shared Embeddings

Reduce memory by sharing tables across layers:

from bla_gpt.engram import create_shared_simple_engram_embeddings

# Create shared tables
shared_embs = create_shared_simple_engram_embeddings(
    table_size=500_000,
    embed_dim=256,
    ngram_sizes=(2, 3),
)

# Use in multiple layers
engram_layer_1 = SimpleEngram(hidden_size=512, vocab_size=32000,
                               shared_embeddings=shared_embs)
engram_layer_2 = SimpleEngram(hidden_size=512, vocab_size=32000,
                               shared_embeddings=shared_embs)

Layer-wise Mixing

# Create model-level wrapper
ngram_engram = NgramLambdaEngram(
    vocab_size=32000,
    model_dim=512,
    num_layers=6,
    ngram=2,  # bigrams
    ngram_vocab_mult=5,
)

# Get n-gram embeddings once
x0_ngram = ngram_engram.get_ngram_embedding(input_ids)

# Mix at each layer
for layer_idx in range(num_layers):
    x = ngram_engram.mix_at_layer(x, x0, x0_ngram, layer_idx)
    x = transformer_block(x)

🛠️ Build With It

The full implementation of all three Engram variants—including the high-performing MinimalEngram—is available now in the BlaGPT.

I have designed the module to be plug-and-play, so you can drop it into your own Transformer blocks with just a few lines of code.

👉 View the Engram Implementation on GitHub

Plus, feel free to open a PR!

Kudos for reading 🙏 …

Step3-VL-10B: Compact Yet Frontier Multimodal Intelligence 🖼️🧠
Step3-VL-10B is a 10B-parameter lightweight open-source multimodal base model focused on visual perception, complex reasoning, and human alignment, using unified pretraining and large-scale multimodal RL to beat peers in its size class.

if liked && want_next_issue:
   subscribe();

Subscribe now

openbmb/AgentCPM-Report · Hugging Face 📊✍️
AgentCPM-Report is an open-source LLM agent specialized for long-form report generation from instructions, co-developed by THUNLP, Renmin University of China, and ModelBest.

Sparrow-1: Human-Level Conversational Timing in Real-Time Voice 🗣️⏱️
Sparrow-1 is a multilingual audio model that predicts when to listen, wait, or speak to match human conversational timing in real-time voice interactions.

LightOnOCR-2-1B: a lightweight high-performance end-to-end OCR model family 📄🔍
LightOnOCR-2-1B is a 1B-parameter end-to-end vision-language OCR model that converts document pages to clean text and figure/image bounding boxes, optimized for fast, practical document workflows.

zai-org/GLM-4.7-Flash · Hugging Face 💬⚡
GLM-4.7-Flash is a ~30B-class model tuned for lightweight deployment, aiming to balance strong performance with much lower inference cost.

nvidia/personaplex-7b-v1 · Hugging Face 🗣️🤝
Personaplex is a 7B real-time speech-to-speech conversational model that supports streaming understanding and generation with interruptions and rapid turn-taking.

black-forest-labs/FLUX.2-klein-4B · Hugging Face 🎨⚙️
FLUX.2 [klein] 4B is a 4B-parameter image generation and editing model designed for real-time use on consumer GPUs with ~13GB VRAM.

SeedFold 🧬🔬
SeedFold is a protein folding model that uses width scaling and large-scale data distillation to boost capacity and sets new state-of-the-art on FoldBench.

microsoft/FrogMini-14B-2510 · Hugging Face 👨‍💻🪲
FrogMini is a 14B-parameter coding agent optimized for multi-turn debugging, trained on a mix of real-world and synthetic bug datasets.

zai-org/GLM-Image · Hugging Face 🖼️🌀
GLM-Image is a text-to-image and image-to-image generator with a hybrid autoregressive–diffusion decoder that focuses on semantic understanding and fine detail.

MedGemma: Our most capable open models for health AI development 🩺📷
MedGemma is a 4B/27B multimodal generative family for healthcare, supporting text and medical images for tasks like report generation and image classification.

CALM Samples – kyutai · Hugging Face Space 🔊📡
CALM is a Continuous Audio Language Model that operates directly on continuous audio with a Transformer backbone, enabling high-quality speech and music generation without lossy compression.

📎 Papers

🧠 Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge

What’s new

• A stochastic soft reasoning mechanism for LLMs.

• Lets models explore multiple next-token hypotheses without blowing up sequence length.

How it works

• At each reasoning step, samples K candidate next tokens.

• Aggregates their embeddings into a single continuous multiplex token, preserving the vocabulary prior and sampling dynamics.

• Induces a tractable probability distribution over multiplex rollouts.

• Self-adaptive behavior:

  • Confident: multiplex tokens become nearly discrete, behaving like standard CoT.

  • Uncertain: compactly encodes several plausible continuations in one step.

• Trained via on-policy reinforcement learning on reasoning tasks.

Results

• Outperforms strong discrete CoT and RL baselines on hard math benchmarks.

• Gains hold from Pass@1 all the way to Pass@1024.

• Produces shorter reasoning traces than traditional multi-sample methods.

Why it matters

• Increases reasoning capacity without linear growth in token count.

• Provides a principled way to “branch and merge” thought processes inside the LM.

• Directly addresses the inefficiency of long, low-bandwidth token sequences for complex reasoning.

🔊 SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision

What’s new

• A self-supervised speech representation model for spoken language modeling.

• Significantly outperforms HuBERT, DinoSR, etc., on multiple benchmarks.

• Trains in about one day on 16 GPUs, much faster than prior work.

How it works

• Trained directly on raw waveforms with a masked prediction objective.

• Uses self-distillation with online clustering:

  • A teacher network produces intermediate representations.

  • A student predicts cluster assignments, stabilizing the clustering process.

• Focuses on extracting phonetic and linguistic units directly from speech (no text).

• Evaluates unit quality (ABX, PNMI) and correlates it with language modeling performance.

Results

• Strong zero-shot performance on textless spoken LM tasks (sWUGGY, sBLIMP, tSC).

• Lower ABX error and higher MAP vs. previous models.

• Better training stability and representation quality at lower compute.

Why it matters

• Enables building powerful speech-language models directly from audio, bypassing text.

• Especially promising for low-resource languages where transcriptions are scarce.

🔊 What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study

What’s new

• Systematic analysis of speech tokenizers for LLM-centric speech generation.

• Multi-token prediction (MTP) to speed up decoding and reduce WER.

• Releases RoleTriviaQA, a benchmark for role-playing QA with diverse speaker identities.

How it works

• Compares three tokenizer designs: coupled, semi-decoupled, fully decoupled.

• Finds decoupled tokenizers improve cross-modal alignment and synthesis quality.

• MTP allows each hidden state to decode multiple speech tokens:

  • Increases information density in speech tokens.

  • Better matches the density of text representations.

• Adds speaker-aware generation by conditioning on speaker identity features.

• Evaluates on role-playing knowledge QA, measuring knowledge retention and speaker consistency.

Results

• Decoupled tokenizers consistently outperform coupled/semi-decoupled in alignment and quality.

• MTP speeds decoding by up to 12× and cuts WER from 6.07 → 3.01.

🔊 Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers

What’s new

• A single autoregressive framework for TTS, ASR, and voice conversion.

• Unifies major speech tasks into one model instead of separate architectures.

How it works

• Operates over a shared discrete audio token space.

• Uses dual tokenizers (GLM + BiCodec) for robust speech discretization.

• Instruction-driven task switching: same architecture, different prompts for TTS/ASR/VC.

• Joint multi-task training shares representations across tasks.

• Fully autoregressive architecture supports streaming and high-throughput generation.

Results

• Competitive or better than task-specific models on standard speech benchmarks.

• High concurrency and throughput suitable for real-time deployment.

• Gains from multi-task training vs. siloed models.

📏 Recursive Language Models

What’s new

• Recursive Language Models (RLMs) for long-prompt processing.

• RLMs dramatically outperform standard LLMs on extreme long-context tasks.

How it works

• Treats the long prompt as an external environment rather than a flat sequence.

• Uses a Read–Eval–Print Loop (REPL) to interact with the prompt.

• Programmatically decomposes long inputs and recursively calls itself on sub-snippets.

• Recursion allows handling context lengths far beyond the raw context window.

• Maintains accuracy as context grows, where standard models degrade.

Results

• Strong gains over base LLMs and common long-context methods.

• Handles inputs up to 10M+ tokens effectively.

Why it matters

• Provides a scalable path to long-horizon reasoning, retrieval, and coding tasks.

• Tackles the core limitation of context windows without needing massive architectural changes.

📏 RePo: Language Models with Context Re-Positioning

What’s new

• A differentiable mechanism for context re-positioning in LLMs.

• Loosens the rigid positional index structures used by standard transformers.

How it works

• Motivated by Cognitive Load Theory: reduces “extraneous” load from bad positional layouts.

• Learns a module to assign token positions based on contextual dependencies.

• Positions live in a dense, non-linear space instead of fixed integer indices.

• Continual pretraining on OLMo-2 1B with RePo improves context handling.

Results

• Better performance on noisy contexts, structured data, and long contexts.

• Maintains competitive accuracy on usual short-context benchmarks.

Why it matters

• Gives models more freedom to allocate attention where it matters, not where position encoding forces it.

🧱 STEM: Scaling Transformers with Embedding Modules

Available in BlaGPT

What’s new

• STEM (Scaling Transformers with Embedding Modules).

• Replaces FFN up-projection with a layer-local embedding lookup indexed by token.

How it works

• Keeps dense gating and down-projection but removes runtime routing (no MoE-style dispatch).

• Enables CPU offload with asynchronous prefetch of embedding parameters.

• Decouples model capacity from per-token FLOPs and cross-device communication.

• Trains stably despite extreme sparsity in accessed parameters.

• Learns embedding spaces with large angular spread for efficient knowledge storage.

• Supports interpretable knowledge editing/injection directly in the embedding space.

• Long contexts activate more distinct parameters, boosting long-context capacity.

Results

• ~3–4% accuracy gains at 350M and 1B scales.

• Strong improvements on ARC-Challenge, OpenBookQA, GSM8K, MMLU.

Why it matters

• Offers a practical path to scale parametric memory without linear compute growth.

• Improves interpretability and targeted knowledge editing.

• A compelling alternative to classic FFNs and MoE for larger, smarter models.

🧠 Demystifying the Slash Pattern in Attention: The Role of RoPE

What’s new

• Identifies and analyzes a Slash Pattern in transformer attention.

• Clarifies how Rotary Positional Encoding (RoPE) shapes attention structure.

How it works

• Uses a theoretical framework based on shallow transformers to isolate effects.

• Defines a Slash-Dominance Frequency Condition to characterize specific attention patterns.

• Designs architectures and training setups to highlight slash-dominant heads (SDHs).

• Separately evaluates short-range vs long-range offsets in attention scores.

• Measures SDHs across models like Qwen2.5, Llama 3, etc.

• Explores parameter compression while preserving useful slash patterns.

Results

• Provides detailed empirical maps of SDHs across different ∆ offsets and model families.

• Shows that preserving these heads can retain performance with fewer parameters.

Why it matters

• Deepens mechanistic understanding of how RoPE shapes transformer attention.

• Helps explain why some attention heads matter disproportionately for model quality.

📏 Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

What’s new

• DroPE (Dropping Positional Embeddings) to extend context length without finetuning from scratch.

• Challenges the standard wisdom that positional embeddings must be kept at inference.

How it works

• Key observations:

  1. PEs help convergence during pretraining but hurt generalization to unseen lengths.

  2. Over-reliance on PEs blocks extrapolation to longer contexts.

  3. After pretraining, PEs can be removed following a short recalibration phase.

• DroPE:

  • Removes PEs post-pretraining.

  • Runs a brief recalibration on a fraction of the original context length.

• This lets LMs generalize to much longer sequences zero-shot, while preserving original performance on shorter ones.

Results

• Outperforms traditional RoPE scaling methods in zero-shot long-context tasks.

• Strong gains across multiple models and data scales.

• Big improvements on long-context and needle-in-a-haystack benchmarks.

Why it matters

• Provides a cheap, post-hoc method for context extension.

🎥🔊 JoVA: Unified Multimodal Learning for Joint Video-Audio Generation

What’s new

• A unified framework for joint video–audio generation with human speech.

• Uses joint self-attention across modalities instead of complex fusion modules.

• Introduces a mouth-area-specific loss to improve lip–speech synchronization.

How it works

• Video and audio tokens are processed in a single transformer with joint self-attention.

• No extra alignment modules: cross-modal interaction happens directly in every layer.

• Mouth-area loss:

  • Detects facial keypoints.

  • Focuses loss on the mouth region to enforce better lip-sync.

• Two-stage training: first separate modality branches, then joint video–audio training.

• Uses temporally aligned RoPE to synchronize positional encodings across modalities.

Results

• State-of-the-art on lip-sync accuracy, speech quality, and overall audiovisual fidelity.

• Outperforms unified and audio-driven baselines on UniAvatar-Bench and Universe-Bench.

• Achieves lip-sync LSE-C of 6.64, beating prior work.

🧠📚 Fast-weight Product Key Memory

What’s new

• A dynamic, memory-augmented layer.

• Upgrades static Product Key Memory (PKM) into a context-responsive component.

How it works

• Performs local chunk-level gradient descent updates during both training and inference.

• Combines:

  • Large, sparse key–value storage of PKM.

  • Rapid adaptation of fast weights.

• Acts as episodic memory:

  • Memorizes new key–value pairs from recent inputs.

  • Retrieves them via Top-k key lookups.

• Uses a gating mechanism to blend FwPKM outputs with static slow weights.

• Supports iterative memorization by re-processing inputs multiple times.

Results

• Substantially reduces perplexity on long-context datasets vs. standard PKM.

• Generalizes to 128K tokens even when trained on 4K sequences.

• Improved retrieval accuracy, especially with iterative reading.

Why it matters

• Brings online, inference-time learning closer to practicality in LMs.

• Provides a flexible memory module that can quickly absorb new information.

⚙️📐 How to Set the Learning Rate for Large-Scale Pre-training?

1. Train small models (e.g., 500M, 1B, 2B, 3B) with various LRs
   - Use ~5-7 LR values per model size
   - Train to moderate token counts (e.g., 80-120B)

2. Apply Equation 4 to extrapolate loss to target data size
   - Fit: L(D) = L₀ + A·D^(-γ)
   
3. Apply Equation 2 to find optimal LR per (N, D) combination
   - Fit: L(η) = L_min + C·(log(η) - η_min)²
   
4. Fit your own version of Equation 6
   - Lr(N, D) = C_η · N^(-α) · D^(-β)
   
5. Extrapolate to target scale

What’s new

• Studies learning rate (LR) configuration for large-scale pretraining at industrial scale.

• Compares two paradigms: Fitting (scaling-law-based) vs. Transfer (proxy → target).

• Extends analysis across architectures, depths, weight decay, token horizons, and MoE.

How it works

• Fitting paradigm:

  • Empirically models optimal LR as a function of model size and training data.

  • Uses scaling laws to predict good LRs, reducing search complexity.

• Transfer paradigm:

  • Tunes hyperparameters on smaller proxy models.

  • Transfers to larger models via heuristic rules.

• Uses a Warmup–Stable–Decay (WSD) LR schedule as the common base.

• Tests global vs. module-specific LRs (including MoE) across many settings.

Results

• Fitting consistently yields better training quality than Transfer.

• No strong evidence that complex module-specific LR schemes significantly beat a well-chosen global LR.

Why it matters

• Offers concrete, data-backed guidance for LR tuning in very large pretraining runs.

⚙️📦 How to Set the Batch Size for Large-Scale Pre-training?

What’s new

• Revisits batch-size scaling under the Warmup–Stable–Decay (WSD) LR scheduler.

• Defines two key concepts: minimum threshold batch size and optimal batch size for data efficiency.

• Proposes a dynamic Batch Size Scheduler.

How it works

• Reconstructs batch-size theory specifically for WSD schedules.

• Derives:

  • Threshold constraint: smallest batch size that can reach a target loss.

  • Efficiency optimality: batch size that minimizes total tokens consumed.

• Produces a formula linking data consumption and optimization steps.

• Designs a dynamic schedule that changes batch size over training to optimize both speed and final performance.

Results

• Dynamic scheduling outperforms fixed batch sizes across many setups.

• Gains hold in both stable and decay phases of training.

• Improves both efficiency and final model quality.

Why it matters

• Gives practical recipes for teams running massive pretraining jobs.

🎥🔊 LTX-2: Efficient Joint Audio-Visual Foundation Model

What’s new

• LTX-2 is an open-source text-to-audio+video (T2AV) foundation model.

• Jointly generates synchronized video and audio from text prompts.

• Combines a 13B Video Diffusion Transformer with 3B audio stream.

• Uses modality-aware classifier-free guidance and progressive joint training.

How it works

• Asymmetric dual-stream transformer:

  • A 13B video stream.

  • A 3B audio stream.

• Uses separate VAEs for video and audio to compress raw signals into latents.

• Bidirectional cross-attention tightly aligns audio and video tokens in time.

• Text is encoded with a high-capacity, multilingual encoder (Gemma 3) for rich semantics and phonetics.

• Implements bimodal classifier-free guidance to control audiovisual alignment and content strength.

• Uses multi-scale, multi-tile inference to efficiently synthesize 1080p video with audio.

Results

• State-of-the-art open-source audiovisual quality and prompt adherence.

• Beats several proprietary systems in human preference tests (visual realism, audio fidelity, sync).

• Ranked 3rd for Image-to-Video and 4th for Text-to-Video tasks (as of Nov 6, 2025).

• Generates up to ~20 seconds of synced audio–video.

Kudos for reading 🙏 …

Enjoyed this issue? Share it with a friend.

Share

🤖 Model Releases

🎙️ nvidia/nemotron-speech-streaming-en-0.6b · Hugging Face - Unified English ASR model for low-latency streaming and high-throughput batch transcription with punctuation and configurable chunk sizes.

if liked && want_next_issue:
   subscribe();

Subscribe now

🎬 Lightricks/LTX-2 · Hugging Face - DiT-based audio–video foundation model with open weights that generates synchronized video and audio, designed to run locally on practical hardware.

🔊 GitHub - ekwek1/soprano: Soprano — Instant, Ultra-Realistic Text-to-Speech - Ultra-fast, open-source TTS that uses <1 GB VRAM and reaches ~2000× real-time, generating 10 hours of speech in under 20 seconds.

💻 Ex0bit/MiniMax-M2.1-PRISM · Hugging Face - 229B-parameter language model tuned for coding, multi-step reasoning, and multilingual tasks, using PRISM methodology to improve performance and refusal behavior.

🎧 GitHub - ysharma3501/LinaCodec - Neural audio tokenizer that compresses audio to 12.5 tokens/sec and decodes to 48kHz, boosting speed and quality for TTS and ASR pipelines.

🧮 LiquidAI/LFM2-2.6B-Exp · Hugging Face - Experimental 2.6B LLM checkpoint trained with RL on instructions, knowledge, and math, targeting agentic behavior and multi-turn conversations.

🎧 GitHub - ysharma3501/FlashSR - Fast audio super-resolution model that upsamples 16kHz audio to 48kHz at 200–400× real time, depending on quality settings.

🎥 KlingTeam/MemFlow · Hugging Face - Streaming video generation model with a dynamic memory bank that retrieves relevant past frames from text prompts to maintain long-context consistency and narrative coherence.

🎮 nvidia/NitroGen · Hugging Face - Vision-to-action model that turns video game footage into gamepad actions using large-scale imitation learning for controllable agents.

📄 Introducing Mistral OCR 3 | Mistral AI - Production-grade OCR model for extracting text and images from diverse document types—including forms, handwriting, and complex tables—with markdown output and API integration.

🧪 GSAI-ML/ReFusion · Hugging Face - Masked diffusion LLM that supports any-order generation and full KV cache reuse, bringing diffusion-style flexibility together with efficient parallel decoding.

📎 Papers

🧠 DAPO: An Open-Source LLM Reinforcement Learning System at Scale

What’s new

• Group Relative Policy Optimization (GRPO) as a simpler RL optimizer for LLMs.

• DAPO as an improved variant addressing stability and efficiency issues in vanilla GRPO.

• Importance Sampling and Truncated Importance Sampling (TIS) to keep training effectively on-policy.

How it works

• GRPO estimates advantages by sampling multiple completions per prompt and using a group-based baseline, avoiding a separate critic network.

• This reduces memory and compute relative to PPO while still supporting RL-style optimization.

• DAPO modifies GRPO with four key tricks:

  1. Clip Higher: Decouples lower and upper clipping bounds to prevent entropy collapse.

  2. Dynamic Sampling: Over-samples prompts and drops those with perfect accuracy to improve sample efficiency.

  3. Token-Level Loss: Aggregates loss at the token level to remove bias toward shorter responses.

  4. Overlong Reward Shaping: Masks truncated samples instead of punishing them, stabilizing training.

• TIS corrects discrepancies between rollout and policy-update engines, ensuring the updates remain on-policy even when using different systems for sampling and training.

Results

• DAPO improves accuracy on reasoning benchmarks, surpassing prior GRPO-based systems such as DeepSeek-R1.

• Shows faster convergence and better sample efficiency versus vanilla GRPO.

• Training curves indicate stable growth in response length, entropy, and reward, suggesting more robust RL behavior.

🧮 Value Residual Learning

What’s new

• ResFormer and SVFormer, two Transformer variants that modify how value states are handled.

• ResFormer enhances information flow with value residual connections.

• SVFormer shares a single value state across layers, cutting KV cache size by nearly half.

How it works

• ResFormer:

  • Adds value residual connections from the first layer’s value vectors to each subsequent layer’s value vectors before attention.

  • Both sets of value states share the same attention matrix, preserving fine-grained token information even in deep layers.

  • Mitigates over-smoothing by keeping early-layer token features accessible throughout the network.

• SVFormer:

  • Decouples value vectors from the attention computation so all layers can share a single value state.

  • Greatly reduces memory overhead for KV cache during inference, especially on long sequences.

• Empirical evidence suggests these architectures maintain or improve performance with fewer parameters and less training data than standard Transformers.

Results

• ResFormer reaches equivalent validation loss with _{16.1% fewer parameters and} 20.3% less training data versus vanilla Transformers.

• SVFormer significantly shrinks KV cache usage while keeping model quality intact.

• Both outperform prior variants like DenseFormer and NeuTRENO across multiple tasks.

🧱 mHC: Manifold-Constrained Hyper-Connections

What’s new

• Manifold-Constrained Hyper-Connections (mHC) as a new way to structure residual-like connections.

• Directly addresses issues with existing Hyper-Connections (HC), such as loss of identity mapping and training instability at scale.

How it works

• Projects the residual space of Hyper-Connections onto a well-chosen manifold, enforcing desirable topological properties.

• Restores the identity mapping behavior that makes standard residual connections so effective and stable.

• Adds infrastructure optimizations so the method remains efficient at scale, not just a theoretical improvement.

• Maintains the performance gains of HCs while improving robustness and scalability.

Results

• Large-scale experiments show mHC can train effectively where naive HCs become unstable.

• Delivers tangible performance improvements over baselines that either lack HCs or use unconstrained versions.

🗣️ What Does the Speaker Embedding Encode?

What’s new

• Comprehensive study comparing i-vector, d-vector, and s-vector speaker embeddings.

• A new i-s-vector framework that combines i-vectors and s-vectors via multi-task learning.

• Significant gains in text-dependent speaker verification.

How it works

• i-vector:

  • Models speaker and session variability in a low-dimensional subspace derived from a universal background model (UBM).

  • Strong for speaker discrimination but weak on temporal/sequential information.

• d-vector:

  • Learns speaker-discriminative features with a DNN; frame-level embeddings are averaged into a single utterance-level vector.

  • Balances multiple properties but discards sequence information through averaging.

• s-vector:

  • Uses LSTMs to treat speaker recognition as a sequence modeling problem.

  • Preserves temporal structure and is jointly trained to predict both speaker identity and text content.

• i-s-vector:

  • Concatenates i-vector and s-vector embeddings and trains them in a multi-task, end-to-end framework.

  • Encourages richer feature interactions and leverages strengths of both generative (i-vector) and sequence-based (s-vector) methods.

  • BLSTM variants further exploit bidirectional temporal context.

Results

• i-s-vector yields >50% relative EER reduction versus an i-vector baseline on content-mismatch trials.

• Improves performance across multiple tasks: speaker verification, speaker ID, and text classification.

• BLSTM-based models deliver additional gains by capturing longer temporal dependencies.

🎵 Aliasing-Free Neural Audio Synthesis

What’s new

• Pupu-Vocoder and Pupu-Codec, two anti-aliased neural audio modules.

• High-quality pre-trained checkpoints for singing voice, music, and general audio generation.

How it works

• Targets aliasing artifacts that commonly appear in neural vocoders/codecs, especially when using ConvTranspose layers.

• Applies oversampling and anti-derivative anti-aliasing (ADAA) to activation functions to suppress aliasing.

• Replaces ConvTranspose with a resampling layer built from zero-interlacing plus low-pass filtering:

  • Avoids tonal artifacts and eliminates aliased components in the upsampling process.

• Overall architecture: encoder → residual vector quantizer (RVQ) → decoder → discriminators.

• Pupu-Vocoder takes mel-spectrograms as input, focusing on high-fidelity waveform reconstruction from spectral features.

Results

• Pupu-Vocoder and Pupu-Codec outperform state-of-the-art systems on singing voice, music, and general audio quality metrics.

• Reach comparable performance to leading baselines on speech tasks while significantly reducing aliasing artifacts.

📐 Decoupling the “What” and “Where” With Polar Coordinate Positional Embeddings

What’s new

• Polar Coordinate Positional Embeddings (PoPE) as a drop-in alternative to RoPE.

• Explicit disentanglement of content (“what”) and position (“where”) in the attention mechanism.

How it works

• Analyzes how RoPE entangles content and positional information, which can confound tasks needing precise indexing by position or content.

• PoPE encodes positions in a polar coordinate system designed so that content similarity and positional relationships can be matched independently.

• Keeps the Transformer architecture largely intact but changes how positional information is injected into queries/keys.

• Enables attention to better reason about “same content, different positions” and “different content, same positions.”

Results

• PoPE beats RoPE in diagnostic tasks that require accurate indexing by either position or content alone.

• Improves autoregressive modeling across diverse domains: music, genomics, and natural language.

• Gains hold from 124M to 774M parameters and include stronger zero-shot length extrapolation than RoPE and YaRN.

🖼️ CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion

What’s new

• CASA (Cross-Attention via Self-Attention) for vision-language models (VLMs).

• Efficient fusion mechanism that supports both text-to-text and text-to-image attention.

• Substantial reduction in memory and compute versus traditional image token insertion.

How it works

• Integrates visual information via dedicated attention layers without inserting image tokens into the text stream.

• Text tokens self-attend to other text tokens within local attention windows, acting as an implicit gating mechanism.

• CASA layers operate in local windows defined by image occurrences, preserving relevant context while remaining efficient.

• Easily pluggable into existing architectures with only a modest parameter increase.

Results

• Outperforms conventional cross-attention models and closes much of the gap to full token-insertion methods.

• Strong performance on document and chart understanding benchmarks.

• Achieves similar quality to larger models with lower memory usage and faster inference.

📏 Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models

What’s new

• HSA-UltraLong, an 8B-parameter MoE model for ultra-long context.

• Hierarchical Sparse Attention (HSA) for efficient 16M-token context handling.

• Robust length generalization from standard training windows to truly massive contexts.

How it works

• HSA combines:

  • Sparsity: Selective activation retrieves relevant fragments instead of processing full context.

  • Random-Access Flexibility: Intrinsic retrieval mechanism for end-to-end optimized access.

  • Length Generalization: Retrieval ability learned on short contexts and transferred to long ones.

• Combines sliding-window attention with HSA; warm-up training uses a short (512-token) window to learn long-range dependencies.

• Chunk-wise attention: each token attends to past chunks separately and fuses them via retrieval scores.

• Uses No Positional Encoding (NoPE) for better extrapolation at extreme lengths.

Results

• Comparable to full-attention models on in-domain tasks despite sparse structure.

• 90% accuracy on in-context retrieval tasks with contexts up to 16M tokens.

• Successfully extrapolates from a 32K pre-training window to 16M tokens.

🎥 MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives

What’s new

• MemFlow, a memory mechanism for streaming video generation.

• Focus on maintaining content consistency over long video contexts.

How it works

• Maintains a dynamic memory bank updated before generating each video chunk.

• Retrieves relevant historical frames based on the current chunk’s text prompt.

• Activates only the most relevant memory tokens in attention, keeping costs bounded.

• Preserves narrative coherence across new events or scene switches.

• Compatible with any KV-cache-based streaming video generation model.

Results

• Strong long-context consistency in generated video narratives.

• Only ~7.9% slower than a memory-free baseline.

⚙️ Cautious Weight Decay

What’s new

• Cautious Weight Decay (CWD), a modification to standard weight decay.

• Optimizer-agnostic and hyperparameter-free enhancement for methods like AdamW, Lion, and Muon.

• Selective decay based on parameter–update alignment.

How it works

• Applies weight decay only when the parameter and its update have the same sign.

• Preserves the original loss landscape, avoiding extra implicit regularization from decoupled weight decay.

• Enables “sliding-mode” behavior: the optimizer can explore Pareto-optimal points on the stationary manifold.

• Theoretical Lyapunov analysis shows asymptotic stability and unbiased optimization with CWD.

Results

• Consistently lower validation loss and higher accuracy across language modeling and ImageNet-scale tasks.

• Demonstrated gains from millions to billions of parameters.

• Improves performance without additional tuning over standard decoupled weight decay.

🧪 SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations

What’s new

• SonicMoE, a memory-efficient training algorithm for Mixture-of-Experts (MoE) models.

• Co-designed MoE architecture and GPU kernels for better utilization.

• Tile-aware token rounding to reduce wasted compute under high sparsity.

How it works

• Cuts activation memory footprint by up to 45% for fine-grained MoEs.

• Custom GPU kernels overlap memory I/O with computation for higher throughput.

• Tile-aware token rounding routes tokens so each expert receives a multiple of the GEMM tile size.

• Efficient forward/backward without activation caching for router gradients; heavily optimized kernel-level operations.

Results

• 213B tokens/day training throughput on 64 H100s.

• 1.86× compute throughput over ScatterMoE for a fine-grained 7B MoE.

• Up to 1.16× additional speedup on kernels under high sparsity from token rounding.

🔁 From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

What’s new

• A principled path to adapt autoregressive (AR) models into block-diffusion models.

• Context-Causal Attention Mask to preserve AR inductive bias during adaptation.

• Gradual block growth for smoother transitions.

How it works

• Views AR models as a special case of block-diffusion within the same architecture family.

• Uses:

  • Context-Causal Attention Mask to retain AR structure while adding bidirectional refinement inside blocks.

  • Parallel training with an auxiliary AR objective for stability early in adaptation.

  • Gradually increasing block size to expand within-block reasoning.

• Block-diffusion generates text block-by-block, enabling parallel token updates and richer local reasoning.

• During training, the model denoises masked tokens in the last block while keeping earlier context fixed.

Results

• NBDiff-7B reaches state-of-the-art performance after adaptation.

• Handles long contexts up to 32K and strong reasoning benchmarks.

• Outperforms methods like LLaDA, Dream, and SDAR across multiple evaluations.

🎬 End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

What’s new

• Resampling Forcing, a teacher-free framework for autoregressive video diffusion.

• Targets exposure bias and train–test mismatch without relying on strong teacher models.

How it works

• Simulates inference-time errors during training via self-resampling of degraded history frames.

• Next-frame predictions are conditioned on corrupted histories, improving robustness.

• Uses a sparse causal mask to enforce temporal causality while allowing parallel training over frames.

• History routing retrieves the most relevant history frames with controlled attention complexity.

• Keeps a clean prediction objective while hardening the model against perturbations.

Results

• Matches or exceeds state-of-the-art distilled models.

• Better temporal consistency in long video generations.

• Outperforms extrapolated baselines on long-horizon video tasks.

⚡ Fast and Accurate Causal Parallel Decoding using Jacobi Forcing

What’s new

• Jacobi Forcing, a training technique that turns AR models into native causal parallel decoders.

• Addresses the mismatch between AR and diffusion-style decoding for speed and quality.

How it works

• Keeps the causal AR backbone but trains the model to handle noisy future blocks along Jacobi decoding trajectories.

• Noise-conditioned training:

  • Splits responses into blocks with assigned noise levels.

  • Selects Jacobi intermediate states matching each block’s noise level.

  • Packs blocks to cycle through noise regimes efficiently.

• Multiblock Jacobi decoding:

  • Maintains several blocks “in flight,” with one real-active and others pseudo-active.

  • Pseudo-active blocks update without committing KV cache until validated.

  • Rejection recycling caches promising n-grams to reuse computation.

Results

• Up to 4× speedup on coding and math tasks versus standard AR decoding.

• Better accuracy–speed trade-off than diffusion-based decoders.

• Maintains near-AR quality at much higher throughput.

🌀 SoFlow: Solution Flow Models for One-Step Generative Modeling

What’s new

• Solution Flow Models (SoFlow) for one-step generative modeling.

• Replaces multi-step denoising with a direct solution function.

How it works

• Learns the solution of the velocity ODE for one-step generation instead of simulating trajectories.

• Uses:

  • Flow Matching loss to estimate velocity fields for classifier-free guidance.

  • Solution consistency loss to avoid Jacobian–vector products, reducing compute.

• Trains a parameterized neural net from scratch to approximate the ground-truth solution function.

• Combined objective (Flow Matching + consistency) supports accurate one-step sampling.

Results

• Better FID-50K than MeanFlow on ImageNet 256×256 across model sizes.

• For example: FID-50K of 4.85 (SoFlow-B/2) and 3.20 (SoFlow-L/2), outperforming baselines under matched conditions.

🧊 ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding

What’s new

• ReFusion, a masked diffusion LLM (MDM) with parallel autoregressive decoding.

• Full KV cache reuse plus global generation flexibility.

How it works

• Elevates parallel decoding from token level to slot level (fixed-length contiguous sub-sequences).

• Two-step decoding:

  1. Diffusion-based planning to identify weakly dependent slots from a fully masked sequence.

  2. Autoregressive infilling of selected slots in parallel.

• Training mirrors inference:

  • Hybrid objective combining AR loss on clean slots with denoising loss on masked slots.

  • Random masking and permutation of slots to improve robustness.

• Uses a certainty score on predictive distributions to choose which slots to decode in parallel.

Results

• 34% performance gain and >18× throughput speedup versus prior MDMs.

• Challenges and often surpasses strong autoregressive baselines across seven benchmarks.

Kudos for reading 🙏 … This one was heavy and if you read it to the end extra Kudos to you.

Enjoyed this issue? Share it with a friend.

Share

🌐 browser-use/bu-30b-a3b-preview — BU-30B-A3B-Preview is a 30B MoE browser agent model with enhanced DOM understanding and visual reasoning, designed to run on a single GPU with native browser-use OSS library integration.

if liked && want_next_issue:
   subscribe();

Subscribe now

🎭 meituan-longcat/LongCat-Video-Avatar — LongCat-Video-Avatar is a unified model for audio-driven character animation, supporting audio-text-to-video and video continuation with single and multi-stream audio inputs.

🧊 microsoft/TRELLIS.2-4B — TRELLIS.2-4B is a 4B-parameter 3D generative model for image-to-3D generation using sparse voxel structures and flow-matching transformers.

🎵 SAM Audio — Meta’s SAM Audio uses text prompts to separate target sounds from audio or audiovisual sources, enabling isolation of specific audio elements through span prompting and interactive selection.

💬 allenai/Bolmo-7B — Bolmo is a byte-level autoregressive language model from AI2, available in 1B and 7B parameter scales for research and educational use.

🎬 thu-ml/TurboDiffusion — TurboDiffusion accelerates video diffusion models 100–205× on a single RTX 5090 with minimal quality loss.

🖼️ apple/Sharp — SHARP enables photorealistic view synthesis from a single image using 3D Gaussian representation with real-time rendering on standard GPUs.

🗣️ Chatterbox Turbo — A 350M-parameter TTS model designed for low-latency voice applications, featuring a distilled audio diffusion decoder and paralinguistic tag support.

🗣️ FunAudioLLM/Fun-CosyVoice3-0.5B — Fun-CosyVoice 3.0 is an LLM-based TTS system for zero-shot multilingual speech synthesis with pronunciation inpainting and bi-streaming capabilities.

💬 nvidia/gpt-oss-120b-Eagle3 — NVIDIA’s 120B-parameter Eagle model uses mixture-of-experts architecture optimized for high-concurrency inference.

🧮 NousResearch/nomos-1 — Nomos 1 specializes in mathematical problem-solving and proof-writing in natural language, developed in collaboration with Hillclimb AI.

🗣️ zai-org/GLM-TTS — GLM-TTS is a two-stage TTS system using LLMs for speech token generation and Flow Matching for waveform synthesis, supporting zero-shot voice cloning.

🗣️ openbmb/VoxCPM1.5 — VoxCPM1.5 is an end-to-end diffusion autoregressive TTS system for continuous speech generation with high-quality voice cloning.

🖼️ ByteDance-Seed/Adversarial-Flow-Models — Pretrained ImageNet-256px models unifying adversarial and flow-based generative approaches.

👁️ zai-org/GLM-4.6V Collection — GLM-4.6V is a vision-language model collection for image-text tasks, available in 10B and 108B parameter configurations.

🎬 vita-video-gen/svi-model — Stable-Video-Infinity (SVI) generates arbitrary-length videos with controllable storylines and high temporal consistency.

🖼️ meituan-longcat/LongCat-Image — LongCat-Image is a 6B-parameter open-source bilingual foundation model for image generation with multilingual text rendering and photorealism.

🎤 Ultravox v0.7 — Ultravox v0.7 is a 355B-parameter speech understanding model built on GLM 4.6 that processes speech directly in noisy environments without separate transcription.

📎 Papers

🎵 SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

What’s new

• A neural speech codec that disentangles semantic and acoustic modeling into two independent streams with dual-stream quantization.

How it works

• Dual-Stream Design: Semantic stream uses a pre-trained speech tokenizer for linguistic content; acoustic stream employs a neural audio codec for timbre and emotional attributes.

• Tokenization: Semantic tokens from self-supervised models; acoustic tokens from codecs trained with reconstruction objectives.

• Training: Streams optimized separately with speaker feature supervision for enhanced timbre modeling.

• Decoding: ConvNeXt-based prenet fuses both streams before codec decoder reconstructs the waveform.

Results

• Achieves state-of-the-art speech reconstruction and semantic representation.

• Outperforms existing codecs in intelligibility (WER) and naturalness (UTMOS).

• Demonstrates effective disentanglement for superior downstream task performance.

💬 Bolmo: Byteifying the Next Generation of Language Models

What’s new

• A byte-level language model that operates directly on raw bytes rather than tokenized text.

How it works

• Processes language at the byte level using advanced algorithms for efficient data handling.

• Focuses on reducing computational resources while maintaining performance.

• Designed for research applications with 1B and 7B parameter variants.

Results

• Results pending full evaluation.

🖼️ Distribution Matching VAE

What’s new

• A framework that explicitly aligns the encoder’s latent distribution with arbitrary reference distributions beyond conventional Gaussian priors.

How it works

• Uses Distribution Matching Distillation (DMD) to train a diffusion model on the reference distribution, learning its score function.

• VAE posterior trained to match reference distribution’s score for flexible latent space shaping.

• Joint training with a fake score model captures evolving latent distribution.

• Composite objective combines reconstruction loss, fake score model loss, and distribution matching loss.

Results

• Achieves state-of-the-art gFID of 3.22 on ImageNet with only 64 training epochs.

• SSL-derived features provide best balance of reconstruction quality and generative performance.

🖼️ Bidirectional Normalizing Flow (BiFlow)

What’s new

• Removing the need for exact analytic inverses in normalizing flow models and enabling flexible architectures and loss functions.

How it works

• Forward Process: Transforms data to noise using any tractable NF model (e.g., improved TARFlow).

• Reverse Process: Learns to approximate inverse mapping from noise to data via separate model.

• Two-Stage Training: Train forward model with Maximum Likelihood Estimation; train reverse model with fixed forward weights.

• Hidden Alignment: Uses intermediate forward states for supervision.

• 1-NFE Generation: Produces samples in a single forward pass.

Results

• Achieves state-of-the-art FID of 2.39 on ImageNet 256×256.

• Up to two orders of magnitude faster sampling than improved TARFlow.

🎤 Vevo2: Unified Framework for Speech and Singing Voice Generation

What’s new

• A unified controllable framework for speech and singing voice generation with novel prosody and content-style tokenizers.

How it works

• Two-Stage Architecture:

  • AR Content-Style Modeling: Takes text and prosodic source, generates content-style tokens.

  • Flow-Matching Acoustic Modeling: Converts tokens to Mel spectrograms guided by timbre reference.

• Prosody Tokenizer: Captures melody from speech, singing, and instrumentals at 6.25 Hz without expert annotations.

• Content-Style Tokenizer: Encodes linguistic content, melody, and style at 12.5 Hz with timbre disentanglement.

• Multi-objective Post-training: Aligns intelligibility and prosody similarity.

Results

• Superior zero-shot TTS and singing voice synthesis performance.

• Outperforms baselines in intelligibility, naturalness, and subjective evaluations.

⚙️ Soft Adaptive Policy Optimization (SAPO)

What’s new

• Replacing hard clipping in policy optimization with smooth, temperature-controlled gating.

How it works

• Adaptively attenuates off-policy updates while preserving useful learning signals.

• Maintains sequence-level coherence with soft gating forming a continuous trust region.

• Selectively down-weights highly off-policy tokens instead of suppressing all sequence gradients.

• Smooth temperature-controlled scaling replaces hard token-level clipping.

Results

• Improved training stability and higher Pass@1 on mathematical reasoning benchmarks.

• Consistent gains across diverse tasks and model sizes on Qwen3-VL series.

🧠 Gated Attention for LLMs

What’s new

• Introduces gating mechanisms in softmax attention with comprehensive comparison of 30 variants across 15B MoE and 1.7B dense models.

How it works

• Applies head-specific sigmoid gate after Scaled Dot-Product Attention.

• Introduces non-linearity in low-rank mapping within softmax attention.

• Uses query-dependent sparse gating scores to modulate SDPA output.

Results

• Simple modification consistently improves performance and training stability.

• Tolerates larger learning rates with improved scaling properties.

• Sparse gating mitigates “attention sink” and enhances long-context extrapolation.

💬 CALM: Continuous Autoregressive Language Models

What’s new

• Shifting from discrete token prediction to continuous vector prediction with a likelihood-free training framework.

How it works

• Autoencoder: Compresses K tokens into single continuous vector with >99.9% reconstruction accuracy.

• Next-Vector Prediction: Predicts continuous vectors instead of tokens, reducing generative steps by factor K.

• Energy Transformer Head: Single-step generation avoiding iterative sampling bottlenecks.

• Likelihood-Free Training: Uses energy loss as training objective.

• BrierLM Metric: Novel evaluation metric based on Brier score for likelihood-free contexts.

Results

• Superior performance-compute trade-offs vs. traditional Transformers.

• Matches or exceeds discrete baselines at lower computational costs.

🎭 Live Avatar: Real-time Audio-Driven Avatar Generation

What’s new

• Introduces Live Avatar, a framework for real-time, infinite-length audio-driven avatar generation using a 14B-parameter diffusion model.

How it works

• Timestep-forcing Pipeline Parallelism (TPP): Distributes denoising steps across multiple GPUs, breaking autoregressive bottleneck for low-latency streaming.

• Rolling Sink Frame Mechanism (RSFM): Maintains sequence fidelity by dynamically recalibrating appearance using cached reference images.

• Self-Forcing Distribution Matching Distillation: Enables streamable adaptation without sacrificing visual quality.

Results

• Achieves 20 FPS end-to-end generation on 5 H800 GPUs.

• First framework enabling practical, real-time, high-fidelity avatar generation at this scale.

🖼️ Adversarial Flow Models

What’s new

• Unifies adversarial and flow generative models, supporting single-step and multi-step generation with improved training stability.

How it works

• Deterministic Mapping: Generator learns deterministic noise-to-data mapping, stabilizing adversarial training.

• Adversarial Objective: Trained using adversarial objective enabling single-step training without intermediate timesteps.

• Flow Matching: Minimizes squared Wasserstein-2 distance between prior and data distributions.

• Gradient Normalization: Improves optimization across model sizes.

• Classifier Guidance: Enhances generation quality using learned classifier gradients.

Results

• B/2 model approaches consistency-based XL/2 models.

• XL/2 achieves new best FID of 2.38 on ImageNet.

• Superior few-step generation compared to existing models.

🧑‍💻 Open Source

🗣️ sarulab-speech/Sidon — Training code and dataset cleansing tools with Sidon.

🗣️ CorentinJ/TorchStream — A library for making PyTorch audio models streamable.

🗣️ microsoft/VibeVoice — Microsoft’s frontier open-source TTS framework for expressive, long-form multi-speaker audio (up to 90 mins, 4 speakers), with a real-time 0.5B variant achieving ~300ms latency.

🤖 block/goose — Block’s extensible AI agent platform (24K+ stars), a founding project of the Agentic AI Foundation alongside Anthropic’s MCP and OpenAI’s AGENTS.md.

⚙️ activepieces/activepieces — AI-powered workflow automation with 400+ MCP server integrations and a no-code builder for enterprise automation.

📚 tmgthb/Autonomous-Agents — Continuously updated curated list of autonomous AI agents research, covering multi-agent systems and agentic pipelines.

Thanks for reading … Enjoyed this issue? Share it with a friend.

Share

• Mistral Large 3 - Model Collection - A general-purpose multimodal model built on a granular Mixture-of-Experts architecture.

if liked && want_next_issue:   
   subscribe();

Subscribe now

• apple/CLaRa-7B-Instruct - An instruction-tuned unified RAG model with built-in semantic document compression for fast, memory-efficient QA.

• black-forest-labs/FLUX.2-dev - 32 billion parameter rectified flow transformer designed for generating, editing, and combining images based on text instructions, with open weights for research and artistic applications.

• apple/starflow - An open-source transformer autoregressive flow model for high-quality image and video generation, combining AR models with normalizing flows.

• deepseek-ai/DeepSeek-V3.2 - A compute-efficient reasoning model using sparse attention and scalable RL for improved agentic task synthesis.

• nvidia/Orchestrator-8B - An 8B-parameter orchestration LLM that coordinates expert models and tools for multi-turn agent workflows.

• PrimeIntellect/INTELLECT-3 - A 106B MoE reasoning model trained with SFT + large-scale RL, competitive on math, coding, and reasoning benchmarks.

• Tongyi-MAI/Z-Image-Turbo - A 6B image generator with variants for photorealism, bilingual text rendering, and image editing—optimized for consumer GPUs.

• FastVideo/CausalWan2.2-I2V-A14B-Preview-Diffusers - A preview image-to-video model supporting 8-step inference, tuned for diverse GPUs and Apple Silicon.

📎 Papers

Glance: Accelerating Diffusion Models with 1 Sample

• What’s new: Proposes “Glance,” a method to accelerate diffusion models with a single sample at inference.

• How it works: Reformulates sampling to extract maximal signal in one pass, reducing steps without retraining.

• Results: Reports large speedups while retaining competitive perceptual quality.

• Why it matters: Brings diffusion closer to real-time applications across image, video, and audio.

Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout

• What’s new: A training-free framework for infinite-horizon, action-controllable video generation.

• How it works

  • Block-Relativistic RoPE reframes temporal encoding as a moving reference frame.

  • KV Flush renews cache by keeping only two frames for instant control without re-encoding.

  • RoPE Cut enables controlled temporal discontinuities for cinematic transitions.
    Reparameterizes temporal RoPE and KV caching at inference time for dynamic rollouts.

• Results: Consistently outperforms prior methods on VBench; improved subject/background consistency, motion smoothness, and dynamics over long videos.

• Why it matters: Enables scalable long-form video without retraining and unlocks creative editing/transition control.

FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks

• What’s new: A low-bitrate speech codec (0.16–0.65 kbps) using focal modulation with a single binary codebook.

• How it works: VQ-VAE-style compressor–quantizer–decompressor with WavLM features; focal modulation replaces self-attention to model fine-to-coarse speech cues; binary spherical quantization for efficient codebook use; two-stage training improves resynthesis.

• Results: Strong resynthesis and voice conversion; better intelligibility and speaker similarity than SOTA, especially in noise; lowest dWER across multilingual tasks.

• Why it matters: Cuts bandwidth/storage for speech applications while maintaining quality—useful for low-resource settings and real-time comms (with responsible-use considerations for voice cloning risks).

Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs

• What’s new: Integrates a frame-level continuous-token diffusion head into an autoregressive MLLM for variable-length, speaker-referenced TTS.

• How it works: Dual-head design - LM head predicts speech token boundaries; diffusion head generates continuous acoustic embeddings decoded to waveforms. Masked training reduces exposure bias; two-stage training stabilizes optimization (joint training, then LM freeze).

• Results: SOTA autoregressive performance on LibriSpeech(PC) test-clean with WER 1.95%, speaker similarity 0.54, UTMOS 4.00; two-stage training cuts WER by 46% vs. one-stage.

• Why it matters: Avoids discrete-token quantization bottlenecks, improving naturalness and enabling unified multimodal speech capabilities.

Hyperparameter Optimization - Book

• What’s new: Comprehensive review of HPO methods with a focus on Bayesian, multi-fidelity, population-based, and gradient-based techniques.

• How it works

  • Bayesian optimization with surrogate models for guided exploration.

  • Multi-fidelity strategies to accelerate evaluations with cheaper approximations.

  • Population-based methods evolve candidate solutions.

  • Gradient-based approaches leverage hypergradients for high-dimensional tuning.

• Results: Improved tuning efficiency and performance; multi-fidelity and population-based methods cut evaluation time; gradient-based methods converge faster in high dimensions.

• Why it matters: Better HPO reduces compute/time costs and boosts model performance across domains.

LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

• What’s new: Combines next-token prediction with a JEPA objective to enhance LLM abstraction and robustness.

• How it works: Dual objectives: standard generative loss + JEPA predicting different views (e.g., text/code) of shared knowledge; custom attention mask prevents view leakage; loss balancing via λ and stochastic JEPA-loss dropout.

• Results: Significant gains across NL-RX, GSM8K, Spider, RottenTomatoes; stronger fine-tuning and pretraining performance with reduced overfitting (notably in LoRA).

• Why it matters: Bridges generative learning with predictive embeddings, improving representation quality for reasoning-heavy tasks.

Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale

• What’s new: A multi-reward GRPO framework addressing prosody instability, speaker drift, and naturalness in single-codebook TTS LLMs.

• How it works: Multi-objective rewards: length penalty, entropy regularization, LLM-annotated prosody alignment, and pause prediction via an external reasoning LLM; scaled across data (1K–1M) and model sizes (1B–8B).

• Results: GRPO-optimized LLaSA achieves best CER on test-zh and competitive multi-lingual results; higher speaker similarity and MOS.

• Why it matters: Enables more natural, robust TTS—useful for assistants, accessibility, and media.

Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy

• What’s new: A unified framework for synchronized audio–video generation, establishing new SOTA alignment.

• How it works: Cross-Task Synergy Training (joint + auxiliary tasks) combats correspondence drift; Global–Local Decoupled Interaction with RoPE-aligned frame-wise attention for timing and a global style module for consistency; SyncCFG amplifies alignment signals at inference with negative anchors.

• Results: Best-in-class sync metrics (Sync-C 5.61, Sync-D 7.53) with coherent, realistic outputs across scenarios.

• Why it matters: Improves content creation, digital avatars, and immersive media with reliable A/V alignment.

Terminal Velocity Matching

• What’s new: One/few-step generative modeling by matching terminal (not initial) flow velocities, with explicit upper bounds on Wasserstein distance.

• How it works: Generalizes flow matching between diffusion steps; regularizes terminal-time behavior; stabilizes diffusion transformers with architectural tweaks and fused attention kernels for efficient JVPs; scaled parameterization handles varying guidance strengths.

• Results: State-of-the-art single-step ImageNet generation with FID ≈ 3.29; outperforms MeanFlow and diffusion baselines with fewer NFEs; stable training without curricula.

• Why it matters: Brings high-quality few-step generation within reach, cutting inference cost while retaining fidelity.

STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows

• What’s new: A normalizing flow-based video generator supporting T2V, I2V, and V2V with end-to-end likelihood training.

• How it works: Spatiotemporal latent space with global–local architecture (global latents for long-range context; local blocks for per-frame detail); flow-score matching via a lightweight causal denoiser; video-aware Jacobi iteration enables parallel updates without breaking causality.

• Results: Strong visual fidelity and temporal consistency; outperforms diffusion-based baselines on multiple benchmarks.

• Why it matters: Positions normalizing flows as a compelling alternative to diffusion for scalable, coherent video generation.

🧑‍💻 Open Source

• opendatalab/LabelLLM - Open-source data annotation platform powered by LLMs.

• karpathy/llm-council - A “council” of LLMs collaborate to tackle hard questions via self-critique and debate.

• cased/kit - Toolkit for AI devtools context engineering: codebase mapping, symbol extraction, and advanced code search.

Thanks for reading… Enjoyed this issue? Share it with a friend.

Share