Machine Learns - 75
Kimi K3 runs on a laptop. DeepSeek released a model for agents. MiniMax's open-weighted video-sound generation model MiniMax-H3. Simple BM25 beats complicated agentic file-systems by 20 points.
🤖 Model Releases
🤖 DeepSeek-V4-Flash-0731
A sparse MoE model for agentic coding with a DSpark speculative-decoding module. DeepSWE 54.4, against 58.0 for Opus-4.8 and 46.2 for GLM-5.2. DeepSeek’s own V4-Pro preview scores 12.8 there and the Flash preview 7.3, low enough that the harness was plainly not a target for either, so read this as a post-training result and not as small beating large. MIT licence, weights on Hugging Face, natively int8 rather than post-hoc quantized. I use it daily. Take it for long agent loops where token cost matters.
🎬 MiniMax-H3
A 33B dense transformer that generates video and stereo audio in one stack, with no modality-specific structure anywhere in the attention or FFN layers. H3-AudioVAE takes 32 kHz stereo down to 40 latent frames per second per channel, slow enough that a sequence model can carry sound alongside video without a separate audio tower. For vision, H3-VisualVAE downsamples space sixteenfold and time fourfold into 24 causal latent channels, which are patchified down to 32x spatial compression before entering the transformer. For conditioning, a full Qwen3-VL-32B encoder feeds its fiftieth-layer hidden states directly as input. About 13B of the 33B parameters sit in AdaLN modulation branches whose outputs can be precomputed and cached, so 40% of the model never loads for inference-only deployment. Custom community licence, base model weights out.
🗣️ Audio8-TTS-Preview-0.6b
A 0.6B text-to-speech model that gets the words right better than anything in its own comparison table and the voice wrong worse than all of them. Transcribe its output back and 1.506% of words come out wrong, ahead of Fish S2 Pro and Higgs Audio v2 at nearly eight times the size. Measure how close the synthesized voice sits to the reference clip and it scores 63.2, last in that table, against 75.3 for VoxCPM2. Comes with Apache-2.0. Use it where the words must land on a small budget, not for voice cloning.
🛠️ WASTE
A dependency-free C engine that streams a mixture-of-experts model’s active experts off NVMe instead of holding the whole thing in memory, which is normally what decides whether a machine can run a model at all. Kimi K3’s published 1.42 TB of weights become a 982 GB container that opens in 29.06 GB of RAM and decodes at 0.45 to 0.62 tok/s on a 64 GB M5 Pro MacBook. Apache-2.0, source on GitHub. Try it if you want a frontier model on your own disk.
# Skip the expensive reasoning layer
machine_learns.subscribe()
Papers
🧠 Kimi K3, Open Frontier Intelligence
What’s new
An open-weight 2.78T-parameter MoE with 104.2B active and a 1M-token context, in which 69 of 93 attention layers are linear Kimi Delta Attention and only 24 are Gated MLA.
How it works
Position comes entirely from KDA’s gating decay. No explicit positional embedding anywhere, so the model reaches 1M tokens with no RoPE rescaling.
Attention Residuals give each layer a learnable pseudo-query attending over the RMSNorm’d outputs of all preceding layers, so a deep layer reads an early layer directly instead of through one residual state accumulated over depth.
SiTU-GLU replaces SwiGLU, soft-capping both branches of the product with a scaled tanh, beta 4 on the gate and 25 on the up branch.
LatentMoE runs 896 routed experts, 16 active plus 2 shared. Quantile Balancing reads each expert bias off the router-score quantile matching its target load, via a Top-(k+1) pass rather than a fixed-step sign update.
Context grows 8K to 64K in pretraining, 256K to 1M in cooldown.
Results
Strongest at using information it is handed. Best long-context retrieval score in its own table, 74.7 on AA-LCR, and 91.2 on BrowseComp.
Weakest at open-ended research, 43.5 on HLE-Full without tools.
Coding runs near-frontier at a fraction of the price, 4.0 points behind Claude Fable 5 for 38% of the cost.
🗣️ Stable AR speech generation with low-frame-rate continuous tokens
What’s new
ByteDance Seed’s Locodec tokenizer emits 8.5 Hz, 768-dimensional continuous tokens on a fixed-norm sphere, with no SSL tokens, no pretrained text LM and no post-training stage.
How it works
Fixing the token norm removes the radial degree of freedom, so the AR model never predicts a magnitude and the decoder only has to survive angular error.
A learnable row-orthogonal projection maps the 768-d sphere onto a low-dimensional core sphere and lifts back, preserving angles, aligned by a bidirectional cosine commitment loss.
Postfix dimension dropout trains with only a prefix of coordinates available. Under a fixed energy budget that turns availability into energy, and log per-dimension energy decays roughly linearly with index. No target profile is prescribed.
The generator, MP-ELD, splits classifier-free guidance into separate self-consistency and alignment-consistency residual paths on a schedule over bridge time.
Results
Encode and decode real audio and every configuration sounds equally good, STOI 0.98, ViSQOL 4.63.
Generate from them and they diverge wildly, long-form WER spanning 4.64% to 34.77% with no warning from reconstruction.
Postfix dimension dropout closes most of it, cutting WER at core 32 from 20.80% to 9.73%.
🔎 BM25 Wins at Scale
What’s new
A 28-tier nested corpus ladder from 1,144 to 511,959 documents, 1.25x per rung, with 500 questions and adversarial distractors held constant, to find where retrieval methods cross over.
How it works
Every tier is a strict superset of the one below, so a question’s evidence never moves and the only variable is how much noise surrounds it.
All methods get the same reader, Qwen3.6-27B, and the same 80-call budget, so no agent buys accuracy with iterations.
The retrieval-primitive control swaps only the discovery mechanism inside one agent loop, holding harness, reader and 150 questions fixed.
Build cost is counted in tokens, not wall clock, so indexing and querying land in one currency.
Results
On a small corpus the agent reading files wins, 77.4 against 74.7. Past roughly 10M tokens it reverses and widens.
At 511,959 documents BM25 scores 50.5 against 30.7 and 29.9, same reader and budget.
Change only how the agent finds candidates and it jumps 36.9 to 69.4.
My 2 cents
I’m using simple BM5 in my harness too. I tried many different conceptually satisfying techniques but benchmarks proved BM5 is good to go.
🧠 Scaling Mixture-of-Experts Diffusion Language Models
What’s new
The first scaling-law study for MoE diffusion language models, finding that optimal compute splits toward data over model size, inverting the model-dominated frontier of dense diffusion and AR MoE.
How it works
Three controlled sweeps cover optimization, compute allocation, and MoE architecture, from 158M to 3.6B parameters under budgets of 10^17 to 3x10^20 FLOPs.
Optimal batch size grows steeper with compute than in AR, and optimal learning rate decays faster, because masked denoising supervises only sampled positions, so nominal batch size overstates effective supervision.
The MoE recipe that wins at every scale is the same: keep activation sparse, use 8 to 16 experts per layer, and dedicate one third of activated capacity to a shared expert. AR designs use 25% or none.
Results
A 30B-A3B trained on 23.5T tokens, 65% of Qwen3’s budget, approaches it on reasoning and coding, 2.22 behind on OlympiadBench.
Also shipped
🗣️ SwanTale
Thanks for reading… Until next one 👋