Jekyll2026-10-06T13:37:32-07:00https://gangliao.me/feed.xmlGang Liao builds the systems that make ML work at scale. Source-backed deep dives on LLMs, GPUs, and AI infrastructure, in English and Chinese.Gang Liao2026-10-05T00:00:00-07:002026-10-05T00:00:00-07:00https://gangliao.me/blog/2026/deepseek-v4-deep-diveGang Liao

At a million tokens of context, the model’s weights stop being the expensive part. The bill is the key-value (KV) cache, the per-token memory that every attention layer keeps so later tokens can look back, plus the work each new token does to read it. DeepSeek-V3.2 already cut the reading cost: its DeepSeek Sparse Attention (DSA) lets each query read only a small, selected set of past tokens. But it still cached an entry for every token in every layer.

2026-10-04T00:00:00-07:002026-10-04T00:00:00-07:00https://gangliao.me/blog/2026/glm-5-3-deep-diveGang Liao

Z.ai created the Hugging Face repos for two models on the same day, August 25, 2026: GLM-5.3 and GLM-5.3-Flash. On paper they look like siblings. Both use a 154,880-token vocabulary and a 1,048,576-position context window. Both pick the 2,048 most relevant past tokens with a small scorer of the same shape, the lightning indexer (32 heads of 128 dimensions). Both are mixture-of-experts (MoE) models that send each token to 8 experts chosen by sigmoid gating (each expert’s score is squashed to 0–1 on its own, rather than through a softmax across all experts).

2026-09-29T00:00:00-07:002026-09-29T00:00:00-07:00https://gangliao.me/blog/2026/miles-v0-1-deep-diveGang Liao

The most dangerous failure in production reinforcement learning (RL) is often not a crashed job. It is a job that keeps running while it optimizes a trajectory the system never actually generated.

2026-04-01T00:00:00-07:002026-04-01T00:00:00-07:00https://gangliao.me/blog/2026/amd-gpu-architecture-deep-diveGang Liao

AMD’s MI350 series, built on the CDNA 4 architecture, marks a significant leap in datacenter GPU compute. This post breaks down its architecture, programming model, hardware features, and what makes it distinct — both from its predecessor MI300X and from NVIDIA’s competing Blackwell lineup.