Understanding Kronos: How a Foundation Model Reads the Language of Financial Markets
Generic Time Series Foundation Models fail on financial data. Kronos fixes this by treating K-line data as a discrete language. A stage-by-stage breakdown of its architecture, from BSQ tokenization to detokenization.
Understanding Kronos: How a Foundation Model Reads the Language of Financial Markets
Financial time series are notoriously hard to model. Low signal-to-noise ratios, non-stationarity, and complex dependencies between price and volume make most general-purpose Time Series Foundation Models (TSFMs) fall short. The Kronos paper shows that generic TSFMs often underperform even simple non-pre-trained models on financial tasks.
So what makes Kronos different? It treats K-line data (Open, High, Low, Close, Volume, Amount) as a discrete language and builds a dedicated foundation model for it.
1. Why General TSFMs Fail on Financial Data
A single 5-minute K-line looks like this:
[2500.00, 2520.00, 2490.00, 2510.00, 1000000, 2510000000]
The core problem:
- Low signal-to-noise — many movements are pure random noise
- Non-stationary — statistical properties change over time
- High-order dependencies between OHLCVA columns
- Most TSFMs trained on less than 1% financial data — wrong inductive biases baked in from training
Generic models fail to generalise across volatility forecasting, synthetic data generation, or simple price prediction. Kronos is built specifically to solve this.
2. The Kronos Two-Stage Framework
Kronos follows the universal LLM pipeline — input, mid, head, output — but adapts every stage to K-line data.
3. Stage 1: K-line Tokenization
Raw OHLCVA values are normalised using z-score and clipped to [-5, 5]. A transformer encoder converts each K-line into a 256-dimensional latent vector. Binary Spherical Quantization (BSQ) then produces a 20-bit code split into two subtokens:
- Coarse subtoken — 10 bits (e.g. decimal 891) — captures trend and regime
- Fine subtoken — 10 bits (e.g. decimal 370) — captures precision and microstructure
Why two subtokens? A single token would need a vocabulary of 2^20 (over one million entries). Two tokens of 1024 each makes embedding tables manageable while preserving full precision.
Tokenization pipeline:
Raw [Open, High, Low, Close, Volume, Amount] → Z-score normalisation, clipped to [-5, 5] → Transformer encoder → latent vector (256-d) → BSQ (20-bit code) Coarse subtoken (10 bits → decimal 891) Fine subtoken (10 bits → decimal 370) → HierarchicalEmbedding emb_s1 (1024 → d_model=512) emb_s2 (1024 → 512) fusion_proj (1024 → 512) → Add temporal embedding (minute, hour, weekday, day, month) → RoPE applied inside each attention block → Output: 512-d vector ready for the decoder-only transformer
4. Stage 2: Transformer Layers (Feature Extraction)
Shape remains unchanged throughout all layers: [batch, seq_len, d_model].
Kronos-small configuration:
| Setting | Value |
|---|---|
| Transformer layers | 8 |
| d_model | 512 |
| Attention heads | 8 (head_dim = 64) |
| FFN width | 512 to 1024 (SwiGLU) to 512 |
| Normalisation | RMSNorm before attention and FFN |
| Connections | Residual after each sub-layer |
What the model learns: hierarchical dependencies between coarse tokens (trend and regime) and fine tokens (intra-bar precision and microstructure).
Each layer runs:
Input hidden state → RMSNorm → Self-Attention with RoPE positional encoding → Residual add → RMSNorm → SwiGLU Feed-Forward Network (512 → 1024 → 512) → Residual add → Output hidden state (shape unchanged)
5. Stage 3: Dual LM Head
After the final transformer layer, a DualHead produces separate logits for coarse and fine subtokens.
Head pipeline:
Final RMSNorm → DualHead proj_s1 (512 → 1024) → coarse logits proj_s2 (512 → 1024) → fine logits → Softmax + argmax (temperature + top-p sampling) → Next token IDs: [coarse_id=891, fine_id=370]
Loss function: Hierarchical cross-entropy — L_coarse + L_fine. Forces the coarse head to learn rough structure first, while the fine head learns residual precision independently.
6. Stage 4: Detokenization
Detokenization pipeline:
Token IDs [coarse_id=891, fine_id=370] → Combine 10-bit coarse + 10-bit fine → 20-bit BSQ code → BSQ dequantization → latent vector (256-d) → Tokenizer decoder (3 transformer layers) → continuous values → Inverse z-score normalisation → Predicted OHLCVA: [2515.00, 2530.00, 2505.00, 2520.00, 1050000, 2646000000]
7. Key Architectural Choices
| Component | Kronos Implementation | Why It Helps |
|---|---|---|
| Tokenization | BSQ with coarse + fine (n=2) | Turns 2^20 vocab into 2x1024 — manageable tables |
| Loss | Hierarchical (L_coarse + L_fine) | Coarse learns structure, fine learns residuals |
| Positional encoding | RoPE inside each attention block | No separate encoder, better extrapolation |
| Attention | MultiHeadAttentionWithRoPE | GQA in larger variants (kv_heads=4) |
| Activation | SiLU via SwiGLU | Better gradient flow than ReLU |
| Normalisation | RMSNorm throughout | Faster and more stable than LayerNorm |
8. Complete Forward Pass
Input (single 5-min K-line): [2500, 2520, 2490, 2510, 1000000, 2510000000]
Universal LLM stages mapped to Kronos:
Universal Stage → Kronos Implementation Stage 1 Input Tokenization: BSQ (coarse + fine subtokens) Embedding: HierarchicalEmbedding + temporal embedding Positional: RoPE applied inside attention Stage 2 Mid (N layers, shape unchanged throughout) RMSNorm → Self-Attention with RoPE → Residual RMSNorm → SwiGLU FFN → Residual Stage 3 Head Final RMSNorm DualHead (two linear projections) → softmax → argmax Stage 4 Output Detokenization decoder → inverse normalisation → continuous OHLCVA
Forecast output (next 5-min period): [2515, 2530, 2505, 2520, 1050000, 2646000000]
9. Why This Matters for Practitioners
| Task | Kronos Result | vs Baseline |
|---|---|---|
| Price forecasting (RankIC) | Beats all leading TSFMs | +93% improvement |
| Synthetic K-line generation | Higher fidelity output | +22% better |
| Volatility forecasting (MAE) | Lower error | 9% lower MAE |
| Model size range | 4M to 102M parameters | Open-source on HuggingFace |
Loading any Kronos variant:
from model import Kronos, KronosTokenizer
tokenizer = KronosTokenizer.from_pretrained("NeoQuasar/Kronos-Tokenizer-base") model = Kronos.from_pretrained("NeoQuasar/Kronos-small") print(model)
The architecture follows the same 4-stage blueprint as GPT-4, but every component is tailored for financial K-lines. That is the secret behind its success on tasks where every other TSFM fails.
10. The Key Insight
Most ML practitioners treat financial forecasting as a regression problem. Kronos reframes it as a next-token prediction problem — the same objective that made GPT-4 work — applied to price bars instead of words.
The result: a model that understands the grammar of markets, not just the statistics.
References
- Kronos paper — arXiv 2025, shiyu-coder et al.
- GitHub — shiyu-coder/Kronos (open source)
- HuggingFace — NeoQuasar/Kronos-small, Kronos-base, Kronos-large
This article summarises a deep-dive analysis of the Kronos paper, its code, and its place in the universal LLM stage blueprint.
quizforml.com — Learn. Build. Fail. Learn Again.