Kronos: A Foundation Model for the Language of Financial Markets¶

1. Introduction¶

Financial markets is a critical and challenging application area for TSFMs, given their inherent data richness, high-frequency observations, and complex, non-stationary temporal dynamics. At the core of this domain are K-line sequences, multivariate time series derived from candle-stick charts that record Open, High, Low, and Close prices, along with trading Volume and Amount(Turnover) over fixed intervals(OHLCVA). These sequences constitute a highly compact, information-dense "language" through which market participants interpret prices movements, volatility regimes, liquidity shifts, and colletive sentment.

2. Motivation¶

Despite the promise of TSFMs. Old application to finance faces significant hurdles.

1. Unique statistical properties of K-line data
   • Low signal-to-noise ratio, strong non-stationarity, and complex interdependencies among OHLCVA features.
   • These properties are poorly aligned with the inductive biases of general-purpose TSFMs.
2. Underserved financial domain in pre-training
   • Most TSFMs are trained on broad, multi-domain corpora where financial data accounts for only a tiny fraction.
   • As a result, financial-specific dynamics are overlooked or averaged out.
3. Task gaps
   • Critical finance-specific tasks such as volatility prediction, synthetic data generation, and risk management remain largely unaddressed by existing TSFMs.

Empirical evidence shows that general-purpose TSFMs often underperform even non-pre-trained, specialized models (like iTransformer) on financial tasks. This reveals a gap: the field lacks a foundation model built specifically for financial time series.

3. Preliminary¶

Let \(D\)-dimensional vector \(x_t \in \mathbb{R}^D\) denote the K-line observation at discrete time \(t\), comprising \(D\) key financial indicators. In this work, the dimension was fixed \(D=6\) to represent OHLCVA attributes. Given a historical sequence \(x_{1:T}=(x_1,x_2,...,x_T)\), our objective is to predict the following \(H\) observations \(\hat{x}_{T+1:T+H}=(\hat{x}_{T+1}, \hat{x}_{T+2},...,\hat{x}_{T+H})\).

Rather than operating on raw continuous inputs, Kronos first quantizes each multivariate observation \(x_t\) into a discrete token \(b_t\) via a learnable codebook \(\mathcal{C}\). Consequently, the original sequence \(x_{1:T}=(x_1,x_2,...,x_T)\) is mapped to \(b_{1:T}=(b_1,...,b_T)\). The forecasting task then reduces to an autoregressive token-sequence modeling problem:

4. Methodology¶

Kronos implements this via a two-phase framework: (1) K-line Tokenization and (2) Autoregressive Pre-training

4.1 K-line Tokenization¶

The tokenization is achieved using a Transformer-based autoencoder composed of an encoder \(E_{\text{enc}}\), a quantizer \(Q\), and a decoder \(E_{\text{dec}}\).

Kronos adapt Binary Spherical Quantization(BSQ) for this task. BSQ quantizes a continuous latent vector \(\xi_t\) into a \(k\)-bit binary code \(b_t \in \{-1,1\}^k\) by projecting it onto a set of learnable hyperplanes.

4.1.1 Introduction to BSQ¶

Binary Spherical Quantization (BSQ) optimizes over an implicit codebook \(\textbf{C}_{BSQ}=\{-\frac{1}{\sqrt{L}}, \frac{1}{\sqrt{L}}\}^L\), a hypercube projected onto a unit sphere. Each corner \(\textbf{c}_k \in \textbf{C}_{BSQ}\) of a hypercube corresponds to a unique token \(k\). The quantizer works as follows: it projects some high-dimensional latent embedding \(\textbf{z}\) to a lower-dimensional unit hypersphere \(\textbf{u}\), applies binary quantization per axis \(\hat{\textbf{u}}=sign(\textbf{u})\) and back-projects to the quantized vector in the original latent space \(\hat{\textbf{x}}\). Specifically, we start with an encoded visual input \(\textbf{z}=\mathcal{E}(\textbf{x}) \in \mathbb{R}^d\). We first linearly project the latent embedding to \(L\) dimensions \(\textbf{v} = Linear(\textbf{z}) \in \mathbb{R}^L\), where \(L \ll d\). Next, we obtain project \(\textbf{v}\) onto the unit sphere \(\textbf{u}=\frac{\textbf{v}}{|\textbf{v}|}\) and perform binary quantization to each dimension of \(u\) independently \(\hat{\textbf{u}}=\frac{1}{\sqrt{L}} sign(\textbf{\textbf{u}})\), where \(sign(x)\) is the sign function. To keep outputs on the unit sphere, we map \(sign(0) \rightarrow 1\). We use a Straight-Through Estimator (STE) make the operator differentiable, \(sign_{STE}(x) = sg(sign(x)−x) + x\), where \(sg(·)\) denotes the stop-gradient operation. Finally, we back-project the quantized \(\hat{\textbf{u}}\) to the \(d\)-dimensional space \(\hat{\textbf{z}} = Linear(\hat{\textbf{u}}) \in \mathbb{R}^d\).

4.1.2 Effectiveness of BSQ¶

The effectiveness of BSQ tokenizer can be analyzed from two key perspectivs: its inherent noise suppression and its ability to create a structured, discrete state space suitable for sequence modeling.

Noise Suppression And Stability¶

BSQ's projection of embeddings onto a unit sphere prior to binarization guarantees that the expected distortion is strictly upper-bounded:

This bound tightens as the codebook dimension \(L\) increases. In contrast, simple methods like sign-based quantization without normalization(e.g., LFQ) lack such a guarantee, leaving them vulnerable to arbitarily large errors from outlier inputs.

Learning in a Compact and Discrete State Space¶

High-frequency financial data exists in a high-dimensional, continuous state space. This tokenizer maps these in a finite, discrete vocabulary of tokens. This discretization serves as a powerful form of regularization with two main benefits.

Inproved Sample Efficiency and Generalization: Instead of learning a complex function over a continuous space, a downstream model like a Transformer learns to predict transitions and patterns among a finite set of abstract states(tokens). This simplifies the learning tasks and also allows the model to learn robust patterns from fewer examples, which is particularly critical for modeling rare market phenomena.

Reduced Overfitting: The quantization process inherently discards fine-grained, potentially noisy variations within each quantization cells. This prevents the model from fitting to spurious artifacts in the tarining data.

Hyperspherical geometry for tail sensitivity¶

In financial contexts, market returns and price changes often exhibit heavy tails(or fat tails). The heavy-tail distribution of price changes is one of the key sources of trading profits in quantitative investment and cannot be ignored.

Unlike standard vector-quantization on the Euclidean sphere, BSQ's binary encoding preserves angular information very efficiently, making it more sensitive to fat-tail data that manifest as sharp directional changes in feature space. This aligns well how microstructure events often appear as abrupt shifts in the "direction" of the joint price-volume vector.

4.1.3 Subtoken Factorization¶

While a large number of bits \(k\) (e.g., \(k=20\)) is desirable for capturing rich financial patterns, it results in an exponentially large vocabulary of size \(2^k\). To mitigate this, we use the subtoken factorization and factorize the \(k\)-bit code into \(n\) subspaces. Motivated by the trade-off between parameter savings and latency costs, we set \(n=2\). We partition the code into a coarse subtoken \(b_t^c\) and a fine subtoken \(b_t^f\) of equal bit length. The resulting code \(b_t\) is a concatenation of these two subtokens: \(b_t=[b_t^c,b_t^f]\).

We train the tokenizer with a composite objective that combines a hierachical reconstruction loss and a commitment loss for BSQ:

where \(\lambda\) is a balancing hyperparameter. The components are defined as:

• \(\mathcal{L}_{\text{coarse}}=\mathbb{E}[\|\textbf{x}-E_{\text{dec}}(\textbf{b}^c)\|^2]\), which trains the coarse subtoken \(\textbf{b}^c\) to form a low-fidelity reconstruction.
• \(\mathcal{L}_{\text{fine}}=\mathbb{E}[\|\textbf{x}-E_{\text{dec}}(\textbf{b})\|^2]\), which evaluates the high-fidelity reconstruction using the complete token \(\textbf{b}\).
• \(\mathcal{L}_{\text{quant}}\) is the quantization loss from BSQ that regularizes the learning process. It penalizes the L2 distance between continuous latent vector \(\xi\) and their binary codes \(\textbf{b}\), aligning the encoder's outputs with the learned codebook to ensure stable training.

4.2 Hierarchical Autoregressive Modeling¶

The primary objective is to estimate the joint distribution over the token sequence \(\textbf{b}=\{b_1,...,b_r\}\). A simplified form of Equation 1 can be derived as:

where \(\textbf{b}_{<t}\) denotes all preceding tokens up to time \(t-1\).

Given the hierarchical token design, in which each token is structured as \(b_t=[b_t^c,b_t^f]\), we further decompose the conditional probability using the chain rule to explicitly capture the inherent coarse-to-fine dependency:

At time \(i\), the subtokens \(b_i^c\) and \(b_i^f\) are independently projected into vector representations using two distinct embedding layers, resulting in representations \(e_c(b_i^c)\) and \(e_f(b_i^f)\), respectively. These embeddings are then concatenated and linearly projected to produce a fused input vector:

where \([\cdot ; \cdot]\) denotes concatenation, and \(W_{fuse}\) is a learnable weight matrix responsible for projecting the combined representaion into the model's latent space.

Coarse Subtoken Prediction¶

The history vector \(\textbf{h}_t\) is projected by a linear head \(W_c\) to produce logits for the first subtoken's distribution:

Fine Subtoken Prediction¶

During the trainiing, we use the model's own prediction from the previous step, \(\hat{b}_t^c\), which is sampled from the predicted distribution \(p(b_t^c|\textbf{b}_{<t})\), rather than using the ground-truth subtoken(i.e., teacher-forcing). We use a cross-attention mechanism where the embedding of \(\hat{b}_t^c\) acts as the query, and the history \(\textbf{h}_t\) provides the key and value. The result is projected by the second head \(W_f\):

The overall training objective \(\mathcal{L}_{ar}\) is the negative log-likelihood of the data, summed over both prediction steps:

where \(\mathcal{D}\) represents the data distribution.

5. Results and Conclusion¶

评论