How do Transformers compare to LSTMs for time-series forecasting?

All answers to this question.

The primary advantage of Transformers is the "Self-Attention" mechanism, which allows the model to look at the entire sequence simultaneously, capturing long-range dependencies that LSTMs often miss due to the sequential nature of their hidden state. However, Transformers don't have an inherent sense of order, so you must use "Positional Encoding" to give the model temporal context. For very long sequences, Transformers are faster because they allow for parallelization during training. LSTMs are still very competitive for smaller datasets or where the temporal patterns are relatively short-term and simple to capture.

   Answered 2024-05-22 by Nancy Carter

That parallelization is a huge plus, but what about the "Informer" or "Temporal Fusion Transformer" variants? Are you planning to use the vanilla Transformer, or are you looking at these specific modifications that were designed to handle the multi-horizon forecasting challenges specific to financial data?

   Answered 2024-05-24 by Richard Scott

• Richard, we are currently testing the Temporal Fusion Transformer (TFT). It’s performing better than our old LSTM because it handles static covariates (like stock symbols) and time-varying inputs much more elegantly. The attention weights also give us some level of interpretability, which our stakeholders really appreciate when we explain our model's predictions.

     Commented 2024-05-26 by Lisa Thompson

Don't ignore the computational cost. Transformers have a quadratic complexity $O(n^2)$ relative to sequence length, which can be a bottleneck if you are trying to process high-frequency tick data in real-time.

   Answered 2024-05-28 by Kevin Parker

• Great point, Kevin. We actually had to implement a "Sparse Attention" mechanism to keep our inference times under the required millisecond threshold for our high-frequency trading platform.

     Commented 2024-05-30 by Nancy Carter