Description:

Since the famous paper "Attention is All You Need" came out in 2016, the exposure neural network models have received has been unprecedented, be it computer vision, speech, or natural language. But among all the applications, one area or a broad spectrum of areas remained quite untouched and linear models have religiously outperformed them.

[Image Source: NVIDIA]

Time-series forecasting (be it long-horizon or not) has been the play area for linear models for a long period, where uni-variate estimations outperform multivariate. The primary reason might be that time-dependent models are different from data-dependent models. Famous M5 competitions for real-world time-series data have famously been dominated by linear models. Slowly but surely, that trend has changed. Newer methods have started to outperform linear models in the M5 competition, and a lot of promising research has been published on the same.

1. Time Series Transformer from HuggingFace
2. Decoder-only foundation model for time-series
3. Multi-horizon time-series forecasting

In this talk, I will focus on the existing nature of time-series forecasting, why transformers can be useful and where they might fail, and building a transformer model for time-series forecasting.

Talk Outline

1. Introduction
2. Advanced understanding of transformers?
   • Multi-head attention at the Encoder/Decoder level
3. Transformer for time-series
   • Current state-of-the-art
   • How transformer can improve forecasting?
   • Quadratic Complexity Issue
   • The new time-series foundation model
4. Improving transformers for time-series
   • Positional encoding
   • Attention module
5. Feasibility Study for transformer models for time-series
   • Latency
   • Comparison with existing state-of-the-art methods
   • Challenges
6. Exploring an usecase

Prerequisites:

Understanding of time-series (time-ordered) forecasting and transformers.

Content URLs:

Presentation [Draft]: Transformers are coming for time-series: Exploring transformers for time-series forecasting

GitHub: saradindusengupta/transformers_for_time-series

Speaker Info:

I am working at Nunam, an energy analytics startup based in Bangalore, India, where my primary area of work is building health and life-cycle forecasting of Li-ion batteries in EV and energy storage. I have over 4 years of professional experience in building ML systems from the ground up after finishing my master's from IIITM, Kerala. I have spoken at both physical and virtual conferences where my primary area of focus has been on Computer Vision, MLOps, model interpretability and model compression and quantization. More details about me and my previous talk can be found here: website

Previous Talks

1. "Managing data quality issues in ML production, especially for time-series" - Link at Google Developer Group Community Day, 2022
2. "Things I learned while running neural networks on microcontroller" - Link at PyData Global 2022
3. "Bessel's Correction: Effects of (n-1) as the denominator in Standard deviation" - Link at PyData Global 2022
4. "Interpretable ML in production" Slides at Google Developer Group Community Day, 2023

Speaker Links:

Speaker Links

LinkedIn GitHub Website Twitter