# Liquid Neural Networks

We invented liquid neural networks, a class of brain-inspired systems that can stay adaptable and robust to changes even after training [[R. Hasani, PhD Thesis](https://repositum.tuwien.at/bitstream/20.500.12708/1068/2/Hasani%20Ramin%20-%202020%20-%20Interpretable%20recurrent%20neural%20networks%20in%20continuous-time...pdf)] [[Lechner et al. Nature MI, 2020](https://www.nature.com/articles/s42256-020-00237-3)] [[pdf](https://publik.tuwien.ac.at/files/publik_292280.pdf)] (2016-2020). We then analytically and experimentally showed they are universal approximators [[Hasani et al. AAAI, 2021](https://ojs.aaai.org/index.php/AAAI/article/view/16936/16743)], expressive continuous-time machine learning systems for sequential data [[Hasani et al. AAAI, 2021](https://ojs.aaai.org/index.php/AAAI/article/view/16936/16743)] [[Hasani et al. Nature MI, 2022](https://www.nature.com/articles/s42256-022-00556-7)], parameter efficient in learning new skills [[Lechner et al. Nature MI, 2020](https://www.nature.com/articles/s42256-020-00237-3)] [[pdf](https://publik.tuwien.ac.at/files/publik_292280.pdf)], causal and interpretable [[Vorbach et al. NeurIPS, 2021](https://proceedings.neurips.cc/paper/2021/file/67ba02d73c54f0b83c05507b7fb7267f-Paper.pdf)] [[Chahine et al. Science Robotics 2023](https://www.science.org/doi/10.1126/scirobotics.adc8892)] [[pdf](http://cap.csail.mit.edu/sites/default/files/research-pdfs/Robust%20flight%20navigation%20out%20of%20distribution%20with%20liquid%20neural%20networks.pdf)], and when linearized they can efficiently model very long-term dependencies in sequential data [[Hasani et al. ICLR 2023](https://arxiv.org/abs/2209.12951)].

# Nonlinear Neural Differential Equation Models

In addition, we developed classes of nonlinear neural differential equation sequence models [[Massaroli et al. NeurIPS 2021](https://arxiv.org/pdf/2002.08071)] and generalized them to graphs [[Poli et al. DLGMA 2020](https://arxiv.org/pdf/1911.07532)]. We scaled and optimized continuous-time models using hybrid numerical methods [[Poli et al. NeurIPS 2020](https://arxiv.org/pdf/2007.09601)], parallel-in-time schemes [[Massaroli et al. NeurIPS 2020](https://proceedings.neurips.cc/paper/2021/file/89b9c689a57b82e59074c6ba09aa394d-Paper.pdf)], and achieved state-of-the-art in control and forecasting tasks [[Massaroli et al. SIAM Journal](https://arxiv.org/pdf/2101.05537)] [[Poli et al. NeurIPS 2021](https://proceedings.neurips.cc/paper_files/paper/2021/file/5291822d0636dc429e80e953c58b6a76-Paper.pdf)] [[Massaroli et al. IEEE Control Systems Letters](https://arxiv.org/pdf/2106.03780)]. The team released one of the most comprehensive open-source libraries for neural differential equations [[Poli et al. 2021 TorchDyn](https://physical-reasoning.github.io/assets/pdf/papers/03.pdf)], used today in various applications for generative modeling with diffusion, and prediction.

# Parallel Scan-based Architecture

We proposed the first efficient parallel scan-based linear state space architecture [[Smith et al. ICLR 2023](https://arxiv.org/abs/2208.04933)], and state-of-the-art time series state-space models based on rational functions [[Parnichkun et al. ICML 2024](https://arxiv.org/abs/2405.06147)]. We also introduced the first-time generative state space architectures for time series [[Zhou et al. ICML 2023]](https://proceedings.mlr.press/v202/zhou23i/zhou23i.pdf), and state space architectures for videos [[Smith et al. NeurIPS 2024](https://arxiv.org/abs/2310.19694)].

# Neural Operators Framework

We proposed a new framework for neural operators [[Poli et al. NeurIPS 2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/342339109d7d756ef7bb30cf672aec02-Paper-Conference.pdf)], outperforming approaches such as Fourier Neural Operators in solving differential equations and prediction tasks.

# Deep Signal Processing Architectures

Our team has co-invented deep signal processing architectures such as Hyena [[Poli et al. ICML 2023](https://arxiv.org/pdf/2302.10866)] [[Massaroli et al. NeurIPS 2023](https://proceedings.neurips.cc/paper_files/paper/2023/file/371355cd42caaf83412c3fbef4688979-Paper-Conference.pdf)], HyenaDNA [[Nguyen et al. NeurIPS 2023](https://arxiv.org/pdf/2306.15794)], and StripedHyena that efficiently scale to long context. Evo [[Nguyen et al. 2024](https://www.biorxiv.org/content/10.1101/2024.02.27.582234v1)], based on StripedHyena, is a DNA foundation model that generalizes across DNA, RNA, and proteins and is capable of generative design of new CRISPR systems.

# Language Models and Beyond

We were the first to scale language models based on both deep signal processing and state space layers [[link](https://huggingface.co/togethercomputer/StripedHyena-Nous-7B)], and have performed the most extensive scaling laws analysis on beyond-transformer architectures to date [[Poli et al. ICML 2024](https://arxiv.org/abs/2403.17844)], with new model variants that outperform existing open-source alternatives.

# Open-source Contributions

The team is behind many of the best open-source LLM fine-tunes, and merges [[Maxime Lebonne, link](https://mlabonne.github.io/blog/)].

# Research Contributions

Last but not least, our team’s research has contributed to pioneering work in graph neural networks and geometric deep learning-based models [[Lim et al. ICLR 2024](https://arxiv.org/abs/2312.04501)], defining new measures for interpretability in neural networks [[Wang et al. CoRL 2023](https://arxiv.org/pdf/2210.06650)], and the state-of-the-art dataset distillation algorithms [[Loo et al. ICML 2023](https://proceedings.mlr.press/v202/loo23a.html)].