Mastering diverse domains through world models
arxiv.org
arxiv.org
Danijar has DreamerV2 and robot-dog application of Dreamer on his github, for those interested in implementation.
First topic: AutoEncoder
An autoencoder is a type of neural network architecture that is typically used for dimensionality reduction or feature learning. It is composed of two main parts: an encoder, which maps the input data to a lower-dimensional representation, and a decoder, which maps the lower-dimensional representation back to the original dimension. The goal of the autoencoder is to learn a compressed representation of the input data that captures the most important features, while discarding the less important ones. This is achieved by training the network to reconstruct the input data as accurately as possible, typically using a mean squared error or cross-entropy loss function.
Ok, so autoencoder means program that does: K = compress(state) state' = decompress(K)
You teach it by minimising: state - state', the goal is for it to learn K that's useful knowledge about data.
Next to solve: "What is Recurrent State-Space Model in ML?"
ChatGPT: A Recurrent State-Space Model (RSSM) is a type of statistical model that is often used for time series data analysis. It is a combination of a Recurrent Neural Network (RNN) and a State-Space Model (SSM).
A SSM is a mathematical model that describes the evolution of a system over time in terms of a set of state variables. The state variables are typically considered hidden variables, meaning that they cannot be directly observed, but their values can be inferred from observations of the system. A SSM is often used to model dynamic systems, such as physical or biological systems.
A RNN is a type of neural network architecture that is particularly well-suited for processing sequential data, such as time series or sequences of words. The key feature of an RNN is that it contains a set of recurrent connections, which allow information to flow through the network across multiple time steps.
An RSSM combines the strengths of RNN and SSM. The RNN is responsible for handling sequential, temporal dependencies in the data, and the SSM is responsible for inferring the underlying state variables of the system. These state variables are often considered as the internal representation of the data, which is compressed form of data, the state-space can be considered as low-dimensional encoding of the data.
An RSSM can be used for a wide range of tasks, such as forecasting, prediction, and control of dynamic systems, sequential decision making, and generative modelling of time series data.
> Can you explain how each of those work in layman terms and give pseudo code examples?
....
Auto-encoders, Transformers and Recurrent models don't pop up much in RL except for advanced topics and cutting-edge research.
[1] https://www.youtube.com/watch?v=o75ybZ-6Uu8 (Yannic Kilcher's overview)
[2] https://www.youtube.com/watch?v=_e3BKzK6xD0 (Presentation by lead author)
[3] https://ai.googleblog.com/2021/02/mastering-atari-with-discr...