ML Speciality: Transformers - Part 1

 It begins with RNNS and LSTMs 

They introduced a feedback loop fr propagating information forward. 

This is useful for modelling sequential things. 

Useful for modelling sequential things 

When doing language translation, we have encoder and decoder architecture  Below is the architecture on this

Below is explanation of diagram, which is use case for translating the English Sentence to Spanish 

English Input:

The process begins with an English sentence, such as "Hello, world."

Embedding Layer:

The words in the English sentence are converted into numerical representations called word embeddings. Each word is mapped to a vector that captures its semantic meaning.

This layer turns the text into a format the RNN can process.

RNN Layers (Recurrent Neural Network):

The embedded words are fed into one or more RNN layers.

Each RNN layer consists of RNN cells.

RNN Cell:

Each cell takes the current word embedding and the previous hidden state as input.

It processes this information and produces a new hidden state and an output.

The hidden state carries information from previous time steps, allowing the RNN to remember context.

The hidden state is passed along to the next RNN cell, and the output is passed to the next layer. This is why it is called a recurrent neural network.

Dense Layer (Fully Connected Layer):

The output from the final RNN layer is passed through a dense layer.

This layer transforms the RNN's output into a probability distribution over the Spanish vocabulary.

The output of the dense layer is a probability distribution that shows the likelihood of each spanish word being the correct translation.

Spanish Output:

The word with the highest probability is selected as the translated word.

The process is repeated until a complete Spanish sentence is generated, such as "Hola, mundo."

Key Concepts:

Word Embeddings: Numerical representations of words that capture their semantic meaning.

Hidden State: A vector that carries information from previous time steps, allowing the RNN to remember context.

RNN Cell: The basic building block of an RNN, which processes the current input and the previous hidden state.

Dense Layer: A fully connected layer that transforms the RNN's output into a probability distribution.

Important Notes:

This diagram represents a basic RNN architecture. More advanced architectures, such as LSTMs (Long Short-Term Memory) or GRUs (Gated Recurrent Units), are often used in practice to improve performance.

Attention mechanisms are also commonly used in modern translation systems to allow the model to focus on the most relevant parts of the input sentence.

Sequence to sequence models, are also very common. They use an encoder and a decoder. The encoder encodes the english sentence, and the decoder decodes the spanish sentence.