← All tutorials
Skill Zuno · skillzunos.com · AI
Transformers tutorial
Learn Transformers on Skill Zuno. This Transformers tutorial covers New, Easy, Medium, Hard, and Diff levels so you can study practical Transformers problems on skillzunos.com.
Open Transformers on Skill ZunoAlso see the Transformers interview questions.
The transformer is the neural network architecture behind modern language models, introduced in the 2017 paper 'Attention Is All You Need'. It replaces recurrence with self-attention, which lets every token attend to every other token and allows highly parallel training. Interviews cover self-attention, multi-head attention, positional encoding, encoder, decoder and encoder-decoder variants, and the cost of long contexts.
Transformers topics covered
- Self-attention — Queries, keys and values, and scaled dot-product attention.
- Multi-head attention — Attending to different kinds of relationship in parallel.
- Positional information — Sinusoidal encodings, learned positions and rotary embeddings.
- The transformer block — Attention, feed-forward layers, residual connections and layer normalisation.
- Encoder, decoder and encoder-decoder models — BERT, GPT and T5, and the tasks each suits.
- Efficiency and long context — Quadratic attention cost, KV caching and FlashAttention.
What you can learn
- Transformers tutorial for beginners and experienced developers
- Level-based practice: New, Easy, Medium, Hard, and Diff
- AI topics alongside Transformers on skillzunos.com