youtube
In this video, I explain the softmax function and the concept of temperature in the context of language models and neural networks. I start by revisiting the Markov chain coding challenge, then break down how softmax converts raw neural network outputs (logits) into probabilities, and how temperature can be used to adjust those probabilities making outputs more random (higher temperature) or more deterministic (lower temperature). I also briefly cover top-K and top-P sampling. Everything is demonstrated with p5.js sketches and a transformers.js language model running in the browser. Code:
🚀 Watch this video ad-free on Nebula
p5.js Web Editor Sketches:
🕹️ Softmax and Temperature Visualization:
🕹️ Markov Chain with Temperature:
🕹️ Markov Chain with Softmax and Temperature:
🕹️ LLM with Temperature:
🎥 Previous:
🎥 Next:
References:
📓 Softmax function on Wikipedia:
📓 Markov Chains explained visually:
📓 Web Trigrams by Chris Harrison:
📓 Euler's number on Wikipedia:
📓 p5.js exp() reference:
💻 Transformers.js Documentation:
🤗 SmolLM2 1.7B on Hugging Face:
📕 Nature of Code: Genetic Algorithms:
Videos:
🚂
🚂
🚂
🚂
Related Coding Challenges:
🚂
🚂
Timestamps:
0:00:00 Hello!
0:02:50 Markov chains as language models
0:07:48 From logits to probabilities
0:10:47 The softmax formula
0:14:18 Implementing softmax in p5.js
0:15:59 Weighted selection (sampling)
0:20:20 Temperature
0:23:02 Applying softmax and temperature to the Markov chain
0:28:01 Temperature with a Transf
|
2026/08/03
youtube
