AI lab
Tokenizer
Split text into tokens with a toy vocabulary and see why code and rare words cost more.
Common English words become one token each, while digits, punctuation-heavy code and characters the vocabulary lacks break into many small pieces. Real tokenizers split differently, but you pay for the same pattern.
Learn the concept
The lesson this lab comes from, with the explanation, a practice task and a quiz.
More labs
All labs- AINext-token samplingAdjust temperature, switch between greedy and sampled decoding, and watch each candidate token’s odds shift.
- AIEmbedding spaceDrop a query into a 2-D embedding space and rank its nearest neighbours by cosine similarity.
- Linuxchmod calculatorToggle read, write and execute per class and read the octal mode and ls -l string together.