Glossary

Token

The fundamental unit of text that AI models process. A token might be a complete word, a word fragment, a punctuation mark, or a code symbol. Roughly, 1 token equals about 4 characters or 0.75 English words. (Ch. 2, 9)

Learn More

Related Terms