← Jargon

Jargonmachine learning· 52 claims· 33 episodes

tokens

The basic units of text (words or word-fragments) that AI language models process and learn from.

As defined in Brendan Foody


Other definitions
  • Units of text that a language model processes, which can be individual words or parts of words (like punctuation or word fragments); the text shown to a model is converted into tokens

    The Future of Artificial Intelligence

Where it comes up · 33

and 25 more


Most important claims · 10 of 52Search all claims for “tokens” →