factual
Code is more predictable than language
Language model loss measured as bits per byte is lower for code than for natural language, meaning code contains many highly predictable tokens, and this predictability is magnified when not just autocompleting but predicting edits.
factualpending
Speaker
Aman SangerEvidence Quote
“the bits per byte, which is a kind of character normalize loss for code is lower than language, which means in general there are a lot of tokens in code that are super predictable”
Created: 6/13/2026, 3:31:02 AM
Your notes
Loading notes...