Multi-headed attention involves splitting the query, key, and value vectors into multiple sub-vectors before applying self-attention, where each split undergoes the same self-attention process as a separate 'head' that learns different attention representations.

factualpending

Speaker

Unidentified Speaker — Illustrated Guide to Transformers Neural Network: A step by… [4Bdc55j80l8]

Evidence Quote

“to make this a multi-headed attention computation you need to split the query key in value into adding vectors before applying self attention to split vectors that goes through the same self attention process individually each self attention process is called a head”

Source

Illustrated Guide to Transformers Neural Network: A step by step explanation— The AI Hacker
Created: 8/13/2026, 9:51:15 AM

Your notes

Loading notes...