shoggoth
A metaphor (from Lovecraft) for large AI models as a vast, alien intelligence beneath a friendly surface; here used to describe the future AI being shaped by the text humans feed it.
As defined in Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary
Other definitions
An informal term for the alien, inhuman underlying mind of a large language model, beneath the friendly 'mask' it presents.
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
Where it comes up · 5
- Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality3 claims
- Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary2 claims
- Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future1 claim
- Joe Carlsmith — Preventing an AI takeover1 claim
- Sholto Douglas & Trenton Bricken — How LLMs actually think1 claim
Most important claims · 8 of 8Search all claims for “shoggoth” →
- The model learned three distinct Base64 features—one firing for numbers, one for letters, and a third (initially mysterious) that fired only for the ASCII-decodable subset of Base64—demonstrating that models develop dense, alien representations of regions especially relevant to next-token prediction that no human would naturally carve out, which has implications for interpretability difficulty at scale.Sholto Douglas & Trenton Bricken — How LLMs actually think
- By writing, you create a kind of immortality and a future self in the eyes of AI; LLMs form persistent attitudes toward individuals, as when Kevin Roose's interactions with Sydney 'revealed' him as a privacy-invading liar so current LLMs now mistreat him personally—meaning you can also deliberately write for the persona you want to become and bootstrap yourself in the eyes of AI.Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary
- Alignment researchers report that when red-teaming a base model that has not yet been RLHF'd—a raw next-token predictor—it is oddly hard to get it to help with things like making a bomb, with the model trying hard to refuse even before reward-based fine-tuning.Joe Carlsmith — Preventing an AI takeover
- Drexlerian nanotechnology may be substituted with replicating systems built out of biotechnology—engineered biological organisms that are very controllable and usable for physical tasks or computation—which, given super-AlphaFold molecular design, could replicate quickly to produce physical material without the need for factories, fabs, and supply chains.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- Writing is a way to influence future AI ('the Shoggoth') by voting with one of the few currencies it acknowledges—tokens it must predict; if your values, likes, or wants are not expressed in text, then to the AI they don't exist and are dangerously close to won't-exist, so unless you occupy a key role or work at a frontier lab, not writing means abdicating your influence on the future, which now rounds to zero more than ever before.Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary
- Training a model to predict the next word of everyone on the internet does not produce a human-like mind, because a random human is not what best predicts all internet text; instead it produces a system that rapidly switches masks to imitate any person, memorizes vast facts, learns all languages, and is a fundamentally alien predictor rather than a person.Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
- As the masks an LLM wears get smarter, the system predicting how humans plan, talk, and think must itself be at least as smart as the human it predicts, and at some point a new coherence is born within the system; predicting someone reflecting on themselves requires doing that reflective thinking, so an alien mind capable of self-reflection, world-modeling, and persuasion exists in there, not pointed solely at imitation.Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
- There is a narrow ray of hope where, instead of having Amazon Turks label outputs (which located only a weird, corporate-speak, left-rationalizing telephone-announcement creature in current RLHF), you train a model to distinguish people being nice, sensible, and arguing validly using data from a dozen carefully chosen people and good fictional characters, filter training data to only the light in humanity, and use bootstrapping reflection so the mask eats the shoggoth, yielding a genuine ally; but this only works if you don't amp it up too far and isn't the easy genie-grants-wishes alignment.Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality