What this covers

This is a conversation between Max Tegmark and Sana founder Joel Hellermark, recorded at Tegmark's MIT office in April 2024. It begins with his move from cosmology into the study of intelligence and follows the argument through to questions of meaning. Tegmark opens with how little his colleagues expected: many thought something like GPT-4 was decades away, on the assumption that machines could not master language until the brain was understood. He describes their surprise that large language models (LLMs)) trained on text alone picked up translation and coding, and he cites internal geometric "maps" of geography and rotation-matched word embeddings across languages as evidence that these systems build real structure.

The middle of the conversation turns to what comes after current systems. Tegmark forecasts that Transformers will be remembered like vacuum tubes, and that better architectures will need far less data and power than today's data centers, which he reads as hardware compensating for weak software. He contrasts them with the brain, a recurrent neural network with loops, and notes a contested theory that such loops matter for consciousness. He also describes his group's work on symbolic regression, which recovered famous physics formulas from data and recently produced a new result on ozone chemistry. The problem is generally considered NP-hard, and his group made progress by testing whether networks decompose into modules.

The final stretch is about governance. Tegmark recounts the 2015 Puerto Rico conference, where figures including Demis Hassabis, Elon Musk and Eliezer Yudkowsky disagreed on timelines yet signed a statement on safety research. He recalls Alan Turing's 1951 warning that losing control would be the default outcome. His most contested claims are that the US has essentially no AI regulation, that safety standards would slow only the roughly 1% of work aimed at superintelligence, and that this is AI's "Enrico Fermi moment." The conversation closes on AI-driven science, education, and what a good outcome might look like.

Sharpest takeaway

Max Tegmark argues that AI systems are rapidly approaching human-level intelligence through fundamentally different architectural paths than the brain, and we must implement safety standards and interpretability research now to ensure beneficial outcomes before we lose control over increasingly autonomous systems.

  • Transformers are likely temporary solutions that will be superseded by better architectures combining neural networks with symbolic reasoning, similar to how humans think
  • We are at the 'Enrico Fermi moment' for AI (equivalent to 1942 nuclear reactor) and have only years before systems become significantly smarter than humans
  • Safety standards and mechanistic interpretability are both feasible and necessary to trust powerful AI systems, and should not meaningfully slow beneficial applications

The claims · ranked66 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.75

Tegmark's group recently discovered a new physics result in climate chemistry about ozone that was published, representing the first time they used these symbolic regression tools to advance science beyond merely rediscovering known results.

factualhigh valueestablishednovelty 2/4durability 3/4· Max Tegmark

“and we actually even managed to very recently discover a new physics result in uh in climate chemistry about ozone which we actually got published so that was the first time we used these tools to advance science a little bit”

0.75

Tegmark's group managed to discover automatically many of the most famous physics formulas just from data, and could have scooped Einstein and others on some discoveries if equipped with a time machine.

factualhigh valueestablishednovelty 2/4durability 3/4· Max Tegmark

“we actually were able to ReDiscover many of the most famous physics formulas just just from data so if we could go back in a time machine we could have scooped Einstein and others on some some cool stuff”

0.74

Large language models trained on all text on the internet learn to internally represent geographic spaces using geometric patterns—literally creating a two-dimensional map where Stockholm is located correctly relative to Beijing, even though they were never shown explicit maps.

factualhigh valueestablishednovelty 3/4durability 3/4· Max Tegmark

“have um large language mod read all the text on the internet and then it realizes that actually all these places it makes more sense if it chooses to represent them in a two-dimensional space a map even if it's never seen a picture of a map and then we wrote a paper where we looked inside llama 2 and there is a map sitting there literally where Stockholm is here and yet the B is there”

0.71

The task of automatically figuring out what formula a neural network has learned is called symbolic regression, and while it's generally believed to be NP-hard (could take longer than the age of the universe), Tegmark's group managed to get state-of-the-art performance by using ideas from physics to discover whether neural networks are modular and can be decomposed into smaller pieces.

factualhigh valueestablishednovelty 2/4durability 3/4· Max Tegmark

“the task of figuring out what the formula actually is that it's learned learned is called symbolic regression if it's for the Nerds who listening to this if it's a linear formula it's just linear aggression which is super easy but if it's some complicated formula like some of the physics formulas there on my by my window it's generally believed to be NP hard it could take longer than age of the universe just because there's exponentially many formulas that are of of length and but but that that one we managed to get stateof the art performance on we use a lot of ideas from physics where we were able to discover automatically if this neural network is actually modular”

0.69

Colleagues working on training large AI models were shocked to discover that models trained on text alone could learn capabilities they never explicitly taught them, such as translating English to Chinese or coding in Python.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“when I talked to to colleagues of mine who were working on training these models they they tell me themselves they were pretty shocked like oh my gosh this thing can translate English to Chinese we never taught it to do that oh my gosh it can code in Python like how did it figure that out”

0.69

AI is already enabling AI development to go faster through tools like coding autopilots that make developers more productive, creating a kind of recursive self-improvement with humans still in the loop, similar to how farming went from employing most people to employing 1% of people.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“today you have must have a lot of people doing coding in your company right who are using autopilots of some form I don't know if it's making them twice as productive or 1.5 times or three times as productive but this is already an example of how AI is enabling AI development to go faster so it is a kind of self recursive self-improvement but there's still humans in the loop uh it's just that they over time we get less and less humans in the loop when we had farming before like almost all people in Sweden were in that Loop farming now there's like 1% in that Loop”

0.69

In 2015, Tegmark organized a major AI safety conference with Demis Hassabis (CEO of DeepMind), Elon Musk, and top AI safety researchers including Eliezer Yudkowsky and Nick Bostrom, using social pressure (proximity to a beach in Puerto Rico) to get leading AI researchers to engage with safety concerns they had previously dismissed as taboo.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“I had this Vision that we should actually try to bring them together and you know one thing that Sweden is strangely good at is throwing good parties from the Nobel Fest to the more informal ones so I I pull out all the stops to to entice people to come to this conference put it not in Sweden but in Puerto Rico in January I sent out an invitation with a picture of a guy digging his car out from one meter of snow right next to the beach by the hotel and be like where would you rather be in January of 2015 and um I went first after the most high-profile people I thought maybe I could actually persuade and then when you get them you know you can start building a Swedish Snowball the RADS down the hill and you get more and in the end we got uh amazing people we got Demis aabis the CEO of deep mine we got Elon Musk we got really top professors from Academia and also all the top people who had been worrying you know Ellie from elaz owski to Nick boam”

0.69

Tegmark's 2015 conference catalyzed a field shift where AI safety stopped being taboo to discuss at technical AI conferences, with dedicated sessions on safety emerging as a normal part of academic discourse.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“pretty quickly um this field it stopped being taboo to talk about AI safety and the technical AI conferences would start having the nerd sessions basically on on these topics”

0.69

As AI gets more powerful and we can use AI to figure out how to make better AI, future AI systems will probably discover much better architectures than current Transformers, and current oversized data centers with nuclear reactors are temporary because we are overcompensating for poor software architectures with ridiculous amounts of hardware and training data.

forecasthigh valuecontestednovelty 3/4durability 2/4· Max Tegmark

“the neat thing is if we start getting ever more powerful Ai and we can to the point where we can use AI to figure out how to make much better AI it will probably disc over those really really clever things so my prediction is actually that even though now we're getting these ever more ginormous data centers the size of an airplane hanger where you have to almost put a nuclear reactor next to it soon to power the whole thing that's just temporary I I suspect that we're overcompensating for a really poor software architectures with this ridiculous amounts of hardware and training data once get over that hump we'll realize that you could do it all with much less Hardware much less energy and much less training data”

0.68

The human brain is a recurrent neural network where information flows around in loops, while Transformers have no loops, and one detailed theory of what causes consciousness (subjective experiences of colors, sounds, love, etc.) is that you need these loops for it.

factualhigh valuecontestednovelty 2/4durability 3/4· Max Tegmark

“our our brain for example is a so-called recurrent neural network where information flows around in Loops there are no Loops in in a Transformer and um that's very interesting actually because one of the more detailed theories of what causes Consciousness subjective experiences of colors and sounds and love and so on is that you need the loops for it”

0.68

Once a language model learns to represent geographic information as a geometric map, it can answer novel questions it never saw during training, such as 'Is Yosemite west of Katmandu', by using its internal geometric representation.

factualhigh valueestablishednovelty 2/4durability 3/4· Max Tegmark

“once it's realized that it can represent things like this it can now start answering questions that it never saw in the training data like if you ask it is Yo west of Katmandu you know it probably never saw that question but it has a map it's represented it this way because it to answer all the some other questions right and now yeah it can figure these things out”

0.68

There is a very common misconception that we must choose between quickly reaping benefits of AI and avoiding going extinct, but 99% of exciting applications are quite harmless and have nothing to do with building superintelligent systems, so safety standards would only slow the last 1% of risky work.

normativehigh valuecontestednovelty 2/4durability 3/4· Max Tegmark

“I think a very common misconception is that um somehow we have to choose between quickly reaping all sorts of benefits of AI on one hand and on the other hand avoiding going extinct the the the the the fact is 99% of the PE of things that the people I talk with are excited about which I think includes you with AI are things that are quite harmless that that have nothing to do with building smarter than human AI that we don't know how to control can cure we can help spread knowledge we can help make companies more efficient we can we can do great progress in science and medicine etc etc so if we put safety standards in place that just end up slowing down a little bit that that last percent the stuff that we might lose control over”

0.68

Word embeddings from English text and word embeddings from Italian text can be matched by rotating and shifting them in high-dimensional space, allowing creation of a pretty good English-Italian dictionary purely from geometric pattern matching without explicit translation training.

factualhigh valueestablishednovelty 2/4durability 3/4· Max Tegmark

“there was a if you look at how it represents words it puts them in a sort of high dimensional map word embeddings and if you take there was a nice paper recently where someone took the word embedding of English words from reading a bunch of English text and word embeddings in Italian from reading a bunch of different Italian text and figured out how do you need to like rotate and shift these things so that they kind of match well and they got out a pretty good English Italian dictionary just from these P matching up the these patterns”

0.65

Large language models can be extended from outputting tokens to outputting commands that control robots, humanoid robots, or self-driving cars, making them not just language systems but controllers of physical agents.

forecasthigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“this also is very powerful for um building uh robots whether they be robots on Wheels like self-driving cars or or more humanoid ones where it can fundamentally it can be controlled the some level by a large language model then once some of the tokens and outputs just motor commands and things like that”

0.65

At the 2015 AI safety conference, there were major disagreements about timelines for achieving superintelligent AI and about whether superintelligence would be net positive or negative for humanity, but broad agreement that it might happen soon enough to warrant precautions.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“there were big there were big differences in in the forecasts people had for like how long is it going to take to get super smarter than human AI there were also big dis disagreements as whether people thought it would probably be fine or great or whether would probably suck um but everybody pretty much agreed that um you it might happen and it might happen soon so so you know with that humility makes sense to take some precautions”

0.65

Human brains learn from often a lot fewer examples and need much less training data than large AI systems today, with the brain using only 20 watts of power compared to data centers using many millions of watts.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“human brains learn from often a lot fewer examples we need a lot less training data than the big AI systems of today and you see these data centers going up using many many millions of Watts you know your brain 20 watts”

0.64

We should not be thinking too small in terms of AI architecture design; we should look beyond Transformers and minor tweaks, as history shows repeated quantum leaps in computational architecture from mechanical to electronic to programmable to neural networks, and the brain uses yet another different paradigm.

normativehigh valuecontestednovelty 2/4durability 3/4· Max Tegmark

“if there is something really bananas that we're not paying enough attention to it's probably that we're thinking too small in terms of architecture design we're looking at the Transformer and make thinking about like minor tweaks but there could easily be completely different types of architectures even just look at the history of computing how many times we've had like a Quantum Leap in architecture”

0.64

All other technologies with potential to cause harm (airplanes, medicines) have safety standards and regulations, but AI in the US has basically no regulation, allowing anyone to release powerful systems tomorrow without safety review.

factualhigh valueestablishednovelty 1/4durability 2/4· Max Tegmark

“you look at all other technologies that have the potential to cause harm we've we have a solution for how we do it with all of them whether it be airplanes or medicines we always have safety standards you know you can't have a if if asra come comes up with says we have a new miracle drug that's going to cure cancer and five you know and we're going to start selling it in E tomorrow La m v would be like okay where is your clinical trial oh you you couldn't be bothered make you haven't had time to make one you just come back when when you've done the clinical trial and then we'll see if you meet the standards right so that buys time for everybody involved it's the responsib the companies now have an incentive to figure out what the impact on society is going to be how many percent of people are going to get this side effect that effect they quantify everything in a very nerdy way and that way we can trust our biotech that's why ultimately asena has a pretty good reputation uh same thing with airplanes same thing with cars same thing with basically any Tech that cause harm except for AI where here in the US there's basically no regulation if some of them wants to release dpd5 tomorrow he can right”

0.63

Transformers are going to be remembered as the vacuum tubes of AI—the first technology that really let us build electronic computers, but then we found better ones—and we should expect much better AI architectures in the near future that do the same thing with much less data, much less power, and much less electricity use.

forecasthigh valuecontestednovelty 2/4durability 2/4· Max Tegmark

“I think that's way too narrow-minded I think uh Transformers are going to be remembered as the vacuum tubes of AI you know vacuum tubes were the first technology that really let us build electronic computers but then we found better ones and I'm pretty confident that in not too many years we'll have found much better AI architectures than what we have now which let you do the same thing with what much less data much less power much less electricity use”

0.63

At Tegmark's 2015 AI safety conference, all major figures signed a statement agreeing that AI safety risks are real and committing to research not just on making AI more powerful but also on making it safe, transparent, and controllable.

factualhigh valueestablishednovelty 0/4durability 3/4· Max Tegmark

“everybody came together and signed the statement saying you know yeah the risks are real uh let's let's deal with them and um let's do research not just on how to make AI more powerful but also on how to make it safe and transparent and controllable”

0.62

The sooner the US switches to treating AI like we treat other powerful technologies (with safety standards and regulation), the better, as this would not interfere with the vast majority of beneficial AI applications that don't involve building superintelligent systems.

normativehigh valuecontestednovelty 1/4durability 3/4· Max Tegmark

“so I think the sooner we switch over to treating AI the way we treat all other powerful Tech the better”

0.62

The current moment is roughly at the Enrico Fermi moment for AI (like the 1942 nuclear reactor), and it is high time to take seriously that big things are going to happen soon and to prepare now.

normativehigh valuecontestednovelty 1/4durability 3/4· Max Tegmark

“when um it's high time to yeah just take seriously that big things are going to happen soon and um let's get it right let's prepare”

0.61

Most colleagues thought we were decades away from something as smart as ChatGPT4 because they believed we could never make machines master language and human knowledge until we figured out how the brain works, but this turned out to be wrong because there was an easier way to make machines think well.

factualhigh valueestablishednovelty 1/4durability 3/4· Max Tegmark

“just a four years ago most of my colleagues thought that uh we were decades away from something as smart as chat G pt4 because they thought we can never make machines of M Master language and human knowledge until we figured out how our brain works and we're nowhere near it still then it turned out there was an easier way to make machines and think well”

0.60

In 1942, Enrico Fermi built the world's first nuclear reactor in Chicago, and when physicists found out they freaked out not because it was dangerous, but because they realized we were only a few years away from nuclear bombs, and 3 years later Hiroshima and Nagasaki happened.

factualhigh valueestablishednovelty 0/4durability 4/4· Max Tegmark

“so in in 19 1942 enrio fary built the world's first nuclear reactor in Chicago under a football stadium and um the phys when physicists found out about that they totally freaked out why was it because they thought this reactor was really dangerous no was really small low energy output it's because they realize that now we're only a few years away from the bomb and 3 years later Hiroshima nasaki happen right”

0.60

Around 1951, Alan Turing said that one day machines would become as smart as people and then very quickly they'd become way smarter than people (since machines are biological computers and there's no reason machines can't do much better), and the default expectation would be losing control over the machines.

factualhigh valueestablishednovelty 0/4durability 4/4· Max Tegmark

“it's there's a nice analogy there because around that time in 1951 Alan Turing said that one day machines will become as smart as people and then very quickly they'll become way smarter than people because we're biological computers and there's no reason machines can't do much better and then the default is we lose control over the machines”

0.60

The 'Enrico Fermi moment' for AI—the warning sign that superintelligence is imminent—is when machines become good enough at language and knowledge that they can fool many people into thinking they are human, and Yoav Bengio (one of the most cited AI researchers) argued that GPT-4 has passed the Turing test.

factualhigh valuecontestednovelty 2/4durability 2/4· Max Tegmark

“I'll give you a little Canary in the coal mine warning so you know when you're close the ter test once machines become good enough at language and knowledge that they can fool a lot of people that um thinking that they are human that's when you're close that's the enrio FY moment when you might have a few years and U last year yosu Benjo even one of the most cited AI researchers in the world argued that gp4 passes the touring test”

0.57

Exponential growth in productivity and technology is not growing at a constant exponential rate but is growing faster than exponentially, with the time it takes to get twice as good continuing to shorten.

factualhigh valuecontestednovelty 1/4durability 2/4· Max Tegmark

“what but what we also see is in this exponential growth if you like look at the world's GDP over time or or most measures of of tech um the time it takes to get twice as good to keep shortening so it's actually growing faster than exponentially”

0.57

Similar to how farming went from employing almost all people in Sweden to 1% and how car factories have far fewer people per car produced than 50 years ago, software development will gradually require fewer and fewer humans in the loop, and eventually might have no humans in the loop at all, at which point things will take off even faster.

forecasthigh valuecontestednovelty 1/4durability 2/4· Max Tegmark

“similarly with software development they'll gradually be fewer and fewer if you look at a car factory today there way fewer people per car produced than there were 50 years ago so I I think that's how it's going to go we'll we'll see that uh you need fewer and fewer and fewer humans in loop at some point there'll be no might be no human in the loop and then things will take off even faster”

0.56

The situation with flight shows that we can build machines that do complex tasks in fundamentally different ways than biological systems: someone could have incorrectly claimed we would never figure out flying machines until we figured out mechanical birds, but there was a much easier way to build machines that could fly much faster than birds.

normativehigh valuespeaker onlynovelty 2/4durability 4/4· Max Tegmark

“this reminds me a lot of U the situation with flight you know in the year 1900s someone could have said oh we're never going to figure out how to make flying machines until we figured out mechanical Birds but that was completely wrong because there was a much easier way to build machines that could fly much faster than Birds even”

0.56

Even a very few number of people can often make a big difference because many things that need to happen don't happen just because of the bystander effect—if you have an idea for a startup or social movement and think it's impossible because otherwise someone else would have done it, it might very well be that nobody else actually tried very hard yet.

normativehigh valuespeaker onlynovelty 2/4durability 4/4· Max Tegmark

“even very few people can often make a big difference many times um things that need to happen don't happen just because of the bystander effect you know and um I think that's a message to anyone listening to this if they have an idea for a new startup or or just some new social movement or anything and they're like ah this must be impossible because otherwise someone else would have done it no it might very well be that nobody else actually tried very hard yet”

0.52

AI systems' ability to generalize and answer entirely new questions that they were never trained on comes from their discovery of geometric patterns, which is one of the key reasons they are often able to answer novel questions and transcend their training data.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“I think these AI systems are learning a lot of these geometric patterns and uh that's one of the key reasons that they are often able to generalize and actually um answer whole new questions that you have never that they've never been trained on cuz that seems to be the way”

0.52

There is too much pessimism about whether we can build AI systems that we can actually understand and prove things about, because people assume all interpreting and proving work must be done by humans, but AI systems are getting really good at that work and can help us.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“I I think there's been too much pessimism on um this question about whether we'll able be able to build systems that we can actually trust and even prove things about because people have made this mistake of assuming that all the work in interpreting and proving has to be done by people but AI systems are getting really good at that stuff now and they can help us”

0.52

The rise of many autonomous systems acting in the world shifts the nature of the challenge from a 'new technology like electricity' to something more like 'a new species', requiring fundamental reconsideration of how we approach it.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“this can of course be great for me for many purposes but this is also obviously something we have to be careful with because if we have a a lot of autonomous systems act in the world you know it starts to feel less like just a new technology like electricity and more like a new species”

0.52

The brain was optimized by evolution under constraints to develop only a biological computer that could self-assemble using the most common atoms in the periodic table, whereas engineers have no need for such constraints, leading to limitations in our current approach to AI.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“when uh the brain was optimized by Evolution it was constrained to only develop a biological computer that could self assemble and that could do it with only the most using the most common atoms in the periodic table you know really weird constraints that Engineers just don't care about whereas um there was no limit on there was no need that for it to be simple and easy to understand stand whereas that's where we tend to be limited by as engineers”

0.52

What is special about humans is not that we can see better than an eagle or have better olfactory systems, but that we can reason with symbols and communicate with human language, mathematical language, and programming languages, and we seamlessly combine old AI (symbolic) and new AI (neural) capabilities.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“what's special about humans is not that we can see better than an eagle it's that we can also reason with symbols and communicate with human language and mathematical language and programming languages and somehow we can combine the old Ai and the new AI in a seamless way”

0.52

Tool use by AI systems can overcome intrinsic weaknesses in large language models by combining them with old-fashioned AI systems that can make very powerful tools, allowing the LLM to learn to use them and get the best of both worlds.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“this is this is something we humans of course do routinely if you're driving you know Sly almost becomes like an extensive extension of your body this can this can start to Bridge I overcome a lot of the intrinsic weaknesses I think in large language models because we have all these old-fashioned AI systems that can make very powerful tools so if the llm learn to use them then um they can get the best of both worlds”

0.52

It is a mistake to think that AI systems are not self-improving now, and then suddenly become self-improving and trigger a singularity; rather, self-improvement in technology is a gradual transition where today's technology is always used to build tomorrow's technology, leading to exponential growth.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“I think it's a mistake to think that now over time now it's self it's not self-improving now it's not self-improving and all of a sudden it is and boom Singularity it's a gradual transition technology has always been self-improving we always use yes today's technology to build tomorrow's technology which is why we've seen an exponential growth in productivity in in technology of most measures”

0.52

The brain solves the problem of arranging atoms to be very smart by evolving solutions to biological constraints, whereas engineering doesn't have these constraints, suggesting we might discover completely different architectural paradigms if we think beyond current frameworks.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“if you just take a step back and say I'm going to give you a big blob of atoms you know what's the best way to arrange these to be really smart you know it might be something we haven't thought about at all”

0.52

If you have an AI system that will affect people's lives and you want high trust, it's a good idea to let the machine learning do the learning (since there's no better way), but then distill what it learned into python or another language where you can actually prove the code meets your specifications.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“if you have a system an AI system that's going to affect people's lives and some way where you really want to have a high trust then um it's a really good idea if you let the machine learning do the learning we don't have any better way than that but then can distill out what it's learned into python or something else where you can actually prove that the the code meets the specs that you have”

0.52

Top AI thinkers and business leaders often don't have as much time to step back and reflect as they probably wish they did, and they often have the default strategy of 'first make it work, then figure out how to make it good for society'—and many were surprised when ChatGPT and Stable Diffusion came decades before expected.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“most of uh top AI my thinkers and Business Leaders that I know don't have as much time for just to step back and reflect as they probably wish they did it's tough to run a company for example and um I get this as many of the most have had this idea that well first I just need to get this thing to work and then I'm going to figure out my strategy for how to make sure this is good for society and all and then many of them were really taken by surprise that chat gpts and stable diffusion and so on came like decades before many expected”

0.52

Safety standards and clinical trials for other technologies work well because they buy time for everybody involved and give companies incentives to quantify impacts on society, understanding side effects and safety profiles, leading to trustworthy technologies like biotech with good reputation.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“that buys time for everybody involved it's the responsib the companies now have an incentive to figure out what the impact on society is going to be how many percent of people are going to get this side effect that effect they quantify everything in a very nerdy way and that way we can trust our biotech that's why ultimately asena has a pretty good reputation”

0.52

You should have a strong bias toward humans over AI models because you are human, you have a one-year-old child you should feel more loyal to him than to a random machine, and humans have collectively figured out how to build AI and therefore have the right to have some influence over the future we build.

normativehigh valuespeaker onlynovelty 1/4durability 4/4· Max Tegmark

“cuz I am a human you know I I I have a lovely little one-year-old Leo yeah and when I look into his eyes you know I feel he's he's my son of course I should feel more loyal to him than to some random machine shouldn't I and uh I feel um we humans have figured out collectively how to start building Ai and so I think we have the right to have some influence over what future we build I'm on team human here not on team machine”

0.52

AI models might eventually supersede current human knowledge not just by drawing analogies between different disciplines (which would be infeasible for humans to master all), but also by fitting vast amounts of knowledge into their short-term context windows.

forecasthigh valuespeaker onlynovelty 2/4durability 2/4· Max Tegmark

“that seems to be the way at which these models could also supersede current human knowledge uh where they're able to draw these analogies between been very different disciplines in a way which would have been infeasible for humans maybe you're clever enough to sort of Master all human knowledge but it seems like that's quite a difficult thing to to do today but that this models canot only do that but they can even fit it into their shortterm memory soon with like the context context windows”

0.52

If safety standards are implemented that slow down the last 1% of work on superintelligent systems, and it takes a bit longer to reach that milestone, this is acceptable because we're not in any great rush—we can have life flourishing for billions of years if we get this right, so it's not worth risking squandering everything just to get it one year sooner.

normativehigh valuespeaker onlynovelty 1/4durability 4/4· Max Tegmark

“then we can really have this age of abundance for a number of years now where we can enjoy revolutions and health care and education and so many other things without having to lose sleep that this is going to all blow up on us and then when we if we can get to the point eventually where we can see that even more powerful systems meet the safety standards great now if it takes a little longer fine we're not in in any great rush we can have life flourishing for billions of years if we get this right so there's no point in risking squandering everything just get it one year sooner”

0.50

Scaffolding—thinking of the LLM as just one component in a bigger architecture (combining system 1 fast/intuitive neural networks with system 2 logic-based symbolic reasoning)—is very promising for creating more powerful AI than today's systems.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“what's very promising is is scaffolding where you think of the llm as just one component in a bigger architecture kanaman who sadly passed away around age 90 just just very recently used to talk about system one and system two in our human brains system one the fast thinking the intuitive thinking is a lot like neural networks where system 2 is the more logic based symbolic reasoning we have which lets us speak Swedish and English and math and Python and um our if you think about the future where the neural network the Transformer say is system one you can imagine that it's just part of this bigger architecture which can enable clever more clever database structures all sorts of tools Loops um various other techniques”

0.48

AI systems will absolutely supersede current human knowledge and there is no doubt about that, and if humans could read all the internet and everything written in all languages and keep it in mind, they would see a lot of patterns too just like these AI systems do.

forecasthigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“I think it's absolutely going to happen no doubt about that and um I mean if if you and I could just read all the internet and everything I've written in all the languages somehow and keep this in our minds I'm sorry I cannot we would see a lot of patterns too”

0.48

Humans fundamentally understand how to distill knowledge because as children learning, humans first intuitively figure things out (like computing parabolic trajectories to catch a ball), then learn to articulate and formalize the knowledge (like the y=x^2 formula), and finally can program it into systems like moon rockets.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“I fundamentally believe that it's possible because we humans can do it you know like when when when you were a little kid if your dad threw you a tennis ball you could catch it because your brain had figured out how to compute parabolic trajectories but when you got older you're like oh this is a parabola yal x^2 you know formula and this is how scientists generally first intuitively figure out stuff even though they don't know how their brain works and then they learn to distill out the the knowledge in a way that you could actually program into a moon rocket”

0.48

The rapid growth in the field of mechanistic interpretability and Tegmark organizing the biggest conference so far at MIT on this field makes him hopeful that we don't have to be resigned to never understanding AI systems—the ones we really want to understand we can, and other AI systems can help us understand them.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“the fact that we managed to make so much progress on this already and the fact that this field is rapidly growing I organized the biggest conference so far here at MIT on this this field of it's called mechanistic interpretability makes me pretty hopeful actually that um we don't have to be resigned to the idea that we'll never understand AI systems the ones that we really want to understand I think we can use other AI systems to help us understand them”

0.48

AI systems that have deep insights and broad knowledge can revolutionize education by understanding where students are, what they already know, what misconceptions they have, and then determining the most helpful way to convey knowledge using appropriate metaphors, analogies, and presentation strategies.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“clearly that can revolutionize education you know my job is not just that of a researcher even though that's what I spend most of my time on here and this office but also very much as an educator and um to me the first thing I always have to do before I start teaching something is make sure I really have a deep understanding of it much deeper than what the students need to get later and that lets me and then I then I need an understanding also of the students where are they that's also something AI can get better at understanding what what the people who are trying to learn actually know already and what they don't know and what their misconceptions are and then finally I can tackle this question what is the most helpful way for me you know to convey this knowledge where do I start what are the metaphors and analogies are going to work”

0.48

Given the enormous amounts of money currently spent on training ever bigger models and making them more powerful, it is reasonable even for skeptics about AI risks to dedicate a substantial amount to figuring out how to make these systems safe.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“and you know given the enormous amounts of money we're spending right now on training ever bigger models and making a more powerful it's pretty reasonable even for people who are sort of skeptical of the risk to say well let's spend lead some substantial amount also on figuring out how to make these systems safe”

0.47

The fundamental desire is for machines to work for humans, not the other way around, distinguishing between being on 'team human' versus 'team machine' in terms of who should control the future.

normativehigh valuespeaker onlynovelty 0/4durability 4/4· Max Tegmark

“I think we want to have the machines work for us not the other way around”

0.47

Tegmark's vision is to be able to take any blackbox AI system and figure out what algorithm it has learned and what knowledge it has acquired, with the goal of ultimately being able to distill what it learned into python code that can be provably verified to meet specifications.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“more broadly what we want to do is ultimately be able to take any blackbox AI system and figure out what what algorithm it's learned and what what knowledge it's really learned the our last paper on this we was one where we we took um about 60 different algorithms and we trained a neural network to just learn to do these tasks so now you have a blackbox but what did it do and then we had an automated AI system which was able to figure out exactly what it was doing and turn it into python code”

0.47

Rather than classifying exactly what current AI systems are (oracles vs agents vs token predictors), it is easier to look at the endpoint of where this technology is going, which is toward autonomous agents that take information, figure out what actions to accomplish some goal, and then go out and do it.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“it's pretty diverse I I I I think it's almost easier to look at the end point of where this is going rather than trying to classify exactly what happens right now there's a huge commercial pressure to build systems that are agents traditional llms are more like an oracle you ask it a question and gives you an answer but there's so much commercial value to have agents which will actually do stuff take information and figure out what what actions to take to accomplish some goal and then go outad and do it”

0.47

Even if you can't read a very long AI-generated proof or code (because it's too long to fully understand), you only need to understand your proof checker (which can be 300 lines of python) to fully trust a very powerful system made by an AI, because it's much easier to verify an existing proof than to find one.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Max Tegmark

“you might think oh how am I ever gonna trust a system if that was produced by an AI with a proof produced by an AI if I don't understand if the both the proof and the code is too long to read that's okay um because it turns out that it's just like it's much harder to find a needle in a hay stack than it is to prove that it's a needle once you found it it's much harder to find the correct code and a proof that it meet your spec that it is to verify that that it works once you have it so all you have to do is actually understand your proof Checker which can be 300 lines of python”

0.45

Even if someone is skeptical that superintelligent AI would lead to human extinction, they should still buy 'fire insurance' by supporting AI safety research, similar to how people buy fire insurance on houses they don't think will burn down.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Max Tegmark

“you know these are people who still buy fire Insurance on their house that doesn't mean they think their house is probably going to burn down but just in case you know why not prepare a little bit maybe have also put in a smoke alarm and have a fire extinguisher handy”

0.43

The history of AI can be divided into two stages: stage one was Good Old-Fashioned AI with symbolic logic-based systems, and stage two was self-learning neural networks that largely crushed the old approaches, creating a false impression that neural networks are inherently superior.

definitionestablishednovelty 1/4durability 3/4· Max Tegmark

“if you um very crudely think of the history of AI as two stages stage one was GOI good oldfashioned AI where you have these symbolic logic based systems and then stage two was the self-learning neural networks that largely crushed the old stuff this has made us think that the the neural enworks are like the better cooler one”

0.41

Douglas Engelbart challenged Marvin Minsky when Minsky claimed computers would reason, be conscious, etc., by asking 'You're going to do all of that for computers—what are you going to do for humans?', highlighting that early AI researchers sometimes neglected human augmentation.

factualestablishednovelty 0/4durability 4/4· Max Tegmark

“there's this one story about Douglas engelbart running into Marvin Minsky and Marvin Minsk is telling engelbart about all of the things he will make computers do they will reason and they will be conscious and so on to which engelbart replied you're going to do all of that for computers what are you g to do for humans”

0.40

The best case scenario is that in two decades, humans are still alive and happy, remain in charge of the planet, and have solved famine, wars, and climate problems, managing to address the biggest problems that have stumped humanity throughout history.

forecasthigh valuespeaker onlynovelty 0/4durability 2/4· Max Tegmark

“The best case is we're still alive and we're happy and the humans are still in charge of this planet uh except there's no more famine no more Wars the climate is all good and we have um basically managed to solve the biggest problems that have stumped people throughout the ages”

0.39

2024 might be remembered as the year of the agents, when we start seeing a lot of more autonomous systems, first purely software ones on the internet and then gradually more and more physical agents.

forecasthigh valuespeaker onlynovelty 1/4durability 1/4· Max Tegmark

“I think 2024 will probably also maybe be remembered as the year of the agents when we start seeing a lot of more autonomous systems first purely software ones on the internet and um and then gradually we'll get more more and more physical agents too”

0.39

In 2024, we might see a lot of patterns and connections being discovered not just in terms of basic knowledge but also in terms of AI systems learning to use tools, such as calculators for math, dictionaries, databases, and other external tools to overcome intrinsic weaknesses in large language models.

forecasthigh valuespeaker onlynovelty 1/4durability 1/4· Max Tegmark

“what I I think uh we might be seeing a lot of in 2024 is um patterns and see connecting things not just in terms of the basic know but also in terms of using tools um so right now these llms they tend to Output tokens new word or pieces of words but people are realizing you can also give them the option to Output commands to a a calculator that can do really good math or to a dictionary or to all sorts of other tools or to a datab base”

0.37

Elon Musk committed 10 million dollars to a grants program to get researchers working on AI safety, demonstrating how a conference can catalyze funding for risk research.

factualestablishednovelty 0/4durability 3/4· Max Tegmark

“Elon Musk said okay I'll give you 10 million to do a grants program to start getting a bunch of nerds working on this so we launched that”

0.35

Neural networks trained on multiple tasks can find both simple heuristic solutions and more complex precise formulas, placing them on a spectrum from simplicity to accuracy, similar to how humans approach problems with both quick heuristics and more precise methods.

factualestablishednovelty 1/4durability 3/4· Max Tegmark

“our system actually does both so it actually finds many form is it's it's we put them in a a plot where on one axis is how complicated they are and on the other axis is how inaccurate they are and it would find like in this case the the simplest one that just the parabola which where there is no air resistance whatsoever and then but then it would find a more complicated it could find a more complicated one which is more precise”

0.33

The universe's fundamental equations don't have explicit mention of meaning, so meaning-making is up to conscious living beings to create our own meaning in the universe rather than having meaning given to us.

normativespeaker onlynovelty 1/4durability 4/4· Max Tegmark

“something we've discovered by finding all these fundamental equations of physics is that none of the equations have any explicit me mention of meaning put into them so I think it's rather up to us conscious living beings to to create our own meaning”

0.32

The fact that we weren't using mixture of experts and weren't sufficiently curating datasets in prior AI model designs suggests significant room for improvement through better architectural and data selection choices.

factualcontestednovelty 1/4durability 3/4· Max Tegmark

“I mean it's quite absurd that we weren't using like mixture of experts and so on in the prior designs of of some of this models and and also that we weren't curating the the data sets sufficiently”

0.29

Consciousness and positive experiences are at the heart of meaning, including experiences like beauty, love, passion, kindness, and hope, which are conscious experiences, whereas inanimate objects like desks don't have these experiences and thus can't give meaning.

normativespeaker onlynovelty 0/4durability 4/4· Max Tegmark

“I think it's rather up to us conscious living beings to to create our own meaning frankly I think that um Consciousness and positive experiences is is at the heart of this because even Beauty and love and passion and kindness and Hope are also conscious experiences after all I don't think my desk here is having any of those experiences and um in other words I don't think that our universe gives meaning to us we can give meaning to our universe”

0.26

Having access to AI systems that can teach in the way that will really make you understand something—answering questions like a real expert who gives you their time—would be one of the great joys and benefits of AI technology.

normativespeaker onlynovelty 0/4durability 3/4· Max Tegmark

“I certainly love the idea of talking to some real expert in something who's willing to give me some of their time and answer questions in a way that makes sense to me it's one of the great joys of working at this University and uh if um there are AI systems that will play that role also and teach me things the way I will really get it I would love that”