heuristic argument
A non-rigorous but structured reasoning that gives a best-guess estimate of whether something is true, subject to revision by new information; ARC seeks to formalize these.
As defined in Paul Christiano — Preventing an AI takeover
Where it comes up · 1
Most important claims · 5 of 5Search all claims for “heuristic argument” →
- ARC's research seeks a formal notion of 'explanation' for a model's behavior as a deductive argument from the weights — proceeding step by step from properties of the network (e.g. two vectors have large inner product, so activations correlate) to conclusions about outputs — but with the rigor standards of proof relaxed, since proofs are too restrictive to apply to interesting neural nets while still yielding a structural causal explanation.Paul Christiano — Preventing an AI takeover
- The key hope of ARC's project is that finding a logical-style explanation for why a neural net works is of similar difficulty to finding the net itself, because the same way neural nets manage to embed discrete logical reasoning into continuous searchable spaces, explanations of that reasoning can be embedded and searched over in parallel — contrary to the common intuition that finding the net is much easier than explaining it.Paul Christiano — Preventing an AI takeover
- A unifying perspective behind heuristic arguments is that it's reasonable to start by treating an object as random and then revise from that initial guess as you notice structure — e.g. reasoning about prime numbers as if they were a random set of numbers, since primes happen to have little additive structure — rather than treating randomness-based reasoning as a special fact about a particular object.Paul Christiano — Preventing an AI takeover
- Most claims mathematicians believe true (like the Riemann hypothesis) already have fairly compelling informal heuristic arguments — the Riemann hypothesis should hold unless the primes have some surprising periodic structure — so formalizing heuristic estimation would not shock mathematicians, though perhaps 10% of accepted compelling heuristic arguments would turn out not quite right if formalized.Paul Christiano — Preventing an AI takeover
- Christiano estimates roughly a 10-20% chance the explanation-search project fully succeeds (explanations that accurately reflect reality and work for all imagined applications), with a higher probability of intermediate results providing value; the overwhelming reason to expect failure is that the desired notion of explanation may be incoherent or intractably difficult.Paul Christiano — Preventing an AI takeover