Nate Soares

❧

Wikipedia

Nathaniel Soares is an American artificial intelligence author and researcher known for his work on existential risk from AI.[1][2] In 2014, Soares co-authored a paper that introduced the term AI alignment, the challenge of making increasingly capable AIs behave as intended.[3][4] Soares is the president of the Machine Intelligence Research Institute (MIRI),[5] a research nonprofit based in Berkeley, California.

In 2025, Soares co-authored If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. In the book, he argues that creating vastly smarter-than-human AI (or superintelligence) “using anything remotely like current techniques” would very likely result in human extinction; he further contends that AI alignment remains nascent and that international regulatory intervention will likely be required to prevent developers from racing to build catastrophically dangerous systems.[6][7][8][9] The book received mainstream attention and mixed reviews: The Guardian selected it as "Book of the day" and described it as "clear" though its conclusions are "hard to swallow"; The Washington Post characterized it as a polemic that offers few concrete instructions; and The New Yorker featured it in "Briefly Noted." Coverage also included reporting in Wired, Semafor, and SFGATE.[10][11][12][13][14][15]

Life

Soares received his Bachelor of Science degree (in computer science and economics) from George Washington University in 2011.[16] Soares worked as a research associate at the National Institute of Standards and Technology and as a contractor for the United States Department of Defense, creating software tools for the National Defense University, before spending time at Microsoft and Google.[17][18]

Soares left Google in 2014 to become a research fellow at the Machine Intelligence Research Institute.[19] He served as the lead author on MIRI's research agenda, which in January 2015 was cited heavily in Research Priorities for Robust and Beneficial Artificial Intelligence, an open letter calling for AI scientists to prioritize technical research “not only on making AI more capable, but also on maximizing the societal benefit of AI.”[20][21] This included “research on the possibility of superintelligent machines or rapid, sustained self-improvement (intelligence explosion).”[20]

Shortly after joining MIRI, Soares became the institute's executive director.[19][22] In 2017, he gave a talk at Google outlining open research problems in AI alignment, and arguing that the alignment problem looks especially difficult.[23]

In 2023, MIRI shifted from a focus on alignment research to a focus on warning policymakers and the public about the risks posed by potential future developments in AI.[24] Coinciding with this change, Soares transitioned from the role of executive director to president, with Malo Bourgon serving as MIRI's new CEO.[24][25]

Publications

  • Garrabrant, Scott; Benson-Tilsen, Tsvi; Critch, Andrew; Soares, Nate; Taylor, Jessica (2017). "A Formal Approach to the Problem of Logical Non-Omniscience". Proceedings Sixteenth Conference on Theoretical Aspects of Rationality and Knowledge. pp. 221–235.
  • Soares, Nate (2018). "The Value Learning Problem" (PDF). In Yampolskiy, Roman (ed.). Artificial Intelligence Safety and Security. Chapman & Hall. ISBN 9780815369820.
  • Soares, Nate; Fallenstein, Benya (2017). "Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda" (PDF). In Callaghan, Victor; Miller, James; Yampolskiy, Roman; Armstrong, Stuart (eds.). The Technological Singularity. Springer. ISBN 9783662540312.
  • Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer; Armstrong, Stuart (2015). "Corrigibility" (PDF). AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 25–26, 2015. AAAI Publications. pp. 74–82.
  • Soares, Nate; Levinstein, Benjamin (2020). "Cheating Death in Damascus" (PDF). The Journal of Philosophy. 117 (5): 237–266. doi:10.5840/jphil2020117516.
  • Yudkowsky, Eliezer; Soares, Nate (2025). If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown and Company. ISBN 9780316595643.

References

  1. ↑ Spirlet, Thibault (2025-09-27). "Superintelligence could wipe us out if we rush into it — but humanity can still pull back, a top AI safety expert says". Business Insider. Archived from the original on January 15, 2026. Retrieved 2026-01-22.
  2. ↑ Waters, Richard (2014-10-31). "Artificial intelligence: machine v man". Financial Times. Archived from the original on March 6, 2025. Retrieved 2026-01-22.
  3. ↑ Yudkowsky, Eliezer; Soares, Nate. "Shutdown Buttons and Corrigibility". If Anyone Builds It, Everyone Dies Online Resources. Archived from the original on December 6, 2025. Retrieved January 21, 2026. We named this broad research problem 'corrigibility,' in the 2014 paper that also introduced the term 'AI alignment problem' (which had previously been called the 'friendly AI problem' by us and the 'control problem' by others).
  4. ↑ Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer; Armstrong, Stuart (2015). "Corrigibility" (PDF). AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, January 25–26, 2015. AAAI Publications. pp. 74–82. Archived from the original (PDF) on January 1, 2026. Retrieved January 22, 2026.
  5. ↑ Whipple, Tom (2025-09-12). "AI — it's going to kill us all". The Times. Archived from the original on November 22, 2025. Retrieved 2026-01-22.
  6. ↑ Canfield, Kevin (2025-09-02). "'Everyone, everywhere on Earth, will die': Why 2 new books on AI foretell doom". San Francisco Chronicle. Archived from the original on 2025-12-29. Retrieved 2026-01-22.
  7. ↑ Louallen, Doc (2025-09-19). "New book claims superintelligent AI development is racing toward global catastrophe". ABC News. Archived from the original on 2026-01-06. Retrieved 2026-01-22.
  8. ↑ Roose, Kevin (2025-09-12). "A.I.'s Prophet of Doom Wants to Shut It All Down". The New York Times. Archived from the original on 2025-09-12. Retrieved 2026-01-22.
  9. ↑ Bordelon, Brendan (2025-10-01). "Why D.C. should be way more worried about AI". Politico. Retrieved 2026-01-22. In his view, companies like OpenAI and Anthropic are racing toward the profoundly dangerous goal of artificial superintelligence — an AI that far outstrips the capabilities of any one human, or even all humans combined. Once that comes into existence, Soares argues, it will be impossible to control[....] Soares sketched out a policy vision to stave off humanity's demise. His ask is in one sense simple: a global ban on advanced AI research.
  10. ↑ Shariatmadari, David (2025-09-22). "If Anyone Builds It, Everyone Dies review – how AI could kill us all". The Guardian. Retrieved 2026-01-22. …as clear as its conclusions are hard to swallow.
  11. ↑ Byron, Grace (2025-09-28). "Could AI be a truly apocalyptic threat? These writers think so". The Washington Post. Retrieved 2026-01-22.
  12. ↑ "Briefly Noted". The New Yorker. 2025-10-13. Retrieved 2026-01-22.
  13. ↑ "The Doomers Who Insist AI Will Kill Us All". Wired. 2025-09-05. Retrieved 2026-01-22.
  14. ↑ "Researchers give doomsday warning about building AI too fast". Semafor. 2025-09-12. Retrieved 2026-01-22.
  15. ↑ "Bay Area researchers argue that tech industry is 'careening toward disaster'". SFGATE. 2025-09-24. Retrieved 2026-01-22.
  16. ↑ "News of Area Students". White River Valley Herald. 2011-05-19. Retrieved 2026-01-26.
  17. ↑ Soares, Nate. "Nathaniel Soares" (PDF). Résumé. Archived (PDF) from the original on 14 September 2024. Retrieved 2026-01-26.
  18. ↑ "If Anyone Builds It, Everyone Dies: How Artificial Superintelligence Might Wipe Out Our Entire Species". The Great Simplification (Podcast). 3 December 2025. Retrieved 27 January 2026.
  19. 1 2 O'Connell, Mark (2017-02-28). "If Silicon Valley Types Are Scared of A.I., Should We Be?". Slate. Archived from the original on March 5, 2019. Retrieved 2026-01-27.
  20. 1 2 Russell, Stuart; Dewey, Daniel; Tegmark, Max (2015). "Research Priorities for Robust and Beneficial Artificial Intelligence" (PDF). AI Magazine. 36 (4). Association for the Advancement of Artificial Intelligence: 105–114. doi:10.5840/jphil2020117516.
  21. ↑ Soares, Nate; Fallenstein, Benya (2017). "Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda" (PDF). In Callaghan, Victor; Miller, James; Yampolskiy, Roman; Armstrong, Stuart (eds.). The Technological Singularity. Springer. ISBN 9783662540312.
  22. ↑ Gallagher, Brian (2018-03-12). "Scary AI Is More "Fantasia" Than "Terminator"". Nautilus. Archived from the original on December 14, 2025. Retrieved 2026-01-28.
  23. ↑ Chivers, Tom (2018-07-16). "How Disney shows an AI apocalypse is possible". UnHerd. Archived from the original on January 3, 2026. Retrieved 2026-01-28.
  24. 1 2 Duleba, Gretta (2023-10-10). "Announcing MIRI's new CEO and leadership team". MIRI Updates. Machine Intelligence Research Institute. Archived from the original on January 4, 2025. Retrieved 2026-01-28.
  25. ↑ Gold, Ashley; Curi, Maria (2023-12-05). "Senate AI forums wrap for 2023 this week". Axios. Retrieved 2026-01-28.