Yuval Noah

tendency to summon powers we cannot control stems not from individual psychology

never summon powers you cannot control.

Today one might hope that AI could provide such a mechanism, as when in April 2023 Elon Musk announced, “I’m going to start something, which I call TruthGPT or a maximum truth-seeking AI that tries to understand the nature of the universe.”[2] We will see in later chapters why this is a dangerous fantasy. In previous eras, such fantasies took a different form—religion.

The lesson to the apprentice—and to humanity—is clear: never summon powers you cannot control.

The main argument of this book is that humankind gains enormous power by building large networks of cooperation, but the way these networks are built predisposes us to use that power unwisely. Our problem, then, is a network problem.

unchecked AI advancement could culminate in a large-scale loss of life and the biosphere, and the marginalization or even extinction of humanity

In the next few decades, it will likely gain the ability even to create new life-forms, either by writing genetic code or by inventing an inorganic code animating inorganic entities.

never summon powers you cannot control.

Why are we so good at accumulating more information and power, but far less successful at acquiring wisdom?

The tendency to create powerful things with unintended consequences started not with the invention of the steam engine or AI but with the invention of religion. Prophets and theologians have summoned powerful spirits that were supposed to bring love and joy but occasionally ended up flooding the world with blood