Artificial Intelligence as a positive and negative factor in global risk

needed to address that challenge

scientific ignorance is hallowed by ancientness. Each and every absence of knowledge dates back to the dawn of human curiosity; and the hole lasts through the ages, seemingly eternal, right up until someone fills it.

Homo sapiens represents the first general intelligence.

suspect that, pragmatically speaking, our alternatives boil down to becoming smarter or becoming extinct.

And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI. Those two statements are not at all equivalent!

If you throw too much money at a problem that a field is not prepared to solve, the excess money is more likely to produce anti-science than science

If we want people who can make progress on Friendly AI, then they have to start training themselves, full-time, years before they are urgently needed.

there is no privileged reason why AIs would slowly climb all the way up the scale of intelligence, and then halt forever exactly on the human dot

The nightmare scenario is that we find ourselves stuck with a catalog of mature, powerful, publicly available AI techniques which combine to yield non-Friendly AI, but which cannot be used to build Friendly AI without redoing the last three decades of AI work from scratch.

The perception I have encountered is that anyone who claims to be researching Friendly AI is implicitly claiming that their AI design is powerful enough that it needs to be Friendly.
It should be obvious that this is neither logically true, nor practically a good philosophy. If we imagine someone creating an actual, mature AI which is powerful enough that it needs to be Friendly, and moreover, as is our desired outcome, this AI really is Friendly, then someone must have been working on Friendly AI for years and years.