TSIN and questions of emphasis

The release of Ray Kurzweil's The Singularity is Near seems to be generating a fair amount of press. I received a pre-release copy in the mail from the SIAI (for being a nontrivial donor) a couple of weeks back. Although I haven't had time to read the book properly, I expect my general position on this is similar to that of most specialists reviewing a popular science book on their field. That is to say, I'm trying to decide whether the progress made in introducing key concepts to a wider audience is worth the distortions and oversimplifications in the book and resulting misunderstandings.

I also expect that most of what I would say on the subject has been or will be said by a reviewer somewhere. So I will restrict myself to a single point which I think is particularly important.

The Singularity concept, which is to say the idea of a 'predictive horizon' and the general notion that 'life as we know it' is going to evaporate sometime soon, is based on accelerating change in four different domains;

• Tools, i.e. technology.
• Knowledge, i.e. science.
• Goals, or rather basic motivational mechanisms, i.e. 'human nature'.
• Intelligence; raw ability to predict outcomes and achieve goals.

These are listed in ascending order of power and decreasing order of specificity and comprehensibility. The real problem with most serious Singularity debate, including Kurzweil's book, is that the authors focus on technology while skimming over knowledge and paying lip service to goals and intelligence. People talk about 'superintelligent computers', and then proceed to waste time arguing about Moore's law, Internet take-up rates and silly graphs of 'technological progress'. The basic reason why people do this is because it's easier, more fun and more familiar than trying to debate poorly defined and poorly understood questions such is 'what is intelligence'. It's also true that the actual implementation of the Singularity will be a recursive loop that has technological progress as the most visible, comprehensible and dramatic-to-write-about stage; goals and intelligence drive the acquisition of knowledge which drives the development of technology which permits greater intelligence and new cognitive architectures.

But when it comes down to it, technology alone doesn't cause the Singularity effect. The reason science fiction is a mass market form of entertainment is that while we can't reliably predict what cool toys will be invented, given a description of what a device can do we have a fair chance of guessing how it will be used and what a society that has access to that device will be like. Certainly we can understand a description of such a society, primarily because there are recurring patterns of human behavior and social organisation that are present (in different mixes and shades) regardless of technological level. Technology has a vast power to make our lives better, but a relatively weak power to render them incomprehensible.

Knowledge has a greater impact than technology alone because it changes how people model the world, and thus how they perceive and react to situations and ultimately how they behave. Both science and culture fit this description, though obviously the former is much more directly linked to technology. It is fundamentally harder to understand someone who has a different world view to yours than it is to understand someone who has access to a technology you don't have. The incomprehenisibility level of more advanced technology quickly plateaus at the level of 'magic' (in the Clarke sense), and people seem to have surprisingly little difficultly predicting what a given human would do given a specific magical power. Meanwhile something as trivial as having a different religion can cause vast gulfs of incomprehension, while serious differences in world view that involve many inferential steps (i.e. medieval alchemy compared to modern biochemistry) are unbridgeable without years or decades of serious effort. Even at the same level of general education, novices in any field are usually unable to predict how an expert would solve a given problem.

Both of the above cases are well within the realm of human experience. Differences in cognitive architecture are largely beyond it. The brains and mind set of humans with serious mental disorders are still very similar to other humans in absolute terms, yet the difference is enough to cause vast difficulties in comprehension. We have severe difficulties in understanding the minds and behavior of animals despite the fact that they lack general intelligence, are considerably simpler than our own minds and still have a relatively similar neural substrate, gross brain architecture, environment and instincts. Only the best sci-fi authors can begin to conceive and describe a truly alien mind (the vast majority of sci-fi aliens are humans with some anatomical differences, different relative strengths of the standard human emotion set and a novel seeming culture), and accurately predicting the structure of a whole society of such minds is more or less impossible. Thus any technique that seriously modifies human nature, much less the creation of de novo artificial intelligences, will create a sharp predictive horizon unless exceptional techniques (i.e. SIAI FAI theory) are used to prevent it.

Finally we come to intelligence itself. Far too many people think of differences in intelligence among existing humans and try to write this off, though even that can produce quite dramatic examples of incomprehensibility (ignoring problems with IQ as a measure, how successful would the average IQ 70 human be at predicting how an IQ 140 human handles a problem at the limit of the latter's ability?). The actual situation is that humans are just above the bare minimum threshold for general intelligence, achieving competence by making the best use we can of a mixed bag of innate capabilities that each contributed to reproductive fitness at some point in our evolutionary history. We already know that the space of physically possible intelligent entities includes intelligences vastly beyond our capabilities, by many orders of magnitude on any scale you care to name, and functional cognitive architectures we find incredibly hard to even envision. Some have stressed that the actual metaphor for comparing a human to a (moderately) transhuman AI should be comparing a dog to a human, but even that is misleading because most of the salient difference is the addition of general intelligence, not raw reasoning capability. Humans simply don't have any experience dealing with general intelligences orders of magnitude more powerful than our own, and it takes serious abstract thought to even understand the depth of our ignorance in this regard.

So in summary, it takes a great deal of very rapid technological progress to create a moderately impeneterable predictive horizon, a large amount of radical cultural or scientific progress to create a pretty serious one, a moderate amount of tinkering with human nature (or creating an infrahuman AGI, or uplifting dolphins...) to create a sharp predictive horizon, and the existence of a single transhuman AGI that is a mere order of magnitude smarter than humans to render the future utterly impenetrable to passive analysis. /That/ is the power of the Singularity; not Moore's law, not Internet culture, which are important but only as means to an end. It is transhuman intelligence and to a lesser extent alien cognitive architectures that create the unknowability at the core of the Singularity, the evaporation of life as we know it and the incredible risks and potential benefits.

This is why I still recommend Eliezer Yudkowsky's Staring Into The Singularity as a primer (and generally essential reading), despite it being years out of date and wrong in places; it is one of the very few summaries of the Singularity concept that nails transhuman intelligence as the key factor. And it is the possibility and extreme desirability of influencing that transition towards benevolent ends that has caused me to lend my support to the SIAI, and devote my life to working on the FAI problem.