Nick Bostrom's Superintelligence is a book that imagines how we should go about dealing with a super-AI, should it come about. The thesis of the book seems to be this: if a superintelligence were to be constructed, there would be certain dangers we'd want to apprise ourselves of and prepare ourselves for, and the book is a precis, essentially, for dealing with some of those risks. Assuming, for the sake of argument, that the thesis of the book is correct, what is of interest to me is how a superintelligence could be constructed. If someone wanted to construct a superintelligence, it seems to me they'd have to understand human intelligence at a deep level, but I doubt we'll ever come to understand how intelligence works.
Bostrom defines a superintelligence as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.” He believes that a superintelligence is on the way, writing that it'll probably arrive within this century. He also thinks that the different forms that superintelligence could take are practically equivalent, and they follow into these general categories: speed superintelligence, collective superintelligence, and quality superintelligence. Speed superintelligence is a “system that can do all that a human intellect can do, but much faster.” Collective superintelligence is a “system composed of a large number of smaller intellects such that the system's overall performance across many very general domains vastly outstrips that of any current cognitive system.” Quality superintelligence is a “system that is at least as fast as a human mind and vastly qualitatively smarter.” In other words, a superintelligence could either be very intelligent across the domains we care about because it's really fast or because it works really well with the subsystems that compose it, or because it could just works better within those domains than we do. On Bostrom's formulation, superintelligence is a general concept, to be amended later, and Bostrom assumes right now we don't have to spell it out too much because we'll know a superintelligence when we see it. But what of intelligence?
There's no currently universally accepted understanding for what constitutes intelligence, but it seems to be something like knowing how to do something relative to some domain. Even a concept like general intelligence as measured by an intelligence assessment can be broken down to checking for knowledge in specific domains, often assessing some combination of (1) logical and mathematical knowledge, (2) linguistic knowledge, and (3) visual and spatial knowledge. We might also add other domains of interest, those domains approximating what Howard Gardner means when he writes of intelligences, plural, or Steven Pinker when he refers to mental modules. These domains of interest might include knowledge of (4) self and others, (5) music and rhythm, (6) motion, (7) morality, (8) how the world works. These domains would bear a few qualities: (a) they could be found universally across human beings and (b) abilities with respect to these domains could vary widely from individual to individual, but (c) these abilities would ultimately follow the law of distribution—that is, most people will be average with respect to use of such knowledge and few will be exceptionally low or high in ability. If these were to be the domains of interest and they'd have these sorts of features, then anything that outperforms in all these domains would constitute a superintelligence.
The only way to know how intelligence works is to know how the subsystems work. And although there's progress in these domains, we don't have anything like a rich explanatory framework for these subsystems such that we could recreate them—apart from having a child (and even then we don't know how their subsystems work, otherwise it might be much easier raising a child). Bostrom acknlowledges that programming an entity with an intelligence is a seriously difficult feat on the engineering side of things:
[A]ccomplishing even the simplest visual task—finding the pepper jar in the kitchen—requires a tremendous amount of computational work. From a noisy time series of two-dimensional pattenrs of nerve firings, originating in the retina and conveyed to the brain via the optic nerve, the visual cortex must work backwards to reconstruct an interpreted three-dimensional representation of external space. A sizeable portion of our precious one square meter of cortical real estate is zones for processing visual information, and as you are reading this book, billions of neurons are working ceaseless to accomplish this task (like so many seamstresses, bent over their sewing machines in a sweatshop, sewing and re-sewing a giant quilt many times a second).
Besides showing what a great science writer Bostrom is, this passage also suggests the immense difficulty of visual programming. He writes, “The main reason why progress has been slower than expected is that the technical difficulties of constructing intelligent machines have proved greater than the pioneers foresaw." But he's optimistic, continuing: "But this leave open just how great those difficulties are and how far we now are from overcoming them. Sometimes a problem that looks hopelessly complicated turns out to have a surprisingly simple solution (though the reverse is probably more common).”
Let's assume that it's correct, that we have those eight subsystems I listed above, and probably more. If we're also to assume that the subsystems operate computationally, we'd have to try to go about figuring out how the algorithms are carried out in the physical plumbing of the brain, assuming also we think these operations are taking place at least mainly in the brain. Some headway has been made on linguistic knowledge through the likes of Noam Chomsky et al., as well as progress with vision through the likes of David Marr. Certain forms of symbolic logic and axiomatic systems have been created to account for logical-mathematical knowledge, just to name a few examples. But to build an artificial intelligence proper, if the goal is to mirror something like the way human beings think and behave, you'd have to program it with principles or algorithms that approximate to those that human beings use when they think and behave. Nothing has been developed yet, however, that could be deemed to be anything like that in terms of a rich computational understanding. Nobody has been able to reduce Psychology, for instance, to a handful of principles such that we could program them into a computer nor has there been any deep investigation of the principles that govern our moral knowledge apart from the creation of a taxonomy of what moral concepts are activated in certain ecologies.
My own view is not that I think there couldn't be such covering laws or principles, but that human inquiry is inherently limited such that we can't even investigate some domains properly. Some things we can conceive of but have no way of knowing what they would look like. For instance, we're pretty good at imagining three spatial dimensions—forward-backward, up-down, left-right—and we could even imagine a fourth: clockwise-counterclockwise. But try to imagine a fifth spatial dimension. Of course, it's just my hedge that we might be near the end of human inquiry in some of these domains. Smart people like Bostrom, on the other hand, are hopeful. And maybe he's right. Maybe it's not the end of inquiry. Maybe the task just requires people who are more intelligent in one of those domains. But as for me, I definitely don't foresee progress in those domains to the point we could imagine how they would integrate and we'd be able to program an AI with them.