On 12/7/14, Dylan and I had a conversation with Oxford philosopher Nick Bostrom about his new book Superintelligence: Paths, Dangers, Strategies
Nick's Future of Humanity Institute studies "existential threats" (a la our past discussion with David Brin), and he presents a model (e.g. in this 1997 paper that we discussed a bit) of philosophical work that is continuous with but with a more general scope than science. We previously discussed (none too charitably) one of his papers on genetically or mechanically enhancing human capacities, and Superintelligence does treat that subject briefly, but the focus is on artificial intelligence and the threat that it poses if engineered improperly.
You can hear (and watch) Nick explain the project here, but here's the run-down: There are no in-principle reasons preventing us from developing an artificial intelligence that's as smart as we are (whether or not it really has consciousness or any other uniquely human trait is beside the point; "intelligence" just refers to means-end strategic ability here), and once we do that, whatever its programmed goals, it will have reason to work to enhance its own abilities, meaning that it's only a matter of time before there's an AI out there that's MUCH smarter than we are. Nick doesn't predict when this will happen, but thinks it more or less inevitable given current research programs.
What motivations will this AI have? Obviously that depends on how it's programmed, but whatever its goals, it will have an instrumental interest in enhancing itself, and also in maintaining goal integrity, i.e. preventing us whatever its goals are from being changed.
Nick thinks that unless strenuous effort is put into carefully defining these goals BEFORE FULL INTELLIGENCE IS ACHIEVED, then we're in trouble. Researchers trying to engineer machine intelligence would more likely than not set just any old goal, e.g. coming up with as many digits of pi as possible or building as many paper clips as possible, as their focus would be to get the machine to innovate and learn in coming up with ways to achieve that goal. But if they succeed, then goal integrity means that we're stuck with a super-intelligence who will now think "outside the box" to take whatever steps it deems necessary to meet its goal, e.g. converting all the matter on earth into paper clip material or computing material to calculate digits of pi. Even if its goal is not infinite, e.g. "manufacture 100 paper clips," there's always more action it can try to take to increase the certainty that it has in fact performed this goal, i.e. go ahead and make the universe into paper clips.
So the primary problem to sole here is what Nick calls "the control problem," which involves measures to either prevent a potential superintelligence from being able to ruin the world (e.g. crippling or containing it) or, more directly, come up with ways to set its motivation in ways that we would find acceptable. This gives rise to many problems more familiar to philosophers: e.g. if you want to tell it to refrain from actions that aren't in our interest, then you have to both figure out what our interest is and figure out how to state this unambiguously. For example, if we tell it to maximize human happiness, it's likely to work to rewire us so that only our pleasure centers function and everything else is removed, if this proves the most efficient way to keep us happy, or better yet, kill us all and replace us with creatures that are easier to keep happy.
Moreover, Nick thinks that we can in effect outsource our more thorny philosophical problems to this much greater intelligence if we can tell it to "take whatever actions we would ask you to take if we were better informed and thought about it for long enough." All this is of course connected with the kind of ethical theorizing that philosophers are familiar with.
The story told, we're faced with a number of issues: Is this something that philosophers have any business messing with, or on the contrary is this a real and pressing enough existential threat that it's much more worth our time than just about anything else? Do we buy the control problem as legitimate, or can we deny that goal integrity would be a central feature of AI motivation? Are we in any position to make the kinds of predictions and analyses Nick makes in this book?
You can hear Nick on a number of other podcasts talking both about this topic and about existential threats more generally. Here are a few from Oxford, or you can just search on his name under podcasts or iTunes U and find plenty of them. We did our best in talking to him this time to engineer an interactive conversation so he's not simply repeating what he had to say in these plentiful previous media appearances.
http://www.geektime.com/2014/12/01/how-intelligent-is-artificial-intelligence-we-ask-the-big-thinkers/
My favorite quote from that article comes from Hofstadter: “We have already seen flash crashes in the stock market, for example, induced by machines with no will and little ability to think with the flexibility of human cognition. Computers can already cause chaos if they are mis-programmed.”
This seems pretty similar to some of Bostrom’s reasoning—bad effects are just mistakes, the mistakes are just human misapplications of the correct technology, there’s inevitably a correct way to program these mistakes away, etc. But I like Hofstadter pointing out that AI doesn’t need to get super smart to wreak havoc in the human world (Weizenbaum’s Eliza). The media generally frames the issue as choice between AI the existential threat or AI humankind’s salvation. Lanier also has some sharp observations as usual.
http://www.motherjones.com/politics/2013/08/eric-schlosser-command-control-excerpt-nuclear-weapons
The problem with the weak-AI super-intelligence holocaust scenario is the same as all the other singularity-related scenarios; unsupported conspicuous optimism. The idea that we will somehow survive long enough to be wiped out by our own software seems to be an extraordinary and dubious claim – I can name twenty or thirty other research program-slash-existential threats that will do us all in long before any of this science fiction stuff gets to take a whack at us.
I think it’s fine however that philosophers consider this stuff; if it sells books or monetizes websites, it’s a solid investment of time or whatever.
The multitude of ways these technologies could go wrong seem obvious and imminent, but with many of them, it’s not clear to me what getting it right would mean. Curing cancer, aging or death with no downside? Well, the planet is already overrun with people, many of whom are tearing each other apart to acquire conditions that others are willing to wager against an existential threat in the name of progress. It doesn’t exactly inspire confidence in the satiation of our desire, but listening to Bostrom in the C-SPAN video, you might think we could roughly chart our standard of living by iphone roll outs. If I live long enough to see one of his graphs shoot vertical, what are my prospects? A Jupiter sized brain, or being uploaded to a computer so that I can “live” forever as software? My dreams fulfilled.
Even if Bostrom and friends’ predictions are on the right track, and they could somehow come up with a miraculous set of protocols for keeping “superintelligence” on the straight and narrow, what’s the likelihood that these would be used in the necessary 100% of cases? Very unlikely, I would think. Why should we expect some international committee on programming ethics to fare better than the Geneva Conventions?
This is not exactly on topic, but I must note the ease with which Mark (but by no means only Mark) has inadvertently transformed human beings into machines. When talking of the AI control problem, he writes: “…if we tell it [the artificial super-intelligence] to maximize human happiness, it’s likely to work to rewire us so that only our pleasure centers function and everything else is removed…” Here the human being is metaphorically (or perhaps not so metaphorically) understood as a complex organization of wires and functions, some sort of biologically evolved computer. Why should this by now trite figure of speech bother me? For the very reason that it HAS become trite, so trite, in fact, that it’s now nearly a dead (invisible) metaphor, increasingly used as if literally true. This habit of thinking of ourselves as computers originates, of course, in a vast amount of cognitive-scientific research whose argot and reductive ways of thinking have filtered down into the common parlance and imagination. Much of this research adopts the computer as at once the heuristic model of what we humans in fact are (in perishable wet form) and as the literal prototype of what we can eventually become (in more durable dry form). I imagine that as we come more and more to talk and think of ourselves in this way, and as this self-image undergoes increasing reinforcement by the cog-sci industry, we will eventually build a computer that matches this self-conception, and we will be so impressed with it that we will rashly conclude we’ve successfully replicated ourselves in non-biological form. In other words, we will forget what it means to be human because we will no longer be able to talk about it. From there, it will seem very desirable to reduce our personal identities to code and thus migrate from these bodies of death to a world of digital immortality. All we will actually have done, however, is built a machine that mirrors our distorted self-image, and, Narcissus like, fallen into it and drowned.
First, I was more or less quoting the way it’s talked about in the book, so the turn of phrase doesn’t necessarily reflect my thinking.
Second, I think that the threat that routine speech patterns pose against out-of-the-box thinking is overblown. One can, e.g., have fully in mind feminine equality while still using the linguistic convention of “his” for the generic singular possessive pronoun… not that I mind “his or her” or simply “her” or (a recent grammatical innovation meant to side-step the issue) “their” either, but for in particular a metaphor like “brain wiring” I think it obvious that it IS a metaphor and not to be taken as a literal diminution of man into machine with the evil effects that would entail, at least not for anyone even the least bit concerned with philosophy and/or self-reflection.
However, interestingly, one of the side issues in the Superintelligence book is the issue of uploading people, and how Bostrom (like Brin) thought that experientially, the uploaded persons would definitely consider themselves people and moreover the SAME people as those who had been uploaded, so, therefore, for all practical purposes those uploads will BE us. Since the point wasn’t central to the issues discussed in my summary here, we didn’t go into this, but you’ve certainly pointed out a key point of disagreement that deserves to be tackled in some other discussion.
So this is supposed to be different, I think, than the argument about teleportation. In that case, the end result is a molecule-to-molecule biological replica of my old self, which at least shouldn’t have a problem being classified as a human person, whether or not you want to agree with its assessment that it is still the same person that stepped into the teleporter at the other end (the latter problem made manifest if the “before” version isn’t destroyed in the process, so that there ends up being two copies of the same person). However, in sci-fi, i.e. via the experiential argument, the case for self-identity is the same for teleportation and for uploading. So if you want to argue that they’re different, and the uploaded person isn’t even a person, then you have to come up with a stronger (Searlian or Dreyfusian or something) argument that the individual person is NOT defined by information patterns (even massively parallel or opaque ones such as explored in neural-net-type computer architectures) that could then be realized in different media, i.e. you have to argue that functionalism is false. We talked about this a little in the Mind episode long ago, but it deserves another look in some future episode. Any recommendations from people re. what to read pro and con for such a discussion?
Though this is a spoiler for the discussion, that was my ending point: I suspect that there’s an in-principle reason why we can confidently say that there can’t be an AI Superintelligence (or even “normal” intelligence) of the type described, but I don’t know what that reason might be, and the versions of the argument against this by Dreyfus are as far as I understand the issue aimed at pre-neural-net types of programming and don’t really impact more current approaches.
A point I should have made but didn’t on the podcast was that I think that a hallmark of human-level-intelligence is the ability to reevaluate ones goals globally. If I’m right, then goal-content-integrity will NOT be a hallmark of superintelligent systems, and any attempt we might make to lock down its motivations will fail (and at the same time, we’re not in danger of our literal programming creating a monster of the type Bostrom fears). In effect, I think that the behavior of such a system is LESS predictable than even Bostrom thinks, so I am personally not drawn to put time into theorizing about it much (much less funneling research dollars into it) merely based on the possibility that he’s right.
http://aima.cs.berkeley.edu/
Thanks, Mark, for your thorough and very lucid response. My point about the prevalence of reductive speech patterns is that they reflect vast cultural changes that are actually taking place, changes that are transforming our understanding of what it means to be human. If the analogy between neurological systems (brains) and information processing systems (computers) gets a sufficient hold on our collective imagination, and assuming that the AI singularity eventually does take place, we will already be culturally and psychologically primed to take the upload leap if it’s presented to us. All the more so if the prospect of digital migration includes escape from personal mortality. What a motivation! This altered self-image, I’m suggesting, will predispose us to embrace uploading as the realization our true nature, our evolutionary telos, our historical fate, our personal destiny. We already feel some uncertainty about whether our gadgets are extensions of us or we’re extensions of them. Give this trend a few more decades, and who knows where we’ll be? There are many people right now who, given the real option to go upload, would say, yeah, let me at it, the sooner the better. My point, in other words, is less about the technological feasibility of uploading, or the soundness of the philosophy of mind that supports it, than about our moral preparedness for it. My further point is that super AI technology, if and when it does erupt, will be of such a very high order and so completely beyond our mere human capacity to evaluate that we won’t know whether the upload prospect is real or not. We’ll simply have to accept it on faith. “The super AI says it’s ready for upload. Are you?” I recoil in horror. Others do not. What is the difference here?
http://www.wsj.com/articles/artificial-intelligence-isnt-a-threatyet-1418328453
Let’s be brutally honest. Nick Bostrom is not a philosopher of any worth. His “arguments” are really bad without exceptions (e.g. the simulation argument is worse than even any purpoted proof of god by medieval philosophers). He just seeks attention by drawing seemingly shocking conclusions which are not at all supported by any solid arguments. Very disappointed by the choice of this charlatan as featured guest.
He’s a professor of philosophy at oxford, I think that’s good enough warrant to be a guest. I suspect that his actual philosophical output (in journals) is much better than his public contributions.
Not sure that being a professor is an automatic proof of quality. Could he be perhaps externally funded? I just went through the list of his publications and don’t find anything that looks “serious” philosophy. Looks more like a media attention monger, peddling half-baked singularity, transhumanism bs.
Why is the simulation argument bad? I suspect you have strong biased towards materialism and haven’t considered the potential alternate definitions of simulation. For me the language naturally extrapolates into “Intelligent beings that are capable of creating universes, would indeed create universes. Our universe at the very least contains beings that are almost capable of creating a universe. Therefore our universe was created by an intelligent being.”
> I think that the threat that routine speech patterns pose against out-of-the-box thinking is overblown
Between analysis of language and accusing one another of privilege, what else will the left have to do?
> functionalism is false
Functionalism is false about everything else in the universe, i’d be odd if it were true about consciousness (eg. only water is watery, not carbon arranged water-wise).
> I think that the behavior of such a system is LESS predictable
This is the key point. Any system sufficiently complex will be mysterious to its inventor, not “known” in the simple mechanical sense of the 19th C. and before. I don’t see us being able to formulate the solution to the problem of conscious AI enough to come up with a system to produce it, and I dont see non-conscious systems as being “not dangerous” in any ways conscious ones would be. Indeed, adding consciousness to the mix just means you can play a scifi sound track over your 5min TV documentary slot.
> Functionalism is false about everything else in the universe, i’d be odd if it were true about consciousness (eg. only water is watery, not carbon arranged water-wise).
Functionalism is true about computation. A computer can be made out of anything that can carry out certain abstract processes, and leaving aside practical considerations of speed, size, and cost, that includes a vast array of stuff. Instead of the binary 1/0 state being instantiated by a transistor, it could in principle be instantiated by a wood fire that is either burning or extinguished, a bucket of water that is full or empty, a bird that is singing or quiet, a mousetrap that is set or sprung, etc., etc., etc. Functionalism is also true about static states of information. The information that constitutes the phrase “To be or not to be” can be manifested in ink, in pixels, in skywriting, in a pattern drawn in the snow, in an ignited pattern of poured gasoline, etc., etc., etc.
The claim made by functionalists is typically that consciousness is computation — a particular kind of computation that our brains do. So, it follows that since functionalism is true about computation, it is also true about consciousness. (People often get hung up on irrelevant dissimilarities between brains’ biological computation and the computation performed by human-made electronic computers. The contemporary concept of computation is broad enough to encompass more than electronic machines with a Von Neumann architecture. Physicists talk about the computation performed by black holes. Biologists talk about the computation performed by cell membranes. Computation isn’t defined according to the contingent features of the electronic computers invented last century.)
The difference between a carbon atom and a water molecule — two hydrogen atoms bonded to an oxygen atom — is not some essential, innate characteristic of carbon or water. It’s simply the arrangement of their respective parts. A carbon atom is simply 6 protons and 6 neutrons (which are themselves made of quarks) orbited by 6 electrons. A hydrogen atom is 1 proton orbited by 1 electron. An oxygen atom is 8 protons, 8 neutrons, and 8 electrons. Water is simply subatomic particles arranged water-wise, and carbon is simply subatomic particles arranged carbon-wise.
Why would only things made out of carbon be capable of consciousness? Only if by some strange magic putting 6 protons, 6 neutrons, and 6 electrons together somehow made the difference between consciousness and unconsciousness.
Two separable issues here:
1. Why did we pick Nick and this book? There’s a long chain here. I remain OK with choosing David Brin previously and only bemoan the fact that he wasn’t more interactive, not that he didn’t have lots of philosophically interesting ideas (and is by any objective measure some sort of genius). That led to the follow-up transhumanism episode (a topic requested by multiple listeners, so I figured what the hell), for which we took suggestions from our listeners through the Facebook group, and multiple people said that Nick was the guy for that. We covered him half-assedly in that episode, and I reached out to Nick via email to alert him and sort of apologize. He said “well, then why don’t you have me on the show,” and his publisher (assistant?) followed up offering to send us free copies of the new book. Only Dylan and I were interested, but given Luke’s (baffling to me at the time) devotion to this topic, and the fact that it was also covered by David Chalmers (another guy who’s really obviously a genius whether you agree with his theories or not, and with more conventional philosophical bona fides), I figured it was a nice bonus episode for us to record (you should know that ep. 109 on Karl Jaspers is already now recorded too). Now, usually we like to keep our author-guests to “big fish” (i.e. people who are already pretty damn famous in some way, though we break this rule on occasion when one of us especially wants to), and I was feeling funny about covering Nick on those grounds, but as I researched him as we approached the date, I discovered, e.g. that he’s actually right now one of the most influential philosophers on the planet, e.g. check out this list of thinkers (just a readers’ poll) that puts him at #15 just below Zizek and above Dennett): http://www.prospectmagazine.co.uk/features/world-thinkers-2014-the-results. Per our previous coverage of Ayn Rand, if someone is making a big splash and his or her work is arguably philosophy, then I feel it’s within our purview to cover, even if the result is that you think it full of crap.
2. As with Brin and Chalmers (and Dennett, whom Wes considers a charlatan, and Zizek, who is sometimes accused of the same thing, and Derrida for that matter), Nick is obviously a genius in measurable terms. He’s not a friggin’ Internet columnist or commentator like me, but a guy with a lot of exhaustively researched (that was our complaint about the transhumanism paper… too many footnotes, too much interaction with “the literature”) material who puts himself out in the world and jumps through hoops enough to impress the various philosophical gate-keepers so that he did get himself this unique foundation at one of the world’s foremost universities (i.e. a cushy job where he doesn’t have to teach) and is asked to speak all over the world, etc. etc. etc. “The Simulation Argument” is a topic for a full post that I’m not going into, but I think he’s very clear that it doesn’t mean what some news outlets say it means, i.e. that we really are likely living in a simulation. (http://www.simulation-argument.com/faq.html) At the very least, that little puzzle is a fun 15 cent toy for philosophers out of the machine in the entry chamber of the Walgreens, given that Descartes’s skeptical challenge is still one of the first things we ever learn about.
So this is not about Bostrom being an idiot or actually a faker (e.g. someone who’s discovered a revolutionary new way to lose weight and wrote 10 books and a 5-CD set to sell you about it), but about whether what he’s doing is legitimate philosophy or not, and it was the goal of this discussion to give people enough to decide exactly that. Philosophers engage in speculation. So instead of speculating about metaphysical things like space and time, or meta-ethics, or the grounds of aesthetic judgment, or how ideas cause or don’t cause revolutions, or how self-consciousness relates to brain states, Nick says that the most PRACTICAL kind of speculation we can do, the kind with a deadline, is about things that are likely to kill us, and the way that you do that is to read all that you can about it, including what lots of very non-philosophical scientists have to say, and learn any analytical tools that seem like they’d be helpful in studying it (e.g. game theory, which like many of the conceptual tools of economics people keep telling me I should learn more about even though I kind of suspect that they’re too bullshitty to bother with, but that’s another discussion), have a bunch of colloquia and other communications with people about it, and do your best to write it all down. To me, this seems as stimulating an intellectual plan as any, and though it’s not exactly my cup of tea, I now understand why Luke (a self-educated guy and voracious learner who spent a few years through his podcast surveying a wide swath of philosophy) would basically devote his life to this.
None of this is likely soothing to those that think that there are so many more worthy philosophers that we should interview, but the best way to make one of those alternate choices happen is for you to reach out to said choice, make him or her aware of PEL and excited about interacting with us, and get such a person (or his or her publisher or other representative) in touch with us. Our list of planned guests of that sort is actually not that long, and if it’s a topic that at least one of us is very interested in and a figure that’s already making some sorts of waves (or a figure that’s willing to come on and talk to us about a historical figure instead of his or her own current book), then there’s a good chance it’ll happen w/in six months or so.
Could you get Wes to do a Bill O’Riley pin-head style breakdown of everyone he considers a charlatan?
This was a good response to any concerns/complaints, IMO. I am anxious to hear the group of you discuss
any relevant issues of potential threats from new technology–albeit, primarily AI–as my optimism remains
far shorter than that of, say, David Deutsch or Peter Diamandis, though it remains higher than that of
Bostrom, James Barrat, or Martin Rees, the latter certainly being another very important contemporary
mind regarding existential threats.
Two quick suggestions for other guests, though I expect that you are anxious to return to the classics:
I was thinking that the aforementioned David Deutsch would be a good suggestion though I imagine that
he would be much in line–perhaps opposed–with the upcoming Bostrom talk. I’m 2/3’s through his
2011 work The Beginning of Infinity and, while rather drawn out, I would still recommend it for those
interested in the philosophy of science. I am, however, just through the section on the multiverse and I
must confess that I find such arguments as unappealing as the simulation arguments mentioned elsewhere
in this blog, whether as a fabled interpretation of quantum mechanics or otherwise.
Otherwise, I’d love to hear a discussion with James Ladyman, a philosopher of science who recently put
out the book Every Thing Must Go: Metaphysics Naturalized (yes, two words up front), co-written with Don
Ross. Like most seemingly interesting ideas, I agree with some of his ideas of structural realism while
disagreeing with others; such is probably a healthy reaction but at times it is quite frustrating…perhaps
a head injury might cure me. Anywho, he is quite interesting and there are a few talks of his available
at youtube as well as a Rationally Speaking interview with him (Massimo Pigliucci & Julia Galef).
’nuff said. keep on keepin’ on
http://www.technologyreview.com/view/532156/googles-secretive-deepmind-startup-unveils-a-neural-turing-machine/
Putting aside my aversion to Nick Bostrom’s outlook, I’m thrilled that you guys are doing episodes like this (especially since you guys never got around to either a potential philosophy of technology or media ecology episode, which I think were both brought up at one point or another).
I hope we’ll continue to see more philosophers take seriously the issues surrounding the practical uses of and possibilities for novel technologies like AI, biotech, information systems, etc. We need minds better equipped than technologists’ to recognize the ethical ramifications, and we need more impressions circulating in the public than just those put forward by marketers and governments. The discussion is pertinent:
http://www.roughtype.com/?p=3371
http://www.pdcnet.org/techne/free
I’m excited. This topic is in the news pretty often, and has a solid hold on the public’s imagination (if box office receipts are any indication). Comments by Hawking and Elon Musk are enough to make it interesting.
The gap between the Turing/Dennett/et al episode and the Brin episode was so long, I didn’t expect a related topic to come up again for years. And you announced it on Christmas Day. Woohoo! Philosophy of art/beauty episodes are pleasant to listen to (and somewhat interesting) but aren’t ones I get hugely excited over, so this is super welcome.
http://www.psmag.com/navigation/nature-and-technology/googles-philosopher-technology-nature-identity-court-legal-policy-95456/
I think one of the most interesting and underdiscussed aspects of Artificial Intelligence today that was obliquely touched on by the Nick Bostrom episode was the notion that we may have already given birth to AI in the form of our institutions. He discussed it in talking about assigning individually uncontrolled agency to governments. I wonder if you guys have a plan to discuss that type of consciousness, a collective will that is composed of individual agency, a la Walter Wink’s series on “The Powers”, Naming the Powers (1984), Unmasking the Powers (1986), Engaging the Powers (1992), When the Powers Fall (1998), and The Powers that Be (1999).
it’s non-sense to speak literally about “institutions” (or nations or such) as agents , what is the mechanism/means that would wire together (or otherwise encode) the assembled individuals in the way that we might circuits/code in an actual machine?