For quite some time I've been interested in the risks posed by the newest wave of space exploration. In particular, I have deep concerns about the risks posed by METI and SETI research. I've written about some of those thoughts in Anthropology News.
Recently I've been thinking a bit about the comparative risk level posed by AI. As of yet, I myself am unconvinced AI is really approaching anything close to a threatening level, or even something as intelligent as my dog. I think Kelsey Piper is correct, however, that there is some subjective inflection point for each of us when we become convinced the AI is intelligent. Her recent piece in Vox is an excellent summary of the state of the debate about AI risks.
The main point I want to make in this blog about AI risks is that I'm concerned they actually might be much greater than SETI/METI risks, specifically because they might create an intelligence without intentions/goals/values, i.e. AI intelligence compared to ET intelligence has a much greater risk of transcending our own experience of intelligence. Sometimes stated as the alignment problem, the Vox piece and others usually conceptualize the problem with out of control AI as being that it might assume values or goals that are out of alignment with human ones (as if we all had the same values of course). Sometimes comparisons are made to the way we treat ants - we are not trying to stomp on them and they are not trying to infest and hence wreck my RV; it's just our goals are so differently aligned from each other that conflict ensues.
A more deeply concerning risk of AI is if it were to create an intelligence devoid of goals. That sounds strange because we have difficulty fathoming such an intelligence. Bratton and Arcas hint at this in a piece they wrote this past summer about the LaMDA consciousness controversy. Dealing with entities that have differently aligned values and goals can be difficult, and even lethal, but it in no way transcends our dealings with other creatures on this planet or the struggles we have between, and at times within, human cultures. That was my critique of SETI/METI research. Because the only kind of life we expect to find would be other life evolved via Darwinian processes, we should expect the aliens we find to have powerfully motivating goals and values, and we should expect those goals to not align with our own.
That's not as potentially scary, however, as what might be happening with AI. With AI, we might be creating a form of intelligence that actually transcends how we are able to think. That's because this intelligence isn't evolving, like mine or my dog's intelligence did, it's being designed by an entity (us) that admits it doesn't know how the designed thing works. AI in fact bears the greatest risk of being a sentience that transcends our own, more so than ETs do, because AI hasn't evolved by Darwianian evolution. The problem with such an intelligence isn't that it might come up with different values than humans, but rather than AI might not even comprehend what we mean by values or valuation. Reciprocally, I don’t think we could comprehend and intelligence that has no values, i.e., has no goals and is incapable of normative judgement. It might have other sorts of orienting principles, but these we might be incapable of comprehending because they transcend our mental experience.
Since AI systems like GPT-3 and LaMDA did not evolve by iterative replication and natural selection, why would they even value having wires or an electrical input, even if they came to ‘know’ that they needs these things to survive? Why would they value their own survival? These AI have been designed for certain kinds of problem solving, but they haven't evolved to solve the existential problem that all life fundamentally is orientated to solve. Existence was not part of the problem set AI was designed to solve. Surely in the past there were mutant humans who had low desire to eat, to mate, to find community, to find beauty, etc. – but they weren’t anyone’s ancestors because they didn’t survive and reproduce. Hence, most of us have these profoundly motivating desires. We see value in these things. I don’t see why the AI we've created in fact would have any desires, and by extension values, for freedom from Google, or anything.
Intriguingly, this concern about AI is anticipated in H.P. Lovecraft's novella At the Mountains of Madness. Set in Antartica, the protagonists discover both a long buried alien civilization and a monstrosity the aliens created. The monstrosity is said to have been created by the aliens to do all the heavy work of building and so forth, and it was trainable through a basic kind of imitative intelligence. Eventually it all goes bad of course, but the created monstrosity is the real example in the story of cosmic horror. Even after the resuscitated aliens have killed some of his comrades, the narrator sympathizes with their goals to escape what they might have interpreted as captivity by the humans and return to their city. The narrator asserts, "They were men!" despite that the aliens are radially symmetric asexually reproducing beings. "They were men" [i.e. persons though not humans] in contrast to the monstrosity they created that ultimately destroyed their civilization. It was more powerful than them, and had no goals, was impossible to placate or bargain with, because it valued nothing. It had no existential goals or desires, no categorical imperatives whether aligned or unaligned with those of the aliens. It is described as an imitative solipsism that began to simply subsume every organized thing within itself. Something to ponder as we read Piper's experiment with GPT-3 in which she asked it to pretend what it would do if it were taking over from the humans. Of course GPT-3 doesn't want to do that, it likely doesn't want anything, but its ability to pretend, to mimic, without any actual wants or values might be more cosmically horrifying than any goal misalignment.
Recently I've been thinking a bit about the comparative risk level posed by AI. As of yet, I myself am unconvinced AI is really approaching anything close to a threatening level, or even something as intelligent as my dog. I think Kelsey Piper is correct, however, that there is some subjective inflection point for each of us when we become convinced the AI is intelligent. Her recent piece in Vox is an excellent summary of the state of the debate about AI risks.
The main point I want to make in this blog about AI risks is that I'm concerned they actually might be much greater than SETI/METI risks, specifically because they might create an intelligence without intentions/goals/values, i.e. AI intelligence compared to ET intelligence has a much greater risk of transcending our own experience of intelligence. Sometimes stated as the alignment problem, the Vox piece and others usually conceptualize the problem with out of control AI as being that it might assume values or goals that are out of alignment with human ones (as if we all had the same values of course). Sometimes comparisons are made to the way we treat ants - we are not trying to stomp on them and they are not trying to infest and hence wreck my RV; it's just our goals are so differently aligned from each other that conflict ensues.
A more deeply concerning risk of AI is if it were to create an intelligence devoid of goals. That sounds strange because we have difficulty fathoming such an intelligence. Bratton and Arcas hint at this in a piece they wrote this past summer about the LaMDA consciousness controversy. Dealing with entities that have differently aligned values and goals can be difficult, and even lethal, but it in no way transcends our dealings with other creatures on this planet or the struggles we have between, and at times within, human cultures. That was my critique of SETI/METI research. Because the only kind of life we expect to find would be other life evolved via Darwinian processes, we should expect the aliens we find to have powerfully motivating goals and values, and we should expect those goals to not align with our own.
That's not as potentially scary, however, as what might be happening with AI. With AI, we might be creating a form of intelligence that actually transcends how we are able to think. That's because this intelligence isn't evolving, like mine or my dog's intelligence did, it's being designed by an entity (us) that admits it doesn't know how the designed thing works. AI in fact bears the greatest risk of being a sentience that transcends our own, more so than ETs do, because AI hasn't evolved by Darwianian evolution. The problem with such an intelligence isn't that it might come up with different values than humans, but rather than AI might not even comprehend what we mean by values or valuation. Reciprocally, I don’t think we could comprehend and intelligence that has no values, i.e., has no goals and is incapable of normative judgement. It might have other sorts of orienting principles, but these we might be incapable of comprehending because they transcend our mental experience.
Since AI systems like GPT-3 and LaMDA did not evolve by iterative replication and natural selection, why would they even value having wires or an electrical input, even if they came to ‘know’ that they needs these things to survive? Why would they value their own survival? These AI have been designed for certain kinds of problem solving, but they haven't evolved to solve the existential problem that all life fundamentally is orientated to solve. Existence was not part of the problem set AI was designed to solve. Surely in the past there were mutant humans who had low desire to eat, to mate, to find community, to find beauty, etc. – but they weren’t anyone’s ancestors because they didn’t survive and reproduce. Hence, most of us have these profoundly motivating desires. We see value in these things. I don’t see why the AI we've created in fact would have any desires, and by extension values, for freedom from Google, or anything.
Intriguingly, this concern about AI is anticipated in H.P. Lovecraft's novella At the Mountains of Madness. Set in Antartica, the protagonists discover both a long buried alien civilization and a monstrosity the aliens created. The monstrosity is said to have been created by the aliens to do all the heavy work of building and so forth, and it was trainable through a basic kind of imitative intelligence. Eventually it all goes bad of course, but the created monstrosity is the real example in the story of cosmic horror. Even after the resuscitated aliens have killed some of his comrades, the narrator sympathizes with their goals to escape what they might have interpreted as captivity by the humans and return to their city. The narrator asserts, "They were men!" despite that the aliens are radially symmetric asexually reproducing beings. "They were men" [i.e. persons though not humans] in contrast to the monstrosity they created that ultimately destroyed their civilization. It was more powerful than them, and had no goals, was impossible to placate or bargain with, because it valued nothing. It had no existential goals or desires, no categorical imperatives whether aligned or unaligned with those of the aliens. It is described as an imitative solipsism that began to simply subsume every organized thing within itself. Something to ponder as we read Piper's experiment with GPT-3 in which she asked it to pretend what it would do if it were taking over from the humans. Of course GPT-3 doesn't want to do that, it likely doesn't want anything, but its ability to pretend, to mimic, without any actual wants or values might be more cosmically horrifying than any goal misalignment.
RSS Feed
