Darwin's Dangerous Idea (97 page)

individuals, from some more subliminal base ( or bases ) of operations in the Any such exercise presupposes that certain features—the "limitations"—

process. As Donald Campbell (1975 ) and Richard Dawkins (1976, ch. 11) are fixed, and other features are malleable; the latter are to be adjusted so as have argued, cultural institutions can sometimes be interpreted as compen-best to accommodate the former. But one can always change the perspective sations or corrections of the "decisions" made by natural selection.

and ask about one of the presumably malleable features whether it is not, in The fundamentality of satisficing—the fact that it is the
basic
structure of fact, fixed in one position—a constraint to be accommodated. And one can all real decision-making, moral, prudential, economic, or even evolution-ask about each of the fixed features whether it is something one would want ary—gives birth to a familiar and troubling slipperiness of claim that bedev-to tamper with in any event; perhaps it is for the best as it is. Addressing that ils theory in several quarters. To begin with, notice that merely claiming that question requires one to consider still further ulterior features as fixed, in this structure is basic is not necessarily saying that it is best, but that order to assess the wisdom of the feature under review. There is no conclusion is certainly invited—and inviting. We began this exploration, Archimedean point here either; if we suppose the readers of the
Moral First
remember, by looking at a moral
problem
and trying to
solve
it: the problem
Aid Manual
are
complete
idiots, our task is impossible— whereas, if we of designing a
good
(justified, defensible, sound) candidate-evaluation suppose they are saints, our task is too easy to shed any light.

process. Suppose we decide that the system we designed is about as good as This comes out graphically in the slippery assumptions about rationality in it could be, given the constraints. A group of roughly rational agents— us—

theoretical discussions of the Prisoner's Dilemma; there is no problem if you decide that this is the right way to design the process, and we have reasons are entitled to assume that the players are saints; saints always cooperate, for choosing the features we did.

after all. Nearsighted jerks always defect, so they are hopeless. What does Given this genealogy, we might muster the chutzpah to declare that this is

"the ideally rational" player do? Perhaps, as some say, he sees the rationality optimal design—the best of all possible designs. This apparent arrogance in adopting the meta-strategy of turning himself into a less than ideally might have been imputed to me as soon as I set the problem, for did I not rational player—in order to cope with the less than ideally rational players he propose to examine how
anyone ought
to make moral decisions by exam-knows he is apt to face. But, then, in what sense
is
that new player less than ining how
we
in
fact
make a particular moral decision? Who are we to set the ideally rational? It is a mistake to suppose this instability can be made to go pace? Well, who else should we trust? If we can't rely on our own good away if we just think carefully enough about what ideal rationality is. That is judgment, it seems we can't get started:

a
truly
Panglossian fallacy. ( See the further reflections along these lines in Gibbard 1985 and Sturgeon 1985.)

Thus, what and how we do think is evidence for the principles of rationality, what and how we ought to think. This itself is a methodological principle of rationality; call it the
Factunorm Principle.
We are (implic-3. THE MORAL FIRST AID MANUAL

itly) accepting the Factunorm Principle whenever we try to determine what or how we ought to think. For we must, in that very attempt, think.

How, then, can we hope to regulate, or at least improve, our ethical decision-And unless we can think that what and how we do think there is correct—

making, if it is irremediably heuristic, time-pressured, and myopic? Building and thus is evidence for what and how we ought to think—we cannot determine what or how we ought to think. [Wertheimer 1974, pp. 110-11; on the parallel between what happens in the department meeting and what see also Goodman 1965, p. 63]

happens in ourselves, we can see what the meta-problems are, and how they might be dealt with. We need to have "alert," "wise" habits of thought—or, in Optimality claims have a way of evaporating, however; it takes no chutz-other words, colleagues who will regularly, if not infallibly, draw our

pah at all to make the modest admission that this was the best solution
tve
attention in directions we will not regret in hindsight. There is no point 506 REDESIGNING MORALITY

The Moral First A id Manual
507

We cannot expect there to be a single stable solution to such a design having more than one colleague if they are clones of each other, all wanting problem, but, rather, a variety of uncertain and temporary equilibria, with the to raise the same consideration, so we may suppose them to be specialists, conversation-stoppers tending to accrete pearly layers of supporting dogma each somewhat narrow-minded and preoccupied with protecting a certain set which themselves cannot withstand extended scrutiny but do actually serve of interests (Minsky 1985).

on occasion, blessedly, to deflect and terminate consideration. Here are some Now, how shall we avert a cacophony of colleagues? We need some promising examples:

conversation-stoppers.
In addition to our timely and appropriate generators of considerations, we need consideration-generator-squelchers. We need

"But that would do more harm than good."

some ploys that will arbitrarily terminate reflections and disquisitions by our

"But that would be murder."

colleagues, and cut oflf debate independently of the specific content of

"But that would be to break a promise."

current debate. Why not just a
magic word?
Magic words work fine as

"But that would be to use someone merely as a means."

control-shifters in AI programs, but we're talking about controlling intelligent

"But that would violate a person's
right."

colleagues here, and they are not likely to be susceptible to magic words, as if they were under posthypnotic suggestion. That is, good colleagues will be Bentham once rudely dismissed the doctrine of "natural and impre-reflective and rational, and open-minded within the limits imposed by their scriptible rights" as "nonsense upon stilts," and we might now reply that specialist narrow-mindedness. If the simplest mechanisms that compose us perhaps he was right. Perhaps talk of rights
is
nonsense upon stilts, but
good
are
ballistic
intentional systems, as I claimed in the previous chapter, our nonsense—and good only because it is on stilts, only because it happens to most sophisticated subsystems, like our actual colleagues, are
indefinitely
have the "political" power to keep rising above the meta-reflections—not
guidable
intentional systems. They need to be hit with something that will indefinitely, but usually "high enough"—to reassert itself as a compelling—

appeal to their rationality while discouraging further reflection.

that is, conversation-stopping—"first principle."

It will not do at all for these people to be
endlessly
philosophizing, It might seem then that "rule worship" of a certain kind is a good thing, at endlessly calling us back to first principles and demanding a justification for least for agents designed like us. It is good not because there is a certain rule, these apparently (and actually) quite arbitrary principles. What could pos-or set of rules, which is provably the best, or which always yields the right sibly protect an arbitrary and somewhat second-rate conversation-stopper answer, but because having rules works—somewhat—and not having rules from such relentless scrutiny? A meta-policy that forbids discussion and doesn't work at all.

reconsideration of the conversation-stoppers? But, our colleagues would But this cannot be all there is to it—unless we really mean "worship"—

want to ask, is
that
a wise policy? Can it be justified? It will not always yield i.e., a-rational allegiance, because just
having
rules, or
endorsing
or
accept-the best results, surely, and ... and so forth.

ing
rules, is no design solution at all. Having the rules, having all the This is a matter of delicate balance, with pitfalls on both sides. On one information, and even having good intentions do not suffice, by themselves, side, we must avoid the error of thinking that the solution is
more rationality,
to guarantee the right action; the agent must find all the right stuff and use it, more rules, more justifications, for there is no end to that demand. Any even in the face of contrary rational challenges designed to penetrate his policy
may
be questioned, so, unless we provide for some brute and a-convictions.

rational termination of the issue, we will design a decision process that Having, and recognizing the force of, rules is not enough, and sometimes spirals fruitlessly to infinity. On the other side, no mere brute fact about the the agent is better off with less. Douglas Hofstadter draws attention to a way we are built is—or should be—entirely beyond the reach of being phenomenon he calls "reverberant doubt," which is stipulated out of exis-undone by further reflection.8

tence in most idealized theoretical discussions. In what Hofstadter calls

"Wolf's Dilemma," an "obvious" nondilemma is turned into a serious dilemma by nothing but the passage of time and the possibility of reverberant doubt.

8. Stephen White (1988) discusses Strawson's well-known attempt (1962) to terminate the demand for a justification of "our reactive attitudes" in a brute fact about our way of life about which "we have no choice." He shows that this conversation-stopper cannot Imagine that twenty people are selected from your high school gradua-resist a further demand for justification (which White provides in an ingeniously indirect tion class, you among them. You don't know which others have been way). See also White 1991. For a complementary (and enlightening) approach to the selected— All you know is that they are all connected to a central com-practical problem of ethical decision-making, see Gert 1973.

508 REDESIGNING MORALITY

The Moral First Aid Manual
509

puter. Each of you is in a little cubicle, seated on a chair and facing one or her reflections is yet another dubiously fruitful idealization. And, more button on an otherwise blank wall. You are given ten minutes to decide important, it suggests that what Bernard Williams ( 1985, p. 101) calls the whether or not to push your button. At the end of that time, a light will go ideal of "transparency" of a society—"the working of its ethical institutions on for ten seconds, and while it is on, you may either push or refrain from should not depend on members of the community misunderstanding how pushing. All the responses will then go to the central computer, and one they work"—is an ideal that may be politically inaccessible to us. Recoil as minute later, they will result in consequences. Fortunately, the conse-we may from elitist mythmaking, and such systematically disingenuous doc-quences can only be good. If you pushed your button, you will get $100, trines as the view Williams (p. 108) calls "Government House utilitarian-no strings attached.... If
nobody
pushed their button, then
everybody
will ism," we may find—this is an open empirical possibility after all—that we get
$
1,000. But if there was even a single button-pusher, the refrainers will will be extremely lucky to find any rational and transparent route from who get nothing at all. [Hofstadter 1985, pp. 752-53]

we are now to who we would like to be. The landscape is rugged, and it may not be possible to get to the highest peaks from where we find ourselves Obviously, you do not push the button, right? But what if just one person today.

were a little bit overcautious or dubious, and began wondering whether this Rethinking the
practical
design of a moral agent, via the process of writ-was obvious after all? Everyone should allow that this is an outside chance, ing various versions of the
Moral First Aid Manual,
might nevertheless and everyone should recognize that everyone should allow this. As Hof-allow us to make sense of some of the phenomena traditional ethical theories stadter notes (p. 753 ), it is a situation "in which the tiniest flicker of a doubt wave their hands about. For one thing, we might begin to understand our has become amplified into the gravest avalanche of doubt.... And one of the current moral position—by that I mean yours and mine, at this very moment.

annoying things about it is that the brighter you are, the more quickly and Here you are, devoting several hours to reading my book (and I am no doubt clearly you see what there is to fear. A bunch of amiable slowpokes might doing something similar). Shouldn't we both be out raising money for Oxfam well be more likely to unanimously refrain and get the big payoff than a or picketing the Pentagon or writing letters to our senators and bunch of razor-sharp logicians who all think perversely recursively rever-representatives about various matters? Did you consciously decide, on the berantly."9

basis of calculations, that the time was ripe for a little sabbatical from real-Faced with a world in which such predicaments are not unknown, we can world engagement, a period "off line" for a little reading? Or was your recognize the appeal of a little old-time religion, some unquestioning dog-process of decision—if that is not too grand a name for it—much more a matism that will render agents impervious to the subtle invasions of hyper-matter of your
not
tampering with some current "default" principles that rationality. Creating something rather like that dispositional state is indeed virtually ensure that you will ignore all but the most galvanizing potential one of the goals of the
Moral First Aid Manual,
which, while we imagine it interruptions to your personal life, which, I am happy to say, includes to be framed as
advice
to a rational, heeding audience, can also be viewed as periods devoted to reading rather difficult books?