Superintelligence: Paths, Dangers, Strategies (69 page)

Tetlock, Philip E., and Belkin, Aaron. 1996. “Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives.” In
Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives
, edited by Philip E. Tetlock and Aaron Belkin, 1–38. Princeton, NJ: Princeton University Press.

Thompson, Adrian. 1997. “Artificial Evolution in the Physical World.” In
Evolutionary Robotics: From Intelligent Robots to Artificial Life
, edited by Takashi Gomi, 101–25. Er ’97. Carp, ON: Applied AI Systems.

Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., et al. 2006. “Stanley: The Robot That Won the DARPA Grand Challenge.”
Journal of Field Robotics
23 (9): 661–92.

Trachtenberg, J. T., Chen, B. E., Knott, G. W., Feng, G., Sanes, J. R., Welker, E., and Svoboda, K. 2002. “Long-Term In Vivo Imaging of Experience-Dependent Synaptic Plasticity in Adult Cortex.”
Nature
420 (6917): 788–94.

Traub, Wesley A. 2012. “Terrestrial, Habitable-Zone Exoplanet Frequency from
Kepler
.”
Astrophysical Journal
745 (1): 1–10.

Truman, James W., Taylor, Barbara J., and Awad, Timothy A. 1993. “Formation of the Adult Nervous System.” In
The Development of
Drosophila Melanogaster, edited by Michael Bate and Alfonso Martinez Arias. Plainview, NY: Cold Spring Harbor Laboratory.

Tuomi, Ilkka. 2002. “The Lives and the Death of Moore’s Law.”
First Monday
7 (11).

Turing, A. M. 1950. “Computing Machinery and Intelligence.”
Mind
59 (236): 433–60.

Turkheimer, Eric, Haley, Andreana, Waldron, Mary, D’Onofrio, Brian, and Gottesman, Irving I. 2003. “Socioeconomic Status Modifies Heritability of IQ in Young Children.”
Psychological Science
14 (6): 623–8.

Uauy, Ricardo, and Dangour, Alan D. 2006. “Nutrition in Brain Development and Aging: Role of Essential Fatty Acids.” Supplement,
Nutrition Reviews
64 (5): S24–S33.

Ulam, Stanislaw M. 1958. “John von Neumann.”
Bulletin of the American Mathematical Society
64 (3): 1–49.

Uncertain Future, The. 2012. “Frequently Asked Questions” The Uncertain Future. Retrieved March 25, 2012. Available at
http://www.theuncertainfuture.com/faq.html
.

U.S. Congress, Office of Technology Assessment. 1995.
U.S.–Russian Cooperation in Space
ISS-618. Washington, DC: U.S. Government Printing Office, April.

Van Zanden, Jan Luiten. 2003.
On Global Economic History: A Personal View on an Agenda for Future Research
. International Institute for Social History, July 23.

Vardi, Moshe Y. 2012. “Artificial Intelligence: Past and Future.”
Communications of the ACM
55 (1): 5.

Vassar, Michael, and Freitas, Robert A., Jr. 2006. “Lifeboat Foundation Nanoshield.” Lifeboat Foundation. Retrieved May 12, 2012. Available at
http://lifeboat.com/ex/nanoshield
.

Vinge, Vernor. 1993. “The Coming Technological Singularity: How to Survive in the Post-Human Era.” In
Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace
, 11–22. NASA Conference Publication 10129. NASA Lewis Research Center.

Visscher, P. M., Hill, W. G., and Wray, N. R. 2008. “Heritability in the Genomics Era: Concepts and Misconceptions.”
Nature Reviews Genetics
9 (4): 255–66.

Vollenweider, Franz, Gamma, Alex, Liechti, Matthias, and Huber, Theo. 1998. “Psychological and Cardiovascular Effects and Short-Term Sequelae of MDMA (‘Ecstasy’) in MDMA-Naïve Healthy Volunteers.”
Neuropsychopharmachology
19 (4): 241–51.

Wade, Michael J. 1976. “Group Selections Among Laboratory Populations of Tribolium.”
Proceedings of the National Academy of Sciences of the United States of America
73 (12): 4604–7.

Wainwright, Martin J., and Jordan, Michael I. 2008. “Graphical Models, Exponential Families, and Variational Inference.”
Foundations and Trends in Machine Learning
1 (1–2): 1–305.

Walker, Mark. 2002. “Prolegomena to Any Future Philosophy.”
Journal of Evolution and Technology
10 (1).

Walsh, Nick Paton. 2001. “Alter our DNA or robots will take over, warns Hawking.”
The Observer
, September 1.
http://www.theguardian.com/uk/2001/sep/02/medicalscience.genetics
.

Warwick, Kevin. 2002.
I, Cyborg
. London: Century.

Wehner, M., Oliker, L., and Shalf, J. 2008. “Towards Ultra-High Resolution Models of Climate and Weather.”
International Journal of High Performance Computing Applications
22 (2): 149–65.

Weizenbaum, Joseph. 1966. “Eliza: A Computer Program for the Study of Natural Language Communication Between Man And Machine.”
Communications of the ACM
9 (1): 36–45.

Weizenbaum, Joseph. 1976.
Computer Power and Human Reason: From Judgment to Calculation
. San FrancYork, CA: W. H. Freeman.

Werbos, Paul John. 1994.
The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting
. New York: Wiley.

White, J. G., Southgate, E., Thomson, J. N., and Brenner, S. 1986. “The Structure of the Nervous System of the Nematode
Caenorhabditis Elegans
.”
Philosophical Transactions of the Royal Society of London
.
Series B, Biological Sciences
314 (1165): 1–340.

Whitehead, Hal. 2003.
Sperm Whales: Social Evolution in the Ocean
. Chicago: University of Chicago Press.

Whitman, William B., Coleman, David C., and Wiebe, William J. 1998. “Prokaryotes: The Unseen Majority.”
Proceedings of the National Academy of Sciences of the United States of America
95 (12): 6578–83.

Wiener, Norbert. 1960. “Some Moral and Technical Consequences of Automation.”
Science
131 (3410): 1355–8.

Wikipedia
. 2012a, s.v. “Computer Bridge.” Retrieved June 30, 2013. Available at
http://en.wikipedia.org/wiki/Computer_bridge
.

Wikipedia
. 2012b, s.v. “Supercomputer.” Retrieved June 30, 2013. Available at
http://et.wikipedia.org/wiki/Superarvuti
.

Williams, George C. 1966.
Adaptation and Natural Selection: A Critique of Some Current Evolutionary Thought
. Princeton Science Library. Princeton, NJ: Princeton University Press.

Winograd, Terry. 1972.
Understanding Natural Language
. New York: Academic Press.

Wood, Nigel. 2007.
Chinese Glazes: Their Origins, Chemistry and Re-creation
. London: A. & C. Black.

World Bank. 2008.
Global Economic Prospects: Technology Diffusion in the Developing World
42097. Washington, DC.

World Robotics. 2011.
Executive Summary of 1
.
World Robotics 2011 Industrial Robots; 2. World Robotics 2011 Service Robots
. Retrieved June 30, 2012. Available at
http://www.bara.org.uk/pdf/2012/world-robotics/Executive_Summary_WR_2012.pdf
.

World Values Survey. 2008.
WVS 2005-2008
. Retrieved 29 October, 2013. Available at
http://www.wvsevsdb.com/wvs/WVSAnalizeStudy.jsp
.

Wright, Robert. 2001.
Nonzero: The Logic of Human Destiny
. New York: Vintage.

Yaeger, Larry. 1994. “Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context.” In
Proceedings of the Artificial Life III Conference
, edited by C. G. Langton, 263–98. Santa Fe Institute Studies in the Sciences of Complexity. Reading, MA: Addison-Wesley.

Yudkowsky, Eliezer. 2001.
Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures
. Machine Intelligence Research Institute, San Francisco, CA, June 15.

Yudkowsky, Eliezer. 2002. “The AI-Box Experiment.” Retrieved January 15, 2012. Available at
http://yudkowsky.net/singularity/aibox
.

Yudkowsky, Eliezer. 2004.
Coherent Extrapolated Volition
. Machine Intelligence Research Institute, San Francisco, CA, May.

Yudkowsky, Eliezer. 2007. “Levels of Organization in General Intelligence.” In
Artificial General Intelligence
, edited by Ben Goertzel and Cassio Pennachin, 389–501. Cognitive Technologies. Berlin: Springer.

Yudkowsky, Eliezer. 2008a. “Artificial Intelligence as a Positive and Negative Factor in Global Risk.” In
Global Catastrophic Risks
, edited by Nick Bostrom and Milan M. Ćirković, 308–45. New York: Oxford University Press.

Yudkowsky, Eliezer. 2008b. “Sustained Strong Recursion.”
Less Wrong
(blog), December 5.

Yudkowsky, Eliezer. 2010.
Timeless Decision Theory
. Machine Intelligence Research Institute, San Francisco, CA.

Yudkowsky, Eliezer. 2011.
Complex Value Systems are Required to Realize Valuable Futures
. San Francisco, CA: Machine Intelligence Research Institute.

Yudkowsky, Eliezer. 2013.
Intelligence Explosion Microeconomics
, Technical Report 2013–1. Berkeley, CA: Machine Intelligence Research Institute.

Zahavi, Amotz, and Zahavi, Avishag. 1997.
The Handicap Principle: A Missing Piece of Darwin’s Puzzle
. Translated by N. Zahavi-Ely and M. P. Ely. New York: Oxford University Press.

Zalasiewicz, J., Williams, M., Smith, A., Barry, T. L., Coe, A. L., Bown, P. R., Brenchley, P., et al. 2008. “Are We Now Living in the Anthropocene?”
GSA Today
18 (2): 4–8.

Zeira, Joseph. 2011. “Innovations, Patent Races and Endogenous Growth.”
Journal of Economic Growth
16 (2): 135–56.

Zuleta, Hernando. 2008. “An Empirical Note on Factor Shares.”
Journal of International Trade and Economic Development
17 (3): 379–90.

A

Afghan Taliban
215

Agricultural Revolution
2
,
80
,
261

AI-complete problem
14
,
47
,
71
,
93
,
145
,
186

AI-OUM,
see
optimality notions

AI-RL,
see
optimality notions

AI-VL,
see
optimality notions

algorithmic soup
172

algorithmic trading
16
–
17

anthropics
27
–
28
,
126
,
134
–
135
,
174
,
222
–
225

definition
225

Arendt, Hannah
105

Armstrong, Stuart
280
,
291
,
294
,
302

artificial agent
10
,
88
,
105
–
109
,
172
–
176
,
185
–
206
;
see also
Bayesian agent

artificial intelligence

arms race
64
,
88
,
247

future of
19
,
292

greater-than-human,
see
superintelligence

history of
5
–
18

overprediction of
4

pioneers
4
–
5
,
18

Asimov, Isaac
139

augmentation
142
–
143
,
201
–
203

autism
57

automata theory
5

automatic circuit breaker
17

automation
17
,
98
,
117
,
160
–
176

B

backgammon
12

backpropagation algorithm
8

bargaining costs
182

Bayesian agent
9
–
11
,
123
,
130
;
see also
artificial agent
and
optimality notions

Bayesian networks
9

Berliner, Hans
12

biological cognition
22
,
36
–
48
,
50
–
51
,
232

biological enhancement
36
–
48
,
50
–
51
,
142
–
143
,
232
;
see also
cognitive enhancement

boxing
129
–
131
,
143
,
156
–
157

informational
130

physical
129
–
130

brain implant,
see
cyborg

brain plasticity
48

brain–computer interfaces
44
–
48
,
51
,
83
,
142
–
143
;
see also
cyborg

Brown, Louise
43

C

C. elegans
34–
35
,
266
,
267

capability control
129
–
144
,
156
–
157

capital
39
,
48
,
68
,
84
–
88
,
99
,
113
–
114
,
159
–
184
,
251
,
287
,
288
,
289

causal validity semantics
197

CEV,
see
coherent extrapolated volition

Chalmers, David
24
,
265
,
283
,
295
,
302

character recognition
15

checkers
12

chess
11
–
22
,
52
,
93
,
134
,
263
,
264

child machine
23
,
29
;
see also
seed AI

CHINOOK
12

Christiano, Paul
198
,
207

civilization baseline
63

cloning
42

cognitive enhancement
42
–
51
,
67
,
94
,
111
–
112
,
193
,
204
,
232
–
238
,
244
,
259

coherent extrapolated volition (CEV)
198
,
211
–
227
,
296
,
298
,
303

definition
211

collaboration (benefits of)
249

collective intelligence
48
–
51
,
52
–
57
,
67
,
72
,
142
,
163
,
203
,
259
,
271
,
273
,
279

collective superintelligence
39
,
48
–
49
,
52
–
59
,
83
,
93
,
99
,
285

definition
54

combinatorial explosion
6
,
9
,
10
,
47
,
155

Common Good Principle
254
–
259

common sense
14

computer vision
9

computing power
7
–
9
,
24
,
25
–
35
,
47
,
53
–
60
,
68
–
77
,
101
,
134
,
155
,
198
,
240
–
244
,
251
,
286
,
288
;
see also
computronium
and
hardware overhang

computronium
101
,
123
–
124
,
140
,
193
,
219
;
see also
computing power

connectionism
8

consciousness
22
,
106
,
126
,
139
,
173
–
176
,
216
,
226
,
271
,
282
,
288
,
292
,
303
;
see also
mind crime

control methods
127
–
144
,
145
–
158
,
202
,
236
–
238
,
286
;
see also
capability control
and
motivation selection

Copernicus, Nicolaus
14

cosmic endowment
101
–
104
,
115
,
134
,
209
,
214
–
217
,
227
,
250
,
260
,
283
,
296

crosswords (solving)
12

cryptographic reward tokens
134
,
276

cryptography
80

cyborg
44
–
48
,
67
,
270

D

DARPA,
see
Defense Advanced Research Projects Agency

DART (tool)
15

Dartmouth Summer Project
5

data mining
15
–
16
,
232
,
301

decision support systems
15
,
98
;
see also
tool-AI

decision theory
10
–
11
,
88
,
185
–
186
,
221
–
227
,
280
,
298
;
see also
optimality notions

decisive strategic advantage
78
–
89
,
95
,
104
–
112
,
115
–
126
,
129
–
138
,
148
–
149
,
156
–
159
,
177
,
190
,
209
–
214
,
225
,
252

Deep Blue
12

Deep Fritz
22

Defense Advanced Research Projects Agency (DARPA)
15

design effort,
see
optimization power

Dewey, Daniel
291

Differential Technological Development (Principle of)
230
–
237

Diffie–Hellman key exchange protocol
80

diminishing returns
37
–
38
,
66
,
88
,
114
,
273
,
303

direct reach
58

direct specification
139
–
143

DNA synthesis
39
,
98

Do What I Mean (DWIM)
220
–
221

domesticity
140
–
143
,
146
–
156
,
187
,
191
,
207
,
222

Drexler, Eric
239
,
270
,
276
,
278
,
300

drones
15
,
98

Dutch book
111

Dyson, Freeman
101
,
278

E

economic growth
3
,
160
–
166
,
179
,
261
,
274
,
299

Einstein, Albert
56
,
70
,
85

ELIZA (program)
6

embryo selection
36
–
44
,
67
,
268

emulation modulation
207

Enigma code
87

environment of evolutionary adaptedness
164
,
171

epistemology
222
–
224

equation solvers
15

eugenics
36
–
44
,
268
,
279

Eurisko
12

evolution
8
–
9
,
23
–
27
,
44
,
154
,
173
–
176
,
187
,
198
,
207
,
265
,
266
,
267
,
273

evolutionary selection
187
,
207
,
290

evolvable hardware
154

exhaustive search
6

existential risk
4
,
21
,
55
,
100
–
104
,
115
–
126
,
175
,
183
,
230
–
236
,
239
–
254
,
256
–
259
,
286
,
301
–
302

state risks
233
–
234

step risks
233

expert system
7

explicit representation
207

exponential growth,
see
growth

external reference semantics
197

F

face recognition
15

failure modes
117
–
120

Faraday cage
130

Fields Medal
255
–
256
,
272

Fifth-Generation Computer Systems Project
7

fitness function
25
;
see also
evolution

Flash Crash (2010)
16
–
17

formal language
7
,
145

FreeCell (game)
13

G

game theory
87
,
159

game-playing AI
12
–
14

General Problem Solver
6

genetic algorithms
7
–
13
,
24
–
27
,
237
–
240
;
see also
evolution

genetic selection
37
–
50
,
61
,
232
–
238
;
see also
evolution

genie AI
148
–
158
,
285

definition
148

genotyping
37

germline interventions
37
–
44
,
67
,
273
;
see also
embryo selection

Ginsberg, Matt
12

Go (game)
13

goal-content
109
–
110
,
146
,
207
,
222
–
227

Good Old-Fashioned Artificial Intelligence (GOFAI)
7
–
15
,
23

Good, I. J.
4

Gorbachev, Mikhail
86

graceful degradation
7

graphical models
11

growth
1
–
7
,
48
–
55
,
69
–
75
,
83
,
163
,
261
,
281

H

Hail Mary approach
198
–
200
,
207
,
293
,
294

Hanson, Robin
2
,
160
,
261
,
270
,
271
,
287