Superintelligence: Paths, Dangers, Strategies (70 page)

hardware overhang
73
,
240
–
243
,
274
,
289
,
301
,
302

hedonism
140
,
210

hedonium
140
,
219

Helsinki Declaration
188

heuristic search
6

Hill, Benny
105

hippocampus
47

HLMI,
see
machine intelligence, human-level

Hodgkin–Huxley model
25

human baseline
62
–
77
,
82

human extinction,
see
existential risk

Human Genome Project
86
,
253
,
276

human intelligence
5
–
14
,
24
–
58
,
98
–
99
,
159
–
184
,
242
–
254
,
255
–
257

human–machine interface,
see
cyborg

I

impersonal perspective
228
–
246

in vitro
fertilization,
see
embryo selection

incentive methods
131
–
143

cryptographic reward tokens
133

social integration
131
–
132
,
156
–
158
,
159
,
202
,
283

indirect normativity
141
–
150
,
209
–
227
,
262
,
298

indirect reach
58

inductive bias
9
–
10

Industrial Revolution
2
,
80
,
161
–
163

infrastructure profusion
123
–
125
,
153
,
187
,
226
,
282

institution design
202
–
208

instrumental convergence thesis
105
–
116

intelligence explosion
2
–
5
,
22
–
51
,
62
–
77
,
78
–
90
,
95
–
104
,
108
,
115
–
126
,
127
–
128
,
136
,
151
,
160
,
165
,
178
,
198
,
205
,
227
,
228
–
254
,
256
–
260
,
274
,
276
,
282
,
284
,
289
,
300
,
301
–
302

Internet
16
,
45
–
49
,
67
–
77
,
85
,
94
–
98
,
130
,
146
,
241
,
271

inventory control systems
16

IVF,
see
embryo selection

J

Jeopardy!13

K

Kasparov, Garry
12

Kepler, Johannes
14

Knuth, Donald
14
,
264

Kurzweil, Ray
2
,
261
,
269

L

Lenat, Douglas
12
,
263

Logic Theorist (system)
6

logicist paradigm,
see
Good Old-Fashioned Artificial Intelligence (GOFAI)

Logistello
12

M

machine intelligence;
see also
artificial intelligence

human-level (HLMI)
4
,
19
–
21
,
27
–
35
,
73
–
74
,
207
,
243
,
264
,
267

revolution,
see
intelligence explosion

machine learning
8
–
18
,
28
,
121
,
152
,
188
,
274
,
290

machine translation
15

macro-structural development accelerator
233
–
235

malignant failure
123
–
126
,
149
,
196

Malthusian condition
163
–
165
,
252

Manhattan Project
75
,
80
–
87
,
276

McCarthy, John
5
–
18

McCulloch–Pitts neuron
237

MegaEarth
56

memory capacity
7
–
9
,
60
,
71

memory sharing
61

Mill, John Stuart
210

mind crime
125
–
126
,
153
,
201
–
208
,
213
,
226
,
297

Minsky, Marvin
18
,
261
,
262
,
282

Monte Carlo method
9
–
13

Moore’s law
24
–
25
,
73
–
77
,
274
,
286
;
see also
computing power

moral growth
214

moral permissibility (MP)218–
220
,
297

moral rightness (MR)217–
220
.
296
,
297

moral status
125
–
126
,
166
–
169
,
173
,
202
–
205
,
268
,
288
,
296

Moravec, Hans
24
,
265
,
288

motivation selection
29
,
127
–
129
,
138
–
144
,
147
,
158
,
168
,
180
–
191
,
222

definition
138

motivational scaffolding
191
,
207

multipolar scenarios
90
,
132
,
159
–
184
,
243
–
254
,
301

mutational load
41

N

nanotechnology
53
,
94
–
98
,
103
,
113
,
177
,
231
,
239
,
276
,
277
,
299
,
300

natural language
14

neural networks
5
–
9
,
28
,
46
,
173
,
237
,
262
,
274

neurocomputational modeling
25
–
30
,
35
,
61
,
301
;
see also
whole brain emulation (WBE)
and
neuromorphic AI

neuromorphic AI
28
,
34
,
47
,
237
–
245
,
267
,
300
,
301

Newton, Isaac
56

Nilsson, Nils
18
–
20
,
264

nootropics
36
–
44
,
66
–
67
,
201
,
267

Norvig, Peter
19
,
264
,
282

O

observation selection theory,
see
anthropics

Oliphant, Mark
85

O’Neill, Gerard
101

ontological crisis
146
,
197

optimality notions
10
,
186
,
194
,
291
–
293

Bayesian agent
9
–
11

value learner (AI-VL)
194

observation-utility-maximizer (AI-OUM)
194

reinforcement learner (AI-RL)
194

optimization power
24
,
62
–
75
,
83
,
92
–
96
,
227
,
274

definition
65

oracle AI
141
–
158
,
222
–
226
,
285
,
286

definition
146

orthogonality thesis
105
–
109
,
115
,
279
,
280

P

paperclip AI
107
–
108
,
123
–
125
,
132
–
135
,
153
,
212
,
243

Parfit, Derek
279

Pascal’s mugging
223
,
298

Pascal’s wager
223

person-affecting perspective
228
,
245
–
246
,
301

perverse instantiation
120
–
124
,
153
,
190
–
196

poker
13

principal–agent problem
127
–
128
,
184

Principle of Epistemic Deference
211
,
221

Proverb (program)
12

Q

qualia,
see
consciousness

quality superintelligence
51
–
58
,
72
,
243
,
272

definition
56

R

race dynamic,
see
technology race

rate of growth,
see
growth

ratification
222
–
225

Rawls, John
150

Reagan, Ronald
86
–
87

reasons-based goal
220

recalcitrance
62
–
77
,
92
,
241
,
274

definition
65

recursive self-improvement
29
,
75
,
96
,
142
,
259
;
see also
seed AI

reinforcement learning
12
,
28
,
188
–
189
,
194
–
196
,
207
,
237
,
277
,
282
,
290

resource acquisition
113
–
116
,
123
,
193

reward signal
71
,
121
–
122
,
188
,
194
,
207

Riemann hypothesis catastrophe
123
,
141

robotics
9
–
19
,
94
–
97
,
117
–
118
,
139
,
238
,
276
,
290

Roosevelt, Franklin D.85

RSA encryption scheme
80

Russell, Bertrand
6
,
87
,
139
,
277

S

Samuel, Arthur
12

Sandberg, Anders
265
,
267
,
272
,
274

scanning,
see
whole brain emulation (WBE)

Schaeffer, Jonathan
12

scheduling
15

Schelling point
147
,
183
,
296

Scrabble
13

second transition
176
–
178
,
238
,
243
–
245
,
252

second-guessing (arguments)
238
–
239

seed AI
23
–
29
,
36
,
75
,
83
,
92
–
96
,
107
,
116
–
120
,
142
,
151
,
189
–
198
,
201
–
217
,
224
–
225
,
240
–
241
,
266
,
274
,
275
,
282

self-limiting goal
123

Shakey (robot)
6

SHRDLU (program)
6

Shulman, Carl
178
–
180
,
265
,
287
,
300
,
302
,
304

simulation hypothesis
134
–
135
,
143
,
278
,
288
,
292

singleton
78
–
90
,
95
–
104
,
112
–
114
,
115
–
126
,
136
,
159
,
176
–
184
,
242
,
275
,
276
,
279
,
281
,
287
,
299
,
301
,
303

definition
78
,
100

singularity
1
,
2
,
49
,
75
,
261
,
274
;
see also
intelligence explosion

social signaling
110