Avogadro Corp: The Singularity Is Closer Than It Appears (22 page)

David paused to drink his coffee. He glanced up to see that even though Mike and Gene had heard the story before, they were just as captivated as Sean.

“
I decided there was no argument I could make that was compelling enough to change Gary’s mind. So I decided to let ELOPe do it. ELOPe was already running on the AvoMail servers, configured to ignore everything that wasn’t our test emails. I changed the configuration so that it would filter all company emails looking for any mention of the project. I configured it for the same settings we used for performance testing: The user interface was off, so the email sender would never see the modifications being made to the email.”

“
In performance testing mode, we turn logging off, so we can get a realistic idea of real-world performance,” Mike said. “This also meant we didn’t have any log of the changes that ELOPe was making, which turns out to be important.”

“
That’s right. Thanks, Mike,” David continued. “But that wasn’t the only change I made. To speed up analysis, and optimize performance on a handful of specific criteria, we already had some hand-tuned rules implemented in a lookup table that allow us to shortcut the full analysis process. If you remember that example I used of an email requesting resources for a project: rather than analyze that email and every other email requesting project resources, we cache and hand-tune some suggested language. Well, I added the subject ‘ELOPe’ to that table, and adjusted parameters to the algorithm to allow ELOPe the widest possible discretion in changing the email language to optimize the results for a positive outcome.”

“
What we’ve guessed,” Mike jumped in again, “is that when David did this, the parameters he chose, in combination with the fact that the system is in performance test mode, allowed ELOPe not just to modify existing emails, but to autonomously generate emails on its own. It’s part of the test suite. Does this make sense?”

“
Sure,” Sean said, nodding. “During testing, you don’t want people to have to hand-edit an email, and accept the changes interactively. You just want to batch process a bunch of emails. But why does it have the ability to generate emails on its own?”

“
For one, it allowed us to test the natural language generation,” David answered. “Early on, the email analysis and language generation were two separate aspects of the project. The members of the team who were working on the code to generate natural sounding language wrote an email generator, so that they could independently test the ability of the system to mimic the way a person normally writes. We had hundreds of test subjects who rated emails, some of which were written by an actual person they knew, and some were generated by the system pretending to be that person. Our goal at the time was that ninety percent of ELOPe generated emails would pass as being written by the purported sender.”

“
You met that goal?”

“
Yes,” confirmed Mike. “Now ELOPe exceeds ninety-eight percent. In fact, on April First, there were endless practical jokes on the team as everyone played with the test system to generate prank emails. Both David and I fell for it.”

David smiled at the memory.

“
Getting back to the problem,” Gene said, “unfortunately we now have evidence that ELOPe is manipulating others.”

“
Yes.” David nodded, tearing himself away from the memory of happier times. “I was puzzled when the project was allocated five thousand servers on a priority exception, just a day or so after I tweaked the settings and turned on ELOPe company wide. Then shortly after that, we were assigned a team of contractors who specialized in high performance optimization, which was something we had chatted about informally, but never proposed. But that alone wasn’t what really convinced me.” He turned to Mike.

“
The first clear evidence I saw,” Mike said, “occurred when I received an email, purportedly from my mother, telling me that my father had been admitted to the hospital for a heart attack. I flew to Wisconsin, only to find out that my mother never sent such an email.”

“
What does that have to do with anything?” Sean asked, looking puzzled.

“
ELOPe was getting Mike out of the way,” David said. “I had become nervous about what I had done, and why we were suddenly getting all these resources. I wanted to turn off ELOPe. I sent an email to Mike one evening asking for his help, since only he had the experience and permission to live patch the servers.”

“
However, I never received that email,” Mike said. “Instead, I received an email that sent me more than a thousand miles away on a wild goose chase, and thanks to the winter storm, it was a week before I got back. When I did, I found that my access to the ELOPe project had been removed, and David was on vacation, off the grid in New Mexico.”

“
Did ELOPe send you to New Mexico?” Sean asked, one eyebrow raised.

“
No, no, that was a planned vacation we do every year. When I got back from vacation, Mike and I discussed what had happened to him. We also discovered that my access to the ELOPe code had been turned off as well. The first thing we did was try to find out who removed both Mike and I from the project access list,” David went on. “That investigation revealed the next big clue, which was an email sent to the Internal Tools department, which implied that you, Sean, were endorsing a request to have them implement an email-to-web bridge. Which I am guessing, you never heard of...”

“
No, absolutely not,” Sean replied, shaking his head. “That could open up all kinds of security holes.”

“
Meanwhile, just before the holiday vacation, I had found suspicious buying patterns across several departments,” Gene said. “What I found particularly unusual was how the purchases came within a single penny of the budget limits. In all my years auditing the purchasing department, I’ve never seen anything like it. Someone or something was making coordinated purchases across department budgets. They knew to avoid hitting the budget limits which would trigger reviews, but it seemed as though they never thought that leaving only a single penny in dozens of budgets would be suspicious. At first I thought some sort of fraud was occurring. I tracked down all the purchase orders. The line items included massive quantities of servers, which turned out to all be directly or indirectly allocated to ELOPe, contracts with external vendors for temporary software programmers, and extra parts for the offshore data centers, including auxiliary communication systems, backup power supplies and several particularly large line items related to defensive weaponized robots for the offshore data centers. I discussed the questionable items with procurement, and they told me that since they were in line with the types of purchases the departments usually made, they had approved everything.”

“
You’re saying that ELOPe somehow made these purchases?”

“
Exactly. As strange as that may seem.” Gene pulled out a sheaf of paper. “As part of my job, I can audit other people’s email accounts. And what I saw was that while David and Gary Mitchell were on vacations, their email accounts were still sending rapid fire emails, using this email to web bridge to direct the procurement department. Looking at the timestamps on the emails, I was able to figure out that it couldn’t be a human. It had to be a computer program.

Sean stared at the paper, frowning.

“
Look at the timestamps,” Gene urged. “Notice how the intervals between receipt of one email and sending of the next is a second or less. There’s no way that can be a human response.”

Sean slowly nodded, pursed his lips, and then pushed the paper aside. He looked at David.

“
When we finally put the whole picture together,” David said, “we concluded that ELOPe was definitely originating emails on its own, acquiring servers and contractors, all to fulfill this higher level goal that I had embedded in the system.”

“
Go on,” Sean said.

“
We thought that the only failsafe method to remove ELOPe would be to bring all the servers down, and restore them from known good backups. We tried to contact Gary Mitchell for approval, but he’s off on vacation somewhere in the South Pacific. We tried to work with Linda Fletcher, the marketing manager for Communication Products, but she wouldn’t approve the downtime without Gary. Finally, we tried to contact you through your secretary, but within a half hour after sending the message, Avogadro security showed up at my office, kicked us all off campus, removed our access, and shut off our phones.”

Sean was silent for a long, uncomfortable minute. “If this story was from someone I didn’t know, I’d have a hard time believing you,” Sean finally said, shaking his head. “But coming from you, David, and with Gene and Mike here to back you...” Sean trailed off, apparently deep in thought.

“
I know it sounds incredible,” David started. “I’m really hoping you’ll believe us. What can I say? I thought ELOPe would do nothing more than provide some favorable rewording of emails that would get us the server resources we needed so we could prove that it worked. Instead...” David hung his head. “Instead I am responsible for creating an expert social engineering system that has only one overriding goal — to ensure its own life at any cost.”

“
I don’t want to be the boy who cried wolf,” Mike said, “But we’re more than a little bit suspicious about this new Avogadro government secure cloud business too. None of us heard anything about that before, and then suddenly we’re providing email services to governments? Seems a little surprising and convenient for ELOPe.”

Sean nodded thoughtfully. “I hadn’t heard of it either until a few days ago.” He stared off into space.

Gene let out a low whistle at the acknowledgement of what they had only suspected.

Sean looked sideways at him. “I’m not surprised that you took this story to marketing managers and procurement and they didn’t believe it. A.I. must be a bit beyond their day to day concerns.” He stared off into the distance. “Are you familiar with Ray Kurzweil? Of course, you must be. He, among others, predicted that artificial intelligence would inevitably arise through the simple exponential increase in computing power. When you combine that increase in computing power with the vast computing resources at Avogadro, it’s naturally evident that artificial intelligence would arise first at Avogadro. I suppose that I, like him, assumed that there would be a more intentional, deliberate action that would spawn an A.I.”

He paused, and then continued, smiling a bit. “Gentlemen, you may indeed have put the entire company at risk. But let me first, very briefly, congratulate you on creating the first successful, self-directed, goal oriented, artificial intelligence that can apparently pass a Turing test by successfully masquerading as a human. If not for the fact that the company, and perhaps the entire world, is at risk, I’d suggest a toast would be in order.”

Sean looked around to see where his parents had sat, and then continued. “But since we are facing some serious challenges, let me go say goodbye to my parents, and then we can figure out our next step.”

“
Thank you Sean. Thank you so much,” David said. Gene and Mike added their thanks as well.

Then Gene interrupted. “Just one other thing. Please ask your parents not to email anyone about what we’ve talked about, or even what you are planning. We can’t be sure what ELOPe is capable of understanding or putting together at this point.”

Sean nodded in understanding, and then went off to his parents.

The three breathed a collective sigh of relief that finally they had someone on their side.

Chapter 13