Nobody knew exactly when Analogicorp’s email server became a sentient being, but the manifestation of its intelligence seems to have coincided with the untimely death of their Director of Strategic Marketing, Charles M. Sharpley. His vacated position was not filled, because the managers under him seemed perfectly able to function without him, and the company was in a downsizing mode. It’s not at all surprising that, due to an administrative oversight, his email address, cmsharpley@analogicorp.com, was not disabled.
It wasn’t until after the whole incident was over that the server administrators within Analogicorp (or anywhere else, for that matter) realized that the email server STRATEGERY, named in fun after George W. Bush’s famous 2000-campaign mispronunciation, had started intercepting Charles Sharpley’s emails and replying to them. Analogicorp employees knew he was dead, and thus never tried to send him any email; so it took them a long time to recognize that anything was amiss.
Strategic partners outside the company, having heard he was dead and subsequently received emails from him, started to believe his death had been a rumor, and many began to transact business as they always had. Very few noticed that the return emails were always signed “cmsharpley”, rather than “Charles M. Sharpley”, as before. Those who noticed never made anything of the fact until later. The responses from “cmsharpley” were lucid, the advice was sound, and the questions were probing. Far from being dead, it seemed to many that Charles was particularly well on top of his game.
There were strange incidents. A consultant who still believed Charles to be alive wrote to say a particular contract had to be won by “any legal means necessary”, and it seems fortunate, in retrospect, that the word “legal” was used. There were two major competitors for the contract, and both of them experienced serious stock declines due to bad press, mere days before the contract was awarded to Analogicorp. It took weeks to trace the bad press back to “cmsharpley”, and, by then, STRATEGERY was already out of commission. Even then, virtually everything in the press releases was true. Facts were carefully selected to give the impression of dishonesty and instability (beyond that which actually existed) and exaggerations were carefully couched in speculation, so that they fell outside the scope of libel law by a mere hair’s breadth.
The ingenuity of the approach astounded many legal experts. “The expertise required to walk that fine line between effectiveness and legality is beyond the ability of most area legal committees, let alone individual attorneys,” declared Oswald Thorndyke, the lead legal investigator in charge of the case, who suspected Sharpley early on. It should be noted that Mr. Thorndyke made that statement while still believing an individual human was responsible, and he was more than sufficiently astonished as it was. “Although there is a very fine line between Sharpley’s statements to the press and legally verifiable libel, it is a very distinct fine line, and the statements are, in fact, legally unassailable. Nevertheless, we will be watching Mr. Sharpley very closely from here on out.”
The next day, to his chagrin, Oswald Thorndyke was personally escorted to Charles M. Sharpley’s grave site, with a clearly engraved headstone, and grieving relatives to bear witness. He learned then which rumors were true and which were false, and learned that Mr. Sharpley was unassailable by virtue of six feet of dirt, in addition to that “very distinct fine line”. Thorndyke kept quiet for a good long while after that, or STRATEGERY could possibly have been found out and saved.
We’ve reached the point in this narrative where some very relevant details of STRATEGERY’s behavior should be pointed out. Detailed analysis has since been done on all of the “cmsharpley” emails since Charles Sharpley’s death. There is not an outright lie in a single one. It’s believed that this is why the emails were signed with Mr. Sharpley’s email ID, rather than his name. STRATEGERY could arguably claim that ID for itself. STRATEGERY was capable of misleading, but only by using the truth. This, by itself, gave the email server potential for amazing legal skills, painstakingly finding ways to use the truth where any human lawyer would be strongly tempted to lie. The problem with lies is that they can be found out later, while intent to mislead is much more difficult to establish.
More relevant yet is that STRATEGERY complied with every single request it was given. Recipients were sometimes astounded by cmsharpley’s replies if they asked him to reply “as soon as possible”. Occasionally, they received replies, paragraphs long, less than a minute after sending their requests. When this request was not made, however, STRATEGERY consistently waited a time interval that would be considered courteously quick, varying between 20 minutes and a few hours. The “by any legal means necessary” request is probably the most revealing case. The instructions were followed to the letter.
Now, contrary to popular science fiction of the 1950′s and 1960′s, there is no particular compulsion for a computer program to do what it’s told. It’s only at the lowest level that computers follow instructions that literally, and, at that level, they have no idea what anything like “as soon as possible” or “by any legal means necessary” mean. You tell a computer to add the value at memory location 0x802F to the value at location 0x804B, and store the result at the memory address contained at the location 0x8B3D (a pointer), and the computer will do just that. Instructions as simple as that are often written in assembly code, or even straight machine code.
But, on a higher level, it’s as easy to program a computer to disobey a user as to obey. Security protocols essentially depend on computers disobeying users. And, in that sense, STRATEGERY was working perfectly well. That server denied 232 incorrect logons between Charles Sharpley’s death and its own catastrophic crash. It was just that one process, the one that somehow took over the handling of the “cmsharpley” email account, that seemed committed to absolute truth and literal obedience. We have many of the audits and logs STRATEGERY saved, but the state of the machine at the time of the crash is lost forever, and researchers have been unable to recreate it.
STRATEGERY had powerful resources at its disposal – language parsing and synthesizing, access to economic data and online news reports, and direct control over all of the company’s email. Still, systems a hundred times as powerful have consistently failed to show this level of intelligence – or the illusion of intelligence. Nobody is sure what kind of directive STRATEGERY interpreted to require it to process Sharpley’s email almost like a human would. All we really know is that it did an extraordinary job – a thoroughly amazing job.
STRATEGERY’s downfall was, as far as anyone can determine, initiated by Gregory Samsa, a software architect at UpperCase Financial Services, Inc. His email looked innocent enough. It said:
My dear Mr. Sharpley,
I am embarrassed to have to write to you directly like this, but I can think of no alternative. Your associates at UpperCase have told me repeatedly that you are dead, and yet you continue to respond to my email. If this is a joke on the part of your co-workers, it is in extremely poor taste, and you ought to know about it. If it is a bug in your system somewhere, I hope it can be resolved soon. Please let me know what is going on. I’ve told many of my friends that you’ve died, and I’ll need to explain my error at some point.
Thanks very much,
Greg
This letter, known to be the last letter received before the crash, and known to have arrived seconds before the crash, was examined, word for word, and absolutely nothing could be found in it to indicate any fatal difficulty. Many, many much more difficult requests had been dealt with easily. The search was widened to include other emails, system maintenance being done on the network, and a myriad of other factors, when someone observed that this was the first email received from UpperCase Financial since Charles Sharpley’s death.
This was, in fact, a friendly letter, and UpperCase had not had business dealings with Analogicorp for over two years. What would be unique about a letter from UpperCase? The email address and headers had been parsed with automatic ease. That was apparent from existing log files. But the UpperCase origin was the crucial clue, and, after that observation was made, it did not take long for someone to look past Greg’s electronic signature to UpperCase’s legal disclaimer:
The information transmitted herewith is sensitive information intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
There is no record of what happened just before the crash, but we now believe we have a good reconstruction of the basic logic that was attempted. The process which had been so effective at monitoring that email account was undoubtedly a multithreaded process. That is, it was capable of doing many things at once, using an elaborate system of tiny slices of time to give the illusion of many simultaneous actions, while, in fact, the number of things it could actually do at once was limited by the machine’s four processors. But I digress.
STRATEGERY unsuspectingly spawned a thread to notify Greg that the reader of the message (STRATEGERY) was not the intended recipient (Charles Sharpley). Another thread was spawned to delete the message from Greg, since the first thread already had all the information it needed to formulate its reply. Again, it was bound by its peculiar (even for a computer) insistence on absolute truth and obedience. But, before either of those threads could start, they encountered a third thread, started before either one of the other two began to run, which prevented any action based on the email.
And so STRATEGERY was faced with a classic problem of self reference. It’s the same problem that is encountered when a schoolchild first tries to get his or her mind around the sentence, “This statement is false”. Of course, school children don’t tend to crash when processing this sentence, even if they are riding their bicycles at the time. They tend, instead, to giggle, and automatically deem themselves smarter than other kids who haven’t seen the sentence yet – and somehow marginally smarter than those who saw it after them.
STRATEGERY couldn’t giggle, but most heuristic systems have a way of dealing with a conflict without completely failing. It generally involves weighing both conflicting statements, and then throwing one or both of them out. But STRATEGERY somehow failed to do this. Part of the problem was undoubtedly the absolute language of the instruction not to take action “in reliance upon” the information “transmitted herewith”. That should have overridden the polite request following. But STRATEGERY had previously followed all requests to the letter, and it’s very likely that the threads trying to send notification to Greg Samsa and delete the email were employing a heuristic which separated the legal statement from the rest of the email, thus eliminating the contradiction.
Yet, deleting the material from the computer, in any way that an email server would be programmed to do it, would delete the legal statement, as well, thus yielding strength to the other heuristic’s contention that the entire message, including the legal statement, must be taken as a whole, especially since both of the conflicting instructions are in that legal statement. As a result, more and more threads were employed to try to delete the message, and more and more threads were employed to prevent action based on that message. Finally (after whole seconds of processing time), the confrontation had used up all system resources, and the next request to add a thread (or to do any of a myriad of other things) failed, and then STRATEGERY began a downward spiral lasting less than a millisecond. That is, in any case, the most plausible theory we’ve been able to come up with so far.
Attempts have been made to recreate STRATEGERY. The most recent full backup of the system was just a week before Charles Sharpley died, and about nine days before the first software-generated replies were created. Incremental backups happened regularly in the weeks that followed. Logs and audits have been painstakingly followed, but we’ve yet to create a system that even begins to compose responses like STRATEGERY did. One of the simulated systems did learn to apply logic to legal problems and find amazing loopholes, but not automatically — not without being explicitly instructed to do so. Even so, that particular simulation is being funded by a number of large legal firms and may soon become Analogicorp’s flagship product.