Is Truth-Value A Strange Attractor?

"What is Truth?"

Formal logic is interesting because it captures both the essential qualities and the essential deficiencies of how human beings think. We have a natural tendency to draw sharp lines and draw (frequently binary) distinctions, while nature tends force us to revise the boundaries we use to draw human-navigable maps of the world. Logic is pragmatic: it makes sense, and yields results. However, like all practical expedients, logic is quite fallible, and often entails subtle complexities even in the pursuit of relatively simple goals. Some authors [3] have characterized logic as a way of thinking about thinking. This is a very interesting view. It means that if we soberly and seriously attend to what goes on in our logical constructions, we may learn something about how we think and what our thinking can and can't and tell us. This, however, means we must understand logic as a human construct with both human relevance and human imperfections.

There has been some excellent writing by some superb minds on the subject of formal logic, its relevance and its connection to informal logic, and I leave it to better experts than to elucidate this matter [5]. Instead, because I try to write as much as possible to the level of the lay-person, I would like to give a few simple and informally constructed examples to give the reader a flavor of basic logic, as a lead-in to a somewhat surprising, and unexpectedly colorful analogy between the abstractions of logic and the deceptively simple behavior of a certain class of phenomena that are both intuitively sensible and concretely physical.

Binary distinctions are everywhere in human thought: yes and no, up and down, light and dark, before and after, inside and outside, present and absent. They're a basic staple of how we see the world. It turns out, they're also a very efficient way to encode and store a lot of information. Consider the familiar game "Twenty Questions", where one player thinks of something and the other players try to determine what it is the first player is thinking of by asking him or her a series of no more than twenty yes-no questions. A few years back, some enterprising folks manufactured and marketed a electronic version of this game, packaged in a unit small enough to fit in the palm of one's hand. The game, and its clones, seemed to be astonishingly skilled at guessing what human players were thinking of -- provided, of course, they did not change the thought-of object midway through the game, or choose something extremely specific or idiosyncratic. The secret to the little gadget's success, however was no different than the strategy commonly used by human players: ask very broad questions at the start (e.g. "Is it an animal?"), and gradually narrow the scope until the set of possibilities is small enough to allow successful guessing (e.g. "Is it a cat?").

It appears that a large class of familiar things (and even many unfamiliar things) can be identified by a series of yes-no questions. This is, for instance, why Twenty Questions is not too hard to win, and not even too hard, with the help of modern technology, to implement as an electronic circuit. This observation is also at the heart of classical logic. In such traditional systems of logic, every proposition is either true or false (No exceptions!) and propositions can be connected using a few simple operators to express the truth or falsity of more complex expressions. In this context, 'operator' is perhaps an over-glorified word. The operations I am referring to correspond (tellingly) to very common words that are staples of ordinary reasoning about everyday things: 'and', 'or', and 'not'. Letting '1' stand for 'true' and '0' stand for 'false', we can succinctly express 'and', 'or', and 'not' in the following tables:

AND	0	1
0	0	0
1	0	1

---

OR	0	1
0	0	1
1	1	1

---

NOT
0	1
1	0

These correspond to basic intuition: if p and q are statements about something, p AND q is true only if the truth of p coincides with the truth of q. For instance, suppose p stands for 'eats grass' and q stands for 'says moo'. If we apply p AND q to a cow, then p AND q = 1 certainly, since we've seen cows eating grass, whence p = 1, and since everybody knows the cow says "moo", whence q = 1 as well. On the other hand, if we apply p AND q to a sheep, we have p AND q = 0, since sheep eat grass but generally have other things to say.

We don't, however, have to restrict ourselves to just conjunctions, disjunctions, and negations of simple true-false statements; we can use 'and', 'or', and 'not' to connect formulas to other formulas. Recursively, if P is any (arbitrarily complex!) formula, and Q is any other formula, we can connect them using any of our operators to get a new formula whose value depends on the respective values of P and Q in a way that respects our simple truth-tables above. The basic construction of logical formulae thus uses only very simple building-blocks; the formulae themselves, however, can become huge and immensely complicated.

An example is instructive. Suppose I come to you and say "I'm a secret super-spy!" Your first inclination might be, "Well, if it looks like a spy and it acts like a spy, it's a spy." That is, you might represent your belief that I'm a spy by the formula:

let
p = "looks like a spy"
q = "acts like a spy"
in
p AND q

However, after a while, you might think to youself, "Gee, I've never met a real spy before, so don't really know how a spy looks or acts", so you decide to refine your idea of the situation a little further. Spies keep a lot of secrets, so you decide that if I don't act secretively enough, I'm probably not a spy, or I'm not a very good spy, or I must really trust you to keep my secrets. You also decide that spies are pretty busy working for someone, and so I need to be off doing spy things as frequently as possible, and not ordinary stuff, and so your idea grows:

let
p = "looks like a spy"
q = "acts like a spy"
r = "is secretive"
s = "really trusts you"
t = "is busy"
in
(p AND q AND (r OR s) AND t)

After some further reflection, though, you realize that it's also possible that I'm deep undercover, and so even though I might be part of some super-secret operation, I might be going to great lengths to appear as if I'm leading an ordinary life so I don't blow my cover That means that either I'm deep undercover or I'm not telling the truth about being a spy, and so things get even more complicated:

let
p = "looks like a spy"
q = "acts like a spy"
r = "is secretive"
s = "really trusts you"
t = "is busy"
u = "is deep under cover!"
v = "is telling a tall tale ..."

in
(p AND q AND (r OR s) AND t AND (u OR (NOT v)))

If I regale you with tales of super-spy exploits, you'll have even more information that you'll have to take into account: if I say I was part of a secret plot to blow up Professor Nightmare's death-ray on Flaming Death Island, then that means that either I was part of a super-awesome adventure and it didn't make the news and death-rays exist, or I'm telling you a tall tale, in which case maybe I'm not trustworthy and I'm making the whole thing up:

let
p = "looks like a spy"
q = "acts like a spy"
r = "is secretive"
s = "really trusts you"
t = "is busy"
u = "is deep under cover!"
v = "is telling a tall tale ..."
x = "super awesome adventure!"
w = "none of the exciting news is ever fit to print"

in
(p AND q AND (r OR s) AND t AND (u OR (NOT v)) AND (x OR (w AND (NOT v))))

As you can see, things may get arbitrarily complicated, and as your idea of me as the super-spy depends upon more and more variables, you find your belief pulled ever more chaotically back and forth between amazement and incredulity. However, I don't have to cook up an incredible story in order to exhibit an instance of a phenomenon with simple parts and simple rules that nonetheless behaves in strange and wildly unpredictable ways.

Sitting on the Fence

"Chaos theory" broadly refers to a large area of research in mathematical physics that originated in the 1960s and flowered in the 1970s and 1980s as science turned its attention to physical systems that exhibit large changes in response to small variations. Like logic, chaos and non-linear dynamics are an active area of study with their own deep and fascinating literature, and so I leave it to those more accomplished to exposit their virtues and mysteries. (The interested reader might consult [4] for an informal but very readable overview of the field, and [6] for a more formal but equally readable presentation of the basic mathematics.) Instead, I would like to borrow one very simple device from the field, in the hopes that perhaps it leads us to a interesting analogy.

A physical system is typically said to be bistable if always tends toward one of two stable states as time passes. Such systems are interesting to non-linear dynamicists because, although they exhibit stability after enough time has elapsed, it is often very difficult to predict which state the system will ultimately end in. A classic example is a ball perched on a very thin divider, as in:





Common experience should be enough to convince the reader that the ball will always fall to one side of the divider, or the other. If the divider is relatively wide -- almost but not quite wide enough to allow the ball to be balanced -- the experimenter should be able to make fairly reliable predictions about which side the ball will fall to when placed. If the ball is sufficiently off-center to cause it to roll to the right or to the left, the imbalance will be visible at the outset. However, if the divider is narrow enough relative to the diameter of the ball, it will be very difficult to predict to which side the ball will fall, no matter how much is taken in placing. (The reader is encouraged to go play with some blocks, and so become really thoroughly convinced.) In this case, the very same physical forces are acting on the ball (e.g. gravity, the normal force exerted by the divider), but slight variations in how the ball is placed will be much harder to detect. As if that didn't make predictions hard enough, the many tiny irregularities in the ambient air currents or in the surface of the ball and the divider have a much larger proportional effect on the motion of the ball than they did when it rested upon a relatively wide divider. By making the divider very narrow relative to the diameter of the ball, a huge number of almost invisibly small variables become relevant to the final outcome, and much smaller inaccuracies in the initial placement of the ball may have a much larger impact on its motion and hence its final state, i.e. whether it comes to rest to the right or the left of the divider.

One interesting thing about this example is that predictions are easy when the relevant variables are few and the forces at work are large and easy to observe, but hard when many variables must be accounted-for and the forces at work obscure. Phrased this way, it doesn't seem excessively imaginative to note the wide-versus-narrow comparison made in the ball example is somewhat like the difference between judging the truth of "x is a cow", which requires relatively little information about relatively few features, and judging the truth of "x is a secret super-spy", which seems to require a great deal of information about very hard to discern features.

Getting the Ball Rolling

Suppose I wanted to construct oracle that answers simple yes-no questions. (Think "Magic 8-Ball", not "Delphi") The ball-and-divider gizmo described above is one very good candidate. All we need to do is let "left" stand for "yes" and "right" stand for "no"; if the ball rolls off the divider to the left, that means the oracle says "yes" to our question, whereas if the ball rolls to the right, that means "no". If we want the oracle's advice (in "yes-no" format, of course), we just utter the appropriate incantation, ask our question, then perch the ball atop the divider and see which side it rolls to. In keeping with the venerable old tradition of superstitious parlor games, we could keep the divider very thin, which would give the ball's motion an appropriately oracular irregularity. (Nobody likes an oracle that always say the same thing.) On the other hand, we could make the divider wide enough that the ball's motion would be easy to predict. (As long as we "clear our minds" sufficiently before playing, getting what we expect might make our oracle more suited to the company of the popular Ouja Board and the Magic 8-Ball, human psychology being what it is.) If our construction was precise and careful enough that we could control which way the ball rolled according to its initial placement, we would have something less like a Magic 8-Ball and more like a transistor.

The transistor is the textbook case of a bistable system. Very loosely speaking, a transistor acts as a material whose overall conductivity is "balanced atop" a semiconductor in a way that can be pushed either to conduct or resist an electrical current. This allows the transistor to be used as a two-state switch. (A light switch is a two-state switch, in that it is generally only "up" or "down".) The importance of semiconducting technology to the development of modern technology cannot be overstated; the invention of the transistor set in motion the explosive advance of the digital computer, which, at its most basic, is nothing more than a very complicated assemblage of two-state switches. Thus, our simple ball-and-divider oracle actually shares and interesting (And not coincidental!) kinship with a basic building block of the computer as we know it.

In its present, very simple state, the ball-and-divider oracle can be used as a machine that computes the answers to exceedingly simple yes-no questions. If we "ask" the oracle to check a statement we know to be false, we place the ball slightly to the right, so that it rolls of the divider onto the "no" side; if our statement is true, we place the ball slightly to the left. At this level, of course, the exercise seems silly: the little gizmo only does what you expect it to do. At the same time, in a world that's full of seemingly random occurrences and surprising things, it is actually extremely noteworthy when something behaves the way we expect it to behave. We can thus use our little oracle as a an external model of our internal judgements.

Nobody's going to be too impressed at a ball that rolls off a wall, but suppose we gave our construction a little more refinement and complexity. Suppose we have at our disposal a team of master craftsmen, and we ask them to modify our oracle as follows:



Essentially, our oracle now has a small replica of itself built onto its left and right sides. Instead of controlling our device by placing the ball, let's also ask our craftsmen to give us some way to modify at will the slope on top of the various dividers, so that we know which side the ball will roll to when it encounters a divider. (Perhaps each divider has a sloped piece that can be snapped on and off the top, so that the direction of motion can be reversed by turning the piece around.) Obviously, the measurements must be (excruciatingly) precise, and the device very carefully constructed, but if all goes according to plan, we can now ask our oracle more complicated questions. But how?

In the original construction, we assigned a truth-value (that is, 'true' or 'false') to each side of the divider. I chose 'left' for 'true' and 'right' for false, but we could have easily chosen the other way. In essence, the original construction corresponds to the simplest logical formula of all, namely, the formula with one variable and no connectives, e.g. just p. However, our new oracle now has two smaller copies of the original constructed into it. We can use this! Suppose I arrange the device so that the ball rolls to the left of the center-most (that is, highest) divider. After it rolls to left, it will fall a short distance and (if the device is correctly constructed) encounter another divider. If this second divider can also be arranged to direct the motion of the ball, we can make it stand for a second statement whose truth depends upon the first. For instance, suppose that we have a statement '(x eats grass) and (x says moo)'. We let the first divider stand for 'x eats grass' and let the two other dividers stand for 'x says moo' (we do need to use them both). If we adopt the same semantics for our two secondary dividers as we did for the first, i.e. if the left side of each stands for 'true' and the right side 'false', our device now computes not just p but p AND q:



It is no mistake that the labels along the bottom of the device correspond to the truth table given for 'and' in the above. We can similarly arrange devices that behave as OR and as NOT:






Now things will really start to take off, provided of course that we have the continued support of our craftsmen. Suppose that our team is able to construct machines with as many dividers as we please -- even up to very huge numbers --- and suppose that these machines have the same stable, predictable, reproducible behavior as the simple ball-and-divider construction we began with. The tremendous importance and difficulty of this stability to the function of the overall machine should not be underestimated and cannot be overstated. Small influences cannot be overlooked, and we must ensure that every possible force is very precisely accounted for in the design and construction of the machine. If we don't, there is no way that our thoughts can follow the bouncing ball -- its motion will be chaotic and random! (This challenge is not unlike tremendous effort that has been required -- and continues to be required -- to design and construct reliable solid-state electronics.) If, however, we can overcome the difficulties of physical construction, our little oracle will have grown into a programmable machine that, given a set of truth-judgments, can evaluate the truth of arbitrarily complex logical formulae. How might this be done?

Suppose we've overcome the construction challenges, and we can add as many dividers as we please. Suppose also that we have a logical formula with N many propositions (e.g. "x eats grass", "x says moo") and a collection of N many truth-judgments about our respective propositions (e.g. "it's true that x eats grass", "it's false that x says moo"). Starting with just one divider in the center of the board, we add 2^k additional dividers for each kth additional proposition. That is, we add 2 additional dividers for the first additional variable, 4 for the second, 8 for the third, and so on. We add the dividers in tiers of the same height, and associate to each divider a collection of dividers with the same height. Thus, for example, if p, q and r are propositions in our formula, we arrange our dividers as:



Note that our first tier, which consists of only a single divider, splits the plane of the device in half. The second tier, consisting of two additional dividers, taken together with the first, splits the plane in quarters. The third tier, consisting of four additional dividers, and taken together with the first and second tiers, splits the plane into eighths. Thus, for each proposition, we take a set of dividers all of which stand lower on the board than the last tier, and place them on the board so that split each segment of free space in half.

Now we want our machine to express some relation between the variables. That is, we want to program our machine with a chosen logical formula. This is done by labeling the slots in the board that lie between each pair of dividers. For instance, we obtained 'p and q' as well as 'p or q', both of which have two propositional variables, by changing the labels along the bottom of the board. If the ball falls into a slot labelled '0', this indicates that our formula expresses something false; if it falls into a slot labelled '1', this indicates something true. The machine is now configured in a way that models some logical formula with N propositional variables. For example:



Our machine is now programmed and ready to go. How do we give it input, that is, how do we ask it to compute the truth of our logical formula given a set of true-false judgments about the variables? This is accomplished by tilting the dorsal surfaces of our dividers, in order to govern which way the ball should roll when it encounters a particular divider. If we stick to our true-left, false-right convention (which we might as well, to keep things simple), we slope a given divider to the left or to the right according to whether the proposition corresponding to its tier is true or false. For example, if 'p' and 'r' are true but 'q' is false:



Now we're ready to go! Once we've gathered all of our information and set up the machine, we set the ball in the middle and just let it roll; the label attached to the ball's final state corresponds to the truth or falsity of our original statement.

(It should be noted how strongly the operation of our machine resembles that of the famous and very real "Plinko" device, which was prominently featured on the popular game show "The Price Is Right". The important difference between Plinko and our machine is that Plinko seemed to purposefully admit a fairly high degree of random behavior, as evidenced by the conspicuous bounciness of the pegs. This seemed to be an important part of both its appeal and its challenge.)

If all goes well (Does it ever?) our machine can successfully automated a potentially very elaborate arrangement of logical judgments. Assuming that the work in setting up the machine is not too difficult or time consuming (which, in our example, it almost certainly would be for all but the most trivially simple formulae), we can now model our basic truth-judgments as we understand them and apply them to very complex situations that would otherwise be humanly impossible to reason through. This is effectively what makes the modern digital computer so powerful and useful: it can apply our own basic "and/or" and "not" intuitions to arrangements that are complicated vastly beyond our own human cognitive abilities. Unlike our Plinko-like machine, however, an electronic computer is much easier to program and will execute much more quickly, though, again, this difference is relative and not at all absolute. Any computer programmer or engineer of even modest experience will most certainly agree. (Despite the computer comparisons, I cannot resist a technical note that our deterministic Plnko-machine is not a Turing Machine, since it lacks a random-access memory.) These differences aside, our imaginary machine is now much more than silly toy: it is a very complicated system that behaves the way we expect it to. Again, given the dear scarcity of things in life that we can expect, this very noteworthy.

In the same way that a bridge allows us to use ordinary human ambulation to cross expanses that we could not traverse on foot, our machine is a tool that allows us to apply our ordinary reasoning to systems whose complexity is far beyond human comprehension. This is all good and fine but (again inviting the reader to go play with some blocks), the bigger bridges get, the harder they are to construct, and the same law of increasing difficulty applies to the size and complexity of the ball-and-stick logic machine.

"Truth is stranger than fiction."

If Newton stood on the shoulders of giants to see as far as he did, it would seem that none of us are in a position to refuse the invitation for a boost above eye-level by someone of greater stature. So, let's push our little logic machine its logical conclusion, using a venerable old trick: limits at infinity.

In the preceding section, we saw that we could use our little logic machine to compute logical statements in as many variables as we pleased, simply by adding enough dividers. As we add variables, a forest of dividers springs up on the plane of our device, becoming (exponentially!) more dense with each variable we add, so that this:



rapidly becomes this:



(Actually, with better artistic skills and more powerful drawing tools, this would
come out something like a collapsed Sierpinski Gasket.)

As the number of dividers increases, the outline of our device rapidly converges to a pair of continuous slopes in opposite directions, both leading down from the same elevated point. A few features, both abstract and concrete, are immediately apparent.

Firstly, the functional details of our machine are now humanly unmanageable. How are we supposed to read infinitely small labels, or manipulate infinitely small parts? The parts do not even need to be infinitely small for the machine to become impractical; they just need to be very small relative to human eyesight. However, the machine is only really interesting if it can handle formulae with lots of variables, which necessarily entails exactly this kind of tortured, inhuman precision. If we wanted badly enough to use the machine, we could construct still other machines to program and use it, etching ball-slots with lasers, or watching the progress through a microscope. Be that as it may, even these methods have limits, and the density of trajectories within the machine grows explosively with each additional variable.

Secondly, the size of the ball that actually runs the machine now matters. An infinitely small ball (that is, one that has zero diameter but somehow "still exists") could be dropped on the machine, somehow falling through the infinitely dense forest of dividers to come to rest in an infinitely small notch, thus evaluating the truth or falsity of a logical statement somehow depending upon an infinite amount of information. Of course, this always mattered, but we glossed over it in the construction. The ball obviously must be large enough to fall through the space between dividers. This means, however, that computing a logical formula with more variables requires adding more dividers and hence requires a small ball. Not only that, the size of the ball must shrink exponentially as the number of variables grows. Basically, there is a vicious dependency between components of our machine. This dependency moreover tightens (Worsens?) as the machine grows in power. The ball we drop will rapidly shrink to the size of a dust-speck as we increase the informational output, no matter whether it starts out the size of an orange or the size of a planet.

There's more. Notice that any sufficiently large ball (really, any ball with a diameter greater than zero), when dropped into our infinitely powerful logic-engine, will appear to roll all the way to the left or all the way to the right. Moreover, because our dividers must be very (i.e. infinitely) small in order to fit onto a finite board, the starting divider which sits at the center will always be very small relative to the diameter of the ball. But that means that our infinite logic-engine, when used with a finite-diameter ball, will actually behave exactly like the very chaotic ball-and-divider construction we started with!

Suppose our board has finitely many dividers, but the space between them is sufficiently small relative to the diameter of the ball. When we try to run our logic engine, the ball will eventually reach a stable state but will fail to fall all the way to the bottom of the board because the spaces below will be too narrow for it to fit. What this means is that, given a ball of a certain spatial extent R, there is a limit to the number variables that a given (finite) logic-machine M can model, and hence to the number of logical formulae it can compute. Moreover, the upper bound on the number of variables in the formulae M can compute is proportional R! Essentially, a finite-variable logic-machine is approximated by an infinite-variable logic machine using a ball with non-zero diameter. What happens in such a case is that the ball simply settles into a rut somewhere between the central divider and one of the edges of the board. Everything below this height corresponds to the "random noise" that is assumed away by the modeled formula.

What does all this mean? Infinite information looks like randomness, and classically logical systems can only model systems with finitely many variables, where the difficulty of model-construction grows exponentially as information about the system is added. Somewhat pretending to understanding, this vaguely resembles a certain theory of Chaitin's [2]. Without pretending, this simple, mechanical analogy greatly resembles familiar human reasoning, in which too many details makes things fuzzy, so that hypotheses that require too much information to evaluate are indistinct from an uninformed assertion of "random" behavior.

In an alternate construction, we might notice that infinitely small labels along the bottom of our infinitely dense board somewhat resemble a Cantor dust or a one-dimensional Julia Set [1]: a ball might roll arbitrarily close to the "true" side of the board but come out false, or arbitrarily close to the "false" side and come out true. Thus life always manages to surprise us, and categorizations always prove intractably blurry about the edges, no matter how logical we choose to be.

Logic thus gives us a bridge beyond some of the limitations of working memory and attention span, which mechanism moreover can extend. However, this extent quickly and easily collapses under its own weight when stretched too far, in exactly the way that a real bridge does. Either the ideas we consider abstractions are very much like our supposedly "concrete" perceptions of the "real world", or supposedly abstract constructions, no matter how empyrean or pure, are subject to the same noisy unpredictability of the physical world we actually inhabit.

---

[1] Barnsley, Michael F. Fractals Everywhere. Academic Press Inc., 1988.

[2] Chaitin, G. J. Algorithmic Information Theory. Cambridge University Press, 1987.

[3] Ernst, Zachary. Free Logic Now! Available here.

[4] Gleick, James. Chaos: The Making of a New Science. Penguin Group, 1987.

[5] Quine, W. V. Philosophy of Logic. Harvard University Press, 1950.

[6] Strogatz, Steven H. Nonlinear Dynamics and Chaos. Perseus Books Publishing, 1994.

When Is It Time to Turn Out the Lights?

I happened to be reading "The Illusion of Conscious Will" by Daniel Wegner (which I highly recommend -- it is an excellent book) and in the text I came across the following excerpt from a book by Julian Jaynes:

Consciousness is a much smaller part of our mental life than we are conscious of, because we cannot be conscious of what we are not conscious of ... How simple that is to say; how difficult to appreciate! It is like asking a flashlight in a dark room to search around for something that does not have any light shining upon it. The flashlight, since there is light in whatever direction it turns, would have to conclude that there is light everywhere. And so consciousness can seem to pervade all mentality when it actually does not.[3]

On reading this, I immediately thought to myself that the solution is, in fact, very easy: turn the flashlight off.

This reminds me of a certain other venerable old quotation:

It is said that things coming in through the gate can never be your own treasures. What is gained from external circumstances will perish in the end. However, such a saying is already raising waves when there is no wind. It is cutting unblemished skin. As for those who try to understand through other people's words, they are striking at the moon with a stick; scratching a shoe, whereas it is a foot that itches. What concern have they for the truth?[2]

This is the interesting thing about carrying a light with you: it is an invitation to explore deep and extremely dark places. This makes me think of the last time I ventured into a cave. (Missouri is, after all, "The Cave State". Because it's like living in a cave?), I went deep enough inside that I needed a light, and shining it about I saw all manner of strange and wonderful things. Still, I could not get past the sense that I was in some sense an anomalous occurrence in that place, if not an intruder outright. It's true, there were things before my eyes that I could perceive and understand, but they were made perceptible and understandable by my own planning and device. It's true that there's genuine information in a beam of light, in both the technical and the colloquial sense of "information". Moreover, such information would not be unavailable otherwise. Even so, investigating what's in the dark by getting rid of the inconveniently dark part seems very unlike the fabled objectivity that classical science strives to attain.

Without admitting romanticism or mysticism, we already know that the Universe contains at least two kinds of knowledge: things that are computable, and things that are not. How many other divisions may there be besides? If we admit that the mind is a physical process and not some magic causeless cause that can create information ex nihilo, there surely must be knowledge that it cannot produce, or even that certain of its subfunctions are unable to produce. (Aphasias are one such interesting and highly celebrated case.) This seems like a little explored and very challenging but very interesting domain for scientific investigation and philosophical inquiry. On the other hand, it might turn out that this is a domain wherein scientific investigation cannot possibly answer the questions we are trying to ask. What then?

We acknowledge that 'knowing' is a physical process, caused by physical processes. Still, we intuitively think of it as a "meta-process", that pertains to certain other processes. The time may yet come when we have to collapse all of the "meta-" distinctions in our modes of thought into a unified whole. A science that successfully transcends all barriers to knowledge will have to remove the distinction between the events in our brains that constitute "knowing" and the events elsewhere that seem, perpetually, to confront, confound and challenge us as human beings.

I realize that this is a very bold and very sweeping proposition, requiring a very unwieldy camel to be threaded through a very delicate needle. Even so, there seems to be a deep conceptual incongruity in our macro-scale world knowledge, and even our personal self-knowledge, that we habitually lean upon but seldom acknowledge. If the vivid sense of identity and "there-ness" that each human being experiences is dependent upon and essentially the same as phenomena in the world at large, why is the feeling of difference and identity so strong? This is the essence of the so-called Hard Problem of cognitive science, and of such puzzling questions as Hofstadter's "identical human copy" problem [1], and no doubt of many troubled personal introspections by many people. I would argue, however, that the question has a much broader, transpersonal significance: Are there facts that are scientifically unknowable? And have we reached a stage in the progress of science where such a question may be regarded as serious and respectable? Perhaps, perhaps not. If not, it seems science is moving more slowly hoped. Even so, this isn't a victory for romanticism either. The romantics, for all their rhapsodizing about essences and sneering at analytic problem-solving, have yet to give any clear, convincing, and accessible explanation of of the key practical point:

When and how do we turn the light off?

---

[1] Hofstadter, Douglas. I Am A Strange Loop. Basic Books, 2007.

[2] Mumon's Introduction to The Gateless Gate, Katsuki Sekida (trans.)

[3] Wegner, Daniel. The Illusion of Conscious Will. MIT Press, 2002.

Editor's Notes for October 2009

I've done a great deal of writing lately but, much to my chagrin, none of it has been here. There is good reason for the digression (proposals, submissions), but I am troubled by the appearance of neglect or lifelessness in this corner, and so, to avoid this, a few passing points of interest and some encouraging news:

1. Return-Oriented Programming: a colleague recently pointed out to me an extremely interesting finding reported at last year's Black Hat USA Briefings. The authors present a exploit that allows them to perform arbitrary computation without injecting code. If an attacker can gain control the stack, the authors show, he can influence program control flow to link together certain segments of code into instruction sequences of his choosing. This is very neat because it is one of those things that seems as if it should possible in principle. However, we so often hear in computer science about humanly impractical things that are "possible in principle"; here is someone who has taken the principle and successfully used it. Well done!



2. Iterated Function Systems and Control Languages: I was very pleasantly astonished when, in the course of combing the literature aimlessly, I discovered the recent work of Henning Fernau and Ludwig Staiger, which appears to be a link between automata theory and fractal geometry. Intuitively (and offering no justification other than that), I feel as if such correspondence makes sense, and I admit to being thrilled that skilled mathematicians have undertaken to actually investigate such a correspondence. This also suggests the surprising possibility that abstract computer science might eventually have substantive theoretical contributions outside of its own domain. Well done!



3. Why doesn't Wendell Berry own a computer, and why don't I own a mule?



4. The Liar Paradox Redux: It occurred to me yesterday that the situation I constructed in the Truth Machine thought experiment was actually an elaborate variant of the famous Twin Paradox. The paradox is not really a paradox so much a puzzle, stated as follows: you meet a pair of identical twins, one of whom always tells the truth, and one of whom always lies. What single question can you ask them to determine which is which? (I would also note that this familiar paradox appears to be a favorite intellectual toy of accomplished animator Genndy Tartakovsky, as the puzzle and its solution has appeared in episodes of both The Powerpuff Girls and Samurai Jack.) That's not to say that there isn't a great deal more to the Truth Machine example than there is to this familiar puzzle, only that I hadn't realized the connection before. Interestingly, it appears that I unwittingly attempted to construct something like the solution to the Twin Paradox as one stage in the elaboration of the Truth Machine.



5. A Second Editor's Note on the Truth Machine: Another thing I had not realized until recently is that The Truth Machine is also the title of a novel by science fiction writer James Halperin. It appears that Mr. Halperin's truth machine is not quite so general in its function, acting in the restricted (but still remarkable) role of an infallible human lie detector, with horrible dystopian consequences. Why is it always the case that catching falsehoods leads to so many unforeseen complications? This seems like a question worth answering.



6. What happens to the story of Job if you reverse the "good" and "evil" roles, i.e. make Job an extremely wicked man who only persists in evil because it's so materially rewarding?



7. Is there such a fallacy as "appeal to chaos"?

I'm afraid that's all the time we have for now. Until next time!

A Concise Refutation of the So-Called Objectivist Philosophy

Ayn Rand and her followers acknowledge that certain exercises of power may infringe upon the basic libertarian rights of others, but fail to either recognize or to acknowledge that economic power may be concentrated and abused in exactly the way as brute strength or cunning deception. For all their talk about the property crimes of tyrannical governments, these would-be philosophers fail to acknowledge that private entities are capable of exactly the same kinds of abuses as public institutions.

So-called "objectivism" has all the marks of a bad abstraction. The entire body of thought fails not because of what it incorporates, but because of the important issues that it leaves out.

The Truth Machine: A Thought Experiment

---

A Beautiful Dream

Suppose, one day, you stray off your usual path and wander off into some out-of-the-way place. After a time, you realize you're lost. Looking around for someone who can show you the way back, you come across me. You're about to ask me for directions, but then you notice that in front of me is a fantastic machine the likes of which you have never seen before.

Naturally, you ask me, "What does that machine do?"

"It's a truth machine," I say.

This sounds outrageously impossible, but you're intrigued enough that, with some coaxing from me, you put aside your objective of getting directions and decide to ask the machine a question. So you think of an easy question and we feed it into the machine. Lo and behold, it comes back true.

"Ah, well that was too easy," you say.

"Well, ask as many questions as you want," I reply.

So, we proceed to feed some harder questions into the machine. Each one seems to come back with a truthful and accurate answer. Emboldened, you start to ask the machine some really wild and speculative questions. These come back with answers that are far too big to verify right away, but you're really intrigued now, and you want to make sure this isn't a scam or a prank.

"How does it work?" you ask me.

"It's very simple," I say.

I proceed to give you a thorough and convincing explanation of the basic scientific principles at work, and the engineering methods I've applied to them in order to construct the machine. (The details of this explanation are left as an exercise to the ambitious reader.) It becomes clear that the machine is not just a high-class Mechanical Turk, or a nice bit computational ventriloquism on my part. Moreover, there's no sorcery, no divine intervention, and nothing weird or unexplained. The workings of the machine are unambiguous consequences of straightforward and readily accessible scientific facts, and they are all perfectly clear to you.

"That's amazing!" you say.

You're overcome with a wave of excitement and decide to get back to the rest of the world so you can test out the machine's answers to your really difficult questions. I give you directions back to the beaten path, and you go on your way. Perhaps, even more interestingly, it is the Machine that gives you the directions.

I might try to tell the rest story but, as wild and fantastical daydreams go, the uses and consequences of the Truth Machine would far surpass even the Machine itself. Would all those beautiful utopian dreams of the Enlightenment finally come true? Or would we turn our knowledge destructively in on ourselves and extinguish the entire universe? Would truth become another commodity to be bought, sold and advertised on the market? Or would the accumulated knowledge of humanity finally be free to anyone curious enough to ask? I won't speculate, because speculation is not my purpose here.

Instead, consider a slight variant of the scenario above.

A Variation on a Theme

Suppose that everything happens as before; we meet, and I demonstrate the Truth Machine. After you've had your fill, we part ways, but you have so many more questions that you decide to come back the next day. You find me and the Machine in the same place, and can hardly contain your excitement.

"I have more questions," you say. "Can I ask the machine?"

"Well, I'd love to oblige," I reply, "but it's too hot today. The heat will interfere with the way the machine works, and it will give false output."

I go on to explain why this is, and it's immediately clear that this is not only consistent with the previous day's explanation, but a necessary consequence of the way the machine works.

You turn to leave, but you're very clever and perceptive, and it occurs to you that this kind of situation often gives rise to paradoxes or infinite regressions, or other such weirdness. Deciding to really put the machine to the test, you turn back to me and propose the following:

"Suppose I ask the machine a question today, take the answer I get, and come back on some day when the machine is working and ask 'How would the Truth Machine answer such-and-such question on a day when the temperature such-and-such many degrees?' If you really do have a truth machine here, and if the ambient temperature does affect its function, then even though the machine answers the question wrong today, it should be able to reproduce and explain its wrong answer later, on some day that it is working."

"That's true," I say. "Why don't you ask your question, write down the answer, and come back later when the machine is working again."

And so, you ask your question and write down the answer. A few days later, you come back, and together we ask the machine how it would respond to your earlier question on a hot day. Much to our surprise, nothing weird or paradoxical happens: the machine correctly reproduces its wrong answer, and gives an accounting for its failure. We're both so thoroughly amazed that you become totally convinced that the Truth Machine works as promised, and I suffer a minor injury from patting myself on the back for my accomplishment.

Some Casual Dream Analysis

There are several ways to read this variation of the story. The story's description of the machine replicating and explaining its own mistakes is something like what happens when one scientific theory subsumes another: a new scientific theory must always explain why its predecessor seemed to be true. This is one interesting feature.

Another interesting feature is brought out if we continue to vary the story by adding additional constraints on the correct function of the machine. For instance, suppose that the machine is sensitive not only to temperature but also, say, electrical fields so that a passing storm cloud might disrupt its function. We can go further and suppose that the machine is sensitive to temperature, electrical force, and perhaps noise. Maybe certain levels of background radiation have an effect. Perhaps very slight changes in gravitational force at the point on the Earth's surface where we are running the machine, due to elevation or the moon's orbit, may also disrupt its function. This doesn't seem unreasonable; after all, the Truth Machine must be extremely delicate. It would seem that we could go on adding necessary constants indefinitely (if you say that we couldn't, then it seems you know more about truth engineering than me) until running the truth machine and getting a valid response requires carefully planned and controlled conditions. Of course, the machine still gives us correct answers; it's just that we must be very careful that exactly the right external conditions are in place to ensure that all the necessary processes within the machine work as expected.

What do we get if we add enough constraints? At some point, the Truth Machine will start to look less like a stunning technological oracle and more like an ordinary experimental apparatus, in the spirit of Boyle's air pump or the Large Hadron Collider. Of course, our fictional construction here differs from typical experimental apparatuses in that it seems we can pose it any question we please and, given the right circumstances, get a correct reply. By contrast, we may be able to ply the LHC for answers about elementary particles or black holes, but it is not immediately apparent how it could give us direct answers about how animal cells specialize into organs, whether the Navier-Stokes equations are a faithful model of fluid dynamics, or how exactly a pair of sociopathic teenagers burning a cat to death in an Internet video could give rise to the NEDM phenomenon. This is an important distinction. Most experimental machines are constructed to answer one specific question, or a specific class of questions.

(On the other hand, if it's perfectly clear to you how the details of elementary particle theory relate to Internet memes, you should really be talking to the Nobel Committee.)

What I've sketched out here is something like the outline of traditional scientific knowledge. Truth comes from relating reproducible phenomena that we do understand to other phenomena that we know less about. In this case, we leveraged what we knew about the workings of the machine into new information that we didn't know. I've purposefully resisted the strong temptation to involve exotic, modern abstractions like algorithmic information theory (Would our machine have to contain all the information in the history of the Universe?), or classic tropes like a Turing-style non-termination scenario (What happens if we ask the machine to predict the future of its own execution?), but not because I think such considerations are irrelevant. Mathematical logic has produced some truly strange and remarkable artifacts in the last century, and as deserving of recognition as these are, it is not immediately clear how they relate to the physical world as we know it. If I brought such abstract machinery into play here, I feel that I would be admitting unfounded back-door assumptions about how the physical world in general, and Truth Machines in particular, must work.

However, I'm not done just yet. Supposing that we do obtain a context-sensitive Truth Machine as described above, is the only work left for the thinkers and discoverers of the world to work out a calculus ratiocinator and start dreaming Leibniz's dream?

Dream On, Leibniz

We meet, I demonstrate the Truth Machine. I explain that it doesn't work on hot days (or stormy days, or at high altitudes, etc. etc.), and you're satisfied that this is indeed a necessary consequence of the appropriate Laws of Nature. As before, you ask the Machine a question on a hot day, in order to really put the Machine to the test by checking the consistency of its answers. Realizing that it may be important what question to ask, you think for a while, and then come up with something.

"Let's ask these two questions," you say. "(1) Is A true? [where A is any meaningful proposition] (2) How will a truth machine answer the question 'Is A true?' under conditions suitable for a correct answer'"

"Well that's a weird question," I say, "but alright. Let's feed it into the machine."

We ask the Machine your question. The Truth Machine gives the following answer: "A is false [without loss of generality, since we can transform any true proposition into a false one by negation], and under conditions suitable for a correct answer, a truth machine will find that A is false." I suggest that you write down the answer and come back tomorrow when the machine will be working again, but you're not satisfied just yet. You go on to ask the machine another question:

"What are necessary conditions for a truth machine, as constructed here, to give correct answers, and why are these conditions necessary for a correct answer?"

The machine responds by describing exactly the physical conditions under which it is operating at the moment. It goes on to give a response that explains, at great length and with considerable detail, why these conditions are essential to producing a correct answer. You and I both seem to understand the response, and after looking over it carefully, neither of us is able to find any internal inconsistency to it. Moreover, the truth or falsity of A is not immediately verifiable without the Machine's help.

"Well," I say, "the machine isn't working right anyway. Of course it would say that. Come back tomorrow, and we'll see what it says."

The next day, you return and we repeat your questions to the machine. According to my earlier claims, conditions are favorable to truth machines that day, and the machine gives the following response: "A is true, and under conditions suitable for a correct answer, a truth machine will find that A is true."

"There you have it," I say, " I guess A is true after all."

But you've become a little more skeptical now. You decide to repeat your other question as well, and you ask the Machine a second time:

"What are necessary conditions for a truth machine, as constructed here, to give correct answers, and why are these conditions necessary for a correct answer?"

As before, the machine responds by describing exactly the physical conditions under which it is operating at the moment. It goes on to give a response that explains, at great length and with considerable detail, why these conditions are essential to producing a correct answer.

"And that's what I said before," I volunteer. "It's all settled."

But you're really troubled by all this now. A shadow's fallen over the enterprise, and although you're not prepared to dismiss me or the Machine out of hand, something seems unaccounted-for.

"How do we know that it wasn't working correctly yesterday?" you say. "Maybe you're wrong about the external conditions, and the machine is actually malfunctioning today."

"But who do we trust? You? Me? The Machine?"

"Brother, let us reason."

Whose claims should we trust? Presumably, we're all interested in nothing more than getting to the truth of the matter. We've all sworn "Calculemus!", true to the spirit of ol' Leibniz, and we're willing to trust the Machine so long as we can verify that it's working as expected. Still, there is a clear problem of which source of assertions to trust. The Machine throws things out of balance because, presumably, it is a new and very large source of information that has come into conflict with a body of information that, as far we know, is well-founded and correct.

This conflict is distantly reminiscent of Poincare's skepticism toward Frege's project of formalizing all knowledge and placing it on a clear and unambiguous axiomatic foundation[1]. Frege seemed to believe in the possibility of realizing a genuine calculus ratiocinator, whereas Poincare argued that the soundness of such a system could never be empirically proven, since it would require the verification of infinitely many theorems. One could always argue for a sort of "generative proof" of the sort one sees in formal logic: show that the axioms of the system do not conflict with one another, then show that it is impossible for any application of the proof rules to produce a contradiction, whence any theorem proved from the axioms cannot be a contradiction. This is fine for an abstraction but, Poincare points out, our abstraction is of interest only to the extent that it reflects some material reality, and no proof of internal soundness can guarantee the faithfulness of a model. Phrased another, we don't know if or when an empirical contradiction might jump out and surprise us.

How can we decipher the conflicting responses of our troublesome Machine? The problem arises because the Machine seems to have distinct and incommensurate frames of reference: it answers the same question one way on one day, and the other way on a different day. One way we might get to the bottom of all this would be to keep plying the Machine with questions, both on "on"-days and on "off-"days to see if we could catch it in a contradiction. If the machine gave a reply that was plainly false to us, the observers, or that contradicted an earlier response we would know that the external conditions under which the machine was operating could not be the right circumstance for truth machines to be truthful. However, there seems to be no way to tell how many questions we might have to ask the machine before we caught it in a contradiction; we might question the Machine until the end of time and never see anything but self-consistency.

As long as the machine continues to explain its own operation and to do so in a way that reveals no contradictions that aren't directly tied to changes in operating conditions, we have no reason to believe that its answers are anything other than the necessary consequences of essential Laws of Nature.

Worse still, there's no way we could ask the machine for help in the matter, since we've already cast doubt on its judgments.

Who do we trust? You? Me? The Machine? And how should we understand the Machine's explanations of its own contradictions?

Though I won't pretend that these questions are impossible or intractable, I also don't see an obvious answer. There is, however, one last variant that may say something important, though it may not resolve this particular dilemma.

Ground Zero

Recall that in the paragraphs above, we took a completely idealized and perfect Truth Machine and replaced it with one that was affected by its physical operating conditions. We observed that we could elaborate a succession of factors that might all contribute to the machine's correct operation, and that so doing lead to something reminiscent of an ordinary experimental apparatus. In schematic, we have a succession like so:

(1) Given input A, the machine produces output B and B is true.

(2) Given input A, the machine produces B and B is true if the machine doesn't get too hot.

(3) Given input A, the machine produces B and B is true if the machine doesn't get too hot and if the machine is insulated from strong electric forces.

...

And so on, until we've taken into account all of the factors that might affect the Machine's operation. This is really a sequence of refinements to our idea of of the variables upon which the machine's output depends. That is, we could look at our succession as:

(1) The machine's output depends on its input.

(2) The machine's output depends on its input and the ambient temperature.

(3) The machine's output depends on its input and the ambient temperature and net electric forces acting on its mechanisms.

(4) The machine's output depends on its input and the ambient temperature and net electric forces acting on its mechanisms and ...

And so on. Looked at this way, it seems that there's one variable that we've forgotten, in our rush to account for necessary operational constants. Consider what we get if we take a step backward from (1):

(0) There is output.

Even the ideal machine we started with had one factor that affected its operation, namely, the question that its user asked. We began to doubt the Machine once its operating conditions allowed it to give different answers and also to account for them. However, we would in general expect our machine to give different answers to different questions. (Unless, of course, it just "holds up a finger."[2]) If we hold that (1) humans are physical beings, and (2) their actions comprise physical phenomenon, and (3) asking a question to the Truth Machine is a physical phenomenon, then since (4) the Machine's output depends upon the question it is asked, it must be that a question is an external physical condition affecting the operation of the Machine. This leads us to one last question:

Is is it function or malfunction for the Truth Machine to give different answers to different questions?

---

[1] Poincare, Henri, Francis Maitland (Trans.). Science and Method, Barnes & Noble Publishing, 2004 (original 1908).

[2] See case 19 of The Blue Cliff Record

Outputs: When There is Madness to the Method

Considering that algorithms are human artifacts, it is very interesting that there even is such a thing as a bad algorithm. (It's also very interesting that apparently it is possible to make a joke in the form of an algorithm.) The phenomenon would seem to tell us something about the "essence" of an algorithm since even a severely deformed sorting algorithm is still recognized as a sorting algorithm. Consider a classic example:


START: given an array A[1..n]

STEP 1: IF (A is in sorted order) THEN [HALT] ELSE [GOTO STEP 2]

STEP 2: DO [randomly shuffle A[1..n]]

END


, which is more commonly known as "bogo-sort" and can be traced back at least to 1984[1]. There are some amusing quantitative analyses of such algorithms[2], but what stands out to me is that this can be called a sorting algorithm at all. I'm not disputing that bogo-sort is indeed a well-defined procedure, or that it eventually sorts its input array. What seems noteworthy is that we have here something that really is a sorting algorithm, but only just barely.

The only sense in which bogo-sort is a sorting algorithm is that, if and when it stops, the input array will be sorted. If we were to change the condition in STEP 1 to anything other than "Is A sorted?", we might say we have some kind of algorithm, but it would certainly not be a sorting algorithm. Put another way, we could change the condition to (almost) anything else and get an algorithm that is not a sorting algorithm. Bogo-sort is nothing other than a slightly constrained version of a very generic and very disorderly procedure (i.e. moving things around at random); its "sorting" is really nothing other than a discrimination applied to an otherwise indiscriminate process.

Perhaps another appropriate (but less snappy) name for this procedure would be "tautological sort". The array is sorted when the program says they're sorted, or rather, when the programmer says the program should stop. By way of analogy, this would be something like "straightening up" a messy room by going in and hurling objects about at random, and stopping whenever it was that you declared the room to be clean. ("Mission accomplished!") In this case, the output makes the program, and the notions of the programmer make the output. Maybe this is not especially surprising in and of itself, but it makes for a stretch of thin and blurry line separating the orderly and the chaotic, or the desirable and the undesirable.

The authors who have already commented on this curiosity raise a class of interesting questions: for any given objective, what is the worst possible means of attainment? ("The Pyrrhic Problem") This is worth asking if for no other reason than to point out that there are exceptionally bad ways of attaining almost any object. Although it seems little more than an academic curiosity when applied to algorithms, the question becomes strangely illuminating when applied to any number of real-world enterprises. Consider, for instance, how useful toilet paper is, but consider also that huge swaths of forest are cut down to produce it. It seems not only that are ends sometimes used to justify means, but that utter chaos circumscribed by an end can become a means.

---

[1] Broder, Andrei, Stolfi, Jorge. "Pessimal Algorithms and Simplexity Analysis", ACM SIGACT News, 16:49-53, 1984.

[2] Gruber, Hermann, Markus Holzer, Oliver Ruepp. "Sorting the Slow Way: An Analysis of Perversely Awful Randomized Sorting Algorithms", FUN 2007, LNCS 4475, p183-197, 2007.

Recent UAV Incidents and the Human-Computer Problem

Over the weekend, the Association of Computing Machinery (ACM)'s RISKS news digest passed along a report about a United States Air Force (USAF) drone that became unresponsive to operator commands and was accordingly shot down by a manned USAF aircraft before it could cross the Afghan border. It's unfortunate that ACM RISKS chose to package a rather lightweight and sensational report that seemed to insinuate that the drone may have become self aware, and ended with a provocative quotation about the drone being motivated by its feeling of being "sicked by reaping hapless fleshies."

The USAF's official statement on the incident is unsurprisingly terse, and an investigation is still pending. However, there seem to be no serious grounds for supposing anything like self-realization or rebellion (compassionate or otherwise) on the part of the drone, and there are several logical and factual problems with even suggesting this to be the case. Most seriously, there is no good reason to believe that a drone designed and built to be controlled by a remote human operator would have anything more than minimal capabilities of autonomous operation. Giving the machine unnecessary autonomy of any kind would be counterproductive to whole aim of building and operating an aircraft with a remote human pilot. More to the point, its seems excessively whimsical to suppose that such a highly constrained and special-purpose computer-control system could possess the richness and complexity necessarily underlying the spontaneous emergence of such a miraculous and unprecedented machine awareness. As far as becoming sick of "reaping hapless fleshies", I would hope that the human operators would take it upon themselves to conceive such a sensibility, rather than offloading the work of ethics onto a machine.

For the record, I believe that machine intelligence is entirely possible, even probable, and I do even admit the possibility of its spontaneous emergence. What I object to is that suggestion that a special-purpose machine containing the same sort of embedded computer systems common to most modern aircraft and designed to be controlled by a human pilot, albeit a remote one, would suddenly be the first machine to become not only willful, but conscious enough to be compassionate or vengeful.

As is always the case, though, truth is stranger than fiction. This most recent failure of a U.S. military unmanned aerial vehicle (UAV) has some striking features that tell us, surprisingly or unsurprisingly, that we have more difficult problems with ourselves and how we use our machines than we do with the machines themselves. Five years ago, the Federal Aviation Administration (FAA) compiled and analyzed all information available on accidents involving U.S. military UAVs (there are no civilian UAVs in widespread use, that I know of), and found that the MQ-9 Reaper (also known as the MQ-9 Predator B), the UAV most commonly used in current U.S. operations in Afghanistan, suffered a noticeable preponderance of accidents due to human error[1]. In particular, operator difficulties with the poorly designed interface used for the remote control of the Reaper were cited as contributing factors in just under half of all reported accidents. These included an improperly executed attempt to transfer control of the UAV between ground control stations that resulted in the turning off of the aircraft's engines, and another episode wherein a pilot accidentally executed a routine that erased the random-access memory of the control computer while the UAV was in flight. The FAA report cites another source claiming that a sequence of keystrokes used to control the lights on the Predator UAV is almost the same as a sequence that cuts the aircraft's engine. Anyone who has ever played a video game knows how easy it can be to hit the wrong keys and get your computerized proxy metaphorically killed as a result. It would be naive to suppose that such mishaps are impossible just because the proxy is a $53.3 million[2] aircraft and the killing is quite literal.

I should note that the most of the other UAVs reviewed in the FAA report showed a much lower incidence of human error attributed to accidents, and these could usually be localized to a single eccentricity of the particular aircraft that made its operation counter-intuitive. (For instance, having to a turn a knob to the left in order to make the aircraft turn right.) However, this only seems to emphasize how the generally poor quality of the Reaper interface contributes to accidents. The episodes described by the FAA report sound eerily reminiscent of the infamous Therac-25 incidents[3], wherein a minor confusion at the interface between human and machine was all it took for someone to be seriously injured or killed. Perhaps we should worry not so much about the consciousness of our machines as about the consciousness of ourselves.

It's noteworthy that the wayward UAV was destroyed once it stopped responding to commands from its operator. This is a dramatic illustration of a basic technological principle: the thing has form and value only insofar as it serves a human purpose. The destruction of the unresponsive Predator can also be viewed, perhaps, as a tacit admission that the thing is dangerous in the absence of a human controller. Of course, there were sensible and sound reasons for USAF's action: the drone was on course to cross international boundaries, which could be easily and rightly read by others as an act of callous negligence, if not outright aggression. In all likelihood, this particular failure of a Predator was due to a component malfunction and not to any error by a human operator or maintainer. However, the episode itself and the responses it has drawn at large all call attention to a basic concern with the notion of a powerful technological artifact being allowed to drift freely out of human control. I would argue that such concern is a mark of sanity. The ease with such errors can happen, by simple misunderstandings or oversights, should highlight the profound difficulties in navigating the interface between human intentions and the machinery built and deployed to execute them.

---

[1] Williams, Kevin. "A Summary of Unmanned Aircraft Accident/Incident Data: Human Factors Implications". DOT/FAA/AM-04/24, Office of Aerospace Medicine, Washington DC. (available here)

[2] USAF Fact Sheet on the MQ-9 Reaper (available here)

[3] Leveson, Nancy, Clark Turner. "An Investigation of the Therac-25 Accidents", IEEE Computer, 26(7):18-41. (Also available here)