The Spin Stops Here!
  Decision-Making Under Severe Uncertainty
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

Voodoo Decision-Making Under Severe Uncertainty

Last modified: Wednesday, 25-Nov-2009 20:41:56 MST

  Latest News  
   
  Supplementary Material  

This page is the nerve center of my campaign to contain the

The Rise and Rise of Voodoo Decision Theories

in Australia — and elsewhere — and it is related to my Info-Gap Campaign.

Table of contents

Overview

Before I turn to the technical aspects of the topic, let me explain what I mean by the term "voodoo decision theory". This is important because some of my colleagues take great umbrage at my labeling Info-gap decision theory a "voodoo decision theory".

According to the Encarta online Encyclopedia,

Voodoo n

  1. A religion practiced throughout Caribbean countries, especially Haiti, that is a combination of Roman Catholic rituals and animistic beliefs of Dahomean enslaved laborers, involving magic communication with ancestors.

  2. Somebody who practices voodoo.

  3. A charm, spell, or fetish regarded by those who practice voodoo as having magical powers.

  4. A belief, theory, or method that lacks sufficient evidence or proof.

My usage of the term "voodoo decision theory" is in line with the last meaning listed above. So roughly, in this discussion voodoo decision-making is a decision-making process that is guided and/or inspired by a voodoo decision theory, that is a theory that lacks sufficient evidence or proof and/or is based on utterly unrealistic and/or contradictory assumptions, spurious correlations, and so on.

This reading is also in line with the widely used terms Voodoo Economics, Voodoo Science and Voodoo Mathematics.

I should point out, though, that the term "Voodoo Decision Theory" is not my coinage (what a pity!):

The behavior of Kropotkin's cooperators is something like that of decision makers using the Jeffrey expected utility model in the Max and Moritz situation. Are ground squirrels and vampires using voodoo decision theory?
Brian Skyrms (1996, p. 51)
Evolution of the Social Contract
Cambridge University Press.

To illustrate a voodoo decision theory in action, consider this.

Example

Suppose that your task is to determine how a given function, f=f(x), behaves on the interval X=[-1000,1000]. For instance, assume that the issue is the constraint f(x) ≥ 0. That is, assume that you want to know how robust this constraint is over the interval X=[-1000,1000].

Also, assume that evaluating function f on X is difficult and/or costly.

Then, a voodoo decision theory would come to the rescue as follows: instead of examining the constraint f(x) ≥ 0 over X, it would prescribe testing it only over a small subset of X, say X'=[-1,1].

Now suppose that the constraint f(x) ≥ 0 performs well on X'=[-1,1]. What can we say about the performance of this constraint on X=[-1000,1000]?

No Man's LandNo Man's Land
[-1,1]

Well, if you espouse voodoo decision theory, you would argue that the performance of f(x) ≥ 0 on X'=[-1,1] provides a good indication of the performance of f(x) ≥ 0 on X=[-1000,1000], and therefore the constraint f(x) ≥ 0 performs well on X. In other words, you would argue that the performance on X=[-1,1] is representative of the performance on X=[-1000,1000].

You can save a lot of $$$$$$$ this way: instead of evaluating the performance of a system over the required large space, you quickly evaluate its performance only on a relatively small subset of the required space.

However, seeing through the nonsensical argument made by voodoo decision theory, you would argue that this is absurd because:

  • X'=[-1,1] constitutes a tiny part of X=[-1000,1000], in fact only 0.1 percent of it.

  • All the points in X' are in the same neighborhood.

  • Therefore X' is not representative of X insofar as the performance of f(x) ≥ 0 is concerned.

  • Therefore, hardly anything can be deduced about the performance of f(x) ≥ 0 on X from the performance of f(x) ≥ 0 on X'.

  • All we can say is that f(x) ≥ 0 performs well on X'.
Got the drift?

Of course, some readers may question the significance of this example, arguing that it is hyperbolic. After all, who would so much as contemplate suggesting that X=[-1,1] is representative of X=[-1000,1000] with regard to the constraint f(x) ≥ 0 (unless f has some very unique properties)?

My answer to this is that -- as we shall see -- this example is not an exaggeration. It is implicit in the type of argument used by experienced senior analysts in academia and business/industry (eg. banks) to justify the application of the methodology that they propose/develop.

Scope

Here I consider a special case where the decision-making environment is subject to severe uncertainty and where voodoo type thinking is used to tackle/manage the severity of the uncertainty.

As you would no doubt gather from the above example, a voodoo decision theory will tackle severe uncertainty by ... ignoring the severity of the uncertainty. For instance, it would prescribe that the severe uncertainty be tackled through an analysis of the immediate neighborhood of a wild guess of the parameter of interest whose true value is subject to severe uncertainty.

It is important, therefore, that you read the following carefully:

Read Me First

The discussion here is not a practical joke!

Scientists, analysts, and decision makers do resort (sometimes) -- often unwittingly -- to voodoo decision theories. Therefore, if you are an analyst, it is important that you be able to identify voodoo decision theories as this can be of great help on your job!

A good way to guard against mishaps in this area is to always keep in mind the Universal

GIGO Axiom

Garbage In ---> Garbage Out

Specifically, when someone proposes that you use a theory that can generate meaningful / useful results that are based on a very poor estimate that is likely to be substantially wrong, recall the following corollary of the GIGO Axiom:

GIGO Corollary

The results of an analysis are only as good as the estimates on which they are based.

The strains one has to endure when doing science are due to these and similar precepts. The bottom line is that we have to justify our theories.

But, the point about voodoo decision theories is that they take no notice of such precepts. They openly violate the GIGO Axiom and its many corollaries without so much as giving the slightest consideration to the consequences. They promise meaningful/useful/reliable results even though their analyses are knowingly based on poor estimates that are likely to be substantially wrong.

And the most interesting thing here is that some analysts justify the use of voodoo decision theories by appealing to the following excuse:

But this is the best estimate we have!?!

If you do not appreciate how unacceptable this argument is, I strongly recommend that you read my criticism of Info-Gap decision theory and my contribution to

WIKIPEDIA article on Info-Gap Decision Theory

The paper

The Mighty Maximin!

is also relevant here.

Severe uncertainty

Recall that classical decision theory distinguishes between three levels of knowledge pertaining to a state-of-affairs, namely

The "Risk" category refers to situations where the uncertainty can be quantified by standard probabilistic constructs such as probability distributions.

In contrast, the "Uncertainty" category refers to situations where our knowledge about the unknown parameter under consideration is so meager that we cannot quantify the uncertainty even by means of an "objective" probability distribution.

It is clear then that, according to this classification, "Uncertainty" eludes "measuring". It is simply impossible to provide a means by which we would "measure" the level, or degree, of "Uncertainty" to thereby indicate how great or daunting it is. To make up for this difficulty, a tradition has developed whereby the level, or degree, of "Uncertainty" is captured descriptively, that is informally, through the use of "labels" such as these:

In this discussion I prefer to use the term "Severe Uncertainty".  I understand "Severe Uncertainty" to connote a state-of-affairs where uncertainty obtains with regard to the true value of a parameter of interest. That is, the true value of this parameter is unknown and the estimate we have of this true value is:

Some analysts even claim that the estimate can be based, among other things, on:

So, the point here is that decision-making under severe uncertainty deals with extreme situations where the estimates we have can be based on no more than ... rumors.

Needless to say, this is a formidable challenge, especially if we are expected to provide robust decisions.

The following example illustrates a typical situation of severe uncertainty.

We want to deliver a personal note to a young kangaroo, known to his friends as Jack. Unfortunately, we do not know Jack's exact whereabouts. All we know is that he was last seen in a huge game reserve somewhere in Australia.

In case you do not know Jack, he is the one on the left in the picture shown on the right hand side of the page. We suspect that this picture was taken about a year ago, but we are not sure. His passport photo, taken 3 years ago, is shown on the left.

We do not know where these pictures were taken.

Actually, some even argue that these are not Jack's pictures, but never mind.

In short, we do not know Jack's exact location. All we have is a very poor estimate of the location, a kind of wild guess. This estimate is likely to be substantially wrong.

But, no worries!

Using voodoo decision theory we can easily handle the severe uncertainty pertaining to Jack's true location. For, its recipe is simplicity itself: ignore the severe uncertainty in Jack's location and regard the estimate we have as good. In fact, we can assume that the estimate is so good that we can restrict the search to its immediate neighborhood.

More generally, to explain how the method of voodoo decision-making tackles severe uncertainty of this type, it is instructive to look at decision-making under severe uncertainty from two perspectives. Hence, the following comparison:

The picture is then as follows:

Reality Voodoo Reality
This picture shows the poor estimate we have as well as a fictitious true value. The point of displaying the (unknown) true value is to drive home the fact that under severe uncertainty the estimate is a poor indicator of the true value and is likely to be substantially wrong. This picture illustrates how voodoo decision theory views the situation under consideration. The point to note here is that it gives not the slightest thought to what severe uncertainty actually entails. This is expressed (in the picture) in the glaring absence of all reference to the true value. Instead, the uncertainty is confined to the immediate neighborhood of the poor estimate. So, the net result necessarily is that no account whatsoever is given of the severe uncertainty which, of course, pertains to the entire space.

The main objective of this picture is to drive home the fact that voodoo decision theory prescribes decision-making that is based solely on an analysis that is confined to the immediate neighborhood of a single poor estimate of a parameter in question - an estimate that is likely to be substantially wrong.

Practically speaking, one is expected to ignore completely the severe uncertainty pertaining to the situation and to pretend that the estimate one has is so good that it is sufficient to conducted the analysis in its immediate neighborhood.

Practice what you preach

Of course it is hard to see how such a self-contradictory methodology that clearly makes no sense can be sold. So for marketing purposes it may have to be dressed up with jargon and rhetoric.

Here is an outline of a possible explanation of the logical progression that leads from the statement of the problem to the Voodoo Reality associated with it.
Step 1:

Display the region of severe uncertainty in its full glory. The larger the better! And do not forget to use impressive colors.

In fact, you may want to stress that usually the region of uncertainty is unbounded.

  
Step 2:

Display the true value (location) of the (unknown) parameter of interest. Of course, you do not know where this point is, but display it anyhow as an indication that you mean business, big business!

  
Step 3:

To impress your client, display the location of the estimate you have somewhere in the region of uncertainty. As a reminder that this is decision-making under severe uncertainty, make sure to place the estimate not too close to the true value.

This is the right moment to advise your client that the situation is very difficult precisely because under severe uncertainty the estimate we have is poor and is likely to be substantially wrong.

  
Step 4:

To stress the consequences of the fact that the estimate you have is poor and is likely to be substantially wrong, display the immediate neighborhood of the estimate. Explain to the client that under severe uncertainty it is not good enough to confine the analysis to this neighborhood.

The picture looks great. It vividly displays the difficulty associated with decision-making under severe uncertainty: it does not make much sense to base the decision on an analysis conducted on the immediate neighborhood of a poor estimate.

  
Step 5:

To deal with this difficult dilemma, it is useful at this stage to delete any reference to the true value of the parameter of interest. Since we plan to conduct the analysis in the immediate neighborhood of the poor estimate, it would be best at this point to delete any reference to the true value of the parameter of interest.

So, drop it from the picture.

  
Step 6:

You must admit that the picture looks much better now!

In fact, following this logic it would be best at this point to delete any reference to the severity of the uncertainty. So, drop the header from the picture!

  
Step 7:

It is embarrassing to show that our analysis boils down to an investigation of the immediate neighborhood of a poor estimate, so it would be best to drop the qualifier "poor".

  
Step 8:

Last but not least, we must get rid of the black background color. It is a grim reminder of the embarrassing fact that we in fact completely ignore the severity of the uncertainty under consideration.

So, invert the color scheme.

  

Of course, you should feel free to improvise as you go along and to use as many buzzwords as you can for extra effect.

To make a long story short, here is the essence of the method. It is a play in two Acts:

Observe that this outline is quite flexible given that it does not specify in detail the methods and techniques to be used to:

In short, there is plenty of scope here for creative thinking.

Info-Gap decision theory

In the framework discussed above, info-gap decision theory is a classic example of voodoo decision-making. Because, while it acknowledges that, under severe uncertainty, the estimate we have is essentially a wild guess, the theory deploys a robustness model that explores only the immediate neighborhood of the estimate.

The flaw in this theory is so glaring that it can be easily described pictorially. So, here is the tale of the "Treasure Hunt" that narrates Info-Gap's fundamental flaw.

Treasure Hunt
  • The island represents the stipulated region of uncertainty (the region where the treasure is located).

  • The tiny black dot represents a wild guess of the parameter of interest (location of the treasure).

  • The large white circle represents the region of uncertainty affecting info-gap's robustness analysis.

  • The small white square represents the true (unknown) value of the parameter of interest.

Clearly then, under severe uncertainty Info-Gap may conduct its robustness analysis in the vicinity of Brisbane, whereas for all we know the true location of the treasure may be somewhere in the middle of the Simpson desert or perhaps in down town Melbourne. Perhaps.

The fundamental question is this:

How can a theory claim that it seeks robust solutions under severe uncertainty when its robustness model ignores the stipulated region of uncertainty and instead focuses exclusively on the immediate neighborhood of a wild guess?

Well, in voodoo-land, questions like this are completely irrelevant.

And if you persist in asking this question, the answer you get is this:

But this is the best estimate we have, mate!?

So it is important to point out that the issue here is not the poor quality of the estimate. Indeed, in many cases the estimates we have are poor and their quality cannot be improved.

Rather, the issue is: what should be done given this state-of-affairs?

For example, the fact that the estimate is bad should not prevent you from -- indeed should encourage you to -- sample the region of uncertainty under consideration with a view to incorporate in the analysis a representative sample of possible values of the parameter of interest.

But suppose that this cannot be done. What then?

Whatever is done, if one has to settle for an analysis that is based exclusively on a poor estimate of the true value of the parameter of interest, then it must be stated openly that the quality of the results of this analysis are on a par with the estimate namely: "poor".

For, an analysis earns the honorary title "voodoo" precisely when no such warning is attached to the results. Such an analysis effectively misleads the public to believe that the results are reliable. Especially when it is also claimed that the analysis, or methodology, are reliable, or that the decisions they yield are responsible.

And so, Info-Gap decision theory earns the honorary title voodoo decision theory not only because it instructs using a poor estimate as the fulcrum of the analysis. It earns this honorary title precisely because its rhetoric gives the false impression that the results yielded by this analysis are reliable, or that the methodology itself is reliable. For instance, that a "rumor" can generate a "reliable" output.

The picture speaks for itself:

rumor ---->     Info-Gap Robustness Model     ----> reliable result

 

No Man's LandrumorNo Man's Land
<----------------- Complete region of uncertainty ----------------->

Isn't this too good to be true ?

Of course, adherents and advocates of this theory would strongly protest against labeling this theory a voodoo decision theory even tough this designation is based on a formal analysis of the theory (eg. see my discussion on info-gap and the references therein).

Some may contend that although my criticism is valid from a purely mathematical point of view, still info-gap has a role to play, and can help, in practice. In other words, the argument is that we have to distinguish between the theoretical and practical aspects of voodoo decision theories.

Others may even claim that my criticism is unfair and hyperbolic. For instance:

Sniedovich has stridently criticized info-gap theory [2; cf. 3]. Some of this criticism is unfair and hyperbolic (e.g., calling it "voodoo decision-making"), and some of it seems to miss the message and intent of the info-gap approach (e.g., whether its solutions are robust, and the sense in which the analysis is local). But some of the criticism is fair and germane. For instance, Sniedovich points out formal similarities with Wald's maximin decision theory that suggest info-gap decision models are really instances of classical models [cf. 4]. He also complains that info-gaps cannot really express "severe uncertainty". This seems correct if one means by this phrase uncertainty beyond the formulation of the system on which info-gap models are based. Despite these criticisms, info-gap theory has been heartily embraced in several quarters, and the literature on the subject is exploding [see 3].

Ferson, S. and Tucker, W.T. (2008)
Probability boxes as info-gap models
Proceedings of NAFIPS 2008, pp.1-6
Fuzzy Information Processing Society.

In case you wonder how Ferson and Tucker (2008) substantiate their claims, particularly the claim that labeling info-gap a voodoo decision theory is unfair and hyperbolic, the answer is simple.

They do not. They do not have too.

You see, this is how scientific discussions upholding voodoo theories are conducted. Claims are not sought to be substantiated. Assertions, allegations, conclusions, suffice.

In the words of Cole Porter (1891-1964), Anything Goes!
Anything Goes (Cole Porter, 1934)
Times have changed,
And we've often rewound the clock,
Since the Puritans got a shock,
When they landed on Plymouth Rock.
If today,
Any shock they should try to stem,
'Stead of landing on Plymouth Rock,
Plymouth Rock would land on them.

In olden days a glimpse of stocking,
Was looked on as something shocking,
But now, God knows,
Anything Goes.

Good authors too who once knew better words,
Now only use four letter words
Writing prose,
Anything Goes.

The world has gone mad today
And good's bad today,
And black's white today,
And day's night today,
When most guys today
That women prize today
Are just silly gigolos
And though I'm not a great romancer
I know that I'm bound to answer
When you propose,
Anything goes

When grandmama whose age is eighty
In night clubs is getting matey with gigolo's,
Anything Goes.

When mothers pack and leave poor father
Because they decide they'd rather be tennis pros,
Anything Goes.
If driving fast cars you like,
If low bars you like,
If old hymns you like,
If bare limbs you like,
If Mae West you like
Or me undressed you like,
Why, nobody will oppose!
When every night,
The set that's smart
Is intruding in nudist parties in studios,
Anything Goes.

The world has gone mad today
And good's bad today,
And black's white today,
And day's night today,
When most guys today
That women prize today
Are just silly gigolos
And though I'm not a great romancer
I know that I'm bound to answer
When you propose,
Anything goes

If saying your prayers you like,
If green pears you like
If old chairs you like,
If back stairs you like,
If love affairs you like
With young bears you like,
Why nobody will oppose!

And though I'm not a great romancer
And though I'm not a great romancer
I know that I'm bound to answer
When you propose,
Anything goes...
Anything goes!

It must be admitted though that this gives voodoo decision theories a major advantage over the traditional scientific approaches to problem solving. This may also explain the popularity of some voodoo decision theories.

Perhaps.

Which brings us to one of the most important characteristics of voodoo theories, a characteristic that it would seem enables to explain why such theories find an audience at all.

Rhetoric and Spin

If there is one thing that voodoo theories truly excel in it is rhetoric or, to use the more current term: "spin".

The true forte of voodoo theories is the verbose narratives that they use to describe their subject matter, their objectives, their capabilities etc. These narratives usually abound with grand declarations and promises that can give the impression that these theories indeed offer solutions to the most difficult (even intractable) problems.

Of course when this bombastic narrative is stripped away, you discover that the "methodologies" proposed to solve the problems, that these theories are presumably concerned with, have got nothing in common with the verbiage.

For it is typical of such theories that the "methodologies" that they propose are utterly unable to deliver the promised goods and/or services. It is typical of such theories that the mathematical models that they offer for this effort are loosely formulated, the interpretation of the relation between the model's components and those of the problems concerned is incorrect, the prescription providing concrete tools for solving the problems in question are totally unsuitable for the task, and so on and so forth.

All of this is manifest in the rhetoric that is used to promote such theories.

For example, consider this:

Making Responsible Decisions (When it Seems that You Can't)
Engineering Design and Strategic Planning Under Severe Uncertainty

What happens when the uncertainties facing a decision maker are so severe that the assumptions in conventional methods based on probabilistic decision analysis are untenable? Jim Hall and Yakov Ben-Haim describe how the challenges of really severe uncertainties in domains as diverse as climate change, protection against terrorism and financial markets are stimulating the development of quantified theories of robust decision making.

When you read the full paper you discover that the silver bullet is ... info-gap decision theory. In other words, the paper's thesis is that responsible decision-making, in the face of severe uncertainty, can be accomplished by ... conducting a robustness analysis in the immediate neighborhood of a wild guess.

Also, using this voodoo decision theory, you can generate reliable forecasts of say the local and international share markets!

and of course, GOLD:

In the words of Cole Porter (1891-1964), Anything Goes!

And talking about the share and gold markets. How about a proposal for formulating a policy yielding Confidence in Monetary Policies?

"... We adopt the technique of Info-Gap Robust Satisficing to first define confidence under Knightian uncertainty, and second quantify the trade-off between quality and robustness explicitly. We apply this to a standard monetary policy example and provide Central Banks with a framework to rank policies in a way that will allow them to pick the one that either maximizes confidence given an acceptable level of performance, or alternatively, optimizes performance for a given level of confidence. ..."
Ben-Haim and Demertzis (2008, p. 1)
Confidence in Monetary Policy
DNB Working Paper No. 192, December 2008

So, the idea here is to formulate a robust monetary policy for Central Banks based on an analysis that is conducted in the neighborhood of a wild guess of the parameter of interest!

Will you have confidence in such a monetary policy?

And how about your health? Yes, your health!

" ... Info-gap excels in conditions where the data are sparse, uncertain, and the result of failure is catastrophe. Standard clinical decision making strategies do not protect patients against errors in the clinical data. However, because info-gap is not probability-based, it provides a strong foundation for making decisions when there is not enough information -- such as in the highly individualized world of the clinic. Info-gap also allows "real-time" decision making and rapid assessment, thus allowing for targeted answers without wasted time. ..."

A quote from the description of
The Info-Gap Theory Program
The Abramson Center for the Future of Health
Houston, TX, USA

If you are of the opinion that a theory based on an analysis of the neighbrhood of a wild guess provides a "... strong foundation for making decisions when there is not enough information ..." then I suggest that you do some reading on Decision-Making Under Severe Uncertainty, Robust Optimization, and Black Swans. Particularly, make sure to examine carefully the GIGO issue.

Furthemore, take a close look at the following (emphasis is mine):

2. Managing uncertainty
Decisions depend on the best estimates of past performance, assessments of the current situation and visions into the future. Where the past performance may be known, the current is clouded by its immediacy and the future is a best guess. The robustness of any decision and the risk incurred in making that decision is only as good as the estimates on which it is based. Making estimation even more challenging, virtually all estimates that affect decisions are uncertain. Uncertainty can not be eliminated, but it can be managed.

Top Ten Challenges for Making Robust Decisions
The Decision Expert Newsletter™ -- Volume 1; Issue 2

And, of course, you might also be interested in:

'... Finally, a reliability theory is only as good as the information upon which it rests. A reliability theory should exploit all relevant verified information, but should treat speculative information and "reasonable assumptions" with caution. ...'

Ben-Haim (1996, p. 206)
Robust Reliability in the Mechanical Science Springer, Berlin.

I urge all those in health-care organizations who actually use Info-Gap decision theory to browse through my compilation of FAQs about Info-Gap to understand why this theory is a voodoo decision theory par excellence.

And as an aside regarding "individualizing/personalizing" health-care (emphasis is mine):

BTS is based on Bayesian decision theory. It offers a well-founded computational theory for applying general knowledge to individual situations that are characterized by varying degrees of uncertainty and risk.

As classical statistics revolutionized the discovery of knowledge in the early 20th century, Bayesian decision theory is revolutionizing the application of knowledge in the 21st. This revolution is already underway:

  • Most major spam tools are based on Bayesian methods; As a classic example involving uncertainty, it gives the software the ability to learn which mail is spam and which isn't.
  • All speech recognition tools are Bayesian.
  • Medical diagnosis is increasingly relying on Bayesian methods.
  • Counter terrorism is using Bayesian method to determine who are the bad guys.
  • Robotics and navigation use Bayesian method to navigate their environment.
What makes Bayesian methods hot stuff for making the most Robust Decisions?
The Decision Expert Newsletter™ -- Volume 2; Issue 7

Just in case: I have been a reformed Bayesian for over thirty years now.

Modern Alchemy, Freudian Slips, Quick-Fixes and Suchlike

If you are taking it for granted that the quest for a magic formula capable of transforming severe lack of knowledge / information into substantial knowledge was abandoned with the Enlightenment, I have news for you!

Apparently, against all scientific odds, Info-Gap scholars were successful in imputing likelihood to results generated by a non-probabilistic model that is completely devoid of any notion of likelihood!

Recall that Info-Gap decision theory prides itself on being non-probabilistic and likelihood-free. Yet, Info-gap scholars -- the Father of Info-Gap included -- now claim that Info-Gap's robustness model is capable of identifying decisions that are most likely to satisfy a given performance requirement.

Consider for instance the following quote from ACERA Endorsed Core Material (emphasis is mine):

Information-gap (henceforth termed 'info-gap') theory was invented to assist decision-making when there are substantial knowledge gaps and when probabilistic models of uncertainty are unreliable (Ben-Haim 2006). In general terms, info-gap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty, termed robust satisficing. It provides a platform for comprehensive sensitivity analysis relevant to a decision.

Burgman, Wintle, Thompson, Moilanen, Runge, and Ben-Haim (2008, p. 8).
Reconciling uncertain costs and benefits in Bayes nets for invasive species management
ACERA Endorsed Core Material: Final Report, Project 0601 - 0611.
(PDF file, Downloaded on March 21, 2009)

This is a major scientific breakthrough.

For, until now we have been warned repeatedly by Info-Gap scholars that no likelihood must be attributed to results generated by Info-Gap decision models. Indeed, we have been advised that this would be deceptive and even dangerous (emphasis is mine):

However, unlike in a probabilistic analysis, r has no connotation of likelihood. We have no rigorous basis for evaluating how likely failure may be; we simply lack the information, and to make a judgment would be deceptive and could be deceptive and dangerous. There may definitely be a likelihood of failure associated with any given radial tolerance. However, the available information does not allow one to assess this likelihood with any reasonable accuracy.

Ben-Haim (1994, p. 152)
Convex models of uncertainty: applications and implications
Erkenntnis, 4, 139-156.

This point is also made crystal clear in the second edition of the Info-Gap book (emphasis is mine):

In info-gap set models of uncertainty we concentrate on cluster-thinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.

Ben-Haim (2006, p. 18)
Info-Gap Decision Theory: Decisions Under Severe uncertainty
Academic Press.

So the question is: have Info-gap scholars managed to accomplish a major feat in the area of decision-making under severe uncertainty?

Of course the answer is that this new claim (Burgman et al's 2008) is not due to a breakthrough in decision-making under severe uncertainty, but rather to a serious misrepresentation of Info-gap's robustness model, culminating in a thoroughly incorrect representation of the results.

My view on this episode -- based as it is on numerous discussions with Info-Gap scholars over the past five years -- is that this new claim is simply -- but not surprisingly -- ... a Freudian slip.

The point is that -- see my FAQs about Info-Gap -- without imputing some sort of "likelihood" to Info-Gap's decision model, Info-Gap decision theory is, and cannot escape being, a voodoo decision theory.

So, all that this Freudian slip manages to do is to extend the already existing error -- an alternative that some Info-Gap scholars seem to prefer to an admission to a mistake.

It is interesting to note, though, that some Info-Gap scholars have taken note of my criticism of Info-Gap's robustness analysis to the effect that they now introduce an assumption that explicitly imputes "likelihood" to Info-Gap's uncertainty model. For instance, consider this (emphasis is mine):

An assumption remains that values of u become increasingly unlikely as they diverge from û.

Hall, J. and Harvey, H. (2009, p. 2)
Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
Eighth International Conference on Hydroinformatics
(January 12-16, 2009, Concepcion, Chile)
(PDF file)

Although this attempt at a quick-fix fails to fix the problem (see FAQ # 78), it does attest to a recognition that without such an assumption, conducting an analysis of the kind prescribed by Info-Gap's robustness model is utterly senseless.

One can only wonder then: how long will it take other Info-Gap scholars such as Burgman et al (2008) to reach this unavoidable conclusion?

Only time will tell (March 21, 2009).

The Black Swan

 

Only time will tell what impact (if any) Nassim Taleb's recent popular and controversial book The Black Swan: The Impact of the Highly Improbable will have on the field of decision-making under severe uncertainty.

I, for one, hope that the issues raised in this book and in its predecessor, Fooled by Randomness: The Hidden Role of Chance in the Markets and in Life, will be instrumental in helping decision-makers to identify voodoo decision theories -- such as Info-Gap decision theory -- that promise robust decisions under severe uncertainty.

I fear though -- in view of my experience of the past 40 years - that the danger is that the huge success of the Black Swan will inspire a new wave of voodoo decision theories, purportedly capable of ... "domesticating" black swans and preempting the discovery of ... purple swans!

We shall have to wait and see.

For those who have "been in hiding" I should note that Taleb has become quite a celebrity. According to the Prudent Investor Newsletters (Tuesday, June 3, 2008):

  • Mr. Taleb charges about $60,000 per speaking engagement and does about 30 presentations a year to "to bankers, economists, traders, even to Nasa, the US Fire Administration and the Department of Homeland Security" according to Timesonline's Bryan Appleyard.

  • He recently got $4million as advance payment for his next much awaited book.

  • Earned $35-$40 MILLION on a huge Black Swan event-on the biggest stockmarket crash in modern history-Black Monday, October 19,1987.

So, if you haven't heard him in person you can easily find on the WWW numerous videos of his interviews.

Here is a link to a very short (2:45 min) clip, recorded by Taleb himself, apparently at Heathrow Airport, of 10 tips on how to deal with Black Swans, and life in general.

  1. Scepticism is effortful and costly. It is better to be sceptical about matters of large consequences, and be imperfect, foolish and human in the small and the aesthetic.

  2. Go to parties. You can't even start to know what you may find on the envelope of serendipity. If you suffer from agoraphobia, send colleagues.

  3. It's not a good idea to take a forecast from someone wearing a tie. If possible, tease people who take themselves and their knowledge too seriously.

  4. Wear your best for your execution and stand dignified. Your last recourse against randomness is how you act -- if you can't control outcomes, you can control the elegance of your behaviour. You will always have the last word.

  5. Don't disturb complicated systems that have been around for a very long time. We don't understand their logic. Don't pollute the planet. Leave it the way we found it, regardless of scientific 'evidence'.

  6. Learn to fail with pride -- and do so fast and cleanly. Maximise trial and error -- by mastering the error part.

  7. Avoid losers. If you hear someone use the words 'impossible', 'never', 'too difficult' too often, drop him or her from your social network. Never take 'no' for an answer (conversely, take most 'yeses' as 'most probably').

  8. Don't read newspapers for the news (just for the gossip and, of course, profiles of authors). The best filter to know if the news matters is if you hear it in cafes, restaurants ... or (again) parties.

  9. Hard work will get you a professorship or a BMW. You need both work and luck for a Booker, a Nobel or a private jet.

  10. Answer e-mails from junior people before more senior ones. Junior people have further to go and tend to remember who slighted them.

It is interesting to juxtapose Prof. Taleb's thesis in The Black Swan that severe uncertainty makes (reliable) prediction in the Socio/economic/political spheres impossible, with the polar position taken by his colleague, Prof. Bruce Bueno de Mesquita, who actually specializes in predicting the future.

New Nostradamuses

One need hardly point out that not only professionals who make "decision under uncertainty" their metier, but also the proverbial "man in the street", take it for granted that the ability to accurately predict future events is one of the most onerous challenges facing humankind — especially persons in authority, persons responsible for the management of business or economic organizations etc.

But apparently no longer.

For, according to Good Magazine, predicting future events — at least in the area of international conflicts — is now possible thanks to the efforts of the New Nostradamus: Prof. Bruce Bueno de Mesquita, a political science professor at New York University and Senior Fellow at the Hoover Institution.

The claim is that this distinguished political scientist can actually predict the future — more specifically the outcome of any international conflcit!

And this he does not with the aid of the age old Crystal Ball, but through the use of a scientific method that, apparently, is grounded in a branch of applied mathematics called Game Theory.

According to GoodReads.com,

" ... Bruce Bueno de Mesquita is a political scientist, professor at New York University, and senior fellow at the Hoover Institution. He specializes in international relations, foreign policy, and nation building. He is also one of the authors of the selectorate theory.

He has founded a company, Mesquita & Roundell, that specializes in making political and foreign-policy forecasts using a computer model based on game theory and rational choice theory. He is also the director of New York University's Alexander Hamilton Center for Political Economy.

He was featured as the primary subject in the documentary on the History Channel in December 2008. The show, titled Next Nostradamus, details how the scientist is using computer algorithms to predict future world events ..."

Here is an interview with Prof. Bueno de Mesquita (with Riz Khan - The art and science of prediction - 09 Jan 08):


And here is a 20-minute lecture on the ... future of Iran (TED, February 2009):

Apparently, all you need to accomplish this is a computer, expert-knowledge on Iran, and game theory!

Some of the predictions attributed to Prof. Bueno de Mesquita are:

  1. The second Palestinian Intifada and the death of the Mideast peace process, two years before this came to pass.

  2. The succession of the Russian leader Leonid Brezhnev by Yuri Andropov, who at the time was not even considered a contender.

  3. The voting out of office of Daniel Ortega and the Sandanistas in Nicaragua, two years before this happened.

  4. The harsh crack down on dissidents by China's hardliners four months before the Tiananmen Square incident.

  5. France's hairs-breadth passage of the European Union's Maastricht Treaty.

  6. The exact implementation of the 1998 Good Friday Agreement between Britain and the IRA.

  7. China's reclaiming of Hong Kong and the exact manner the handover would take place, 12 years before it happened.

Impressive, isn't it!

As might be expected, these and similar claims by Prof. Bueno de Mesquita have sparked a vigorous debate not only in the professional journals but also on the WWW. Interested readers can consult this material to see for themselves, whether Bueno de Mesquita's claims attest to a major scientific breakthrough or ... voodoo mathematics.

Also, in addition to consulting this material you may want to have a look at a short video clip by Matt Brawn (right) which, he compiled in response to a short note entitled This man can actually predict the future!.

Of particular interest is, of course, the "success" rate of the Prof. Bueno de Mesquita's predictions: over 90% — yes over ninty percent!

Here is Trevor Black's common sense reaction to this claim:

I am a little skeptical about anyone who claims to have a 90% success rate. I just don't buy it. Especially when they say that they can explain away a lot of the other 10%.

If you come to me and tell me you have a model that gets it right 60% or 70% of the time, I may listen. Skeptically, but I will listen. 90% and I start to smell something.

All I wish to add here is that Prof. Bueno de Mesquita (left) makes his predictions under conditions of "severe uncertainty" which of course render them hugely vulnerable to what Prof. Naseem Taleb (right) dubs the Black Swan phenomenon.

Hence, the very proposition that such predictions can be made at all, let alone be reliable, is diametrically opposed to Nassim Taleb's categorical rejection of any such position. For his thesis is that Black Swans are totally outside the purview of mathematical treatment, especially by models that are based on expected utility theory and rational choice theory.

Interesting, though, this is precisely the stuff that Prof. Bueno de Mesquita's method is made of: expected utility theory and rational choice theory!

Even more interesting is the fact that Nassim Taleb (right) and Bueno de Mesquita (left) are staff members of the same academic institution, namely New York University. So, all that's left to say is: Go figure!

As indicated above, the debate over Bueno de Mesquita's theories is not new. It has been ongoing, in the relevant academic literature, at least since the publication of his book The War Trap (1981).

Note, therefore, that Bueno de Mesquita's work has attracted a considerable amount of criticism. For an idea of the kind of criticism sparked by his work, take a look at the quotes I provide from articles that are critical of Bueno de Mesquita theories.

Of course, there are other New Nostradamuses around.

According to the Associated Press, the latest (Mar 4, 4:39 AM EST) news from Russia about the future of the USA is that

" ... President Barack Obama will order martial law this year, the U.S. will split into six rump-states before 2011, and Russia and China will become the backbones of a new world order ..."

Apparently this prediction was made by Igor Panarin (right), Dean of the Russian Foreign Ministry diplomatic academy and a regular on Russia's state-controlled TV channels (see full AP news report).

Regarding the future of Russia,

"You don't sound too hopeful".
"Hopeful? Please, I am Russian. I live in a land of mad hopes, long queues, lies and humiliations. They say about Russia we never had a happy present, only a cruel past and a quite amazing future ..."
Malcolm Bradbury
To the Hermitage (2000, p. 347)

We should therefore be reminded of J K Galbraith's (1908-2006) poignant observation:

There are two classes of forecasters: those who don't know and those who don't know they don't know.

And in the same vein,

The future is just what we invent in the present to put an order over the past.

Malcolm Bradbury
Doctor Criminale (1992, p. 328)

So, we shall have to wait and see.

And how about this more recent piece by Heath Gilmore and Brian Robins in the Sydney Morning Herald (March 27, 2009):

"... COUPLES wondering if the love will last could find out if theirs is a match made in heaven by subjecting themselves to a mathematical test.

A professor at Oxford University and his team have perfected a model whereby they can calculate whether the relationship will succeed.

In a study of 700 couples, Professor James Murray, a maths expert, predicted the divorce rate with 94 per cent accuracy.

His calculations were based on 15-minute conversations between couples who were asked to sit opposite each other in a room on their own and talk about a contentious issue, such as money, sex or relations with their in-laws.

Professor Murray and his colleagues recorded the conversations and awarded each husband and wife positive or negative points depending on what was said. ..."

Such interviews should perhaps be made mandatory for all couples registering their marriage.

More details on the mathematics of marriage can be found in The Mathematics of Marriage: Dynamic Nonlinear Models by J.M. Gottman, J.D. Murray, C. Swanson, R. Tyson, and K.R. Swanson (MIT Press, Cambridge, MA, 2002.)


On a more positive note, though, here is an online Oracle from Melbourne (Australia: the land of the real Black Swan!).

You may wish to consult this friendly 24/7 facility about important "Yes/No" questions that you no doubt have about the future.

Enter your "Yes/No" question:
    

 
 

More on this and related topics can be found in the pages of the Worst-Case Analysis / Maximin Campaign, Severe Uncertainty, and the Info-Gap Campaign.

Your comments, please!

I am most interested in feedback on the contents of this web page. I promise to respond to comments within ... a week or so!

This feature is experimental, so I am not sure how robust it is. I checked a number of platforms and they seem to work fine. But, if it does not work well for you, .... let me know!

Recent Articles, Working Papers, Notes

Also, see my complete list of articles
  • Caserta, M., Voss, S., Sniedovich, M. (2009) Applying the corridor method to a blocks relocation problem, OR Spectrum in press
    • .
  • Sniedovich M.(2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich M.(2008) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.

  • Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.

  • Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

  • Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
    This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

  • Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.

  • Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.

  • Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.

  • Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
    In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

  • Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
    This paper is dedicated to the modeling aspects of Maximin and robust optimization.

  • Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .

  • Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
    In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

  • My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)

    This is an exciting development!

    • Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

    • Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.

      So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.

      Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!


  • A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
    This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
    It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).

  • A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
    This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
    It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

  • A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
    This is a very short article entitled The GAP in Info-Gap (PDF File) .
    It is a math-free version of the paper above. Read it if you are allergic to math.

  • A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
    This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).

Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: Wednesday, 25-Nov-2009 20:41:56 MST