Working In Uncertainty

Risk, uncertainty, and limited knowledge and control

Sometimes we are certain we are right but in fact we are wrong. This phenomenon has profound implications for theory and practice, which are explored in this article.

The nature of (un)certainty

A simple implication of this is that certainty is, at least sometimes, certainty about the truth of some proposition. (A proposition is a statement that is true or false, even if we don't know which.) This is implied by the idea of being right or wrong about it.

(Un)certainty versus reality

A second implication is that, since our feeling or perception of certainty can be wrong, certainty as to a proposition is not an objective fact. For example, I might be certain that it will rain this afternoon. My prediction is for rain. Reality might be different.

Our certainty as to the truth of a proposition is the result of an assessment of our knowledge – usually a self-assessment. If I feel certain of the truth of a proposition then that's because I think my knowledge about it is completely reliable. If I feel uncertain about a proposition then that means I think my knowledge about it is not entirely reliable.

The main problems

Two obvious problems related to this clearly exist. They are:

1. unreliable knowledge of relevant propositions (regardless of assessed uncertainty); and

2. incorrect assessments of our knowledge (resulting in incorrect uncertainty).

Both are potentially dangerous but the first problem – unreliable knowledge of relevant propositions – is the really big, common one. We usually experience it when we need to decide what to do because, usually, we are not certain what will happen if we take actions we are considering. To deal with this we often need to consider a variety of possibilities, develop our plans to cope with them, and evaluate the whole mess of possibilities, rather than just one certain future.

We also experience unreliable knowledge of propositions when we need to establish what happened in the past (e.g. in a criminal court case) and struggle for evidence that conclusively proves what happened.

Incorrect assessments of knowledge can also be an important problem. They were a problem for some banks in 2008. Their models of markets and their own businesses were often quite good, at least compared to businesses in other industries, but they were not as good as the bankers thought. That was the problem, because their excessive certainty encouraged them to place much larger and more aggressive bets than they should have.

Problems related to the experience of uncertainty itself and not within the two problems listed above seem minor by comparison. For example, perhaps you might experience some anxiety due to feeling uncertain.

Managing when knowledge is unreliable

If our only goal was to manage our uncertainty then we could do so with tricks that let us feel certain when we should not feel certain. Clearly, blind optimism is a poor attitude when sensible precautions should be taken. So, strictly speaking, dealing with limited knowledge of propositions when managing is more important than dealing with ‘uncertainty’ itself.

And, as ‘risk’ is so often defined in terms of uncertainty, managing ‘risk’ is also less important than managing our limited knowledge of the truth of propositions.

Instead, we should be looking for ways to assess objectively our current state of knowledge and manage appropriately (in various ways).

We can assess our relevant knowledge by considering the inherent difficulties of knowing, the efforts we have made to do so, and our past success in doing so. Here is a framework of points to consider:

To better understand the separate contributions of research and control, consider these two examples:

Mainly by research/understanding: Suppose the challenge is to predict the temperature at the top of Nelson's Column in London at midday tomorrow. This outdoor temperature is not controlled by anyone and varies often, depending on the weather. The weather itself is notoriously hard to forecast but over the past decades weather forecasts in the UK have improved significantly, on some measures at least, and so a prediction can be made. It is not a certain prediction of course, but it is better than a wild guess. This prediction is based on a huge amount of knowledge gained from satellites and weather sensors, fed into powerful computers and combined within detailed models of the physics of the atmosphere, simulated into the future. In this prediction task we have no control, but knowledge of how the weather works and what the weather has been recently allows for valuable predictions. Better information and understanding would allow better predictions.

Mainly by control: Suppose instead that the challenge is to predict the temperature in the heart of a well designed museum of antiquities at midday tomorrow. This indoor temperature is controlled within a narrow range by systems designed for precisely that purpose. To make our prediction all we have to do is telephone the museum and ask them what the temperature is supposed to be and we have a good prediction. The museum's computerised climate control system, insulated structure, myriad sensors, back-up power supply, and fall-back thermostats will take care of the rest. In this prediction task we have great control and, once we know the museum's intended temperature, more knowledge of the weather outside has only the remotest chance of contributing to our prediction.

Use of language on this website

If focusing on managing despite unreliable knowledge of propositions is a better aim than managing risk or uncertainty, why do I use the words risk and uncertainty so often on this website? There are two reasons. One is that I started doing that before the above insights were reached. The other is that, without the words risk and uncertainty on the site, most relevant readers would not find it.

Appendix A: Certainty

We use the words ‘certainty’, ‘certain’, ‘uncertainty’, and ‘uncertain’ in at least two important ways.

• About the truth of a proposition: For example, ‘I am certain that 2 + 2 = 4.’ The proposition is that ‘2 + 2 = 4’. My assessment of the reliability of my knowledge on this point is that I am certain of it.

• As a general feeling: For example, ‘Since I was named employee of the month I've felt so certain of myself.’ The underlying reality here may be nothing more than a freedom from worry. Alternatively, the basis might be increased certainty about a set of related propositions, hazily thought about, but including items like ‘I will keep my job for at least another year.’, and ‘I will get a promotion in the next two years.’

There are some other words that we use in similar ways: sure, confident, and trust.

Appendix B: Calibration

It is quite rare to feel completely certain of something that then proves to be wrong – though this is surely an experience most of us have had. Politicians seem to be experiencing it all the time.

However, the idea has been extended and studied using scientific experiments. These look at many occasions where we felt a particular level of uncertainty and at how often we were in fact correct. The conclusion is that our feelings of certainty, expressed in terms of probabilities, often do not match our actual ability to predict. Typically, we are overconfident when our knowledge is poor but underconfident when our knowledge is very good.

Reality and our feelings of certainty have to be compared over many occasions. Being right on a particular occasion and being certain are not directly linked. I might predict rain for this afternoon but not be certain of it. If it does rain then that does not mean my assessment of uncertainty was inappropriate. Perhaps my track record of weather predictions is poor.

Exciting progress has been made by scientists studying the properties of good probabilistic forecasts.

Made in England