Working In Uncertainty (WIU)

For students and researchers

Here are some ideas for research studies related to working in uncertainty. I don't guarantee these are original, or that there has already been some related work that you could build on. However, I think these are interesting ideas for research. If you do a literature review on any of them, or a study, I would be delighted to hear about it.

Communication with numbers and words: An argument often rages between the risk-listing school of risk management and the management science school over whether 'risk levels' should be expressed with numbers or with words like 'high', 'medium', and 'low'. Various advantages are claimed for each. One aspect of this is how well information is conveyed by these alternative techniques.

This could be explored with a simple design. A series of numbers between 0 and 100 are chosen at random and given, one by one, to an experimental subject, along with a definition of the number. For example, 'The number of students who enrolled in course ABC101 is 43.' The subject has to pass that message on verbally or in writing to another person, who will then pass it on to another person, who will then pass it on to another person. With some messages the first person in the line has to convert the number into a quantity word like 'high', 'medium', or 'low', using whatever approach they think is best. With other messages the subject just has to use the number given. Immediately after passing on each message each subject has to write down the message as they understood it, including a specific number or number range rather than any quantity word used.

What we should be able to see is exactly where and how the message gets degraded as it journeys through the chain. It is likely that the numerical messages will suffer changes to the definition of the number but usually not the number value itself. In contrast, conversion to a quantity word will cause immediate loss of accuracy of the number as well as changes to the definition. Communication in writing should be more accurate and reliable than verbal communication, even if people have to copy out the message themselves each time. There may be other interesting effects too. The data should provide nice graphs showing degradation of the value and definition by stage in the chain.

Probability words and context: Previous studies have tried to establish the typical quantities people associate with quantity phrases like 'highly probable', 'likely', and 'rare'. However, if you think about how quantitively vague phrases are used more generally it seems that the numbers we associate with phrases tend to vary with context. For example, we tend to understand the number relative to our expectations. If someone said 'There are a lot of politicians living in my road' and 'Last weekend we went to the beach, where there were a lot of shells.' we would not assume that there are as many politicians in the road as there were shells on the beach.

The study design requires a questionnaire in which people are asked to put numbers to quantitively vague phrases expressing probabilities. In each case the phrase is used in a sentence that establishes a context. The study might test five different phrases, each in three contexts. One context should be designed to create the expectation of very low probabilities, one should create the expectation of very high probabilities, and the third should be somewhere in the middle. If the meaning of the phrases is affected by context it should be possible to create a graph showing a line for each phrase that slopes according to the expectation created by the context.

Prompts for management method improvements: How can risk management be 'integrated' with management? The Working In Uncertainty perspective is that this is a misleading question because it creates the impression that there is something separate that needs to be 'integrated'. Instead, the way management is done can be tweaked to better deal with limited knowledge and consequent uncertainty. The risk-listing school currently says that 'integration' can be achieved by implementing tailored versions of their risk management process throughout an organization.

In a previous survey (Leitch, 2012) I asked people which type of prompt they thought would be more effective in generating ideas for better management methods, choosing between (a) a generic risk management process description, and (b) a short list of ideas for particular practices likely to be useful in that context. There was a strong majority view that the more specific prompts would be more effective. Are people correct?

A study could be conducted in which people are given an imaginary management situation describing how managers currently work, then asked to suggest improvements. One group gets the generic risk management process as a prompt. The other group gets a short list of specific practices likely to be relevant in that context. After people have written their suggestions, a second set of subjects is asked to score the suggestions made. Which group gets the higher scores?

Reducing Ignorance in probabilistic forecasts: Using subjective probabilities can be helpful in some situations, but people make mistakes when choosing probability numbers. The right approach is to assign 0 when you are sure some event will not happen, 1 when you are sure it will, and numbers in between when you are unsure. One problem is that some people choose zero when they are unsure of the outcome. Another problem is that people choose 0 or 1 when they are not in fact entirely sure. The second problem is particularly important when their probability numbers are evaluated as probabilistic forecasts using the logarithmic skill score known as Ignorance. The effect of saying the probability of a discrete event is 1 but being wrong is that your Ignorance is scored as infinite! The same happens if you say the probability is zero but the event happens.

What is needed are techniques to help people give subjective probabilities that reduce these errors. For example, rather than allowing people to choose 0 or 1, how about asking them to supply the X in the phrase 'the chance is 1 in X'?

Experimental tests of statistical methods: In a simple study, Graham and Glencross (2007) generated pseudo-random numbers using a chosen normal distribution, then fitted a normal curve to that data and studied the resulting curve parameters. (Obviously this was done thousands of times and the results averaged.) They showed that the properties of the tails of the distribution are much less accurately recovered than other aspects of the curve. This makes some sense because the tails are the part of the curve about which we have least experience. Typical measures of fit reflect the fit of the whole curve, but it is easy to study the fit at various points on the curve and create graphs showing how it varies along the curve.

It would be interesting to repeat this experiment and to take it further by looking at other distributions. More generally, this technique of generating data then using statistical techniques on it, then testing their conclusions to see if they are what they should be, is simple yet powerful. In a similar study, reported by David A Freedman in appendix A of Good and Hardin (2003) stepwise linear regression was shown to create the impression of statistically significant relationships in synthetic data that had none. Similar tests of other statistical hypothesis tests could be done.

Responding to circumstances in judgements under uncertainty: Good decisions under uncertainty should respond to such objective circumstances as current wealth, expected future income, certainty of future income, ability to respond to events, and so on. However, if a person believes that their decisions under uncertainty are primarily driven by their personality, and specifically by their personal propensity towards risk taking, then they may pay less attention to objective circumstances than they should.

A study could be done with a questionnaire that describes a sequence of similar decisions under uncertainty (e.g. how much to lend or guarantee) in which the objective circumstances are varied from one question to the next. To get at the influence (if any) of beliefs about risk taking propensity the experimental subjects could be given some questions about their beliefs about it after they have done the test. Alternatively, they could be induced to focus on personality by a short priming task or statement given before they do the decisions. For example, they could be asked to read descriptions of fictitious people and say what attitude to risk each person seems to have, or they could be asked to read a very short article about 'risk appetite'.

If there is a relationship between focus on personality and attention to objective circumstances then it should be possible to show that the differences in decisions (e.g. the amount lent in each hypothetical situation) are less when the focus is on personality than when it is not.

Risk register experiences survey: This study is very simple in principle but requires access to a pool of people who have been required to work with risk registers, but not to promote them (e.g. because of being in the job of corporate risk manager). The survey simply asks questions about the perceived value of the exercise, clarity over various technical points, extent of new ideas arising, and so on. Although most people who had been required to work with risk registers will have made positive statements about them at work (e.g. to the corporate risk manager or their boss), their real views are likely to be very different, particularly if they have been doing them for a long time. A challenge in this study is to convince people that they can safely express their real views.

Ease of understanding Bayesian data fitting: A highly effective and easily done approach to fitting certain types of model to data is to use 'conjugate priors' within a Bayesian framework. Despite many apparent technical strengths this technique is still relatively little known. This may be just inertia or because of some problem in using or explaining the technique.

A useful study would be to find out whether outputs from conjugate prior techniques are easier or harder to understand than frequentist alternatives. The study would involve a data fitting problem solved using an appropriate conjugate prior technique or a frequentist statistical analysis or test. Subjects would be shown one or the other and asked questions to test their understanding of the outputs. A variation of this would be to compare a static and an animated demonstration of the conjugate prior technique. This would be interesting because the way conjugate prior techniques work produces some beautiful and powerful graphics, and animation should provide an understanding that static images cannot. A more sophisticated research design might shed light on what needs to be explained before people can respond appropriately to results from the conjugate priors technique.

Stress and planning: My theory linking stress and uncertainty says that we instinctively begin a (usually small) fight or flight response whenever we detect a sign that some additional effort by us is needed. This is precautionary, and when we discover that perhaps no effort is needed after all, or that the effort is not now but later, or is entirely mental and not physical, our bodies start to wind down again. One implication of this idea is that it is stressful to think about things you need to do without thinking about when they need to be done. Just reading your 'to do' list is stressful because it reminds you of the things you have to do and it seems at that moment that they are all efforts needed now. Peter Gollwitzer's fascinating and useful work on implementation intentions shows how intentions can be linked to future situations in our minds, allowing us to let go of them until that situation occurs, whereupon they are activated in an almost automatic way. Perhaps forming implementation intentions in this way would allow a person to consider the things they have to do in a less stressful way. A basic study design would be to have people do a planning activity in one of two ways. One group just list what they have to do that day, without thinking about when. The other group think about when they have opportunities to do things (which will be later) and then plan to do things in those opportunities. Ask participants to rate their feelings of stress before and after the activity and make comparisons.

Implementation intentions and memory: This isn't really uncertainty related but it is related to implementation intentions. Gollwitzer's studies show that we can form intentions to perform particular acts in particular future situations, and that we do quite well at remembering these acts when the time comes. One of the studies shows that thinking "When [situation] then I will [action]" is more effective for this than thinking "I will [action] when [situation]". Is this also telling us something interesting about how to form memories? For example, if you wanted to learn that the capital city of England is London (assuming you don't know this already of course) then which thought would be most effective at doing this: (1) "The capital city of England is London", (2) "When I want to recall the capital city of England then I will remember that it is London.", (3) "London is the capital city of England.", or (4) "I will remember London when I want to recall the capital city of England. These four versions vary two factors. First, the explicit resemblance to an implementation intention. Second, the order of items in the intention. Which is most effective for promoting memory and how do they compare to just leaving people to think what they like? Do some tests to find out.

References

Good, P.I., and Hardin, J.W., 2003. Common Errors in Statistics (and How to Avoid Them). Hoboken, New Jersey: John Wiley & Sons, Inc.

Graham, M. and Glencross, A., 2007. The case of the credulous actuary: Rediscovering the importance of judgement. A paper from the General Insurance Convention, 2008. Available at: <http://www.actuaries.org.uk/research-and-resources/documents/case-credulous-actuary-rediscovering-importance-judgement-handout> [Accessed 17 October 2012].

Leitch, M., 2012. Results of a survey on corporate programmes to improve 'risk management'. Available at: <study_prog_report.shtml> [Accessed 17 October 2012].

Made in England