Working In Uncertainty

Results of a survey on 'risk' decision support

Contents

Many thanks to everyone who participated in this survey. Without your generous donations of time and thought this would have been impossible.

Summary

In an earlier study (Leitch, 2011) it emerged that almost everyone thinks project risk management should include all significant decisions, not just decisions about actions perceived as responses to risk(s). I have no doubt that a similar question about risk management generally would get a similar response. This brings into scope such decisions as choosing between suppliers, selecting new products to market, and choosing between a bridge or a tunnel to get a railway past a river. In these decisions it is obvious that all relevant factors should be considered, including uncertain effects (or 'risks' if you prefer).

In this new survey a situation was presented in which the actions under consideration (the safety investments) were primarily responses to risk. Would people still think that conventional decision-making logic should be applied?

Yes, most did think that conventional decision-making logic applies to those decisions too. They wanted information and summaries that directly helped them design and select the best bundle of investments, and they wanted to know about all relevant effects, not just cost and risk reduction.

The survey described a hypothetical exercise to select investments in safety for a factory. This setting is typical of 'risk management' exercises. Respondents were asked questions about which summaries of information they would find helpful in thinking of safety improvements and in deciding which to do given limited (but somewhat uncertain) resources.

The results of the survey provide rich information about what summaries people want in this sort of situation. They show that, in general, people prefer information that bears directly on the decisions they must make, and that includes all relevant considerations, not just cost and risk reduction. The most popular summaries were cost-benefit analyses of each possible investment in safety, which would include costs, risk reduction, and other benefits and dis-benefits.

Summaries that were oriented towards 'risks' rather than towards the investments to be chosen were less popular, despite being the familiar focus of many corporate risk management processes and getting most of the attention in guidance and standards on risk management.

These results should be helpful to risk managers who want to know what they should provide, to writers who want to know what to give guidance on, and to tool vendors who want to know what their tools should do for people.

A note on the statistics and graphs

Most of the graphs in this report look like a dark red bar chart with a grey overlay at the end. This overlay represents uncertainty about the true value in the population, based on the sample evidence, using Bayesian methods.

    

The dark red part of the bar shows the range up to the 10th percentile of a probability distribution representing belief about the true percentage or average of the entire population (i.e. people with an interest in management). The pink part shows the range from the 10th to the 50th percentile. The light grey part shows the range from the 50th to the 90th percentile. In effect, the graph says that, given various assumptions (as usual in statistics), we can be 80% certain that the truth lies within the grey overlay area (i.e. pink and grey segments combined).

In all cases the initial prior distribution was uniform. For questions requiring a choice between two answers the prior distribution was the beta distribution. For questions requiring a choice between more than two answers the prior distribution was the Dirichlet distribution. For the distribution of risk-listing focus a gamma distribution was used because the scores appeared to be roughly Poisson distributed.

The survey questions and answers

In this section the text in italics is the text used in the survey itself. Other text discusses the implications for decision support.

The first two questions in the survey were about the first language and skills of respondents. These are given and discussed in a later section. The survey then continued as follows.

Imagine you will be required to select a set of investments in safety designed to make a factory safer for workers. These might include such things as protective clothing, training, protective screens, sensors and alarms, replacing machines, changes to wiring and pipes, revised rest times and shift patterns, and so on. The total budget available to spend on these improvements is not settled, but probably around £200,000 (say USD 320,000 or Euros 240,000).

3. If you were to write down a list of safety 'risks', then a list of safety investments, and then link the 'risks' to the safety investments that reduced them, which pattern of relationships would you most expect to see?:

Pattern% of respondents
No safety investment reduces more than one of the safety 'risks'2%
    
Some safety investments reduce more than one safety 'risk'35%
    
Most safety investments reduce more than one safety 'risk'45%
    
All safety investments reduce more than one safety 'risk'18%
    

Implications for decision support: This question asked people to imagine making a list of 'risks', which many will not normally do. It was, as with all the questions in this survey, a hypothetical situation. Nevertheless, this is a crucial question because, if at least some safety investments address more than one 'risk', then summaries for decision-making need to reflect that. It is not sufficient to list all information by 'risk'. It is not sufficient to make decisions one 'risk' at a time. A risk-register format, with 'risks' as the main organizing item and safety investments listed by the side of each 'risk' is not adequate on its own. It is necessary to map the 'risks' to safety investments and to summarise decision factors against the investments, or against bundles of investments.

If the 'risks' are interpreted as objectives in a value model, or something similar, to support decision-making then it is even more obvious that decision support needs to focus on investments rather than only on the consequences for each objective, one objective at a time.

4. What proportion of safety investments do you think would give additional benefits, beyond safety improvements, that should be taken into consideration? (These extra benefits, if any, might include allowing faster working or increasing flexibility, for example.):

Pattern% of respondents*
None3%
    
Some79%
    
All safety investments18%
    

Implications for decision support: This is another crucial question. Since almost all respondents anticipated that at least some safety investments would have implications beyond their cost and reduction of risk then these other factors should be brought into the decision-making somehow.

One respondent, in comments, pointed out that safety investments might have dis-benefits other than cost. Another pointed out that some safety investments might increase danger in some ways while decreasing it in others. Although the survey question did not mention these other effects they might also be common and important enough to need inclusion in any good decision support approach.

Suppose just for a moment that you have only two alternative investments of the same cost, and that you have divided the causes of injury and death into two categories. The two investments have different effects:

  • Investment A eliminates less danger (in terms of injuries and death from all causes) but its effect is spread across both categories of causes.

  • Investment B reduces danger by 10% more in total than investment A, but it only reduces the danger in one category of causes, leaving the other unchanged.

5. Which course of action would you prefer?:

Course of action% of respondents*
Select Investment A, which has effects on both categories of cause.34%
    
Select Investment B, which reduces injury and death the most.47%
    
Select Investment B, but recategorise the causes so that Investment B gives reductions in both of the new cause categories.19%
    

Implications for decision support: Most respondents thought that the safety investment providing the greatest improvement in safety overall was the best. In order to choose safety investments on the basis of their overall contribution to safety, as most respondents would want to, it is necessary to summarise the effect across all 'risks' / objectives. Just showing the total reduction in risk levels against each 'risk' will not be sufficient.

In testing this question before the survey was launched it emerged that some respondents might think that spreading the effect of a safety investment over both cause categories would be 'politically' better in some way. The survey results suggest that at least some respondents may have seen it that way. A substantial minority chose the safety investment that was best overall but would revise the cause categories in order to show a spread of effect. A larger minority chose the safety investment that had effects spread over the initial cause categories, even though its effect was less overall.

The reason some people prefer (the appearance of) spread effects is not clear. Ideally, the way we analyse risk into categories should not affect our ability to identify the best investments. We should arrive at the same choices regardless of how risk is initially carved up. In the risk-listing approach to risk management (Leitch, 2012a) the initial division into 'risks' does affect the choice of investments because of targets or thresholds for risk reduction applied to each 'risk'. It may be that this approach influenced some respondents to prefer the less effective investment, and other respondents chose to recategorise in order to persuade anyone following the risk-listing approach to agree with their selection of the more effective investment.

These survey results suggest that decision-makers should not be so concerned about other people wanting to see spread effects across causes. Most people don't think this is important to them. (This is different from spreading effects across people. To focus resources on making some people much safer but ignoring the safety of others raises different issues.)

6. Returning to the more realistic situation where you have lots of potential safety investments, which of the following analyses do you think would help you select a good set of safety investments? Click all you think would be helpful.

Summary formatShort name% of respondents*
A scatterplot graph showing how serious each of the safety 'risks' is that might be addressed by the safety investments, with the axes being 'probability of occurrence' of each 'risk', and 'impact' if it does occur. The background is coloured so that you can see which are more worrying and there is a list of brief descriptions of each risk so that you can see what the dots on the graph represent.Probability Impact Graph63%
    
A table of the safety 'risks' that might be addressed by the safety investments, with the risk level, fraction that can be reduced, risk reduction in absolute terms, and cost per unit of risk reduction shown against each 'risk'.Risk and Risk Reduction list57%
    
A table of the safety investments showing for each its cost, the total risk reduction provided across all the safety 'risks' considered, and the risk reduction per unit cost.Investment cost and risk reduction list64%
    
A scatterplot graph where each dot represents a possible set of safety investments and the axes are total cost of the set and total risk reduction provided. A table lets you see which investments are included in each set.Investment bundle cost and risk reduction graph56%
    
Assessments of the potential effects on safety and other objectives, and of costs, for each potential safety investment.Investment cost-benefit analyses68%
    

Implications for decision support: Many respondents simply chose to take all available summaries, but even so a slight difference exists with the full cost-benefit analysis being selected more often than other items. Such analysis was considered helpful by 68% of respondents. So, decision support summaries for this kind of task should accommodate analyses of costs and (dis)benefits for each potential safety investment. This preference is consistent with the answers given earlier to questions 3 and 4.

The statistical persuasiveness of the results is as follows. If we assume that the respondents were representative of the population of people interested in management then we can be about 77% confident that more of such people think the full assessments are more helpful than the Probability-Impact Graph. We can also be 99.98% sure that more than half of these people think the full assessments are helpful.

Note that the Risk and Risk Reduction list is much more than a list of 'risks' because it has summary information about mitigation, including the amount of potential risk reduction per unit cost. It is the sort of summary whose properties were studied by Cox (2012), who showed it supports better decisions than just addressing the 'biggest' risks first.

7. Which do you think would be the most helpful?

Summary formatShort name% of respondents*
A scatterplot graph showing how serious each of the safety 'risks' is that might be addressed by the safety investments, with the axes being 'probability of occurrence' of each 'risk', and 'impact' if it does occur. The background is coloured so that you can see which are more worrying and there is a list of brief descriptions of each risk so that you can see what the dots on the graph represent.Probability Impact Graph16%
    
A table of the safety 'risks' that might be addressed by the safety investments, with the risk level, fraction that can be reduced, risk reduction in absolute terms, and cost per unit of risk reduction shown against each 'risk'.Risk and Risk Reduction list21%
    
A table of the safety investments showing for each its cost, the total risk reduction provided across all the safety 'risks' considered, and the risk reduction per unit cost.Investment cost and risk reduction list11%
    
A scatterplot graph where each dot represents a possible set of safety investments and the axes are total cost of the set and total risk reduction provided. A table lets you see which investments are included in each set.Investment bundle cost and risk reduction graph21%
    
Assessments of the potential effects on safety and other objectives, and of costs, for each potential safety investment.Investment cost-benefit analyses31%
    

Implications for decision support: This question was more discriminating than the previous one and shows more clearly the preference for cost-benefit analysis for each investment over other formats. Decision support for this kind of task should accommodate such analyses.

Now imagine that you are also required to develop proposals for investments in safety, so now you have to think of the investments and analyse them as well as making the final choice of which investments to select.

8. An expert in the field of factory safety has offered to help with some summaries of information to get you started. Which of the following summaries would help you in your task? Select all that you think would be helpful.

Summary formatShort name% of respondents*
A scatterplot graph showing how serious each of the safety 'risks' is that are typically addressed by safety investments, with the axes being typical 'probability of occurrence' of each 'risk', and 'impact' if it does occur. The background is coloured so that you can see which are more worrying and there is a list of brief descriptions of each risk so that you can see what the dots on the graph represent.Probability Impact Graph48%
    
A table of the safety 'risks' that are typically addressed by safety investments, with the typical risk level, fraction that can be reduced, risk reduction in absolute terms, and cost per unit of risk reduction shown against each 'risk'.Risk and Risk Reduction list60%
    
A table of typical safety investments showing for each its typical cost, the total risk reduction provided across all the usual safety 'risks', and the risk reduction per unit cost.Investment cost and risk reduction list56%
    
A scatterplot graph where each dot represents a possible set of safety investments and the axes are typical total cost of the set and total risk reduction provided. A table lets you see which investments are included in each set.Investment bundle cost and risk reduction graph46%
    
Indications of the typical effects on safety and other objectives, and of costs, for each of a range of typical safety investments.Investment cost-benefit analyses61%
    

Implications for decision support: Again, many respondents chose to take all available summaries, but some slight differences are still visible. Again, the assessment of all costs and (dis)benefits was the most popular. This means that decision support materials provided early in the task to give people a start in the task should provide this sort of analysis.

Again assuming that respondents were representative, we can be about 96% sure that the full assessment is judged helpful more often than the Probability-Impact Graph. We can be 98.3% sure that most people think full assessments are helpful.

In this task, suppose the expert was much less willing to provide detail and just offers you one of two lists.

9. Which of these lists would be more helpful?:

Content% of respondents*
a detailed list of typical safety 'risks' in factories54%
    
a detailed list of typical safety investments in factories46%
    

Implications for decision support: Slightly more respondents chose the list of 'risks' than chose the list of safety investments. Most likely both should be provided if possible to help people with this kind of task. The slightly higher selection of the set of risks contrasts with the answers to the previous question, where more respondents thought the list of typical investments would be helpful than thought the Probability-Impact Graph (with supporting list) would be helpful.

With the usual assumptions, we can be 90.2% sure that at least 40% of interested people think a detailed list of typical safety investments would be more useful than a detailed list of typical safety 'risks'. That is an important minority.

Now imagine that you have to think up safety investments for your factory but this time there is no expert to help you. However, you have an assistant with time to do any three of the following four things for you. One item must be left out and not done.

10. Which of these would probably be the least helpful?:

Summary formatShort name% of respondents*
A brainstormed list of safety 'risks', ranked based on their 'probability' and 'impact', for your factory (as judged by the brainstormers).Risk List37%
    
A list of safety investments that people in the factory have already thought of but about which no decision has been made so far.Existing Investment Ideas28%
    
A list of the injuries and other safety related incidents that have been recorded at the factory over the last few years.Past Incident List6%
    
A list of the changes in the factory since investment in improved safety was last seriously considered (i.e. places where safety might need rethinking).Recent Changes List29%
    

Implications for decision support: Objective information was the big winner from this question. Very few respondents were willing to go without the information on past incidents, while the most often left out summary was the brainstormed risk-list. The list of recent changes also did quite well, perhaps helped by the remark in the question explaining why the list might be helpful. This remark was added because, in another study (Leitch, 2012b), I found that people are very keen on efficient allocation of resources in risk-control projects but tend not to realise that the advantage of focusing on the most risky areas is greatly reduced by the fact that previous risk management exercises will already have done so. The list of recent changes was another source of objective information.

Now imagine that some research has been done by you and your capable and hard-working assistant but now you have to choose what summary to ask your assistant to prepare to support the final decisions.

11. What single summary would you ask for if it had to be one of these?:

Summary formatShort name% of respondents*
A table showing how much each safety 'risk' on a list could be reduced if all investments that have been considered were selected, showing also the risk reduction per unit cost of safety investments.Risk and risk reduction list9%
    
A table of safety investments showing the improvement in safety that would be achieved by each safety investment, its cost, and its safety improvement per unit cost.Investment cost and risk reduction list23%
    
A scatter plot where each dot represents a set of safety investments and the axes are 'total cost of safety investments' and 'total risk reduction'. A table lets you see which investments are in each set.Investment bundle cost and risk reduction graph22%
    
An assessment of the various effects on risk, on cost, and on other benefits and dis-benefits, for each potential safety investment.Investment cost-benefit analyses46%
    

Implications for decision support: The clear implication is that summaries to support decision making in this type of task should accomodate cost-benefit analyses of each potential investment.

(Note: A Probability-Impact Graph was not offered in this question because it contains no information about safety investments at all, and therefore could not be a candidate as the sole decision support summary.)

This question is very similar to question 7, with the main difference being that in question 11 there is no Probability Impact Graph on offer, and yet there is a very obvious difference in the results. The votes for the risk list with risk reduction information are much lower in question 11 than in question 7. I can offer no compelling reason for this difference but I speculate that it may be something to do with the cumulative experience of thinking about the survey questions.

The idea is that, as people worked through the questions, they were made to think more and more about the practical reality of the task and this gradually reduced the initial tendency of some to select answers that seemed familiar and theoretically appropriate. Answers to question 6 showed more people thinking the Probability-Impact Graph would be helpful than thinking the risk list with risk reduction information would be helpful. However, in question 7 the ranking of these two items was reversed. Now people realised that the risk reduction information meant the list with risk reduction information would be more useful. In answer to question 8 they continued to see the risk list with risk reduction information as better than the Probability-Impact Graph, and almost as useful as a cost-benefit analysis of each investment. Question 9 then asked for a simple choice between information about 'risks' and information about safety investments. Many chose the safety investments, and this may have highlighted the advantages of focusing on the investments.

There may also have been some confusion between the risk list with risk reduction information and a brainstormed risk list with probability and impact ratings, which was mentioned in question 10.

12. This box is for any comments, explanations, or suggestions you would like to make. Include your email if you would like me to respond.

Comments received (other than friendly greetings) were as follows:

"Brainstorming is an excellent way of creating a large quantity of duplicated drivel quickly. Some people like this and consider it valuable.
Q3. is dependent on the level of breakdown/detail.
Q9. A list of typical investments would be useful as it probably contains a number of low cost options that are no brainers (e.g. PPE, tidy site) and benefit from the learning of others. One drawback might be that choices are 'cultural' rather than strictly rational.
Q11. I am envisaging a senior management presentation, selecting investments on a bangs for the buck basis. Possibly graphic."

"I found the questions in this survey difficult to understand. The combination of passive voice and compound-complex sentences was at times frustrating. I found myself lost in the answers, as well; the complexity of the answers you provide made it difficult to analyze between them while attempting to determine an answer." (In fact the survey did not use the passive voice.)

"I can't visualize those scatter plots without an example. I don't know how one dot can represent a set of safety investments. To me a dot might represent one." (It seems it was the less familiar scatter plot of bundles of safety investments that was harder to understand for this respondent.)

"The important part in risk is analysing the 'resilience and recovery' from a unknown risk. Not all risks are recorded but the processes in place need to reflect the scope for the organisation to recover from the unknown risks - not just specified risks."

"For Q6 and Q7, it would be much better to provide the examples rather than having the reader 'visualize' what each choice would provide; that would make choice by this selector much easier.
I personally cannot visualize this: "A scatter plot where each dot represents a set of safety investments and the axes are 'total cost of safety investments' and 'total risk reduction'. A table lets you see which investments are in each set." Therefore, I would not choose it - but maybe it is worthwhile? Don't know." (Again, it is the unfamiliar scatter plot that has baffled this respondent.

"Interesting survey. I guess I would try and monetize the risk with a range of potential costs of each and a related probability, and then try and relate or match these with specific investments."

"A good set of thought-provoking questions. I wonder if you have seen the DEFRA portfolio work we had at one of our IRM NW meetings: http://www.theirm.org/events/documents/5AchilleasMavrellis-DEFRAportfoliomanagement.pdf."

"My instinct would be to make investment decisions only after having considered and evaluated the relative likelihoods and impacts (human and financial) of safety risks."

"In my personal opinion, an effective risk/control model would have been evolving if and when circumstances allow us to enhance the baseline model to reflect the ever changing reality."

"I found this difficult as my belief is if you want improved safety/risk statistics you need to change the safety/risk culture. Culture, in my opinion, is minimally influenced by graphs and lists, it takes listening and responding to what is being said (see answer to Q10.) to have safety/risk investments produce maximum returns." (In fact this respondent nevertheless made choices that were clear cut and consistent with the most popular views. Emails exchanged after the survey revealed that, in his experience, safety had been an important industrial relations issue, hence the emphasis on listening and responding.)

"I would also have regard to the appetite for risk in deciding."

"The challenge of decision support in risk management is that 'scientific' approaches are limited (in availability), particularly the data with which to decide between different possible control options. I have direct experience of doing this in the InfoSec where it was hugely beneficial, albeit then captured into tooling that was not sustained. There is nothing equivalent now available and the profession / discipline is all the poorer for it.
NB: control options (in a library) were indexed to or by:
1. The asset types for which they were appropriate
2. The threat types for which they were appropriate
3. The risk level at which they became justified and were no longer justified.
4. The control options' 'nature' (preventive, detective, reduce impact, threat, vulnerability, recovery).
5. The relative 'efficacy'.
6. The relative cost (VH <> VL) where metrics were the same as for the impact assessment element of the RA." (Comment by Stephen Daniels.)

"Some of the choices were hard to understand."

"Q10 was a tough question. It was very hard to decide what to leave out. I was tempted to leave out the list of injuries and safety incidents as it is likely that the root cause of such incidents would already have been addressed, but then I concluded that you couldn't really risk missing something by ignoring this area. Also, it's tempting to think that there is a reason that something remains on the list of unactioned safety investments (i.e. it's not very good), but that assumes rational decision-making. I found it interesting that although I personally preferred to compile data in a graphical format to make the decision, I didn't like relying on it to make the decision (I would want to dig down to the underlying data) and might have reservations about communicating to others in that format (because it would raise too many questions)."

"Very interesting - some of the choices were forced as I believe in real life I would not have had to make them."

"One of the key factors to consider when presenting risk decisions is how people take on information. A majority of people take on information visually i.e if you give them a diagrammatic or pictorial representation (e.g. scattergraph) they can better understand the point you are making. For me personally this is not particularly useful but is always important in considering what I need to show the audience. For me, I will use as much information as I can get - but how to summarise and present the conclusions is often the key to getting the right decisions."

"* Cost-benefits are much better than lists of risks or lists of investments.
* If it's necessary to choose, lists of investments are better than lists of risks. (I suspect the latter is more easily obtained through brainstorming or internet research than the former; effective solutions are the valuable bit.)
* A picture is worth many words; seeing what it really mean in practice would have made me more certain over my responses especially in Q7."

"Risk is about the likelihood of occurrence, and impact on the organisation that determines how it should be addressed."

"Interesting area to explore. Health and Safety (in the built environment at least) has become a top reporting/agenda item in recent years yet cost effectiveness is rarely considered in my experience. Cost benefit analysis and cost effectiveness is recommended in guidance to achieve ALARP status, yet this is rarely done due to difficulty in quantifying risk and following a precautionary approach. Interesting to see the results. The bottom line is that safety risk investment could be reduced if a more analytical approach to cost effectiveness were followed."

"In my opinion brainstorming and accurate likelihood estimation are mutually exclusive."

"A very interesting survey. I look forward to seeing your conclusions. It makes you think which is good and following this approach make you think about what measures should be taken; rather than having a series of un-costed mitigations."

"Q5. seems to presuppose that the level of uncertainty in the analysis is negligible. With such a cursory description, I couldn't bring myself to accept that assumption, but it might be appropriate if some additional background was provided to indicate its accuracy/reliability."

"Not sure that this reflects 'decision support under uncertainty' or Monte Carlo simulation to the extent that would be helpful. Also, no mention of enterprise goals, objectives, strategic initiatives and their impact on safety investment, etc."

"Perhaps it is just the presentation, but these approaches seem very foreign compared my experience of evaluating systems and then making decisions and actions to ensure safe operation." (from a risk analyst/scientist who selected cost-benefit analyses both times)

"Q3. I'd like to have said most safety investments affect non-target risks making them either bigger or smaller, but this was not an option.
Q4. Some safety investments will lead to additional unintended consequences which may be good OR bad.
Q5. Research by Slovic and others has looked at similar questions to these.
Q11. This question gets in the issue of other benefits and disbenefits which I missed in the earlier questions."

Respondents

Respondents were recruited through postings on internet discussion lists for risk analysts, risk managers, auditors, and performance managers, and then by direct email requests to about 850 people who have shown an interest in my work over the years.

The survey was completed by 97 people.

1. Your first language:

Most respondents had English as a first language:

% of
respondents
First language
77%English
3%German
3%French
2%Arabic
1%Afrikaans
1%Cantonese
1%Urdu
1%Chinese
1%Indonesian
1%Turkish
1%Spanish
1%Italian
1%Tooro
1%Greek
1%Japanese
1%Thai
1%Finnish
1%Dutch

2. Do you consider yourself to be a professional in any of these? (click all that apply)

Role% of respondents
Performance management18%
  
Risk analysis53%
  
Risk management57%
  
Audit36%
  

Differences between groups of respondents

When classified by the type of professional specialism, some groups were more focused on risk-lists and Probability-Impact Graphs than others.

A composite risk-list focus index was created using answers from several of the survey questions, giving values between 0 and 9.5 inclusive. Comparisons were possible with the larger groups, and for this purpose the specialism of performance management was ignored because it was uncommon. The least risk-list focused were risk analysts who also saw themselves as risk managers but not auditors (average risk-list focus of 2.6). The most risk-list focused where auditors who also saw themselves as risk managers but not as risk analysts (average risk-list focus 4.5). It seems that adding risk management to a risk analyst's expertise reduces risk-list focus (from 3.3 down to 2.6), but adding risk management to an auditor's expertise increases their risk-list focus (from 2.9 to 4.5).

Areas of expertiseAverage risk-list focus*
Risk analyst & risk manager
(but not auditor)
2.59
    
Pure risk analyst3.3
    
Pure auditor2.92
    
Auditor & risk manager
(but not risk analyst)
4.5
    

Taking just the most risk-list focused group, is there still a preference for cost benefit analysis over a risk-led decision support? Overall, yes. Even with the 8 auditors who also claimed expertise in risk management but not risk analysis, the following results were found:

  • Q3. All anticipated at least some safety investments would affect more than one 'risk'.

  • Q4. All expected benefits beyond risk reduction from safety investments.

  • Q5. Most preferred the safety investment that was most effective, regardless of its distribution of effects across causes.

  • Q6. Just as many thought the cost-benefit analysis of investments would be helpful as thought the probability-Impact Graph would be helpful, and in fact most thought that the cost-benefit analysis of investments would be helpful.

  • Q7. Most thought the most helpful summary would not be the Probability-Impact Graph, with half thinking it would be the scatter graph of investment bundles or the cost-benefit analysis.

  • Q8. In contrast to other groups, although most in this group thought the expert's cost-benefit analysis of typical safety investments would be helpful, even more thought the expert's typical Probability-Impact Graph would be helpful.

  • Q10. Also in contrast to other groups, more held on to the brainstormed risk-list than the list of safety investment ideas, though a quarter would still leave out the brainstormed risk-list.

  • Q11. However, their most popular summary for the final decision on safety investments was the cost-benefit analyses of safety investments, getting half their votes.

Although this extreme group is only small, these results suggest that, with almost any group of people interested in management, suggesting an analysis of all the effects and costs of each investment will be broadly supported, usually more so than suggesting risk-list oriented exercises.

Limitations of the survey

The survey suffered from a number of typical limitations.

  • Respondents were self-selected and from a group that is not entirely representative of people with an interest in management. They were people with an interest in risk and many were people who had shown a specific interest in my work. That makes them, to some extent, a special group. However, the analysis of different respondents in the survey shows that even the group most oriented towards risk-listing methods was still more interested in broad cost-benefit assessments of each investment and understood the reasons why this was a better choice.

  • Some respondents reported difficulty understanding some of the items and one person wrote to say he had not completed the survey because he found it too difficult to understand. This is typical for surveys about risk management methods because of the rather abstract and confusing subject matter. It means that respondents either have excellent powers of understanding or tolerate not being able to understand the questions fully. I suspect both were true for many respondents.

  • The descriptions of alternatives presented to respondents were not equal in length or complexity. The alternatives were also not of equal familiarity. This was probably to the advantage of the Probability-Impact Graph and overall cost-benefit analyses, relative to the other options.

Wider implications

Summaries for decision support under uncertainty, even in such areas as safety where risk-listing is a common approach, need to provide summaries organized by courses of action, to deal with complex relationships between risk and actions. The summaries also need to include benefits and disbenefits other than those perceived as risk modification.

The findings also suggest that the allocation of space within guides and standards for risk management should be changed. ISO 31000:2009, the international standard on risk management (which is largely a risk-listing approach) has 7.5 pages on its 'risk management process'. These include two pages on thinking about risks but only half a page on thinking of actions that could be taken. In the survey, respondents were about equally interested in expert ideas on 'risks' as they were in expert ideas for safety investments. This suggests that thinking of things to do deserves about as much space as thinking of 'risks' / objectives.

References

Cox, L.A. (2012). Evaluating and Improving Risk Formulas for Allocating Limited Budgets to Expensive Risk-Reduction Opportunities. Risk Analysis, 32(7).

ISO 31000:2009. Risk management: Principles and guidelines.

Leitch, M. (2011). Results of a survey on 'project risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_pram_report.shtml.

Leitch, M. (2012a). The risk-listing school. Available online at: http://www.workinginuncertainty.co.uk/risk_listing.shtml.

Leitch, M. (2012b). Results of a survey on corporate programmes to improve 'risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_prog_report.shtml.

 

Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.

Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.

Words © 2013 Matthew Leitch