Matthew Leitch, educator, consultant, researcher SERVICES OTHER MATERIAL |
Working In UncertaintyResults of a survey on 'risk' decision support
Many thanks to everyone who participated in this survey. Without your generous donations of time and thought this would have been impossible. SummaryIn an earlier study (Leitch, 2011) it emerged that almost everyone thinks project risk management should include all significant decisions, not just decisions about actions perceived as responses to risk(s). I have no doubt that a similar question about risk management generally would get a similar response. This brings into scope such decisions as choosing between suppliers, selecting new products to market, and choosing between a bridge or a tunnel to get a railway past a river. In these decisions it is obvious that all relevant factors should be considered, including uncertain effects (or 'risks' if you prefer). In this new survey a situation was presented in which the actions under consideration (the safety investments) were primarily responses to risk. Would people still think that conventional decision-making logic should be applied? Yes, most did think that conventional decision-making logic applies to those decisions too. They wanted information and summaries that directly helped them design and select the best bundle of investments, and they wanted to know about all relevant effects, not just cost and risk reduction. The survey described a hypothetical exercise to select investments in safety for a factory. This setting is typical of 'risk management' exercises. Respondents were asked questions about which summaries of information they would find helpful in thinking of safety improvements and in deciding which to do given limited (but somewhat uncertain) resources. The results of the survey provide rich information about what summaries people want in this sort of situation. They show that, in general, people prefer information that bears directly on the decisions they must make, and that includes all relevant considerations, not just cost and risk reduction. The most popular summaries were cost-benefit analyses of each possible investment in safety, which would include costs, risk reduction, and other benefits and dis-benefits. Summaries that were oriented towards 'risks' rather than towards the investments to be chosen were less popular, despite being the familiar focus of many corporate risk management processes and getting most of the attention in guidance and standards on risk management. These results should be helpful to risk managers who want to know what they should provide, to writers who want to know what to give guidance on, and to tool vendors who want to know what their tools should do for people. A note on the statistics and graphsMost of the graphs in this report look like a dark red bar chart with a grey overlay at the end. This overlay represents uncertainty about the true value in the population, based on the sample evidence, using Bayesian methods. The dark red part of the bar shows the range up to the 10th percentile of a probability distribution representing belief about the true percentage or average of the entire population (i.e. people with an interest in management). The pink part shows the range from the 10th to the 50th percentile. The light grey part shows the range from the 50th to the 90th percentile. In effect, the graph says that, given various assumptions (as usual in statistics), we can be 80% certain that the truth lies within the grey overlay area (i.e. pink and grey segments combined). In all cases the initial prior distribution was uniform. For questions requiring a choice between two answers the prior distribution was the beta distribution. For questions requiring a choice between more than two answers the prior distribution was the Dirichlet distribution. For the distribution of risk-listing focus a gamma distribution was used because the scores appeared to be roughly Poisson distributed. The survey questions and answersIn this section the text in italics is the text used in the survey itself. Other text discusses the implications for decision support. The first two questions in the survey were about the first language and skills of respondents. These are given and discussed in a later section. The survey then continued as follows. Imagine you will be required to select a set of investments in safety designed to make a factory safer for workers. These might include such things as protective clothing, training, protective screens, sensors and alarms, replacing machines, changes to wiring and pipes, revised rest times and shift patterns, and so on. The total budget available to spend on these improvements is not settled, but probably around £200,000 (say USD 320,000 or Euros 240,000). 3. If you were to write down a list of safety 'risks', then a list of safety investments, and then link the 'risks' to the safety investments that reduced them, which pattern of relationships would you most expect to see?:
Implications for decision support: This question asked people to imagine making a list of 'risks', which many will not normally do. It was, as with all the questions in this survey, a hypothetical situation. Nevertheless, this is a crucial question because, if at least some safety investments address more than one 'risk', then summaries for decision-making need to reflect that. It is not sufficient to list all information by 'risk'. It is not sufficient to make decisions one 'risk' at a time. A risk-register format, with 'risks' as the main organizing item and safety investments listed by the side of each 'risk' is not adequate on its own. It is necessary to map the 'risks' to safety investments and to summarise decision factors against the investments, or against bundles of investments. If the 'risks' are interpreted as objectives in a value model, or something similar, to support decision-making then it is even more obvious that decision support needs to focus on investments rather than only on the consequences for each objective, one objective at a time. 4. What proportion of safety investments do you think would give additional benefits, beyond safety improvements, that should be taken into consideration? (These extra benefits, if any, might include allowing faster working or increasing flexibility, for example.):
Implications for decision support: This is another crucial question. Since almost all respondents anticipated that at least some safety investments would have implications beyond their cost and reduction of risk then these other factors should be brought into the decision-making somehow. One respondent, in comments, pointed out that safety investments might have dis-benefits other than cost. Another pointed out that some safety investments might increase danger in some ways while decreasing it in others. Although the survey question did not mention these other effects they might also be common and important enough to need inclusion in any good decision support approach. Suppose just for a moment that you have only two alternative investments of the same cost, and that you have divided the causes of injury and death into two categories. The two investments have different effects:
5. Which course of action would you prefer?:
Implications for decision support: Most respondents thought that the safety investment providing the greatest improvement in safety overall was the best. In order to choose safety investments on the basis of their overall contribution to safety, as most respondents would want to, it is necessary to summarise the effect across all 'risks' / objectives. Just showing the total reduction in risk levels against each 'risk' will not be sufficient. In testing this question before the survey was launched it emerged that some respondents might think that spreading the effect of a safety investment over both cause categories would be 'politically' better in some way. The survey results suggest that at least some respondents may have seen it that way. A substantial minority chose the safety investment that was best overall but would revise the cause categories in order to show a spread of effect. A larger minority chose the safety investment that had effects spread over the initial cause categories, even though its effect was less overall. The reason some people prefer (the appearance of) spread effects is not clear. Ideally, the way we analyse risk into categories should not affect our ability to identify the best investments. We should arrive at the same choices regardless of how risk is initially carved up. In the risk-listing approach to risk management (Leitch, 2012a) the initial division into 'risks' does affect the choice of investments because of targets or thresholds for risk reduction applied to each 'risk'. It may be that this approach influenced some respondents to prefer the less effective investment, and other respondents chose to recategorise in order to persuade anyone following the risk-listing approach to agree with their selection of the more effective investment. These survey results suggest that decision-makers should not be so concerned about other people wanting to see spread effects across causes. Most people don't think this is important to them. (This is different from spreading effects across people. To focus resources on making some people much safer but ignoring the safety of others raises different issues.) 6. Returning to the more realistic situation where you have lots of potential safety investments, which of the following analyses do you think would help you select a good set of safety investments? Click all you think would be helpful.
Implications for decision support: Many respondents simply chose to take all available summaries, but even so a slight difference exists with the full cost-benefit analysis being selected more often than other items. Such analysis was considered helpful by 68% of respondents. So, decision support summaries for this kind of task should accommodate analyses of costs and (dis)benefits for each potential safety investment. This preference is consistent with the answers given earlier to questions 3 and 4. The statistical persuasiveness of the results is as follows. If we assume that the respondents were representative of the population of people interested in management then we can be about 77% confident that more of such people think the full assessments are more helpful than the Probability-Impact Graph. We can also be 99.98% sure that more than half of these people think the full assessments are helpful. Note that the Risk and Risk Reduction list is much more than a list of 'risks' because it has summary information about mitigation, including the amount of potential risk reduction per unit cost. It is the sort of summary whose properties were studied by Cox (2012), who showed it supports better decisions than just addressing the 'biggest' risks first. 7. Which do you think would be the most helpful?
Implications for decision support: This question was more discriminating than the previous one and shows more clearly the preference for cost-benefit analysis for each investment over other formats. Decision support for this kind of task should accommodate such analyses. Now imagine that you are also required to develop proposals for investments in safety, so now you have to think of the investments and analyse them as well as making the final choice of which investments to select. 8. An expert in the field of factory safety has offered to help with some summaries of information to get you started. Which of the following summaries would help you in your task? Select all that you think would be helpful.
Implications for decision support: Again, many respondents chose to take all available summaries, but some slight differences are still visible. Again, the assessment of all costs and (dis)benefits was the most popular. This means that decision support materials provided early in the task to give people a start in the task should provide this sort of analysis. Again assuming that respondents were representative, we can be about 96% sure that the full assessment is judged helpful more often than the Probability-Impact Graph. We can be 98.3% sure that most people think full assessments are helpful. In this task, suppose the expert was much less willing to provide detail and just offers you one of two lists. 9. Which of these lists would be more helpful?:
Implications for decision support: Slightly more respondents chose the list of 'risks' than chose the list of safety investments. Most likely both should be provided if possible to help people with this kind of task. The slightly higher selection of the set of risks contrasts with the answers to the previous question, where more respondents thought the list of typical investments would be helpful than thought the Probability-Impact Graph (with supporting list) would be helpful. With the usual assumptions, we can be 90.2% sure that at least 40% of interested people think a detailed list of typical safety investments would be more useful than a detailed list of typical safety 'risks'. That is an important minority. Now imagine that you have to think up safety investments for your factory but this time there is no expert to help you. However, you have an assistant with time to do any three of the following four things for you. One item must be left out and not done. 10. Which of these would probably be the least helpful?:
Implications for decision support: Objective information was the big winner from this question. Very few respondents were willing to go without the information on past incidents, while the most often left out summary was the brainstormed risk-list. The list of recent changes also did quite well, perhaps helped by the remark in the question explaining why the list might be helpful. This remark was added because, in another study (Leitch, 2012b), I found that people are very keen on efficient allocation of resources in risk-control projects but tend not to realise that the advantage of focusing on the most risky areas is greatly reduced by the fact that previous risk management exercises will already have done so. The list of recent changes was another source of objective information. Now imagine that some research has been done by you and your capable and hard-working assistant but now you have to choose what summary to ask your assistant to prepare to support the final decisions. 11. What single summary would you ask for if it had to be one of these?:
Implications for decision support: The clear implication is that summaries to support decision making in this type of task should accomodate cost-benefit analyses of each potential investment. (Note: A Probability-Impact Graph was not offered in this question because it contains no information about safety investments at all, and therefore could not be a candidate as the sole decision support summary.) This question is very similar to question 7, with the main difference being that in question 11 there is no Probability Impact Graph on offer, and yet there is a very obvious difference in the results. The votes for the risk list with risk reduction information are much lower in question 11 than in question 7. I can offer no compelling reason for this difference but I speculate that it may be something to do with the cumulative experience of thinking about the survey questions. The idea is that, as people worked through the questions, they were made to think more and more about the practical reality of the task and this gradually reduced the initial tendency of some to select answers that seemed familiar and theoretically appropriate. Answers to question 6 showed more people thinking the Probability-Impact Graph would be helpful than thinking the risk list with risk reduction information would be helpful. However, in question 7 the ranking of these two items was reversed. Now people realised that the risk reduction information meant the list with risk reduction information would be more useful. In answer to question 8 they continued to see the risk list with risk reduction information as better than the Probability-Impact Graph, and almost as useful as a cost-benefit analysis of each investment. Question 9 then asked for a simple choice between information about 'risks' and information about safety investments. Many chose the safety investments, and this may have highlighted the advantages of focusing on the investments. There may also have been some confusion between the risk list with risk reduction information and a brainstormed risk list with probability and impact ratings, which was mentioned in question 10. 12. This box is for any comments, explanations, or suggestions you would like to make. Include your email if you would like me to respond. Comments received (other than friendly greetings) were as follows: "Brainstorming is an excellent way of creating a large quantity of duplicated drivel quickly. Some people like this and consider it valuable. "I found the questions in this survey difficult to understand. The combination of passive voice and compound-complex sentences was at times frustrating. I found myself lost in the answers, as well; the complexity of the answers you provide made it difficult to analyze between them while attempting to determine an answer." (In fact the survey did not use the passive voice.) "I can't visualize those scatter plots without an example. I don't know how one dot can represent a set of safety investments. To me a dot might represent one." (It seems it was the less familiar scatter plot of bundles of safety investments that was harder to understand for this respondent.) "The important part in risk is analysing the 'resilience and recovery' from a unknown risk. Not all risks are recorded but the processes in place need to reflect the scope for the organisation to recover from the unknown risks - not just specified risks." "For Q6 and Q7, it would be much better to provide the examples rather than having the reader 'visualize' what each choice would provide; that would make choice by this selector much easier. "Interesting survey. I guess I would try and monetize the risk with a range of potential costs of each and a related probability, and then try and relate or match these with specific investments." "A good set of thought-provoking questions. I wonder if you have seen the DEFRA portfolio work we had at one of our IRM NW meetings: http://www.theirm.org/events/documents/5AchilleasMavrellis-DEFRAportfoliomanagement.pdf." "My instinct would be to make investment decisions only after having considered and evaluated the relative likelihoods and impacts (human and financial) of safety risks." "In my personal opinion, an effective risk/control model would have been evolving if and when circumstances allow us to enhance the baseline model to reflect the ever changing reality." "I found this difficult as my belief is if you want improved safety/risk statistics you need to change the safety/risk culture. Culture, in my opinion, is minimally influenced by graphs and lists, it takes listening and responding to what is being said (see answer to Q10.) to have safety/risk investments produce maximum returns." (In fact this respondent nevertheless made choices that were clear cut and consistent with the most popular views. Emails exchanged after the survey revealed that, in his experience, safety had been an important industrial relations issue, hence the emphasis on listening and responding.) "I would also have regard to the appetite for risk in deciding." "The challenge of decision support in risk management is that 'scientific' approaches are limited (in availability), particularly the data with which to decide between different possible control options. I have direct experience of doing this in the InfoSec where it was hugely beneficial, albeit then captured into tooling that was not sustained. There is nothing equivalent now available and the profession / discipline is all the poorer for it. "Some of the choices were hard to understand." "Q10 was a tough question. It was very hard to decide what to leave out. I was tempted to leave out the list of injuries and safety incidents as it is likely that the root cause of such incidents would already have been addressed, but then I concluded that you couldn't really risk missing something by ignoring this area. Also, it's tempting to think that there is a reason that something remains on the list of unactioned safety investments (i.e. it's not very good), but that assumes rational decision-making. I found it interesting that although I personally preferred to compile data in a graphical format to make the decision, I didn't like relying on it to make the decision (I would want to dig down to the underlying data) and might have reservations about communicating to others in that format (because it would raise too many questions)." "Very interesting - some of the choices were forced as I believe in real life I would not have had to make them." "One of the key factors to consider when presenting risk decisions is how people take on information. A majority of people take on information visually i.e if you give them a diagrammatic or pictorial representation (e.g. scattergraph) they can better understand the point you are making. For me personally this is not particularly useful but is always important in considering what I need to show the audience. For me, I will use as much information as I can get - but how to summarise and present the conclusions is often the key to getting the right decisions." "* Cost-benefits are much better than lists of risks or lists of investments. "Risk is about the likelihood of occurrence, and impact on the organisation that determines how it should be addressed." "Interesting area to explore. Health and Safety (in the built environment at least) has become a top reporting/agenda item in recent years yet cost effectiveness is rarely considered in my experience. Cost benefit analysis and cost effectiveness is recommended in guidance to achieve ALARP status, yet this is rarely done due to difficulty in quantifying risk and following a precautionary approach. Interesting to see the results. The bottom line is that safety risk investment could be reduced if a more analytical approach to cost effectiveness were followed." "In my opinion brainstorming and accurate likelihood estimation are mutually exclusive." "A very interesting survey. I look forward to seeing your conclusions. It makes you think which is good and following this approach make you think about what measures should be taken; rather than having a series of un-costed mitigations." "Q5. seems to presuppose that the level of uncertainty in the analysis is negligible. With such a cursory description, I couldn't bring myself to accept that assumption, but it might be appropriate if some additional background was provided to indicate its accuracy/reliability." "Not sure that this reflects 'decision support under uncertainty' or Monte Carlo simulation to the extent that would be helpful. Also, no mention of enterprise goals, objectives, strategic initiatives and their impact on safety investment, etc." "Perhaps it is just the presentation, but these approaches seem very foreign compared my experience of evaluating systems and then making decisions and actions to ensure safe operation." (from a risk analyst/scientist who selected cost-benefit analyses both times) "Q3. I'd like to have said most safety investments affect non-target risks making them either bigger or smaller, but this was not an option. RespondentsRespondents were recruited through postings on internet discussion lists for risk analysts, risk managers, auditors, and performance managers, and then by direct email requests to about 850 people who have shown an interest in my work over the years. The survey was completed by 97 people. 1. Your first language: Most respondents had English as a first language:
2. Do you consider yourself to be a professional in any of these? (click all that apply)
Differences between groups of respondentsWhen classified by the type of professional specialism, some groups were more focused on risk-lists and Probability-Impact Graphs than others. A composite risk-list focus index was created using answers from several of the survey questions, giving values between 0 and 9.5 inclusive. Comparisons were possible with the larger groups, and for this purpose the specialism of performance management was ignored because it was uncommon. The least risk-list focused were risk analysts who also saw themselves as risk managers but not auditors (average risk-list focus of 2.6). The most risk-list focused where auditors who also saw themselves as risk managers but not as risk analysts (average risk-list focus 4.5). It seems that adding risk management to a risk analyst's expertise reduces risk-list focus (from 3.3 down to 2.6), but adding risk management to an auditor's expertise increases their risk-list focus (from 2.9 to 4.5).
Taking just the most risk-list focused group, is there still a preference for cost benefit analysis over a risk-led decision support? Overall, yes. Even with the 8 auditors who also claimed expertise in risk management but not risk analysis, the following results were found:
Although this extreme group is only small, these results suggest that, with almost any group of people interested in management, suggesting an analysis of all the effects and costs of each investment will be broadly supported, usually more so than suggesting risk-list oriented exercises. Limitations of the surveyThe survey suffered from a number of typical limitations.
Wider implicationsSummaries for decision support under uncertainty, even in such areas as safety where risk-listing is a common approach, need to provide summaries organized by courses of action, to deal with complex relationships between risk and actions. The summaries also need to include benefits and disbenefits other than those perceived as risk modification. The findings also suggest that the allocation of space within guides and standards for risk management should be changed. ISO 31000:2009, the international standard on risk management (which is largely a risk-listing approach) has 7.5 pages on its 'risk management process'. These include two pages on thinking about risks but only half a page on thinking of actions that could be taken. In the survey, respondents were about equally interested in expert ideas on 'risks' as they were in expert ideas for safety investments. This suggests that thinking of things to do deserves about as much space as thinking of 'risks' / objectives. ReferencesCox, L.A. (2012). Evaluating and Improving Risk Formulas for Allocating Limited Budgets to Expensive Risk-Reduction Opportunities. Risk Analysis, 32(7). ISO 31000:2009. Risk management: Principles and guidelines. Leitch, M. (2011). Results of a survey on 'project risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_pram_report.shtml. Leitch, M. (2012a). The risk-listing school. Available online at: http://www.workinginuncertainty.co.uk/risk_listing.shtml. Leitch, M. (2012b). Results of a survey on corporate programmes to improve 'risk management'. Available online at: http://www.workinginuncertainty.co.uk/study_prog_report.shtml. |
Company: The Ridgeway Expertise Company Ltd, registered in England, no. 04931400.
Registered office: 29 Ridgeway, KT19 8LD, United Kingdom.
Words © 2013 Matthew Leitch