Matthew Leitch, educator, consultant, researcher
SERVICES
OTHER MATERIAL
Working In Uncertainty
Results of a survey on 'integrated risk management'
Contents |
Many thanks to all who participated in this survey. As always, without your donations of time and thought this would not have been possible.
Summary
Respondents were asked to choose the more 'integrated' approach to 'risk management' out of a pair in 10 different scenarios, and asked if they would recommend either or both of the approaches.
The results of the survey provide rich information about what 'integrated' risk management means to people in practice. It shows that most people recognize it when they see it, and would be much more likely to recommend risk management that was integrated in that way.
None of the preferred, more integrated, more recommended approaches included listing risks, while seven of the less preferred approaches did. Clearly, listing risks is not essential for integrated risk management in the eyes of most people.
The scenarios and alternative approaches are commonly occurring in practice so even where generalizations across scenarios cannot be made the results of individual scenarios are still useful.
The survey method
The survey was completed by 111 people. Each respondent considered three simple scenarios (drawn randomly from a bank of 10) about management methods and was asked to choose for each the more 'integrated' example of risk management from two alternatives. They were also asked, for each alternative approach, whether or not they would recommend it.
Before the study began one approach from each pair was designated as the integrated approach. This was based on a variety of features but a commonality is that none of the designated integrated approaches involved making lists of risks. However, this was not revealed to respondents at any stage. Also, no explanation of the term 'integrated risk management' was provided and respondents were told that exploring the meaning of this was a goal of the survey.
Results
In all 10 scenarios it was the approach initially designated as more integrated that was chosen by most respondents. It was also the approach more likely to be recommended in each pair.
None of the preferred, more integrated approaches involved making lists of risks.
All 10 integrated approaches would be recommended by most respondents, but only 3 out of 10 of the other approaches would be recommended by most, and then the majorities were very small.
Across all scenarios, the more integrated approach was chosen by 75.1% of respondents, the more integrated approach would be recommended by 78.7%, and the less integrated approach would be recommended by only 39.0%.
It seems that people can recognize the more integrated approach, most of the time, and they much prefer it.
Here are the summary numbers for each scenario, starting with the scenarios in which the selection majority was largest:
| Scenario | Approaches | Chosen as best example of integrated risk management | Would recommend | ||||
|---|---|---|---|---|---|---|---|
| Explaining measurement uncertainty | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Devising policies on risk taking | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Applying policies on risk taking during planning | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Forecasting under uncertainty | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Appraisal of a proposed project | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Developing a plan | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Approaching a difficult decision | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Policy on project acceptance | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Strategy development | Integrated approach |
|
| ||||
| Other approach |
|
| |||||
| Searching for opportunities | Integrated approach |
|
| ||||
| Other approach |
|
|
The tendency to select the more integrated approach as being more integrated correlates well with the difference in recommendation percentages between the integrated and other approach. The linear correlation coefficient is 0.80. This confirms that people prefer the more integrated approach as well as recognizing it.
In the survey, respondents were asked to indicate if they considered themselves to be professionals in performance management, risk analysis, risk management, or audit. Regardless of these specialities, all groups of respondents showed the same pattern of preferences across scenarios and approaches. However, the risk oriented professionals (risk analysts, risk managers, and auditors) were slightly more likely to select the integrated approach.
Respondents whose first language was English (the language of the survey) were also slightly more likely to select the integrated approach.
This was an exploratory study, designed to cover a variety of situations and characteristics of integration. Each scenario is informative.
Explaining measurement uncertainty
Here is the scenario and the two approaches offered, with results from the 32 respondents offered this scenario:
| "A division’s top level performance report shows numbers on many aspects of performance, month by month, for the past year." | |
|---|---|
| "A) Measurement uncertainty for each line of numbers is explained in various ways e.g. sample size for survey data, confidence bands, narrative statements of known limitations." | "B) In a separate risk register document one of the risks is that the performance report may include incorrect data, leading to poor decisions." |
| Most integrated: 84.3% | Most integrated: 15.7% |
| 81.3% would recommend | 28.1% would recommend |
| Possible integration cues: + single document + tailoring to fit + high focus on uncertainty | Possible integration cues: - separate documents - listing of risks |
There were no comments by respondents on this scenario.
Devising policies on risk taking
Here is the scenario and the two approaches offered, with results from the 36 respondents offered this scenario:
| "A company wants to develop policies to prevent excessive risk taking." | |
|---|---|
| "A) It identifies the important types of decision it expects to take, considers the major uncertainties involved, and designs policies to suit those decisions." | "B) A team of executives brainstorms a list of risks and then policies are written stating limits on each of those risks." |
| Most integrated: 83.3% | Most integrated: 16.7% |
| 77.8% would recommend | 33.3% would recommend |
| Possible integration cues: + reference to decisions + use of 'uncertainties' + tailoring to fit | Possible integration cues: - use of 'risks' - fixed limits - meeting dedicated to 'risks' - listing of risks |
The task is what has, confusingly, been called 'articulating a risk appetite', by some. The British Standard on risk management, BS 31100:2011, is unusual in describing the process as one of formulating and applying policies designed to control risk taking decisions. This contrasts with the approach more often suggested prior to this, which was to put limits on each of a list of risks.
Respondents emphatically preferred the decision-focused approach as more integrated and were far more likely to recommend it.
One respondent made a comment specific to this scenario. With the advantage of hindsight we know that the first part of this comment is not consistent with the strong collective preference for the decision-focused approach:
"Not sure one is any better or more integrated than the other; I would recommend both risks and likely decisions are identified with a prioritised risk tolerance developed based on both. This overcomes lots of unnecessary limits for risks that are unlikely to be encountered - or can't be controlled anyway!"
Applying policies on risk taking during planning
The results are from the 27 respondents offered this scenario:
| "A company holds meetings to devise a plan for the year ahead and..." | |
|---|---|
| "A) ...during those meetings people check suggested plans against a list of policies designed to avoid excessive risk taking." | "B) ...after those meetings the plans are considered in a risk committee meeting to see if they comply with a list of policies designed to avoid excessive risk taking." |
| Most integrated: 81.5% | Most integrated: 18.5% |
| 74.1% would recommend | 37.0% would recommend |
| Possible integration cues: + single meeting + same group of people | Possible integration cues: - separate meeting - different group of people - group dedicated to 'risk' only |
This scenario is related to the previous one about devising policies on risk taking.
One respondent, auditor, made a comment specific to this scenario:
"While it appears A) is more integrated, I would feel more comfortable if, after A) was acted upon, B) was utilized to review/determine if decisions made in A) were appropriate. My reasoning is that the individuals involved in B) would most likely have more experience in risk analysis."
Forecasting under uncertainty
Here is the scenario and the two approaches offered, with results from the 37 respondents offered this scenario:
| "The board of directors of a company receives..." | |
|---|---|
| "A) ...a probabilistic forecast showing distributions of possible outcomes for profit and other measures, with narrative explaining the uncertainties involved and how alternative outcomes could arise." | "B) ...a financial forecast giving a best estimate of year end profit and other measures and then, later in the meeting, a risk register listing risks to the company’s objectives." |
| Most integrated: 81.1% | Most integrated: 18.9% |
| 86.5% would recommend | 35.1% would recommend |
| Possible integration cues: + single document + probabilistic forecast + use of 'uncertainties' + alternative outcomes | Possible integration cues: - separate documents - best guess - separate times in the meeting - listing of risks |
Two respondents commented on this scenario:
"B) is most likely, A) is a nice idea I'd go for but I would expect a board to send it back saying 'what do you really think'?"
"Human capacity to understand is also a temporal attribute; risk needs to be tightly integrated with potential financial implications."
The second part of the first comment suggests a slightly world-weary view of boards, but the results of various surveys I have carried out usually show that people generally do understand that uncertainty exists and is important. Most people, even board members, probably would not think it was a good idea to say "Yes, but what do you really think?" thus pushing the forecaster into making a guess and presenting it as an opinion.
Appraisal of a proposed project
The results are from the 35 respondents offered this scenario:
| "A charity is deciding whether to go ahead with a major project, or not. Their head of finance has prepared..." | |
|---|---|
| "A) ...an analysis of how the charity would be affected financially and in other ways by alternative possible outcomes of the project." | "B) ...a best-estimate prediction for how the charity will be affected financially by the project, with a note saying that there are risks involved and listing the assumptions made." |
| Most integrated: 77.1% | Most integrated: 22.9% |
| 85.7% would recommend | 40.0% would recommend |
| Possible integration cues: + alternative outcomes | Possible integration cues: - best guess - use of 'risks' - low emphasis on uncertainty |
One respondent commented:
"A) is most sensible. B) would be good IF the board had the experience to assess the risks or to know how likely they were or which had already happened to other people."
Developing a plan
The results are from the 30 respondents offered this scenario:
| "Marketers hired to launch a sugary drink must decide who to try to sell it to, and for what use (e.g. sports, parties, refreshment). They do some initial, general market research, brainstorm some ideas, then..." | |
|---|---|
| "A) ...try to predict the results of each idea, noting the biggest uncertainties. Then they do some extra research on the biggest uncertainties and try again, refining their ideas as they go." | "B) ...pick the one they like best and pass it on for a ‘marketing audit’. Part of this audit is listing risks and opportunities for the marketing idea." |
| Most integrated: 76.7% | Most integrated: 23.3% |
| 70.0% would recommend | 33.3% would recommend |
| Possible integration cues: + use of 'uncertainty' + iteration + same group of people | Possible integration cues: - best guess - use of 'audit' - different group of people - listing risks |
One respondent commented:
"Case does not indicate 'time to market' constraints; nor is there an option to address health issues of sugar in drinks, which is not a typical risk concern."
Approaching a difficult decision
The results are from the 25 respondents offered this scenario:
| "A government minister faces a difficult decision on a new banking regulation involving a lot of people, money, and uncertainty. As a first step in clarifying her thoughts, and before involving her advisors, she..." | |
|---|---|
| "A) ...draws a decision tree showing her alternatives and different things that could happen as a result." | "B) ...makes a list of risks she sees related to banking regulation and assesses their importance." |
| Most integrated: 72.0% | Most integrated: 28.0% |
| 84.0% would recommend | 52.0% would recommend |
| Possible integration cues: + focus on decision making + alternative outcomes | Possible integration cues: - listing risks |
Two respondents commented:
"I would have thought a combination of A) and B)."
"What a wonderful world if our minister had the talent (and inclination) to execute option A)."
Policy on project acceptance
The results are from the 39 respondents offered this scenario:
| "A company’s policies on approving projects use a ‘risk score’ based on a checklist of factors related to proposed projects. The highest acceptable risk score for a project is..." | |
|---|---|
| "A) ...somewhat more for projects with higher benefits." | "B) ...the same regardless of the benefits of the project." |
| Most integrated: 71.8% | Most integrated: 28.2% |
| 76.9% would recommend | 25.6% would recommend |
| Possible integration cues: + tailoring to fit | Possible integration cues: - not tailoring to fit |
One respondent commented:
"My response ... assumes the risk score is wholly negative since the postulation is risk score versus benefits. IF, however, the 'risk' score considers both +ve and -ve (as in 'risk' being both threat and opportunity) then my answer would be B)."
This comment brings to the light the fact that neither of the two approaches offered in the survey would be logical for all ways of creating a risk score from both positive and negative outcomes. The logical approach would be to vary the level of risk for higher benefits, not necessarily raise it.
Strategy development
The results are from the 44 respondents offered this scenario:
| "A company develops its strategy..." | |
|---|---|
| "A) ...through scenario planning i.e. thinking about drivers, imagining very different alternative futures, and developing strategies to work in some or all of those scenarios." | "B) ...on the basis of a best estimate forecast of the future. It then holds a risk management meeting to identify risks related to the strategy and actions to respond to those risks." |
| Most integrated: 63.6% | Most integrated: 36.4% |
| 79.5% would recommend | 54.5% would recommend |
| Possible integration cues: + alternative outcomes | Possible integration cues: - best estimate - separate meetings - meeting just on risk - risk listing |
One respondent commented:
"Although I believe A) is the most integrated - and have recommended that approach - I can also see elements of B) that I would also recommend. My thoughts are that after identifying scenarios any strategy must ultimately take the direction of the 'best estimate forecast' albeit whilst always recognising the limitations of that forecast and having alternative strategies for emerging situations. In addition I also recognise different hierarchies of risk: and whilst scenario planning is an integrated way of mitigating strategic risk, in my mind it will also be necessary to then consider tactical and operational risks for the implementation of that strategy."
Searching for opportunities
The results are from the 28 respondents offered this scenario:
| "A team of tax inspectors is given the names of 10 large companies and told to look for extra tax revenue from them. Perhaps some should pay more tax but there is no evidence yet. The team..." | |
|---|---|
| "A) ...does a quick assessment of all the companies using information it already has, picks the most promising 7, then does a more detailed analysis using existing information, picks 5 for limited reviews on site, and so on." | "B) ...makes a list of risks and opportunities involved in a typical company tax review then picks three companies at random and does a detailed review of their financial records and tax payments." |
| Most integrated: 60.7% | Most integrated: 39.3% |
| 67.9% would recommend | 53.6% would recommend |
| Possible integration cues: + iteration + decision focus | Possible integration cues: - risk listing |
One respondent commented:
"B) random sample is fine when you've already decided to audit, in real life tax inspectors would look for bad smells that guide A)"
Respondents
Invitations to participate in the survey were sent to the PMA Forum (an internet discussion list about performance management), the Auditnet discussion list (for auditors), the RISKANAL discussion list (for risk analysts), and a varied selection of my contacts. 75% of respondents stated their first language as 'English' or 'British English', with the rest having a variety of other first languages. Total response rate cannot be calculated but will have been low.
Clearly this is a limitation of the survey and the selection of respondents will have affected the pattern of answers given to some extent. However, two points suggest that this was not a significant problem:
The overall pattern is a very strong one, with massive majorities in most cases.
When different sub-groups were compared there seemed to be no major differences in their views.
Implications
The study did not identify all the underlying principles that determine whether an approach is 'integrated' or not, or that determine whether people like it.
However, it is clear from these results that approaches do not have to involve making lists of risks to be regarded as 'integrated risk management.' None of the 10 preferred approaches involved making lists of risks, while 7 of the less preferred approaches did.
The results are also useful because the scenarios are common ones and the competing approaches are also common. Some respondents, in their comments, indicated that they liked the integrated approach but thought others would choose the less integrated approach. Perhaps this is because they see the less integrated approaches as more common in practice or more commonly suggested. The survey results show that in fact most people prefer the integrated approaches, even though some may be less common at present.
Getting more integrated approaches adopted may be easier if they are suggested and then the views of all concerned are polled, rather than relying on the first reactions of a few to decide what is to be done.