Matthew Leitch, educator, consultant, researcher
SERVICES
OTHER MATERIAL
Working In Uncertainty
Results of a survey on corporate programmes to improve 'risk management'
Contents |
Introduction and summary
Many thanks to everyone who tackled this survey. Without your choices and many comments this study would have been impossible.
Most respondents thought that ideas for improved working practices to deal with uncertainty would not be devised only at the start of a programme of work. They were very much interested in directing resources at the most worthwhile improvements. The prompts most likely to give rise to improvement ideas such as those that have tested well in a previous survey (see Results of a survey on 'integrated risk management') were the more specific ones, and the most popular number of ideas to include on a prompt list for a meeting was seven.
So, it looks like if you want to get approval for a programme to look for and implement changes to ways of working that improve performance under uncertainty then you should suggest something with:
a heavy emphasis on focusing resources on the most worthwhile changes (not necessarily the biggest 'risks');
the ability to keep gathering, selecting, and implementing new ideas throughout the programme, not just at the start; and
bundles of seven specific prompts to help people think of good ideas rather than just putting up a generic risk management cycle.
Also, do not suggest that the objective of the programme is to manage risk levels down below limits listed against 'risks' on a risk register. Only 19% of respondents thought that was a 'helpful' objective.
The survey results
The survey was completed by 73 people. Here are the main questions that were asked, with percentages of respondents choosing each available answer:
"Do you consider yourself to be a professional in any of these? (click all that apply)"
| 25% | Performance management |
| 41% | Risk analysis |
| 45% | Risk management |
| 42% | Audit |
(A further 15% of respondents clicked none of the above.)
"1. In a corporate programme to improve the way risk/uncertainty is managed, in an integrated way, when will ideas for improvements arise? (Please select the most likely in practice.)"
| 1% | All the good ideas will be found in an initial analysis and design phase of the programme. |
| 99% | Good ideas will continue to arise throughout the life of the programme as more is learned from trials and other research. |
"2. Select all those you think are acceptable ways for ideas for improvement to arise in practice."
| Way for ideas to arise | % | % as graph | ||
|---|---|---|---|---|
| The most valuable activities are identified and studied for improvements | 73% | |||
| Processes are drawn up and studied for possible improvements | 67% | |||
| A disappointment/shock leads to a specific search for a better way of working | 88% | |||
| A search is made for places to use an idea found outside an organization | 55% | |||
| A search is made for places to use a good practice found inside an organization | 58% | |||
| A search is made for places to use an idea worked out in principle | 42% | |||
| An outsider suggests a particular application for a particular idea | 58% | |||
| Improvements are searched for in areas where people are most willing to change | 49% | |||
| A defined risk that is a particular concern is used as a prompt for ideas | 89% | |||
| An area of uncertainty of particular importance is used as a prompt for ideas | 75% | |||
| Major areas of uncertainty are listed and studied | 68% | |||
| Specific risks are listed and studied | 79% |
"3. Select all those you think are reasonable ways to encourage improvements within a corporate programme."
| Way to encourage improvements | % | % as graph | ||
|---|---|---|---|---|
| Collect promising ideas and encourage people to find places to use them | 48% | |||
| A process led analysis | 71% | |||
| A risk/uncertainty led analysis | 81% | |||
| An activity priority led analysis (i.e. starting with the most important activities) | 82% | |||
| Responding when people have an appetite for improvements (opportunistic) | 63% | |||
| Agree standard practices and implement them consistently | 52% |
"One way to help people have specific ideas for improvements is to make a suggestion, or a number of suggestions. These might be written or spoken."
"The following ideas for integrated risk management practices (shown in italics) are each next to two possible suggestions (A and B) that might be used to help people think of those ideas. In each case, choose the suggestion that, if used, would be most likely to lead to the idea."
| Scenario | % of respondents | Prompt |
|---|---|---|
| "4.1. A division’s top level performance report shows numbers on many aspects of performance, month by month, for the past year. Measurement uncertainty for each line of numbers is explained in various ways e.g. sample size for survey data, confidence bands, narrative statements of known limitations." | 21% | Suggestion A: Implement the risk management process in all your management processes (i.e. establish context → assess risks → treat risks → monitor and review). |
| 79% | Suggestion B: Where management information is used, implement practices that help people understand and respond to measurement uncertainty. | |
| "4.2. The board of directors of a company receives a probabilistic forecast showing distributions of possible outcomes for profit and other measures, with narrative explaining the uncertainties involved and how alternative outcomes could arise." | 75% | Suggestion A: Where forecasts or predictions are made, implement practices so that the forecasts show and explain the alternative outcomes that are possible. |
| 25% | Suggestion A: Implement the risk management process in all your management processes (i.e. establish context → assess risks → treat risks → monitor and review). | |
| "4.3. Marketers hired to launch a sugary drink must decide who to try to sell it to, and for what use (e.g. sports, parties, refreshment). They do some initial, general market research, brainstorm some ideas, then try to predict the results of each idea, noting the biggest uncertainties. Then they do some extra research on the biggest uncertainties and try again, refining their ideas as they go." | 19% | Suggestion A: Implement the risk management process in all your management processes (i.e. establish context → assess risks → treat risks → monitor and review). |
| 81% | Suggestion B: When developing plans, use uncertainties around predictions of results as a guide to what research and risk mitigation to do. |
"5. If a set of suggestions such as the more specific ones in the previous question were used, what would be a reasonable number to provide to a person or group (e.g. to help them in a workshop)? Please select the best number from this list:"
| No. of prompts | % of respondents | % as graph | ||
|---|---|---|---|---|
| 1 | 0% | |||
| 3 | 30% | |||
| 7 | 40% | |||
| 10 | 16% | |||
| 20 | 8% | |||
| 50 | 1% | |||
| 100 | 3% | |||
| 200 | 0% | |||
| more | 1% |
"6. What objectives should a corporate programme designed to promote better risk management have? (Select all those that seem helpful.)"
| Objective | % of respondents | % as graph | ||
|---|---|---|---|---|
| Achieve consistent standards of practices | 51% | |||
| Increase the pace of worthwhile changes to ways of working | 52% | |||
| Get risk level assessments for a list of risks down below specified levels | 19% | |||
| Focus resources on the most worthwhile improvements | 90% | |||
| Focus resources on the improvements leaders most want | 26% | |||
| Stimulate a strong flow of good ideas for improvements | 79% |
Implications
The implications of responses to question 1 are clear. All respondents but one thought ideas for improvement would not be discovered completely in an initial phase of design, and this clearly indicates that any method for running a programme of improvements in the area of uncertainty will need to allow for, and probably encourage, an ongoing flow of ideas. Even if it is possible to generate all ideas in an initial design phase, few people will happily attempt it.
Answers to questions 4 show that most people think specific prompts are more likely to generate the ideas for change preferred in a previous survey (see Results of a survey on 'integrated risk management') than a generic 'risk management' process prompt. Just six respondents out of 73 chose the generic risk management prompt for all three scenarios, whereas 45 respondents chose the specific prompt for all scenarios.
Answers to question 5 show that the most attractive number of prompts for a single workshop is about 7 (or slightly fewer), so most likely a larger bank of prompts would have to be used for the programme as a whole, with sub-sets being selected for particular design meetings.
The implications of answers to questions 2, 3, and 6 are much less clear. The table below shows the results again, but this time with question 2, 3, and 6 side by side, and with items listed in descending order of preference. The coloured backgrounds pick out items whose answers seem contradictory to some extent across the three questions.
Although the responses give a broad indication of preference, many respondents apparently did not answer question 2 as it was worded, and a similar range of interpretations may have affected question 3 and perhaps even question 6. Although most respondents seem to have interpreted question 2 as asking if they would consider ideas from a source, others seem to have interpreted it as asking if they would accept the practices that had led to the ideas, and others thought it was about how likely it was that ideas would arise that way, or that such ideas would be accepted for implementation.
However, even taken as broad indications of preference, the responses seem contradictory at first. The best explanation I can offer is that respondents' strongest desire is for resources to be focused on the most worthwhile improvements. (The top answer in Q6.) This, combined with a slight preference for familiarity, may explain a lot of the contradictions, as follows:
| Apparent contradiction | Possible explanation |
|---|---|
| Risk limitation (coloured pink) was the least popular objective, by far, but a risk/uncertainty led analysis was the second most often 'reasonable' approach, and risk/uncertainty sources dominated Q2. | Risk limitation lacks a sense of focusing resources efficiently, giving it a low rating, while risk/uncertainty in the other questions also benefited slightly from being the most familiar idea for many respondents. |
| Stimulating a strong flow of good ideas for improvements (coloured blue) was the second most popular objective, and yet the related approach of collecting promising ideas was deemed 'reasonable' by slightly fewer than half of respondents and idea-related items rested at the bottom of the Q2 table. | The idea-related items in Q3 and Q2 suffered from a lack of apparent focus on value. Perhaps the description of the ideas in Q3 as 'promising' was not enough to suggest focus on value while the Q2 items offered even less emphasis on efficient use of resources. |
| Focusing resources on the improvements leaders most want (coloured green) was the second least popular objective, and weak in Q3, but the second most 'acceptable' source of ideas in Q2 was in response to a shock or disappointment, which is surely a common reason for there being a willingness to change. | The idea of doing what people want perhaps conflicts with the ideal of doing what is really most valuable, but in the case of responding to a shock or disappointment, the recent experience is taken as evidence that there is a real, important issue that needs to be addressed and this confers a sense of efficient focus of resources. |
| In Q3 an activity priority led analysis (coloured yellow) was the most often reasonable approach, ahead of a risk/uncertainty led analysis, but in Q2 it was less often acceptable than most of the risk/uncertainty items. | The differences are mostly small, but it is the top item in Q2 that really pulls ahead and it contains the phrase "a particular concern", which may have created the desired sense of focus on priorities. |
Focusing scarce resources where they can do most good was highly favoured by respondents and seems to have been the main driver of opinions. This was the programme objective most often considered helpful and the element of focus probably explains some of the apparent contradictions in responses.
If these explanations are correct then the main implication is that, whatever approach is taken for the programme as a whole, it needs to feature mechanisms that focus resources efficiently in ways that are explicitly stated. Without this an approach is unlikely to get support. In particular, I suggest the following implications:
Elements of the programme designed to generate a strong flow of good ideas need to be guided in some obvious way to favour ideas likely to be of high value.
If the approach to prioritization uses any of the following effects then they need to be explicitly stated or many people will not recognize them:
Effect 1: Obvious higher priority activities and uncertainties tend to have been thought about more already, reducing the chance of finding further worthwhile improvements. The focus should be on changes and on activities whose importance has previously been underestimated.
Effect 2: New ideas are more likely to lead to new improvements than ideas that have already been considered and perhaps are in use already.
Effect 3: The cost of making changes in areas where people are unwilling to change is likely to be higher on average, particularly when you take account of the initiatives that fizzle out despite protracted effort.
Effect 4: When people are reluctant to make improvements that may be for good reasons not known to others.
Comments by respondents
A number of respondents made comments that explained or added to answers given. The key parts of comments were as follows:
Explanations of answers given
"Q2 [on acceptable ways for ideas to arise] - none of the methods are unreasonable in themselves but might be unreasonable in some contexts e.g. A disappointment/shock leads to a specific search for a better way of working might be a safety context."
"Q5 [on the best number of prompts] is difficult - it depends on what is found that needs to be done. I suppose 3 is realistic, but in practice you need to supply 10 so they can knock out the straw men and arrive at what they want. I'll pick the magic 7. For most of the others I picked what seemed to be the most important rather than just clicking All as most seem to be relevant."
"Q2 [on acceptable ways for ideas to arise] and Q3 [on reasonable forms of encouragement] - Among the various approaches that may be acceptable and reasonable, some may turn people on and others may turn people off, but a lot will depend on the personalities involved."
"Curiously on Q4.3 [on best prompt for research] I was quite marginal between the two options (but not so on Q4.1 [on best prompt for measurement uncertainty] and Q4.2 [on best prompt for forecasting]). I think this was to do with the likely audience for what gets developed and their expertise (and possibly interest) down either route. In Q4.1 [on best prompt for measurement uncertainty] and Q4.2 [on best prompt for forecasting] the connection to the business may need to be almost childishly simple to generate management attention! On Q6 [on helpful objectives] I am assuming (a) that the goodwill in getting leaders' improvements is worthwhile and that their desires are sensible and (b) getting risks down to a level involves at least implicit cost-benefit considerations i.e. is not arbitrary."
"At Q4 the presentation of suggestions 'A' & 'B' seemed to me at first glance to suggest that they are mutually exclusive. However, when I looked at these more closely I recognised them to be two sides of the same coin. One is the 'bare bones' text book 'risk management process' - the other seems to me to be the 'fully dressed' result of following that process, albeit perhaps subconsciously and informally rather than necessarily consciously following a prescribed process. I think the key thing is the delivery of good quality of thinking rather than whether this is ideally achieved through prescribed formal (risk) processes or perhaps more informally through subconscious competence."
"I didn't quite 'get' Q4 [on best prompts] - as both suggestions are in fact the same throughout - in that if you implement the risk management process then the other suggestion (A or B) should also be happening as part of that process."
"I have answered Q2 [on acceptable ways for ideas to arise] and Q3 [on reasonable forms of encouragement] on the basis that the points are acceptable to me and reasonable to me. Most of them strike me as extremely unlikely so presumably they are not acceptable or reasonable to some others."
"Regarding Q2 [on acceptable ways for ideas to arise], Q3 [on reasonable forms of encouragement], and Q6 [on helpful objectives], all your options are acceptable, reasonable and should be promoted, respectively. However, the capacity to do all of that commonly forces prioritisation wholely dependent on the resources available."
"There's a bit of background as to why I've responded to Q4 [on best prompts] the way I have. The 'risk management' process is still seen as being separate to the way of doing business - it is an add on and not part and parcel of how people approach things. I'm basing this on ten years in OpRisk - risk profiles/registers are reviewed and used infrequently and projects still overly focus on project risks (time/budget/deliverables) and not on the end impact once delivered. So my perception is that it is better to build in uncertainty thinking into the actual processes that management use in a way that makes sense to them as opposed to specifically using the standard risk management steps. Oh and as for the number of prompts - bit of a toss up between 3 and 7. The challenge a lot of people would find is coming up with more than 3-5 different scenarios. Beyond that level I think that they would become all variations on a theme."
"I struggled a little to understand what 'Implement the risk management process in all your management processes' would mean in practice. I shied away from searching for places to use good practice [in Q2, on acceptable ways for ideas to arise] as it sounded like a recipe for duplicating effort and/or increasing complexity."
"In Q2 [on acceptable ways for ideas to arise] and Q3 [on reasonable forms of encouragement] I haven't ruled anything out, although I think that clearly some routes are preferable to others. Just because there is desire or enthusiasm for change / improvement doesn't make it right that there should be some, or that it is in those areas that improvements should be prioritised. Good ideas can come from anywhere at any time, focus should always be on improvement in areas of key objectives, or to take out least value add, or maximise benefit / efficiency. Risk management is a great technique to assist in improvement, but is only ever a part of the process. Clear definition of a problem, and analysis and measurement of what underlies that problem are critical before development of a solution can take place. Risk management is a key part of this end-to-end process - and continues right through to ensuring the solution / improvement is operating and delivering what it was expected to deliver (or not). Q5 [on the best number of prompts] was difficult to answer; too many options tends to confuse, however if the process I have set out above is followed it is likely that a clear solution delivering improvement will emerge without requiring many options to be presented."
"Q2 [on acceptable ways for ideas to arise]. In healthy organizations complementary tactics for improvement are applied. Some opportunism is OK, since it might increase the momentum of success e.g. search for improvements where people are willing to change. Most search answers are way too much "push"; we have a hammer, let's start looking for anything to hammer on. Q3 [ on reasonable forms of encouragement] is for me close to Q2 [on acceptable ways for ideas to arise]. Q4.1 [on best prompt for measurement uncertainty] I have seen too many "cockpits" with huge amounts of data, but no interpretation and follow-up, hence B. Maybe when a company is "mature and ripe" then A can work in such company. Q4.2 [on best prompt for forecasting] idem, hence A. Q4.3 Many people get lost in too much data, unable to filter out the relevant. The risk management process as described here could function when all participants live and understand it. I have never seen that. Mostly I see that people 'go through the motions' as prescribed. Hence B. Q5 [on the best number of prompts]: I would in due time provide employees with a much richer toolbox with suggestions, e.g. tens or even hundreds. In a workshop, a practical limit is ~3. Q6 [on helpful objectives]: This list seems quite limited. My first priority would be increased awareness and understanding of the status quo, both for leaders and employees."
"Q5 [on the best number of prompts] does not allow for differences between individual and group. Three good suggestions are a good start if they cover the waterfront (i.e. are constructed so as to look specifically at independent solutions). Three is definitely not enough where group facilitation does not set out to achieve this objective. If ideas are randomly generated, and especially if they are shared as they are conceived, 50 may not be enough. In other words, this depends very strongly on how you phrase the task and define the conditions of satisfaction."
"I tended to feel 'all' in your listed options questions. Have therefore been a little more discerning."
Additional comments
"Given a competitive, changing business environment, I'm thinking about the links between effective, integrated risk management with cultural factors e.g. 'the learning organisation' (which switches people onto a spirit of continuous improvement) with the openness to ideas from anyone in the organization. I think about blind spots and gaps (for example, in the vision, perceptions and focus of the leadership team...and in what communication is shared more widely). Experience also demonstrates that feeling ownership is key (with clear roles and responsibilities), and then it's back to culture and levels of people-engagement to 'glue' an integrated approach together. I also wonder about risk management maturity models, to paint a 'word picture' of where an organization is now and where to head next, in terms of best practice and 'excellence'."
"It may be useful to clarify what the minimum audit criteria for risk management are e.g. IIA or ISO 31000?"
"We can't lose sight of the fact that risk is a good thing in business. Risk leads to opportunities. Unrecognized risk or mismanaged risk are threats to the business and should be corrected. However, it is a sound business strategy to recognize and leverage risk."
"Need to consider tracking and response methods and processes, and the attributes of such methods and processes: usability, reliability, maintainability and so on. Need to consider incentives and rewards related to achieving goals with respect to risk management and performance management. Time is important, as shorter life cycles for products and services reduce predictability."
"I think this survey depicts an overall ERM process (like ISO 31000) in the worst possible light. Certainly its prescriptive approach to process is of little (or more likely negative) benefit when the risk and uncertainty relate to a well defined objective as in the specific examples given here. Still, I can imagine framing a survey like this to reflect the value of a coherent overall ERM process as a kind of high level uniting perspective. To my mind, the biggest problem is that the current situation does not enable assembly of good technical risk analysis in specific areas into an overall enterprise risk management approach. Has there been any work done on considering whether or not that is even theoretically possible? Or are there complexity related constraints on our ability to ever do that?"
"As time has passed and I have had experiences developing risk assessments and then being on the other side, I have two comments for you to ponder: (1) due to the high degree of familiarity risk one major consideration for risk management and assessment must come from an external resource. People are too much like a fish in water ... can't see what they can't/don't want to see. (2) The more tools that can be designed to measure performance metrics and provide a baseline to remove subjectivity the better. I think that the activity based costing model from cost accounting could be revamped to the risk world."
"I think that risk management needs to decide what it is. By embracing the management of opportunities - all uncertainties that matter - as a result it is really turning itself into a 'theory of business' / 'theory of the firm'. Namely, this is how a good an enterprise should be managed. However, I think that its 'methodology' is too simplistic - the five risk steps, plus setting the risk appetite."
"I am generally in favour of adopting the practice of implementing risk management as a part of all governance and day-to-day management activities. That said, I too frequently see it just not happen. Some cases in point: a network penetration test conducted a year ago revealed an area of potential weakness that the responsible technology manager later deemed irrelevant. A month ago, a client stumbled upon the issue, 'triggering' the risk. Now the manager is scrambling to fix things but can't come out of this looking good. Another example: we had certain data on our network that belonged to our clients' account-holders. The tax regulators have very strict penalties if that very data is exposed due to theft/blackmail/etc. The problem persisted for years until a specific risk management process defined the risks (e.g. executives doing prison time). And this is just a small software firm. At a previous employer in a different regulatory environment I witnessed my employer run afoul of AML issues time after time - there simply wasn't any ownership. As maddening as it is, that mindfulness of risk mitigation just isn't there until someone's done a round-up of risks independent of 'business as usual'."
Respondents
Invitations to participate in the survey were sent to RISKANAL (an internet discussion list about risk analysis), the PMA Forum (an internet discussion list about performance management), the Auditnet discussion list (for auditors), and a varied selection of my contacts. Most respondents were from my contacts. 81% of respondents stated their first language as some kind of English, with the rest having a variety of other first languages. Total response rate cannot be calculated but will have been low.
As usual, this is a limitation of the survey and the selection of respondents will have affected the pattern of answers given to some extent.
However, the patterns of responses were broadly similar regardless of what background respondents claimed, and regardless of whether they gave their name or not. (Giving a name was typical of respondents known to me personally.)