Matthew Leitch, educator, consultant, researcher
SERVICES
OTHER MATERIAL
Working In Uncertainty
Alternative risk lists
by Matthew Leitch; first published 18 September 2007
Contents |
Thank you
First, thank you to everyone who responded to this survey, particularly if risk isn't particularly your speciality or you provided useful information in your comments, as many did.
Summary
The survey probed basic beliefs connected with risk and revealed a surprise. Although a lot has been written about risk analysis as if the risks already exist and need only be discovered, over 80% of respondents believe that there are alternative ways to list risks that are not just more or less complete versions of a true list, or the same risks with different names or in a different order. Most people believe this for a variety of reasons, including different models/perspectives, different ways to break down outcomes into risks, and different knowledge. Also, most people believe that some lists are more useful than others even when equally valid and complete.
The same high degree of belief in alternative risk lists was apparent in respondents from all three sources and did not differ greatly between risk specialists and others, or between people with some mathematical skill and people with little or no mathematical skill.
This article describes the study and its results, then describes alternative views so that you can understand better why people think as they do. It also points out some practical implications of these findings for the way we talk about risk analysis and the topics that need to be covered in any guidance on how to analyse risk.
Design of the survey
Respondents were invited to participate using postings to three professional discussion lists. The invitation contained a link to a very short online survey that posed the following questions:
What country are you living in now? | |
Do you consider yourself a risk or risk management specialist or expert? | Yes No |
Do you think you could solve the following equation in less than 20 seconds? | Yes No |
Suppose a risk analysis is needed to create a list of risks for a given application (e.g. the risks of a project). Which of these statements do you agree with most? | There is only one valid list of risks. (Though people might give different names to risks, get more of the total list of risks, or put them in different orders.) There are alternative valid lists of risks. (This goes beyond just alternative names, orders, or degrees of completeness.) |
IF you think there are alternative valid risk lists, why might that be? Check all that could apply, e.g. when listing risks for a project. | Different perspectives / different models of the project and its environment. Different ways to split causes or outcomes into risks. Different knowledge about the project and its environment. Other. |
IF you think there are alternative valid risk lists, which do you agree with most? | Any equally valid and complete list is equally useful. Some lists may be more useful than others even when equally valid and complete. |
| Are there any comments or explanations you would like to make? (Include your email address if you would like a reply.) | |
Sources of respondents
To ensure a range of expertise in risk assessment among respondents they were recruited from three discussion lists to one of three identical versions of the survey. This meant that responses could be roughly separated between the more and less risk oriented respondents based on source, as well as by looking at survey responses. Here are the discussion lists involved:
RISKANAL: A discussion list about risk analysis that has a large membership, mainly in the USA, and includes many academics and practitioners.
AuditNet: A list mainly for internal auditors.
PMA Forum: A list about performance measurement, with many members from the UK.
Survey respondents
The volunteers from each source can be summarised as follows:
| List | Respondents | Risk specialists | Equation confident |
|---|---|---|---|
| RISKANAL | 78 | 79% | 78% |
| AuditNet | 31 | 35% | 65% |
| PMA Forum | 34 | 18% | 71% |
| All | 143 | 55% | 73% |
Judging by their confidence in solving the equation in 20 seconds it looks as if RISKANAL members are either slightly more mathematically oriented than members of the other lists, or are younger. The equation used in the question is simple if you have just done an examination in mathematics at school but baffling if you did that years ago and haven't had to think about algebra since.
Responses about beliefs
Asking questions about beliefs concerning abstract ideas like ‘risk’ is extremely difficult. The comments from some respondents show something of the range of unexpected interpretations of words in the survey. However, the main results are clear. Most people think that alternative lists of risks are possible and vary in usefulness. The reasons for alternative lists go beyond different orders, descriptions, or degrees of completeness, and beyond different objectives or points in time.
| List | Believe alternative risk lists possible | Because of different perspective / models* | Because of different splits* | Because of different knowledge* | Because of other reason* | Alternative lists can have different usefulness* |
|---|---|---|---|---|---|---|
| RISKANAL | 88% | 87% | 61% | 74% | 28% | 80% |
| AuditNet | 84% | 88% | 50% | 77% | 19% | 96% |
| PMA Forum | 85% | 83% | 69% | 66% | 21% | 90% |
| All | 87% | 86% | 60% | 73% | 24% | 86% |
* This is the percentage of the respondents believing alternative lists to be possible. These follow up questions were not applicable for respondents believing only one valid and complete list can exist.
It seems to make no consistent difference whether respondents consider themselves to be risk or risk management specialists. It also makes no consistent difference whether respondents can do simple algebra. In the table below ‘Risk’ means the respondent is a risk specialist and ‘Math’ means the respondent is able to do simple algebra.
| List | % for Alternative lists: not Risk, not Math | % for Alternative lists: not Risk, Math | % for Alternative lists: Risk, not Math | % for Alternative lists: Risk, Math |
|---|---|---|---|---|
| RISKANAL | 100% | 78% | 80% | 90% |
| AuditNet | 86% | 77% | 75% | 100% |
| PMA Forum | 100% | 72% | No Data | 100% |
| All | 96% | 75% | 79% | 92% |
Understanding different beliefs
Obviously this survey does not give a detailed understanding of each respondent's rationale. However, while designing the survey I was aware of three main reasons why alternative, valid risk lists might be considered possible, and more have come to light from comments made by respondents.
Perspectives considered in designing the survey
Here are the three main reasons I started with.
[In addition, different lists might result from different objectives/interests and from creating the risk lists at different times, though arguably these are part of the circumstances and not reasons in principle for alternative lists within the same circumstances.]
Risks as derived from models
Some people derive risk lists from models of the system, activity, or whatever they are making a risk list for. For example, an accountant might build a financial model of a project and then use a Monte Carlo simulation tool such as @RISK to represent uncertain inputs as probability distributions, and compute the implications for output variables and intermediate variables.
Each of these variables can be seen as a ‘risk’ and a risk list could include all these, plus risk items for model uncertainty and a variety of other things.
The same can happen with non-quantified models and variation in risk lists results from the choice of model and the choice of method of deriving risks from the model.
With this perspective the risk list depends on what model you start with, and alternative models are common. Some risk lists are likely to be more useful than others in a given situation.
In the survey the reason ‘Different perspectives / different models of the project and its environment’ was referring to this view, though respondents will not necessarily have read it that way.
Risks as potential events
In the usual textbook introduction to probability theory ‘events’ are defined as sets of outcomes. For example, the outcomes from throwing a six sided die can be represented by the numbers 1, 2, 3, 4, 5, and 6 but an ‘event’ is a sub-set of these, such as ‘less than 4’, ‘an odd number’ or simply ‘six’ (because sets can have just one member).
Clearly there are alternative ways to split the outcomes into events.
The same thinking can apply to ‘risks’ on a risk register if you think they are the same as the mathematician's events. Risks can usually (perhaps always) be seen as sets of potential outcomes. For example, ‘Losses from vehicles this year’ might appear on a company's risk register, but if they were more interested in these losses they might have captured the same outcomes within a larger number of ‘risks’ perhaps for losses of different types of object, or even losses from individual vehicles, or for shorter periods of time.
Some risks are effectively infinite sets of outcomes. For example, the risk item ‘loss of market share’ could refer to any extent of lost market share up to total loss, which is an interval of a continuous variable that mathematicians would usually regard as having infinitely many members.
Listing every outcome is not a practical possibility and again there are alternative ways to split down the total set of potential outcomes. Some splits are likely to be more useful than others in a given situation.
In the survey the reason ‘Different ways to split causes or outcomes into risks’ was an attempt to refer to this view.
Knowledge
In many views of probability knowledge is crucial. New information leads to revisions of probabilities.
Applied to risk listing this gives another reason for recognising alternative lists. For example, suppose some situation with uncertain outcomes has already taken place. Some risk analysts already know the outcome but some do not. The analysts who know the outcome cannot have it as a risk any more because there is no uncertainty. However, the analysts still in ignorance can have it as a risk because, for them, the outcome is still uncertain.
In the survey the reason ‘Different knowledge about the project and its environment’ was referring to this view.
Further beliefs inferred from respondents' comments
Comments by respondents pointed towards yet more reasons for believing that alternative, valid risk lists are possible.
Personal perceptions
The idea seems to be that people have different perceptions of situations and these can be equally valid.
Levels of detail
Risks need to be addressed with different risk lists at different levels in an organisation because otherwise low level lists are too detailed for high level people and high level lists are too broad to be useful to low level people.
Practical limitations
Lists of risks are never complete because of the sheer number of possible risks and the difficulty of understanding the future.
Some respondents gave this explanation but also commented that they thought there was, in theory an ultimate and complete list reflecting all perspectives but in practice this was unobtainable and too complex to use. Some respondents with this view said alternative lists were possible and some said they were not.
Decision analysis perspective
The validity and usefulness of a risk lists depends on the decision questions being addressed.
Depends on the audience
The risk list provided depends on who the audience is. A list for the public might be different from one for private use.
Risk analysis is an art
Risk analysis is an art that depends as much on the analyst as on the facts.
Practical implications
The high proportion of respondents who believe that alternative risk lists are possible was a surprise – to me at least – because the traditional language of risk management and content of the best known guidance on risk management suggests that alternative risk lists are not possible.
If you think alternative risk lists are possible then you may be interested in the following practical implications.
Guidance
Although it is increasingly common for guidance to acknowledge alternative processes for arriving at a risk list it is rare to see explicit discussion of alternative lists.
Usually there is considerable scope to give more advice on:
Why alternative lists are possible.
What characteristics make some lists more useful than others in a given situation.
When choices arise, even within a particular process.
How to make those choices e.g. what to consider.
How to change lists to meet new requirements and circumstances.
Language
One way that the traditional language of risk management tends to suggest a unique risk list stands out in particular. It is the use of the phrase ‘Risk Identification.’
If you are any kind of risk expert you probably don't see the problem with the phrase ‘Risk Identification’ because we seem to have become so used to it that the conflict is invisible. However, consider the view from outside the risk world. Some uses of the word ‘identification’ make sense but some do not. The following examples illustrate important points about this word.
| Situation | Appropriate use |
|---|---|
| The police discover the name of a suspicious man seen near a crime. | ‘The police identified the suspect.’ |
| A commuter solves a Sudoku puzzle. | ‘The commuter identified the numbers that solved the puzzle.’ |
| A bird watcher sees a bird fly between two trees. | ‘The bird watcher identified the bird as a jay.’ |
| A manager is faced with a difficult choice. | ‘The manager identified the best option.’ |
| Situation | Inappropriate use |
| An artist paints a picture. | ‘The artist identified the picture.’ |
| An architect designs a new building. | ‘The architect identified the new building.’ |
| A team of software developers creates a new graphics program. | ‘The team identified the new software.’ |
‘Identification’ is appropriate when a name is being put to something that exists, or when a limited range of possibilities is studied to pick the one that meets some criterion e.g. ‘the best’. The thing to be identified exists already.
‘Identified’ is not appropriate when something is being created, invented, or developed. In other words, when the thing involved does not exist already.
The more it seems that developing a risk list involves choices and creates things (‘risks’) that did not exist before the analysis began, the more appropriate it is to use words like ‘develop’, ‘analyse’, ‘define’, and ‘create’ instead of ‘identify.’ More appropriate terms to replace ‘Risk Identification’ include ‘Risk Set Definition’, ‘Risk Analysis’, ‘Risk Analysis Development’, ‘Risk List Development’, ‘Risk List Creation’, ‘Risk Hierarchy Development’, and so on.
Other language habits that subtly conflict with the idea of alternative lists include these:
Using words that are similar to ‘identify’ e.g. ‘We need to find risks related to this objective.’
Talking about ‘the risks’ without reference to a particular analysis, as if ‘the risks’ existed before any thinking was done.
Talking about sets of risks as if they exist separately from any analysis e.g. ‘Strategic risks are risks that ...’ said without reference to analysis so that is seems that strategic risks exist beforehand.
Giving the name of a risk with no definition in a situation where the definition hasn't already been established, and expecting people to know what it is. For example, ‘Hey, how about technology acceleration risk?’
Comments by respondents
These are the comments by respondents related to beliefs about risk lists:
| Comments from RISKANAL respondents |
|---|
| Regarding the last question, when there are alternatively valid risk lists, how useful they are mostly depends on what 'decision questions' are being asked. |
| Though I believe there is only one valid list of risks, many times that list is unknowable. |
| The usefulness of the alternative risk lists is a very big topic of discussion and I think it is up to the specific issue under stake..... Although not a risk manager myself, but still if you consider that in any kind of project each ones does a little bit of risk management, then we need all kinds of risk lists we can come up in order to secure ourselves against all possible risky scenarios. |
| I answered that there are multiple valid lists of risks because the list of risks will depend on the context of what decision is being informed and from which perspective of personal/group incentives and values is the list being constructed. A list constructed by one group may be only useful for that group and for a specific decision. Another set might be applicable to a broader set of the population or to inform a more diverse set of decisions. Florig et al. wrote a paper in Risk Analysis on approaches to categorizing risks in the early 2000s. The conclusion was that there is no ‘correct’ categorization, only better or worse categorizations based on the decisions that need to be made. This is my line of thought. |
| Often the list provided depends on who the audience is and the overall purpose of this list. A list of valid risks presented to the public may be completely different than an internal risk register used for decision making. |
| I have a problem with the question. I believe that there is only one valid list of risks, but it must include risks from all perspectives, all causes and consider all outcomes, including risks based on different knowledge about the project and its environment. Thus all risk lists are subsets of that larger list that includes all. |
| Risks are often a subjective medium whereby the greater knowledge held by the individual can produce either a casual approach or alternatively a pedantic approach. The term ‘a reasonable or competent person’ goes some way to ensure that a common sense approach is implemented but as long as there are human inputs there will be grey zones. Some will complain that OH and S risk management is excessive and others just as vehemently insufficient. You cannot mitigate all risk as humans adapt so that injury may occur in situations it never would have previously, and of course you need to take into consideration cultural and generative (gen x, y, baby boomer) issues. |
| If risk is frequency and severity of adverse impacts on something that matters to me, then, obviously, what matters to me is crucial in characterising risk. What matters to me now is not necessarily the same as what mattered to me yesterday, or what matters to others. |
| I don't think I have ever seen a complete list of risks! For example, truly large fires. The Peshtigo fire burned four million acres and killed 1,100 people, many by oxygen starvation. Ever heard of it? It happened the same day as the Chicago fire. I have never persuaded any client to put such a fire on the list. |
| The tendency is to split risks into different categories (perceptions V physical phenomena etc). While convenient, all of the possible risks bear on the actvity and need to be considered – even if (for example) then no action is taken (i.e. choice not to treat the risk). |
| I believe that even though there are several equally valid lists of risks they as a whole complete the true total list of project risks. But after risk identification is performed it might not be motivated (for different reasons) to act on all of the identified risks irrespective of their priority. My 2 non-professional cents. |
| Different stakeholders with different objectives are likely to have different lists of risks, as well as different ratings for similarly-worded risks. That is why the context-setting aspect of the proposed RM standard is so important. |
| The harm/consequence of risk has a social context and this alone gives it multiple possibilities. The subjective estimation of likelihood also provides alternative possibilities. |
| I think that it is possible to create a unique and complete list of all known risks, but is it useful? Most of the time you face only one aspect of the problem because the reality is too complex to be handled all in one time. |
| How would you compare lists by validity? Is there a way to measure validity except for subjective personal opinion? |
| I believe that risk assessment is an art such that the determination of ‘risk’ relies as much on the practicioner as it does on relevant information used in the process. |
| Your use of word ‘risk’ creates a problem for me. In our work we use risk as defined in a lead article, first issue of Journal of Risk Analysis (by Kaplan and Garrick): complete set {(si,Li,Ci)} where for each risk triplet(si,Li,Ci), si is the ith scenario, Li is likelihood of ith scenario, Ci is consequence(s) of ith scenario. The complete set encompasses everything except the ‘as planned’ scenario in which everything goes as planned. In this context, a ‘risk’ is not some adverse event, but a set of descriptors for some particular unwanted scenario. And just as Tolsoty observed that all happy families are alike and each unhappy family is unhappy in its own (unique), most projects or activities have a very small set of desired (we should probably include the category ‘acceptable&rquo;) outcomes and an essentially infinite set of possible undesireable outcomes. Using the word risk always starts the Babel of everyone talking in their own language. The word ‘valid’ is also a problem. If you identify a possible outome that isn't part of the as-planned scenario, and that outcome is valid (i.e., based on logic or truth) but not one we are concerned about, we have a valid but irrelevant (un-useful) item for the list. All these words...complete for example: complete is obviously context specific. ask a cardiologist what can go wrong in heart surgery and you will get different list of outcomes than that enumerated by HMO business manager. Each may be valid and complete from a particular observer's perspective. So we end up in an Orwellian world where some lists are more equal than others. |
| Some lists may be more oriented to risk prevention or reduction. For example, understanding the CAUSES of the risks is necessary for risk management, while other ways of categorizing risks may not yield the same insights. |
| Lists of risks are all limited. There are a multitude (maybe an infinite number) of risks, the question is risk of what? There are health risks, economic risks, political risks, social risks, etc. The most valid list of risks is the one that addresses your interest in the project, task, endeavor, whatever you are analysing. If your interest is vested in the failure of a project, the risk is that it will succeed. |
| I am not sure what it means for a list of risks to be ‘valid’. To be complete any list of risks would be infinite, so there has to be some sort of criterion to decide what is relevant/significant enough to get on the list. How small is too small? What about high-magnitude, low probability risks? How big is too big? |
| By ‘other’ in the 5th question I refer to the possible necessity to create sub-lists within a risk list (e.g. a construction project will surely imply some safety risks for workers, and safety risk is too generic a term which requires further detail.) |
| Comments from AuditNet respondents |
| ‘Different knowledge about the project and its environment’ would not give different VALID (by which I would mean complete) list of risks. ‘Different ways to split causes or outcomes into risks.’ is the same as alternative names or orders. ‘Different perspectives / different models of the project and its environment.’ is the most likely alternative but then we'd really need to get these different perspectives talking. After all, who has to DO something based on these risks? |
| I work for a large financial institution corporation, highly regulated by banking, securities and insurance laws. Too often, someone far, far away makes risk decisions at a big picture level to satisfy the regulator and shareholder gods then homogenize the risk and control processes across the businesses. What makes sense at 50,000 feet does not always make sense at ground zero. That's why risks should be identified at various levels, especially in large companies so the big picture risks are addressed, but so are the local business unit/legal entity ones that actually keep the company in business. |
| Unless you have a very static (business, technology, systems) environment you can never be sure you understand, recognise and contain the risks from ever evolving threats. Simply consider the following IT/data/privacy threats (from a recent in-house assessment) and then revisit the premise of your survey questionnaire. (I'm sure you'll acknowledge that there are a bunch of ‘obvious/traditional’ risks, and then there are a whole bunch more you can only guess at unless you can do a full due-diligence of the outsourcer organisation.) – Outsourced Operations, which might include: A. An outside organization supplying staff to operate/manage some or all of your in-house IT facilities; B. An outside organization operating/managing all your IT facilities at their premises (or some combination); C. An outside Security company managing offsite backups (collecting, cataloging, storing, cycling, replacing, destroying when worn/outdated) D. An ASP hosting your company applications and data on their systems through a dedicated network; E. An ASP hosting an online package (such as Salesforce.com) which processes and stores your strategic company information alongside that of 200 other companies. This suggests there are a multitude of additional threats to the confidentiality, security and integrity of a company's data (transactions, reports, etc) over which its executives have given up control, and are in fact beholden to the ‘Security Disciplines and Practices’ of the outside service provider. |
| Risk lists must be dynamic to have any hope of usefulness given that few organisations are static or operate in a static environment anymore. Even the most comprehensive risk lists are limited by perceptions of possible future events. |
| Comments by PMA Forum respondents |
| Regarding the 4th question, my interpretation is that all risk is a factor of environment (in the broadest sense of internal and external factors and parties). The biggest impact on the creation of a reliable list of risks (and I'm thinking especially in project management terms here, which is my speciality) is the individual perceptions of risk from the participants compiling the list. |
| Techniques of reliability analysis are useful tools for risk analysis too! |
| The complexity with risk lists is that there is no equal and opposite effect. One risk may lead to perverse (and unexpected) outcomes in another part of the system. |
| No list of risks is ever complete. |
| Different risks/risk levels may actually exist for different groups of stakeholders viewing the same project/item. |
| An issue that is not explicitly raised here but is important in the public and voluntary sector is risks to whom – the organisation, service users, the general public, the government. This applies of course in the private sector, but if you consider the prison or probation service for example this becomes more complicated. |
Related reading
‘Making sense of risk appetite, tolerance, and acceptance’ discusses another area at the heart of risk management thinking where alternative perspectives proliferate.
‘Favourite ways to characterise risks: Results of an online survey’ reports recent research on the range of techniques that people prefer.