Working In Uncertainty

The fundamental flaws of “ISO 31000:2009 Risk management – Principles and guidelines”

Contents

In 2009 the International Organization for Standardization (ISO) published its first generic standard for risk management. It was named ISO 31000:2009 Risk management – Principles and guidelines. Shortly after that I wrote a critique of this document (Leitch 2010) which was published in the journal of the Society for Risk Analysis, pointing out several serious flaws in the Standard. In the years since then the committee responsible for this standard has not been able to agree on improvements (despite a lot of effort) and no new edition has emerged. At the same time, my understanding of the relative importance of the various defects of the Standard has improved.

This new article provides a revised and improved critique of this important but fundamentally flawed international standard. These are not just my opinions. They are the result of a great deal of analysis and research. If you have access to the Standard then you can, if you want to, follow the references to it and verify the points I make.

The key flaws

Very narrow range of methods permitted

The Standard permits only a very narrow range of methods to be used in managing risk. It advocates one approach and does not allow others to be used instead. Most importantly it asserts that:

all risk should be managed by making lists of risks and managing them.

In other words, this is a Risk Listing standard (Leitch 2012). This is not stated in one explicit sentence and the italicized text above is not a quote. However, this assertion is clear from a combination of explicit statements and implications.

  • Implementing risk management is identified with the "risk management process" (Figure 1 page vii).

  • Managing risk is to be done by implementing the risk management process at varying levels (clause 4.1, paragraph 1, sentence 2).

  • The aim of the risk identification step is said to be to generate a comprehensive "list of risks" based on "events" (clause 5.4.2, paragraph 1, sentence 2).

  • This is reinforced by the headings throughout the risk management process clause (clause 5).

  • The purpose of risk evaluation is said to be to help decide which risks needed treatment (clause 5.4.4, paragraph 1) and the procedural advice in sub-clause 5.4 is consistent only with this approach.

  • This is further reinforced by the statement of the purpose of risk treatment (clause 5.5.1) and the procedural advice in sub-clause 5.5 is also only consistent with this approach.

  • Accountability for risks further implies a focus on managing risks (A.3.2, paragraph 1 and 2).

Over the years some members of the ISO committee responsible for this Standard have argued that the text does not imply this at all and that the approach described is much more encompassing than the summary above. This argument requires an imaginative interpretation of the text (particularly sub-clauses 5.4 and 5.5) and, once you start ignoring the literal meaning of the words of the Standard, it is very difficult to discuss flaws and improvements of any kind. The literal words should be the only ones relevant when it comes to revising the Standard.

Having said that, a fair test of the text would be to find out what a sample of potential users of the Standard think it says once they have spent some time reading it carefully. They could be presented with a set of descriptions of different methods of managing risk and asked to identify the elements of the ISO 31000:2009 process within them, if they can. They could be asked to say which of the descriptions is compatible with the ISO 31000:2009 approach, if any. They could be asked which description best agrees with the advice given in ISO 31000:2009.

I have already written several such descriptions and, although I do not have the resources to actually test the reactions of a reasonably large sample of potential users, I am confident that Risk Listing is the only approach they would be able to relate to the advice in ISO 31000:2009 and that other well established and widely used methods would be virtually impossible to relate accurately to ISO 31000:2009.

The consequences of prescribing this approach of listing risks and then managing them are profound:

  • This method is suitable only for decisions about actions that are responses to a risk. It does not handle decisions with several associated 'risks' (or, more generally, relevant consequences that are uncertain).

  • It excludes some long-established, widely-used, and highly-respected methods that do not involve itemising discrete 'risks' but instead involve creating probability distributions for outcomes and calculating from them a risk measure (e.g. Value at Risk, variance), or making a choice based on probability weighted average outcomes (e.g. expected value, expected utility), reflecting risk concerns using risk-adjusted discount rates or hurdle rates, or simply presenting decision-makers with the distributions and allowing them to explore their valuations.

  • It also excludes one of the most common methods in practice, which is to create a decision support model and plug different assumptions in to see what might happen (i.e. "what if" calculations).

  • And it excludes methods that do not involve explicit considerations of possible futures at all, but instead involve applying policies that promote organizational adaptability and resilience and thereby help to manage a wide range of unanticipated futures.

  • Excluding methods also means excluding the people who use those methods, such as actuaries, quants working for financial institutions, decision analysts, NASA engineers, and so on. In some cases it may even mean excluding methods that are required by law or by international regulations. I have not studied the extensive regulations on risk analysis for nuclear engineering or for analyzing pollution in various countries, but I strongly suspect that they are not compatible with the prescriptions of ISO 31000:2009. Such an analysis should be done before an international standard such as ISO 31000:2009 is released but as far as I know this has not happened.

If ISO 31000:2009 stopped at prescribing listing risks then managing the risks on the list then it would already be far too prescriptive as to methods. However it does not stop there. ISO 31000:2009 goes on to assert that:

all risk management must be done by repetitions of the following process steps, performed in this order: (1) Establishing the context, (2) risk identification, (3) risk analysis, (4) risk evaluation, and (5) risk treatment.

This means no jumping around between steps. If you have just done risk evaluation, for example, and realized that you made a mistake in risk analysis then you need to go through risk treatment, and then re-do establishing the context and risk identification before you can fix your mistake.

This is shown clearly in Figure 3. Also, there is no language elsewhere that explicitly opens the door to variations of any kind. If there were then it should be in clause 5.1, General.

The advice in ISO 31000:2009 about exactly how to analyze risks is even more detailed and, as usual, no variations are permitted. It asserts that:

all risk management must involve representing the relevant information in terms of at least the following objects: (1) sources of risk, (2) areas of impacts, (3) events, (4) the causes of events, (5) the potential consequences of events, and factors that affect (6) consequences and (7) their likelihood.

So, if you design a template for risk analysis then your template will need headings and boxes for all those objects. If you use a diagram for risk analysis then the diagram will have to show at least one of each of those objects for each risk on your list. If you build a mathematical or simulation model then it will have to analyze each risk in such a way that each of those objects is explicitly represented and identifiable. However you do it, every 'risk' on the list will need to be elaborated within an analysis that features at least one of all those object types.

The relevant references are as follows:

  • Risk identification (clause 5.4.2, paragraph 1, sentence 1).

  • Terms and definitions (2.15, Note 1).

  • Risk analysis (clause 5.4.3, paragraph 2, sentence 2).

Furthermore, it asserts that:

for each risk the following must be determined: (1) a level of risk, along with (2) a confidence in the level of risk, and its sensitivity to (3) preconditions and (4) assumptions.

So, in addition to a way to express "level of risk" you also need a way to express confidence in each level of risk, and sensitivity to preconditions, and sensitivity to assumptions. You will need to determine each of these for every risk.

The relevant references are as follows:

  • Risk analysis (clause 5.4.3, paragraph 3, sentence 1).

  • Risk analysis (clause 5.4.3, paragraph 4, sentence 1).

  • Risk evaluation (clause 5.4.4, paragraph 2, sentence 1).

How many risks might you need to do this for? This is not stated but the Standard asserts that:

in this risk assessment, all risks must be listed and all significant causes and all significant potential consequences must be considered.

What exactly this implies in practice is not clear. What about causes very far back in the past? What about consequences very far in the future? They too could be significant.

The relevant references are as follows:

  • Risk identification (clause 5.4.2, paragraph 1, sentence 2).

  • Risk identification (clause 5.4.2, paragraph 1, sentence 4 - the last one).

  • Risk identification (clause 5.4.2, paragraph 2, sentence 5 - the last one).

If this level of prescription was used in a document whose aim was to describe a particular method to be applied in a particular situation then it might be appropriate. However, it is completely inappropriate in a high level, international standard intended to apply to all risk and all organizations.

All this detail, and especially the idea that all risk must be managed by making a list of risks and then managing them, is too narrow because it excludes many well established, highly effective methods. It is as inappropriate as, for example, writing the whole standard as if all risk is to be managed by calculating value-at-risk numbers, or all risk is to be managed by constructing multi-attribute utility models, or all risk is to be managed by calculating mean and variance numbers from probability distributions.

This level of prescriptive detail is also inconsistent with the statement in the Scope of ISO 31000:2009, which describes the content of the standard as "principles and generic guidelines on risk management". You might imagine doing all the things listed in the above summaries occasionally, but would you do them always, for all risk? Of course not.

Narrow scope

The ISO 31000:2009 approach does not include setting objectives but it does require that objectives be set. This implies an overly narrow scope in three ways. First, it excludes situations where it is impractical to represent explicitly the interests of stakeholders, which is quite common in political decision-making. Second, it excludes organizations that do not set objectives even though they could, and express the interests of stakeholders in other ways. Third, it means that risk management provides no help in setting objectives.

The integration method is not universally feasible

ISO 31000:2009 takes a strong position on the integration of risk management and is again prescriptive as to methods. It asserts that:

risk management must be integrated into all the organization's processes and practices and this must be done by making the complete risk management process part of each of those organizational processes.

The relevant references are as follows:

  • Integration (clause 4.3.4, paragraph 1, sentences 1 and 2).

  • Application to all decision-making (A.3.3, paragraphs 1 and 2).

  • Principles (clause 3 principle b, paragraph 1, sentence 2).

Remember that this risk management process is the Risk Listing one with many quite specific and onerous elements that are to be used on every occasion. Just to be clear, ISO 31000:2009 asserts that:

making the risk management process part of processes means that all the steps of the risk management process must be identifiable within each process in the organization, using the objects (risks, causes, consequences, sources of risk, etc) of the risk management process, including making lists of risks and managing them.

This is implied by the forgoing and also reinforced by the material on application to all decision-making at A.3.3, paragraphs 1 and 2.

It is not necessary for me to spell out how impractical it would be to try to implement the full details of the ISO 31000:2009 risk management process in all processes of an organization.

However, it is also crucial to understand that there are problems with trying to achieve integration into processes using just one process and particularly if it is Risk Listing. In a survey study (Leitch 2011) I found that it is easy to suggest methods for particular steps in particular processes that most people prefer to Risk Listing. Most people also consider these alternatives to be better examples of integration of risk management. However, these alternative methods are not the same in each situation.

Risk Listing is much better suited to standalone risk workshops where the task is to look at what has been decided in the past (or is proposed) and then to try to find risk problems with it and suggest improvements.

Narrow language and illogical definitions of key terms

The Standard contains a long list of terminology with definitions and has nothing to say about using alternative language. Consequently, we can deduce that the Standard asserts that:

the terminology used in risk management must be that used in ISO 31000:2009.

Those definitions include redefinitions of familiar words in new and unfamiliar ways. It asserts that:

"risk" means the same as "effect on uncertainty", and so (1) risk is an effect and (2) what is affected by risk is the choice of objectives, not outcomes or predictions. (Ref. Terms and Definitions, 2.1)

Almost certainly this is not exactly what the authors meant, but the words are clear. (The problems of this wording are analyzed in more detail in Leitch, 2015.) A similar mistake is made in the definition of consequences:

"Consequence" means "outcome of an event affecting objectives". Hence, it is the effect on the choice of objectives that is important, not outcomes or predictions. (Ref. Terms and Definitions, 2.18)

Use of language is one of the worst areas of ISO 31000:2009, with many more flaws I could list.

Why these flaws exist

TC 262, the committee that produced ISO 31000:2009, is not the most harmonious or productive committee at ISO and it is trying to write about something that is difficult to write about. The text of its standard, and the procedures the text describes, have not kept up with the expanding ambition of its scope.

Over the years, what was originally just an idea for a simple method to be used instead of rigorous Operational Research methods became a method to be used on some public sector projects in New South Wales (Australia), and then a standard for all risk management across Australia and New Zealand, and then for the whole world and all subsidiary risk management standards produced by ISO.

Because the text has not kept up with that growing status and scope, the level of prescriptive detail that once seemed a helpful guide to a particular method people might like to try has become a narrow approach to a big subject. What was once a prescription for separate risk workshops to add to a project is now put forward as a process that must be applied as an integral part of every activity in every organization.

Eliminating the flaws

Fixing the problems described above is relatively simple from a drafting point of view because most of what needs to be done is deleting existing text. The text is too prescriptive and deleting some of it would help solve that problem.

Specifically, the risk management process should be eliminated entirely and references to it should be replaced by references to methods that the user needs to choose. This would be a variety of different methods, not just one method, and specific methods do not need to be given in the Standard.

The Risk Listing language should be eliminated throughout the Standard by removing any text that suggests or implies that risk management is synonymous with managing risks.

Text that reinforces the narrow position on objectives should be removed.

This would open up the Standard to a wider range of methods, expand the scope, and remove requirements that are absurdly onerous. The Standard would get much closer to the high level guidance that is appropriate for the document and its content would more closely match the intended "principles and generic guidelines".

With a lot of the prescriptive details removed it should also be possible to remove several of the definitions, including some flawed ones. The definition of risk should be removed, leaving users to rely on ordinary dictionary definitions. The scope of the standard can be stated as including all uncertain outcomes, not just disappointing ones, which would achieve the main purpose of the faulty definition of risk and be easier to understand for most people.

Commenting on ISO 31000:2009

This standard has already been influential and the longer it stays in publication in its current form the more damage it will do. Directly or, more likely, indirectly it will affect your life eventually, especially if you already manage risk in a better way than is described in the Standard. If you have the opportunity to comment on ISO 31000:2009 then please do so. The more people who speak up about it the better. Many of the members of the committee responsible for this standard still appear to be unaware of its defects.

If your work involves scientific risk analysis or perhaps decision analysis and the text of ISO 31000:2009 seems irrelevant to you, please do not be put off commenting. It is precisely your work that is under threat from this international standard. You may not think ISO 31000:2009 is good enough to even comment on but politicians and regulators are not as discriminating and they like sources that seem authoritative.

Please do not be put off if someone says that your comments will not be acted on. Making comments yet failing to point out major flaws endorses the existing text and will perpetuate the existing situation. Your comments will provide much-needed support to the minority on the ISO committee and national committees involved who are already prepared to speak up. Your comments will also give confidence to committee members who currently lack the confidence to speak up.

References

ISO 31000:2009. Risk management: Principles and guidelines.

Leitch, M. (2010). ISO 31000:2009—The New International Standard on Risk Management. Risk Analysis, 30: 887–892.

Leitch, M. (2011). Results of a survey on ‘integrated risk management’.

Leitch, M. (2012). The Risk Listing school

Leitch, M. (2015). The fundamental flaws in ISO's definition of ‘risk’.






Made in England

 

Words © 2016 Matthew Leitch.