Working In Uncertainty

Results of a performance management survey

by Matthew Leitch, first published 19 February 2004.

Thank you

First, thank you to everyone who responded to this survey. Whether you like ‘Beyond Budgeting’ or think it's dangerous nonsense I hope you will find something in these results of interest. There are lessons in advocacy here, and hints about what people are really looking for from their management systems, whether Beyond Budgeting or not. You've certainly given me a lot to think about.

Summary

The survey was conducted online and respondents were self-selected. The first significant question asked if the respondent was already convinced that the Beyond Budgeting model or something like it was a good idea. Then the respondent read 30 arguments in favour of such management models and rated each for how persuasive it was. A negative rating indicated that the respondent thought the argument backfired and argued against Beyond Budgeting and similar models.

Overall, there was a lot of disagreement between respondents. There are many views on these issues.

Arguments that criticised more conventional management models tended to provoke more disagreement, especially among respondents who favoured conventional models. For this and other reasons some arguments were generally accepted by respondents while others split respondents.

The findings suggest that anyone wishing to promote a new idea in management control should explain what to do instead and justify the new ideas on the basis of generally accepted premises, if possible.

Arguments driving for faster, more proactive management were the most popular and persuasive. Arguments in favour of decentralisation, a key part of the Beyond Budgeting model, were not well supported. In addition, arguments often used by the Beyond Budgeting Round Table were barely more persuasive than others used in the survey.

One argument not tested in this survey is that fixed targets should be replaced by targets relative to competitors. This argument tends to provoke a lot of angst and confusion in debates. Two arguments in the survey shed light on this. One argument was that ‘Working towards a goal of relative superiority over the competition encourages people to consider a wider range of future scenarios and make more flexible and effective plans.’ Another was that ‘Re-planning just once a year is far too slow. Quarterly is better. Monthly is better still.’ Both were popular suggesting that respondents are happy to revise their thinking more often than once a year and to use relative goals, but not happy to abandon absolute targets.

It is sensible to have specific targets and many planning procedures demand it, but it is not sensible to continue driving towards targets even when experience has shown that they need to be adjusted.

The results of this survey suggested to me that it would be helpful to view management models in terms of how fast and proactive they are, and recognise a wide range of methods for achieving excellence on these performance characteristics. I've written a paper along these lines called ‘A framework for accelerating management systems’.

Favourite ratings

Respondents were given 30 arguments in favour of 'adaptive' management systems and asked to rate how compelling each was. A rating of 0 meant it had no persuasive effect. A rating of +5 meant the argument was compelling in favour. A rating of -5 meant the argument backfired, and was compelling against 'adaptive' methods.

A look at the distribution of ratings across all arguments and all respondents shows something about how people chose their ratings. From the histogram you can see that the most popular ratings, along with my interpretation of them, were:

• 0 – ‘don't understand/not relevant’

• 5 – ‘strong argument and I totally agree’

• +3, -3 – ‘agree/disagree to some extent but not sure how much so I'll just be moderate’

There is a strong skew towards positive ratings which is to be expected as the arguments were chosen specifically because they have often been used in favour of Beyond Budgeting and similar management models.

Extent of disagreement

Of the 30 arguments all but one were rated as compelling in favour of something like Beyond Budgeting by at least one respondent, with a score of +5. However, 13 arguments were rated as compelling against something like Beyond Budgeting by at least one respondent with a score of -5, and all arguments had at least one respondent who thought they backfired to some extent.

Before I explain which arguments provoked the most disagreement there is one aspect of the way the arguments were put that strongly affected ratings. This is a lesson in advocacy, especially if you are promoting a minority viewpoint.

Criticism and suggested alternatives

Some of the arguments used in the survey stated an alleged disadvantage of budgeting or a similar approach based on setting fixed targets annually. Some arguments combined this with a positive suggestion on what should be done instead. There were also two arguments that simply stated a positive reason for doing something more like Beyond budgeting, without criticism of alternatives. These were abbreviated to n – negative, m – mixed, and p – positive respectively.

The results showed that the presence of criticism and suggested alternatives was important.

• Arguments that were purely criticism evoked the greatest variation in ratings (measured by standard deviation), the highest number of negative ratings, and the lowest average ratings. (Additional analysis showed that the stronger the wording of the criticism the greater these effects were. Average ratings declined and standard deviation and negative ratings increased as the wording got stronger.)

• Arguments that were purely a positive statement about what should be done evoked the lowest variation in ratings (measured by standard deviation), the lowest number of negative ratings, and the highest average ratings.

• Arguments that mixed criticism with a positive alternative were rated nearly as well as purely positive arguments.

Even though a mixed argument has more content and so you might think it would be more compelling, the results seem to show that a purely positive argument is slightly more effective. However, the low number of purely positive arguments in the survey makes this finding unreliable. The helpful effect of talking about the preferable alternative is clear, however.

The differences between positive and negative arguments were most marked for respondents who, overall, were not convinced of the advantages of something like Beyond Budgeting. People who were very strongly in favour were quite happy with purely negative arguments. The graph below is an analysis of average ratings between p, m, and n arguments, split between respondents who were ‘authors’ in the field and strongly in favour of Beyond Budgeting or similar, people who were ‘pro’ i.e. supportive of the idea but not authors, and ‘anti’ i.e. people who were not already convinced.

Common ground and areas of disagreement

Overall, the level of disagreement between respondents was high. However, certain arguments were welcomed by nearly all respondents indicating that they are common ground, whereas other arguments divided the respondents.

This graph shows the effect. The green bars show the percentage of respondents agreeing to some extent with the argument. The red bars show the percentage disagreeing with the argument. The white bars below show the percentage rating zero (scale is on the right).

The common ground (defined as statements that 10% or fewer respondents disagreed with to any extent) included:

The arguments that most sharply divided respondents (defined as 25% or more respondents disagreeing) were:

Notice how the way an issue is raised can make all the difference. While few people disagreed that ‘Fixed targets encourage gaming behaviour, especially near the year end.’ many more disagreed with the statement that ‘Fixed targets encourage unethical behaviour ranging from small lies about what is an achievable target to Enron-style fraud.’

Also, both the purely positive statements (arguments 10 and 24) appear in the ‘common ground’ list.

However, there is more to these lists than mere presentation.

Further evidence as to which arguments split the respondents is provided by analysing the average ratings given by people classified into the following groups: Authors (typically very positive on most arguments), Pro (said they were convinced, but not authors), and Anti (said they were not convinced).

The averages for Author and Pro groups stay positive for all arguments, but the averages for the Anti group dips negatively for the arguments that split the respondents.

Conceptual grouping

Within the list of 30 arguments there were certain themes that came up more than once. Here are the conceptual groups and their members, as I saw them, and ratings within these groups tended to correlate quite well.

The Games group is for arguments to the effect that budgetary control leads people to do dysfunctional things. This group is the largest and easiest to identify. It contains some arguments that were generally agreed with (marked above with asterisks) and others that provoked high disagreement.

The Speed group is for arguments about how management need to review things more often. These have quite good ratings and low disagreement.

The Risk Management group is for a collection of issues related to risk, and the ideas are often expressed in terms of being proactive rather than reactive. This is the most popular group overall, though it includes one of the least popular arguments, which is the little-known observation that ‘Budgets get in the way of good risk management. People think they know what is going to happen in the future.’ This group has the advantage that it contains the two purely positive arguments.

The Decentralisation group contains arguments related to decentralisation. It suffers from the disadvantage of including the argument about budgets being too detailed, which does not seem to correlated with other arguments related to decentralisation, and to the argument referring to ‘Soviet style’ government, which some probably found provocative. Nevertheless, even the moderately worded argument number 27, ‘Decentralisation is needed if you want to be responsive and budgetary control tends to block empowerment.", scores poorly with a low average rating and high disagreement.

When the effects of positive, negative, and mixed forms of argument are adjusted for the concept groups retain their relative position: Games – mixed, Risk Management – top, Speed – close second, Decentralisation – last.

There seems to be more agreement that management needs to be frequent and proactive than there is that it should be decentralised.

A related message comes from the ratings of arguments generally favouring more control versus arguments generally favouring less control. In the graph below, arguments have been divided into three groups: m – more control, l – less control, and o – other arguments. I have classified arguments as favouring less control if they favour decentralisation (without specifically claiming that this leads to greater control) or they argue for not doing budgets on grounds of being time wasting without suggesting an alternative. Since it has already been shown that decentralisation arguments and purely negative arguments are unpopular the result of this analysis is no surprise, though it does suggest another reason why those groups were unpopular.

Best and worst arguments

Here are the top 10 arguments in terms of average rating, in descending order, with their average ratings.

These are the top 11 arguments in terms of percentage of respondents disagreeing.

And these are the bottom 10 arguments in terms of average rating, in ascending order (i.e. worst arguments first).

Finally, these are the bottom 12 arguments in terms of percentage of respondents disagreeing.

Effect of Beyond Budgeting Round Table advocacy

The Beyond Budgeting Round Table has been operating for some years now, running courses and seminars, distributing papers, and generally promoting its management model. In 2003 a book was published called ‘Beyond Budgeting", written by Robin Fraser and Jeremy Hope, published by Harvard Business School Press.

Arguments often used by the BBRT should fare well in the survey because they are more likely to be familiar and well understood by at least some of the respondents, and because over the years the BBRT will have learned to concentrate on the most effective arguments.

Dividing the 30 arguments in the survey between those commonly used by the BBRT and those not was not easy as so many arguments have been advanced by the BBRT in its various publications and speeches. I did it using my knowledge of their materials.

A look at the scattergram shows that there is little if any difference in impact between arguments used often by the BBRT and other arguments.

However, the average ratings and percentage of disagreement show a slight effect in the expected direction. The most sensitive indicator, percentage of respondents disagreeing, averaged 17.8% for arguments which are often used by the BBRT, and 21.4% for arguments not often used by the BBRT.

Comments by respondents

At the end of the survey respondents had the opportunity to make other comments. Here are the ones not related to the design of the survey, edited in places to preserve anonymity:

‘Quite interesting answering this from a public sector point of view. Is it not possible to have annual budgets etc. (useful because given annual amounts of money from government and funders and looking to achieve long term rather than short term goals) and flexibility, proactivity, acknowledgment of changing conditions etc.? This doesn't seem like an either or to me (but I don't claim to be particularly well-informed!)’

‘Deviation from aim control charts are very helpful to make budgeting/forecasting more meaningful – especially if the measurement process is bottom-up.’

‘Agree with concept of adaptive systems but remain unconvinced of success unless culture in many organisations is changed otherwise back to old story of using metrics as a measurement tool rather then a tool to help improve; metric driven versus goal driven.  If you can change culture then you will reap benefits of systems that will help decision support in reacting to market demands etc and benefit the overall organisation rather then reacting to internal budget needs.’

‘Beyond budgeting is mainly applicable in private sector companies. What about public sector organisations? Any example of beyond budgeting in public sector organisations at hand? They need it probably much more than private sector.’

‘Organisations only have a finite amount of cash. An awareness of how we are spending is essential.’

‘I don't think it's as black-and-white as ‘budgetary control and other systems with fixed performance targets should be ditched and replaced with more adaptive management systems". Why not have both and use whichever is more appropriate? Standard systems as a baseline, with adaptive systems in order to be responsive to change? I particularly disagree with Q8 – you need a baseline against which to identify risk.’

‘In budget controls we are looking at the rationality of spending the money we already have. However, we need to look at the market and explore areas where we can create business and make money – increase our business and worth. Like, Act on opportunity costs.’

‘In a (UK) Government Department situation, where (in general) it is not easy to carry unspent funds forward to the next financial year, budgetary control is essential. The answer is to link budget monitoring more closely with measuring how far objectives are being met.’

‘All of the criticisms above are addressed in well-managed budget systems.  It is not so much budgets that cause the problems as they way in which they are used – normally in businesses which are not particularly well-managed anyway.  I am not saying we definitely have to keep budgets, but I have yet to see a proposal which significantly improves on the situation without raising its own weaknesses.’

‘In our City Government, City Commissioners and the citizens are used to hearing the City Manager talk about what the City plans to do based on the budget.  I believe it would be difficult to eliminate the budget process in our City.  However, using the concepts in the beyond budgeting model may help the City Manager make the transition away from a purely budget basis for discussions regarding what the City will do each year or over the next 5 years.’

‘It is essential that the schedule and budget reflect a best estimate of the actual work to be accomplished. If this cannot be projected 1 year, six months or whatever, ahead then you must move to near real-time planning (actual planning window determined by the advance time that task are known, together with their expected duration) Note: I use the term ‘task’ to reflect a series of activities associated with expenditure of revenues and the development of end products.’

‘We are privately held, so as a non-public firm we do not encounter many of the issues noted in the survey.’

‘In order to decentralise expenditure target setting and control, all employees with expenditure delegations must have an up to date understanding of the strategic goals of the organisation, and be prepared to be accountable for achieving these goals. Too few organisations do this, and are afraid to ditch the budget process as a result.’

Respondents

There were 40 respondents. All completed the survey online. Most have a background in performance management, audit, or accountancy. Two were classified as authors, 20 more were in favour of Beyond Budgeting or something similar but were not authors in the field, 12 were not convinced/against, and 6 could not be classified. This classification was based on the question in the survey but in three cases this answer was lost due to a technical fault but I inferred agreement from the high average ratings. In nearly all cases of known opinion, an average rating of more than about 1.75 indicated overall support.

The 30 arguments

These are the arguments used in the survey, in the order they were presented:

Further information

If you would like to analyse the original data yourself I can provide a matrix of the ratings given, classified between Author, Pro, Anti, and unknown. The information will not allow you to identify respondents or their organisations.

Copies of the original survey are also available. Please contact me at matthew@workinginuncertainty.co.uk.

Words © 2004 Matthew Leitch. First published 19 February 2004.