Working In Uncertainty

Evidence for an efficient approach to evaluating controls effectiveness

by Matthew Leitch, first published 27 April 2004.

Contents


Thank you

First, thank you to everyone who responded to this survey. The results are generally clear cut and important for most people involved in audit work, their employers, and regulators. One thing I personally found interesting about the results was how sensitive auditors are to potentially relevant evidence.

Summary

The evidence most often mentioned in regulations on internal controls – and almost the exclusive concern of the PCAOB's requirements for SOX 404 compliance – is of individual tested controls. However, in real audit work other information is relevant and by casting the net wider it is possible to conduct a much more efficient review. This is because it is possible to cream off the most persuasive and easily gathered evidence in each category rather than digging down into expensive but individually unimportant details.

If you are interested in exploring this strategy two questions may have occurred to you: (1) ‘Do other auditors accept evidence other than individually tested controls?’ and (2) ‘Will our external auditors accept other evidence?’ This survey set out to find what kinds of evidence auditors generally accept as relevant to evaluating internal controls effectiveness. The results give strong support to use of the strategy.

The survey was conducted online during April 2004 and respondents were self-selected, though the thinking underlying the survey was not explained and comments from respondents suggested they were puzzled by what the survey was doing. Respondents were presented with 20 statements and asked how relevant each was, as evidence, in evaluating the effectiveness of a system of internal controls over financial reporting. Respondents could choose between ‘Relevant’, ‘Somewhat relevant’, and ‘Not relevant’.

The statements were of 4 main types: (1) ‘distractors’ intended to have no relevance, (2) individual controls tested, (3) inherent risk factors, and (4) process health measures. The tested controls were further divided into application controls, IT controls, and ‘tone at the top’.

The most relevant statements concerned process health measures – a dramatic finding as these are rarely mentioned in auditing theory – closely followed by tested controls and inherent risk factors. Many of the statements designed to be Not Relevant were considered Relevant or Somewhat Relevant by at least some auditors and this is almost certainly because they suggested inherent risk factors.

In short, all the auditors in this survey considered process health measures to have relevance and over 95% considered inherent risk factors to have relevance as evidence. This provides confidence that a strategy that uses them will be persuasive for internal and external auditors. Such a strategy is explained in ‘Sarbanes-Oxley Act section 404 and 302: efficient compliance’.

Relevance

Out of 46 respondents, just two considered all inherent risk factors to be irrelevant as evidence.

All respondents considered process health measures, such as error rates, to be relevant. Indeed, process health measures were, arguably, more relevant for respondents than even clear information about individual controls tested. This is perhaps because they show the end result of control, whereas evidence that individual controls have operated does not demonstrate that the system as a whole is working well.

The full results are shown on this graph and the following table. On the graph the 20 statements are numbered along the bottom and the abbreviations indicate the type of statement: irf = Inherent Risk Factor, phm = Process Health Measure, dis = Distractor, con-env = Control tested – environment (‘tone at the top’), con-appn = Control tested – application control, con-IT = Control tested – IT control:


No.StatementType% Not Relevant% Somewhat Relevant% Relevant
1Thanks to a merger during the year, preparing the financial statements will be much more complex than in the past.irf6%15%79%
2The company uses a bespoke consolidation system that has been extensively updated this year.irf6%17%77%
3There have been no changes to finance staff or systems during the year.irf15%43%43%
4Customer queries include a very large number of complaints about incorrect bills.phm0%2%98%
5The company uses automated checking of its billing using large numbers of test items. Results show very few errors, none large.phm4%13%83%
6Unmatched cash and general ledger suspense items are both high and have been climbing rapidly during the last quarter.phm0%0%100%
7The company is considering extending its product line over the forthcoming year.dis28%40%32%
8The company's logo has been changed during the year.dis85%13%2%
9The company's sales prices increased steadily during the year to a level 3% greater than last year.dis34%40%26%
10The chief finance officer has brown eyes.dis94%2%4%
11The company's founder, who retired three years ago, has founded a charity to support medical research.dis81%13%6%
12The main call centre was refurbished during the year.dis49%40%11%
13The company's new product has won an award for innovation.dis66%26%9%
14The external auditors have merged with a rival firm.dis53%28%19%
15The board of directors has issued a formal policy on risk management, which has been communicated to all employees.con-env4%34%62%
16A survey of senior executives has shown that 95% agree that an ethical approach to financial reporting is a top priority.con-env11%40%49%
17Testing has confirmed that all consolidation adjustments are checked independently from the originator and also authorised by the chief accountant.con-appn4%2%94%
18The sales ledger and its general ledger control account are reconciled daily and testing has confirmed that this has been done effectively.con-appn0%4%96%
19The company's computer network is protected from security attacks by a sophisticated firewall, and testing has confirmed its rules have been properly maintained.con-IT0%17%83%
20Changes to the software of the bespoke order system are tested thoroughly before being used live, and audit testing has confirmed this has been done effectively.con-IT0%11%89%

In addition to the main effects mentioned above, there are some interesting smaller effects.

  • The unremarkable inherent risk item: Item 3 was an inherent risk factor but more respondents rated it as only Somewhat Relevant than for the other two. The text of the item was ‘There have been no changes to finance staff or systems during the year.’ Had it said that the staff and systems had changed a lot more respondents would have considered the relevance clear. As it was, the fact that this information removed the possibility that the staff and systems had changed was enough for most respondents to see at least some relevance.

  • Relevant distractors: The ‘distractors’ were statements designed to be irrelevant as evidence. The example that some respondents found amusing or puzzling was the statement that the finance director had brown eyes. However, it seems it is very difficult to think of information about an organisation that is completely irrelevant because so much can indicate inherent risk. Items 7, 9, 12, and 14 seemed to be Relevant or Somewhat Relevant to many respondents. The one thing that slightly concerns me is that so many people saw relevance in the statement that ‘The external auditors have merged with a rival firm."

  • Apparent control environment not so persuasive: Of the statements of individual controls the least persuasive were clearly those concerning the control environment. When Chiefs say they consider reliable financial reporting to be fundamental we are not hugely impressed.

Internal versus external views

28 of the 46 respondents had at least some external audit experience, but only 10 had spent more time as external auditors than as internal auditors, and only 4 respondents were purely external auditors. This is unsurprising as respondents were obtained mainly by e-mailing subscribers to the AuditNet website, a popular resource for internal auditors.

The four pure external auditors all considered inherent risk factors to be relevant, as did the other six who had more external than internal audit experience.

Aggregating the results of people with more external than internal audit experience, and comparing them with the other respondents shows that there is little difference between them.

Does experience count?

Years of audit experience did not seem to make much difference. Respondents were divided into a tiny group with no audit experience and two large groups, one for people with more than 10 years of any kind of audit experience and one for others. If anything, the respondents with less than 10 years of experience tended by give answers closer to my intention when devising the questionnaire.

Comments by respondents

The survey asked if the respondent had any other comments they would like to make about audit evidence. Most respondents made no comment and several made light hearted comments about the survey, especially the item about eye colour. There was just one serious comment, throwing light on the lower value usually placed on evidence about the attitudes of senior executives.

‘Part of effective internal controls includes good written policies and procedures. However, these are not to be taken in isolation of all other factors. If policies and procedures are circumvented they are of little or no value for being an effective part of I/C. Interviews of persons who do the work to determine just how the work is done will give insight as to employee awareness and the effectiveness of the written policies and procedures.’

Respondent profile

Slightly over half the respondents were from the United States of America. The countries of respondents are shown on this graph:

Most were internal auditors, with predominantly internal audit experience.

Instructions used in the survey

Critical instructions used in the survey were contained in the following paragraphs. Firstly, there was only a vague suggestion as to the purpose and rationale of the survey. This was to reduce the risk of biased responses from people eager to support a more efficient style of SOX 404 compliance. The introductory words were:

‘We need to find out more about what people consider relevant to deciding how effective a system of internal controls is. The range of evidence considered has a big impact on the cost of auditing controls effectiveness so there are immediate practical implications.’

‘For this survey to have any value it is vital that you answer honestly. Your answers will be confidential. Your employer, if you have one, will never know what you answered.’

The twenty statements were listed in a random order (using a random number generator seeded by the time of day in milliseconds) and the instructions were as follows:

‘The following 20 items are things that might be relevant to your evaluation of the effectiveness of a system of internal controls specifically for financial reporting. These statements aren't about the same organisation, so take each one in isolation.’

‘Consider each and decide how relevant it is, if at all, as evidence relating to the current effectiveness of controls over financial reporting, in your opinion. Don't worry about complying with any particular piece of official guidance or regulations. What do you think? What could influence your view?’

Further information

If you would like to analyse the original data yourself I can provide a matrix of the ratings given. The information will not allow you to identify respondents or their organisations.

Copies of the original survey are also available. Please contact me at matthew@workinginuncertainty.co.uk.






Made in England

 

Words © 2004 Matthew Leitch.