Audit of Selected Monitoring and Oversight Activities

Final Report: November 2013

This report has been reviewed in consideration of the Access to Information and Privacy Acts. The published information is UNCLASSIFIED.

Table of Contents

Acronyms and Abbreviations

APPS
Annual Performance Planning System
CAEE
Chief Audit and Evaluation Executive
CO
Commanding Officer
COSO
Committee of Sponsoring Organizations of the Treadway Commission
IAER
Internal Audit, Evaluation, and Review
MR
Management Review
NRS
National Review Services
RBAP
Risk-Based Audit Plan
RCMP
Royal Canadian Mounted Police
RDC
Regional Deputy Commissioner
SEC
The RCMP's Senior Executive Committee
ULQA
Unit Level Quality Assurance

Executive Summary

The Royal Canadian Mounted Police (RCMP) conducted an audit to assess the extent to which the Management Review (MR) and Unit Level Quality Assurance (ULQA) processes provide value and support effective administration of programs and services.

MRs are intended to assess the adequacy and effectiveness of a unit's management and investigative performance. These reviews are to be conducted by employees independent of the unit subject to review. The ULQA is a formal self-assessment process and monitoring tool through which unit managers conduct periodic risk-based assessments of activities. It is designed to assess compliance with operational, financial, administrative and program responsibilities.

The results of the audit determined that MR and ULQA processes assisted local commanders and managers to identify and monitor higher-risk activities faced by their units. These processes focused on compliance to policy and procedures and local deficiencies were being identified and addressed. However, in many cases, the processes identified symptoms rather than the root causes of the deficiencies, therefore not preventing the reoccurrence of the errors. In addition, the current processes provided results on divisional or unit conformity, which had limited benefits being realized at the national level. Without the organization's ability to analyze overall results and to communicate the information to senior levels of management, RCMP is missing opportunities to better integrate these processes with respect to quality and risk management of programs and services.

Furthermore, Senior Management should assess where the responsibility for these processes resides within the organization to provide useful information for organizational improvement. In addition, accountabilities, roles and responsibilities related to the MR and ULQA processes need to be clarified in order to update policy and methodology accordingly.

The above noted recommendation will lead to improvement of the value-added of the MR and ULQA processes to better support effective administration of programs and services. The Management Response demonstrates the commitment from senior management to address the recommendation. A detailed Management Action Plan will be provided and RCMP Internal Audit will monitor its implementation.

Management's Response to the Audit

Many RCMP resources are committed to a number of on-going monitoring and oversight activities. Two such activities, Management Reviews (MRs) and Unit-Level Quality Assurance (ULQA), are the subject of this audit.

If all monitoring and oversight activities, including MRs and ULQAs, are to add-value and lead to organizational improvements, they must identify areas of concern and assist the organization in addressing them in an effective and timely manner.

Based on the results of the audit, improvements are needed to maximize the benefits that can be derived from these two monitoring and oversight activities. One of the key findings of the audit is that some recommendations and their corresponding action plans do not address the root cause of the observed deficiencies, increasing the likelihood that noted deficiencies will reoccur. If actions are taken to address symptoms of deficiencies as opposed to the root cause, there is limited benefit in undertaking such activities.

For the period of FY 2004 to April 2013, Internal Audit, Evaluation and Review (IAER) Branch, led by the Chief Audit and Evaluation Executive (CAEE), was the policy centre for the MR and ULQA processes. Effective April 2013, following a Senior Executive Committee decision, the IAER branch ceased to be the policy centre for these two processes.

Subsequent to this decision, a business case was developed to establish a Continuous Auditing Team (CAT) within the IAER branch. The business case, approved by the Commissioner in September 2013, approves the establishment of a team that will provide a more cost-effective means of monitoring a greatly increased quantity of data by leveraging the use of technology. By analyzing the results of ULQA's and MR's, and using these as only one input to a more comprehensive continuous monitoring regime, CAT will provide global oversight of these processes.

Dennis Watters

Chief Audit and Evaluation Executive

1. Background

In June 2012, the Commissioner approved an Audit of Selected Monitoring and Oversight Activities as part of the 2012-15 Risk-Based Audit Plan (RBAP). The RBAP highlights that resources are dedicated to monitoring and oversight of activities at all levels of the Royal Canadian Mounted Police (RCMP). During the planning phase of the audit, a risk-based prioritization exercise was conducted to determine which of these monitoring and oversight activities required audit attention. Through this exercise, the Management Review (MR) and Unit Level Quality Assurance (ULQA) processes were selected. These processes are intended to monitor the internal quality of operational, administrative and financial activities, focusing on the higher-risk activities and ultimately supporting accountability throughout the RCMP.

Administration Manual I.7 - Audit and Evaluation Programs serves as national policy for the MR and ULQA processes. It describes the responsibilities for the coordination and delivery of these quality assurance processes. In addition, there are local policies, the MR and ULQA handbooks, and several review guides which provide guidance on process delivery.

National Review Services (NRS), a unit within Internal Audit, Evaluation and Review (IAER) led by the Chief Audit and Evaluation Executive (CAEE), serves as the National Policy Centre. It is responsible for developing policies and procedures, coordinating the development of adequate tools, and monitoring the effectiveness of the MR and ULQA processes. Recently, NRS' oversight role and responsibilities were significantly impacted following two Senior Executive Committee decisions. These decisions and their impact were considered during the audit.

Each Commanding Officer (CO) Footnote 1 is responsible for establishing and maintaining an adequately resourced Regional/Divisional Review Group to promote and monitor the implementation of the two processes. Also, Regional/Divisional Committees are responsible for monitoring overall results of MRs, approving review plans, ensuring implementation of ULQA process, and that trained resources are available to meet the demands of the regional review plan.

National Headquarters policy centres, in coordination with NRS, have to ensure that review guides are maintained and reflect current RCMP and Government policies and procedures as they affect operations and administration of the RCMP.

Management Review Process

MRs are intended to assess the adequacy and effectiveness of a unit's management and investigative performance. These reviews are to be conducted by employees independent of the unit subject to review.

Each year, the MR process begins within a division when its risk-based plan is established by a committee comprised of divisional senior management. This risk-based approach includes input from the unit level all the way through divisional senior management. As the first step in the process, input is provided from the unit level, by means of risk self-assessments, which are subsequently validated by their line officers. To establish the plan, the divisional committee members consider their own risk input in conjunction with the unit self-assessments in order to obtain a holistic view on risk within the division. Once the plan is established, it is sent to the Regional Review Group to begin scheduling MRs for the year.

Once a MR is complete, recommendations are issued by the review team, and action plans are established by unit management. Implementation of action plans is to be followed up by the Regional Review Group, and status of implementation is shared with the divisional committee.

Unit Level Quality Assurance Process

The ULQA is a formal self-assessment process and monitoring tool through which unit managers conduct periodic risk-based assessments of activities. It is designed to assess compliance with operational, financial, administrative and program responsibilities. As the manager is ultimately responsible for the management and supervision of a unit, he/she is accountable for the effectiveness and integrity of the ULQA process.

The first step in the ULQA process is to conduct a unit risk self-assessment. This requires a unit to first identify 15 activities, and then conduct a risk assessment. As per the ULQA handbook, divisional mandatory activities, which are refreshed and established annually by divisional senior management, must be included in these 15 activities. This process helps ensure that divisional priorities remain relevant and that they are considered by each unit when selecting which activities will be subject to review during the year.

Once a ULQA is complete, recommendations are generated, and action plans are established by unit management. Implementation of action plans is to be tracked by the unit's line officer. Throughout the ULQA process, the unit is required to document the work electronically through the Annual Performance Planning System (APPS).

2. Objective, Scope and Methodology

2.1 Objective

The objective of this audit was to assess the extent to which the MR and ULQA processes provide value and support effective administration of programs and services.

2.2 Scope

The audit was national in scope, covering the MR and ULQA functions in each region and at national headquarters. To ensure relevance of audit evidence, a judgmental sample of MRs and ULQAs completed in the 2011/2012 fiscal year was reviewed. The 2011/2012 fiscal year was chosen as the audit team needed access to files which have been fully completed through the follow-up process.

2.3 Methodology

Planning for the audit was completed in February 2013, and included preliminary interviews with the Strategic Policy and Planning Directorate, divisional and regional management services, NRS, and a selection of Criminal Operations Officers and COs. In this phase, the audit team examined information related to several monitoring and oversight activities. A prioritization exercise was required to identify activities to audit, and risks outlined in the RBAP were used for the assessment.

Sources used to develop audit criteria included RCMP policies and procedures, the Audit Criteria related to the Management Accountability Framework developed by the Office of the Comptroller General, and the COSO Internal Control Model. The audit objective and criteria are available in Appendix A.

The examination phase, which concluded in April 2013, used various auditing techniques including interviews, documentation reviews, file reviews, and analysis of information. For MRs, a total of 40 files were selected on a judgmental basis. Visits were conducted at four sites and National Headquarters. The visits allowed for testing of both contract and federal divisions. Upon completion of each visit, the audit team held exit meetings to debrief management of the relevant findings.

For the ULQA process, the majority of the testing was conducted through interviews, file testing, and analysis. A national random sample of 100 electronic files from the APPS was selected and reviewed. In addition, a sample of ULQA hardcopy files was also tested during the site visits.

2.4 Statement of Conformance

The audit engagement was planned, conducted and reported in accordance with the Internal Auditing Standards for the Government of Canada.

Standard 1130.A2 of the International Standards for the Professional Practice of Internal Auditing states that: "Assurance engagements for functions over which the chief audit executive has responsibility must be overseen by a party outside the internal audit activity."

As such, this audit engagement required an oversight authority outside of the CAEE in order to conform to standards. The Chief Financial and Administrative Officer agreed to serve as the appropriate authority over this engagement.

3. Audit Findings

3.1 Results Obtained from the MR and ULQA Process

While the processes lead to local correction of reported deficiencies, they do not contribute to sustainable improvement across the entire organization.

Management Review

MR is a process which assesses the adequacy and effectiveness of a unit's management. According to the methodology, the review should identify conditions of non-conformance or deficiencies in areas that represent a greater risk exposure and impact. In order to ensure deficiencies found during the review do not reoccur, recommendations and corrective measures should ensure that the root causes of deficiencies are identified and addressed.

Of the 40 MR files examined, many instances were observed where the corresponding recommendations and management action plans were not sufficient to prevent the reported deficiencies from reoccurring. Accordingly, if sound recommendations and corresponding management action plans are not developed properly, it increases the likelihood that the deficiencies will reoccur.

Consequently, the value added of an MR depends on the effective application of review methodology and implementation of a sound action plan to solve deficiencies and prevent reoccurrence. However, when the root cause of deficiencies is unknown or not identified during an MR, action plans may have corrected partially or temporarily the deficiencies. A robust follow-up process could identify this situation as deficiencies are likely to reoccur if not adequately addressed. The follow-up processes were limited to only obtaining written confirmation that action plans were completed. Further monitoring by Regional Review Groups was not consistently done to ensure that action plans were in fact implemented and that they solved deficiencies.

As MR findings, recommendations and management action plans are typically limited to one unit, Regional Review Groups consolidated results of multiple MRs in order to provide the divisional committees with a summary of common issues throughout the divisions. While results are shared within the divisions, they are not typically shared with other divisions or with national policy centres. The sharing of results would allow for the analysis of trends or identification of common issues. All three National Headquarters policy centers that were interviewed indicated that they are not receiving results of MRs, and two of them indicated that this created a gap in monitoring. These two national policy centres indicated that national consolidation of MR results would assist them with their monitoring requirements. Given the local nature of the MR results, national policy centres would need to further analyze the findings and recommendations in order to ensure that the root causes of deficiencies are in fact identified and recommendations are sufficient.

Unit Level Quality Assurance

ULQA is a self-assessment tool that is used to assess the level of conformance to existing policies and procedures. With corrective actions being generated at the unit level, it is not always feasible to address the root cause of deficiencies and therefore to prevent their reoccurrence. Similar to the MRs, several corrective actions resulting from ULQAs addressed the symptoms as opposed to the root cause of the noted deficiencies, thus not preventing their reoccurrence. For example, 88 of the 100 files reviewed identified deficiencies that required corrective actions. Of the 88 files, only 32 contained recommendations that addressed the root cause of deficiencies.

Furthermore, there are varying levels of commitment to the process which impact its effectiveness. For example, there are different interpretations of which units are required to conduct ULQA activity, and this has an impact on ULQA completion rates.

As previously mentioned, the first step in the ULQA process is to conduct a unit risk self-assessment. This requires a unit to first identify 15 activities, and then conduct a risk assessment. As per the ULQA handbook, divisional mandatory activities, which are refreshed and established annually by divisional senior management, must be included in these 15 activities. Some divisional management have misinterpreted this requirement as having each unit review all mandatory activities. This approach ultimately reduces the extent to which the process is truly risk-based at the unit level. In addition, not all units considered the mandatory activities when conducting the risk assessment.

ULQA results are shared with divisional management through the committees, but to varying extents, depending on the composition of the committees. Information provided to the committees is not typically shared with other divisions or with the national policy centres, and this has an impact on the RCMP's ability to analyze results which could lead to organizational improvement.

In addition, although ULQAs are documented through the APPS, analysis of overall results using data from the system at the divisional and national levels is limited.

3.2 Governance of the MR and ULQA Processes

Current governance of the MR and ULQA processes is unclear.

National Oversight

The chapter pertaining to MR and ULQA in Administration Manual 1.7 - Audit and Evaluation Programs was last updated in 2004. The chapter has not been revised to reflect the current practices and recent management decisions.

NRS, a unit within IAER, served as the National Policy Centre for MR and ULQA processes; however the CAEE expressed concerns related to MR and ULQA responsibilities. According to TB Policy on Internal Audit, the Chief Audit Executive should not have any departmental management or operational responsibilities which may compromise independence and objectivity in respect of internal audit responsibilities. The MR and ULQA are viewed as management monitoring activities which could have an impact on the ability to maintain independence of the internal audit function from line management. In addition, there was a view that the monitoring performed by NRS would be more efficient if it was undertaken by national policy centres, as they would be better positioned to assess the results from MRs related to their policies and address any common themes. Also, it would facilitate the update of review guides with current policies and procedures. Consequently, in 2011, the RCMP's SEC approved a proposal that limited the CAEE's role in relation to the MR and ULQA processes. Furthermore, in April 2013, SEC agreed that NRS would no longer be the National Policy Centre, and that policy would soon be updated. As such, the MR and ULQA processes are currently functioning without national policy centre oversight.

Notwithstanding the current oversight changes, it would appear that the overall monitoring of the effectiveness of MR and ULQA processes was not being performed by NRS as expected. Although NRS produced annual reports on MR and ULQA results, it was unclear how the reports were used to assess overall effectiveness. For example, in the methodology section of the MR and ULQA handbooks, it is stated that MR and ULQA focus on root cause of deficiencies being identified however, as previously mentioned, most MRs and ULQAs did not or could not always address the root cause. Unless the root causes of observed deficiencies are addressed, the value added of both the MR and ULQA processes will be limited.

In addition, there are inconsistent interpretations on what is expected when implementation dates are established. It is unclear if the date represents the completion date or the date when a status report is to be provided, thus preventing the ability to determine if and when action plans are actually to be implemented. Furthermore, various interpretations of mandatory activities for ULQA as well as the interpretation of who is required to perform the ULQA, are additional examples of methodology issues that had not been addressed.

It is to be noted however, that the recent changes related to the national oversight for the MR and ULQA processes had an impact on roles and responsibilities related to the maintenance of review guides, liaison with national policy centres and MR training.

Without sound national oversight, continuous coordination with national policy centres to ensure policy changes are reflected in review guides in a timely manner is limited. Divisional employees took on the responsibility of updating review guides before each review. Without coordination or appropriate decentralization of this role, duplication of effort exists with divisions updating review guides individually and not necessarily sharing the updates. Accordingly, if updates are not made, management risks making decisions based on invalid results.

In addition, without a liaison with national policy centres to integrate and share divisional MR plans and results with national policy centres, extensive overlap is more likely to occur across the organization. Also, because results from MR and ULQA processes are not being shared with the national policy centres, the organization's ability to identify and address systemic issues is limited.

Finally, with respect to MR training, without continuous coordination with the Learning and Development Policy Centre to deliver national training, standardization of service delivery will decrease, there will be an increased burden on divisions to deliver training, and duplication of effort may exist with respect to the development of training material.

Local Oversight

Since the elimination of the RDC positions, the divisional COs have effectively taken on this responsibility. As such, the divisions have established different organizational structures and service delivery models for the MR and ULQA processes. In addition, Review Services within four divisions have yet to determine how the services will be provided, or where they will report organizationally. This uncertainty may have an impact on the sustainability of the processes and it could be decided that the MR and ULQA processes within those divisions are no longer used.

While these structures and delivery models suit the needs of the respective COs, there are several implications, such as: MR process costs; MR overlap with other monitoring activities; and ULQA responsibilities that could impact on the overall effectiveness and efficiency of the processes.

Some Review Groups utilize permanent review staff for each MR that is conducted within its divisions, while others utilize trained reviewers whose permanent position is located in close geographical proximity to the location of the review. Using permanent review staff increases costs as reviewers' salaries are a significant component of the local Review Groups' budget. Travel costs to conduct site visits for reviews also increase under this model. If locally-based reviewers are used, travel costs are minimized. The extent of follow-up on management action plan implementation also impacts costs of the process. Some divisions perform extensive follow-up, as opposed to others who rely on line officer verifications that action plans have been implemented. Regardless of the chosen delivery model, several different budgets are impacted within the divisions. While Review Groups' budgets typically included salary and operations and maintenance, travel costs for site visits come from various budgets including: the District Commanders', Divisional Criminal Operations', or the CO's. The use of various costing methods prevents the RCMP from knowing the total costs of MR processes at the divisional and national levels.

In addition, with delegation of the MR process implementation to divisions, overlap varied with other monitoring activities. For example, some Review Groups indicated that they avoid testing areas of finance that are being monitored by local Internal Control groups. Others indicated that they wanted additional assurance, through provision of an MR, over these activities. Ultimately, the intention is to bring risks to an acceptable level, even if overlapping monitoring activities is required. However, the process becomes more inefficient as overlap is increased.

Implementation of the ULQA process, and the delivery of its training, differed significantly across the divisions. The groups administering the process within the divisions reported to different sections, and their level of involvement varied. Unclear responsibility has led to poor clarity over which units are required to conduct ULQA activity, inconsistent application of mandatory activities, and accountability not being exercised over initiating or completing the process. While the results of the ULQA and data input into the APPS are being monitored in some divisions, such monitoring is inconsistent.

4. Conclusion

MR and ULQA processes are intended to help commanders and managers identify and monitor higher-risk activities faced by their units. These processes focused on compliance to policy and procedures and local deficiencies were being identified and addressed. Although the current processes provided results on divisional or unit conformity, there were limited benefits realised at the national level.

To maximise the value and support continuous improvement, the sharing of common issues outside of the Division would assist in addressing deficiencies and lead to more efficient administration of programs and services.

Furthermore, clear accountabilities for the management and monitoring of MRs and ULQAs are essential to ensure appropriate delivery of the two processes.

5. Recommendation

Following the organizational changes related to the national policy centre and regional delivery for the MR and ULQA processes, the SEC should assess where the responsibility for these processes resides within the organization.

Using the results of the assessment, the following should be clarified:

  1. MR and ULQA objectives, and
  2. Accountabilities, roles and responsibilities for these processes.

Appendix A - Audit Objective and Related Criteria Footnote 2

Objective:

The objective of the audit is to assess the extent to which the MR and ULQA processes provide value and support effective administration of programs and services.

Criterion 1:

A governance process is in place which supports achievement of MR and ULQA objectives.

Criterion 2:

Overlap with other monitoring and oversight activities is reduced to the extent possible.

Criterion 3:

MR and ULQA methodologies are risk-smart.

Criterion 4:

Results obtained from MR and ULQA activities lead to organizational improvement.

Appendix B - Detailed Management Action Plan

Recommendation:

Following the organizational changes related to the national policy centre and regional delivery for the MR and ULQA processes, the SEC should assess where the responsibility for these processes resides within the organization.

Using the results of the assessment, the following should be clarified:

  1. MR and ULQA objectives, and
  2. Accountabilities, roles and responsibilities for these processes.

Management Action Plan

Agreed. Following SEC discussion, a business plan was prepared, approved by the Commissioner on September 4, 2013, which outlines the establishment of a Continuous Auditing Team (CAT) within the IAER branch, in response to this recommendation.

The overarching objective will be to ensure that more timely, effective corrective action is taken when control gaps or deficiencies are identified by any of the internal monitoring mechanisms. As articulated in the business case, this will be accomplished by ensuring that:

  • Findings of significance are communicated to Senior Management in a timely manner;
  • Appropriate follow-up measures are put in place;
  • SEC is briefed at least annually; and
  • Policy centres are reminded annually to review National Review Guides (NRGs) to ensure they remain relevant and up-to-date.

By analyzing the results of ULQA's and MR's, and using these as one input to a more comprehensive continuous monitoring regime, CAT will provide global oversight of the process. Overall, the CAT will provide a more cost-effective means of monitoring a greatly increased quantity of data. It will focus on identifying potential risks, and leave management to propose solutions.

Completion Date: September 2014

Position Responsible: Chief Audit and Evaluation Executive

Date modified: