Audit of Federal Policing Major Project Governance Tools

Final vetted report

March 2022

• Acronyms and abbreviations
• Executive summary
• Management's response
• 1.0 Background
  • 1.1 Audit context
• 2.0 Objective, scope and methodology
  • 2.1 Objective
  • 2.2 Scope
  • 2.3 Methodology
  • 2.4 Statement of conformance
• 3.0 Observations
  • 3.1 Oversight Mechanisms
  • 3.2 Divisional Processes
  • 3.3 Monitoring and Reporting
• Appendix A – Audit objective and criteria
• Appendix B – Management action plan
• End notes

Acronyms and abbreviations

CrOps

Criminal Operations Officer

DCFP

Deputy Commissioner Federal Policing

DG

Director General

FP

Federal Policing

FPCO

Federal Policing Criminal Operations

FP AU

Federal Policing Assessment Unit

FP RM

Federal Policing Resource Management

NHQ

National Headquarters

PDF

Portable Document Format

PPD

Projects and Prioritization Dashboard

Prioritization Committee

FP NHQ Standing Committee on Major Project Prioritization and Governance

Prioritization Tool

Federal Policing Major Project and Governance Tool

PROS

Police Reporting and Occurrence System

RCMP

Royal Canadian Mounted Police

ROD

Record of Decision

SASU

Situational Awareness Support Unit

SPROS

Secure Police Reporting and Occurrence System

Executive summary

Background

National Headquarters (NHQ) Federal Policing Criminal Operations provides oversight and governance for investigations related to serious and organized crime, financial crime, cybercrime and border integrity. The Major Project Prioritization and Governance Tool (Prioritization Tool) was established in 2013 to assist Federal Policing (FP) management in determining how best to apply its investigative resources against the most important priorities, criminal threats and activities facing Canada. The Prioritization Tool is intended as a cost benefit assessment mechanism that takes into consideration variables to weigh the overall importance, profile, resource cost, and net benefit of a project based on FP priorities. A tier level is assigned to a major project to indicate the priority of addressing the criminal threat.

The implementation of the national Prioritization Tool is a component of the FP Governance Framework. In 2021, the FP Transformation initiative was in the process of examining how FP may be more responsive to the complex and evolving nature of the operating environment.

Why this is important

The current policing environment is complex and fast-changing with global crime networks posing significant risks to the safety and security of Canadians. Major projects require the sustained commitment of a team or teams of investigators over extended periods of time to investigate criminal activity. With limited resources, the prioritization of the most important criminal threats ensures that the RCMP can align resources to concentrate its efforts on high priority criminal activities. The Prioritization Tool supports FP governance by providing situational awareness about Federal investigations across Divisions.

Audit objective and scope

The objective of the audit was to determine whether a sound governance framework over major projects was in place that included effective oversight bodies, clearly established policies and procedures, and mechanisms to monitor and report on key performance indicators. The scope of the audit included an examination of the use of the FP Major Project Prioritization and Governance Tool from April 1, 2018 to March 31, 2020.

Findings

The following aspects of the Federal Policing Major Project Prioritization and Governance Tool require management's consideration. The detailed observations and recommendations are discussed in the report that follow this executive summary.

Observation 1

Federal Policing has established a national prioritization process to support governance. Opportunities exist to enhance oversight mechanisms to ensure that the Prioritization Tool meets its intended purpose of allocating resources to the highest threats.

Observation 2

Divisional processes are in place to apply the national prioritization process requirements. Enhancing the process to provide visibility on the progress of tiered projects would facilitate Federal Policing maintaining and improving oversight over major projects.

Observation 3

Accurate and reliable performance data is not available to support monitoring of the prioritization process. There is an opportunity to enhance monitoring and reporting to support oversight of the prioritization process.

Overall audit opinion

Federal Policing has a national prioritization process in place for major projects to support governance but some gaps exist. The process includes the Prioritization Tool, oversight by the FP NHQ Standing Committee on Major Project Prioritization and Governance, a guide for the use of the Prioritization Tool and requirements for reporting.

There are opportunities to strengthen the process and address the gaps to ensure national oversight is provided and criminal threats are assessed in the national operating context. The prioritization process relies on the discretion of Divisions to determine when they will submit a project for tiering, which diminishes the opportunity to exercise national governance. As a result, Divisions determine the priorities they intend to work on and submit projects for tiering without having a full awareness of national priorities. Without sufficient visibility on which projects are being advanced at any given time, FP may have a reduced ability to provide oversight or determine whether a project should no longer advance. [***]

The prioritization process could be enhanced by strengthening the requirements for early program engagement with Divisions, assessing whether scoring results are relevant to the tier assessment, establishing roles and responsibilities for monitoring, and providing guidance to educate users on enhancements to the process. Also, developing a process to update the status of projects would provide accurate information and increase national visibility on the advancement of tiered projects to support governance.

Next steps

The management response and action plan developed in response to this report demonstrate the commitment from senior management to address the audit findings and recommendations. RCMP Internal Audit will monitor the implementation of the management action plan and undertake a follow-up audit if warranted.

Management's response

Federal Policing's management are in support of the recommendations and observations within this Audit of our Major Project Governance Tools.

There is a recognition that FP's prioritization process is required to meet our needs, but that it also needs to evolve. The audit's findings are not surprising, and rather validate FP's arguments for a need to change.

The changes required, for instance, data quality, or the issue of divisional priorities vs. national ones; are items that support more substantial governance discussions internally, in addition to reviewing our roles and responsibilities.

Federal Policing is thankful for this review, and the audit team has our full support.

The federal policing prioritization process is critical to our future state and operational performance.

FP's next steps are to develop an appropriate management action plan that will address each of the observations and recommendations.

Thank you for the great work by this team.

FP Management.

1.0 Background

1.1 Audit context

National Headquarters (NHQ) Federal Policing Criminal Operations (FPCO) provides oversight and governance for investigations related to serious and organized crime, financial crime, cybercrime and border integrity. Given the complex and sophisticated nature of crime that falls within the Federal mandate to investigate, the Federal Policing (FP) Program must direct operational efforts and resources to the highest risk priorities.

In 2013, FP developed the Major Project Prioritization and Governance Tool (Prioritization Tool) to assist the RCMP's FP program in the application of its investigative resources on major projects against the most important priorities and criminal threats. The Prioritization Tool is a form available on the RCMP Infoweb ^{End note 1} that contains a series of variables to assess the overall priority of a project.

The Prioritization Tool provides a structured approach for Divisional management to assess their investigative projects and direct their efforts towards the highest priority FP activities within their Division. The FP Assessment Unit (FP AU) within FP Operational Information Management supports the FP NHQ Standing Committee on Major Project Prioritization and Governance (Prioritization Committee) and the prioritization cycle for all major projects. The Committee ensures that FP major projects are assigned priority to the criminal threats that pose the greatest harm to Canada. The Prioritization Tool process is summarized in Figure 1.

The Commissioner approved 2019-24 Risk-Based Audit and Evaluation Plan included an Audit of Federal Policing Major Project Governance Tools.

Figure 1 – National Process for the Prioritization Tool ^{End note 2}

Figure 1 – National Process for the Prioritization Tool - Text version

A diagram depicting the Prioritization Tool process.

The Prioritization Tool calculates the Priority Score by subtracting the Resource Requirement Score from the Profile Score.

• The Tool contains a series of variables to assess the overall priority of a project.
• Divisional management assess projects against variables to complete the Prioritization Tool.

There are three groups involved in the process:

• First, Divisions, who:
  • complete the Prioritization Tool
  • assess their project against the variables in the Tool using drop-down fields. The Tool assigns the project a Priority Score
  • send a completed Prioritization Tool and a completed Form 2350 – Investigational Planning and Report (Operational Plan) to the FP Assessment Unit for review
• Next, the FP Assessment Unit, who:
  • is led by an Inspector to support Committee governance and the prioritization cycle
  • reviews the submitted Tool
  • completes the Tool and assigns a Priority Score
  • determines if the project is ready for presentation to the Prioritization Committee
  • assigns Tier 8 to rejected/not prioritized projects or sends accepted projected to the Prioritization Committee for review
• Finally, the Prioritization Committee, which:
  • is chaired by the DG FPCO
  • includes the Director/Officer in Charge level from FPCO Serious and Organized Crime, Cyber Crime and Border Integrity, as well as FP National Security. Representatives from Covert Operations, OIM, FP Strategic Policy, International Policing and the intelligence and support streams are also eligible for membership at the discretion of the Chair
  • assigns a tier level to a project to indicate the priority of addressing the criminal threat (as defined in the Prioritization and Governance of Major Projects Tool User Guide. Note that Tier 3 projects typically represent investigations within the FP mandate that are in the intelligence gathering stage)

Projects reviewed by the Prioritization Committee are assigned one of four tiers:

• Top Priority projects are assigned Tier 1
• Priority projects are assigned Tier 2
• Not a Priority projects are assigned Tier 3
• Not Prioritized projects are assigned Tier 9

Oversight and reporting requirements are determined based on the tier level:

• Tier 1 projects require significant oversight and direction from NHQ and typically must report every 14 days
• Tier 2 projects require some oversight and direction from NHQ and typically must report every 28 days
• Tier 3 projects may not require oversight and the reporting requirement is at the discretion of the Committee
• Tier 8 and 9 projects do not have oversight or reporting requirements

2.0 Objective, scope and methodology

2.1 Objective

The objective of the audit was to determine whether a sound governance framework over major projects was in place that included effective oversight bodies, clearly established policies and procedures, and mechanisms to monitor and report on key performance indicators.

2.2 Scope

The scope of the audit included the use of the Federal Policing Major Project Prioritization and Governance Tool from April 1, 2018 to March 31, 2020.

2.3 Methodology

The audit was conducted between September 2020 and July 2021. The audit team employed various techniques including interviews, documentation reviews, data analysis, and file testing.

Specifically, the audit team:

• Conducted interviews with key personnel in FPCO, FP Resource Management and FP representatives within C, E, K and O Divisions;
• Reviewed process and guidance documents related to the national prioritization process to assess the oversight, use of the Prioritization Tool, and monitoring and reporting in place;
• Performed data analysis to assess whether tiered major projects that Divisions were working on corresponded with national reporting; and
• Sampled 25 major projects tiered between April 1, 2018 and March 31, 2020 to assess compliance with the national prioritization process, as per Table 1 below.

Detailed criteria are contained in Appendix A of this report.

2.4 Statement of conformance

The audit engagement conforms to applicable standards in the Institute of Internal Auditor's International Professional Practices Framework and the Treasury Board of Canada Directive on Internal Audit, as supported by the results of the quality assurance and improvement program.

3.0 Observations

3.1 Oversight Mechanisms

National prioritization process

The audit found that FP has established a national prioritization process in support of governance to determine how best to allocate resources to the most important criminal threats.

Relating to what was earlier identified section 1.1, the Prioritization and Governance of Major Projects Tool User Guide states that overall discretion remains with each Division concerning when a major project should be submitted through the national prioritization process. In accordance with the process guidance, consideration should be given to submitting a project for prioritization as soon as it meets the threshold of a Federal major project.

A Terms of Reference was established to provide the Prioritization Committee with authority to assign a tier for each major project and the reporting requirements for governance and oversight. As of August 2020, Divisional FP Criminal Operations Officers (CrOps) and their teams attended Committee meetings when their projects were being considered. This allowed Divisional representatives to provide recent updates on their projects. The results of the meetings included:

• assigning an FPCO Reviewer from the applicable FPCO program to the project as part of oversight (or an FP National Security Reviewer for National Security projects)
• maintaining Records of Decision (RODs) to note the assigned project Priority score, tier, reporting requirements, and a brief reason for the tiering decision
• assigning a Tier (1, 2, 3) to the major project for the Division to pursue the investigation and document progress in the Police Reporting and Occurrence System (PROS) ^{End note 3} or another RCMP approved records management system. A request to re-prioritize a project can be made if there is a major change to the investigation

The RCMP Infoweb guidance states that it is expected that Tier 1 and 2 projects will be investigated at the discretion of the Division CrOps, but Division management may decide to advance a Tier 3 project for specific divisional reasons. Divisions are ultimately responsible for managing both their investigative resources and their priorities in conjunction with NHQ FP priorities.

While FP has established a national prioritization process for major projects, the audit noted some areas for enhancement. The Prioritization Tool requires Divisions to indicate whether they have consulted with the FPCO program Director or Reviewer. The audit's review of the sample of major projects identified that Divisions did not consult the program for most (18 of 25) of the projects. As a result, the FPCO Reviewers had limited engagement with a Division prior to a project being submitted to the FP AU. Although the Division has discretion to determine when a project should be submitted for prioritization, Divisional interviewees informed the audit team that some investigations were at an advanced stage prior to being submitted for tiering. As a result, Divisions may work on their priorities without the benefit of program engagement.

The status of tiered projects already being worked on by a Division is not provided to the Prioritization Committee when they are reviewing a new project. The Prioritization Committee reviews a project without consideration of the current workload of tiered projects in a Division or how a newly tiered project will impact the resource allocation for all ongoing tiered projects.

Review and validation of major projects

The audit found that FP has established a validation process to support the review of all major projects. This includes steps for the FP AU to:

• review each completed Prioritization Tool and Operational Plan submitted by a Division. These are projects that the Divisional FP CrOps has approved and already decided to work on in the Division
• independently complete a second Prioritization Tool for the project, which generates a second Priority score. This score reflects the overall importance of the project to FP priorities
• decide whether the project is ready for submission to the Prioritization Committee in consultation with the respective FPCO program. If it is determined that a project is not ready or does not meet the major project threshold, the project is assigned Tier 8 by the FP AU and the Division is informed why it will not be submitted. Division management can decide if they will continue to work on this investigation.
• present both the Division and FP AU versions of the completed Prioritization Tool and the Operational Plan to the Prioritization Committee for projects determined to meet the threshold

The audit did not find evidence that the FP AU consults with the Division prior to the Committee meeting when their Prioritization Tool assessment differs with the Division's. Although the Committee reviews both Prioritization Tools, the FP AU Priority score is reflected in the ROD for the tiered project. The audit analysis of the scores from the sample of projects indicated that the FP AU made changes to 22 of 25 Prioritization Tool scores without consulting the Division. There were 17 projects where the FP AU reduced the Division score, 5 projects where the score was increased and 1 project where the score was the same. The other 2 projects were rejected by the FP AU as Tier 8 and were not scored.

Given the difference in the FP AU and Division scores, there is an opportunity for the FP AU to discuss the differences of their assessment to ensure that the Divisions use the Tool in a consistent manner. There is also an opportunity to educate users on the application of the Tool.

Analysis of scoring

The audit found that the Priority score from the Prioritization Tool did not correlate with the project tier. Analysis of the 153 major projects tiered between April 1, 2018 and March 31, 2020 identified that there was a wide variation in the Priority scoring results for projects within a tier. Figure 2 below indicates that Tier 1 project scores ranged from 430 to 2680, Tier 2 ranged from 730 to 2400, and Tier 3 ranged from 330 to 2050.

It would be reasonable to expect that scores would generally descend from high to low across the priority tiers. However, Figure 2 indicates that the range of scores were similar for all tiers with significant overlap. For example, the minimum score assigned to a Tier 1 project was 430 yet the minimum score for a Tier 2 project was 730, which exceeded the score assigned to some Tier 1 projects. Tier 8 projects were rejected as a priority yet the minimum score was 1500, which exceeded some Tier 1, 2 and 3 project scores.

Although the assignment of the tier is based on the judgment of Committee members, large variations in the approved Priority score for the same tier can lead to Divisions losing confidence in the process. Divisional interviewees noted that not all relevant information is included in the Operational Plan due to sensitivities or disclosure requirements, which could contribute to a lower Priority score and may result in a decision that the project is not ready to be tiered. As of August 2020, Divisional CrOps can provide additional information at Committee meetings. Divisions indicated that they would continue to work on projects based on their Divisional priorities regardless of the FP AU or Committee decision. The risk of this increases when the prioritization process is perceived by a Division to be inconsistent.

Conclusion

A national prioritization process has been established to support governance for major projects, which includes the Prioritization Tool to provide NHQ FPCO with visibility on the major projects across the RCMP. Divisions determine when they will submit a project for tiering. A validation process is in place to assess the submission and provide a Priority score for presentation to the Prioritization Committee.

Although Divisions use the Prioritization Tool, they have limited engagement with the FPCO program, resulting in Divisions assessing the priority of their project without national guidance. The Priority scores assigned through the validation process had large variations for projects within a tier and across tiers. Scoring inconsistencies may result in a loss of confidence in the tiering process and Divisions using their discretion to determine the tiered projects they will work on. The variations in score and decisions to work on a project without national governance increases the risk that the RCMP may be unable to provide a consistent rationale for how investigations were assessed using the Prioritization Tool. The Committee is not provided with an update of current tiered projects to consider how an additional project will impact a Division's workload.

Why these findings are important

A structured process to support Federal major projects assists Divisions to identify the most important criminal threats and how best to allocate their investigative resources. Having a process in place allows for a balanced assessment of projects to ensure that FP resources are being applied against the most important national and local priorities.

Divisional engagement with the FPCO program early in the prioritization process would provide guidance to the Divisions to ensure their projects meet FP priorities. The FPCO program could share information from the national context to support the Divisions in determining when a project is ready for tiering and in assessing any horizontal impacts to the project. Increased awareness and early sharing of information could contribute to more consistent scoring during the validation process and increased confidence in the prioritization process. Having confidence in the prioritization process is a key element for Divisions to use the Prioritization Tool and allocate their resources to projects that meet the FP mandate and be able to demonstrate effective utilization of resources.

Providing the status of tiered projects in a Division to the Prioritization Committee would ensure that a tiering decision considered the Division's ability to take on additional work. The Committee would have knowledge of the threats pursued in the Division to assess the most important FP priorities in the national operating context.

Recommendations

1. The Deputy Commissioner Federal Policing (DCFP) should revise the prioritization process to:
   1. strengthen the requirements for FPCO program engagement to ensure early communication between NHQ and Divisions and to share information related to the national context
   2. review the validation process to assess whether scoring is relevant to the tier assessment
2. The DCFP should communicate revisions to the prioritization process and educate users on how to use the revised Prioritization Tool to assess FP priorities.

3.2 Divisional Processes

Divisional processes

Divisions identified that they conducted weekly operational meetings to assess Divisional threats, opportunities, constraints, available resources, progress on investigations, and planning for the weeks ahead. As investigations progressed, the Divisional management determined whether they met the threshold of a major project as defined in the Prioritization Tool and directed the investigative team to prepare an Operational Plan and the Tool.

The audit found that Divisional management within the chain of command reviewed an investigation and FP CrOps approved it as a major project. Investigators in C, E and K Divisions completed the Prioritization Tool themselves whereas a centralized unit within O Division completed the Tool.

Compliance with the National Prioritization Process

The Prioritization and Governance of Major Projects Tool User Guide included with the Prioritization Tool defines the process requirements for completing the Tool. Divisional interviewees informed the audit team that the User Guide provided sufficient guidance to complete the Prioritization Tool and that their roles and responsibilities were well understood.

The audit found that files were generally compliant with key prioritization process requirements provided in the User Guide. The results of audit testing of a sample of 25 major projects are indicated in Table 2 below.

The audit team noted a few areas where projects were not in compliance with the stated requirements. Specifically:

• One project did not have evidence within PROS that the Prioritization Tool was completed.
• 24 projects did not have information on the resources that were currently in use for the investigation. Divisions are required to enter the estimated number of resources for the investigation in the Prioritization Tool and identify resources used in the current phase in the Operational Plan. Although the Operational Plan identifies key roles such as Primary Investigator, Team Commander, Major Case Management and File Coordinator it does not contain a section to provide all resources involved in the current phase of the investigation. Only one project identified the resources that were required within the Operational Plan.
• 16 projects did not use the current Prioritization Tool at the time of submission even though the current version of the Prioritization Tool is available on the Infoweb.

In addition to the process requirements, re-tiering is advised if there is a major change to an investigation. The audit found that two projects in the audit sample were submitted for re-tiering but the objectives of six additional projects were changed without evidence that the need for re-tiering was discussed with the FPCO Reviewer. Divisional interviewees noted that there was no incentive to submit a project for re-tiering.

Analysis of tiered projects in divisions

The audit found that C, E, K and O Divisions had informal processes for ranking their prioritized major projects. The Divisions stated this was completed on an ongoing basis through daily updates and weekly meetings which are intended to track progress and plan work for the weeks ahead.

Interviewees stated that discussions with Divisional management took into consideration the threats and opportunities to address emerging priorities and optimize resource allocation, and assessed potential return on investment and viability of the investigation. Additionally, tiered projects were given priority access to resources as needed, based on the approved Operational Plan. However, Priority scores assigned to the major projects by either the FP AU or the Division were not used by any of the Divisions in ranking the priorities among their tiered projects. Divisions relied on their own informal process.

Audit analysis results

The audit requested C, E, K and O Divisions to provide a list of prioritized major projects that were worked on during the audit scope period to assess if Divisional priorities aligned with the national approved prioritized projects. The audit team compared the lists to the information provided in the Projects and Prioritization Dashboard (PPD), which is a database accessible through an online portal that contains information on all tiered projects across the RCMP.

The audit found discrepancies between the priorities that Divisions stated they were working on and the priorities that were tracked in the PPD as follows:

• Divisions were working on fewer tiered projects than listed in the PPD as per Table 3 below. These projects were considered open investigations or were in the court system. There were 149 'Open/Court' tiered projects in the PPD that the Divisions were not working on.

• Further analysis of the 149 variances in 'Open/Court' tiered projects showed that 44% were Tier 1 projects and 18% were Tier 2 projects, as per Table 4 below.

The audit noted data integrity issues with the PPD, which contributed to the discrepancies. The audit also found that a process has not been established to require Divisions to update the status of their tiered projects to ensure that the PPD is accurate. Divisions have the most current status on their tiered projects, which is not updated on a regular basis in the PPD. As a result, the audit could not determine whether Divisions were not working on projects identified in Tables 3 and 4 or reprioritizing projects due to operational reasons, or if the project had been closed or if the PPD had inaccurate information.

Divisions also reported they were working on 31 projects that had a 'Closed' status in the PPD, which included 8 [***] and 22 [***] projects as noted in Table 5 below. As a result of the data integrity issues and lack of current status on tiered projects from Divisions, the audit could not determine whether the PPD reports of 'Open' projects were inaccurate or if the Division had previously provided incorrect information.

Conclusion

Divisions determine when an investigation is considered a major project and they generally complied with the requirements of the national prioritization process. An informal ranking approach is used within Divisions to reprioritize tiered projects to address changing threats and optimize resource allocation.

Although Divisional management has discretion to decide which projects they work on, the tiered projects Divisions reported working on did not align with the FP national reports. The audit found that Divisions were working on fewer tiered projects than the Committee had previously approved. This presents a risk that FPCO does not have visibility on emerging priorities or on potential resource constraints in the Divisions. Without accurate data, FPCO does not know whether Divisions are working on high priority tiered projects.

Why these findings are important

The Prioritization Committee and FPCO program management have limited awareness of the tiered projects that are being worked on by Divisions. The limited awareness poses a risk that national governance and oversight are not being exercised once the Prioritization Committee assigns a tier. This risk can undermine Committee decisions and the intent of the prioritization process and Tool.

As such, there is an opportunity for the Committee and FPCO program management to strengthen the prioritization process to provide national visibility on which tiered projects are being advanced at any given time, as well as to capture priority work that Divisions are not able to do. This information would provide guidance on reprioritizing projects as needed, ensure that resources are assigned to the highest priorities at any given time and aid in identifying potential resource gaps.

Recommendation

3. The DCFP should establish a process for all Divisions to regularly update the status of their tiered projects.

3.3 Monitoring and Reporting

Monitoring of major projects

The 2018 FPCO Governance Framework states that FPCO DGs are responsible for monitoring and overseeing all Tier 1 and 2 projects and maintaining awareness of all Tier 3 projects. As part of this oversight, the FPCO program assigns a Reviewer to a tiered project. Aside from stated guidance that Divisions should consult with the Reviewer for requests to change the reporting requirement, the audit found that neither the Prioritization Tool User Guide nor the RCMP Infoweb guidance identified the roles and responsibilities of the Reviewer. The national prioritization process does not provide requirements for how prioritized major projects should be monitored for national oversight.

Based on the results of the audit team's file review, most of the sampled projects had limited monitoring by the FPCO Reviewer. The level of monitoring varied by project but typically contained brief summaries of the Investigative Reports uploaded into PROS by the Division. There was limited evidence to indicate that the progress of the investigation was being monitored. 90% (19 out of 21) of the sampled projects with reporting requirements had some evidence of progress reported on the project's PROS file. However, only 43% (9 out of 21) of the projects consistently submitted the reports within the mandatory timeframes. There was no evidence on the PROS file to indicate a change in the reporting or that these reports were being monitored. Divisions explained that there is no consistent follow-up from the FPCO Reviewer when reports are not submitted on time.

Although the development of key performance indicators was initiated by FP in 2016, they have not been formally adopted to monitor the results of a project. FP is currently in the process of implementing Foyer, a web-based collaborative platform that may contribute to the development of performance indicators for the prioritization process. Based on the results of the sample testing, the audit team noted some indicators in the project files that could have contributed to monitoring the results of a project, as shown in Table 6 below:

Based on the indicator results from Table 6, the audit team noted the following:

• [***]
• [***]
• Sixty percent (15 of 25) of projects were not completed within the estimated duration originally identified in the Prioritization Tool. Most (21 of 25) projects estimated completing the investigation [***] but there was no evidence of monitoring to determine if the estimates were reasonable.

Reporting

The FP Situational Awareness Support Unit (SASU) within FP Resource Management (FP RM) maintains the PPD. SASU uses the PPD to produce monthly reports containing statistics on all tiered major projects The reports are shared with the Senior Executive Committee. SASU also prepares ad hoc reports as requested such as a summary of the length of investigations and briefing notes on the impact of the pandemic on tiered projects.

In addition to the PPD, there is a Project Stat Sheet developed by the FP AU, which is a spreadsheet with information on projects submitted for prioritization. The Stat Sheet is used to quickly produce any reports requested by senior management.

Information in both the PPD and the Stat Sheet is manually entered by the FP AU without data input controls or consistent quality review. An analysis of the PPD and the Project Stat Sheet of projects tiered during the audit scope period found data integrity issues that impacted the reliability of the information. Some of the discrepancies noted included:

• 153 projects were listed in the PPD but 170 projects were listed in the Stat Sheet for the same period
• 2 Tier 8 projects were listed in the PPD but 23 Tier 8 projects were listed in the Stat Sheet
• 48 discrepancies were found in projects for C, E, K and O Divisions where information was included in the Stat Sheet but not the PPD and vice versa, or where information did not match

Most of the data in the Project Stat Sheet is captured in the PPD; however, the field for 'FP Priority' is only in the Stat Sheet. This field indicates the type of major project (e.g. Money Laundering, Opioids and Precursors etc.) allowing for more detailed analysis of the prioritized projects if also included in the PPD. The 'FP Priority' field also has the option of 'Non FP-Priority'. [***] which included investigations related to drug and/or firearm importation, money laundering and bribery. The category of 'Non FP-Priority' appears to contradict the definition of a tiered project and impacts accurate reporting.

Information systems

PROS is used for storing key documents for prioritized major projects including the Operational Plan and Investigative Reports status updates. However, these documents are in PDF format and cannot be extracted for reporting or analysis. This includes key information related to the tiering process, such as 'Committee Date, Tier Level, FP Stream and FP Priority'. As a result, this information must be manually entered into the PPD, which as noted above, contributes to data integrity issues.

The information in PROS is not sufficiently accurate to support decision making for major projects. Results from the file review indicated that Divisions did not update PROS on a timely basis and information entered was sometimes inaccurate. In particular, Divisions are required to update PROS to indicate the status of the investigation. 5 of the 25 sampled projects had an inaccurate status, where the projects had been concluded by the Divisions but not recorded as concluded in PROS.

The information in PROS also does not allow for the differentiation between [***] Results from the audit testing indicated that some projects remained open but were not being pursued for reasons including a suspect being out of the country or the Division waiting for information from policing partners.

The FP RM group had previously identified data integrity issues during a 2019 trend analysis review intended to inform senior management of the average duration of tiered projects by investigative outcome. Significant data integrity issues were noted between the information in PROS and the PPD.

Conclusion

FPCO Reviewers performed limited monitoring of the status of major projects which decreases FPCO's ability to exercise governance over the prioritization process. Reports were provided to senior management on the tiered projects. However, the requirements for monitoring major projects are not included in the national prioritization process. Performance indicators to monitor the results of major projects have not been formally adopted. Reports are created by manually entering data without quality control to ensure accuracy. PROS also has data integrity issues and system limitations for extracting key information. As a result, accurate data is not available to support national monitoring of project progress. There is an opportunity to enhance the PPD that will permit the storing and extracting of key information for reporting purposes, which would also support a performance measurement regime for the prioritization process.

Why these findings are important

Monitoring the tiered major projects would ensure that FP could assess the likelihood of a major project meeting its stated goals. Without active monitoring, there is a risk that the Prioritization Tool will not meet its intended purpose of providing governance and oversight for the duration of a project.

Adopting performance indicators would allow FP management to assess if a major project was achieving the expected results for the resource investment. Having regular reporting that contains accurate and timely data to assess and monitor the national results of major projects would enable evidence-based decision-making.

Having an information system that provides accurate information quickly would support timely analysis and evidence-based decision-making. If systems are not updated regularly and in a timely manner, FP senior management may receive information which can negatively impact decision making and credibility as a result of inaccurate reporting.

Recommendations

4. The DCFP should enhance the prioritization process to provide project information that will increase national visibility on the advancement of tiered projects, including:
   1. validating data inputs to ensure accuracy and completeness in both PROS and the PPD for tiered projects
   2. determining the reporting needs for NHQ and Divisions and providing reports on the progress of major projects to support monitoring and governance
5. The DCFP should establish the roles and responsibilities for the national monitoring of the prioritization process to ensure that oversight is provided for the duration of a tiered major project, including the need to re-tier or reprioritize a project due to changes.

Appendix A – Audit objective and criteria

Objective

To determine whether a sound governance framework over major projects was in place that included effective oversight bodies, clearly established policies and procedures, and mechanisms to monitor and report on key performance indicators.

Criterion 1

Effective oversight mechanisms for the Federal Policing Prioritization and Governance of Major Project Tool have been established and clearly communicated.

Criterion 2

Divisional processes are established and are functioning as intended for the prioritization of major projects.

Criterion 3

Monitoring and reporting mechanisms are in place to assess the performance of the national prioritization process.

Appendix B – Management action plan

The MAP makes reference to FOYER, an online environment hosted on Sharepoint that facilitates and standardizes the work of FP employees by guiding them through operational documents and tools. FOYER is not a records management system (RMS); rather, it provides access to the Occurrence Triage Aid (OTA), Pandemic Assessment & Response Tool (PART), the updated Investigation Planning and Report form (2350) and training resources. The Major Project Prioritization Tool and the Project and Prioritization Dashboard will be added to the FOYER platform in the near future. It is accessed via VPN connection to Cisco AnyConnect Secure Mobility.

Date modified:

2022-05-26