Program Evaluation

Fire Prevention Effectiveness Model - Position Paper

Section III Program Evaluation

Program Evaluation Flow Chart

Section III Program Evaluation chart

3.1. INTRODUCTION

An integral and vital aspect of any fire prevention program is an evaluation of its results. This section provides some suggestions for measuring and assessing the effectiveness of programs implemented.

Implementing a program is not the end of the process. A program must be evaluated continually to ensure the best use of resources to address a fire problem in the community. The evaluation process should indicate if a program needs to be modified or whether its goals and objectives have been achieved. Achieving these will likely change the character of a community's fire problems. Therefore, an updated needs analysis may be required to determine how to maintain the improved level of fire safety and what priority should be addressed next subject to the availability of resources.

3.2. PROGRAM EVALUATION METHOD

One method of program evaluation is explained below. There are a number of components to this evaluation model. It is important to understand that these components are interrelated and therefore, all must be applied so that the program can be implemented easily, monitored continuously, and evaluated objectively. The first three components address program planning, the next three address program measurement, and the last three address evaluating program performance.

Program Planning Components

Goal(s)

A program goal(s) is the intended overall benefit of the program. It should state what the program is intended to achieve and should answer an identified need. This would include the reason(s) why a program was created and consideration of how the program will evolve in the future. A program goal is linked to the realization of the overall mission of the department.

The first step to good planning is to set realistic, achievable goals which reflect the parameters under which the program must operate. For example, the availability of fiscal or human resources to apply to a particular activity.

Objectives

Objectives are the measurable outcomes of a program that can be achieved within a specific period of time to contribute to the realization of the goal(s). The objectives should state clearly the quantitative outcomes where appropriate (e.g. 100 % of apartment buildings will be inspected by a certain date), as well as qualitative outcomes, such as raised awareness or attitude shift regarding fire prevention will occur by a certain date.

In cases where a new initiative has been launched or where results have not been previously measured for a program, the first year that the objectives are measured will be considered baseline information against which all future results will be compared.

The following are suggested objectives for programs in the three areas of fire prevention:

Inspections

1) Achieve compliance with fire safety legislation:

  • within a specified time (determined by follow-up inspections);
  • through continued compliance (determined at subsequent routine inspections);
  • through continued compliance for properties not subject to routine inspection (determined by spot checks); and
  • through voluntary compliance of properties not subject to inspection.

2) Reduce the potential impact of fire in properties through:

  • the elimination or reduction of fire hazards to reduce the size and spread of fires;
  • containment measures to limit the size and spread of fires;
  • alarm and detection measures to limit the spread of fire and provide occupants with adequate warning to respond appropriately;
  • suppression measures to limit the damage caused by fires; and
  • fire safety planning information to increase occupants' knowledge and to limit the damage caused by fires.
Public Safety Education

1) Increase the fire safety knowledge and awareness of property occupants:

  • to a defined, acceptable level;
  • appropriate to the risks for the targeted occupants.

2) Reduce the incidence and impact of fires:

  • occupants of properties prevent fire occurrences as a result of fire safe behaviour;
  • limit damage and injuries due to fire by reacting appropriately to fire incidents.
Fire Incident Evaluation

1) Gather sufficient information:

  • to allow for proper needs analysis;
  • for meaningful evaluation of fire department programs.
Activities

Activities are actions conducted to achieve the objectives and goal(s) of a program. A complete description of the key activities of the program can be developed by asking questions such as:

  • What things need to be done to achieve a certain result?
  • What fiscal, human, and physical resources are required to achieve these results?

The activities associated with each type of program are addressed in Section II.

Program Measurement Components

Costs

Both staff and direct operating costs should be included.

Staff Costs

Salaries, benefits and any other related costs for staff who carry out the activities of a program. These costs can be assessed by reviewing the activity reports of staff and determining the amount of time spent on various activities. If an activity reporting system is not in place, then a reasonable estimate of staff time can be used as long as it is clearly identified as an estimate.

Direct Operating Costs

Transportation, communications, services, supplies and equipment costs that are required to carry out the activities of the program.

Success Indicators

Information or data that is tracked and analyzed to assess the progress of reaching objectives and goals.

Success indicators should not be developed in a vacuum as they flow from the objectives and related activities. If the objectives clearly state how far, how much, by when etc., then it will be fairly easy to identify the kind of information to track and analyze to measure a program's performance.

Program Evaluation Components

Success Assessment

An analysis of a program's progress towards the achievement of expected outcomes and goals using success indicators.

A success assessment includes gathering fire loss information for a community and comparing it to previous years and to other jurisdictions. Investigating fire scenes will determine if there is a direct link between a program and improvements in fire losses. Any statistical and background information used in the analysis must be objective and stated clearly to minimize the effects of unrelated factors on the actions of a fire department. For example, the number of fires per thousand population is an objective means of measuring the rate of fire occurrence as it eliminates the impact of changes in population.

Following is an example of a sample success assessment of the previously suggested objectives for the three subject areas of fire prevention:

Inspections

  1. Determine the level of compliance with fire safety legislation by the:
  • incidence of compliance within a specified time (determined by follow-up inspections);
  • incidence of continued compliance (determined at subsequent routine inspections);
  • incidence of continued compliance by properties not subject to routine inspection (determined by spot checks);
  • incidence of voluntary compliance by properties not subject to inspection.
  1. Provide an indication that the fire safety measures implemented in the inspection program reduced the impact of fire on evaluated incidents:
  • fire hazard elimination and reduction measures reduced the size and spread of fires;
  • containment measures limited the size and spread of the fires;
  • alarm and detection measures limited the spread of the fire and provided occupants with adequate warning to respond appropriately;
  • suppression measures limited the damage caused by fires;
  • fire safety planning information received by occupants limited fatalities, injuries and damage caused by fires.
  1. Provide an indication that the inspection program was responsible, in whole or in part, for fire loss reductions. This could be achieved by thorough a fire incident evaluation of all fires in properties subject to inspection.
  2. Determine if fire incidents in the community may have been prevented as a result of the inspection program. This could be achieved conducting surveys to seek information about the fire incidents.

Public Safety Education

  1. Determine if there was an increase in fire safety knowledge as a result of the program by surveying target audiences before and after the implementation of the program. Establish a starting reference point and compare all subsequent results to it.
  2. Determine from the fire incident evaluations that an increased knowledge of fire safety of occupants as a result of the program, reduced the impact of evaluated fires.
  3. Determine if the education program was responsible, in whole or in part, for fire loss reductions by conducting a thorough fire incident evaluation of all fires.
  4. Determine if the fire incidents in the community may have been prevented as a result of the public safety education program. This could be achieved by conducting surveys to seek information about the fire incidents.

Fire Incident Evaluation

Determine whether sufficient information is being gathered through the fire incident evaluations to allow for a proper needs analysis and evaluation of the programs.

Why/Why Not Successful

Commentary should be provided on a program's successes, areas that need improvement and how they could be implemented. These comments would be based upon, but not limited to, the data contained in the success indicators and success assessment which measure a program's level of achievement. Commentary on the objectives, activities, and program costs may also be helpful as this could be used to address how resources are being used and how, or why, resources should be redeployed. Items related to the planning and execution of activities in future fiscal years should be discussed as well.

Projected Adverse Impact Without A Program

This component would highlight the negative affect on stakeholders and public safety if part or all of the program ceased to exist. A realistic analysis will provide senior management with sound information to set priorities.

Conclusion

The on-going evaluation of a department's fire prevention programs and related activities will provide the information necessary to plan properly. Meaningful evaluation results will ensure that a department is achieving its fire safety objectives and is using available resources effectively and efficiently to address the community's fire risks. Also, municipal managers may require the evaluation information when considering whether to allocate or continue funding for fire department programs.