Performance Measurement, Monitoring and Evaluation

Measurement, monitoring and evaluation are essentially three different approaches used to collectively assess the implementation, performance, results and impacts of programmes.¹ These activities enable us to meet performance accountability requirements and establish whether programmes are operating as intended or whether some adjustment is needed.² Each of these three activities has a particular focus, purpose, perspective and use, although there is overlap in some places.

Performance Measurement

Performance measurement involves collecting data and information about aspects of a programme's performance to report to external stakeholders.  It has a financial and managerial orientation, and is broadly aimed at promoting transparency to stakeholders.³ Performance measurement is largely concerned with data and information about the amount of service provided to programme participants, service completion levels and outcome attainment, along with matters of cost effectiveness and cost efficiency.⁴ Reporting against MSD's Output Plan is a performance measurement activity. I would also add due diligence testing to this category of activity, conducted prior to an organisation being awarded a contract, although debatably it may also fit within monitoring.

Monitoring

Monitoring is derived from the Latin word 'monere', which means 'to warn'.⁵ Monitoring focuses on programme operations during the delivery phase, providing data and information so those responsible for the programme(s) can track and adjust activity as needed.  Because monitoring involves comparing what is actually happening with what was intended or expected, a clear articulation of expectation during the programme development phase is critical.  Common monitoring questions relate to the type and number of participants, the type and  amount of assistance being provided to participants, and results being achieved (outputs and outcomes).⁶

Monitoring for the purpose of programme management, as opposed to programme evaluation, would also include areas like risk management and incident reporting.  This kind of activity needs to occur throughout the programme delivery phase.

Evaluation

There are different types of evaluation, each with a particular aim or purpose, focusing on particular aspects of a programme, and conducted during a particular phase of a programme's lifespan.⁷ The type of evaluation carried out ultimately needs to reflect the information needs of decision makers as well as pragmatic issues around feasibility.⁸ Evaluations should help to answer the following questions:

• what kinds of clients
• experiencing what types of issues or problems
• receiving what type and volume of services
• get what results
• at what costs?^{9 10}

An alternative set of questions is presented by Eccles and Gootman, who conceptualise the key questions as follows:

• is the theory of the programme that is being evaluated explicit and plausible
• how well has the programme theory been implemented in the sites studied
• in general, is the programme effective and, in particular, is it effective with specific sub-populations of young people
• whether it is or is not effective, why is this the case
• what is the value of the programme
• what recommendations about action should be made?¹¹

A key focus for programme evaluations is the identification of programme outcomes and impacts.  Programme impact is a very precise, and often misunderstood, concept involving the establishing of a cause-and-effect relationship where a programme, rather than any other factor, is found to be the reason for the programme's outcomes being achieved.  Information about outcomes and impact is needed to help policy makers and planners make decisions about programme value or worth.¹²

It is common but poor practice to wait until late in the planning process, or until the programme is underway, before considering evaluation requirements.  It is extremely difficult to evaluate a programme if the evaluation begins as the programme is ending.¹³ Evaluation needs to be underpinned by good data and good monitoring, and in many cases data cannot be collected retrospectively. The late consideration of evaluation requirements, or the late involvement of evaluators, is often what ultimately causes evaluations to fail to deliver important information. This is a costly, frustrating and avoidable situation.

Management Information System

To support performance measurement, monitoring and evaluation, organisations need a system of collecting and managing their data and information, generally termed a management information system.  This broadly encompasses the specific tools that are used to capture and manage data and to generate reports, and the processes that sit around these activities.  Regardless of the type of system and tools used, every organisation is confronted with three issues:

• what questions do we need the system to answer
• what data elements need to be included in the system in order to answer those questions
• what reports does the system need to generate?¹⁴

The extent to which these issues are thought through will largely determine the quality, relevance and general adequacy of subsequent performance management, monitoring and evaluation activities.

Footnotes

¹ Kettner et al, 2008, op cit

² Ibid

³ Ibid

⁴ Ibid

⁵ Ibid

⁶ Ibid

⁷ For further discussion, see for example, Rossi, P. H., Lipsey, M. W. and Freeman, H. E (2004), Evaluation: A systematic approach (7th Ed). Thousand Oaks, CA: Sage Publications

⁸ Ibid

⁹ Ibid

¹⁰ The consideration of cost, in terms of efficiency and effectiveness, can sometimes sit outside the evaluation process within evaluations of government-funded programmes

¹¹ Eccles and Gootman, 2002, op cit

¹² Ibid

¹³ Cunningham et al, 2008, op cit

¹⁴ Ibid