Evaluating the outcome of a research project in an annual report is difficult because many projects run over multiple years and there is a period of time between when R&D is undertaken, completed and then adopted by end users as to when the total value of the investment is realised.

The time scale can also vary depending on the activity undertaken. While there can be an instant impact from a project — resulting in change of practices or management arrangements for example — the total outcome may take time to accrue and that can only be measured when looking back.

The FRDC has put in place metrics to anticipate potential value (ex ante) and a formal measurement process to evaluate benefit cost (poste ante), which aligns with the Council of Rural Research and Development Corporations (CRRDC) evaluation framework.

The first population of projects was defined in January 2011 and projects were placed into clusters. Where there were high numbers of projects, these divided more specific groupings and this resulted in a total of 25 clusters across the 14 themes being assessed.

Evaluation of the first eight clusters was completed in October 2012 and the evaluation report provided to FRDC. The second round of evaluation was undertaken between July 2013 and December 2014 with nine clusters (2015 assessments outlined below) subjected to impact assessment. The assessment used benefit cost analysis to estimate investment criteria for each cluster of projects. 

RDC impact assessment and performance reporting

The evaluation program being undertaken by the FRDC is part of the CRRDC work to collaboratively implement a framework of benefit cost analysis to evaluate R&D activities.

The FRDC assessment uses the methodology developed by the rural RDCs benefit cost framework which is based on the work of the Department of Finance in Introduction to Cost-Benefit Analysis and Alternative Evaluation Methodologies, and subsequent discussions with the department to refine the methodology.

Generating and documenting evidence of impact and demonstrating performance of the RDCs as a collective is also a key objective for the CRRDC.

CRRDC cross-portfolio evaluation

Over 2015-16 the FRDC, along with the other RDCs in the CRRDC, has funded a consortium of evaluation consultants to update the evaluation analysis to cover the period 2010–15 and develop an updated framework for ongoing monitoring and reporting across the RDCs. This builds on the work done by each RDC, but it is the first major cross-portfolio review in the past five years.

A stocktake of current evaluation activities of rural RDCs for the CRRDC was conducted by Lange Consulting & Software in the second half of 2015. The stocktake aimed to document existing evaluation processes and practices and inform the development of a high-level joint RDC performance framework.

From the recommendations, an opportunity was identified to standardise practices, terminology, measurement of intangibles, metrics and performance measures against Australian Government strategic research priorities for benefit cost analysis. This will better enable aggregation of performance data and provide improved visibility of the cost effectiveness of research programs across each RDC’s portfolio. 

2015 Portfolio Assessment 

The nine clusters evaluated in this second round comprised:

 

2012 Portfolio Assessment

BCA 1- Biosecurity and Aquatic Animal Health
BCA 2- Extension and Adoption
BCA 3- Habitat and Ecosystem Protection Part A
BCA 4- Habitat and Ecosystem Protection Part B
BCA 5- Innovation Skills Part A 
BCA 6- Innovation Skill Part B
BCA 7- Leadership Development

 

2010 Portfolio Assessment

FRDC completed (16 June 2010) Benefit Cost-Analysis on on 18 R&D subprograms comprising over 200 projects which will deliver a total R&D portfolio assessment. This analysis will provide data for which the FRDC and its partners can optimise future investment decisions based on an adaptive management approach.

The 2010 economic evaluation of the 18 randomly selected clusters of FRDC investment across three programs has found that the average return to FRDC investment is 5.6 to 1.
 
The research projects completed or substantially completed between July 2002 and June 2007 were assessed, and this population was divided into 32 clusters of investment by grouping projects into homogenous subject areas. It was from these 32 clusters that the 18 clusters analysed were randomly selected. The total value of FRDC funding for this population was $92.76 million (nominal). The FRDC investment in the eighteen clusters analysed totalled $51.63 million therefore represented 56% of FRDC’s total investment (in nominal terms) in the population. The 18 report from each assessment can be accessed from the list below.
 
The BCA Summary Report 29 July 2010 provides the detail and summation of the total portfolio assessment.
 
BCA Executive Report 29 July 2010
BCA 1 – Diet Development 16 June 2010
BCA 2 – Abalone 16 June 2010
BCA 3 – SBT 16 June 2010
BCA 4 – Salmon 16 June 2010
BCA 5 – Environmental impacts 16 June 2010
BCA 6 – ESD and management 16 June 2010
BCA 7 – MPAs and Spatial 16 June 2010
BCA 8 – Animal health and pests 16 June 2010
BCA 9 – Enhancing wild catch 16 June 2010
BCA 10 – Environmental aquaculture 16 June 2010
BCA 11 – Food safety 16 June 2010
BCA 12 – Marketing 16 June 2010
BCA 13 – Workplace safety 16 June 2010
BCA 14 – Workshops and Conferences 16 June 2010
BCA 15 – Strategic Planning 16 June 2010
BCA 16 – Population dynamics AFMA 16 June 2010
BCA 17 – Population dynamics QLD 16 June 2010
BCA 18 – Population dynamics NSW 16 June 2010
 

Developing a Benefit-Cost Evaluation Framework

Individually, Research and Development Corporations (RDCs) use a range of evaluation approaches to report value to stakeholders and to provide recommendations and guidelines for ongoing and future investment. In 2007 the RDCs agreed to work together to measure and report on the overall return on R&D investment.
The Council of Rural Research and Development Corporations’ Chairs (CRRDCC) prepared the evaluation framework, methodology and approach for this work. These were reviewed by key economic agencies of the Australian Government including the:
  • Treasury
  • Department of Finance
  • Department of Agriculture and Water Resources
  • Productivity Commission
  • Australian Bureau of Agricultural, Resource Economics and Sciences.
Following agreement being reached on the framework the 15 RDCs engaged independent economic consultants to undertake the first phase of evaluations. The results for the first round were released in November 2008. For more information, please see the CRRDCC Summary of the Evaluation Report [PDF].
 

2009 Initial 'Hero Project' Assessment