Evaluating the outcome of a research project in an annual report is difficult because many projects run over multiple years and there is a period of time between when R&D is undertaken, completed and then adopted by end users as to when the total value of the investment is realised.
The time scale can also vary depending on the activity undertaken. While there can be an instant impact from a project — resulting in change of practices or management arrangements for example — the total outcome may take time to accrue and that can only be measured when looking back.
The FRDC has put in place metrics to anticipate potential value (ex ante) and a formal measurement process to evaluate benefit cost (poste ante), which aligns with the Council of Rural Research and Development Corporations (CRRDC) evaluation framework.
The first population of projects was defined in January 2011 and projects were placed into clusters. Where there were high numbers of projects, these divided more specific groupings and this resulted in a total of 25 clusters across the 14 themes being assessed.
Evaluation of the first eight clusters was completed in October 2012 and the evaluation report provided to FRDC. The second round of evaluation was undertaken between July 2013 and December 2014 with nine clusters (2015 assessments outlined below) subjected to impact assessment. The assessment used benefit cost analysis to estimate investment criteria for each cluster of projects.
RDC impact assessment and performance reporting
The evaluation program being undertaken by the FRDC is part of the CRRDC work to collaboratively implement a framework of benefit cost analysis to evaluate R&D activities.
The FRDC assessment uses the methodology developed by the rural RDCs benefit cost framework which is based on the work of the Department of Finance in Introduction to Cost-Benefit Analysis and Alternative Evaluation Methodologies, and subsequent discussions with the department to refine the methodology.
Generating and documenting evidence of impact and demonstrating performance of the RDCs as a collective is also a key objective for the CRRDC.
CRRDC cross-portfolio evaluation
Over 2015-16 the FRDC, along with the other RDCs in the CRRDC, has funded a consortium of evaluation consultants to update the evaluation analysis to cover the period 2010–15 and develop an updated framework for ongoing monitoring and reporting across the RDCs. This builds on the work done by each RDC, but it is the first major cross-portfolio review in the past five years.
A stocktake of current evaluation activities of rural RDCs for the CRRDC was conducted by Lange Consulting & Software in the second half of 2015. The stocktake aimed to document existing evaluation processes and practices and inform the development of a high-level joint RDC performance framework.
From the recommendations, an opportunity was identified to standardise practices, terminology, measurement of intangibles, metrics and performance measures against Australian Government strategic research priorities for benefit cost analysis. This will better enable aggregation of performance data and provide improved visibility of the cost effectiveness of research programs across each RDC’s portfolio.
2015 Portfolio Assessment
The nine clusters evaluated in this second round comprised:
2012 Portfolio Assessment
2010 Portfolio Assessment
FRDC completed (16 June 2010) Benefit Cost-Analysis on on 18 R&D subprograms comprising over 200 projects which will deliver a total R&D portfolio assessment. This analysis will provide data for which the FRDC and its partners can optimise future investment decisions based on an adaptive management approach.
The 2010 economic evaluation of the 18 randomly selected clusters of FRDC investment across three programs has found that the average return to FRDC investment is 5.6 to 1.
The research projects completed or substantially completed between July 2002 and June 2007 were assessed, and this population was divided into 32 clusters of investment by grouping projects into homogenous subject areas. It was from these 32 clusters that the 18 clusters analysed were randomly selected. The total value of FRDC funding for this population was $92.76 million (nominal). The FRDC investment in the eighteen clusters analysed totalled $51.63 million therefore represented 56% of FRDC’s total investment (in nominal terms) in the population. The 18 report from each assessment can be accessed from the list below.
Developing a Benefit-Cost Evaluation Framework
Individually, Research and Development Corporations (RDCs) use a range of evaluation approaches to report value to stakeholders and to provide recommendations and guidelines for ongoing and future investment. In 2007 the RDCs agreed to work together to measure and report on the overall return on R&D investment.
The Council of Rural Research and Development Corporations’ Chairs (CRRDCC) prepared the evaluation framework, methodology and approach for this work. These were reviewed by key economic agencies of the Australian Government including the:
- Department of Finance
- Department of Agriculture and Water Resources
- Productivity Commission
- Australian Bureau of Agricultural, Resource Economics and Sciences.
Following agreement being reached on the framework the 15 RDCs engaged independent economic consultants to undertake the first phase of evaluations. The results for the first round were released in November 2008. For more information, please see the CRRDCC Summary of the Evaluation Report
2009 Initial 'Hero Project' Assessment