Evaluating Research Impact on the UK Economy

1. Assessing Research Impact Dr Ian McMahon

2. Definition for the Research Excellence Framework [UK] An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia I ncludes an effect, change or benefit to: The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding Of an audience, beneficiary, community, constituency, organisation or individuals In any geographic location whether locally, regionally, nationally or internationally E xcludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

3. Economic Social Public Policy and services Health Cultural Quality of Life Environment

4. Advocacy Accountability to taxpayers Funding allocation

5. Demonstrate the importance of university research increasing productivity and wellbeing engaging effectively with current and future national and global challenges. focus on showcasing or measuring the societal benefits from research,

6. Provide assurance to Government and general public of the return from their investment in research Increasingly important in a tight fiscal government environment where returns on investment in research will need to be demonstrated in terms of environmental, economic and social impact.

7. UK Research Evaluation Fund Australia possibility of funding some research block grants

8. Measuring impact is not easy Attribution or contribution Time lags between research and impact Research user engagement Transaction costs of impact assessment That some impact is negative That all research must have impact That impact is only economic Demonstrating impact in the arts and humanities

9. That impact cannot be measured That the requirement to demonstrate it distorts academic behaviour That it takes time for impact to happen That the expectation of impact is a threat to academic freedom That impact will become an industry That measures will become targets

10. Research Quality Framework preparation in Australia 2005 - 2007 Excellence in Research for Australia 2010, 2012 Impact and UK REF pilot in 2010, full REF in 2014 RAND Corporation: Capturing Research Impacts A review of international practice. (for HEFCE) OECD projects 5% funding to demonstrate impacts Economic Impact of Research & Commercialisation at Leiden University.

11. Research Excellence Framework (REF) 2014 requires impact case studies (1 per every 10 academics submitted) and an impact strategy for each unit of assessment submitted. Impact judged on significance and reach. http://www.ref.ac.uk REF will be used at allocate UK Quality-related research block grant funding. Research Councils UK (including Arts and Humanities Research Council) requires applicants to submit a pathways to impact statement

12. Australian Government 2011 Focusing Australias Publicly Funded Research Review recommended a feasibility study on possible approaches for assessing the wider benefits arising from publicly funded research. 2013 New Government with greater emphasis on research with immediate benefit

13. Options for university-based research impact assessment To provide evidence for public investment in research To increase incentives for universities/end-user engagement improve understanding of pathways between research and innovation to inform policy. Findings: feasible to introduce two new research impact mechanisms 1. University assessment based on case studies and metric data, 2. integrating data from government, research agencies and universities.

14. Research skills for an innovative future Smarter Manufacturing for a smarter Australia Review of Collaborations between the Public and Private Sectors Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People The Place of Science in Policy Development in the Public Service

15. Excellence in Innovation for Australia The Conversation: public engagement metrics IP Australia: patent-based metrics University of Melbourne: STAR metrics Higher Education Funding Council of England : Research Excellence Framework Research Councils UK : pathways to impact. UNSW: income from Technology Transfer Cooperative Research Centres: Impact Study CSIRO: Impact 2020

16. Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia. Impact Measurement and operational principles. Respect the diversity in research disciplines/sectors in demonstrating research impact.

17. There are compelling stories that need to be told of research impact arising from research at Australian universities. EIA arose out of a commitment shared by the ATN and the Go8 in the national interest - to produce research that is both excellent in and of itself and also benefits the broader Australian community. The excellence of research produced by Australian universities has been assessed by the Excellence in Research for Australia (ERA) EIA seen as being potentially complementary to ERA in assessing the impact of research

18. A trial assessing the impact of university research by case studies Conducted over 2 nd half of 2012 finished yesterday 12 universities, 162 case studies across Defence, Economic Development, Society, and Environment 7 assessment panels 70% non-academic membership Main conclusions: Can assess impact across a broad range of disciplines using case studies Universities produce high-impact research

19. RQF preparation in Australia 2005 - 2007 ERA: Trial 2009, v1.0 2010, v2.0 2012 Recognise work in UK REF pilot in 2010, full REF in 2014 RAND Corporation: Capturing Research Impacts A review of international practice. (for HEFCE) OECD projects 5% funding to demonstrate impacts DIISRTE feasibility study into assessment of impact ATN/Go8 Symposium in November, 2011

20. 12 universities ATN: UniSA, Curtin, UTS, RMIT, QUT Go8: UQ, UNSW, Melbourne, UWA Newcastle, UTas, CDU VCs, DVCs-R, Research Directors + many more Advisory Board: Phil Clark, Robin Batterham, Patricia Kelly, Laurie Hammond, Craig Roy, David Sweeney + uni representatives 75 assessment panellists 53 non-academics including from Rio Tinto, Chevron, IBM, Microsoft, Salvation Army, CSIRO, Deloitte, KPMG, Australian Government supporting assessment panel process Lots of researchers

21. 1. That the development of the EIA should look to leverage the work performed in developing the UK REF where appropriate 2. That the EIA examine retrospective impact rather than prospective impact 3. That there be no explicit excellence threshold for submissions, but the impact should show a link to underpinning research 4. That the EIA take a verifiable approach to evidence in submissions 5. That impact will be primarily assessed by means of case- studies for all disciplines. 6. That research underpinning a submitted impact should be attributed to the university (ies) at which the research was generated

22. Four broad sectors, Defence, Economic Development, Society, Environment Time period impacts from 1 st January 2007, underpinning research from 1 st January, 1992 Unit of evaluation the impact itself, not discipline, academic unit or individual A maximum of five case studies for each of the four Sectors (i.e. 20 in total) A minimum of two case studies in three of the four Sectors (i.e. 6 in total)

23. Case Studies on prescribed templates 15 pages Information sought on Context Details of the impact Underpinning research and research publications submissions to draw the link to research quality Income and other relevant metrics Validation material available People involved in research and impact

24. Each case study rated A-E based on reach & significance of impact A = outstanding impact B = very considerable impact C = considerable impact D = recognised impact E = limited impact 7 assessment panels based on broad SEO code categories o 1 x Defence o 2 x Economic Development o 2 x Society o 2 x Environment

27. 1. There are compelling stories that need to be told of research impact arising from research at Australian universities. 2. Using a case study methodology with SEO codes to assess research impact is applicable as a way forward to a national assessment of research impact with provisos 3. The use of expert Panels for the impact assessment process, with significant input from representative external panel members , is applicable for a national impact assessment process. 4. The range of impact ratings was wider than anticipated. 5. Many case studies put forward cases based on prospective impact rather than demonstrated impact

28. 6. The link between impact and good quality research was able to be identified. 7. It was appropriate to assess impact using the industry- based grey literature. 8. Panel Chairs all commented on the time involved assessing case studies 9. Universities reported challenges in putting together case studies.

29. High quality research carried out in Australian universities has had enormous benefits for the health, security, prosperity, cultural and environmental wellbeing of Australia, the region and the world. Approximately 87% of the case-study assessments demonstrated at least considerable impact. The case-study method adopted in the Trial enables these benefits to be communicated to governments, business, and the community. The combination of end-user assessment informed by expert advice has made it possible to verify that the underpinning research has contributed to the outcomes.

30. 1. It is possible to assess research impact across a broad range of disciplines. 2. The case study approach can provide a compelling narrative of the impact of research. 3. Research impact could be assessed against an outcomes based system of classification such as the ABS SEO codes with provisos 4. Expert Panels comprising a majority of end-user stakeholders are able to assess research impact. 5. Development of an impact component of any broader research assessment exercise would require further consideration of the number of case studies to be submitted.

31. http://www.atn.edu.au/eia/inde x.htm www.go8.edu.au/eia

32. Review of Trial documentation A critical review of 10 case studies An assessment of Panel scoring Surveys of ATN universities Findings A success but could be improved Consider contribution as an assessment criteria A need to create a culture of impact Scale up issues Amount of money needed for national assessment Liked concise documentation, SEO Codes Engage with non-Trial universities

33. Review of EIA BY RAND (Europe) Positive outcomes from Trial (researchers perspective) Understanding the impact/effect of their research Ability to adapt material for other purposes Appreciating need to collect ongoing evidence Consolidating networks through re-establishing contacts with clients and end-users

34. discussion paper on assessing the wider benefits arising from university-based research released June 2013. The National Research Investment Plan actions to maximise public benefits that arise from publicly funded research Case studies vs metrics Some universities starting Impact stocktakes New Government Would it result in extra funding or help maintain existing funding?

35. Ian McMahon Director, Research Group of Eight ian.mcmahon@go8.edu.au