Join us for a REF-related 12 days of Christmas, where each day we unwrap a different aspect of the REF.
On the first Day of Christmas, HEFCE gave to me…A Research Excellence Framework in 2021
On the second Day of Christmas, HEFCE gave to me… 2 Impact case studies per Unit of Assessment
On the third Day of Christmas, HEFCE gave to me… 3 months from acceptance for Open Access compliance
On the fourth Day of Christmas, HEFCE gave to me… 4 main panels of assessment
On the fifth Day of Christmas, HEFCE gave to me…5 maximum outputs
On the sixth day of Christmas, HEFCE gave to me…6 rules for staff inclusion
On the seventh Day of Christmas, HEFCE gave to me… 7 key tasks for Sub panel chairs
On the eigth day of Christmas, HEFCE gave to me…8 members on the Equality and Diversity panel
On the ninth day of Christmas, HEFCE gave to me…9 Responsibilities of the Interdisciplinary adviser role
On the tenth day of Christmas, HEFCE gave to me…10 UoAs before Computer Science and Informatics.
On the eleventh day of Christmas, HEFCE gave to me…11 letters in portability
On the twelfth day of Christmas, HEFCE gave to me…12 recommendations of the Stern review
A Research Excellence Framework in 2021
The Research Excellence Framework exercise, or REF, is undertaken by the four UK higher education funding bodies and coordinated by the Higher Education Funding Council for England (HEFCE). It is is one of the main ways that the quality of UK research is evaluated with the aim of identifying and rewarding excellence. Rewards come in the shape of Quality-Related (QR) block grants and positioning in a number of important Higher education (HE) league tables. The REF assesses individual HE institutions (HEIs) on the quality profile of their Research Outputs (such as journal articles, book chapters etc.), Research Impact (the economic or societal benefit of research) and the Research Environment, with the relative weighting of each element being 60/25/15% respectively. The next REF period spans from 2014 to 2020, with results announced in 2021.
For more information visit: http://www.ref.ac.uk/about/whatref/
Two Impact case studies per Unit of Assessment
A quarter of the REF quality profile comes from the Impact element of the assessment exercise. But what is impact? Impact is any auditable change your research makes in the wider world. For the purposes of the REF, changes to the economy, society, policy, culture, the environment, health, quality of life and teaching are all eligible impacts, so long as they took place within the current REF period and are evidenced appropriately! The REF assesses impact through case studies, which use a narrative approach to describe both the research and the impact generated from it. They are evaluated according to the reach (i.e. who has been impacted) & significance of the impact. To be eligible, the underpinning research must belong to the submitting HEI, and have taken place within the last 20 years (from 2000 onwards). For each Unit of Assessment, or UoA, a minimum of one case study needs to be submitted, plus one more for each 15 full-time equivalent (FTE) members of staff returned. This means every UoA must have a minimum of two impact case studies.
More information on research impact
Three months from acceptance for Open Access compliance
Open-access to research is vital for dissemination, increasing public understanding of research, research uptake and knowledge exchange. However, it is also important for the REF! In order for research outputs to be eligible for assessment in the next REF, they must be Open Access (OA) compliant. From April 2016, all journal articles and conference proceedings must be available in an open-access format within 3 months of publication to be compliant; from April 2018, the rules change to 3 months from acceptance. There are two routes to making an output open access: Green and Gold. The green route involves uploading a version of your output to an open-access institutional or subject repository; Plymouth University has its own open access repository: PEARL. The version you upload depends on the permissions granted by the publisher but will have to be, at the very least, the final peer-reviewed draft accepted for publication. The gold route involves (usually) paying a fee to the publisher to make the article open access immediately upon publication.
For more information visit the library and Higher Education Funding Council England’s (HEFCE) open access FAQ
Four main panels of assessment
The REF is interested in rewarding research whatever the discipline. However, different disciplines have many different approaches to what constitutes an output, methodology and quality. To address this the REF has four main panels of assessment: Panel A: Medicine, Health and Life Sciences, Panel B: Physical Sciences, Engineering and Mathematics, Panel C: Social Sciences & Panel D: Arts and Humanities. The main panels’ role is to ensure that standards defining research quality are consistent across all four disciplinary areas and to provide overarching leadership and guidance to the sub-panels that sit under them; the sub-panels are responsible for evaluating submissions to a given Unit of Assessment (UoA). As well as the four main panels A-D, there is also an interdisciplinary research advisory panel, which will advise on methods to evaluate interdisciplinary research, and a panel focused on Equality & Diversity.
More information on Panel Chairs
In the previous REF exercise, all academics submitted to the REF had to submit 4 research outputs, except for those with exceptional circumstances. This time around, academics can submit 5… or they can submit one (and in exeptional circumstances zero)! In order to remove the link between the individual and the number of outputs submitted, each UoA will have to submit an average of 2.5 outputs for every FTE staff member returned (as counted on a census date); individual staff can submit anywhere from 1 to 5 outputs. For example, a UoA submitting 10 staff on full time contracts will have to submit 25 outputs. Everyone submitted will have to contribute at least one output, with the remainder contributed by the submitting group There will be the option for exceptions to the rule that everyone must submit at least one output, e.g. in case of illness or parental leave.
One of the key drawbacks from previous exercises is that the REF included a select number of academics only. This caused issues for staff morale and gave a biased picture of the quality of research across the sector. REF 2021 will resolve this by including all staff with significant responsibility for research. That means that if you meet any of the criteria below you will be part of the next REF and will need to have at least one output ready to include:
- academic staff with a contract of employment of 0.2 full-time equivalent (FTE) or greater
- on the payroll of the submitting institution on the census date (31/07/2020)
- with a substantive connection with the submitting institution
- with a primary employment function to undertake either ‘research only’ or ‘teaching and research’
- For staff on ‘research only’ contracts, the eligible pool should only include those who are independent researchers, and not research assistants
- Identified as having significant responsibility for research on the census date.
Many research intensive HEIs will be able to identify staff by contractual status alone, and will be encouraged to return 100% of staff that meet the criteria above. Otherwise, HEIs will have to work with staff to identify those with significant responsibility for research, e.g. an academic may have a ‘teaching and research’ contract but be engaged primarily in knowledge exchange or scholarship, and therefore it’s not appropriate to return them. HEIs that choose this route will have to produce a code of practice and an audit trail.
Seven key tasks for Sub panel chairs
Selection for assessors is done by nominations by Learned and other societies/organisations with an interest in research. Hopefully, you will have been nominated and are even now awaiting your invitation to your chosen sub-panel!
For REF 2021 there will be 34 Sub-panels, corresponding to the different disciplinary areas (or units of assessment). Each sub-panel sits under a main panel and comprises a Chair, Deputy Chair(s), members and assessors for particular elements such as outputs and impact. The Chair has significant levels of responsibility and seven key tasks outlined below
- Develop any necessary criteria and working methods specific to the individual sub-panel, for approval by the main panel
- Explore whether to introduce discrete sub-profiles for outputs to provide increased visibility for distinct areas covered in the UOA,
- Consult on and contribute to the criteria and working methods of the group of subpanels within a main panel
- Apply agreed criteria and methods to assess submissions.
- Advise the main panel and REF team on cross-referrals of submitted material to other sub-panels and on any need for additional expertise required to assess submissions.
- Produce draft assessment outcomes for each submission and associated concise feedback for submissions.
- Advise on sub-panel membership
Sub-Panel chairs have now been appointed and will now begin the task of identifying members and developing assessment criteria, to be published in 2018
Eight members on the Equality and Diversity panel (The importance of E&D)
In REF 2014, an analysis found that some groups (e.g. women, some ethnic minorities, and disabled people), were under-represented on sub-panels, and that barriers to participation existed at all levels, from inclusion in the REF process itself to sub-panel nomination. The establishment of the REF Equality and Diversity (E&D) Advisory panel (EDAP) will ensure consideration of E&D is embedded at all stages of the REF and work towards removal of these barriers. The panel has 8 members, drawn from across the HE sector and beyond, plus a Chair, Prof Dianne Berry, and an observer, Sophie Lowry from the Scottish Funding Council. We look forward to celebrating the diversity of the panels and the breadth of expertise they will bring with them.
More information on EDAP and panel membership
Nine Responsibilities of the Interdisciplinary adviser role
Interdisciplinary research (IDR) plays a crucial role in addressing many of the complex and intractable challenges facing the world today. However, the UoA structure of the REF means that equitably assessing IDR requires special arrangements. One arrangement is the introduction of an Interdisciplinary adviser role to every sub-panel. The IDR advisor will have nine main responsibilities
- Consult on and contribute to the criteria and the associated working methods for evaluating IDR, in collaboration with the interdisciplinary advisory panel (IDAP), the network of main panel interdisciplinary members and IDR advisers across the panels.
- Advise on and provide oversight of the assessment of IDR elements in submissions
- Advising on whether outputs flagged as interdisciplinary meet the definition in the published guidance.
- Advising on the cross-referral of outputs that have been identified as interdisciplinary, where cross-referral is required.
- Moderation and monitoring of output scores for outputs that have been identified as interdisciplinary.
- Advising on the assessment of any IDR elements of Impact and Environment.
- Participating in calibration activities both within the sub-panel, and with the network of IDR advisers across the panels.
- Assess submissions, including both outputs identified as interdisciplinary and those that are not, as appropriate to their field of expertise and in accordance with the working methods set out in the panel criteria
- Engage with the network of main panel interdisciplinary members and IDR advisers across the panels to share good practice and consistency of approach as appropriate, and feed this into the sub-panel’s assessment
Other arrangements are the introduction of an IDR section in the Environment Template and an interdisciplinary identifier for outputs. The interdisciplinary advisory panel, chaired by Professor Dame Athene Donald, will provide guidance.
Ten UoAs before Computer Science and Informatics.
Quality assessment in the REF takes place through a process of expert peer review, at the subject or Unit of Assessment (UoA) level. Each UoA represents a distinct disciplinary area and falls under one of the four main REF Panels. In REF 2014 there were 36 UoAs, in 2021 there will be 34. With the main changes being that all the Engineering UoAs are now condensed into one unit (UoA12) and an additional unit for Archaeology (UoA15) has been added. Staff are assigned to the most appropriate UoA for their research but can also cross-refer if there work straddles disciplinary boundaries or is interdisciplinary in nature. For each UoA, the three REF elements, Outputs, Impact and Environment, are graded using a star-rating from zero (unclassified) to 4-star, with the overall quality profile and score for each UoA (and HEI) determined by calculating the grade-point average or GPA. HEIs can submit to any number of the 34 UoAs that best represent their research strengths. In Plymouth, we are expecting to submit to at least 17 UoAs, many of which we submitted to in REF2014.
Eleven letters in portability
Portability of outputs has been one of the most controversial aspects of the new REF rules. In the previous exercise, academics could take all their research with them if they moved HEIs. This led in part to a “REF transfer market” and HEIs buying in quality rather than investing in it. This was to the advantage of academics with high-quality outputs but to the detriment of HEIs, who lost the ‘return’ from their investment in staff and resources. To address these issues, it is likely that future iterations of the exercise will restrict portability, bringing the Outputs element in-line with the Impact element of the REF. In order to let the sector adjust to this new approach, transitional arrangements will be in force for REF 2021. If an academic is included in the census at one HEI but one or more of their research outputs were demonstrability generated at another HEI, both institutions will be able to claim and submit the same output.
Twelve recommendations of the Stern review
July 2016 saw the publication of Lord Stern’s review of the REF. The report made 12 recommendations, which have since been implemented. HEFCE and the other funding bodies are still working on the fine details, but we now have clarity on the key issues of portability and staff selection… just in time for Christmas!
| A: Outputs |
| Recommendation 1: All research active staff should be returned in the REF. |
| Recommendation 2: Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average. |
| Recommendation 3: Outputs should not be portable. |
| Recommendation 4: Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use. |
| B: Impact |
| Recommendation 5: Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment. |
| Recommendation 6: Impact must be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs. |
| Recommendation 7: Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socio-economic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching. |
| C: Environment |
| Recommendation 8: A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel. |
| Recommendation 9: That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment. |
| D: Wider context |
| Recommendation 10: Where possible, REF data and metrics should be open, standardised and combinable with other research funders’ data collection processes in order to streamline data collection requirements and reduce the cost of compiling and submitting information. |
| Recommendation 11: That Government, and UKRI, could make more strategic use of REF, to better understand the health of the UK research base, our research resources and areas of high potential for future development, and to build the case for strong investment in research in the UK. |
| Recommendation 12: Government should ensure that there is no increased administrative burden to Higher Education Institutions from interactions between the TEF and REF, and that they together strengthen the vital relationship between teaching and research in HEIs. |