This July, Research England (RE) and the 3 devolved funding bodies (HEFCW, SFC, & DFE) published the draft guidance on submissions for REF 2021 along with the draft panel criteria and working methods.
These hefty documents (weighing in at roughly 300 pages and 800 paragraphs between them, with 19 Appendices) flesh out many of the initial decisions on the shape of REF2021, staff selection and outputs and as such are vital reading for anyone involved in, or who will be submitted to, the REF.
Most of you, of course, will have instantly cast aside your summer reading list of blockbuster books, bodice rippers and Scandi-noirs and spent the last month poring over every sentence, but for those who haven’t – never fear, the Kaleidoscope is here!
In it to win it.
First up in our digest are the rules for staff selection. Essentially anyone employed by a HEI on the 31st July, 2020 and who has “significant responsibility for research” will be submitted. Research England have produced the handy flow-chart below to take you effortlessly through the process. In the first stage, all HEIs will identify their pool of eligible Category A staff (CatA Staff) and then they will either automatically submit everyone in that pool (low burden option) or go through a fair, transparent and rigorous consultation and selection process (high burden). A number of criteria have been defined to identify the pool of CatA staff but the primary mode of selection, which will be adopted within HEIs, will be based on the Academic Employment Function (AEF) of a staff member.
What is this AEF I hear you ask? Well, every year HEIs return data to the Higher Education Statistics Authority (HESA) on such things as staffing, research income and studentships. The AEF code is a way for HESA to identify how many academics are employed by a HEI and what their job function is: Teaching and Scholarship (AEF: 1), Research Only (AEF: 2) and Research and Teaching (AEF: 3). How these codes are assigned is decided by the individual HEI and may reflect contracts, job descriptions or work load allocations etc. At the University of Plymouth, we expect everyone with AEF: 3 to have significant responsibility for research and will be able to follow the low burden route for selection. Whether high or low burden, every HEI will have to have a code of practice for determining research independence for staff on AEF: 2. At Plymouth, we will be guided by the indicators of research independence outlined in the document guidelines (paragraph 130) :
- Leading or acting as principal investigator or equivalent on an externally funded research project
- Holding an independently won, competitively awarded fellowship where research independence is a requirement.
- Acting as a co-investigator on an externally funded research project
- Leading a research group or a substantial work package
- Significant input into the design, conduct and interpretation of the research.
Special Circumstances
Each UoA has to return 2.5 research outputs per full time member of staff (reduced proportionally for part-time staff). Everyone submitted is expected to contribute at least one research output with the maximum number for any individual being five. However, if a UoA has a lot of staff who are Early Career Researchers (ECR), or who have caring responsibilities, ill-health etc., then they can apply for a reduction in the overall number of outputs they have to return, by up to 1.5 for every member of staff with special circumstances. The Stern review noted that accounting for special circumstance in the last REF proved to be extremely burdensome in terms of administration and also in terms of the cost to academic self-esteem and professional relationships. To simplify the situation in REF 2021, HEIs can now calculate how many outputs they can reduce a submission by, through the tables below; they then need to notify their funding body. However, it becomes more complicated for those with other circumstances that might impact on their research productivity, such as for those with disabilities, chronic illness, issues related to protected characteristics etc. In these cases institutes will have to decide how those circumstances would equate to a period of absence and ask for appropriate reductions.
The guidelines also lay out the approach by which Institutions can request a further reduction of 1 (essentially taking someone’s contribution to zero). The funding bodies feel that a minimum of 1 output would be the expectation for the majority of research active staff, even with extenuating circumstances. However, in exceptional circumstances, a narrative would need to be submitted highlighting how circumstances have prevented an academic from producing any outputs. Naturally this has the potential to become a very burdensome and possibly disruptive and negative process which is why the funding bodies are consulting on their proposals (see below)
One thing to note here is that the guidelines expect all Institutions to develop and implement codes of practice that embed equality and diversity at every stage and which would be adhered to, especially when it comes to selecting outputs and requesting reductions. Therefore even if difficult circumstances need to be discussed with colleagues, the code of practice should ensure that issues are dealt with fairly and sensitively.
R&I will be working hard over the next few months to ensure that codes of practice are fit for purpose and in place both at the Institutional and Unit of Assessment (UoA) level
The three elements: Outputs, Impact & Environment
Outputs
Another useful flowchart to identify which outputs can be submitted is published in the guidelines document, that includes rules around Open Access (link to University guidance and policy) and outputs from staff who no longer work at the submitting UoA.
As well as following the steps below, we will also be producing a code of practice on Output selection that will embed Equality and Diversity into the decision making process.
In terms of eligibility, it is well worth reading through Appendix K in the submission guidelines to see a list of acceptable research outputs. If you are submitting to Panel D, then a read through Appendix C in the Panel guidelines will also be most illuminating, especially if you are a practice-based researcher. For other differences across Panels, in terms of citation data, additional information etc., you can see a summary in the section on Consultation, below.
Environment
The biggest news from the guidelines is that every HEI will have to submit an Institutional Environment Statement, which will be read alongside the UoA Environment statement. Environment statements at the HEI level will contain a number of elements: a. Institutional context, b. Institutional strategy, c. People, d. Income, infrastructure and facilities.
At the UoA level the statement will have the following elements
- Unit context, research and impact strategy.
- People, including:
– staffing strategy and staff development
– research students
– equality and diversity.
- Income, infrastructure and facilities.
- Collaboration and contribution to the research base, economy and society.
The guidelines state that institutional statements will not be formally assessed but will be available to Subpanels and might inform the grading of the UoA’s environment for supporting research and impact. R&I will be starting work on this over the coming months.
Impact
There wasn’t much that was new for impact in the guidelines – most of the decisions had already been made. Instead there was clarity on continuing case studies, the criteria and also some examples of impact, which can be found in Appendix A, Panel Criteria and Working Methods.
For case studies that were submitted to REF 2014 and have had new impact, they can be flagged as continuing if the beneficiaries and impact areas are similar and there is no new underpinning research. This will restrict the number of case studies that can be “continued” to those with underpinning research dated from 2000 onwards, as ALL case studies must meet the standard criteria around research. All Panels make it clear they want to see the strongest examples of impact and that assessment will be made only on the impact generated during this period, i.e. any reference to REF 2014 will be discounted. It could be assumed then that the flag of continued is only of interest to the funding bodies for post-assessment analysis. One thing to note here is that there is no prohibition on submitting case studies that could be based on 2014 submissions but that don’t meet the criteria for continued.
In terms of criteria, case studies will be judged on the reach and significance of their impact on the economy, society and/or culture, underpinned by excellent research conducted in the UoA.
Reach: the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact – where extent is not defined geographically
Significance: the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries
Case studies will need to demonstrate both reach and significance, but both elements will be assessed together (noting that the balance between them may vary at all quality levels).
In 2014, impacts on or through teaching were only allowed if they could be demonstrated beyond the submitting Institution. For REF 2021, impacts on or through teaching within an Institution will now also be allowed. Reach and significance will be assessed in the following way:
Reach: the extent or diversity of the communities affected by the change to teaching practice
Significance: the extent to which teaching practice was enriched, influenced or changed at the organisation(s) involved and/or the extent to which individuals experiencing changed teaching practice were enriched, influenced or changed.
Consultation
It is important to note these are draft guidelines and Research England are running a consultation exercise. Much of the consultation focuses on the clarity of the guidelines and appropriateness of the criteria. They are also looking for responses to their arrangements for output reductions and a variety of other issues such as the usefulness of the information in the appendices and list of independent research fellows. You can see all the consultation questions here
If you do want to respond then please use the word document provided by RE and get in touch with DVCRE, Prof Jerry Roberts, or Head of Research Support and Governance, Dr John Martin, as they are coordinating our response.
Panel requirements
For the panel criteria and working methods, they are also looking for comments on further refinement on UoA descriptors and whether there should be more or less commonality between panels. We at the Kaleidoscope have summarised the main key differences between Panels on a number of issues, in the table below.
| Panel A | Panel B | Panel C | Panel D | |
| 4* research | Word leading in terms of some of the following:
Scientific rigour and excellence, with regard to design, method, execution and analysis Significant addition to knowledge and to the conceptual framework of the field Potential and actual significance of the research The scale, challenge and logistical difficulty posed by the research The logical coherence of argument Contribution to theory-building Significance of work to advance knowledge, skills, understanding and scholarship in theory, practice, education, management and/or policy Applicability and significance to the relevant service users and research users Potential applicability for policy in, for example health, healthcare, public health, animal health or welfare. |
Some of the following:
Agenda-setting Research that is leading or at the forefront of the research area Great novelty in developing new thinking, new techniques or novel results Major influence on a research theme or field Developing new paradigms or fundamental new concepts for research Major changes in policy or practice Major influence on processes, production and management Major influence on user engagement. |
Some of the following:
Outstandingly novel in developing concepts, paradigms, techniques or outcomes A primary or essential point of reference A formative influence on the intellectual agenda Application of exceptionally rigorous research design and techniques of investigation and analysis Generation of an exceptionally significant data set or research resource. |
Some of the following:
A primary or essential point of reference Of profound influence Instrumental in developing new thinking, practices, paradigms, policies or audiences A major expansion of the range and the depth of research and its application Outstandingly novel, innovative and/or creative. |
| 3* research | Internationally excellent in terms of some of the following:
Scientific rigour and excellence, with regard to design, method, execution and analysis Significant addition to knowledge and to the conceptual framework of the field Potential and actual significance of the research The scale, challenge and logistical difficulty posed by the research The logical coherence of argument Contribution to theory-building Significance of work to advance knowledge, skills, understanding and scholarship in theory, practice, education, management and/or policy Applicability and significance to the relevant service users and research users Potential applicability for policy in, for example health, healthcare, public health, animal health or welfare. |
Some of the following:
Makes important contributions to the field at an international standard Contributes important knowledge, ideas and techniques which are likely to have a lasting influence, but are not necessarily leading to fundamental new concepts Significant changes to policies or practices Significant influence on processes, production and management Significant influence on user engagement. |
Some of the following:
Novel in developing concepts, paradigms, techniques or outcomes An important point of reference Contributing very important knowledge, ideas and techniques which are likely to have a lasting influence on the intellectual agenda Application of robust and appropriate research design and techniques of investigation and analysis Generation of a substantial data set or research resource |
Some of the following:
An important point of reference Of considerable influence A catalyst for, or important contribution to, new thinking, practices, paradigms, policies or audiences A significant expansion of the range and the depth of research and its application Significantly novel or innovative or creative. |
| Research Groups | Submissions should use research groups | Outputs can be linked to research groups is they are in the Environment | ||
| Pedagogic research | Only research on education, relevant to uoa3 disciplines, can be submitted to uoa3
Across the Panel, all other pedagogic research should not be submitted (bodies of work should be submitted to uoa23) or will be cross-referred to uoa23 |
Research on pedagogy and educational issues within higher education, related to the uoa disciplines, can be submitted to the relevant uoa
Across the Panel, all other pedagogic research should not be submitted (bodies of work should be submitted to uoa23) or will be cross-referred to uoa23 |
Research on pedagogy and educational issues within higher education, that relate to the disciplines covered by Main Panel C may be submitted in the UOA to which it relates or in UOA 23 | Research on pedagogy and educational issues within higher education that relate to the disciplines covered by Main Panel D may be submitted in the UOA to which it relates or in UOA 23 |
| Multiple submissions | No | No
Uoa 12 will provide sub-profiles for distinct disciplines within engineering |
Uoa22 | Uoa25, 26, 32-34 |
| Co-author contribution statement | Needed if co-author is not lead or corresponding author + 10 or more co-authors | Needed if 25 more authors | None | None |
| Double Weighting | Exceptional, supporting statement required | Exceptional, supporting statement required | Welcomed. Supporting statement | Welcomed. Statement only if not self-evident that this is double-weighted |
| Additional output information | Statement where this is not evident within the output (for non-text or practice-based outputs | Statement where this is not evident within the output (for non-text or practice-based outputs)
Identification of the original research or new insights reported (for reviews) |
Statement where this is not evident from the output (for any type of output)
For practice-based outputs, an explanatory presentation in paper format should be include For software and datasets, a full written description with details how to access |
Statement where the research role of the researcher, or research process, is not evident (any output).
Statement on contribution to translations, anthologies, edited books, special editions of journals and curatorial projects Rationale for grouping short items as a single output |
| Citation Data | Yes | Uoas 7-9, 11 | Uoa16 | No |
| Factual statement on significance | None | Uoas 11, 12, Factual statement wherever available | None | None |
| Continuing case studies | Panel A encourages new case studies, from the UK heis’ biomedical, health and life sciences sector. Continued case studies will be considered, recognising the long lead-in time for certain biomedical and life sciences impacts.
|
Submit strongest case studies only, whether new or a continuation.
Panels will disregard information on how any continued case study relates to that submitted to REF 2014 |
Submit strongest case studies only, whether new or a continuation.
Panels will disregard information on how any continued case study relates to that submitted to REF 2014 |
Submit strongest case studies only, whether new or a continuation.
Panels will disregard information on how any continued case study relates to that submitted to REF 2014 |
| Impact evidence | Wherever possible, quantitative indicators should be included. Verifiable sources for key evidence and indicators should be provided.
Testimony: The sub-panels do not welcome testimonials offering individuals’ opinions as evidence of impact Factual statements from external, non-academic organisations would be acceptable as sources to corroborate claims. |
The subpanels welcome both evidence-based quantitative and qualitative indicators as appropriate.
Where testimony is cited, it should be made clear whether the source is a participant in the process of impact delivery (and the degree to which this is the case), or is a reporter on the process. |
Individual case studies may draw on a variety of forms of evidence and indicators. The sub-panels do not wish to pre-judge forms of evidence. They encourage submitting units to use evidence most appropriate to the impact claimed | Sub-panels will not privilege any type of evidence above another.
Evidence should be focused on substantiating the impact(s) claimed. Sub-panels will consider the extent to which testimonials demonstrate the significance of the claims. Where corroborating evidence is reviewed as a consequence of audit, it will be used solely to verify the claims made about the impact. |
| Underpinning research for impact | Case studies must include references to one or more key research outputs that underpinned the impact.
Outputs must enable panels to decide on quality Additional indicators should be included Panels will review outputs if needed |
Case studies must include references to one or more key research outputs that underpinned the impact.
Outputs must enable panels to decide on quality Additional indicators should be included Panels will review outputs if needed |
A sample of the research should be cited that is sufficient to identify clearly the body of work, or individual project that underpins the claimed impact.
‘underpinning’ does not imply a specific temporal or directly causal relationship to impact Panels will not review outputs unless they have to Encourage addition of indicators of quality to enable assessment |
A sample of the research should be cited that is sufficient to identify clearly the body of work, or individual project that underpins the claimed impact.
‘underpinning’ does not imply a specific temporal or directly causal relationship to impact Panels will not review outputs unless they have to Encourage addition of indicators of quality to enable assessment |
For a digest, this has already been quite long, but hopefully will have saved some of you, at least, from having to ditch all your summer reading list!
In the meantime keep an eye out for further updates on the consultation responses, Codes of Practice, our annual REF Readiness exercise and of course more updates from the Kaleidoscope!
