During the recent JISC Managing Research Data - Benefits & Evidence workshop in Bristol, 29-30 November 2012, we discussed how to evaluate the quality of data management plans (DMPs).
I promised to share our experiences of how in practice we evaluated DMPs for the Rural Economy and Land Use programme (Relu) and recently also for ESRC grant applications.
In an earlier blog post the evaluation methodology was described in more detail.
Relu DMPs
This cross-disciplinary research programme (2005-2012) had its own data policy, with all projects preparing a data management plan at the start of a project (post funding). At the end of research research data were deposited with the UK Data Archive and the Environmental Information Data Centre.
In a data management plan researchers described:
- ( the need for access to existing data sources and any access limitations that may exist )
- datasets planned to be produced by the research project
- planned quality assurance and back-up procedures for data
- plans for management and archiving of collected data
- expected difficulties in making data available for re-use (through data archiving) and measures to overcome such difficulties
- who holds copyright and intellectual property rights of the data
- data management roles and responsibilities within the research team
We reviewed the data management plans prepared by 29 large projects, four pilot projects that created and archived data and three fellowships (totalling 36 plans). For each question we evaluated whether the information provided was insufficient (lacking clarity or detail), sufficient or excellent.
Datasets planned to be produced by the research project
Most plans contain sufficiently detailed lists of the various datasets planned to be produced. In a few cases information was vague and award holders were asked to provide better or more detailed information. For each dataset, the format/software in which data will be created or stored is specified and storage details are provided. Dependent on projects, storage may be solely on an institutional server or on a combination of server, PCs, institutional virtual environments and back-ups on movable media (CD, DVD, …).
During research projects, research activities may change and actual datasets produced at the end of a project can be different from those initially planned.
Planned quality assurance for data
All plans include good information on how data quality will be ensured. Measures include
- institutional quality assurance procedures, ISO standards
- standard data collection protocols
- standardised data recording (data entry sheets, validation rules in databases)
- instrument calibration
- recording metadata, labelling data
- documenting methods and procedures
- training researchers
- pilot studies
- double data entry
- validation check, cross-checking
- random checks
- peer review of data
- data record forms
- file naming standards
Planned data back-up procedures for data
Overall the information provided within this section is excellent. Most data management plans describe institutional data storage and back-up procedures that are in place. Most projects store data on institutional servers, which guarantees regular back-up and transfers the responsibility to institutional IT staff.
Some projects mention additional back-ups researchers plan to carry out (e.g. onto disks or hard drives, or by sharing copies of data between partner institutions) or state that the principal investigator will hold a master copy of all data, besides data held on partner servers.
Three data management plans failed to incorporate information for partner institutions, only listing procedures at the host institute.
Only four projects have specific data management staff allocated to the project, which have a role in overseeing data storage and back-up procedures (besides other responsibilities).
Expected difficulties in making data available for re-use and measures to overcome difficulties
Only 14 plans provide excellent information on this topic; in 10 plans the information is sufficient, whereas in 12 plans the information is vague or contains only a simple statement that ‘no difficulties in making data available for secondary use are anticipated’. In six project where no problems to make data available for archiving were foreseen, researchers did not consider obtaining consent for data obtained through interviews or surveys to be shared, or collected data under unnecessarily strict confidentiality agreements. Data obtained through interviews / surveys could therefore not be archived due to confidentiality restrictions. Researchers thus tend to underestimate potential difficulties to archive and share data, especially for confidential, commercial or sensitive data.
Almost half the plans (17) state that data confidentiality, the inclusion of personal data in research data, and copyright of third party sources may limit the archiving of some research data, with overall valid reasons given. Confidentiality restrictions may be in place due to commercial confidentiality (e.g. business information for farms) or where interviewees are easily identifiable (e.g. public body stakeholders and policy makers). Copyright limitations exist mostly where research projects use licensed data sources within GIS systems, to create derived data or to model research scenarios. Use of OS data in GIS typically limits sharing even many derived data.
Only six plans then provide information on how such difficulties may be overcome by the researchers, e.g. by anonymising data, aggregating data, obtaining consent to share data, or discussing data archiving with owners of licensed data.
Data copyright and IPR
Copyright / IPR of the data is generally with the researchers. At times there is joint copyright through use of third party data.
Data management responsibilities within the research team
Most projects allocate data management responsibilities to various researchers within the research team – typically one person per partner institution or one person per work package.
A few projects allocate only one person with data management responsibility for the entire project. For cross-institutional projects, it is not clear how that is manageable.
Four projects have a dedicated data manager, database manager or project manager with overall data management responsibility.
ESRC DMPs
ESRC introduced the requirement for a data management plan to be submitted with every grant application in April 2011. A data management plan should describe:
- an assessment of existing data that could be used for the research
- information on new data that will be created
- quality assurance of data
- back-up and security of data
- expected difficulties in data sharing, e.g. ethical or legal issues
- copyright and Intellectual Property Right of data
- data management responsibilities
- preparation of data for sharing and archiving
We recently evaluated an anonymous sample of 25 submitted data management plans, evaluating the quality of the information provided for each of those eight topics, by scoring: 1=insufficient; 2=sufficient; 3=excellent; whereby each plan was evaluated twice independently by various staff members of the UK Data Archive's Research Data Management section.
The average quality score for a DMP was 17, with a minimum score of 9 and a maximum score of 23. Nine DMPS scored below 16 (the score for suffcient information being provided for each topic). Six DMPs scored below 12.
DMPs on average provide good to excellent information on assessing existing data and describing new data to be created (average score of 2.4 and 2.3 resp). DMPs perform poorest on information about copyright and IPR of research data (average score 1.8).
Scores of 1 (insufficient information provided) were most common for copyright (7 plans), for data management responsibilities (5 plans) and for data preparation (5 plans).