Key lessons on using social surveys within interdisciplinary research
Earlier this year, ESPA hosted an event to share best practice on social surveys.
Research carried out under the ESPA programme regularly combines the natural and social sciences and can quite often be data intensive, with many projects requiring substantive new field campaigns. The new data for much of ESPA’s social research is underpinned by social or household surveys.
A significant number of ESPA projects have designed and implemented surveys. The ESPA Social Surveys event, which took place in London in October, provided an opportunity to bring together those researchers involved in designing and using social surveys to share and discuss their experiences and findings.
Here, we’ve summarised the main learnings and tips shared at the event for anyone interested in using social surveys.
- Consider ways to supplement ecosystem services surveys with additional metrics in order to provide a more complete picture and understanding of concepts such as wellbeing and poverty.
Ecosystem services are important for mediating the effects of human actions on wellbeing and considering the larger ecosystem, but it is important to consider the ‘blind spots’ of a social survey approach and be explicit in the limitations of particular methods and their indicators in measuring key concepts. Take the time to consider definitions of key concepts, how they are understood locally and how best to measure these to enhance internal validity.
- Supplement survey data with a mixed methods approach in order to produce a more complete understanding of ecosystem services, but be mindful of inherent methodological tensions.
Local decision making can be political, complex and process-based in nature. Ecosystem services may also be used in different ways by different beneficiary groups. This makes it difficult to measure in a standardised way, but there is also a recognition that there is a need to scale up findings to support programme management and policy-makers. The fundamental differences of approach between social and natural sciences make mixed methods a difficult task, but there are examples of where, with careful planning, this can be achieved effectively to produce a better local understanding of ecosystem services and its different dimensions. In one example, ‘water quantity and quality’ were determined through multiple means: biophysical measurements, household survey responses, and focus groups to understand perceived changes and causality.
- Qualitative methods can help navigate definitional and ethical challenges, and help validate wider findings.
Local understanding of concepts such as ‘village’, boundaries and technical phrases can vary, making them difficult to measure in a standardised way (some terminology can be evocative and sensitive). This can be further compounded by cultural differences in understanding concepts such as wealth, social or cultural capital and land ownership. Piloting of household surveys and their indicators is already part of best practice for M&E professionals, and this can be built on by utilising the various examples of qualitative methods highlighted in this report to define concepts and to test internal validity on an ongoing basis.
- Be mindful that the lack of clarity of systems boundaries provides challenges for constructing a sampling frame.
The characteristics of local geographical landscapes, and the differing needs of social and natural scientists, provide challenges for sampling. Difficulties have emerged both quantitatively (in terms of representativeness) and qualitatively (how to discuss and simulate scarcity and geographical and system boundaries). There are numerous examples of innovative approaches to overcoming these, and how careful planning and set up can help to mitigate the associated methodological risks involved.
- Build in additional time for planning and implementation.
More time is needed to better understand local context and to ensure that mixed methods approaches provide robust frameworks for both social and natural scientists to ensure that scientific validity isn’t compromised by the need to provide a complete picture of how ES are used locally. Additional time is need for constructing a sampling frame and choice of indicators, planning of how to isolate the effect of interventions, building trust and considering local ethics.
- Take time to consider how data collection may affect, and be affected by, local dynamics.
Be aware that interaction with participants may have both positive and negative implications for them, their household and their community. Consider what impact compensation will have on the individual (create tensions between families) and on the validity of their response (will they be telling the truth, would they prefer not to respond?). Understand local taboos and legally loaded terminology (e.g. wetlands). Be clear about definitions of key concepts for qualitative work and developing indicators for frameworks. Think about how the findings will be published: it may be desirable for analysis to provide GPS locations (to link socioeconomic with biophysical data) but this may contravene consent agreements by allowing identification of a particular household within a particular sparse landscape.
- Build trust with local participants and explain purpose of research
Participants may have numerous reasons for not trusting the research process (including data collectors, enumerators, translators and policy makers). For example there are sensitivities in ‘slash-and-burn’ agricultures, tensions between participating and non-participating families and worries that responses may be fed back to authorities. Being clear about the purpose of the research, particular questions and what will happen to participants’ responses/data will help to build trust and therefore increase validity of responses.
- Adhere to data management best practice to support effective and efficient data archiving.
To ensure that others can learn from your experiences and data effectively, be explicit in terms of: data sources; assumptions and context where appropriate; which definitions have been used; and, what the ethical and political considerations are. Providing ‘read me’ files and access to raw data are extremely useful, although the latter may be difficult ethically.
- Consider the ethics of data sharing over the entire data life cycle.
To protect participants from risk it is important to consider how to secure personal and local data from various levels of stakeholders (including enumerators and translators), from engagement to the consequences of archiving data, adhering to data protection laws in your country and the country of origin.
The ESPA Social Surveys Event took place on the 23rd and 24th October in London.
A corresponding Working Paper ‘Sharing Social data in multidisciplinary, multi-stakeholder research’ has also been produced, which offers detailled guidance and best practise about designing and implementing social surveys.