An image of China in Africa through the lens of mixed-methods

by Jarosław Jura & Kaja Kałużyńska

The increasing number of digital and digitized content sources (online versions of traditional media, news portals, various websites on myriads of topics, and, of course, social media) has started to influence empirical social research. Huge amounts of easily accessible and almost ready-to-analyze datasets seem to be a dream coming true for social researchers, especially those who prefer to work with unobtrusively-collected data.

Such large datasets ask for being analysed by mixed methods, to avoid wasting their potential by either choosing a sample or focusing on quantitatively obtained information only. Here come other tools that make the life of a contemporary researcher much more comfortable – software solutions. Of course, in the ideal situation, one could just ‘feed’ all the data to AI and wait for the results, but there are many limitations to such an approach, like usability in specific cases, its accessibility, and, of course, the researcher’s nightmare: a limited project’s budget. Moreover, in the case of smaller datasets, consisting of heterogeneous data, analysis’ results might prove unsatisfactory.

Our research project, an exploratory study on the image of China and the Chinese in Zambia and Angola, included also an analysis of textual media content, namely news articles published in these countries and mentioning China or the Chinese. We obtained a mid-sized dataset, consisting of 2477 articles; the material was very heterogeneous, because of the wide scope of topics covered by the texts and the fact that we analysed content from both English- and Portuguese-language media.

In the course of analysis, we realized that a new method would be needed to obtain the best possible results on the basis of the collected data. After a series of trial-and-error approaches, we managed to develop MIHA – Mixed Integrative Heuristic Approach. The application of this method allowed us to create an exhaustive, contextual and precise keyword dictionary for automated classification of text units as well as a set of sentiment indexes.

We have to admit, that even though we did our best to utilize all the possibilities of the software (Provalis QDA Miner and Wordstat), the dictionary creation process was a time-consuming task since it included reviewing each word of frequency higher or equal to 10 in the whole database.

Our classification, similar to the initial conceptualization of theoretical categories within the grounded theory approach, aimed to explore the most frequent contexts in which China was depicted in African e-media. Each examined word was either added to an exclusion list (words irrelevant from the point of view of the research) or assigned to a chosen – sometimes a newly created – category, together with other words of the same root and all the synonyms.

In the next step, we examined the already categorized keywords in their context to refine the categorization results, mainly by removing those keywords that appeared within the text in unexpected contexts. Most of the categories were re-coded, and some of the keywords were re-assigned in the next steps. This heuristic approach resulted in a set of categories, including ‘emotional’ ones, positive and negative, that later on were used to design sentiment indexes. Our indexes are based on a comparison of the results of quantitative and qualitative analysis and coding. They could be used as a tool for improving dictionary-based sentiment analysis by comparing the results of sentiment analysis performed on the basis of automated coding with manually-coded samples.

We believe that MIHA constitutes a conceptual approach applicable by researchers of various backgrounds in projects focused on investigating the general image presented in textual content, especially in case of mid-sized, heterogeneous data sets. We do not overlook the fact that soon, automated machine learning coding methods will constitute the main approach towards text analysis. However, since such procedures are still imperfect and context-sensitive, we presume that MIHA, consisting of a contextualized dictionary, manual coding of chosen parts of the database and index measurements, could be useful for analysis of data sets related to less common study areas (social groups, languages, geographical areas, subcultures, etc.), in which machine learning-based research would contain a low level of construct validity.

Both the dictionary-creation process and the indexes are described in detail in our paper.

Read the full article in the IJSRM here.

featured, Notebook

Using objects to help facilitate qualitative interviews

by Signe Ravn

Doing empirical research on imagined futures is a methodological challenge. As scholars have argued, generating rich insights into how such futures might look can be difficult as participants may produce somewhat generic or stereotypical accounts of what the future might hold or even refuse to engage in such tasks (which of course provides other insights). Over the past decade, these challenges have led many qualitative researchers to explore different forms of creative, arts-based and/or participatory methods to approach the topic in new ways. In some cases, these approaches have been productive, and in other cases they lead to new questions about how to then interpret the findings. And sometimes they don’t really generate more concrete insights after all.

In my longitudinal research on the everyday lives and imagined futures of young women with interrupted formal schooling, I also used various creative methods to break away from the traditional interview format and to seek to approach the ways in which participants imagined their futures from multiple different perspectives. This approach was inspired by Jennifer Mason’s work on facet methodology. In my recent paper for the International Journal of Social Research Methodology I explore one creative method that proved particularly fruitful, that is, an object-based method. In brief, this method was deployed in the third interview with my participants (after one year) and involved asking participants to bring ‘one thing (like a gift, some clothing, a thing you once bought, or something else) that reminds you of your past and a thing that you relate to your future’. Only one participant asked for a clarification of what these items could be, while the remainder were happy to do this task, and some even said right away that they knew exactly what to bring. On the day of the interview, some participants did say that deciding on a ‘future’ thing had been difficult, but nevertheless they all had chosen something. Towards the end of the interview I asked about their ‘things’ and we spoke about each object in turn, exploring why they had brought a particular object, how it related to their past/future, and whether and how this was something they used in their day-to-day lives.

Reflecting on the interviews I was wondering what made this particular exercise helpful for exploring and speaking about ‘futures’. Other scholars have successfully drawn on objects to study memories, but none have turned their attention to the potential of objects for studying futures. In the paper I argue that what makes the object-method productive is to do with materiality. More specifically, I argue that what makes this method unique is the combination of ‘materiality as method’ as well as the ‘materiality of the method’, and that this double materiality at play is what is producing elaborate future narratives. In other words, via the materiality of the objects, specific imagined futures become ‘within reach’ for participants, with the object serving as an anchor for these future narratives. The method suggests a temporal complexity as well: the future objects come to represent futures that the participants have already taken steps towards; they are ‘futures-already-in-the-making. Drawing on Jose Esteban Munoz, we can consider them ‘futures in the present’, that is, futures that already exist, perhaps just in glimpses, in the present.

To make this argument I draw on both narrative research, material culture studies and qualitative research methodology. One key source of inspiration was Liz Moor and Emma Uprichard’s work on material approaches to empirical research, where the authors argue for paying greater attention to the ‘latent messages’ of methods and data, for instance in the form of sensory and emotional responses but also, as I point to in the paper, the messages conveyed by a dirty and bent P plate and a carefully crafted name tag.   Due to limitations of space, the published paper focuses on the ‘future’ objects and the future narratives generated through these, and only briefly mentions the ‘past’ object that participants also brought to the interview. This is due to the paper’s ambition to highlight the potentials of using object methods, and a focus on materiality more generally, in research on futures. However, for a full analysis of the insights gained through this method, both in terms of the settled and unsettled future narratives and the normative dimensions shaping which objects became ‘proper’ objects for the interview situation, both ‘past’ and ‘future’ objects should be analysed together.

Read the full article in the IJSRM here.

featured, Notebook

Measuring Measures during a Pandemic

by Paul Romanowich & Qian Chen,

The spring 2020 semester started like many others before – frantically preparing class materials, finalizing research proposals, and trying to squeeze in one last getaway trip. However, by mid-March 2020 that normalcy had fallen by the wayside. Like it or not, classes were now all remote, disrupting both data collection and plans for any meaningful travel during the summer. But what about that data that was collected? Was it any good, considering what our participants were experiencing? Not surprisingly, little research has focused on the impact major environmental disruptions have on data reliability, given how rare and unpredictable those disruptions are (have you ever experienced a pandemic before 2020?!?). However, we were fortunate to be collecting repeated-measure impulsivity data throughout the spring 2020 semester. Thus, this research note focuses on whether data obtained in the immediate aftermath of the beginning of the COVID-19 pandemic is reliable, from a test-retest perspective.

Our original research question centered around whether decreasing one aspect of impulsivity, delay discounting, would have a positive effect on test scores for Electrical and Computer Engineering students. Like many personality traits, delay discounting rates have been shown to be relatively stable via test-retest data (i.e., trait-like). However, there is also a growing literature that episodic future thinking (EFT) can decrease delay discounting rates, and as a result decrease important impulse-related health behaviors (e.g., smoking, alcohol consumption, obesity). Thus, delay discounting also shows state-like properties. We hypothesized that decreasing delay discounting rates via EFT would also decrease impulse-related academic behaviors (e.g., procrastination), resulting in better quiz and test scores. To accurately measure temporal aspects of delay discounting, EFT, and class performance students completed up to 8 short (27-items) delay discounting tasks from January to May 2020. Multiple EFT trainings significantly decreased delay discounting rates relative to a control group (standardized episodic thinking – SET). However, the impact of EFT on academic performance was more modest.

Although the data did not support our original hypothesis, we did still have repeated-measure delay discounting data throughout the semester, which included data from March 2020 when classes were switched from in-person to fully remote. This repeated-measure data set up a series of Pearson correlations throughout the semester between delay discounting rates at two points in time (e.g., delay discounting rates at the beginning of the semester in January 2020 and end of the semester in May 2020). Importantly, students in the EFT group completed a delay discounting task on March 22, 2020 – 11 days after the official announcement that all classes would be fully remote for the remainder of the semester. In terms of test-retest reliability, the data collected on March 22, 2020 stood out as not like the other. Whereas delay discounting task test-retest reliability was high throughout the semester (supporting previous studies), most correlations using the March 22, 2020 data was nonsignificant, suggesting poor test-retest reliability. Thus, it appeared that the COVID-19 pandemic had significantly, but only temporarily, decreased test-retest reliability for delay discounting rates.

The EFT data also afforded us a way to look at changes more qualitatively in behavior before and after March 22, 2020. As a part of the EFT trainings, students came up with three plausible positive events that could happen in the next month, 6 months, and one year. We coded these events as either having COVID-19 content or not for all students. Predictably, events containing COVID-19 content did not appear until March 22, 2020. However, this event content changed as the semester progressed. On March 22, 2020, most (6 of 7 events) of the content was for the 1-month event. By May 7, 2020 only two students included COVID-19 content, and this was for the 6-month event. Thus, students were more concerned with COVID-19 in March 2020 and as a closer temporal disturbance, relative to May 2020. Perhaps this focus on COVID-19 in the near future disrupted delay discounting rates. We can’t be sure from this data, but the idea is intriguing.

Although this research note was not a rigorously controlled experiment to explicitly examine test-retest reliability for delay discounting, there are still some important points to take from the obtained data. First, it does appear that large environmental disruptions in participants life can significantly change test-retest reliability on standardized measures. Social and behavioral science researchers should be aware of this when interpreting their data. It may also be worthwhile to include a brief measure for significant life events that may be occurring concurrently with their participation in the task. Second, the change in test-retest reliability we observed was only temporary. This is actually good news for researchers, in that even significant environmental disruptions seem to have a minimal impact on test-retest reliability one month later. Perhaps we are more resilient as a species than we typically give ourselves credit for. Lastly, we have no doubt that other social and behavioral science researchers collected similar repeated-measure data throughout the spring 2020 semester. One way to be more confident that our results are not an outlier is through replication. Although we can’t (and don’t want to!) replay the beginning of the COVID-19 pandemic, researchers around the world could profitably begin to combine their data for specific well-validated measures to examine how this large environmental disruption may have systematically affected their results. The same could be done for other large environmental events, such as earthquakes or wars. The end result would be a better understanding of how these environmental disruptions impact those measurement tools that we base many of our theories and treatments off of.

Read the full article in IJSRM here.

featured, Notebook

Bringing “context” to the methodological forefront

By Ana Manzano & Joanne Greenhalgh,

In his latest methodological writings, Prof Ray Pawson (2020) noted that the Covid-19 pandemic:

 “covers everything from micro-biology to macro-economics and all individual and institutional layers in between”.

The current global pandemic could be considered the mother of all contexts. Many will conclude that we could not reduce the impact of Covid-19 in our lives to a limited number of contextual factors such as disease, bereavement, home working, schools closures, travel bans, etc. Covid-19 was and continues to be a force that impacts everything through a complex combination of omnipresent uncertainty, fears, risk management and materiality (masks , PCR tests, and hydrogenic gel). Our paper Understanding ‘context’ in realist evaluation and synthesis (Greenhalgh & Manzano, 2021), just published in the International Journal of Social Research Methodology, reflects precisely on how methodologically complex context is and reviews how context is conceptualized and utilized in current realist evaluation and synthesis investigations.

Perhaps, the most useful of all the quotes mentioned in our paper is one of French sociologist, Raymond Boudon’s (2014, p. 43) who reminds researchers that in the social sciences, it is impossible to talk about context in general terms, since context is always defined specifically:

The question as to “What is context?” has actually no general answer, but answers specifically adapted to the challenging macroscopic puzzles the sociologist wants to disentangle.

Context is somehow everything and, in some ways has become “nothing” with many methodological writings on causality focusing on the more attractive concept of “mechanisms”. Our paper projects context from its eternal background position in peer-reviewed papers, trials and research results, to the foreground.  Although context is a key concept in developing realist causal explanations, its conceptualisation has received comparatively less attention (with notable exceptions e.g. Coldwell (2019)). We conducted a review to explore how context is conceptualised within realist reviews and evaluations published during 2018. We purposively selected 40 studies to examine: How is context defined? And how is context operationalised in the findings? We identified two key ‘narratives’ in the way context was conceptualised and mobilized to produce causal explanations: 1) Context as observable features (space, place, people, things) that triggered or blocked the intervention; assuming that context operates at one moment in time and sets in motion a chain reaction of events. 2)  Context as the relational and dynamic features that shaped the mechanisms through which the intervention works; assuming that context operates in a dynamic, emergent way over time at multiple different levels of the social system. 

We acknowledge that the use of context in realist research is unlikely to be reduced to these two forms of usage only.  However, we argue that these two narratives characterise important distinctions that have different implications for the design, goals and impact of realist reviews and evaluations.  Seeing context as a ‘thing’, that is, as a  ‘feature that triggers’ suggests that one can identify and then reproduce these contextual features in order to optimise the implementation of the intervention as intended.  This reinforces a view that it is possible to isolate ‘ideal’ contexts that determine the success of an intervention. 

On the contrary, seeing context as a dynamic interaction between contexts and mechanisms implies that contexts are infinite, embedded and uncontrollable. Knowledge gained about how contexts and mechanisms interact can be used to understand how interventions might be targeted at broadly similar contextual conditions or adapted to fit with different contextual conditions.  This latter approach eschews the idea that there are ‘optimal’ contextual conditions but argues that successful implementation requires a process of matching and adapting interventions to different evolving circumstances. 

Our paper will disappoint those who seek a practical definition that will help the ever impossible task of distinguishing mechanisms from contexts in causal explanations. We have some sympathy with Dixon-Woods’ claim (2014, p. 98) about distinguishing mechanisms from contexts in realist studies:

 I am inclined towards the view that discussions of what constitutes a mechanism rapidly become unproductive (and tedious), and that it is often impossible, close up, to distinguish mechanism from context.

Since much methodological thinking focuses on mechanisms and funders are (typically, though not exclusively)  interested in outcomes, contexts are, if anything,  rather “annoying”. Context, with its symbiotic relationship with mechanisms,  confuses and distracts researchers in their most important search for mechanisms ‘holy grail’. Our paper demonstrates that the answer to that holy grail pursuit is precisely in that symbiotic relationship, in which contexts are relational and dynamic features that shape the mechanisms through which interventions work. Context operates in a dynamic, emergent way over time at multiple different levels of social systems.

Finally, we are mindful that, in Pawson’s own words when we discussed this paper with him, ‘context’ can mean ‘absolutelybloodyeverything’ and so it is very difficult to perceive that its usage in realist research is reduced to the two forms identified in our review.

Read the full IJSRM article here.


Boudon, R. (2014). What is context? KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie 66 (1), 17-45

Coldwell, M. (2019). Reconsidering context: Six underlying features of context to improve learning from evaluation. Evaluation, 25 (1), 99-117.

Dixon-Woods, M. (2014). The problem of context in quality improvement. Perspectives on context. London: Health Foundation. 87-101

Greenhalgh, J. and Manzano, A. (2021) Understanding ‘context’ in realist evaluation and synthesis. International Journal of Social Research Methodology. Pawson, R. (2020). The Coronavirus response: A realistic agenda for evaluation. RealismLeeds Webinar July 2020.

featured, Notebook

Bystanders and response bias in face-to-face surveys in Africa

By Zack Zimbalist

Public opinion surveys are crucial fonts for understanding the public’s perceptions, values, and attitudes across the world. By conducting such surveys repeatedly with random samples, social scientists are able to track how responses change over time. This allows researchers to capture the dynamics of social perceptions on a range of interesting topics, from the economy, to health, education, crime, corruption, democracy, and government performance.

Ideally, respondents feel secure to disclose accurate information (avoiding reporting bias and item non-response) in the context of a face-to-face interview. Yet, survey research in political science seldom accounts for peer effects caused by bystanders. Much of the existing research focuses primarily on the effects of parents and spouses on self-reporting illicit activities or marriage-related issues. Moreover, these studies have mainly been carried out in industrialized countries with only a few studies that are also confined to similar survey questions in a small sample of developing countries.

This is thus the first study to investigate bystander effects across a large sample of developing countries for a broad set of questions related to social, political, and economic outcomes. Studying the presence of bystanders is important because third parties are often present in population surveys, especially in developing country contexts where extended family members and communities live in close proximity. For example, a bystander is present at 34% of interviews conducted by the Afrobarometer survey. Of those, 16% of respondents in the total sample is accompanied by non-familial bystanders, 6% by their spouses, and 12% by their children.

Using survey data from over 45,000 households across 34 African countries (collected by the Afrobarometer), my new articleBystanders and response bias in face-to-face surveys in Africa” finds that bystanders, especially non-familial ones, substantially affect responses to an array of questions (some sensitive and some not). The paper also demonstrates that these biased responses run counter to biases due to fear (linked to the perception of a state interviewer) and are most likely explained by social desirability when one is among her peers (a few people or a small crowd). The biases are far rarer for interviews conducted among just a spouse or children.

Let me provide a few examples from the article. First, in the presence of non-familial bystanders, respondents understate the extent (or supply) of democracy and their satisfaction with democracy and report higher levels of fear of political violence. These results run counter to respondents’ overstatement of supply and satisfaction with democracy with respect to the perception of a government interviewer. I argue that these overstatements correspond to the fear of criticizing the government’s performance on this dimension. By contrast, in the presence of non-familial bystanders, the opposite effect is most likely driven by social desirability concerns around what one’s neighbors believe to be the appropriate answer.

Second, respondents supervised by their peers express more disapproval for the performance of their MPs, local government, and the mayor. Here, again, it seems likely that expressing disapproval for politicians and government is, on average, the socially desirable response. This result contrasts with reporting systematically more approval in the case of perceiving a state interviewer (a fear-induced bias).

Third, in line with the social desirability of reporting disapproval of elected officials, respondents supervised by non-familial bystanders report higher levels of corruption in the presidential office, among MPs and government officials. Again, this result runs counter to the fear-induced state interviewer response, which is underreporting corruption levels to state interviewers.

In addition to misreporting, bystanders of both kin and non-kin are strongly associated with higher rates of item nonresponse. The levels of nonresponse and the gaps between bystander and non-bystander interviews were largest for arguably sensitive questions wherein a “don’t know” answer could be seen as satisficing and socially desirable.

This article’s results suggest the need to implement additional measures to measure and mitigate bystander presence. To measure bystander bias in contexts outside of Africa, other surveys such as other regional barometers and Pew polls would do well to include a question on the presence of bystanders. Mitigating bystander-induced biases is a thornier challenge that requires further experimentation across contexts. One alternative approach is self-administration in high literacy contexts (which eliminates the biases caused by bystanders overhearing answers) as some research has shown that respondents are more willing to answer sensitive questions when they are self-administered (see Krumpal, 2013 for a review). In addition, indirect modes of administration such as endorsement experiments, list experiments (or item count or unmatched count technique) (Glynn, 2013) or randomized response techniques (RRT) (Coutts & Jann, 2011; Rosenfeld, Imai, & Shapiro, 2015) could also be tested. Despite their limitations, indirect methods may improve data collection on sensitive questions. Moreover, they are more implementable than self-administration in low literacy contexts. Further research would be helpful in bolstering our understanding of whether, and to what extent, these methods obtain more reliable estimates across different contexts.

Overall, the article provides new evidence of substantial effects of bystanders across a range of survey questions in a large sample of democratic and non-democratic African countries. Securing private interviews is a sine qua non for obtaining accurate data. In the absence of this, alternative techniques could be deployed to ensure that respondents are free to provide honest assessments and perspectives on important economic, political and social questions.

Read the full article on IJSRM here.