featured, Notebook

What is Ragin’s Indirect Method of Calibration?

Guest post by Preya Bhattacharya

In the last few years, Qualitative Comparative Analysis (QCA) has become one of the most important comparative data analysis approaches in the field of social sciences. QCA is useful if you have a small to medium number of cases, you are collecting data from multiple levels, and trying to analyze the causal pathway through which a condition or combination of conditions impacts an outcome, cross-cases and/or across time.

In my doctoral dissertation on “Can Microfinance Impact National Economic Development? A Gendered Perspective,” I applied panel data QCA as my causal analysis method, because I had a small number of case studies, and I wanted to analyze how microfinance institutions impact the economic development of women at the level of the national economy, in the context of former Yugoslavian countries (Bhattacharya 2020). But, during my data analysis phase, I came across this concept of “calibration.” At that time, I did not have any formal training in QCA, and it was a little bit difficult for me to understand and apply this concept. So, through this research note, I hope to guide future researchers in their own work on QCA, by explaining the process of calibrating an interval variable.

In my article, I have first defined the concept of calibration, and then applied my own data to demonstrate the steps of data calibration. As described in my article, calibration can be defined as a data transformation process, in which researchers try to transform the data that they have collected into set-membership scores, of 0/1 (crisp-set), or a range of 0 to 1 (fuzzy-sets). This data transformation process helps a researcher interpret their data in the context of the cases studied, and it depends on a variety of factors. As a result, the process of calibration might differ from one variable to another, even within the same dataset.

To demonstrate this, I have described the data calibration process for an interval variable, that does not have established prior theoretical cut-off points. I have divided my calibration process into five main steps:

  • First, define the set
  • Second, visualize the data distribution
  • Third, identify qualitative changes in data distribution
  • Fourth, assign qualitative categories
  • Fifth, transform these categories into set membership scores, ranging from 0 to 1 (fuzzy set) or 0/1 (crisp)

Finally, I have discussed the issue of robustness in calibration, and how researchers can ensure that their calibrated data matches with the reality of the cases they have studied. By describing these steps, I hope to help future researchers in their own process of calibrating interval variables in QCA.

I would also like to thank the 2019 Southern California QCA Workshop, organized by Dr. Charles Ragin (University of California, Irvine) and Dr. Peer Fiss (University of Southern California), for introducing me to the world of Qualitative Comparative Analysis and set theoretic research approach!

Read the full article here

featured, Notebook

25th Anniversary Editorial appendix on methods and data

By Iasonas Lamprianou

One of the sections of the 25th Anniversary Editorial of the International Journal of Social Research Methodology (IJSRM), presents the thematic trends in published contributions, for the whole period of 25 years of the journal’s life. Investigating the thematic trends in published contributions was not an easy task, not only because of the huge number of published papers, but also because of various technical details (for example, the published papers were not accompanied by keywords in the first volumes). 

Coding and charting the thematic trends of published papers proved to be a very laborious task and the workload was shared between four researchers. The aim of this document is to help interested readers understand the nature and the structure of the dataset. This could be useful to those interested in extending our own analysis, which had to be confined in the limited space of an Editorial.  

Methodology 

It is important for prospective users of the dataset to understand how the published content was coded and how reliable the coding was. 

Three coders worked in parallel for three weeks, under the supervision of an experienced researcher. They coded each of the contributions published by IJSRM, not only in the first 25 issues, but also in the ‘latest papers’ section of the journal’s web page, which includes papers which have not yet been assigned to specific volumes/issues.  

The three coders and the experienced researcher, developed a coding scheme, which included all the necessary variables to be coded in an Excel file. Through long online meeting, the group discussed the aims of the coding exercise, the structure and content of the coding scheme etc. The group coded a number of common papers to confirm that they interpreted the coding scheme in the same way. Regular online meetings and email exchanges were necessary to discuss various issues which emerged and to keep the coders in sync. To make sure that the coders did not ‘drift’ over time, they were instructed to ‘blindly’ re-code 5%-10% of each other’s excel file incrementally (every few days). The coders were in communication all the time and they exchanged emails where they would update each other about coding difficulties in order to remain in sync. As a result of this procedure, various issues came to the surface (e.g. there were many papers which could not be easily categorized as Qualitative, Quantitative or Mixed, so a new category was created; more information later). 

When all the coding was completed, the experienced researcher re-coded blindly 50 random papers – around 5% of the total number of papers in the database – but no major discrepancies were detected (for example, in one case, the number of views was miskeyed as ‘867’ instead of ‘861’) .  

Overall, there is no reason to believe that there is widespread bias or errors in the data. We expect the dataset to give a fair interpretation of what has been published in the journal in the last 25 years of its life. 

Variables in dataset 

The dataset includes the following variables: 

Vol Volume 
No Issue number 
Title The title of the paper (no coding, it was just copied and pasted) 
Abstract The abstract of the paper (no coding, it was just copied and pasted) 
Keywords The keywords of the paper (no coding, it was just copied and pasted) 
Paradigm Main research paradigm. Takes four values: Qualitative, Quantitative, Mixed Methods, General/Other.  Note: The General/Other category refers to papers which cannot be described accurately by the three other codes (Qualitative, Quantitative, Mixed Methods) 
Views Number of views (as reported on the journal’s web page) 
CrossRef Number of CrossRef citations (as reported on the journal’s web page) 
Altmetric Altmetric count (as reported on the journal’s web page) 

Data filtering 

The original dataset consisted of 1043 records, but book reviews, editorials and other small items were removed, resulting to a ‘clean’ dataset of 924 published papers (including ‘Research Notes’). 

Dataset format 

The dataset is provided as an R data frame, with the name EditorialData.Rda

You can download the data files here.

featured, Notebook

A virtual collection to celebrate 25 volumes of IJSRM 

The International Journal of Social Research Methodology is celebrating its 25th anniversary!!  In that period, we have become a leading methods journal publishing high quality contributions across the methodological field – both qualitative and quantitative, and including mixed, multi-media, comparative and simulation methods. 

To mark the occasion, we have gathered together a series of methods discussions that have been published in the journal, and our publisher, Routledge, is making them freely available as a collection.  

Choosing which articles to include in our anniversary virtual collection was a hard task.  We inevitably had to leave some important and favourite pieces aside.  The collection below includes contributions that we felt represented the range of methodological articles that we publish in IJSRM, a selection of early career prize winning articles, influential pieces and discussions that deserve more attention for their contributions, and individual editors’ personal choices.  

The methodological reach of our anniversary, then, ranges across survey non-response, behavioural experiments, quantitative pedagogy, the Delphi method, the problem-centred expert interview, the self-interview, narrative and computerised text analysis, qualitative methods transformations, anonymisation, triangulation of perspectives, indigenous data sovereignty, post-humanism, and researcher safety. 

We hope that you enjoy our selection. You can access it at:  

https://www.tandfonline.com/journals/tsrm20/collections/TSRM_25th_Anniversary_SI

Rosalind Edwards, Jason Lamprianou, Jane Pulkingham, Malcolm Williams 

IJSRM Co-Editors 

featured, Notebook

Seeking alternatives: A reflection on conducting online interviews with disabled young people during the COVID-19 pandemic

Angharad Butler-Rees and Stella Chatzitheochari

While scholars have increasingly documented and reflected on their approaches to conducting research during the pandemic, little is still known about the impact of social distancing measures on qualitative research with disabled young people.

Our new paper provides a methodological reflection on undertaking qualitative research with disabled young people as part of the Educational Pathways and Work Outcomes longitudinal study. Our study started in March 2021 during the third national lockdown in England. Due to social distancing measures in place at the time of commencing, it was necessary to revise our original plans to conduct face-to-face interviews with disabled young people and conduct online interviews instead. We conducted a total of 35 online interviews with autistic, dyslexic, and physically disabled young people aged 15-16 years old.

Ensuring Accessibility

The internet has long been deemed as a potentially empowering platform for disabled people, connecting isolated individuals and ensuring access to social, civic and community life. Our focus on young people was particularly useful, as this population tends to be very comfortable with the use of technology. The extensive periods of enforced home-learning during the covid-19 pandemic had further increased young people’s familiarity with online communication platforms, rendering the idea of online interviews a far less daunting prospect. However, it is worth noting that online tools can also present a number of accessibility barriers e.g., poor text layout, little colour contrast and limited keyboard functionality. These were important factors to consider when designing online interviews for our project.

Accessibility has to be incorporated into every part of the research process when working with disabled people. To put participants at ease prior to their interview, we sent them participant information packs as well as a short video of the interviewer introducing themselves and the study. Familiarity with the researcher was greatly valued for autistic young people, making them feel more at ease. Previous literature has suggested that autistic young people may be disconcerted or unresponsive in encounters with strangers, so building a degree of initial trust and rapport was of upmost importance for successful interviewing. In line with this, we also arranged online pre-interview meetings with participants and their parents to build rapport.

Pre-interview meetings also helped us ensure that any accessibility requirements were put into place. We asked participants to choose their preferred communication platform. Several participants opted to use assistive software during their interview e.g., enabling captioning, magnification or modifying volume. Other adjustments included allowing participants to sit off screen or to keep their cameras off while the interviewer remained visible. This made interview far less intrusive and anxiety provoking and was greatly valued by autistic participants. Other adjustments included the presence of a guardian that could provide practical assistance or emotional support, simplification of interview questions, as well as collection of data over several interviews as opposed to one. Overall, we felt that these adjustments made interviews considerably more accessible for disabled young people, ultimately giving voice to a population who may not always be amenable to conventional face-to-face interviewing methods that can be experienced as more restricting and demanding.  

Challenges during Interviewing

While some young people were very comfortable in engaging with the interview process and narrating their lived experiences, others were far more hesitant, requiring regular prompting and reassurance. The online medium made this slightly more challenging for the interviewer, with prompting and encouragement occasionally leading to cross-talking. It was also notably more difficult to interpret emotion and body language online, while the loss of internet connection at times affected the flow of the interview.

Another challenge was the difficulty in maintaining participants’ attention. We sometimes felt that the lack of physical presence meant that participants were far easier distracted by being in their homes, e.g., checking their mobiles, playing with family pet. However, we also recognise that this may be interpreted in a different manner: Indeed, it may be indicative of a greater share of power afforded to disabled young people in online settings. Overall, we did not feel that such distractions affected the quality of our data collection and think that the physical distance may have aided disclosure of personal experiences. A feedback survey confirmed that participants enjoyed the use of the online medium, with the vast majority requesting online interviews for the future waves of data collection.

A final note on accessibility

Our reflections may not speak to studies seeking to interview disabled young people with different accessibility needs such as speech or communication difficulties (e.g., stammering). These participants may find online communication more difficult due to possible misunderstanding and difficulties in lip reading and interpretation. Similarly, it is worth noting that interviews can be experienced as particularly exhausting by some disabled young people, whether face to face or online, preventing them from taking part. Researchers may consider offering alternatives such as email interviews alongside conventional online interviews.

Looking ahead

Our overall experience with online interviews was very positive. We were privileged to be able to access disabled young people’s lived experiences during an unprecedented period of global disruption. Notwithstanding the challenges mentioned above, we feel that online interviewing is a valuable tool that should not be viewed as second best to face-to-face conversational methods. We therefore encourage researchers to explore the use of online methods, especially with regards to young and disabled populations.

Read the full article here: Giving a Socially Distanced Voice to Disabled Young People: Insights from the Educational Pathways and Work Outcomes Longitudinal Study

Notebook

An image of China in Africa through the lens of mixed-methods

by Jarosław Jura & Kaja Kałużyńska

The increasing number of digital and digitized content sources (online versions of traditional media, news portals, various websites on myriads of topics, and, of course, social media) has started to influence empirical social research. Huge amounts of easily accessible and almost ready-to-analyze datasets seem to be a dream coming true for social researchers, especially those who prefer to work with unobtrusively-collected data.

Such large datasets ask for being analysed by mixed methods, to avoid wasting their potential by either choosing a sample or focusing on quantitatively obtained information only. Here come other tools that make the life of a contemporary researcher much more comfortable – software solutions. Of course, in the ideal situation, one could just ‘feed’ all the data to AI and wait for the results, but there are many limitations to such an approach, like usability in specific cases, its accessibility, and, of course, the researcher’s nightmare: a limited project’s budget. Moreover, in the case of smaller datasets, consisting of heterogeneous data, analysis’ results might prove unsatisfactory.

Our research project, an exploratory study on the image of China and the Chinese in Zambia and Angola, included also an analysis of textual media content, namely news articles published in these countries and mentioning China or the Chinese. We obtained a mid-sized dataset, consisting of 2477 articles; the material was very heterogeneous, because of the wide scope of topics covered by the texts and the fact that we analysed content from both English- and Portuguese-language media.

In the course of analysis, we realized that a new method would be needed to obtain the best possible results on the basis of the collected data. After a series of trial-and-error approaches, we managed to develop MIHA – Mixed Integrative Heuristic Approach. The application of this method allowed us to create an exhaustive, contextual and precise keyword dictionary for automated classification of text units as well as a set of sentiment indexes.

We have to admit, that even though we did our best to utilize all the possibilities of the software (Provalis QDA Miner and Wordstat), the dictionary creation process was a time-consuming task since it included reviewing each word of frequency higher or equal to 10 in the whole database.

Our classification, similar to the initial conceptualization of theoretical categories within the grounded theory approach, aimed to explore the most frequent contexts in which China was depicted in African e-media. Each examined word was either added to an exclusion list (words irrelevant from the point of view of the research) or assigned to a chosen – sometimes a newly created – category, together with other words of the same root and all the synonyms.

In the next step, we examined the already categorized keywords in their context to refine the categorization results, mainly by removing those keywords that appeared within the text in unexpected contexts. Most of the categories were re-coded, and some of the keywords were re-assigned in the next steps. This heuristic approach resulted in a set of categories, including ‘emotional’ ones, positive and negative, that later on were used to design sentiment indexes. Our indexes are based on a comparison of the results of quantitative and qualitative analysis and coding. They could be used as a tool for improving dictionary-based sentiment analysis by comparing the results of sentiment analysis performed on the basis of automated coding with manually-coded samples.

We believe that MIHA constitutes a conceptual approach applicable by researchers of various backgrounds in projects focused on investigating the general image presented in textual content, especially in case of mid-sized, heterogeneous data sets. We do not overlook the fact that soon, automated machine learning coding methods will constitute the main approach towards text analysis. However, since such procedures are still imperfect and context-sensitive, we presume that MIHA, consisting of a contextualized dictionary, manual coding of chosen parts of the database and index measurements, could be useful for analysis of data sets related to less common study areas (social groups, languages, geographical areas, subcultures, etc.), in which machine learning-based research would contain a low level of construct validity.

Both the dictionary-creation process and the indexes are described in detail in our paper.

Read the full article in the IJSRM here.