featured, Notebook

What is Ragin’s Indirect Method of Calibration?

Guest post by Preya Bhattacharya

In the last few years, Qualitative Comparative Analysis (QCA) has become one of the most important comparative data analysis approaches in the field of social sciences. QCA is useful if you have a small to medium number of cases, you are collecting data from multiple levels, and trying to analyze the causal pathway through which a condition or combination of conditions impacts an outcome, cross-cases and/or across time.

In my doctoral dissertation on “Can Microfinance Impact National Economic Development? A Gendered Perspective,” I applied panel data QCA as my causal analysis method, because I had a small number of case studies, and I wanted to analyze how microfinance institutions impact the economic development of women at the level of the national economy, in the context of former Yugoslavian countries (Bhattacharya 2020). But, during my data analysis phase, I came across this concept of “calibration.” At that time, I did not have any formal training in QCA, and it was a little bit difficult for me to understand and apply this concept. So, through this research note, I hope to guide future researchers in their own work on QCA, by explaining the process of calibrating an interval variable.

In my article, I have first defined the concept of calibration, and then applied my own data to demonstrate the steps of data calibration. As described in my article, calibration can be defined as a data transformation process, in which researchers try to transform the data that they have collected into set-membership scores, of 0/1 (crisp-set), or a range of 0 to 1 (fuzzy-sets). This data transformation process helps a researcher interpret their data in the context of the cases studied, and it depends on a variety of factors. As a result, the process of calibration might differ from one variable to another, even within the same dataset.

To demonstrate this, I have described the data calibration process for an interval variable, that does not have established prior theoretical cut-off points. I have divided my calibration process into five main steps:

  • First, define the set
  • Second, visualize the data distribution
  • Third, identify qualitative changes in data distribution
  • Fourth, assign qualitative categories
  • Fifth, transform these categories into set membership scores, ranging from 0 to 1 (fuzzy set) or 0/1 (crisp)

Finally, I have discussed the issue of robustness in calibration, and how researchers can ensure that their calibrated data matches with the reality of the cases they have studied. By describing these steps, I hope to help future researchers in their own process of calibrating interval variables in QCA.

I would also like to thank the 2019 Southern California QCA Workshop, organized by Dr. Charles Ragin (University of California, Irvine) and Dr. Peer Fiss (University of Southern California), for introducing me to the world of Qualitative Comparative Analysis and set theoretic research approach!

Read the full article here

featured, Notebook

25th Anniversary Editorial appendix on methods and data

By Iasonas Lamprianou

One of the sections of the 25th Anniversary Editorial of the International Journal of Social Research Methodology (IJSRM), presents the thematic trends in published contributions, for the whole period of 25 years of the journal’s life. Investigating the thematic trends in published contributions was not an easy task, not only because of the huge number of published papers, but also because of various technical details (for example, the published papers were not accompanied by keywords in the first volumes). 

Coding and charting the thematic trends of published papers proved to be a very laborious task and the workload was shared between four researchers. The aim of this document is to help interested readers understand the nature and the structure of the dataset. This could be useful to those interested in extending our own analysis, which had to be confined in the limited space of an Editorial.  

Methodology 

It is important for prospective users of the dataset to understand how the published content was coded and how reliable the coding was. 

Three coders worked in parallel for three weeks, under the supervision of an experienced researcher. They coded each of the contributions published by IJSRM, not only in the first 25 issues, but also in the ‘latest papers’ section of the journal’s web page, which includes papers which have not yet been assigned to specific volumes/issues.  

The three coders and the experienced researcher, developed a coding scheme, which included all the necessary variables to be coded in an Excel file. Through long online meeting, the group discussed the aims of the coding exercise, the structure and content of the coding scheme etc. The group coded a number of common papers to confirm that they interpreted the coding scheme in the same way. Regular online meetings and email exchanges were necessary to discuss various issues which emerged and to keep the coders in sync. To make sure that the coders did not ‘drift’ over time, they were instructed to ‘blindly’ re-code 5%-10% of each other’s excel file incrementally (every few days). The coders were in communication all the time and they exchanged emails where they would update each other about coding difficulties in order to remain in sync. As a result of this procedure, various issues came to the surface (e.g. there were many papers which could not be easily categorized as Qualitative, Quantitative or Mixed, so a new category was created; more information later). 

When all the coding was completed, the experienced researcher re-coded blindly 50 random papers – around 5% of the total number of papers in the database – but no major discrepancies were detected (for example, in one case, the number of views was miskeyed as ‘867’ instead of ‘861’) .  

Overall, there is no reason to believe that there is widespread bias or errors in the data. We expect the dataset to give a fair interpretation of what has been published in the journal in the last 25 years of its life. 

Variables in dataset 

The dataset includes the following variables: 

Vol Volume 
No Issue number 
Title The title of the paper (no coding, it was just copied and pasted) 
Abstract The abstract of the paper (no coding, it was just copied and pasted) 
Keywords The keywords of the paper (no coding, it was just copied and pasted) 
Paradigm Main research paradigm. Takes four values: Qualitative, Quantitative, Mixed Methods, General/Other.  Note: The General/Other category refers to papers which cannot be described accurately by the three other codes (Qualitative, Quantitative, Mixed Methods) 
Views Number of views (as reported on the journal’s web page) 
CrossRef Number of CrossRef citations (as reported on the journal’s web page) 
Altmetric Altmetric count (as reported on the journal’s web page) 

Data filtering 

The original dataset consisted of 1043 records, but book reviews, editorials and other small items were removed, resulting to a ‘clean’ dataset of 924 published papers (including ‘Research Notes’). 

Dataset format 

The dataset is provided as an R data frame, with the name EditorialData.Rda

You can download the data files here.

featured, Notebook

A virtual collection to celebrate 25 volumes of IJSRM 

The International Journal of Social Research Methodology is celebrating its 25th anniversary!!  In that period, we have become a leading methods journal publishing high quality contributions across the methodological field – both qualitative and quantitative, and including mixed, multi-media, comparative and simulation methods. 

To mark the occasion, we have gathered together a series of methods discussions that have been published in the journal, and our publisher, Routledge, is making them freely available as a collection.  

Choosing which articles to include in our anniversary virtual collection was a hard task.  We inevitably had to leave some important and favourite pieces aside.  The collection below includes contributions that we felt represented the range of methodological articles that we publish in IJSRM, a selection of early career prize winning articles, influential pieces and discussions that deserve more attention for their contributions, and individual editors’ personal choices.  

The methodological reach of our anniversary, then, ranges across survey non-response, behavioural experiments, quantitative pedagogy, the Delphi method, the problem-centred expert interview, the self-interview, narrative and computerised text analysis, qualitative methods transformations, anonymisation, triangulation of perspectives, indigenous data sovereignty, post-humanism, and researcher safety. 

We hope that you enjoy our selection. You can access it at:  

https://www.tandfonline.com/journals/tsrm20/collections/TSRM_25th_Anniversary_SI

Rosalind Edwards, Jason Lamprianou, Jane Pulkingham, Malcolm Williams 

IJSRM Co-Editors 

featured, Notebook

Seeking alternatives: A reflection on conducting online interviews with disabled young people during the COVID-19 pandemic

Angharad Butler-Rees and Stella Chatzitheochari

While scholars have increasingly documented and reflected on their approaches to conducting research during the pandemic, little is still known about the impact of social distancing measures on qualitative research with disabled young people.

Our new paper provides a methodological reflection on undertaking qualitative research with disabled young people as part of the Educational Pathways and Work Outcomes longitudinal study. Our study started in March 2021 during the third national lockdown in England. Due to social distancing measures in place at the time of commencing, it was necessary to revise our original plans to conduct face-to-face interviews with disabled young people and conduct online interviews instead. We conducted a total of 35 online interviews with autistic, dyslexic, and physically disabled young people aged 15-16 years old.

Ensuring Accessibility

The internet has long been deemed as a potentially empowering platform for disabled people, connecting isolated individuals and ensuring access to social, civic and community life. Our focus on young people was particularly useful, as this population tends to be very comfortable with the use of technology. The extensive periods of enforced home-learning during the covid-19 pandemic had further increased young people’s familiarity with online communication platforms, rendering the idea of online interviews a far less daunting prospect. However, it is worth noting that online tools can also present a number of accessibility barriers e.g., poor text layout, little colour contrast and limited keyboard functionality. These were important factors to consider when designing online interviews for our project.

Accessibility has to be incorporated into every part of the research process when working with disabled people. To put participants at ease prior to their interview, we sent them participant information packs as well as a short video of the interviewer introducing themselves and the study. Familiarity with the researcher was greatly valued for autistic young people, making them feel more at ease. Previous literature has suggested that autistic young people may be disconcerted or unresponsive in encounters with strangers, so building a degree of initial trust and rapport was of upmost importance for successful interviewing. In line with this, we also arranged online pre-interview meetings with participants and their parents to build rapport.

Pre-interview meetings also helped us ensure that any accessibility requirements were put into place. We asked participants to choose their preferred communication platform. Several participants opted to use assistive software during their interview e.g., enabling captioning, magnification or modifying volume. Other adjustments included allowing participants to sit off screen or to keep their cameras off while the interviewer remained visible. This made interview far less intrusive and anxiety provoking and was greatly valued by autistic participants. Other adjustments included the presence of a guardian that could provide practical assistance or emotional support, simplification of interview questions, as well as collection of data over several interviews as opposed to one. Overall, we felt that these adjustments made interviews considerably more accessible for disabled young people, ultimately giving voice to a population who may not always be amenable to conventional face-to-face interviewing methods that can be experienced as more restricting and demanding.  

Challenges during Interviewing

While some young people were very comfortable in engaging with the interview process and narrating their lived experiences, others were far more hesitant, requiring regular prompting and reassurance. The online medium made this slightly more challenging for the interviewer, with prompting and encouragement occasionally leading to cross-talking. It was also notably more difficult to interpret emotion and body language online, while the loss of internet connection at times affected the flow of the interview.

Another challenge was the difficulty in maintaining participants’ attention. We sometimes felt that the lack of physical presence meant that participants were far easier distracted by being in their homes, e.g., checking their mobiles, playing with family pet. However, we also recognise that this may be interpreted in a different manner: Indeed, it may be indicative of a greater share of power afforded to disabled young people in online settings. Overall, we did not feel that such distractions affected the quality of our data collection and think that the physical distance may have aided disclosure of personal experiences. A feedback survey confirmed that participants enjoyed the use of the online medium, with the vast majority requesting online interviews for the future waves of data collection.

A final note on accessibility

Our reflections may not speak to studies seeking to interview disabled young people with different accessibility needs such as speech or communication difficulties (e.g., stammering). These participants may find online communication more difficult due to possible misunderstanding and difficulties in lip reading and interpretation. Similarly, it is worth noting that interviews can be experienced as particularly exhausting by some disabled young people, whether face to face or online, preventing them from taking part. Researchers may consider offering alternatives such as email interviews alongside conventional online interviews.

Looking ahead

Our overall experience with online interviews was very positive. We were privileged to be able to access disabled young people’s lived experiences during an unprecedented period of global disruption. Notwithstanding the challenges mentioned above, we feel that online interviewing is a valuable tool that should not be viewed as second best to face-to-face conversational methods. We therefore encourage researchers to explore the use of online methods, especially with regards to young and disabled populations.

Read the full article here: Giving a Socially Distanced Voice to Disabled Young People: Insights from the Educational Pathways and Work Outcomes Longitudinal Study