Making Case Study Research More Reliable: How Well Do Information Systems Researchers Follow Best Practices?

🔍 Introduction

Research is how we create new knowledge. But how do we know if research is done properly? This paper looks at a specific type of research called case study research in the field of Information Systems (IS). Case studies help us understand real-world situations by examining businesses, organizations, or specific technologies in detail. However, some researchers have criticized case studies for being inconsistent and lacking structure. This paper examines whether IS researchers are following best practices to make their case studies rigorous and reliable. – (Dubé & Paré, 2003)

📌 What is Case Study Research?

A case study is a research method that involves an in-depth investigation of a subject in its real-life setting. Unlike experiments, where researchers control variables, case studies observe things as they happen naturally. According to Yin (1994), case studies are particularly useful when “the boundaries between phenomenon and context are not clearly evident” (p. 13). This means that when studying complex systems, such as how companies use technology to improve their operations, researchers need to look at the broader environment in which these systems function.

Case research has gained respect in Information Systems because it allows researchers to explore both organizational and technical aspects of IT usage (Benbasat et al., 1987). This method helps uncover patterns, interactions, and trends that might not be visible through other research approaches, making it essential for understanding how businesses adopt and integrate new technologies.

What is a Positivist Case Study? 🧐

A positivist case study is a type of research that aims to be as objective as possible. Researchers using this approach believe that the world follows certain rules that can be discovered through careful observation and analysis. They collect data, test ideas, and try to draw conclusions that can apply beyond the single case being studied.

To be considered rigorous (or trustworthy), a positivist case study should:

• 📌 Have a clear research question that precisely defines what the study aims to investigate and why it is important.
• 🔍 Follow a structured research design, ensuring that all steps—from case selection to data analysis—are systematically planned and executed.
• 📊 Use multiple sources of data such as interviews, documents, and quantitative measures to confirm findings and reduce bias.
• 📝 Explain the methods clearly so others can evaluate the study, replicate its approach, and assess the validity of its conclusions.
• 🏗️ Ensure transparency in data collection and analysis, documenting how decisions were made throughout the research process.
• 📜 Provide justification for case selection, demonstrating why particular cases were chosen and how they contribute to the broader understanding of the subject.

How Was This Study Conducted? 🔬

To assess how well IS case studies follow these principles, the researchers examined 183 case study articles published in seven major IS journals from 1990 to 1999. They looked at how each study was designed, how data was collected, and how conclusions were reached.

Step 1: Selecting the Case Studies 📚

The researchers carefully chose case studies from top IS journals to make sure they were looking at well-regarded work. They only included positivist case studies, meaning that studies using other approaches (like interpretive or critical research) were excluded.

Step 2: Defining What to Look For ✅

They identified key characteristics that a high-quality case study should have. These characteristics were grouped into three areas:

1. Research Design 🏗️: This includes defining the research problem, selecting appropriate cases, and developing a structured plan for the study. It also involves outlining hypotheses, specifying the unit of analysis, and determining whether a single or multiple-case study approach will be used.
2. Data Collection 📥: This step covers how researchers gather data, including the use of interviews, surveys, document analysis, and direct observations. It also considers data triangulation, ensuring reliability and accuracy by using multiple data sources.
3. Data Analysis 📊: This refers to the process of organizing, coding, and interpreting data to identify patterns and relationships. Researchers may use statistical tools, thematic analysis, or comparative analysis to draw meaningful conclusions from their findings.

Each study was examined to see whether it followed best practices in these areas.

What Did the Study Find? 📈

1. Issues with Research Design ❌

• Only 42% of studies had a clear research question, meaning that over half of the studies lacked a well-defined focus, making their objectives ambiguous.
• 40% used multiple case studies, while 60% focused on just one case. While single-case studies can be insightful, multiple-case studies often provide a broader perspective and more generalizable findings.
• Only 8% clearly stated what their unit of analysis was (for example, whether they were studying an entire company or just a department). Without a defined unit of analysis, it becomes difficult to draw meaningful conclusions or compare results across studies.
• Most studies did not explain why they chose a particular case to study, raising concerns about selection bias. Without a justification for case selection, the research findings may not be applicable to a wider audience.

2. Problems with Data Collection 🛑

• 42% of studies did not fully explain how they collected their data, making it hard to judge their reliability. Clear documentation of data collection methods ensures transparency and allows others to replicate the study.
• Only 30% used multiple sources of data to confirm findings (a process called triangulation 🔄). Triangulation helps validate findings by cross-checking information from different sources, reducing biases and increasing credibility.
• Most studies relied heavily on interviews, but many did not explain how interviewees were selected or how the interviews were conducted. Without clarity on selection criteria and interview procedures, it is difficult to assess the reliability of the insights gathered.

3. Weaknesses in Data Analysis 📉

• Only 23% explained how they analyzed their data, leaving gaps in understanding how conclusions were drawn. Properly documenting data analysis methods enhances credibility and allows for verification of results.
• Most studies did not compare their findings with existing research, which is important for understanding broader patterns. Situating findings within the existing body of knowledge strengthens their relevance and contributes to academic discourse.

🚀 Recommendations for Better Case Studies

The authors suggest several ways to improve IS case study research:

1. 📝 Clearly state research questions – A well-defined research question helps in selecting appropriate case studies, guiding data collection, and ensuring a structured analysis (Dubé & Paré, 2003, p. 608). A clear research focus enhances coherence and facilitates meaningful insights.
2. 📜 Provide more details on how data is collected and analyzed – Transparency makes studies easier to evaluate and replicate. Researchers should describe their data collection methods in detail, including interview protocols, observational techniques, and document analysis strategies (Benbasat et al., 1987; Yin, 1994). Providing explicit details ensures that other researchers can assess the reliability of findings.
3. 📚 Use multiple sources of evidence – This increases the reliability of findings by cross-verifying information from different sources. Using interviews, documents, and observations enhances the validity of conclusions by reducing biases and improving comprehensiveness (Dubé & Paré, 2003, p. 612; Yin, 1994).
4. 📊 Consider using both qualitative and quantitative data – Combining different types of data strengthens conclusions by capturing both subjective experiences and objective measures. Mixed-method approaches provide a more holistic understanding of complex IS phenomena, improving explanatory power and generalizability (Eisenhardt, 1989; Dubé & Paré, 2003, p. 615).
5. 🔄 Use triangulation – Checking findings from multiple angles improves accuracy by cross-validating results through different methodologies. Triangulation strengthens theoretical insights, ensures consistency, and enhances methodological rigor, making findings more robust and credible (Dubé & Paré, 2003, p. 613; Eisenhardt, 1989).

🏆 Conclusion

Case studies are an important research method in IS, but they must be conducted with rigor and transparency. This study found that while there has been some progress, there is still room for improvement. By following best practices, IS researchers can produce more reliable and impactful case study research.