atzis RE. One article (SHI139) c

2021/05/21

Boyatzis RE. One article (SHI139) clearly contrasted nomothetic (statistical) generalisation to idiographic generalisation, arguing that the lack of statistical generalizability does not nullify the ability of qualitative research to still be relevant beyond the sample studied. that moved beyond description. Others investigated PhD qualitative studies [31] and grounded theory studies [35]. These are described below and illustrated with excerpts from relevant articles. There was a significant association between the journal a paper was published in and the provision of a justification (2 (2)=23.83, p<.001). A wide range of variants of saturation was evident corroborating the proliferation of the meaning of the term [49] and reflecting different underlying conceptualisations or models of saturation [20]. Moreover, a high degree of consensus had begun to emerge among those interviewed, and while it is always difficult to judge at what point theoretical saturation has been reached, or how many interviews would be required to uncover exception(s), it was felt the number was sufficient to satisfy the aims of this small in-depth investigation (Strauss and Corbin 1990). Recruitment continued until data saturation was reached, defined as the point at which no new themes emerged. The current study has a number of limitations. When authors justified their sample size, our findings indicate that sufficiency was mostly appraised with reference to features that were intrinsic to the study, in agreement with general advice on sample size determination [4, 11, 36]. (BJHP35). Saturation was the most commonly invoked principle (55.4% of all justifications) deployed by studies across all three journals to justify the sufficiency of their sample size. How theory is used and articulated in qualitative research: development of a new typology. Google Scholar. This study could be criticised for using a small and unrepresentative sample. That ones sample might be thought small was often construed as a limitation couched in a discourse of regret or apology. [23] recommend reordering and re-analysing interviews to confirm saturation. Accessed 17 May 2018. Although IPA generally involves intense scrutiny of a small number of transcripts, it was decided to recruit a larger diverse sample as this is the first qualitative study of this population in the United Kingdom (as far as we know) and we wanted to gain an overview. Twenty-three people with type I diabetes from the target population of 133 (i.e. Expert Rev Med Devices. (BMJ08). (BJHP09). Studies that employed individual interviews (n=30) had an average sample size of 45 individuals and none of these explicitly reported whether their sample size sought and/or attained saturation. Finally, three SHI articles explained their sample size with reference to practical aspects: time constraints and project manageability (SHI56), limited availability of respondents and project resources (SHI131), and time constraints (SHI113). The level of analysis i.e. Int J Soc Res Methodol. 2001;24(3):23040. In line with previous studies [22, 30, 33, 34] the findings demonstrate that reporting of sample size sufficiency is limited; just over 50% of articles in the BMJ and BJHP and 82% in the SHI did not provide any sample size justification. Interestingly, BMJ13 included data in the analysis beyond the point of saturation in search of unusual/deviant observations and with a view to establishing findings consistency. In light of the growing methodological evidence in the area, transparent presentation of such evidence-based judgement is crucial and in time should surely obviate the seemingly routine practice of citing the small size of qualitative samples among the study limitations. Qual Health Res. (SHI81). Furthermore, data collection ceased on pragmatic grounds rather than at the point when saturation point was reached. statement and Nurs Health Sci. 0000003258 00000 n (SHI57). Qual Quant. 0000010124 00000 n volume18, Articlenumber:148 (2018) (SHI115). Experts have also offered numerical guidelines tailored to different theoretical and methodological traditions and specific research approaches, e.g. Bowen GA. Naturalistic inquiry and the saturation concept: a research note. This analysis resulted in two overarching thematic areas; the first concerned the variation in the characterisation of sample size sufficiency, and the second related to the perceived threats deriving from sample size insufficiency. 2010;5(1). 2013;13(2):1907. 2010. We stopped recruitment when we reached 3035 interviews, owing to the depth and duration of interviews, richness of data, and complexity of the analytical task. 2017;27(4):591608. Overall, it is clear that the concept of saturation encompassed a wide range of variants expressed in terms such as saturation, data saturation, thematic saturation, theoretical saturation, category saturation, saturation of coding, saturation of discursive themes, theme completeness. Naturalistic inquiry. An open letter to the BMJ editors on qualitative research. 2015;18(6):6856. studies that quantified their qualitative data were excluded). We drew participants from a list of prisoners who were scheduled for release each week, sampling them until we reached the target of 35 cases, with a view to achieving data saturation within the scope of the study and sufficient follow-up interviews and in line with recent studies [810]. All other authors have no competing interests to declare. This gave a target sample of 75100 patients and 3550 caregivers overall. Descriptions of sampling practices within five approaches to qualitative research in education and the health sciences. This research was initially conceived of and partly conducted with financial support from the Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH) programme (EP/F063822/1 and EP/G012393/1). Whilst the themes from the initial interviews seemed to be replicated in the later interviews, further interviews may have identified additional themes or provided more nuanced explanations. Newbury Park, CA: Sage; 1990. van Rijnsoever FJ. On those occasions that authors did not simply cite their small sample size as a study limitation but rather continued and provided an account of how and why a small sample size was problematic, two important scientific qualities of the research seemed to be threatened: the generalizability and validity of results. Constantinou CS, Georgiou M, Perdikogianni M. A comparative method for themes saturation (CoMeTS) in qualitative interviews. PLoS Med 2009; 6(7): e1000097. data, theoretical, thematic saturation) (SHI04; SHI13; SHI30) whilst another four articles explicitly stated that they achieved saturation (SHI100; SHI125; SHI136; SHI137). Int J Soc Res Methodol. 2004;31(1):921. (DOCX 12 kb), List of eligible articles included in the review (N=214). Book Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, et al. (BJHP53). Field Methods. Qual Quant. Dai NT, Free C, Gendron Y. Interview-based research in accounting 20002014: a review. Similarly, Britten [40] notes that large interview studies will often comprise of 50 to 60 people. Those reviewing papers have a vital role in encouraging transparent study-specific reporting. Morse [11] posits that the more useable data are collected from each person, the fewer participants are needed. Green J, Thorogood N. Qualitative methods for health research. Examining additional disciplines (e.g. London: Sage; 2006. Correspondence to However, the emphasis changes from an in-depth individualistic analysis to one in which common themes from shared experiences of a group of people can be elicited and used to understand the network of relationships between themes that emerge from the interviews. 2008;8(1):13752. As many of the issues explored are of relevance to mental health stigma more generally, it may also provide insights into adult engagement in services. Theoretical sampling describes an iterative process of data collection, data analysis and theory development whereby data collection is governed by emerging theory rather than predefined characteristics of the population. (BJHP04). Further, these data do not need to be statistically generalisable for us to draw inferences that may advance medicalisation analyses (Charmaz 2014). Because the order in which interviews are analysed can influence saturation thresholds depending on the richness of the data, Constantinou et al. Nelson J. 2013;20(5):29. the point at which no additional issues are identified) was achieved at 9 interviews, but meaning saturation (i.e. Qualitative inquiry and research design: choosing among five approaches. Forum Qual Soc Res 2010;11(3):8. http://nbn-resolving.de/urn:nbn:de:0114-fqs100387. In the BMJ, one article (BMJ15) appealed to pragmatic reasons, relating to time constraints and the difficulty to access certain study populations, to justify the determination of its sample size. 2010;32(7):105971. (BJHP27). New York: Macmillan; 1986. p. 11961. An experiment with data saturation and variability. Whilst the concerns about internal validity might be legitimate to the extent that qualitative research projects, which are broadly related to realism, are set to examine phenomena in sufficient breadth and depth, the concerns around generalizability revealed a conceptualisation that is not compatible with purposive sampling. Similarly, whilst methodological research that seeks to demonstrate the achievement of saturation is invaluable since it explicates the parameters upon which saturation is contingent and indicates when a research project is likely to require a smaller or a larger sample [e.g. Seven of the 19 SHI articles cited references to support their position on saturation (see AdditionalFile4 for the full list of citations used by articles to support their position on saturation across the three journals). 2005;15(9):127788. No new topics were identified following analysis of approximately two thirds of the interviews; however, all interviews were coded in order to develop a better understanding of how characteristic the views and reported behaviours were, and also to collect further examples of unusual/deviant observations. BJHP studies provided a sample size justification significantly more often than would be expected (z=2.9); SHI studies significantly less often (z=2.4). Other articles appeared to accept and acknowledge that their sample was flawed because of its small size (as well as other compositional deficits e.g. Sample adequacy in qualitative inquiry pertains to the appropriateness of the sample composition and size. It has previously been recommended that qualitative studies require a minimum sample size of at least 12 to reach data saturation (Clarke & Braun, 2013; Fugard & Potts, 2014; Guest, Bunce, & Johnson, 2006) Therefore, a sample of 13 was deemed sufficient for the qualitative analysis and scale of this study. 2001;33(4):18492. Indeed, recent research demonstrates the greater efficiency of purposive sampling compared to random sampling in qualitative studies [9], supporting related assertions long put forward by qualitative methodologists. Those who characterised their sample as small connected this to the limited potential for generalization of the results. (BJHP31). It is an important consideration in evaluations of the quality and trustworthiness of much qualitative research [1] and is implicated particularly for research that is situated within a post-positivist tradition and retains a degree of commitment to realist ontological premises in appraisals of validity and generalizability [2,3,4,5]. authors, title, journal, year of publication etc. Borsci S, Macredie RD, Martin JL, Young T. How many testers are needed to assure the usability of medical devices? Guetterman TC. OReilly M, Parker N. Unsatisfactory saturation: a critical exploration of the notion of saturated sample sizes in qualitative research. The Abstract, Methods and Discussion (and/or Conclusion) sections of each article were examined by one author (KV) who extracted all the relevant information. Hennink, Kaiser and Marconis [29] methodological study sheds further light on the problem of specifying and demonstrating saturation. This research, while limited in size, has sought to capture some of the complexity attached to mens attitudes and experiences concerning incomes and material circumstances. 0000010772 00000 n The principle of saturation was the most commonly invoked argument [22] accounting for 55% of all justifications. California Privacy Statement, In practice, the number of individuals volunteering to participate dictated when recruitment into the study ceased (15 young people, 15 parents). BMC Medical Research Methodology Although our sample size was sufficient for this exploratory study, a more diverse sample including participants with lower socioeconomic status and more ethnic variation would be informative. 0000034683 00000 n Achieving maximum variation sampling in terms of specific interviewee characteristics determined and explained the sample size of two BMJ studies (BMJ02; BMJ16 see extract in section Meet research design requirements). However, 40% did not report the number of participants. Their analysis of interview data showed that code saturation (i.e. (DOCX 14 kb). PLoS One. National Centre for Social Research 2003 https://www.heacademy.ac.uk/system/files/166_policy_hub_a_quality_framework.pdf Accessed 11 May 2018. As such, care should be taken not to overstate the findings. 0000006727 00000 n London: Sage; 1985. Within each of the age-stratified groups, interviews were randomly sampled until saturation of discursive patterns was achieved. Mixed method studies and articles reporting more than one qualitative method of data collection (e.g. Five BJHP articles (BJHP28; BJHP38 see extract in section Qualities of the analysis; BJHP46; BJHP47; BJHP50 see extract in section Saturation) and one SHI paper (SHI73) relied on citing existing sample size guidelines or norms within research traditions to determine and subsequently defend their sample size (7.2% of all justifications). This study had a large diverse sample, recruited through a range of locations and used in-depth interviews which enhance the richness and generalizability of the results. The discovery of grounded theory: strategies for qualitative research. Carlsen B, Glenton C. What about N? One BJHP paper argued that the sample size was appropriate for the exploratory nature of the study (BJHP38). Authors expressed uncertainty about the degree of confidence in particular aspects or patterns of their results, primarily those that concerned some form of differentiation on the basis of relevant participant characteristics. non-representativeness, biases, self-selection) or anticipated that they might be criticized for their small sample size. Onwuegbuzie AJ. This was directly copied from the articles and, when appropriate, comments, notes and initial thoughts were written down. 2011;11(1):26. small, in-depth investigation), aims and nature (i.e. Nevertheless, our study is the first to provide some comparative insights on the basis of disciplines that are differently attached to the legacy of positivism and analysed literature published over a lengthy period of time (15years). One article stated that thematic saturation was anticipated with its sample size (SHI94). Cookies policy. (SHI127). Saturation in qualitative research: exploring its conceptualization and operationalization. Recommendations for designing and reviewing qualitative research in psychology: promoting methodological integrity. generalisation that can be made from and about cases [5]). How many interviews are enough to identify metathemes in multisited and cross-cultural research? Using this approach it was possible to more carefully explore childrens discourse about the I, agency, relationality and power in the thematic areas, revealing the subtle discursive variations described in this article. For example, Green and Thorogood [38] maintain that the experience of most qualitative researchers conducting an interview-based study with a fairly specific research question is that little new information is generated after interviewing 20 people or so belonging to one analytically relevant participant category (pp. 2077 0 obj <> endobj xref 2077 38 0000000016 00000 n JB and TY conceived the study; KV, JB, and TY designed the study; KV identified the articles and extracted the data; KV and JB assessed eligibility of articles; KV, JB, ST, and TY contributed to the analysis of the data, discussed the findings and early drafts of the paper; KV developed the final manuscript; KV, JB, ST, and TY read and approved the manuscript. Quality and trustworthiness in qualitative research in counseling psychology. Similarly if published in the BMJ, the odds of a study justifying its sample size were 4.5 times higher than in the SHI. We believe that such an analysis would yield useful insights as it links the methodological issue of sample size to the broader philosophical stance of the research. November 2016. https://ssrn.com/abstract=2711022 or https://doi.org/10.2139/ssrn.2711022. A thematic analysis [55] was then performed on all scientific narratives that discussed or commented on the sample size of the study. Spencer L, Ritchie J, Lewis J, Dillon L. Quality in qualitative evaluation: a framework for assessing research evidence. 0000012498 00000 n 1530 interviews) projects. : expert voices and early career reflections on sampling and cases in qualitative research. Indicative of an apparent oscillation of qualitative researchers between the different requirements and protocols demarcating the quantitative and qualitative worlds, there were a few instances of articles which briefly recognised their small sample size as a limitation, but then defended their study on more qualitative grounds, such as their ability and success at capturing the complexity of experience and delving into the idiographic, and at generating particularly rich data. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. 0000001056 00000 n On the basis of the researchers previous experience and the literature, [30, 31] we estimated that recruitment of 1520 patients at each site would achieve data saturation when data from each site were analysed separately. 2003;15(4):386404. We also extracted or coded quantitative data regarding the following aspects: Presence of sample size justification(s) (Yes/No), Presence of a particular sample size justification category (Yes/No), and, Number of sample size justifications provided. He is unaware of any direct conflict of interest with respect to this paper. The type of generalisation aspired to (BJHP48) was not further specified however. Clarity and sharpness of emerging themes and alignment with previous relevant work were the arguments employed to warrant the validity of the results. 2015;26:175360. Similarly, BJHP04, which did not provide a sample size justification, argued that it targeted a larger sample size in order to eventually recruit a sufficient number of interviewees, due to anticipated low response rate. Rather, sample size seemed to correlate with factors such as the journal of publication or the region of study (US vs Europe vs Asia). Qual Psychol. group interviews, structured interviews and e-mail interviews over a period of time were excluded), and the data had to be analysed qualitatively (i.e. We set a target of seven to 10 caregivers per site because of time constraints and the anticipated difficulty of accessing caregivers at some home based care services. Though not always explicitly articulated to what form of generalisation the articles referred to (see BJHP09), generalisation was mostly conceived in nomothetic terms, that is, it concerned the potential to draw inferences from the sample to the broader study population (representational generalisation see BJHP31) and less often to other populations or cultures. In terms of principles, Lincoln and Guba [17] proposed that sample size determination be guided by the criterion of informational redundancy, that is, sampling can be terminated when no new information is elicited by sampling more units. Incomplete and imprecise sample size reporting is commonly pinpointed by these investigations whilst assessment and justifications of sample size sufficiency are even more sporadic. The notion of saturation was also invoked by 11 out of the 51 most highly cited studies that Guetterman [27] reviewed in the fields of education and health sciences, of which six were grounded theory studies, four phenomenological and one a narrative inquiry. Additionally, this study attempted to expand previous empirical investigations by examining how qualitative sample sizes are characterised and discussed in academic narratives. a 15-year review period). Qual Health Res. Nevertheless, this more formulaic approach raised criticisms relating to assumptions about the conceptual [43] and ontological status of themes [44] and the linearity ascribed to the processes of sampling, data collection and data analysis [45]. Psychol Health. These articles, all published in the discipline of sociology, defended their findings in terms of the possibility of drawing logical and conceptual inferences to other contexts and of generating understanding that has the potential to advance knowledge, despite their small size. The views expressed are those of the authors alone. 0000001969 00000 n Operationalising data saturation for theory-based interview studies. Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period, https://doi.org/10.1186/s12874-018-0594-7, https://www.heacademy.ac.uk/system/files/166_policy_hub_a_quality_framework.pdf, https://doi.org/10.1007/s11135-017-0574-8, http://nbn-resolving.de/urn:nbn:de:0114-fqs1502256, http://nbn-resolving.de/urn:nbn:de:0114-fqs100387, http://www.joaag.com/uploads/5_1__Research_Note_1_Thomson.pdf, http://eprints.ncrm.ac.uk/2273/4/how_many_interviews.pdf, http://creativecommons.org/licenses/by/4.0/, http://creativecommons.org/publicdomain/zero/1.0/, bmcmedicalresearchmethodology@biomedcentral.com. Data collection ceased on pragmatic grounds rather than when no new information appeared to be obtained (i.e., saturation point). 0000029575 00000 n 0000050482 00000 n Qual Health Res. (BMJ02). Similarly, BJHP38 (see extract in section Qualities of the analysis) claimed that its sample size was within the range of sample sizes of published studies that use its analytic approach. The material collected was considerable and, given the focused nature of the study, extremely detailed. Department of Psychology, University of Bath, Building 10 West, Claverton Down, Bath, BA2 7AY, UK, School of Psychology, Newcastle University, Ridley Building 1, Queen Victoria Road, Newcastle upon Tyne, NE1 7RU, UK, Department of Computer Science, Brunel University London, Wilfred Brown Building 108, Uxbridge, UB8 3PH, UK, You can also search for this author in The study has two principal limitations. Soc Sci Med. It must be noted that samples are small and whilst in both groups the majority of those women eligible participated, generalizability cannot be assumed. Anyone you share the following link with will be able to read this content: Sorry, a shareable link is not currently available for this article. The past decade has seen a growing appetite in qualitative research for an evidence-based approach to sample size determination and to evaluations of the sufficiency of sample size. Figure1, a PRISMA flow diagram [53], shows the number of: articles obtained from the searches and screened; papers assessed for eligibility; and articles included in the review (AdditionalFile2 provides the full list of articles included in the review and their unique identifying code e.g. : a review of qualitative interviews in IS research. Emmel N. Themes, variables, and the limits to calculating sample size in qualitative research: a response to Fugard and Potts. 0000002606 00000 n Mason M. Sample size and saturation in PhD studies using qualitative interviews. A non-parametric test of difference for independent samples was performed since the variable number of interviews violated assumptions of normality according to the standardized scores of skewness and kurtosis (BMJ: z skewness=3.23, z kurtosis=1.52; BJHP: z skewness=4.73, z kurtosis=4.85; SHI: z skewness=12.04, z kurtosis=21.72) and the Shapiro-Wilk test of normality (p<.001). 2017;17(5):57188. 2nd ed. To strengthen the explanation for choosing a non-normative sample size, previous IPA research citing a similar sample size approach is used as a precedent. 17.3%) consented to participate but four did not then respond to further contacts (total N=19). If an article was published in the BJHP, the odds of providing a justification were 4.8 times higher than if published in the SHI. 0000007668 00000 n 0000002135 00000 n 2017;4(1):222. Thus, participants may not be representative of couples that experience postnatal PTSD. Data were saturated. ux2Mn[m}2M(q}[CFeY#5`sf\aI>aX}A6F56Tz|~.v)^:c35~nW|E$Kf4K]bmq42yG|a/sk An examination of 83 qualitative interview studies in leading information systems journals [30] indicated little defence of sample sizes on the basis of recommendations by qualitative methodologists, prior relevant work, or the criterion of saturation. BJHP47 explicitly renounced the notion of saturation within an IPA approach. Accessed 17 May 2018. (BJHP48). Despite these challenges, various conceptual developments have sought to address this issue, with guidance and principles [4, 10, 11, 13,14,15,16,17,18,19,20], and more recently, an evidence-based approach to sample size determination seeks to ground the discussion empirically [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]. Accessed 17 May 2018. The significance of saturation. Qualitative research practice: a guide for social science students and researchers. For example, SHI139, in line with the justification of thematic saturation that it offered, expressed trust in its sample size sufficiency despite the poor response rate. Sample sizes of BMJ and SHI articles did not differ significantly from each other. BMJ. [18] introduced the concept of information power as a pragmatic guiding principle, suggesting that the more information power the sample provides, the smaller the sample size needs to be, and vice versa. Twenty seven agreed and 21 (aged 2164, median 40) were interviewed before data saturation was reached (one tape failure meant that 20 interviews were available for analysis). One paper claimed theoretical saturation (BMJ06) conceived as being when no further recurring themes emerging from the analysis whilst another study argued that although the analytic categories were highly saturated, it was not possible to determine whether theoretical saturation had been achieved (BMJ04). 1995;17(1):89113. 2013;54(1):1122. 2017. https://doi.org/10.1007/s11135-017-0574-8. Finally, SHI112 argued that once it had achieved saturation of discursive patterns, further sampling was decided and conducted to check for consistency of the findings. Qualitative sample sizes were predominantly and often without justification characterised as insufficient (i.e., small) and discussed in the context of study limitations. Pairwise comparisons following a significant Kruskal-WallisFootnote 2 test indicated that the studies published in the BJHP had significantly (p<.001) smaller samples sizes than those published either in the BMJ or the SHI. 2018;13(Suppl 1):14. 2016;18(1):47. (BJHP19). Beyond such guidance, specific numerical recommendations have also been proffered, often based on experts experience of qualitative research.