Validity and Circumstances

"Validity is not a commodity that can be purchased with techniques...Rather, validity is like integrity, character, and quality, to be assessed relative to purposes and circumstances"

(Brinberg & McGrath, quoted in Miles & Huberman, p. 39)

I liked this quote; it seemed somehow reassuring. Our quick ramble through various approaches to validity in qualitative research left me feeling like it was a hodge podge of or wish list of "ideal types" that were each in their own way trying to define a multi-dimensional, historically contingent, ideosyncratic process as a codifiable regime of trustworthy inferences. That's a tall order.

The typologies of validity are useful, however, foremost for directing our attention to the many ways in which our research can fall short in one or more dimensions. Thinking through the details of what makes for authetic findings is critical to the research enterprise. Valid inferences are the goal, but the pitfalls loom everywhere (errors of omission or characterization, for example).

I do wonder about the refrain that holds qualitative studies are doomed to have limited generalizability. This overstates the case, don't you think? What kinds of qualitative studies might result in findings that are generalizable to a wider popluation?

KKV: "The procedures are public"

"We seek not dogma, but disciplined thought."
(KKV, Designing Social Inquiry, p.7)

Item 2, "The procedures are public," in the 4 point list in section 1.1.2 states concisely one of the central pillars of scientific inquiry. Whether it accurately describes how scientists do their work varies from case to case. As we explore what it means to be a good qualitative scientist, it will be important to discipline our thought by constantly asking exactly how far this maxim can be taken. KKV are correct that "investigators often take down the scaffolding after putting up their intellectual buildings," and I believe I can personally be more fluent with qualitative methods if I attend to the full and orderly preservation of just enough of the scaffolding to make my inferences more compelling to a wider audience. The tough question is: how much is enough?

Where practical, I have tried to make the data on which my major e-rulemaking stakeholder report and other scholarly papers are based available. To a minimal extent, the procedures used to reach inferences found in my work have been described in turgid terms, usually reflecting an over-arching tension that my methods are never going to be good enough for some in my profession or even defensible enough to merit the best of my scholarly prose.

But should I do more? I teach my coders and clients working with QDAP to memo extensively about the process and struggle of coming to grips with a qualitative project. My own memoing has always been rather limited nonetheless and I sometimes find it tough to read all the memos on process that a single project generates. It is even more difficult to know exactly how to incorporate that "procedure" in a public manner.

One idea gaining currency is to share not only the primary data with other researchers but also the completed coding and entire memo trail. One can imagine posting archives of coded data for other researchers to examine or attempt to replicate. I wonder: Would they do so?