Walden University Wk 8 Qualitative Research Designs Discussion

Research Theory, Design, and Methods
Walden University
Trustworthiness
(Lincoln & Guba, 1985)
Trustworthiness is
1. The extent to which one can have confidence in the study’s findings
2. Parallel of reliability, validity, and objectivity in traditional “quantitative”
research
Trustworthiness Criteria
Credibility
Findings and interpretations are plausible to the “researched”
(the participants)
Do findings accurately reflect reality as seen by participants?
Transferability
Applicability of findings based on comparability of contexts
Are conditions similar enough to make findings applicable?
Dependability
Account for factors of instability and change within the natural
context
Document naturally occurring phenomena (stability and
change)
Confirmability
Capacity to authenticate the internal coherence of data,
findings, interpretations, and recommendations
Document “researcher as instrument” and potential sources of
bias
© 2016 Laureate Education, Inc.
Page 1 of 3
Research Theory, Design, and Methods
Walden University
Insuring Trustworthiness
Action
Prolonged
engagement
Description
Investing sufficient time to learn the culture,
build trust with stakeholders, understand the
scope of target phenomena, and test for
misinformation/misinterpretation due to
distortion by the researcher or informant
Insures
Credibility
(internal
validity)
Persistent
observation
Continuing data collection process to permit
identification and assessment of salient
factors, and investigation in sufficient detail to
separate relevant (typical) from irrelevant
(atypical)
Credibility
(internal
validity)
Triangulation
Data collection and analysis interpretation
based on multiple sources, methods,
investigators, and theories
Credibility
(internal
validity)
Peer
debriefing
Engage in analytic discussions with neutral
peer (e.g., colleague not involved in the
project)
Credibility
(internal
validity)
Member
checks
Test veracity of the data, analytic categories
(e.g., codes), interpretations, and conclusions
with stakeholders to ensure accurate
representation of emic perspectives
Credibility
(internal
validity)
Thick
description
Describe procedures, context, and participants
in sufficient detail to permit judgment by others
of the similarity to potential application sites;
specify minimum elements necessary to
“recreate” findings
Transferability
(external
validity)
Audit trail
Records that include raw data; documentation
of process and products of data reduction,
analysis, and synthesis; methodological
process notes; reflexive notes; and instrument
development/piloting techniques
Dependability
Confirmability
(reliability and
objectivity)
Negative
case analysis
Investigate “disconfirming” instance or outlier;
Credibility
continue investigation until all known cases are (internal
accounted for so that data reflects range of
validity)
variation (vs. normative portrayal)
© 2016 Laureate Education, Inc.
Page 2 of 3
Research Theory, Design, and Methods
Walden University
Action
Reflexive
journal
Description
Researcher’s personal notes; documentation
of researcher’s thinking throughout the
research process
Insures
Credibility
(internal
validity)
Transferability
(external
validity)
Dependability
Confirmability
(reliability and
objectivity)
Referential
adequacy
Archiving of a portion of the raw data for
subsequent analysis and interpretation, for
verification of initial findings and conclusions
Credibility
(internal
validity)
References
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA:
Sage.
© 2016 Laureate Education, Inc.
Page 3 of 3
Understanding Reliability and Validity in Qualitative Research
Nahid Golafshani, University of Toronto Follow
Abstract
The use of reliability and validity are common in quantitative research and now it is reconsidered in the qualitative
research paradigm. Since reliability and validity are rooted in positivist perspective then they should be redefined for
their use in a naturalistic approach. Like reliability and validity as used in quantitative research are providing
springboard to examine what these two terms mean in the qualitative research paradigm, triangulation as used in
quantitative research to test the reliability and validity can also illuminate some ways to test or maximize the validity
and reliability of a qualitative study. Therefore, reliability, validity and triangulation, if they are relevant research
concepts, particularly from a qualitative point of view, have to be redefined in order to reflect the multiple ways of
establishing truth.
Keywords
Reliability, Validity, Triangulation, Construct, Qualitative, and Quantitative
Publication Date
12-1-2003
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.
Recommended APA Citation
Golafshani, N. (2003). Understanding Reliability and Validity in Qualitative Research. The Qualitative Report, 8(4), 597-606.
Retrieved from https://nsuworks.nova.edu/tqr/vol8/iss4/6
RESEARCH ETHICS PLANNING WORKSHEET
This worksheet is designed to help a student researcher anticipate and manage possible ethical concerns
that are relevant to planning and executing a study. This worksheet contains the same 40 ethics question
that you will find in Form C (the most critical component of your IRB application). These 40 ethical
standards will be evaluated for your study by the IRB (Institutional Review Board), once your proposal is
approved.
INSTRUCTIONS: To ensure a smooth ethics review, build your proposal and your research design so that
the answer to each question below is “yes.” You should be as objective and self-critical as possible during
this self-evaluation in order to spot and resolve any potential ethical challenges in advance.
Researchers proposing to complete research in the following specialized areas should first review the
relevant research ethics FAQs on the IRB website:
Clinical or Intervention Research
Educational Research
International Research
Research in One’s Own Workplace
Research about Bullying or Other Potential Issues Related to Safety
If you don’t know how to address one of the ethical standards below, just email IRB@mail.waldenu.edu for
support or join the IRB office hours at one of the posted times.
Footnotes contain tips, samples, and definitions.
Answer each question below with yes, no, or N/A.
If you cannot easily answer “yes” or “N/A” to
each of the ethical standards below, then you
probably need to build extra protections into your
research procedures.
1. Has each pilot1, recruitment, consent, and data collection
step been articulated such that the responsibilities of the
researcher and any partner organization(s) are clearly
documented?
(Provide a numbered list of the data collection steps that
includes how/who/where2 details for each step. Here is a
sample.)
2. Will the research procedures ensure privacy3 during data
collection? Describe how.
3. Will data be stored securely4? Describe how.
4. Will the data be stored for at least 5 years? Describe how
data disposal will occur.
It is fine to roadtest an interview or survey with friends or family prior to IRB approval and that data may not be used in the study’s analysis. However, any piloting
done outside of friends/family requires prior IRB approval, regardless of whether the data would be included in the final analysis or not.
2
How = Write this like a recipe, including enough details so that a person could replicate your study. Submit copies of any of the following that apply: flyer, invitation
email, ad/posting.
Who = Which parties are involved in each step? In particular, we need details about any partners who might be assisting the researcher in identifying or contacting
participants. Note that doctoral students may not delegate the tasks of obtaining consent or collecting data to anyone else.
Where = Specify whether the interactions will take place via phone, email, online, or in-person at a specific location.
3
Privacy risks might include unintended breach of confidential information (such as educational or medical records); being observed/overheard by others while
providing data; or intrusion on the privacy of others who not involved in study (e.g. participant’s family).
4
Secure data storage requires password protection on electronic files and locks for physical data.
1
5. If participants’ names or contact info will be recorded in
the research records, are they absolutely necessary5?
Describe why or clarify that data collection is 100%
anonymous (which is preferable).
6. The research procedures and analysis/writeup plans must
include all possible measures to ensure that participant
identities are not directly or indirectly6 disclosed.
a. For research topics that possibly involve some stigma
(i.e., workplace bullying): Can you confirm that the
volunteering and data collection process will not result
in others learning of your volunteers’ participation in
the study?
b. For all topics: If participant demographic details (i.e.,
age, ethnicity, number of years in a position) are going
to be shared in the final results, will they be shared in a
manner that will not render certain participants
identifiable?
c. For all topics: The standard for Walden capstones is
to not name the partner organization in published
reports, including Proquest. Will you mask the identity
of any partner organizations that are playing a role in
data collection and/or identification of potential
participants?
Exceptions to the organization-masking practice must be
granted by the Program Director and approved by the IRB.
Place an X here ___ if you were granted an exception as
part of your prospectus approval.
7. Will confidentiality agreements7 be signed by anyone8
who may view data that that contains identifiers? (e.g.,
transcriber, translator) Submit a blank copy.
8. Is there a specific plan9 in place for sharing results with
the participants and community stakeholders? Describe.
9. Have the potential risk categories (privacy10,
psychological11, relationship12, legal13
economic/professional14, physical risks) each been fully
5
Note that consent forms do not require signatures if the participant can indicate consent by some action such as clicking on a link, returning a completed survey, etc.
Participant identities might be “indirectly” and unintentionally disclosed if a researcher’s final research report fails to withhold demographic details or site descriptions
that might permit a reader to deduce the identity of a participant. So the researcher needs to think about which demographic descriptors are most important to collect
and report, while ensuring that the identity of individual participants is protected. For example, readers may be able to deduce a participant’s identity if a qualitative
analysis stated, “One African-American vice-principal with 14 years of administrative experience described her professional development experience as…” A general
rule of thumb is to only include a particular demographic descriptor combination if at least 3 people have that combination of demographic details. So if a district had 4
African-American vice-principals with 10+ years experience but only 2 were female, then an appropriate demographic description would be: “One African-American
vice-principal with 10+ years of administrative experience described the professional development experience as…”
7
A sample confidentiality agreement can be found here.
8
Confidentiality agreements are required for transcribers or interpreters but not for the researcher or Walden faculty/staff who are automatically bound to
confidentiality. Some professional transcribers/statisticians/etc address confidentiality in their work agreement and this is acceptable.
9
It is important that the format is audience-appropriate. Stakeholders may lack the time or inclination to digest a full research article or dissertation. Typically a 1 to 2
page summary or verbal presentation is most appropriate.
10
Privacy risks are present if there is any possibility of data being linked to participants’ identities or if anyone could possibly see or hear responses during data
collection.
11
Psychological risks include stress greater than what one would experience in daily life (e.g., materials or topics that could be considered sensitive, offensive,
threatening, degrading).
12
Relationship risks are present if the recruitment or data collection process are likely to alter the existing dynamics between the researcher and participant (who may be
coworkers or have some professional relationship), among participants (if they know one another), or between the participant and the participant’s friends, coworkers,
or family members.
13
Legal risks are present if data collection might result in a participant’s disclosure of violation of laws.
14
Economic/professional risks are present if data collection could result in the participant disclosing violation of workplace policies, disagreement with leadership
decisions, poor work performance, or anything else that could be damaging to the participant’s position, professional reputation, promotability, or employability.
6
acknowledged and described if the study activities might
exceed the risks of daily life? In the brown column, briefly
describe the specific risk and indicate whether each risk is
minimal15 or substantial16. If a risk category is not
applicable, please explain why.
10. Have the above risks been minimized as much as
possible? In other words, are measures in place to provide
participants with reasonable protection from loss of privacy,
psychological distress, relationship harm, legal risks,
economic loss, and damage to professional reputation? In
the brown column, explain how each risk identified in #9
above will be minimized.
11. Will the researcher be proactively managing any
potential conflicts of interest17 (particularly when researcher
is known to the participants in some professional role)?
Describe how.
12. Are the research risks and burdens18 reasonable, in
consideration of the new knowledge19 that this research
design can offer? Describe why.
13. Is the research partner organization willing20 to provide
a Letter of Cooperation and/or Data Use Agreement
granting permission21 for all relevant data22 access, access to
participants, facility use, and/or use of personnel time for
research purposes? IRB staff will advise which type of
partner agreement is needed. State whether you will be
obtaining written partner organization approval before or
after Walden IRB approval.
14. Is participant recruitment coordinated in a manner that is
non-coercive23? Describe. Coercive elements include:
leveraging an existing relationship to “encourage”
participation, recruiting in a group24 setting, extravagant
Minimal risk is defined as follows in U.S. federal regulations: “that the probability and magnitude of harm or discomfort anticipated in the research are not greater in
and of themselves than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.” Most socialbehavioral studies involve some forms of minimal risks.
16
Substantial risk is when the probability and magnitude of harm or discomfort anticipated in the research are greater than those ordinarily encountered in daily life or
during the performance of routine physical or psychological examinations or tests. Substantial risks are acceptable as long as adequate preventive protections are in
place.
17
A conflict of interest is caused when the researcher has some sort of dual role in the research context, such as being a teacher, therapist, investor, business-owner,
manager, etc. Conflict of interest must be managed to ensure that the research reveals “truth,” not just the outcome that the researcher might desire to see due to their
other role. The simplest way to ensure this impartiality is to conduct research OUTSIDE of one’s own context but other methods are possible (e.g., using anonymous
data collection to encourage honest responses).
18
All research activities place some degree of burden on the participants by asking the participants to share personal information, volunteer time, and assume risks.
19
Examples of “new knowledge” include: effectively addressing a gap in the literature, generating new theory, enhancing understanding of a phenomenon, assessing
effectiveness of a particular professional practice, addressing a local practical problem via data analysis.
20
If a partner organization requires the researcher to obtain Walden’s IRB approval before they can provide their written approval, that’s fine. (Walden can issue a
“conditional IRB approval” letter to the researcher and then Walden’s IRB approval will then be finalized once the Walden IRB receives the partner organization’s
letter of cooperation.)
21
No Letter of Cooperation is required (a) if the researcher will simply be asking organizations to distribute research invitations on the researcher’s behalf, or (b) if the
researcher is using only public means to identify/contact participants.
22
Note that when medical, educational, or business records would be analyzed or used to identify potential research participants, the partner organization needs to
explicitly approve access to data for research purposes (even if the researcher normally has access to that data to perform their job).
23
For example, anonymous surveys and/or low-pressure communications such as email invitations permit potential participants to opt out with minimal fear of
retaliation or other negative consequences.
24
It is not ethically acceptable to invite a “captive audience” to participate in research on the spot (i.e., to ask an entire class or a group of meeting attendees to complete
a survey during their session). Such a dynamic would not provide sufficient privacy or respect for their right to decline research participation. However, a researcher
may use the last few minutes of a class session or meeting to introduce a study and distribute materials, such that the potential participants can then take their time to
decide later about participation.
15
compensation, recruiting individuals in a school/work25
setting, involving a service provider26 in the recruitment
process, etc. A researcher must disclose here whether/how
the researcher may already be known to the participants and
explain how perceptions of coerced research participation
will be minimized27.
15. If you were directed to complete Form D in order to
specifically recruit vulnerable individuals28 as participants,
answer the following question: Is targeting this population
justified29 by a research design that will specifically benefit
that vulnerable group at large? Describe why.
If you were not directed to complete Form D, answer NA.
16. All samples could potentially include adults who are
vulnerable30 (without the researcher’s awareness) and it is
important to include their perspectives. Would the benefits
of including these individuals outweigh the risks?
17. If anyone would be excluded from participating, is their
exclusion justified? Is their exclusion handled respectfully
and without stigma31? Describe.
18. If the research procedures might reveal criminal activity
or child/elder abuse that necessitates32 reporting, are there
suitable procedures in place for managing this? Describe.
19. If the research procedures might reveal or create an
acute psychological state that necessitates referral, are there
suitable procedures in place to manage this? Describe.
20. If the research design has multiple groups, are measures
in place to ensure that all participants can potentially benefit
equally33 from the research? Describe how.
Generally, data collection cannot be approved during work hours or school hours unless a “free period” has been identified (e.g., break, study hall) so the research
activities can be separated from the participants’ regular activities. It is important to maintain an “opt in” dynamic rather than implying that employees/students/group
members are expected to participate.
26
A researcher can ask a service provider (nurse, physician, therapist, etc.) or an aid provider (shelter staff) to give research invitations to clients who meet the inclusion
criteria. However, we cannot approve for the service/aid provider to answer questions about the study, obtain consent, or collect data (unless the data is being collected
by the organization itself for purposes other than the study).
27
Doctoral research directly benefits the student (allowing him or her to obtain a degree), and so the researcher should minimize the potential for either (a) conflict of
interest or (b) perceived coercion to participate. Researchers who are in positions of authority or familiarity must take extra precautions to ensure that potential
participants are not pressured to take part in their study.
Examples:
-a professor researcher may recruit her students AFTER grades have been assigned
-a psychologist researcher may recruit clients from ANOTHER psychologist’s practice
-a manager researcher may conduct ANONYMOUS data collection so that subordinates do not perceive their responses or [non]participation as being associated with
their job standing
28
For this purpose, vulnerable individuals include children, prisoners, people with cognitive impairments, on-duty military personnel, people living in an institutional
setting such as a prison, inpatient care, rehab center, or shelter.
29
Convenience sampling is not approvable. Targeted recruitment of children as participants can only be approved when a majority of the IRB votes that the study’s
benefits justify its risks/costs (such as disruption to instructional time). For recruitment of adult vulnerable populations, IRB staff will determine on a case-by-case basis
whether approval must be issued via the full board’s vote (as opposed to expedited ethics review).
30
It is ethically acceptable to include certain vulnerable adult populations if screening for that particular status would be overly invasive, given the research topic: for
example, a researcher might unknowingly have participants who happen to be pregnant, residents of a facility, mentally/emotionally disabled, victims of a crisis, or
elderly. We don’t expect researchers to screen for these statuses routinely for minimal risk research. However, minors may never be unknowingly recruited; adult
recruitment procedures must deliberately avoid recruiting minors and/or include a reliable way of discerning that participants are 18 or older.
31
When a study has exclusion criteria, they should be listed upfront in the recruitment material (flyer, invitation email,etc.) or consent form to prevent situations in
which the researcher rejects volunteers in a stigmatizing manner.
32
Any limits to confidentiality (i.e., duty to report) must be mentioned in the consent form.
33
Control groups must be eligible to partake in the intervention after the study, if results show the intervention to be beneficial. If the design does not involve a control
group, then the researcher only needs to ensure that all participants have equal access to the study results.
25
21. Applicable for all student researchers: Will this
researcher be appropriately qualified34 and supervised35 in
all data collection procedures? Describe how36.
22. If an existing survey or other data collection tool will be
used, has the researcher appropriately complied with the
requirements37 for legal usage? Describe how and submit
relevant documentation.
Questions 23-40 pertain to the process of ensuring that potential participants make an informed decision
about the study, in accordance with the ethical principle of “respect for persons.”
23. Do the informed consent38 procedures provide adequate
time to review the study information and ask questions
before giving consent?
24. Will informed consent be appropriately39 documented?
25. Is the consent form written using language that will be
understandable40 to the potential participants?
26. Does the consent form explain the sample’s inclusion
criteria in such a way that the participants can understand
how/why THEY are being asked to participate?
27. Does the consent form include an understandable
explanation of the research purpose?
28. Does the consent form include an understandable
description of the data collection procedures?
29. Does the consent form include an estimate of the time
commitment41 for participation?
30. Does the consent form clearly state that participation is
voluntary?
31. Does the consent form convey that the participant has
the right to decline or discontinue participation at any time?
When the researcher is already known to the participant, the
34
Researchers must be able to document their training in the data collection techniques and the IRB might require the researcher to obtain additional training prior to
ethics approval. For most student researchers, the research course sequence is sufficient but some research procedures (such as interviewing people with mental
disabilities) may require additional training. For psychological assessments, the manual indicates specific qualifications required. Data collection from children requires
a background check/clearance through a local agency.
35
Remote supervision is suitable for most studies but onsite supervision may be required for certain types of sensitive data collection (e.g., interviews or assessment
regarding emotional topics).
36
If your study is targeting a vulnerable population or involves a sensitive topic, describe any additional training or experience beyond the research courses and ethics
training you have completed.
37
IF YOU ARE USING A PUBLISHED INSTRUMENT: Many assessment instruments published in journals can be used in research as long as commercial gain is not
sought and proper credit is given to the original source (United States Code, 17USC107). However, publication of an assessment tool’s results in a journal does not
necessarily indicate that the tool is in the public domain. The copyright holder of each assessment determines whether permission and payment are necessary for use of
that assessment tool. Note that the copyright holder could be either the publisher or the author or another entity (such as the Myers and Briggs Foundation, which holds
the copyright to the popular Myers-Briggs personality assessment). The researcher is responsible for identifying and contacting the copyright holder to determine which
of the following are required for legal usage of the instrument: purchasing legal copies, purchasing a manual, purchasing scoring tools, obtaining written permission,
obtaining explicit permission to reproduce the instrument in the dissertation, or simply confirming that the tool is public domain and providing credit by citation. Even
for public domain instruments, Walden University requires students to provide the professional courtesy of notifying the primary author of your plan to use
that tool in their own research. Sometimes this is not possible, but at least three attempts should be made to contact the author at his or her most recently listed
institution across a reasonable time period (such as 2 weeks). The author typically provides helpful updates or usage tips and asks to receive a copy of the results. This
type of communication with the author is not necessary when a website or publisher clearly states that the tool is publicly domain. Many psychological assessments are
restricted for use only by suitably qualified individuals. When in doubt, researchers must check with the test’s publisher to make sure that they are qualified to
administer and interpret any particular assessments that they wish to use.
IF YOU ARE CREATING YOUR OWN INSTRUMENT OR MODIFYING AN EXISTING INSTRUMENT: It is only acceptable to modify data collection tools if
one explicitly cites the original work and details the precise nature of the revisions. Note that even slight modifications to items or instructions threaten the reliability
and validity of the tool and make comparisons to other research findings difficult, if not impossible.
38
Informed consent is not just a form; it is a process of explaining the study to the participant and encouraging questions before the participant makes a decision about
participation. The IRB website provides an informed consent form template that researchers are invited (but not required) to use.
39
While documenting consent via signature is common, note that anonymous surveys can obtain “implied consent” by informing the participant, “To protect your
privacy, no consent signature is requested. Instead, you may indicate your consent by clicking here/returning this survey in the enclosed envelope.”)
40
Walden encourages tailoring the language to the readers as long as a professional tone is maintained.
41
Provide an estimate (in minutes or hours) of each component of data collection (e.g., survey, interview, memberchecking. etc. )
consent form must include written assurance that declining
or discontinuing will not negatively impact the participant’s
relationship with the researcher or (if applicable) the
participant’s access to services.
32. Does the consent form include a description of
reasonably foreseeable risks42 or discomforts?
33. Does the consent form include a description of
anticipated benefits to participants43 and/or others?
34. Does the consent form describe any thank you gift(s),
compensation, or reimbursement (for travel costs, etc.) or
lack thereof?
35. Does the consent form describe how privacy will be
maintained44 and state that that the data will not be used for
any purposes other than research?
36. Does the consent form disclose all potential conflicts of
interest?
37. Does the consent document preserve the participant’s
legal45 rights?
38. Does the consent form explain how the participant can
contact the researcher (for general questions about the
study) and the university’s Research Participant Advocate
(if they have questions about their rights as participants)?
(612-312-1210 from within the USA, 001-612-312-1210
from outside the USA, or email address
irb@mail.waldenu.edu). Only one method of contact
required (phone or email).
39. Does the consent form include a statement that the
participant should keep/print a copy of the consent form?
40. If any aspect of the study is experimental (unproven), is
that stated in the consent form?
42
Describe only the possible harms that go beyond the risks of daily life.
For most social science studies, it is appropriate to state that there are no particular direct benefits to the individual. In this case, just present the benefits to society.
44
The consent form should explain any coding system that will permit the researcher to not use names in the research report; how names,contact info, and research data
will be secured and eventually destroyed; and that the data will not be used for any purposes other than research. It is not always clear to participants how a research
interview is different from a journalistic interview, in which informants might be named. So the consent form should make this distinction clear. For sensitive
interviews, the researcher might also want to assure participants that recordings will be destroyed immediately after transcription.
45
A consent form should not ask a participant to waive any legal rights.
43
Educational Evaluation and Policy Analysis
Winter, Vol. 6, No. 4, 1984, pp. 379-391
The Problem of Criteria for Judging
Interpretive Inquiry
John K. Smith
University of Northern Iowa
Over the years various interpretive (or
qualitative, naturalistic, etc.) investigators have been concerned with developing criteria for judging, and defining procedures for doing, this type of investigation. Apparently the aim is to establish
some set of foundational criteria to separate the good interpretive study from the
not so good. These criteria and procedures
would help advocates of this approach
refute charges of subjectivity and allow
them to lay claim to that certitude that is
available, at least as it is generally presumed, to their quantitatively oriented
counterparts. Given the assumptions conventionally made to distinguish interpretive from quantitative inquiry, this paper
takes the position that the quest for foundations may be a misguided one. If the
assumptions and their implications are
taken seriously, arguments that attempt
to establish such nonarbitrary criteria
and/or procedures will, perhaps inevitably, be marked by confusion and inconsistency. To defend this contention, this
paper will begin with a brief historical
overview of the issue. This discussion will
then be followed by a specific example of
the pro