Research Design & Stats
Basic Scientific Research Methodology
Introduction
Basic scientific research can be applied to achieve the goals of demonstrating the reliability, validity and relevancy of life care planning as a tool for case management of the patient with severe disabilities. In order to do that, we need an understanding of scientific methodology.
Hypothesis-Driven Research
Basic scientific research is driven by the testing of hypotheses. The hypothesis is our best supposition of what we think is happening under a given set of circumstances. While the development of a working hypotheses is applicable to individual client assessment (Reid, 1997) and is employed in daily clinical practice, it can also be applicable to a larger field in general, such as life care planning.
Each scientific study or experiment is designed to ask a particular question about the hypothesis. The results of each study or experiment have the potential to either lend support to the truth of the hypothesis or to disprove and challenge the hypothesis. As the evidence in support of a hypothesis accrues, the hypothesis may become a well-accepted theory on how things work. This does not imply that life care planning is a theory. It is certainly not a theory, but rather a very useful tool. So how might we develop hypotheses about life care planning to test scientifically?
Hypothesis Testing
Traditionally, hypotheses develop from careful observations of a phenomenon or reviews of the published literature in an area leading to a rational assessment of the field. Once an idea is intellectually formed of how things might be working, then a research question can be posed to test whether the hypotheses is true. A scientific study sets forth specific aims and objectives to answer the research question. The specific aims define the response variable that will be recorded as the outcome of the investigation.
In the instance of research for validation of life care planning, the body of published literature is only now emerging. For a comprehensive anthology, see the appendix to the Amicus Curiae Brief (Countiss, 2002) and The Bibliography of Life Care Planning and Related Publications (Weed, Berens & Deutsch 2002), as well as Hamilton’s state of the science paper (1999).
The issues arising from the U.S. Supreme Court ruling on Daubert v. Merrill Dow (1993) serve as an impetus for scientific studies to validate the life care planning process. The ruling has asked three important questions as to whether life care planning, in all its aspects, are 1) reliable, 2) valid, and 3) relevant to each specific patient’s case. Therefore, our hypothesis is that life care plans are indeed 1) reliable, 2) valid, and 3) relevant to each specific patient’s case.
The Research Process
A single hypothesis can generate many research questions. In a research study, the research question is addressed by development of specific aims and the research objectives through which the specific aims are going to be accomplished. The specific aims identify the response variables to be analyzed.
Next, the study protocols and procedures are developed. The protocols and procedures detail the methods to be employed to assure consistent collection of reliable data. After the data is gathered, it must be statistically analyzed. To be meaningful, the results must be interpreted in context of the current state of the profession and its future directions.
The Design of Research Studies
The research process is captured within the overall design of the proposed study. Whatever the hypothesis, design the best possible study to disprove it. Results gathered in this manner have the strongest impact.
A major distinction in design can be made between descriptive and analytical study designs, (Bellini & Rumrill, 1999, Chap. 6). Descriptive studies are non-experimental or “cohort” studies, while Analytical studies test hypotheses.
Descriptive Studies
Descriptive studies gather data of interest about a certain population, a “cohort.” A cohort is a sub-population of patients that share particular characteristics, (e.g. HIV infection or hemiplegia). The outcome of a descriptive study might be a determination of the prevalence of disability within a certain population. After analyzing the results of a descriptive study statistically and making some inference about the meaning of the data, a hypothesis may be generated that can be tested analytically. For example, in a cohort of insulin-resistant type II diabetics, the prevalence of hearing loss might be determined.
The individual case report and case series are always descriptive studies, usually of a singular, interesting nature. These studies can provide a provocative observation justifying a larger, descriptive cohort study.
Analytical Studies
Descriptive studies can inform the design process for analytical studies. A retrospective case review groups similar cases as cohorts and collects specific data about them. They can be either descriptive or analytical. An example of an analytical case review study would be the comparison of life care plans that were updated five to seven years after implementation to determine the predictive validity of the initial life care plans.
Prospective longitudinal studies are more powerful than retrospective case reviews. They always test hypotheses by following a particular endpoint over time in a specially enrolled patient population. If, in the descriptive study, the prevalence of hearing loss in insulin-resistant type II diabetics were found to be very high, then some hypothesis as to why that occurs might be put forward and tested by experimental intervention in a prospective longitudinal study (Elwood, 1998, Chap. 2; Piantadosi, 1997, Chap. 4).
Statistical Design and Power Analysis
Importantly, statistical consultation should be a part of the study design process. Because the data must ultimately be analyzed statistically to be meaningful, it is extremely helpful to consult with a statistician in the design stage of the study. The final methods of analysis should be determined before data collection begins.
After the data collection has been completed, statistical analysis will indicate whether the outcome is significant. However, the qualitative parameters for deciding what is significant must be chosen before data collection begins. The level of the difference detected must be set very low to minimize the chance of identifying a false positive effect, known as a Type I error.
Type I Error
A Type I error occurs when the difference detected in the study is accepted as a true result when it is not. Conventionally, the level of significance is set at p < 0.05, so that the probability of a Type I error is less than 5%. In contrast to the parameter for the Type I error rate, the parameter for the Type II error rate should be set very high.
Type II Error
A Type II error occurs when no difference is detected, but a difference actually does exist, in other words a false negative is identified. The Type II error probability is frequently set as high as 80 – 90% (Bellini & Rumrill, 1999, Chap. 3; Friedman, Friedman, Furberg, & DeMets, 1998, Chap. 7; Piantadosi, 1997, Chap. 4) .
Power Analysis
Statisticians can also help determine whether the proposed study is feasible. This is done by power analysis. “Power” refers to whether the study has the capability to detect a significant difference in the response variables given the levels set for the qualitative parameters discussed above. Power comes from the number (N) of participants included in the study and the magnitude of the effect of interest.
If a sufficient number of cases are not available to power the study adequately, then it is not feasible to conduct the study because no meaningful results can be detected. The N required for the study to detect a difference can be calculated from the expected effect size and the expected variation in the data. If the effect size is small, or the variation large, then the N must be large.
The circular question is “How can the effect size from a study that has not been completed be determined?” The answer is that it cannot be determined, only estimated. Published reports of similar effects or preliminary studies, which are small studies that were not “powered” and may not have detected a difference in outcomes, can inform us about estimating the effect size and the range of variation in the effect (Bellini & Rumrill, 1999, Chap. 6; Friedman, Furberg, & DeMets, 1998, Chap. 7; Senn, 1997, Chaps. 4 & 13.).
Inclusion/Exclusion Criteria
Inclusion/exclusion criteria define the study’s target population, (Bellini & Rumrill, 1999; Piantadosi, 1997, Chap. 8). The baseline characteristics considered by the study are described by the inclusion/exclusion criteria, including any baseline exams the study might deem important to control of potential extraneous confounding variables.
Confounding Variables
A confounding variable, or bias, is some factor that accounts for an effect identified in the study, but masks a true effect. Some commonly identified confounding variables include baseline characteristics of the cohort such as gender, age, cultural background and socioeconomic level. Other confounding variables could be identified as pre-existing medical conditions with pathology similar to the pathology of interest in the study age or with pathology that exacerbates the severity or progression of the pathology of interest in the study.
Controlling for Confounding Variables
One way to control for confounding variables is to set the inclusion/exclusion criteria to limit their presence within the study population. For example, it might be reasonable in a study on the effects on I.Q.of HIV-Associated Dementia (HAD) to exclude those patients with a pre-existing closed head trauma or cerebral stroke. In the same study, age might be limited to young adults aged 21-35 to control for the normal age effects on intellect seen in immature and geriatric populations. The inclusion/exclusion criteria serve as an assessment of eligibility, or checklist, for participant enrollment to the study.
Stratification
Another way to control for confounding variables is to include the confound in the study population, but stratify the study by the levels of the confounding variable. For example, socioeconomic effects are commonly stratified by level of education achieved and earned income. Gender might be an interesting confound within the same study of HAD effects on intelligence, not because males and females have essentially different IQ’s, but because the HIV disease state underlying the observed pathology may progress differently in males and females due to their intrinsically different immune systems. Stratifying for a confounding variable has the potential to identify important and sometimes unanticipated effects.
Stratification and Hypothesis Testing
Stratification can also be used to test hypotheses. Consider the following analytic retrospective case review study: The response variable, (i.e., recommended level of nursing care for a patient with C5 tetraplegia), may be stratified by some factor of interest to test a hypothesis. In order to test for intra-planner reliability, members of the cohort from a single practitioner’s caseload may be grouped according to the purpose and source of referral. Three groups may include: (a) development of a life care plan referred by defense counsel, (b) development of a life care plan referred by plaintiff counsel, or (c) an independent medical examination. The mean response variables may then be compared and group differences identified. In this study the inclusion/exclusion criteria would be set to limit the population to a single life care planner’s caseload and to specifically include patients referred from all three sources.