Research Design & Stats
Reliability and Validity for Life Care Planners
Why are we, as clinical practitioners, interested in research aimed at validating the process of life care planning? There are multiple answers. Through research we can improve the process,raise standards,help to answer ethical questions and resolve ethical dilemmas. Perhaps most importantly we protect the patient’s access to life enhancing care through this very useful case management tool.
Ensuring the future of life care planning is essential to protecting the catastrophically injured patient’s access to quality of life enhancing care through visionary case management practices. In light of the recent challenges presented by the Daubert rulings, the future of life care planning as a forensic tool is dependent upon validating the Life Care Planning process in the eyes of the court (Countiss & Deutsch, 2002).
We can do this through definitive research attesting to consistent methodology employed by life care planning professionals that is reliable, valid and relevant to the individual patient’s case. Because we intend to objectively validate the utility of life care planning as a tool in case management, let’s first consider the facets of reliability that are prerequisite to validation of life care planning.
Issues Related To Reliability
Demonstrating the reliability of life care planning as a case management tool is at the heart of validating life care planning as a specialty area of practice. Reliability is comprised of the dependability and consistency of the life care planning process to yield similar results under similar conditions. In other words, life care plans (LCPs) are reproducible. If life care planning is a reliable tool in case management and the provision of patient care, then the results of a given LCP can be consistently replicated. We can convincingly demonstrate reliability of life care planning by appropriately designed research studies. First we need to discriminate between two aspects of reliability: intra-planner and inter-planner reliability (Bellini & Rumrill, 1999, Chap. 3).
Intra-Planner Reliability
Intra-planner (a.k.a., Intra -rater, or Intra -observer) reliability provides internal consistency to the process much like “test-retest”. Intra-planner reliability attests to the consistent application of an individual life care planner’s processes and the reliability of the results of that process. Given similar circumstances, the process of developing the LCP recommendations and cost estimates are the same. Certainly, similar forms and procedures would be used for collecting the information needed for similar cases. But intra-planner reliability goes beyond that type consistency.
Because the LCP is a document that makes recommendations for case management and estimates the costs of those recommendations, it will produce similar recommendations and cost estimates given patients with similar disabilities and life circumstances. Differences between the individual patients, their families, and geographic locations would be appropriately noted as modifying factors in the comparisons.
For example: The basis for establishing the skill level involved in the provision of care for a C-5 tetraplegic should remain consistent regardless of geographic location and irrespective of subsequent development of cost data. What is being validated is the basis for establishing need for care, level of care, and availability of care. Costs are incidental to these issues. It does not vary depending upon the geographic locale or any other predisposing factor for bias.
The significance of intra-planner reliability is that a given life care planner will produce the same life care plan whether it is produced in Indiana or California, whether it is produced as an Independent Medical Examination (IME), or as a forensic tool of the defense or plaintiff, or even whether funding is available to implement the recommendations.
Inter-Planner Reliability
Inter-planner (a.k.a., Inter -rater, or Inter -observer) reliability provides external consistency to the process. Inter-planner reliability indicates that life care planning is a standardized process, consistently applied by life care planners across the country in a similar manner. Given similar patient disabilities and circumstances, life care planners, in general, present similar recommendations and cost estimates. Wide discrepancies do not occur between plans generated by different life care planners for the same patient.
Let’s consider an example demonstrating these concepts:
What if Dr. Smith who always works for the plaintiff, always gets his cost estimates for custom modification of a vehicle to accommodate a wheelchair from Jaguar, Porsche, and Mercedes? Dr. Smith could show a tight range of costs, and his results would be consistent and dependable over time, for each and every LCP he develops.
On the other hand, his colleague, Dr. Jones, only accepts defense work. Dr. Jones consistently gets his cost estimates from Bubba’s Junkyard, Billy Bob’s Pre-Owned Palace, and Honest Eddie’s. [Note: Honest Eddie’s motto is “We’ll beat any deal or give it to you for free,” so Dr. Jones always checks with him last.]
Dr. Jones’s results are also consistent and dependable over time, for each and every LCP he develops. However, reliability between the LCPs provided by Dr. Smith and Dr. Jones does not exist. The results vary depending upon which life care planner writes the LCP!
Validity is Case Specific
Dependability and consistency in applying the life care planning process will reliably yield similar results under similar conditions. While general reliability is necessary to establish validity, it is not entirely sufficient. Validity is case specific. A standard, “valid” life care plan for all people who are infected with HIV does not exist. Each LCP must be valid for the individual for whom it was developed. If life care planning is a valid process, then a good LCP will accomplish its mission: to decrease the frequency and severity of medical complications for a particular patient, avoiding case management by crisis intervention, and improving the patient’s overall quality of life. Establishing the validity of the life care planning process shows that the basic tenets of life care planning are sound.
Issues Related To Validity
Validity has four major aspects: face validity, content validity, criterion-related validity and construct validity. These four aspects can be used as lenses through which to view validity. For validity to be established, evidence of each of the four aspects should be demonstrated (Bellini & Rumrill, 1999, Chap. 3).
Face Validity
Face validity in life care planning refers to whether the LCP “looks like” it appropriately details the disability-related needs of a given individual. Although face validity is not evidence of whether or not the LCP accurately presents an individual’s needs, it is still important to life care planners.
Because the LCP is a tool for educating people about disability-related needs, if it does not appear to represent those needs accurately, then family members, judges, juries, insurance adjusters, etc. may not accept it as a useful instrument. The establishment of face validity speaks to the lay audience.
Content Validity
Content validity relates to whether the elements included in the LCP actually address all the disability-related needs of an individual with a particular disability and set of circumstances for enhancing their life across their life span. The specific LCP should address all of a particular patient’s needs without providing for extraneous treatments.
For example, an annual urological exam would be an important part of the LCP for a person with a spinal cord injury, but not for an individual who has had her leg amputated. Furthermore, to find a meaningful and useful life, the patient’s needs could reasonably be expected to extend beyond medical care. These needs may include vocational education and retraining.
One approach to examining content validity is to have a group of recognized experts come to consensus about which items are most appropriate. Relating your recommendations to published treatment protocols and standards of care could help you to demonstrate the content validity of elements in your life care plan. An excellent reference is Outcomes Following Traumatic Spinal Cord Injury: Clinical Practice Guidelines for Health-Care Professionals issued by the Consortium for Spinal Cord Medicine (July 1999).
Criterion-Related Validity
Criterion-related validity is the gold standard of validity. Evidence of criterion-related validity is presented when a relationship exists between the LCP’s recommendations and estimated costs and some outside measure, or criterion, relevant to those recommendations and estimated costs. There are three types of criterion-related validity: concurrent validity, predictive validity, and convergent and divergent validity.
Concurrent Validity
Concurrent validity refers to the relationship between elements of the LCP and objective findings available at the time the plan was developed, ( concurrent “at the same time”).
If the client fits the demographic profile and circumstances of individuals studied in Aging With Spinal Cord Injuries (Whiteneck, Charlifue, Gerhart, et al., 1993), it should be possible to demonstrate similarity between the LCP’s recommendations and the recommendations made for the individuals in that published study. As medical research and education advance, the standards of practice for life care planning must necessarily evolve to keep pace. Demonstrating concurrent validity shows that the specific LCP meets the current standards of practice and is not obsolete.
Predictive Validity
Predictive validity is of greatest interest to those utilizing the LCP for reserve setting, budgeting, or in a forensic setting. Demonstrated predictive validity answers these questions:
1) Do the recommendations and cost estimates accurately predict the services that will be needed by the individual for whom the plan was written, and at what cost?
2) If the LCP’s recommendations are implemented in full, and the LCP predicts they will help reduce the incidence, frequency, severity, and duration of complications, will the research study demonstrate a difference from the occurrence of those complications in patients with similar injuries but without LCPs?
3) If projections of life expectancy are included in the LCP, are those projections accurate?
Predictive validity examines the quality and quantity of follow-up on patients with whom LCPs have been completed. The goal is to re-examine these patients and update the plans later in time to establish predictive validity.
Convergent and Divergent Validity
Convergent and divergent validity are specific types of criterion-related validity. They could be addressing either concurrent or predictive variables. Essentially, if you have evidence that two things that should be similar to each other, are similar to each other (converge) you have evidence of convergent validity. Likewise, if two things that should be different from each other, actually are different from each other (diverge), you have evidence of divergent validity.
Evidence of convergent validity is demonstratedwhenthe recommendations in a LCP for an individual who has lower extremity paralysis are similar, in some ways, to those in a plan for an individual who has lower extremity amputation. On the other hand,therecommendations for an individual who has congestive heart failure would be expectedto differfrom those of an individual who has a hearing impairment. If the life care plans for these two individuals are not different, then the divergent validity of the two plans is not demonstrated.
Construct Validity
Construct validity is the essential validity to establishing that the basic tenets of life care planning are sound and are evolving contemporaneously with related health care fields. Construct validity might best be understood as theoretically related validity. Evidence of construct validity is presented when a theory or hypothesis predicts a particular finding, and the results of an analytical study correspond to that prediction.
Although life care planning is not a theory per se, many of the underlying tenets of life care planning could be considered theoretical propositions. For example, we operate under the assumption that the development and implementation of a good LCP will decrease the incidence of medical complications, and increase an individual’s quality of life. To test that assumption, we could compare the rate of complications for people who have implemented a LCP with those who did not have a LCP. We could also interview individuals with implemented LCPs and individuals without such plans, and ask them questions about their quality of life.
Consideration of construct validity leads us to conclude that validation must be an on-going process. There are many different aspects of validity most of which change with time. The large number of variables involved in LCP complicates the process of researching reliability and validity relative to life care planning . The only solution is to reduce the number of variables in any given research project and increase the number of projects being conducted.
No single study will conclusively validate life care planning once and for all. Every research study provides another piece of evidence establishing life care planning as a valid case management tool. We need to begin collecting elements that contribute to establishing the validity life care planning’s place in the management of the catastrophically injured patient’s case.