|
Utility assessments
are subject both to the general problems of any measurement, and
also to some challenges specific to utility elicitation.
- As with any questionnaire
or survey method, investigators should take steps to ensure
that subjects are fully informed, comfortable, and free of distractions.
- Assessment instruments
should undergo pilot testing with subjects representative of
the target subject population.
- Pilot-testing should
include consideration of the level of language difficulty.
- If an interviewer
must take an active role (beyond handing out a questionnaire),
as is usually the case in utility assessment, the interviewers
should follow a written protocol in which they have been trained
and tested, and they should be observed at intervals during
data collection for adherence to the protocol.
Utility assessment
also presents some special challenges.
Validity Testing
for Utility Instruments
Because there is no
"gold standard" for utilities, it is not possible to
determine construct validity or to report test characteristics
such as sensitivity and specificity for utility elicitation techniques.
Test-retest reliability of an assessment instrument typically
sets the upper boundary for validity; yet, even reliability testing
for utility instruments is difficult, because utilities may legitimately
change over time, even over short intervals. Factor analysis and
other methods typically applied in questionnaire design are not
relevant to utility elicitation, which attempts to assess a single
underlying construct. Nevertheless, utility assessment techniques
can have strong face validity.
Health State Descriptions
Health state descriptions
are not at issue when a subject is rating his or her own current
health, except to ensure clarity about whether the subject should
consider all current health problems or only those due to a particular
disease that may be under study.
In contrast, ratings
of hypothetical (imagined) health states require attention to
framing effects; they may vary between first- and second-person
narrations, (Llewellyn-Thomas,
Sutherland et al., 1984) and between descriptions that are
framed positively or negatively. (McNeil,
Pauker et al., 1982; McNeil,
Pauker et al., 1988). Multidimensional health state descriptions
provided verbally may tax the memory of the respondent.
These challenges can
be addressed by providing health state descriptions that are detailed
both from a positive frame and from a negative frame (e.g. "In
this health state, you are not able to bathe yourself without
help from another person, but you are able to perform all the
other activities described."), and providing pictures along
with text, both to provide richer information and also to serve
as a memory aid when shown repeatedly.
Cognitive Burden
Utility elicitation
tasks can be cognitively challenging. Many individuals have difficulty
with numeracy (Woloshin,
Schwartz et al., 2001). Respondents may thereby have difficulty
with the probabilities in the SG, or even with the simple numbers
and subtraction in the TTO. The TTO requires respondents to consider
trading off time rather than events, and it requires assumptions
about the constant proportional trade-off of remaining life years
(Loomes
and McKenzie, 1989).
These problems can
be addressed with careful explanation, breaking the assessment
task into simple steps, and checking with respondents about their
understanding of the task. Automated elicitation methods (described
in more detail in Section 10:
Computerized Utility Assessment) can greatly facilitate the
breakdown into steps and can support the elicitation with graphics.
Interviewer Bias
The question and answer
method with a human assessor who explains each step is subject
to interviewer bias. As above, this can be addressed by careful
training of interviewers in the protocol and periodic checks on
adherence to the protocol. Automated elicitation obviates many
of the concerns of interviewer bias.
With attention to the
precautions described above, utility assessment can be accomplished
successfully, even in challenging populations. For example, the
TTO has been used successfully in hundreds of hospitalized patients
over age 80 years (Tsevat,
Dawson et al., 1998).
|