management health care system

1289
MEDINFO 2004
M. Fieschi et al. (Eds)
Amsterdam: IOS Press
© 2004 IMIA. All rights reserved
An inventory of evaluation studies of information technology in health care: Trends in
evaluation research 1982 - 2002
Elske Ammenwertha, Nicolette de Keizerb
a UMIT - University for Health Informatics and Technology Tyrol, Research Group Assessment of Health Information Systems, Innsbruck,
Austria
b Department of Medical Informatics, Academic Medical Center, University of Amsterdam, The Netherlands
Elske Ammenwerth, Nicolette de Keizer
Abstract
During the last years the significance of evaluation studies as
well as the interest in adequate methods and approaches for evaluation
has grown in medical informatics. In order to put this discussion
into a historical context, we conducted a systematic
overview on trends in evaluation research of information technology
in health care 1982 – 2002.
The inventory is based on a systematic literature search in
PubMed. Each of the found 1.035 papers from 1982 – 2002 was
indexed based on a taxonomy coding type of information system,
clinical domain, research strategy, evaluation methods, setting
and evaluation criteria.
We found interesting developments in evaluation research in the
last 20 years. For example, there has been a shift from medical
journals to medical informatics journals. With regard to methods,
explanative research and quantitative methods have dominated
evaluation studies in the last 20 years. From 1982 to today,
the number of lab studies and technical aspects has declined,
while the number of studies focusing on process quality or outcome
quality of patient care has increased. Based on our results,
we are inclined to talk of a kind of maturation of evaluation studies
in medical informatics research.
Keywords
Evaluation studies, technology assessment, information technology,
health care, health information systems, inventory
Introduction
During the last years the significance of evaluation studies has
grown in medical informatics. Evaluation studies are more and
more considered as integral part of the planning, introduction
and operation of information technology in health care.
Evaluation can be defined as the act of measuring or exploring
some property of a system, the result of which informs a decision
concerning that system in a specific context [1]. Evaluation of
health information systems has to deal with the actors (the people),
the artifacts (the technology), the environment in which it
is implemented as well as with their interaction.
Some discussion is going on in medical informatics about the
best methods and approaches to be used, the need for an evaluation
framework, and the often considered insufficient quality of
evaluation studies. Some of those topics have already been discussed
for some years now (see e.g. in [2], [3] or [4]).
In order to put this discussion into the historical context of evaluation
research, we conducted a systematic overview on evaluation
studies of information technology in health care from 1982
– 2002. The aim of this overview was to get insight into trends
of evaluation research in the last 20 years.
In this paper, we will present some selected results of this inventory
of studies. We will answer the following questions:
1. How does the number of published studies develop?
2. From which countries do the studies come from?
3. In which journals have they been published?
4. Which types of information systems have been evaluated?
5. In which clinical domains did the studies take place?
6. Which research strategy has been chosen?
7. Which evaluation methods have been used?
8. In which setting did the studies take place?
9. Which aspects have been evaluated?
The inventory of evaluation studies described in this paper is
freely available on the internet, to help people planning evaluations
and to find comparable evaluations.
Methods
The inventory is based on a systematic literature search in
PubMed. This PubMed search was done on April 25th 2003; an
update was done on July 28th 2003.
PubMed Search Strategy
We searched for evaluation studies of health information systems.
We defined an evaluation study as the systematic, empirical
assessment of a component of a health information system.
We defined a health information system as comprising all computer-
based components which are used by health care professionals
or the patient themselves in the context of inpatient or
outpatient patient care.
We decided to exclude medical-technical components (such as
robotics or virtual reality systems) and all systems which are
only used to analyze images or signals (e.g. automatic image
analysis system). We also excluded all computer-based training
and education systems for health care professionals since these