Systematic Program Evaluation For
By Daryl
Boucher, NMCC Assessment Committee Chairperson
What
is an educational program evaluation, and why do we have to do it?
With
the recent decision by NREMT to register only those candidates who have
graduated from nationally accredited programs,
“Programs must periodically
assess effectiveness in achieving goals and learning domains… Outcomes
assessments include but are not limited to exit point completion, graduate
satisfaction, employer satisfaction, job placement rates, national registration
rates, etc.”
According
to CAAHEP, program evaluation should be a continuing and systematic process
with internal and external validation in consultation with employers, faculty,
preceptors, students and graduates. In short, the basis for developing a
systematic program evaluation plan is to answer two simple questions:
§ How do we know that the
program is meeting its objectives?
§ How do we know that the
program is producing qualified personnel?
Historically,
most academic programs and institutions have used external reviewers to
validate the quality of a program. This external review provides assurances to
constituents, whether they are students or patients, that the organization is
meeting the very minimum of standards. In
The
purpose of any external review process is to assure compliance with established
standards and to allow program leaders to continually improve. It allows
students seeking a specific education to objectively compare the quality of
programs they are considering. Additionally, having a systematic plan for
evaluation is crucial in preparing for upcoming accreditation site visits and
demonstrating to others the quality of a program, as compared to established
benchmarks, thresholds, or national statistics.
Instructors
in the classroom have used some of the components of program evaluation for
many years. The most common of these is the student opinion survey. From these
survey results, instructors have adjusted and adapted their courses in an
effort to improve. It is common to hear faculty discussing pass rates on
national exams, or to have instructors informally discussing recent program
improvements or changes. Gather a group of state instructors together and
before long the conversation turns to a comparison of admission requirements or
attrition in programs, or the lack of preparation of students, etc. With
changing requirements regarding accreditation, the goal now is to take those
informal discussions and formalize them into a sound program evaluation plan.
Components
of the plan
Before
program leaders develop a program evaluation plan, it is important to identify
key items about which they want to learn. The plan should answer:
§ What do we want to know?
§ What should we measure and
how should we measure it?
§ What are the
thresholds/expected outcomes?
§ What will we do to fix
identified problems?
Typically,
high risk, problem-prone, or commonly occurring issues have been assessed. For
example, if ambulance directors complain that graduates of a program don’t seem
to be “road-ready,” educators may come up with a plan to improve the readiness
of future graduates. Or, if pass rates on national exams are lower for one
instructor as compared to another, the program may choose to have those
instructors collaborate to improve pass rates for that first instructor.
The
first step should be to get together a committee consisting of people who know
the program well and can help identify problems or concerns. Sometimes, because
of a lack of available data, no problems have been identified; in those cases,
the data collection can begin by having the committee look at some of the
required components. At the very minimum, every plan should address the following items:
a. Exit point completion: This
is frequently referred to as attrition or retention. How many students started
the program? How many students finished the program? Typically, most programs
allow for a 10% attrition rate. Higher attrition indicates a need to look at
system processes such as availability of tutoring, admission requirements, etc.
b. Graduate satisfaction:
Student opinion surveys remain an excellent way to evaluate the program. These
should be program-based rather than instructor-based, however.
c. Employer satisfaction:
Employers of graduates should be surveyed objectively to identify strengths and
weakness of entry level employees.
d. Job placement rates:
Programs need to assess whether their graduates are employed in their chosen
field of study, or if they have chosen to continue their education.
e. National Registry/state
licensing exam pass rates: Leaders should assess pass rates on licensure
examinations. Most programs establish thresholds at about 90% first time pass
rate. It is useful to compare these numbers to national and state databases.
Once
the data is collected, then program leaders are obligated to develop a plan to
address outcomes that don’t meet thresholds. These remediation plans should be
written and adhered to, and all program constituents should be aware of the
plan.
In
future articles, we will discuss how leaders can begin the data collection
process and develop remediation plans. For additional information about the
program evaluation process, visit the CAAHEP standards and guidelines. Many
sample evaluation plans are available on the World Wide Web.
© 2008 by Jacqueline B. Vaniotis