Minutes

November 18, 2002

Evaluation Committee Meeting

Members Present: Helen Baker, Jodie Jackson (Co-chair), Judy Koehler, Ken Shannon, Sonnie Strader (Co-chair), Sheryll Tennant,

Others present: Ralph Utzman

Members Absent: Nancy Dunn, Kathleen Bors, Kelly Fritts, Brenda Michael, Cindy Moats, Nancy Nedrow, Joanne Raines, Teresa Richmond, Malinda Turner

1. Jodie announced the drawing winners for $250 prizes (drawing took place in September, 2002). To be eligible for the drawing students had to:

a) have completed an RHEP rotation between August 1, 2001 and July 31, 2002

b) have completed a SERFE for EVERY RHEP rotation completed since 1/1/01

c) Medical students must ALSO have completed the Baseline Data Questionnaire

Winners were:

Ø Brook L. Browning, PT Class of 2002

Ø Lindsay R. Rhodes, DH, Class of 2003 (now a 1^st year dental student)

Ø Cyrus Kavasmaneck, WVU Medicine, Class of 2002

Ø Ryan McCarthy, WVU Medicine, Class of 2002

Ø Neel Karne, WVU Medicine, Class of 2003

Ø Jessica Thompson, WVU Tech School of Nursing, Class of 2002

Data Analysis of the SERFE (students 1^st RHEP rotation data only: n = 867) was presented by Ken Shannon. Some of the items had a lower rating by students than, possibly, desired. In particular, the question regarding the LRCs and the question regarding the quality of the IDS session. Helen Baker was interested in the number of students that didn’t reply to these questions. Helen Baker suggested that focus groups for students to address the following questions might be useful:

a) Do we need LRCs?

b) Are students too spread out to make LRCs useful?

c) Is good internet connection at RHEP housing more useful than LRCs?

d) Most students are used to fast internet connections at their home institution. Is student dissatisfaction LRCs due to slow internet connections?

e) What is Derrick’s assessment of the LRCs?

Helen suggested that these questions, along with the results from the SERFE should be presented to the LRC committee. Helen and Jodie both volunteered to work with student focus groups if committees think this would be useful.

Results to the SERFE question regarding the quality of the IDS sessions should be pulled out and presented to the Curriculum Committee. IF students are dissatisfied with the IDS sessions, we need to get at the root cause of the problem. Every Faculty Development retreat devotes time to IDS training, however, many preceptors who need this training may not be going to the Faculty Development retreats. It was also suggested that we talk to Malinda Turner to get information from the Student Retreat.

We also looked at the results on the SERFE that showed that 1/3 of the students are not living in RHEP communities; they are not having much interaction with community members of their discipline or other disciplines and they are not meeting community leaders. Is it practical for a student to meet with a community leader greater than one time over a 3 month period? Is this important or realistic? The questions pertaining to the amount that students engage in local social activities: do we know how many social activities are even available? Do these questions need to remain on the SERFE?

Suggestions for next data analysis included: looking at gender issues; hometown issues; responses to 2^nd rotation versus responses to 1^st rotation.

Our PA data looks very good. We might want to include this in the Annual Report.

Jodie reported that the revision to SERFE Question 15 recommended at the May, 2002, Evaluation Committee meeting has been done. Question 15 was made into 2 questions: Questions 15 & 16. The old Question 15 read as follows: “If you are a medical student and have made a residency choice, please write your choice here: _________________________________.” The new Questions 15 and 16 read as follows:

Question 15: If you are a medical student and have made a specialty choice, please write your intended specialty here: ____________________________.

Question 16: If you are a fourth year medical student and know where you will be doing your residency, please complete the following:

Name of university or residency program:____________________.

City:____________; State______________.

Discontinuance of IDS data analysis?/Possible Revision of SERFE:

Jodie reported that, currently, site coordinators enter their consortium’s IDS evaluation data into TRACKER. The WVU Office of Rural Health has been responsible for analyzing this data in bar chart format. Currently, consortia only see a comparison of their consortium’s student data against ALL student responses (from ALL consortia) for each question on the IDS. They have not been provided with data that would allow them to compare their consortium with other individual consortium. Jodie and at least one site coordinator have questioned the usefulness of the current bar chart analysis.

Jodie and Sheryll presented a bar chart comparing Question 1.b. of the SERFE (“How would you rate . . . . the quality of the interdisciplinary sessions?”) to Question 8 of the IDS evaluation (“My learning during the session was such that it was worth the time and effort required to attend.”) Approximately 80-90% of students agreed or strongly agreed with Question 8 of the IDS whereas only 30-50% of students rated the quality of the interdisciplinary session as very good – excellent. Since the IDS is not confidential whereas the SERFE is, the question of the quality of the IDS evaluation data presents itself.

Jodie said that one alternative would be for the Evaluation team to cease analyzing IDS evaluation data in bar chart format and let site coordinators simply share their consortium’s results with their faculty and board members. Another alternative would be to add a couple of specific IDS questions to the SERFE (right now there is only one question: “how would you rate the quality of the IDS sessions”. Jodie presented the Committee with the following possible additions to the SERFE:

Disagree Neutral Agree

The Interdisciplinary sessions gave me a better 1 2 3

idea of the [scope of work?] or [contribution?] that

disciplines other than my own can provide in health

care.

Disagree Neutral Agree

The Interdisciplinary sessions gave me a better 1 2 3

understanding of the importance and/or advantages

of interdisciplinary team work in the care of patients

with more complex health problems.

Possible open-ended question:

What improvements would you make in the Interdisciplinary sessions? ________________________________________________________________________________________________________________________________________________

After discussion, the decision was made to NOT make any immediate changes to the SERFE but to send all consortia their data compared to other consortia anonymously, i.e., consortia would be coded and each individual consortia would only be able to identify their own consortium (i.e., they would only be given a code for their own consortium). This will be done on a trial basis and feedback will be sought regarding the usefulness of this format. The possibility of adding more specific questions regarding the IDS on the SERFE may be considered in the future but, first, the Evaluation Team needs to examine the data collected to date and evaluate the usefulness of each question currently on the SERFE. Jodie, Ken Shannon, and Helen Baker plan to meet by conference call to discuss this on December 16.

Jodie reported on the SERFE response rate for 6/1/01 – 5/31/02: 1170 completed rotations; 801 evaluations = 69% response rate. Jodie also reported on the Baseline Data Questionnaire response rate for medical students who will graduate in 2003 and 2004:

2003 Medical School Graduates
	Number of 2003 graduates who have completed a BDQ	Number of 2003 graduates	Response rate
All RHEP	187	238	79%
WVU	65	73	89%
MU	22	52	42%
WVU-CHAS	27	36	75%
WVSOM	73	77	95%

2004 Medical School Graduates
	Number of 2004 graduates who have completed a BDQ	Number of 2003 graduates	Response rate
All RHEP	172	207	83%
WVU	54	61	89%
MU	24	43	56%
WVU-CHAS	24	28	86%
WVSOM	70	75	93%

Jodie reported that the school of dentistry is planning on developing a dental student baseline data questionnaire

Jodie reported that Mike McCarthy will be leading a TRACKER training for site coordinators and campus coordinators in Morgantown 11/25/02, (9:30 AM – 12:00 noon)