Evaluation

I have found myself going back and forth in  the past week, trying to understand implementation and evaluation. Nonetheless, I have found the articles written by Michael Treser on the different phases of  ADDIE model very helpful. You can find them at: https://elearningindustry.com/getting-know-addie-analysis. From what I have been able to glean from the literature, evaluation can be formative or summative.  Formative evaluation is done as part of the learning process and is used to evaluate the quality of the materials being used and their reception by learners. Summative evaluation on the other hand is done at the end of the course to determine whether or not the training had a positive effect.

For formative assessment I intend to use small group evaluation and a short survey (probably use google forms), while for summative evaluation would use the Donald Kirkpatrick evaluation model - arguably the standard for evaluating the effectiveness of training.

Analysis to design

 Analysis to design

The goals of a Problem-based learning (in a problem-based curriculum) is for students to ;
• Acquire, synthesize and apply basic science knowledge in a clinical context;• Engage in critical thinking and problem-solving;• Teamwork;• Critically evaluate literature;• Effectively utilize information technology and identify the most appropriate resources for knowledge acquisition and hypothesis testing; • Self-direct;• Contextualize and communicate their knowledge to others;• Develop the ability to evaluate their own and the group's learning and collaborate with peers.

 What colleagues and I have observation is that the students even in the later years of their medical training students still struggle with demonstrating some of the skills the PBL is trying to infuse or show little evidence that they have acquired or acquiring the above-stated skills. This is despite the training they receive when they join the medical school (first module in year 1). May be if the processes/mechanisms/approaches that are used to help or ensure and  monitor students’ acquisition of the mentioned skills (especially in the first two years of the their training) were improved, there would be an improvement in the acquisition/development of the skills.

Analysis of this situation includes:
• evaluation of the current processes or mechanism/approaches used to monitor and ensure students ACTUALLY acquire the requisite skills;
• Needs analysis of students’ looking at their habits in relation to self-assessment, online interaction ( with regard to academic issues), information literacy and technical skills,  internet access, receiving and giving feedback. This will be done by carrying out surveys and conducting focus group interviews
• Research e-learning methods (that are student-centred) that could be used to monitor skills acquisition or development

What I have been able to glean from the analysis  thus far is that:
 Generally, students do not self assess and are not comfortable with giving or receiving feedback from peers.
There is little online interaction (among students) with regard to academic matters
Students come from a system that is largely teacher-centred
Students have access to internet
There is limited space for reflection particularly in the earlier years of training
No peer feedback
Infrequent one-on-one feedback with the PBL tutor
PBL tutorial sessions are assessed. Rubric is based of some of the goals of PBL mentioned above.

With all these is mind, I want to creating an online social constructivist learning environment. I want to encourage collaborative learning and critical reflection. The plan is to an online activity (will call it Reflective thought) for students to think reflectively and systematically by writing about their development of 'PBL generic skills and attitudes' and exchange their thoughts online with peers. The exchange process will require students to comment on their peers thoughts as well as respond to comments from others on their own thoughts. The facilitator(s) will make comments as well. The plan is for students to use the Borton's framework of reflection. The intervention draws from the following theories: Social constructivism, transformative learning and the theory of critical reflection.

Aim: To enable students to reflect on their PBL learning experiences, articulate what the experiences mean and monitor their development of 'PBL generic skills and attitudes'.

Outcomes (aligned with Borton's framework):

• Reflect on their significant learning and challenges vis-à-vis the PBL generic skills and attitudes

• Articulate what the learning experience reveal about themselves in relation to PBL skills and attitudes.

• Develop a plan to address identified deficiencies or to enhance success achieved.  
• Monitor their development of 'PBL generic skills and attitudes'

I am still thinking about what to use as a delivery mechanism and assessment format