Why I continue to participate even though I’m failing the assessments #medx

For those of you who are following along, you will see that I have been making weekly posts about the content of the Patient Engagement Design MOOC. What you may not know is that I’m failing the assessments. I attribute this to two things: (1) the assessments are poorly designed, and (2) I don’t agree that the content is relevant.

My first gripe with the assessments is the use of “choose all” questions. These are questions like: “Why did Proctor and Gamble slash its survey and focus group approach? You may select more than one response.” The problem with this question is that if you get four out of five correct (or say 80%), you still get zero on that question. So, when this is given even weight as a multiple choice question, which is usually both better written and easier for someone who has watched the videos can get correct, the assessment results do not adequately represent the level of learning the participant has achieved. In general, “choose all” questions represent poor design. There is a reason that we don’t use them when we create official certification exams. Writing good assessment questions is a skill – and it shouldn’t be dismissed as something that easy and that anyone can do. Typically the goal in an assessment is to ensure that those who know the content pass and those who don’t can’t pass simply by guessing. There is actually a science to writing good assessments – Masters and PhD degrees are obtained in this field.

So, although I may be learning more than 70% of the content of the course (needed for the certificate of accomplishment), I’m only getting 33% on the assessments, because I can pretty much guarantee that I’ll make a mistake on the choose all questions. I go back to my previous statement, writing good assessment questions is not easy!

As a quick tip, choose all questions could be re-written as negative multiple choice questions (which also would not be permitted on certification exams, but are fairer way to balance the assessment, and allow the assessment to re-inforce the learning). Just ask “Which of the following is NOT a reason Proctor and Gamble slashed its survey budget?”.

The second reason that I am not doing so well is that I don’t agree with the content. Not that the content is wrong, just that I don’t see it as relevant in patient engagement design. When I fail to see relevance in it, I don’t pay as much attention to the details. I’m focused more on how I would change it to make it better – what approach I would take to make the content more relevant. In doing that, I don’t remember details like “Why did Proctor and Gamble slash its survey and focus group approach? You may select more than one response” or “Which of the following descriptors/analogies does David Eagleman employ to explain how the brain works? You may select multiple answers.”

So, why do I continue to participate when I don’t think the content is good? Partially because I like the outline of the course. I think the overall structure is good, just the choice of lectures isn’t. I participate in MOOCs to improve my knowledge of a topic, but also to help give me ideas on where I can delve further into different topics. The MOOC gives me ideas for blog posts. So, I may not be learning what the course designers want me to learn, I am still learning.

I was reflecting last night on the lost opportunity in this course. Part of what the course is about is “Design”, and yet the expertise of design has not really be brought into the course. I think if the course had been structured a little more around the “Design Thinking Action Lab” model (was offered on the NovoEd platform – by Leticia Britos Cavagnaroalso from Stanford – and yes, I did manage to get a statement of accomplishment for that MOOC), then it could have been very good. I think the course designers underestimate the power of what the participants could have brought to the topic (this is true in many of the xMOOCs – which largely use a knowledge dump pedagogy rather than a connectivist/socio-constructivist pedagogy). The course could have had participants talk about patient engagement – what it is and what it means to them (from that perspective week 1 was good), but then could have introduced the design thinking model, and then had people or teams use the design thinking model to share and explore a patient engagement problem. The results would have been very interesting, would have been participatory in nature, and participants would have learned a lot more about Patient Engagement Design! Perhaps a new course is in order – one on the NovoEd platform which better enables team collaboration – one that is really about Patient Engagement Design – and encourages participants to actually use a design as a way to explore the topic of Patient Engagement. This would also be a course that could qualify for CME credits, as participants would need to engage in the content, not simply memorize/recall (P.S. I’d love to collaborate on creating such a course!).

So, I will continue to follow along with the content – and I will continue to critique it – because it is through that critique that I personally develop a deeper meaning of Patient Engagement – I just wish I didn’t feel so much like I was talking to myself!


6 responses

  1. […] end of the module. I must say that they did a good job writing the questions, unlike my rants about the MedicineX ePatient course. If you want to learn how to write good multiple choice questions, there are some free eLearning […]

  2. amy ahearn Avatar
    amy ahearn

    Dear Rebecca,
    Thank you again for your thoughtful engagement and participation with this course. You can rest assured that you are not talking to yourself! Course staff, faculty and our ePatient moderators have read all of your blog posts and all of your forum posts. We hear your concerns, appreciate your feedback, and are learning from your insights.

    First, I would like to address the central purpose of launching this course as a MOOC. As you know, in addition to performing active clinical duties as a practicing anesthesiologist. Dr. Chu runs Stanford Medicine X and is a passionate advocate of patient engagement. To this end, he has developed a new method of teaching classes at the Stanford School of Medicine whereby he enrolls Stanford students, but also livestreams the classes simultaneously to the broader community so that patients and other stakeholders who can’t attend in person can also participate. Our efforts to turn this livestreamed course into a MOOC were in line with this broader goal to make these ideas and talks available to more people, free of charge, so that they can engage with them on their own terms. This could mean that they walk away a coherent framework for patient engagement, or just selectively identify a few talks and ideas they find useful.

    Dr. Chu’s education team in the AIM Lab at Stanford runs multiple online courses that provide time-shifted, place-shifted models of learning to medical interns, residents and physicians. This work is a full time job by itself, but we also believe in patient engagement and are therefore also working to support this course. I fully appreciate that assessment design is an art and a science, and one that we admittedly could have done better in this MOOC. We appreciate the tips you have passed on and will look to make adjustments as the course move forward. That said, I think that the larger lesson I learned from Stanford’s Graduate School of Education is not the art of designing perfect multiple choice questions, but a framework for learning design that is iterative and emphasizes a joint construction of knowledge between learners and teachers. We view this class as a first prototype. We wanted to see: What could a class on patient engagement look like? How can we start with providing access to talks and resources and forums for engagement? What can learners teach us about what they want to see more of our less of in a course like this? How can we adjust and make changes midstream as needed? What kind of staffing/teaching model does a course serving this volume of learners require? How can we efficiently certify knowledge, even when we have concerns about the reductive nature of multiple choice quizzes?

    People like you who have advanced credentials in the field of education are fully aware that MOOCs in their current form are not the optimal vehicles for teaching and learning. Just as you probably have, we’ve learned far more from reading the discussion forums than tracking the results of the multiple choice quizzes. I like your idea of a shared design project and am a fan of Leticia’s course on NovoEd (I was a course catalyst for that MOOC and it was a great experience). In our other MOOC that we are running right now, we are actively seeking new approaches to medical education and testing out a Peer Studio tool that provides rapid, actionable feedback to learners form other colleagues in the class. If you can believe it, just by putting together a course that collects interdisciplinary stakeholders and foregrounds discussion-oriented responses, Dr. Chu is making a big departure from traditional medical pedagogy, which is premised on lecture based transmission models.

    This probably doesn’t address all of your concerns, but I wanted to make you aware that we are self-reflective, open to feedback, and view online teaching and learning as an iterative design process that we can continue to improve. We hope you will continue to engage with this course and keep providing feedback that respects these intentions.

    Best, Amy

    1. Rebecca - @rjhogue Avatar
      Rebecca – @rjhogue

      Thanks for the note Amy. I appreciate that it is a lot easier to critique something than to create it! I also appreciate that the EdX platform that is currently being used isn’t making life any easier for you. Unfortunately, I think the lack of usability in the forum tools (which don’t encourage engagement) make it that much more difficult to create something that connects people. So for me, I see more of a lost opportunity … but as you say, there is a lot to be learned in creating a first prototype, and you are giving me lots to blog about 🙂

      1. scottx5 Avatar

        Having helped pilot courses before I appreciate what Amy is saying about essentially testing the waters first time out. As a patient I sense the potential and opportunity I’m willing to work hard to realize, but find myself up against “procedure”, the world’s lowest form of engagement. The Canadian system I witness is overworked and reduced to careful delivery with all interaction stripped away. In this atmosphere questions become challenges to trust and answers are merely snippets from policy manuals.
        Maybe being this detached is better for me? I send my cancer in for chemo and then watch and note the side-effects. Separated, we live in the same house, speak in the language of blood tests while my worry, sleeplessness and unpredictable physical reactions are irrational responses (verging on rebellion) to the best of care.
        Having no option but to continue this treatment I’ll fill out my forms, note my vitals and endure the unsupportive lectures. No more pretending this has anything to do with me. Clearly, there’s no engagement here for me to engage with but I am curious why being “cured” necessitates my exclusion?

  3. scottx5 Avatar

    Well, you are talking to me anyway Rebecca. I enrolled in the hope of finding a place for my voice as a patient but feel as disconnected from acknowledged as I am dealing with doctors who ignore me or berate me for using up more of their time than is seemingly allowed.

    Though I agree there’s potential here, it seems only an imaginary mock up of what a patient might feel were they invented by an unreflective doctor. Where is the connection to what I actually experience as a patient dealing with doctors who deliberately (yes deliberately) treat me like a fool? Why would I engage with a system that projects an image of invulnerability to any question or micro-observation?

    Ironically, the course has replicated recognizable characteristics I encounter at every meeting I have with my “care-team.” I’m uncooperative, unappreciative, point out contradictions and overly emotional about being extremely ill. Imagine, a sick person behaving so badly–shocking.

    1. Rebecca Avatar

      I think there is one difference I’ve felt in the American system versus Canadian – is that my doctors never make me feel like I’m wasting their time – I never feel rushed. I am permitted to take as much time as I want to get through my list of questions. Now, this sometimes means I also spend time waiting. Somehow, at my primary care doctor, I seem to be seen on time and almost never wait, and yet I also feel like I am given as much time as I need to cover as many issues as I want (that never happened in Canada – in Canada I never waited, but the trade off was that I was only ever allowed to talk about one thing per visit – so in order to honor the commitment to stay on time, I sacrificed the ability to talk about multiple things in a single visit – which always had me wondering if my primary care doctor had a holistic view of my health). At Stanford I can expect to wait … I’ve created coping mechanisms to deal with it (booking the first appointment in the morning when possible certainly helps – bringing things to do in the waiting room when I know I have an appointment later in the day). However, when I am in the office talking to the doctor, I’m never rushed. I spent over an hour with the plastic surgeon on the initial consult (even though the appointment was booked for 30-minutes). So the trade off I have with Stanford is that I get all my questions answered in a single visit, which matters because we drive an extra 20-30 minutes to get to Stanford, but I spend more time in waiting rooms.

Leave a Reply

Your email address will not be published. Required fields are marked *