When Multiple Choice is Just Not Enough (Anatomy of a Bad Assessment Item)

One of the traditional tasks in instructional design is the creation of “knowledge checks” — standard quiz items usually placed in-context with content. The thought behind these types of assessment items is to provide the learner an opportunity to self-assess in sequence, immediately after information acquisition. A common type of item is multiple choice. Multiple choice items can work if you have a great deal of time and understanding of the material, and you are able to construct items that probe higher levels of reasoning. However, too often the reality is that instructional designers without domain expertise write multiple choice items that are irrelevant to real learning, and in many situations, cause more harm than good. Let’s dissect a multiple choice item and probe a bit deeper on why they can be dangerous, and how to make them better.

The inherent issue with multiple choice items like this is the fact that they only test recall, and, in this case, the recall is requested seconds or minutes after the content is provided. And, to impede the process even further, the learner can’t progress until they answer. In this item, the learner is exposed to wrong responses as well as correct ones. The standard four distractors are offered with radio-style buttons: one correct, and three incorrect. The learner scans the list, and makes their choice by clicking in the radio button and then clicking a “Submit” button to receive a response, such as this one:

In this instance, feedback is displayed immediately because the instructional designer has chosen to allow only one try for the item. On incorrect, a “Sorry, that’s incorrect…” statement appears next to a large red “X”. Visual cues are strong, and in this case the type of visual reinforcement and the placement is critical to how useful the item is for learning. When the learner attempts to recall information supported by this item, an unhelpful visual may appear.

For this to be good instructional design the red “X” should be over (or beside) the wrong selected radio button, and the correct answer is should be highlighted. The learner never has a correct visual to overlay the incorrect visual. The learner leaves with a powerful, incorrect visual, instead of a bolder, corrective one. Proper feedback is critical as well. Sometimes learners are just told that the answer is wrong, without being given the correct feedback. Then they return to the item, knowing that whatever thought process or strategy they used was wrong, and can get stuck, trying to remember what their wrong response was, trying to choose the correct answer. If they have to repeat the process multiple times, they do not come away with a strong sense of knowing the correct answer, they instead feel relief that they finally guessed the right answer and were able to progress. In the example above, the incorrect feedback statement is close to the selection, and the correct choice is highlighted with a feedback confirmation next to it on the screen. Additionally, more feedback may be appropriate adding context.

When creating multiple choice items, ask yourself these questions:

  • Is your goal just to have learners take courses, or are you trying to ensure they learn something?
  • If you need someone to demonstrate mastery of procedures, multiple items may be an inappropriate mechanism for them to demonstrate that they can perform.

If you decide to use multiple choice items to promote learning, here are our recommendations:

  • Provide the correct answer after a wrong response.
  • Supplant the incorrect visual, with a bolder visual of the correct response.
  • Use a tracking system which requires the learner to answer a certain percentage (or all) questions correctly.
  • Provide personalized, meaningful feedback in-place on-screen whenever possible.

We all think knowledge checks are innocent enough — important segues in the content sequence — little “breaks” that let the learner pause and think about what they just consumed. This can be a good thing as long as you make sure you’re putting forth the appropriate test item for both the learner and the business. At the end of the day, you don’t want to waste the learner’s time, and you really don’t want to spend precious resources designing learning experiences that don’t have a demonstrable educational gains.

This post was cowritten by Brandon Carson of the Total Learner Experience.

Advertisements

Instructional Design Rules to Break

The world of instructional design is overrun with endless rules. Many rules are steeped in years of research and driven by appropriate learning theory. However, we think there are some rules that are OK to be bent, twisted, or broken to fit specific needs. There’s an old saying that in order to break the rules, you need to know them. In this series, we are going to take some long-lived rules of Instructional Design and discuss the when and how to break them. Agree or disagree? Let us know!

Rule #1: “Avoid Direct Instruction

Direct Instruction (DI), is the explicit teaching of a skill-set using lectures or demonstrations of the material, rather than exploratory models such as inquiry-based learning (http://en.wikipedia.org/w/index.php?title=Direct_instruction&oldid=526504905). Think of the last “training class” you sat thru while a droning facilitator dumped data from a seemingly endless PowerPoint deck. Many Instructional Designers try to avoid DI at all costs because it’s seen as old-fashioned, ineffective and didactic. Exploratory learning came along in reaction to the DI “sage on the stage” model, and has been overused to such an extent that many see it as the only way to design effective instruction. However, by rejecting DI because of assuming its leading to a poor design, we lose out on its distinct advantages.

Research tells us that learners must construct their own understanding in order to truly learn. In the real world, learners working through exploratory models often go down rabbit holes, learning valuable information and skills, but ones that may not be relevant to the task at hand. Exploratory learning is critically important when you want learners to be invested in the topic and when skills are more important than fact. But for learning data and facts, DI cannot be beat.

DI can save time and effectively transmit data, facts and procedures. We’re not arguing that DI should be used exclusively. Many learning organizations use DI exclusively (and, in our opinion, ineffectively) in the form of PowerPoint-style data dumps with little interactivity. A better model is to create an exercise that encourages the learner to explore a relevant case study, view targeted direct instruction related to the case, conduct problem-solving and reflective group work, and then finally prepare a presentation or teach-back to their peers. Don’t think you have to avoid DI at all costs, just be sure to integrate it effectively with interactive activities and exercises.

This series is co-written with Total Learner. Check back for the next rule to break: Get Buy-in From All Your Stakeholders.

The Intersection of CLAW and ID

One thing that I have struggled with over the past few years is the degree to which the different parts of my virtual and IRL selves should be integrated and advertised. I’ve aligned myself with the K-12 educational world for well over a decade now and have been hyper-aware of the importance of editing my professional presence to be as non-confrontational and non-controversial as possible. However, I feel that my outside interests inform my professional expertise AND are part of what make my instructional design more fresh and interesting.

And I guess the big reveal in this post is that I am a lady arm-wrestler.

Copyright 2012 Billy Hunt For more of Billy's phenom work: billyhunt.com

The Birth of Schoolmarm
Copyright 2012 Billy Hunt
For more of Billy’s phenom work: http://www.billyhunt.com

I’ve got several personas that I wrestle as, but Schoolmarm is my first and most personal expression. Schoolmarm represents the darkest aspects of my teacher and ID personas. Judgmental, unkind, didactic, and unpleasant, Schoolmarm personifies all that I fear I might become, but hope not to. When I design instruction, I design in opposition to Schoolmarm’s angry and insecure disposition.

Rather than be embarrassed about this expression of crazed therapy, I’m going to embrace it. This love of play and pushing boundaries IS what makes me a great instructional designer.

For more information about the Ladies Arm Wrestling movement go to:

CLAW on Facebook

CLAW the Movie

CLAW USA

 

Video

A frog, an iPod, and misplaced skills

I love this video. It’s funny and unexpected, but it also can inspire some thought about the intersection between mobile learning, training and authentic tasks (no, really!).

Mastery Learning is Transferable

That frog is really good at that game. Is anyone surprised? That’s what frogs do.  The main knowledge transfer is from operating in 3D world to a 2D representation on a screen, but the game still exploits the frog’s rapid reflexes and tongue/eye coordination.

Platforms Change, Skills Remain the Same

The frog doesn’t care if that’s an iPhone or an HTC or a real-life fly. Its scanned the movement and the shape and wants to perform. Design for the task, not for the platform or the OS.

Without Satisfaction, Frustration Reigns

Finally the frog attacks the one 3D object that it can. It’s been denied a tasty treat multiple times when all of its experience tells it that that it should be eating a bug– not just a tidbit, but a critical part of its sustenance. As it fails to get its reward, its body language demonstrates greater urgency.

What are our lessons as designers here?

  1. Know what compels our learners — design activities that speak to their desires and strengths. 
  2. Design platform- and device-agnostically– Exploit what makes them unique– touch screens, for example, but make sure that activities are high quality.
  3. Reward with authentic returns. Virtual awards won’t satisfy everyone, or at least not frogs.

Care and Maintenance of Gadgets in the Wild

We sometimes act as though our shiny gadgets are thoroughly reliable constants. By simply living outside of the norm, I constantly expose my computer equipment to difficult circumstances. As a friend said “Your gadgets are tested more than any tool since Deliverance.” Paddle faster?

A few ridiculous equipment circumstances from my own life:

  • My iMac regularly develops a film of condensed water that blurs half of my screen, making HD or retina display options seem completely optional. Do they make Rain-X for computers?
  • An inch and a half long hornet somehow committed suicide in the backside of my eMac. The hornet was much larger than any vent, so it was a mystery how it gained entrance.
  • A mud dauber (another wasp-like flying insect) plugged the headphone jack on my iPad with mud while attempting to build a hatchery for its young.

Living in humid Virginia in a log cabin is at times difficult on the body… but it’s hell on my gadgets. There are other challenges as we move further outside Western suburbia. When traveling to Ghana last year I carried a used laptop as a gift to my host. While she had some correct electrical adapters, she didn’t have a sufficient surge protector. The power cable overheated, burning out the power adaptor and rendering the laptop temporarily unusable. Thankfully as I traveled with the laptop I didn’t run into the problem that my friend had with another laptop in Israel. A Palestinian friend had passed along a laptop in hopes that she would be able to repair it. The Israeli border patrol stopped her and wanted to know why she was traveling with a non-working laptop with a non-US power supply. She was questioned for quite a long time while they ensured that she was not ferrying an explosive device.

It’s easy to think that we can ship extra computers to the underserved corners of the world and fix access problems. It’s not that simple. Any computer, including mine, not housed in a tightly-sealed, climate and environment-controlled building, is at the mercy of insects, humidity, dust, heat and a myriad of other concerns. Electrical surges and differing currents wreak havoc. Security at schools continue to be an issue; computer labs set up previous years by other visitors were robbed. Even when all these hurdles are surmounted, basic maintenance remains a problem. Older computers are likely to have hardware malfunctions and updating software through dial-up Internet connections is challenging.

It’s critical, and inevitable, that computers will continue to penetrate to all parts of our globe, but it’s likely that we’ll bypass many of the challenges that desktop models give us and instead replace it with mobile and cellular technologies. As the world moves to mobile, it’s hopeful that we’ll have more durable gadgets with fewer environmental and maintenance issues.

Too Many Apps

It’s only been four years since the Apple app store was launched, but there are over a half million apps available. In addition there are also all the apps that are embedded in websites and in Google’s store. How do we possibly focus on getting our key tasks done while being told that we need to try this or that new life-changing app? Here are some techniques that I’ve been using to good effect:

Reading Headlines
I don’t read every blog and article the moment it comes out. If I read tech blogs everyday I’d get whiplash. Every new app and tech company is going to change the world. Except they don’t. I wait for the hue and cry to die down and let others weed out the technologies that won’t last. I resist the need to try every last fix-all solution by remembering my History of Ed Tech professor showing us the lantern slide, blackboard, tape player, TV, etc that was going to change the education world and bring in a golden age of human development.
Asking Questions
When a technological break-through lasts long enough to make it worthwhile to invest, I ask friends who I respect what the educational applications are, what the limitations are, and what’s the likelihood that it’s going to be around for a while. Even if there is no cash cost for using a tool, the time and energy that a user spends learning the tool has a price and that becomes a complete loss if the app is going to be absorbed or cut (e.g. Google Wave).
Rejecting Tools (even popular/useful ones)
There are some services that I don’t I need. Spotify is probably great, but I haven’t been convinced that it will add value to my life. I like music, but I’m not an audiophile. I often want it to provide background to other productivity. I mostly don’t want it to bother me. I’ve collected a pretty good iTunes library and I’ve curated some pretty great Pandora stations. I know that Spotify probably is great, but I’m cheap– I don’t want to add another subscription to get the features that really set it apart from competitors. I also want to minimize the amount of time that I spend taking care of my technology– each app that we install means another tool with its own updates, subscriptions to manage, privacy to consider. When we adopt and adopt again we’re devoting more of our time to means not the ends. I’ve got 3-4 ways to listen to music I like already, I’m not sure that I need to add another.
Following Instincts
Sometimes a new app just seems great, a no brainer. If it’s appealing, easy to use, with a clear educational component– then yes, I use it– hello Bitstrips.com. I don’t reject apps just because they’re new. I’m not a crazy luddite. I hope that I am merely a thoughtful luddite. Following my instincts has made me reject any locator type program like foursquare as creepy and unneeded even before I read articles like this.
Investigating on Need
I figure out what I need to do before searching out the app. I don’t find an app and try to shoehorn it into a curriculum or task just to say that I used something new. Using this method we’ve found apps to calculate percentage change, to create secure passwords and teach students how to use javascript.

One technique that has proven invaluable for me is accepting that I can’t stay abreast of every innovation. I shift the focus back from the shininess on the web, and back to the reality of what I really want to do and accomplish. Freeing myself from being an early adopter has definitely reduced my anxiety about what I am missing.

Unplug the Games: The value of human to human play

This summer we’re teaching a series of optional free camps at Computers4Kids. The goal is to provide a fun and engaging experience, but also introduce concepts that might pique the students’ interest. Last week I observed a phenomena that I had noticed before — a definite lack in students’ problem-solving skills. This wasn’t in an unfamiliar content area, but a simple game of 20 questions where the items were familiar animals, foods and objects. It was hard to watch the student stymied by the task. He wasn’t able to construct questions, even with prompts, that would lead to him guessing his hidden animal. He seemed frustrated and embarrassed, but other students lent him support and he was able to succeed eventually. This game that seemed so familiar to me and most of the students was clearly foreign to him.

Observing this interchange made me think about how important games are to children’s development of problem-solving and critical thinking skills. How many hours of my childhood  journeys were occupied with searching for objects that started with certain letters or fit into different categories, guessing a word, or out-spelling the driver? Thinking about strategy, getting tricky, and being clever were all skills that offered gratification and reward. Successful thinking was play and fun. It was valued.

Certainly many of these experiences can be duplicated on smartphones. There are plenty of free or cheap apps with guessing games built in, some lacking a few key components:

  1. An authentic feedback loop. It’s more gratifying to hear a peer or adult congratulate you for fresh thinking or more efficient guessing than canned computer feedback, no matter how intelligent the system is.
  2. Evolution of gameplay. When you’ve planned the same game mile after mile, you begin to shift the rules and boundaries to keep the gameplay challenging and new. The negotiation of the rules to keep play fun, challenging, or to make it appropriate for different levels of players is a critical skill for young players.
  3. Reading the opponent. When people play games against one another, they observe the strengths and weaknesses of the other’s play. Predicting a player’s next steps is often crucial in competitive games. Being able to parse the psychology of a colleague is a valued skilled in real life.
  4. Spontaneity. Humans almost always bring a sense of unpredictability to gameplay. While a computer algorithm can be analyzed and predicted, humans tend to ignore logical steps and respond “irrationally”; sometimes a perceived “losing” move is a strategically savvy one. Many computer programs aren’t equipped to play the long game.

Certainly I don’t disdain computer games and apps completely. I can’t stop playing a Tetris clone on my phone, but I see that games still need to come out of the digital sphere and into our human one. Sitting in the backseat affixed to the smartphone is no substitute for playing a game of I Spy out the window. Just because an activity is lo-fi, doesn’t mean that it’s not valuable.