Category Archives: Uncategorized

When Multiple Choice is Just Not Enough (Anatomy of a Bad Assessment Item)

One of the traditional tasks in instructional design is the creation of “knowledge checks” — standard quiz items usually placed in-context with content. The thought behind these types of assessment items is to provide the learner an opportunity to self-assess in sequence, immediately after information acquisition. A common type of item is multiple choice. Multiple choice items can work if you have a great deal of time and understanding of the material, and you are able to construct items that probe higher levels of reasoning. However, too often the reality is that instructional designers without domain expertise write multiple choice items that are irrelevant to real learning, and in many situations, cause more harm than good. Let’s dissect a multiple choice item and probe a bit deeper on why they can be dangerous, and how to make them better.

The inherent issue with multiple choice items like this is the fact that they only test recall, and, in this case, the recall is requested seconds or minutes after the content is provided. And, to impede the process even further, the learner can’t progress until they answer. In this item, the learner is exposed to wrong responses as well as correct ones. The standard four distractors are offered with radio-style buttons: one correct, and three incorrect. The learner scans the list, and makes their choice by clicking in the radio button and then clicking a “Submit” button to receive a response, such as this one:

In this instance, feedback is displayed immediately because the instructional designer has chosen to allow only one try for the item. On incorrect, a “Sorry, that’s incorrect…” statement appears next to a large red “X”. Visual cues are strong, and in this case the type of visual reinforcement and the placement is critical to how useful the item is for learning. When the learner attempts to recall information supported by this item, an unhelpful visual may appear.

For this to be good instructional design the red “X” should be over (or beside) the wrong selected radio button, and the correct answer is should be highlighted. The learner never has a correct visual to overlay the incorrect visual. The learner leaves with a powerful, incorrect visual, instead of a bolder, corrective one. Proper feedback is critical as well. Sometimes learners are just told that the answer is wrong, without being given the correct feedback. Then they return to the item, knowing that whatever thought process or strategy they used was wrong, and can get stuck, trying to remember what their wrong response was, trying to choose the correct answer. If they have to repeat the process multiple times, they do not come away with a strong sense of knowing the correct answer, they instead feel relief that they finally guessed the right answer and were able to progress. In the example above, the incorrect feedback statement is close to the selection, and the correct choice is highlighted with a feedback confirmation next to it on the screen. Additionally, more feedback may be appropriate adding context.

When creating multiple choice items, ask yourself these questions:

  • Is your goal just to have learners take courses, or are you trying to ensure they learn something?
  • If you need someone to demonstrate mastery of procedures, multiple items may be an inappropriate mechanism for them to demonstrate that they can perform.

If you decide to use multiple choice items to promote learning, here are our recommendations:

  • Provide the correct answer after a wrong response.
  • Supplant the incorrect visual, with a bolder visual of the correct response.
  • Use a tracking system which requires the learner to answer a certain percentage (or all) questions correctly.
  • Provide personalized, meaningful feedback in-place on-screen whenever possible.

We all think knowledge checks are innocent enough — important segues in the content sequence — little “breaks” that let the learner pause and think about what they just consumed. This can be a good thing as long as you make sure you’re putting forth the appropriate test item for both the learner and the business. At the end of the day, you don’t want to waste the learner’s time, and you really don’t want to spend precious resources designing learning experiences that don’t have a demonstrable educational gains.

This post was cowritten by Brandon Carson of the Total Learner Experience.


Creative Play

Part 5 of 6 of an ongoing series about the six activity modes.

Creative mode activities are sometimes hard to find within what we would consider the integral part of game play, but it often appears as a precursor or reward to game play. How many games have the option to dress the avatar before gameplay commences or reward the player by purchasing items and decorating their hideaway? This manipulation has taken the spot that paper dolls or make-up mannequins held during my childhood. Both boys and girls engage in this activity, but I certainly observe girls spending hours dressing their characters.

Marvel’s Create your own Super-Hero certainly appealed to this comic book-loving geek, but it has some of the inherent issues in these templatized activities. The player can only choose the typical female super hero body– pneumatic breasts not effected by this world’s gravity. Tall and lean, both the male and female bodies are completely unrealistic, but you are not limited in your color palette, making it possible to choose features and skin tone of any race. The brief Google search for dress-up games yield TONS of hits, but the majority were white, very thin images. Be cautious when designing your own avatars and be more inclusive, please.

The Sims,  Roller Coaster Tycoon and all the tremendous numbers simulation games were ultimately just vessels to create different worlds. While there were different challenges, a great deal of fun was had creating a personalized world with a color scheme and layout at the discretion of the player. It was not a huge coincidence that it was called a “god’s eye” perspective.

I’ve long thought that Andy Deck’s stuff was odd, interesting and thought-provoking. Here at Collabyrinth, he lets people create icons and then publishes them into a maze that displays the most recent users’ work.

Moving into the music world, this activity allows children to easily create music and sounds.

Finally, in Copy Cat, players recreate the images that appear on the screen. More of a problem-solving than a pure creative task, it does include aspects of color theory and the breaking down of space necessary for certain kinds of artwork.

I’m well aware that the activities for this blog entry are not strictly games, but I do think that these links provide activities that are compelling for a lot of users– and that it’s important to include opportunities for learners to create in free-form ways. Oftentimes as designers, we (or our employers) have a strong desire to lockdown the learner experience to predictable paths. Including creative activities challenge that structure.

Explorative Play

Part 3 of a six-part series reviewing examples of the six activity modes.

Another activity mode widely experienced in games is explorative play, where physical space and travel is simulated through the layout of the game arena. By hiding certain areas from view, the player is allowed to discover new areas and challenges in turn. Explorative play can be easily modified by the addition of other activity modes. Many three-dimensional “shooters” combine active and explorative play, where players find their way through virtual buildings or cities while dodging bullets and shooting enemies. Slower-paced educational games often pair exploration of an area with problem-solving activities.

I personally dislike explorative play. It makes me anxious because I don’t know what’s around that corner or in the hidden parts of the game board. I love Civilization. The first time I played Civ3 I played for 10 hours and then realized that I didn’t know what I was doing. Finally aware of the actual rules, I played for another three hours. Similarly when Civ4 came out I played til dawn one long night. But the initial stages of the game — the time where you send explorers out to scout the edges of the world with a completely hidden topography holds no appeal to me. Sure, there’s treasure and unknowns, but those things can be scary and dangerous as well.

In Little Wheel, the player explores the game space by solving puzzles and problems. Successful players are taken to another game space to undo other puzzles. Myst is the best known of these types of puzzle games.

Second Life is, of course, a social experience at heart, but it also incorporates a large amount of exploration of space and environments. In virtual worlds, people navigate their avatars through areas, discovering new spaces and interacting with objects by flying, walking, running, and jumping.

Bubble Tanks is a shooter that takes place within a series of contiguous bubble realms. Rather than experiencing player death, the player’s bubble tank is spat out to another bubble once its power has diminished.

The key to the explorative mode is understanding that not all options may be available simultaneously, and rarely is there a tiered, linear progression through levels. The explorative mode allows the player to feel in control of their movements through a larger space.

Now is the Time to Do Less of More

This week we struck up a conversation about the general state of eLearning design in the corporate world. We ruminate over our belief that companies should consider doing less of more:

Bwc BRANDON: So I thought this week we would discuss the overwhelming number of requests for “training” that come from business units and stakeholders. It seems like some people think “training solves all the problems.”

I’ve been on the inside and the outside of several corporate learning organizations over the years, and one trend that I’ve seen explode recently is a “factory mentality” designed to “templatize” training. I’ve seen these factories operate almost ’round the clock packaging “rapid eLearning” courses with little regard for formative or summative analysis, or interaction that motivates a learner to participate. It’s really a “page turner” world out there in many instances. Some of this is pure economics — many learning organizations are funded by business units. If a business unit allocates a specific amount of dollars for “training” they expect those dollars to actually be spent, regardless of the necessity or quality of the training.

Why have training if it’s not good? I’ve counseled the organizations I work for/with to do “less of more”. I firmly believe if most training organizations just stopped producing about 40% of what they are doing today — just stopped cold turkey — no one would even notice.

Drj DOLLY: I think one reason that training can seem so irrelevant is because it is so divorced from the day to day activities that the jobs actually require. So, these people who are in sales for instance, talking and corresponding with clients everyday, come to training and spend 4 days listening to someone speak at them. These week-long explorations of Powerpoint slides don’t engage the learner. They just become data dump sessions.

Bwc Yeah, I agree. I personally think sales training should be high-touch, situational, and as contextual as possible. It seems to me that there are two avenues to drive down when producing sales training: basic “transfer of information” about products, services, etc., and scenario-based/role-play simulations that place the participant in authentic situations. At Sun, we leveraged the community for the transfer of information component by providing a user-generated content platform. Sales people ate it up. They could get small chunks of product information or sales techniques from experts in the field and download it to their mobile device. We then had “Sales University” for the mandatory accredited training courses.

Drj And the thing is, if people are just going to be sitting in darkened rooms, why bother going to the expense of shipping them across the country? If it’s just memorization of information, there’s plenty of fairly easy and cheap ways to get that across in an online setting. However, both face to face and online activities can be so much more engaging and rich.

Bwc Agreed. One thing that seems to be missing is an evaluation of actual sales skills done in a formal manner. If we’re training these folks to sell, then we need to really look at each individual and evaluate their readiness to sell. It’s one thing to lecture them, have them role-play, provide feedback, and then send them on their way. Where do we assess their readiness to do their job? Is that for their manager to determine outside of the training? If so, is training’s role just to “provide the foundation”?

Drj And  evaluation doesn’t have to be a test. It can be a demonstration, a portfolio or series of smaller activities built into the curriculum.


Bwc Yes, one vendor I worked with provided role-playing scenarios in an online format using computer webcams and FlipCams. Participants would video themselves doing their pitch, and then upload it for the cohort and the facilitator to critique.

Drj In my eyes, that’s a perfectly valid evaluation, as long as the learning objectives were to improve their sales pitch, not to learn the capitals of African countries. The whole point is to have evaluation that supports your objectives and curriculum. Excellent evaluation can be painless and seamless.

This post can also be found at The Total Learner Experience

Games4Change Festival–Quick Thoughts

This post originally appeared in The Total Learner Experience in May 2010.

I’m attending the Game4Change Festival in NYC for the first time this year. I might do a bit more formal commenting in later posts, but these are a few of the thoughts that keep coming up:

1) STEM– everybody kept talking about STEM content (science, technology, engineering and math) but I always wonder if STEM content in games is accurate and/or complex enough. I don’t know what the answer is to the balance of engagement and science content, but the brief glances I took at some games were encouraging.

2) What is the role of facilitators and teachers for these games? I am not sure that higher level learning outcomes happen for the majority of learners/players without some guided instruction. I know that some choose to become engrossed in this worlds, but the vast majority do not. Also, who will be pushing kids to these worlds. It seems like you will need lots of engaged ambassadors helping get the word out and getting the kids initially turned onto these games.

4) “Everyone”– People kept saying that everyone plays games. There was a good bit of homogeneity to the crowd. Mostly young, male, and white, with a few Indians and women. This was a laptop-carrying crowd, and while access was mentioned during the education days, it wasn’t so much yesterday during the main session. While I spoke with some practitioners who seemed to get it, a lot of the comments I heard tended to assume technological prowess.

By far, the getting kids involved with social games– as both creators and players– was the best takeaway I had from this conference. It reaffirmed and rekindled my belief in the power of kids doing.