Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the eighth and final part of chapter 6:
To summarise, these are the conditions for success with bottom-up learning:
Bottom-up learning is not relied upon to meet needs that are critical or common to large numbers of employees.
The appropriate tools are put in place to support bottom-up learning.
Where necessary, employees are provided with the right training to help them to use these tools.
Employees have sufficient discretionary time to devote to bottom-up learning.
Employees are provided with the authority to engage in bottom-up learning activities.
Workspaces are designed to encourage informal communication and to maximise the opportunities for novices to observe experts at work.
Managers and respected peers model effective bottom-up learning behaviour.
The performance management policy encourages bottom-up learning.
Employees are recognised for taking the initiative in meeting their own learning needs and in helping peers to meet their needs.
When the culture is not supportive right at the top, then chances are diminished but not destroyed. Cultures can differ in divisions or departments, under strong leadership. A learning and development department may influence the culture, through the programmes that it offers (including leadership development programmes and executive coaching), but does not have the mandate to unilaterally change a culture. This must come from the organisation’s leadership. Coming next is chapter 7: Formal learning
Return to Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5 Obtain your copy of The New Learning Architect
In part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are best suited. In part 3, we moved on to the writing of the questions, in particular the traps to avoid. In part 4, we saw how quizzes can be presented as games. In this final part, we look at the steps you can take to make your quizzes robust and reliable.
Being thorough
Assuming your quiz is being used to test knowledge, then you need to take some care to ensure that it performs this function effectively. Prepare at least one quiz question for each of your knowledge objectives. You cannot be sure that a learner has achieved mastery if you test only a sub-set of your objectives. To be absolutely sure the learner has not simply got lucky by guessing answers, you may well prepare more than question for each objective. Don’t write questions to test skills, unless you are absolutely sure quiz questions are capable of assessing these effectively, which is likely to be rarely.
As we discussed in part 2, you need to select a question format that’s appropriate to the type of knowledge you are testing, For example, if you need to test recall of a technical term, use a text input question and not a multiple choice, which only tests recognition of the term. Don’t be tempted to select different formats simply to increase variety – that’s not your purpose here.
If your objective is that a learner is able to come up with a response quickly, then add time limits to your questions.
Discouraging guessing
Some people reckon they can pass any multiple choice quiz by guessing the right answers. Your job is to prove them wrong. In part 3 we looked at techniques you can use to make life difficult for the chancer – no give-away distractors, no obvious right answers. A simple improvement would be to prepare at least four options for each multi-choice question, and even better five. That does make it even harder to write the questions, but then there really is no pain, no gain when it comes to question writing. Hopefully all that deep thinking is to come up with the right answer, not just to make a good guess
Another technique you could try is to include a ‘don’t know’ or ‘not sure’ option for each question. This would score no points. Then penalise wrong answers with negative scores. This ups the stakes for the learner who wants to guess the right answers.
Discouraging cheating
The greater the reward for passing an assessment, the more tempting it becomes to cheat. Really high-stakes assessments are beyond the scope of this guide, but you should be aware of the difficulties in authenticating whether the person answering the questions really is who they say they are. All sorts of complex and expensive technologies are available to authenticate remote users, including finger-printing and retina scanning, but the only way you can be really sure that the learner is who they say they are and is getting no help from a third-party or some reference source is to have them attend a testing centre which has an invigilator present. Most quizzes are not that serious, so there’s no point getting carried away with the security! Sophisticated technologies such as retinal scanning can be used to aid the authentication of learners, but are only really necessary for very high stakes assessments
A more routine way to avoid cheating is to randomise the order in which the questions are presented and the order in which options are displayed within the questions themselves. That way, no-one can simply write down the question and option numbers and pass them on to others. A step further is to create a bank of questions from which the system selects the questions to display, which means that every learner will receive a different set of questions. Yes, this is a lot more work, but the chances of successful cheating will be much reduced. Every quiz tool will provide you with different configuration options. This shows some of the options available in the Moodle quiz module, including the ability to set time limits and shuffle questions and options. Providing feedback
Assuming you are using a quiz as a form of assessment, then if you tell the learner whether they have got each question right or wrong, you are making it easy for them to pass the quiz on a second attempt, without necessarily curing any misunderstandings they may have had. To avoid this problem, you could create a completely different quiz for second attempts, or have the system draw questions from a bank, as described above.
At the end of the quiz, inform learners whether or not they have passed. If your software allows it, let them know how they performed against each of the topics addressed by the quiz. Pass or fail, provide advice to learners on what they should do next.
If the quiz is being used in a formative manner (to help the learner progress towards the learning objectives), rather than summative (to assess mastery), then it is vitally important that you provide helpful feedback for every question. Ideally this should be provided for each option of each question, rather than just for all correct answers and all incorrect answers. The purpose of this feedback is to correct errors and misunderstandings and to reinforce key learning points.
Scoring fairly
Another consideration is how you score correct answers. Most authoring tools will allow you to specify the number of points you will award to each correct answer. In a simple multiple-choice question, this is straightforward enough – you either allocate the same number of points to each question or award more points for particularly difficult questions.
The difficulty comes with questions that ask for multiple responses. The first consideration is whether these questions should score higher than MCQs because they are actually asking the learner for a series of decisions, not just one. Another issue is how you apportion the points across the various options. Let’s say there are five alternative options, three of which are correct. Ideally, each correct option will score 20% of the available points. But the learner should also be rewarded for not choosing incorrect options, so each option not chosen should also score 20% of the total. Whether you can achieve this with your authoring software remains to be seen!
That concludes this practical guide. A PDF version will be available shortly. Next up: how to create reference material.
Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the seventh part of chapter 6:
As any reader of detective novels will know, any self-respecting criminal needs the means, the motive and the opportunity. Now no-one’s suggesting that employees should behave like criminals, but they should be given every chance to learn. We’re left with the issue of motive: Intrinsic motivation: Bottom-up learning is most likely to occur if an employee has a desire to improve their performance, or at least to maintain their performance in the face of changing circumstances. Without motivation, the motive has to be externally provided. Modelling: One of the most powerful influences on our behaviour is the example provided by others that we respect. If managers and high profile peers exhibit the behaviour of effective bottom-up teachers and learners, then the pattern is established. Tangible rewards: Ideally, those employees that contribute most to their own development and to the learning of those around them will benefit from this is some tangible way, whether that’s through an increase in pay or through a promotion. To make the connection between learning and reward absolutely clear, it makes sense for this to be explicitly included in an organisation’s performance management policy. Intangible rewards: A reward does not have to be bankable to act as a powerful incentive; often all that’s needed is a little recognition, whether that’s from managers or peers. An employee whose manager thanks them for taking the initiative in meeting their own learning needs or by helping to meet those of their peers, will be only too keen to repeat the process.
Remember that it is natural for human beings to co-operate with each other, whether on a ‘I’ll scratch your back if you scratch mine’ basis, or because a goal can only be achieved though combined effort. We are usually quite happy to share expertise, to be a teacher as well as a learner – it’s flattering to us. Too much in the way of external incentive may only cause suspicion; too little and it looks like this behaviour is not valued by the organisation.
Coming next in chapter 6: Conditions for success with bottom-up learning
Return to Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5 Obtain your copy of The New Learning Architect
In part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are best suited. In part 3, we moved on to the writing of the questions, in particular the traps to avoid. In this fourth part, we have a little light relief, as we see how quizzes can be presented as games.
Quiz games are still quizzes, in that they can be used assist and assess learning, but they employ gaming techniques to increase learner engagement. We all know how compulsive games can be, so it takes little in the way of imagination to appreciate how much they can add to what would otherwise be a very dry process of drill and practice.
To demonstrate a wide variety of quiz game techniques, I’ve taken examples from a quiz making package called Quizit, unfortunately no longer available. Similar results could be achieved by those with coding skills using Adobe Flash Professional or HTML 5, or by using a number of off-the-shelf quiz game tools. Players try to identify as many as possible of the six pictured people, objects or places.
This first example, a classic ‘picture board,’ requires players to type in the name of the pictured object. Players try to get as close as possible to the right answers for each question. Each option is graded as to how right or wrong it is and scored accordingly.
In this variant of a multiple choice quiz, player get rewarded for how close they can get to the right answer. The rather irreverent feedback is delivered randomly from a pool, depending on the accuracy of the answer. Players attempt questions of increasing difficulty, with the aim of getting to the highest level that they can (10 being the highest). Players have three ‘lives’, which allow them to have another go when they make a mistake.
Levels are a classic gaming feature. As the player moves up the levels, the questions get correspondingly more difficult. Players attempt to guess the identity of a person, object, place or event from the clues provided. The more time they take, guesses they make or clues they ask for, the lower their score.
This game is unusual in that it works entirely as a ‘conversation’ between questioner and player. All input is by natural text. Time pressures add to the level of engagement. Players demonstrate how well they know the subject of the quiz by estimating a series of percentages relating to the subject’s behaviour. They nearer they get to the actual percentages, the higher their score.
This game works with a slider, which the player uses to make estimates. Players answer a series of questions as fast as they can. They can have as many attempts as they like at each question, but this reduces their score accordingly.
This time players can have multiple attempts at every question, but in the process waste time and points. Three teams or three individual players each answer a series of questions, to see who can answer the most correctly.
This competitive game can be used with teams of players in a classroom. Two teams or two individual players each answer a series of questions, to see who can answer the most correctly.
This variant on the competitive game pits two players sitting round the same computer against each other. Coming in part 5: Making your quizzes robust
Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the sixth part of chapter 6:
The most powerful tools in the world won’t help if you don’t have the time or the authority to use them. The second ingredient of an effective bottom-up learning strategy is opportunity: Discretionary time: Bottom-up learning is in most cases a discretionary activity for which time must be made available. Many employees – particularly knowledge workers – have some degree of discretion about how they spend their time; others are strictly rostered and timetabled and can only participate in bottom-up learning activities outside work hours or in time specially allocated by their managers. Authority: Time is not the only issue – even with the time, employees have to be allowed to contribute to bottom-up learning, whether that’s through formal organisational policies or the specific inclusion of these activities in their job descriptions. L&d professionals have their role to play here, by making sure that their own policies don’t leave all the power to control the teaching and learning process in their own hands. A good example would be the restrictions that are often placed on the content that can be published on an organisation’s LMS, making it impossible for rapid or user-generated content to be distributed in this way. Informal spaces provide those without the facility or the inclination to blog with a face-to-face equivalent. Americans talk about the learning that takes place ‘around the water cooler’ and with good reason. Coffee areas and staff restaurants have the same effect, as do the areas where smokers gather to satisfy their addictions. Organisations create these spaces primarily for their functional purpose, but they should also be aware of the learning opportunities that these provide. Experts in the open: throughout history, humans have learned a great deal by observing experts in their everyday work. Organisations can facilitate this process by arranging workspaces in such a way that novices can work alongside the experts, much as apprentices and their masters have done for centuries.
In part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are suited. In this part, we move on the writing of the questions, in particular the traps to avoid.
Question writers are faced with two tricky problems:
They do not want to make the correct answers too obvious but, on the other hand, they don’t want to leave these answers open to challenge. So, they put an awful lot of care into how they phrase the right answers, to make them absolutely clear. In the process, they often give the answers away.
They need to tempt the user with distractors (wrong answers) that are genuinely distracting. To be distracting, they must be plausible. The trouble is that plausible distractors – tempting but still unambiguously wrong – can be hard to come by, particularly when you’re testing knowledge of rules and principles. Still sure you want to write quiz questions?
We’ll explore both of these traps, with the help of some examples.
Obvious answers
In this first case, most learners will pick the fourth option. Why? Because it is the longest. That’s because the question writer will spend a disproportionate time making sure the correct answer is absolutely precise. As a result, it stands out. Also, the question writer was obviously struggling to come up with a plausible fourth option, which is why they threw in option 3 – an amusing throw-away, but certainly not a genuine distractor.
This second example is just carelessness. Only the word ‘elephant’ fits grammatically with the stem of the question.
Here’s another frequent mistake. The absolute answers (‘no men’ and ‘all men’) are clearly less likely than the softer ‘some men’. No learner is going to be fooled by these distractors.
Another lazy cop out is to use ‘none of the above’. It’s a way of telling the learner that you’re having trouble phrasing a correct answer. And option 3 is another throw-away.
Confusing questions
Learners don’t mind obvious answers, because it makes their job easier. But they certainly get annoyed if they are presented with questions that they don’t understand. Take these examples …
It’s bad enough having a negative in the question stem (“Which of the following is NOT …”), but a double negative such as “never unnecessary” makes the question really hard to fathom out.
There is no grammatical consistency in these options. Each one should be phrased in the same way and flow nicely from the question stem.
The problem here is that the learner could enter their answer in so many different ways: “CERN”, “CERN in Geneva”, “Switzerland”, even “Centre Européen de Recherches Nucléaires”. Text input questions are fine, but you must make it absolutely clear what sort of answer you are expecting, for example, “In what city was the research establishment where the World Wide Web was invented?” Even then you be prepared to accept both “Geneva” and “Genève”.
The problem here is that it is not clear whether you are looking for a single response or multiple responses. Experienced computer users will recognise that the use of check boxes implies you can pick any number of options, but that won’t be obvious to everyone. Better to say “You can choose more than one option.”
Chances are you won’t see your own obvious answers and confusing questions. You have no option really but to have your quiz questions tested by typical learners. Believe me, you will learn lots from what they have to tell you. Coming in part 4: Quizzes as games
Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the fifth part of chapter 6:
Bottom-up learning will happen to some extent regardless of the efforts put in by the employer to smooth the process. It’s natural for an employee to take the initiative if they don’t know how to complete a task, because they want to do a good job. It will surely be the exception rather than the rule for someone to simply sit down, fold their arms and wait to be told. However, it’s the role of managers – and the l&d specialists who support them – to do more than just leave things to chance. Bottom-up learning can be positively encouraged, by ensuring employees have the means, the opportunity and the motive to contribute to each other’s learning.
Let’s start with the means and in particular the software tools that can provide the infrastructure to support bottom-up learning: Blogs provide employees with the ability to reflect on their work experiences and to share those reflections with others who have similar work interests. Where an employee has many peers within the organisation, the blogging software can be made available inside the firewall, which has the added benefit of keeping the content away from the prying eyes of competitors. Where an employee works as a specialist and has few internal peers, they may be encouraged to blog on the World Wide Web where they can benefit from the expertise of similar specialists around the world. Assuming they are not critical of the employer – and you would need a policy to cover this – an external blog may even have a positive PR benefit for the organisation, demonstrating thought leadership in a particular discipline. Search engines are an essential component in any bottom-up learning infrastructure. We all know the power of Google to help us hunt down information and, for any organisation which has an intranet or a substantial collection of online documents, a similarly powerful search facility behind the firewall is essential. Yellow pages or their software equivalent, allow employees to seek out experts who may be able to help them solve a current problem. If you don’t have the software to do this in a structured fashion, you can always provide a simple list of who to call for what type of information, on the intranet or in hard copy on a notice board. Once an employee has identified the right expert to contact, they need the right communications medium to put forward their question, whether that’s the telephone, email, instant messaging, web conferencing, SMS messaging or some other format. With this proliferation of communication media, organisations might consider issuing some guidelines to help employees choose the right medium for each particular situation. Forums (or message boards or bulletin boards, as they are sometimes called) provide a simple way for employees to post questions online, with the hope that somewhere in the community another employee will be able to provide a helpful response. Where forums can let you down is when you are depending on other users to visit the forum site in order to see the latest questions. Some element of ‘push’ is required to alert users to new questions, whether that’s email notifications, RSS feeds or lists of recent postings that appear on the intranet home page. Wikis provide a way for employees to collaborate in creating content that can be of use to the whole community. Although subject experts and l&d professionals may prime a wiki with content, the ability of all employees to make contributions based on their own particular experiences, makes a wiki much more than a simple reference manual. Learning management systems (LMSs) can help employees to find e-learning content that is available for open access on an ‘as needs’ basis. It’s important to keep the barriers between the employee and the content to a minimum. Ideally employees should not have to log in separately to the LMS; the available content should be sensibly categorised; the content should be tagged so it can be easily found using the LMS’s own search facilities; the registration/sign-up process should be minimal; and employees should be allowed to rate content, so the most popular and useful content is clearly visible. Social networking is usually seen as an out-of-work activity, where people use software such as Facebook, MySpace or Bebo to maintain their network of contacts for purely social reasons. However, in just a few years, social networking has grown so quickly, and exerted such a powerful influence on its users that many organisations are now looking for ways to achieve similar benefits inside the firewall. An organisation’s own social network could be used to allow employees to connect with others who have similar needs and interests, to find sources of expertise, to form communities of practice and to keep up-to-date on developments in their particular fields.
Of course, tools are not enough in themselves; employees also need the skills to use them. Although many of the tools listed above are extremely easy to use and many employees will be familiar with their use outside work, there is room here for top-down initiatives to ensure all employees know which tool to use in which circumstance and have the confidence to become active users.
In his ‘How to save the world’ blog, Dave Pollard lists a number of ways in which organisations can achieve quick wins with bottom-up knowledge management initiatives, by skilling up employees to make better use of the tools at their disposal:
“Help people manage the content and organisation of their desktop: Most people are hopeless at personal content management but don’t want to admit it. Provide them with a desktop search tool and show them how to use it effectively. Help people identify and use the most appropriate communication tool: Give them a one-page cheat sheet on when not to use e-mail and why not, and what to use instead. Create a simple ‘tool-chooser’ or decision tree with links to where they can learn more about each tool available. Teach people how to do research, not just search: If people are going to do their own research, they need to learn how to do it competently. Most of the people I know can’t.”
If you’re looking for somewhere to spend the first week of November, when the cold and dark have set in and yet Christmas is still two months away, then why not head down to the deserts of Nevada, where a whole load of e-learning folk will be gathering to have some fun and learn a few things.
Well I’m going to be at DevLearn 2011, a conference hosted by the eLearning Guild. I’ll be presenting a one-day pre-conference workshop entitled Designing Next-Generation Blended Learning Solutions on Tuesday, Nov. 1. I’m then running a session on The New Learning Architect on Friday, Nov 4.
If you can’t be there, you can always read the books! See The Blended Learning Cookbook and The New Learning Architect.
Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the fourth part of chapter 6:
The phrase The Long Tail was first coined by Chris Anderson in 2004 to describe the niche strategy of businesses, such as Amazon.com, which sells a large number of unique items in relatively small quantities. Whereas high-street bookshops are forced, by lack of shelf space, to concentrate on the most popular books, shown in the chart below in blue, retailers selling online can afford to service the minority interests shown below in orange. Interestingly, the volume of the minority titles exceeds that of the most popular, yet before the advent of online retailing, these needs would have been very hard to service.
The concept of The Long Tail fits well with the argument for bottom-up learning. Top-down efforts can only seek to address the most common (or, as we have seen, sometimes the most critical) of needs. It is simply not possible, given available resources, for l&d professionals to design and deliver an appropriate solution to satisfy every learning requirement in their organisations. Instead, when it comes to corporate learning and development, it is bottom-up learning that must address training’s long tail.
In the end it comes down to priorities – putting the effort in where the reward is going to be greatest. At risk of over-simplifying the issues involved, you could argue that the prioritisation process could be extended across all eight cells of our model, with the most generic and critical needs met top-down, through formalised courses. Less common/critical needs would be met by less structured proactive methods; if not, then at the point of need; if not, then through a process of structured reflection. As we extend into The Long Tail, bottom-up approaches come to the fore, starting with formal external programmes and continuing across the four contexts:
Tony Karrer argued that: “To play in The Long Tail, corporate learning functions will need to:
find approaches that have dramatically lower production costs, near zero;
look for opportunities to get out of the publisher, distributor role such as becoming an aggregator;
focus on knowledge worker learning skills;
help knowledge workers rethink what information they consume, how and why;
focus on maximising the “return of attention” for knowledge workers rather than common measures today such as cost per learner hour.”
Throughout 2011 we will be publishing extracts from The New Learning Architect. We move on to the third part of chapter 6:
It is conceivable that an organisation could manage without any top-down learning interventions at all, relying on employees who were independent learners, good communicators, willing to share their expertise and keen to support each other. More realistically, an organisation will need to make a judgement about how much will be top-down and how much bottom-up, based on a wide range of factors that will differ enormously both between organisations and within them. Here are some guidelines for identifying the situations in which bottom-up learning will be most appropriate: Where there is constant change and fluidity in tasks and goals: Some organisations are lucky enough to enjoy relatively stable policies, procedures and systems; for many others, particularly those which employ a large number of knowledge workers, the opposite is true – everything seems in a constant state of flux. In this situation, it is almost impossible to create top-down interventions fast enough to satisfy the need; you simply must depend on bottom-up processes to fill the gap. For knowledge and skills that are used infrequently: If we apply the Pareto principle, then it is realistic to assume that 20% of all the knowledge related to a particular job will be adequate to cover 80% of the tasks. The remaining 80% of knowledge is used only occasionally and can probably therefore be handled using more informal, bottom-up methods, including the use of forums, wikis and similar tools. Some care needs to be taken that, within the less-commonly used knowledge and skills, there are none which are nevertheless critical. A good example would be emergency procedures – these should certainly not be left to chance and should therefore be tackled through a formal, top-down intervention. When expertise is widely distributed: It is hard to place too much emphasis on bottom-up processes when expertise is centred around a small number of key individuals – these people would be simply overwhelmed by the never-ending requests for help that they would inevitably receive. In these situations, it makes more sense to capture this expertise using more structured, top-down methods. On the other hand, there are many situations in which it is true to say that ‘nobody knows everything and everybody knows something’. Where expertise is widely distributed, bottom-up methods are much more likely to thrive. Where there are fewer people who need the learning: With scarce resources, l&d departments must focus on those needs which are highly critical or which impact on large numbers of people. There are simply not enough hours in the day to create top-down programmes for every small group that has a shared need. Here again, bottom-up methods fill the gap and ensure that needs don’t go unmet. Where the employees are more experienced: Experienced employees have the advantage of generalised knowledge about situations and events stored in long-term memory, which make it very much easier for them to absorb new information. Whereas novices benefit from more structured learning experiences, in which they are led step-by-step through new information with the aid of an expert facilitator, more experienced employees can be given much more latitude to help themselves.
Coming next in chapter 6: Training’s long tail
Return to Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5 Obtain your copy of The New Learning Architect
