How many training sessions have you gone to where you received a one sheet evaluation form that asked you to rate your instructor, the course, the room, the chair and the snacks provided at ten a.m.? Chances are you have filled out a few of these over time. In my own experience I have probably filled out hundreds and after a while, there is a tendency to just tick off “strongly agree” on everything, especially if it’s getting close to supper time!
Once in awhile, a new method comes along that radically changes the way we do things. Fire, the wheel, smartphones… you get the idea. Have you heard about Dr. Will Thalheimer’s book Performance Focussed Smile Sheets: A Radical Rethinking of a Dangerous Art Form? Will is one of my top go-to guys in evidence-based performance improvement and for myth busting methods being used in the field that aren’t so evidence-based.
Will explains why the current design of end of training evaluations are actually counter-productive, and sums it up nicely with this list of nine points:
They are not correlated with learning results.
They don’t tell us whether our learning interventions are good or bad.
They misinform us about what improvements should be made.
They don’t enable meaningful feedback loops.
They don’t support smile-sheet decision making.
They don’t help stakeholders understand smile-sheet results.
They provide misleading information.
They hurt our organizations by not enabling cycles of continuous improvement.
They create a culture of dishonest deliberation (Thalheimer, 2016, Kindle Locations 137-143).
That’s just in the book’s introduction! Will uses the rest of the book to show us all a better way for “creating smile sheets that will actually help us gather meaningful data— data that we can use to make our training more effective and more efficient” (Thalheimer, 2016, Kindle Locations 2646-2647) by targeting training effectiveness and actionable results.
Here are the results from question #1 which Will calls “The world’s best smile-sheet question.” By asking this question we are getting a measure of potential for the trainee’s improvement back on the job.
37.5% responded “I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.”
62.5% answered “I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.”
Dr. Thalheimer provides a rubric or set of standards in the book to measure the responses for each question. The standards for question 1 are shown in Table 1 below.
Not bad, but not what I can accept either. Two-thirds of the respondents felt they would be able to employ the methods taught in the workplace with more practice. One-third has a general awareness but won’t be able to apply what they learned. This is a one-day workshop that covers a lot of ground and arguably the goal is to raise awareness of performance improvement. There is also a follow on “at-work” component that the learners can do to further increase their skills and earn a certification if they choose. My goal is to have all the learners choosing C or D. Clearly, I have some work to do in the design and delivery departments for this offering.
The result above made me question if the learners had enough hands-on practice with the case study and the exercises. That takes us to question 9 shown below.
The averaged response was 54%. Will believes (and I agree) that the absolute minimum for time devoted to practice is 35%. Given the number of practical exercises, I think this number needs to be higher… in the 65% range, so that gives me some quantifiable results and work to make changes to the design before the next session. More practice – less lecture. Check!
One final example. Have you heard of spaced learning theory? Casebourne (2015) provides a good overview of the body of research that suggests that by spacing learning over time, people learn more quickly and remember better. Will has designed questions such as #11 below to measure spacing.
The results were interesting because one respondent apparently went back to the training facility the following day! Overall, it seems that the spacing designed into the workshop was effective and 69% of the respondents recognized that topics were covered more than once. As noted above, the learners do have the opportunity to apply what they learned back on the job and submit it to ISPI to earn a certification which is another spacing strategy, but one I have little control over.
In order to get a measure of actual performance improvement for question #1, and to accurately measure the spacing effect, I will need to conduct the survey again after the learners have had sufficient time to apply the skills they were taught on the job. That’s still to come.
If you are still using level one evaluations or smile sheets that ask if the learning was fun, if the learner liked the instructor and the facilities were comfortable, it’s time to re-think your approach. If you attend a training session and still receive those old style smiley sheets, you might also ask yourself how effective the training design really was. I hope this example has shown you enough evidence to convince you there is a better way. If it was – please share it with your friends and colleagues. Heck – share it with your enemies, they might become your friends!
Casebourne, I. (2015) Spaced learning: An approach to minimize the forgetting curve. Retrieved from https://www.td.org/insights/spaced-learning-an-approach-to-minimize-the-forgetting-curve 06 December, 2017.
Thalheimer, W. (2016). Performance-focused smile sheets: A radical rethinking of a dangerous art form (). Work-Learning Press. Kindle Edition.
I was recently invited to the NATO School in Oberammergau Germany to deliver instruction on evaluating E-Learning. As an added bonus I was asked to present the closing keynote speech to the class on the subject “The Education Revolution.”
While I am comfortable on the topic of evaluation, I am (was) not as familiar with the keynote topic so research began! The regularly scheduled keynote presenter is a big supporter of learning technology and that in part shapes his view. I too am a supporter of Learning Technology and have been branded by some as a “Techie” but you just have to see the WiFi go down in my house to know that’s not necessarily true. Anyway… as I sat down to begin my research, I wondered “Is there an education revolution underway?”
I have said “words are important” in previous posts. It’s a lesson I keep learning and applying, so no different here. I thought I had better look at the definitions of education and revolution and make sure I understood what I was looking at – and for. Thank you Merriam Webster!
1a: the action or process of educating or of being educated; also: a stage of such a process
1b: the knowledge and development resulting from an educational process
2: the field of study that deals mainly with methods of teaching and learning in schools
Pretty straight forward right? Albert Einstein’s explanation here strikes a chord with me. So what about revolution? Whenever I hear that word, images of civil wars, the Arab Spring and so on, pop into my mind. Merriam Webster‘s definitions of revolution are…
a: a sudden, radical, or complete change
b: a fundamental change in political organization; especially: the overthrow or renunciation of one government or ruler and the substitution of another by the governed
c: activity or movement designed to effect fundamental changes in the socioeconomic situation
d: a fundamental change in the way of thinking about or visualizing something: a change of paradigm
e: a changeover in use or preference especially in technology
Definitions A and B didn’t seem relevant to me. I am not aware of a sudden, radical or complete change in the education system or a fundamental change in political organization (in the West) that has impacted education in a revolutionary way so I struck those off the list and honed in on definitions C to E.
The Australian Government implemented an Education Revolution in 08-09 which had a digital component, and an infrastructure component so when you google the term – you get a lot of hits related to Australia. It was assessed just three years later that this “revolution” was not successful (Author, 2011). As a counter-point to the government’s description of the program, historian Geoffrey Blainey argued that there has only been one education revolution in Australia and it occurred in 1870 (Vanstone, 2009). Now I am no historian because remembering who did what to who – and when – has always been a challenge for me so I set off to find out what happened in 1870.
As it turns out, Blainey’s argument related to definition C “activity or movement designed to effect fundamental changes in the socioeconomic situation.” In the early to mid 1800’s Westerncountries began to mandate education for children and in the later 1800’s state funded school boards, mandatory attendance and secondary schools were implemented. This meant that children subjected to child labour were now moving from mines, factories and fields into school rooms. That’s pretty fundamental socioeconomic change so I agree with Geoffrey’s assertion that there was a revolution in the 1870’s (each country’s timeline is a little different). This gave me a “baseline” to view the education system from.
With the shift in policy, a private system for the privileged was being supplanted by a public system for all (in the west). Engineering of education had to happen to find efficiencies to meet the influx. There are always pros and cons to any system design. A core curriculum that everyone must follow, including standardized testing to ensure that students are learning what the system requires (and the teachers are teaching what they are supposed to) provides efficiency, but there are going to be compromises. The two cartoons show the trade-offs of an efficient education system pretty well, but I digress… I could not find anything to indicate that between the first revolution of the 1870s and now there has been any new activity or movement designed to effect fundamental changes in the socioeconomic situation via the education system. (If you know of something – please feel free to share your thoughts so I can become more educated). Before I struck definition C off the list, I looked into the current child labour situation. In 2014 there were approximately 168 million child labourers in the world! Based on this, it would seem that moving children from mines, factories and fields into school rooms is not yet complete and therefore the socioeconomic change started in the 1870’s (even in parts of the west) was not entirely sudden, radical or completed (definition A).
At this point, I struck off definition C and pondered, between the 1870’s and now, has there been a fundamental change in the way of thinking about or visualizing something [education], a change of paradigm (definition D)? Prior to 1543, man believed that the Earth was the center of the universe (and Toronto was the center of Canada – still a belief). Nicolaus Copernicus proposed a new model with the sun at the center (Wikipedia). A true paradigm shift! What is at the center of the universe is somewhat analogous to training and education… is it the student or the teacher at the center? In my experience there is strong agreement that it should be the student, but as we can see below in many cases the environment today (top) is much the same as it was in 1901 (bottom) and the teacher is still up front being the sage on the stage. So what about content?
Prior to 1870, the core curriculum was the “Three R’s” of reading, writing and arithmetic. I burned more than a few brain cells as a kid wondering why they were referred to as R’s… In the 1870’s the curriculum was expanded to include the sciences, history and geography. The current system still uses the same core curriculum. I’m not seeing a fundamental change in what is being taught. What about the media?
Media, or the replicable “means”, forms, or vehicles by which instruction is formatted, stored, and delivered to the learner has, and continues to change dramatically. Especially in the past 20 – 30 years. With the exception of the Pressey Learning Machine in the top right of the media examples pictured here, I have been subjected to them all. The next aspect I considered was learning methods, or the “conditions which can be implemented to foster the acquisition of competence” (Glaser as cited in Clark, 2011). Examples of learning methods include Action Learning and Coaching. There has been – in my time in the field – a steady flow of “new” methods, however, in my humble opinion, there are many that are a re-packaging or slight tweaking of existing methods aimed at generating revenue for entrepreneurial folks in the field. I am not aware of any revolutionary methods that have turned the education world on it’s ear. Again – if you are reading this and know of something – I’d love to learn about it!
When you put the media and methods together with a learning strategy that identifies activities that motivate and engage learners, formative and summative assessments to provide a program that meets organizational needs – you have a great instructional design. Nothing revolutionary here either. This has been the practice for at least 60-70 years. At this point, I concluded that a fundamental change in the way of thinking about or visualizing something: a change of paradigm in the education system overall has not happened since the 1870s. Yes, media, specifically technology based media (We still have teachers, books and white-boards) continues to advance and influences methods and strategies but is it revolutionary?
Merriam Webster’s last definition of revolution a changeover in use or preference especially in technology has two key words. USE and PREFERENCE. This made me think of the Digital Immigrant and Digital Native debate that was popular at the turn of the century when proponents like Tapscott (1999) and Prensky (2001) believed there was a generation of technologically adept learners that required a radical transformation of the education system. At that time, to me, it felt a bit panicky… like there was a crisis and if we didn’t “revolutionize” the educational system for the digital natives they were all going to fail miserably. Others like researchers Bullen, Morgan and Qayyum (2011) took the view that technology should simply be used to enhance current practices.
No doubt that younger people love their Information and Communication Technology (ICT). But how and why do they use it? What are their preferences with regards to ICT in learning? Before I go further, an important note here about generations and generalizations. “The popular press, scholarly publications, business leaders and social pundits have all used the inherently weak practice of grouping individuals into broad generational categories to support speculation that millennial students enrolled in today’s higher education institutions, as well as different generations of employees in the workplace, require different approaches to education and training” (Pedro as cited in Christensen and Tremblay, 2013). Research has shown that “leisure time use of ICT doesn’t necessarily translate into effective use of technology in education and training” and not all members of a generational cohort have the same access to ICT (Christensen and Tremblay, 2013). Trying to generalize USE by generation is tricky business or maybe bad design (or science!?).
What about PREFERENCE? Do students (of any age) prefer one form of media more than another for learning? Do we need e-Learning for children and Chalkboards for Boomers? Of course not! After chasing e-learning as the holy grail from the mid to late 90’s for at least a decade, the field began to realize in the early 2000’s (Pappas, 2015) that a blended learning approach is often better. Research has shown that “students use a limited range of mainly established technologies such as search engines, e-mail, mobile telephony and SMS messaging frequently, while “Web 2.0″ technologies such as blogs, wikis and social bookmarking tools were only used by a relatively small proportion of students” (Christensen and Tremblay, 2013). Other studies (Corrin, Lockyer & Bennett, 2010; Lohnes & Kitzner, 2007) have shown that younger students use ICT more for social purposes and older students use it more for study. I also wonder if the use of newer media has been limited by instructional designers who stick to what they know and are not using the newer technologies yet. Another potential topic!
If there has been a changeover in use or preference especially in technology (for learning) it seems that it may be with older learners like me? AND it still doesn’t seem to be revolutionary. I did an online course for math in 1996 using a Commodore 64 computer. Khan Academy is certainly slicker but I was learning the same gizintas ( 2 gizinta 4 twice) on a 64K machine with a dial up modem. I took psychology courses in 1997 and 1998 using books and a telephone. The LMS and Skype are also slicker but the same design methods still apply.
There has been steady progress and incremental change but there is a long way to go. Saying that, there are also interesting things happening that have me wondering if we are on the cusp of a true revolution.Finland‘s education system is switching from the standard core curriculum to an interdisciplinary approach. Definitely revolutionary – if it works and is applied broadly.
Another interesting initiative is Victor Saad’s Leap Year Project where he took a year off from work to create his own MBA education, described in this (20 min) Tedx Talk. From this, he created the Experience Institute to “establish experience as a credible form of education and equip students with the tools necessary to transform our world. Through apprenticeships, self-guided projects, meetups, and coaching, we create a space within higher education that helps individuals build creative confidence, agency, and a compelling portfolio.” In addition to the cohorts following the Ei program, Saad has also partnered with Stanford University to integrate his approach into educational institutions. The Leap Course is explained in this short (2 min) video.
If you made it this far – thanks for sticking it out to the end. I know this was a long post. My final conclusion is that at this moment in time while there area lot of good things happening in our field, there doesn’t appear (to me) to be a revolution underway. I am in agreement with Bullen, Morgan and Qayyum (2011) and we should continue to apply new technology (and methods) to enhance current practices – through the deliberate application of instructional design of course!
Bullen, M., Morgan, T., and Qayyuim, A (2011). Digital learners in higher education: Looking beyond stereotypes. Proceedings of the ED MEDIA Conference, Lisbon, 1 Jul 2011.
Christensen, B.D. & Tremblay, R. (2013). Generational learning differences – myth or reality? In Best, C., Galanis, G., Kerry, J., & Stottilare, R. (Eds.) Fundamental Issues in Defence Training and Simulation. Farnham, UK: Ashgate