Data driven learning design and digital body language

As learners, do we have our own digital body language and can this be tracked and measured for learning outcomes? Learning strategist Lori Niles-Hofmann talks on the podcast in a discussion about her approach to data driven learning design, which begins with a few very simple steps.

Links from the podcast

• Data-driven Learning Design ebook by Lori Niles-Hofmann

Transcript

Robin: 
Hi it's Robin here, host of the Learning While Working podcast and the founder of Sprout Labs. In this podcast I return to talking about data - but this time, not xAPI data.

This podcast is an interview with Lori Niles-Hofmann about data driven learning design. Lori has written a great ebook about this topic as well. One of the nice cross-overs with other podcasts is Lori actually came to this approach of data driven learning design from working as an L&D person at a marketing company.

In the podcast, Lori talks about an idea of 'digital body language', and reading what a learner wants and needs from the traces that they leave in digital media.

One of the things that's really refreshing about Lori's approach to these types of ideas is that it's really low-tech. She's using it to drive learning design and inform what's engaging for people. During this particular interview I really felt like I was learning from Lori, and I really hope you have lots of takeaways from this. 

Lori, welcome to the Learning While Working podcast.

Lori:
It is an absolute pleasure to be here, thank you.

Robin:
Lori, you sort of coined this term of 'Data Driven Learning Design', what do you mean by that term?

Lori:
What I mean by data driven learning design is: We often put pen to paper using a lot of the really good theories and methodologies and even neuroscience that we know about learning, which is great, but there's also another source we can go to and that's data, and to start looking at some things that our learners are doing. What types of content do they engage with? How long do they engage with? What devices do they use? And you put all of those things together and you get this really nice pool of data that can help answer some of your design questions, and in my experience has led to higher engagement levels. The fact is when you have higher engagement levels you're more likely to see return on your investment. Rather, the people are actually going to engage in the behaviours you're trying to teach.

Robin:
So, this is really about at that sort of, I call it the understanding analysis phase of the learn design, so drilling into what behaviour people are doing.

Lori:
Exactly, exactly. So, we make a lot of choices even when we design. For example, when we want to put in a quiz, what type of intro do we do, when do we put in a drag-and-drop, or any of those types of exercises. I think if we really are honest with ourselves, we do a lot of that based on what we think will work. But if we start looking at how people engage with our content, we start to uncover that maybe what we assumed is not necessarily correct and maybe we start seeing some trends and insights and when we design to those trends and insights, as I said, we get the better engagement.

Robin:
I know when we've done usability testing and actually been able to observe how people are working through a learning experience, there's been times you've done a whole, "Oh, okay. Didn't expect that at all. Didn't expect that." It's just been really fascinating to drill down into a different level of data, of information.

Lori:
You're absolutely right. There's nothing like seeing that. It's fun to observe, but there's also ways we can get that automated data, but there's nothing like just having a look at it and saying, "Oh, wow. I didn't realise that was happening." Maybe even sometimes you think, "Well, I wonder why that's happening. Do you need to fix something? Is there a flaw in the design?" It's absolutely fascinating what you start to uncover.

Robin:
Essentially I quite often act as Instructional Design lead, and I can sometimes see in the work of other instructional designers, that their own personal learning preference comes through. That can be really dangerous.

Lori:
Completely agree. I also find too that we design for ourselves. We design for the way that we think we would learn the content, or we also too can put - sometimes it's not  even that we design for ourselves. I see a lot of learning where it's almost as if we're designing it in teacher-speak. Meaning that we assume that the learner maybe isn't very bright, or just because they need to know something. We got to get out of some of those habits, because what I'm starting to see in some of the engagement levels that type of content when it's written that way, doesn't get the response that you necessarily want. It's not about us, it's about the audience, it's about the learner.

Robin:
Yes, and it's really subtle things like instructions about how to use an activity,  can be condescending. People know how to use a computer and they know how to use the mouse button to drag stuff around. You don't need to sit there and say explicitly click on the left button to drag this.

Lori:
Completely agree. In fact, I know of one organisation, they did the proverbial, "In this module you will learn ..." And they listed the learning objectives, which we all do. They found that people were spending microseconds on that page and just bypassing it. Yet we get so precious about that page. Does it need to be there if it's not being used? It's only for us.

Robin:
Yes, and the people might sometimes sit there and say, "Oh, but people need to understand the 'Why'." But the learning objectives don't quite often tell you the 'Why', and the emotional, or a productive level, about the why or the real reason for the learning experience.

Lori:
I would completely agree. There really also too told from the business point of view. Whereas, if you reposition it, maybe do some story-telling, do a hook or a metaphor, analogies, things like that. You'll find that people will spend a lot longer on that page because it means something to them. It resonates. That's what we're really looking for with data driven learning design is to understand what types of content, what formats, what media, even what types of headlines or types of content is what engages the learner and makes them stay on the page.

Robin:
When you think about that word 'Engagement', can we just drill into that a little bit in terms of how that shows up in data? What does it mean - how you see that in data the sort of data you get back in learning?

Lori:
Engagement, it is a bit of a fuzzy term and I am using it loosely but what I mean by engagement in some most easiest terms I guess, is people not spending microseconds on the page. By engaging with their content they're spending an appropriate amount of time on a section. They are going through the activities, if an activity is there. They're perhaps coming back to it. It's gold if they go forward in content and then they come back, how often do we do that? All those things to me indicate a higher engagement with content.

Robin:
You have a nice term for this, which is 'Digital Body Language'. How did you come about that term?

Lori:
Well, I came about the term - I can't claim credit for it but I do have permission to use it. It's actually a marketing term that was coined by Steve Woods, and he was the CTO of a company called Eloqua that I worked at. It actually came from a marketing term as I said and it was used to describe how people are engaging with your website, your emails, all of your digital content.

I've adapted it so that digital body language in learning is - if we think of when we were in the classroom as a teacher/facilitator, we are able to see visually whether our students are interested or whether our learners are keen on the content. They're sitting up or they're looking out the window or they're crossing their arms.

What digital body language is, is looking at things like click throughs. Do they open an exercise? How long they spend on a page? What type of device do they log in on? Is it mobile? Is it desktop? All these things together form a digital body language. They signify interest. They signify what they prefer. It also signifies what they dislike, and that's what I mean by 'Learner Digital Body Language'.

Robin:
The traces that people leave behind.

Lori:
Yes.

Robin:
What's also really different - well, the way you're talking about it as well, some time times when people talk about measuring based on time or based on hits, they're talking about measuring the outcome, the end result. What you're talking about is using this to inform the decisions a learning designer makes.

Lori:
Absolutely. To be very clear, when we talk about data driven learning design, I'm not in the space of measuring outcomes. What my theory is and what I am seeing is, if we design to those preferences, to the digital body language, we have a higher level of interest and engagement with the content. That means that people are more likely to be more motivated to actually implement it and that increases the ROI. That's the space that I'm operating in.

Robin:
This also must shift the way learning experiences are built because essentially, you're talking about something that people's eyes glaze at me a little bit when I sit there and say, "You build something but you don't leave it, you keep on refining it." Is this also a culture of constant improvement?

Lori:
It is and that's also too a culture shift, exactly to use your words. We're so used to, we build something and then we watch it. We see did people do it? What were their test scores? Did they maybe do a measurement? And that's great and it's a way to design content but if we do a iterative design and say we start off with maybe not putting out a tonne of content but start with what I can Minimum Viable Learning so just a small piece and we start to see, :"Hey, did people prefer the talking head video to the PDF? How many hits did we get? How many downloads? Oh, Okay. I see the digital body language is trending this way. And you know what? People seem to really like 8 o'clock in the morning for some reason."

Well, now I know when I'm designing my subsequent piece to that, I'm going to start hitting those benchmarks. I might look at the time of day that I put it out, I might do that PDF instead of the video because we got more lift-off with that. That iterative process, it means that you're constantly learning and you're constantly responding. It's just like being that teacher in the classroom who: "Yes, we have our workbook and we have the learning objectives we have to hit,” but if we have an unruly classroom or maybe if people are really excited about a certain topic, you will pivot and it's that pivoting that makes that learning dynamic and exciting so it's translating that into an iterative design.

That's a big change from our ADDIE model. It’s even a change from SAM, or even AGILE, it has a bit of AGILE to but it is a shift, it is a mind shift.

Robin:
Actually it is interesting because we use - at Sprout Labs - design thinking and this is the bit which I've been struggling with talking about because essentially it's the bit - the Minimum Viable Learning bit - is actually a really nice way of putting it because some of this works comes from working with some clients who sit there and go, "Well, actually, we don't just build and leave. We continuously improve and do this constant process, which is quite different to all those other things, which still has quite this end point." One of those people who saw the process in action, she went, "There’s no implementation phase Robin?" I went, "No, there's not. Because it's always shifting. You don't leave it and ‘implement’ it as such. You make it happen."

Lori:
Exactly. The interesting thing too about doing that type of dynamic content is it also increases the chances that your learners will come back. They'll feel re-engaged with it because they see it shifting and they see you reacting and that's a fascinating thing to observe. Give a very practical example, and I alluded to it before, if you see a page that people are skipping over, strip it out. Do they need that content? Well, they're not seeing value in it so pull it out. Now you're making that nice and tight and ultimately that's what learners are looking for.

Robin:
Just to explore a little bit around technology, are you using complicated technology to do this sort of data whatever, what are the tools of your trade?

Lori:
I'm going to be really transparent and it's something that people laugh at because I'm in data. I can't even do a pivot table, I can barely do that so no it's not complicated. What I'm looking at, I'm pulling data basically from anywhere and everywhere that I can get it from. Some of the places that I will go to are, for example, our internal sites that we have. Anything that we have on our intranet and social collaboration sites. I get a very simple feed. What are the top 10 search terms? What are the top 10 topics that are trending that people are talking about? That's just an aggregate of keywords. I can pull out from there too, time of day, how people are accessing it, are they on a mobile or on a desktop? Those are just some really basic feeds.

The other thing that I pull from, I also pull from my LMS and I look at - to be completely transparent we don't have xAPI on ours so I can't get a granularity that I would look for but I can get quite a bit of data and I'm looking, again, for the same types of stuff, what are people searching for when they visit the LMS? How many people continue to go back to the LMS? How long do they spend there? Those types of things.

Another piece that I like to look at, and this is very specific to video, is if I pop my video up on Vimeo on a private channel or on YouTube, you can get very extensive analytics and it will tell you what country the person is viewing from, because I work for an international company. It'll tell you how long they stayed. Did they just view the first three seconds and drop off? All those things. I'm kind of cobbling together all these pieces of data and then looking for trends. Am I doing fancy metadata or big data link and putting it all together? No, I'm not and I'd probably be hopeless at that. This is just basic stuff.

Robin:
Basically, you're just collecting all this data manually, getting it into excel spreadsheets, and then just literally looking at it and not really doing anything complicated with it.

Lori:
Exactly.

Robin:
Doesn't even sound like you do any stats work on that, Lori.

Lori:
No. When it comes to, as I said, the top 10 search terms, that's a very easy one for me to have a look at, it comes in once a week and I say, "Oh, do we have content aligned to that?" "Yes, no, yes, no." If I do, push it out. If I don't, is it a gap? Obviously people are looking for it, I may need to think about that.

That's no Excel, that's nothing. That's just looking at some simple data points.

Robin:
Just a slight diversion, because sometimes there's talk about knowledge management and L&D being linked, that's essentially your internet people just giving you that data so you can actually make that response without having to go to them.

Lori:
Correct. Absolutely. I also use that data to talk to subject matter experts. Often they will come to me and I work in a very data centric - I work in Finance in a bank, and I use that to have really good performance consulting conversations with them when they say, "Well, people are clamouring for this data, this content, they really see a gap." I had an example where when I did the search, this particular content that people supposedly desperately needed was coming in at 152 on intranet searches and library services so it wasn't that much of a hot topic but it was their perception that it was. But yet in the top three there were some gaps of topics that we didn't have content aligned to. It almost becomes like you're operating like a news media newsroom, seeing how the Twitter feeds and seeing what's popular, and what's trending, and how you want to pivot and react.

Robin:
Yes, and getting that flowing through in real time. And this is really nice because essentially you're talking about other data sources outside your LMS, bringing them in the easy way. Are there other data sources you think L&D could access in their organisation, that they're not accessing?

Lori:
Yes, I think a big one - If you have a Talent Management System, one that is a really interesting thing to do is to see if - you have to sanitise the data so I always put the caveat on there. Before you do anything with data, do make sure that what you are doing does not contravene any privacy rules in your organisation or in your country, because they vary from country to country. One thing I have done is pulled out of the Talent Management System an aggregate of keywords that people were using in their goal setting and keywords that people were using in their performance reviews. It was very interesting to see some of the trends there so we could see what the top 10 were in both. It gave us a really good preview, in the case of goal setting, as to what was important to people and in the case of performance review, what gaps were. That's a really good place to go.

I would also urge as well, to have a look at things that maybe aren't even within your organisation. I often - I know it's a dirty word but I often go to marketing blogs and you can get some really interesting statistics and knowledge about how people consume content. At the end of the day that's what we're doing, we're about people consuming content. Ours just happens to be learning content. You can see a lot of statistics there and that might influence some of your decisions but what I would caution is, any time you use some of those general population data, every company's and culture's digital body language will be unique to that system. So you can't simply say, "Millennials will always do this." It doesn't always translate that way, you have to really look at yours as well. That's a caution with the general. But in lieu of not having any other data, that's a great place to also go.

Robin:
You'll actually be in the spot where you're looking at how a certain population, how a certain country, how a group of people, cohort, are using media to be able to access - was it you in the degree talk that I heard you actually put together a mobile website that only had two people - yes, it was you.

Lori:
Yes it was, and this is where I would say that cautionary tale. You think of Millennials - we had a new graduate cohort coming in and of course when you think of that age group, they're attached to their phones, they love being on their mobiles and if we built them a mobile site to onboard them, of course they'll go to it. They'll love it! It'll be everything! No, no.

What happened was, you're exactly right, we had less than 2% engaged with it. We tried everything, we had a fancy video, we had avatars, and we just couldn't get the engagement and when we talked to some of them, we found out a couple of things. One: that even though the company had provided the mobile devices to these new graduates, they still had to pay for their own data plan. Nobody is going to use their own data plan to access something for work, That's an HR hurdle.

The second issue was, they saw mobile as being personal, and that was their personal time. On their train home, they're going to check Instagram, or Snapchat, rather than go onto a work program, and had we done our due diligence and had we looked up the digital body language from other mobile sites that had been launched across that organisation, we would have known that in advance. I want to be very clear, that's not to say never build mobile. It's just to say to do it for the right audiences, the one's that are actually going to consume it.

Robin:
And that's a really interesting and good observation as well. Just because of that story, that case, different organisations that had different things around data, people were using their mobiles during 9-5 as well, for work. There might be a really different dynamic around that.

Lori:
Absolutely. If you had a mobile sales force for example, that the mobile is their desktop and they're on it all the time then absolutely, because you have that happening, so you've got to be careful.

Robin:
Lori, you had some really nice, elegant ideas about really simple ways to getting sound in being more data driven. What are your big gems of advice if people want to start to become more data driven in their learning approach?

Lori:
I'd say the first thing to do is, do make friends with your IT department. They are often the keepers of a lot of data that just come out naturally out of reports but they don't do a lot with it. Ask them what they're already collecting. That's a very good way to start rather than asking for favours. It’s easier to go in and say, "What do you already have?" as opposed to saying, "I want this custom report." Because you're not likely to get very far with that. Make friends there.

I think the other thing too is not to be afraid of data. It doesn't need to be technical, it doesn't need to be deep. Look for the quick hits, the top 10s, the really basic things, and they're going to help you. Also too, if you have a social collaboration site - a lot of companies now who've used Facebook at work or they might have some things similar to that, and internal. Those inherently will have a lot of visible data that you don't even have to ask somebody for a report. On our social collaboration site, which is Agira driven, we're able to see right away, how many likes? What are the sentiments for? How many comments? All those sorts of things. Start with those very basic pieces and use that to start getting a picture as to what the digital body language is.

Robin:
That's a really nice thing. Taking on a filter of things that are already around you and sitting there going, "What can I learn from that particular behaviour?

Thank you so much for joining me today. On the website I'll include a link to Lori's eBook on Data Driven Design or you could probably do a Google search on them as well, to be able to get that. It's a great read.

Thank you so much for joining me today.

Lori:
It's been an absolute pleasure. Thank you so much Robin, have a wonderful, wonderful day.