Evidence-Informed Learning Design for Meaningful Skill Acquisition

[00:00:08] - [Speaker 0]
Everyone, and welcome to the Learning and Development Podcast sponsored by InfoPro Learning. As always, I'm your host, Nolan Hout. Joining me today, we have a very well known thought leader in the space for over thirty years. We have Clark Quinn joining us. He's the Executive Director of Quinnovation, where they help organizations work smarter, aligning with how we think, work, and learn.

[00:00:29] - [Speaker 0]
Today, we're going to talk about a topic that sounds pretty advanced, evidence informed learning design for meaningful skill acquisition, but I promise we're going to break it down. It's going to be a lot more simple than it sounds. But considering we have somebody with over thirty years experience in learning design talk us through this, I don't think there's anybody better to navigate us through this topic than Clark Quinn. So Clark Quinn, without any further ado, welcome to the podcast.

[00:00:54] - [Speaker 1]
Thank you very much, Paul, and a pleasure to be here with you today. And sad to say it's actually about forty years. Forty years. Now.

[00:01:04] - [Speaker 0]
Know, I have to admit sometimes Clark, depending on which guest I have, I'll kind of cap it at something because I don't want them to feel, you know, a certain way. I

[00:01:17] - [Speaker 1]
was a child prodigy. No worries.

[00:01:21] - [Speaker 0]
Great. Well, speaking of child prodigy, you know, Clark, I'd love to just start before we kind of get into the meat of the topic. We'd love to just start and learn a little bit. How did you get into this field forty years ago? You know, back then, gosh, wasn't really, wasn't nearly as big a field.

[00:01:38] - [Speaker 0]
What drew you to this? Where where did you get your start? I

[00:01:45] - [Speaker 1]
won't go into the long, short and twisted tale, but briefly, I saw the I was doing a job doing the computer support for an office that did tutoring, and I had been tutoring on campus as well. I got that job, and I saw the connection between computers and learning. I said, this sounds like something interesting. And our university at that time didn't have a program in it, but they had a program where you could design your own program. And I designed my own major in computers and learning, and essentially it's been my career ever since.

[00:02:17] - [Speaker 1]
It's taken strange twists and turns. My first job out of college was designing and programming educational computer games. So that's remained a recurrent theme is how do we make experiences both effective educationally and engaging? Realized we didn't know enough and went back to get a PhD in basically applied cognitive science. And, you know, did the academic route for a while, got involved in corporate learning and have ended up here.

[00:02:45] - [Speaker 1]
But it was that recognition of the connection between computer supporting learning and that background in the cognitive science of how we think we're going to learn to and saying, the sad thing is we don't do a good enough job of aligning them particularly well. And so that's been essentially career.

[00:03:11] - [Speaker 0]
Yeah. And so, I mean, you mentioned the twists and turns. Tell us, you know, what are you doing today at, at Quinnovations? What is your really, where do you find yourself spending the majority of your time?

[00:03:23] - [Speaker 1]
I continue to unpack the details of learning science because a number of years ago, Cammie Beam wrote a book, The Accidental Instructional Designer. And that's the sad state of affairs is we end up there. You wouldn't want an accidental plumber. You may do it around your house, but when you hire somebody, you don't want to be accidental or an accidental surgeon. And yet we are trying to systematically change arguably the most complex thing in the known universe, the human brain, in reliable ways.

[00:03:59] - [Speaker 1]
And we're sort of, you know, like, this is probably mischaracterizing monkeys, monkey with a hammer, right? This sort of banging away without really knowing what we're doing in way too many instances.

[00:04:17] - [Speaker 0]
Okay. So you're helping them kind of bring back the science to a field that, as you said, is, is kind of, we're trying to take the science out of it. Or, or I guess we're, we're not trying to, we are, through various ways. So, there's

[00:04:36] - [Speaker 1]
lots of pressures that mean we're not doing what we could and should be, and we need to fight against that. I'm working in a number of ways. I help people create, improve the output of their product, which typically involves looking at their process and figuring out ways to what are the, you know, it's really hard to go in and totally revise what you're doing. That's rarely what happens. But instead, if you can find those initial small tweaks you can make that will really improve the learning outcomes you're achieving.

[00:05:07] - [Speaker 0]
Yeah.

[00:05:08] - [Speaker 1]
You get, and if you, you know, track that and evaluate that and have that evidence, you can get the support you need to make bigger changes and eventually start doing what we know, what evidence shows we should be doing.

[00:05:23] - [Speaker 0]
Yeah. Wonderful. Wonderful. Well, talking about what we should be doing. We, you know, I mentioned this very lofty title, Evidence Informed Learning Design for Meaningful Skill Acquisition.

[00:05:34] - [Speaker 0]
Let's start breaking that down. The first thing I want to talk with, you know, is the end. The reason that we do it is for skill acquisition. Now there's been a lot of talk past couple of years, maybe before then, you know, this job versus skill. Are we training to do the job or training to do the skill?

[00:05:51] - [Speaker 0]
Why is it important that we focus on that skill acquisition? Why does that become kind of the, the, the, the benchmark of where we want to be, be monitoring, measuring?

[00:06:02] - [Speaker 1]
There's a couple of reasons and it was somewhat dismaying to hear you describe, you know, evidence based as a, as a heavy title. It really shouldn't be. We should be, you know, based in known practices, as most other professions would be embarrassed not to be. But what we see too often is a lot of information jump. And to go back to your question, I want to get there,

[00:06:34] - [Speaker 0]
but

[00:06:35] - [Speaker 1]
to go back to your question, roles are changing faster and faster. That's not just because of people introducing AI, and I think there are problems with that, but what matters are the skills and different organizations will break up the things they do. So in some organizations, somebody will be responsible for analysis and they'll hand off to somebody else for design and hand off somebody else for development. Whereas in other organizations, one person will take thing all the way through and another person will take something else all the way through. So, what matters are the skills of doing analysis and the skills of doing design and the skills of doing development, as opposed to what is your job?

[00:07:18] - [Speaker 1]
And then we need, what we're now recognizing is that those skills we need, if we have really good definitions around them, we can develop them. If we have vague definitions, we tend to dump a bunch of information on people and think, well, we hope they're going to figure out what that means for them in practice, and we have reliable evidence that that doesn't happen. So we really need to focus on specific skills and how do we develop them and how do we evaluate them so that we can be sure that we're creating the capabilities that our organizations need. How do you, how do you do that? And I, you

[00:08:01] - [Speaker 0]
know, I remember like five years ago going through these types of exercises and they were enormous exercises trying to do skills mapping. And then AI came in and said, we can, we can get you close, right? We can, we can get with our measure of accuracy of what are the top 10 skills that a marketer needs. Now we hope you come in and correct the model, you know, tell us where we've, we've, we've made a mistake, but I feel like it kind of lent itself more into that close enough is good versus nothing, right? Versus having zero definition.

[00:08:38] - [Speaker 0]
So where do you really fit on this? Well, maybe getting it a 100% accurate will never be possible or even 95%. We're better off doing 80% accurate across the 3,000 person org versus a 100% accurate. And then we change it every two years. Cause you mentioned the, you know, the skill decay is, is at its all time high.

[00:09:02] - [Speaker 0]
Shouldn't. Well, is it the highest ever been now? I'm sure tomorrow it will be higher and vice versa. How, where, how do you like know when to stop? When is good enough, good enough?

[00:09:12] - [Speaker 1]
You don't know when to stop. You're doing this continually, what you have to do is figure out what skills are critical that you, for where you're going. So it's a strategic thing that says, where are we going as an organization? What are the directions? What are the skills we absolutely have to master?

[00:09:32] - [Speaker 1]
There are certain things that are, you know, background and things that are critical to the success of the organization. And so I'm listening to people like Karim Pagano, who's coming out with a skill book and Kevin Wheeler, who's in charge of, you know, who's a talent advisor used to do corporate universities. And they're talking repeatedly about going beyond just taking an order for a course and figuring out what does the organization need as critical skills and let's focus on those. So you're taking a subset of the 3,000 odd roles. And it's not that, you know, some of those skills may be distributed across multiple roles,

[00:10:11] - [Speaker 0]
you

[00:10:11] - [Speaker 1]
say, what do we have to understand? What do we have to be able to do? Do we have to understand this material that's core to our product? Do we need to understand this, approach that's core to our services? What is it that's going to be critical and where are things going?

[00:10:27] - [Speaker 1]
Where are directions as an organization? What skills are necessary and focus your efforts there. So you're not trying to boil the ocean. You're being very focused. And then you do a bit more analysis and AI can help as you're pointing out.

[00:10:39] - [Speaker 1]
Right. But you're never going to get it a 100%, but you're going to find those critical skills and develop those properly. And that's, what's going to make, I think the biggest difference to your organization.

[00:10:52] - [Speaker 0]
Yeah. Yeah. So, so, so as we start to, when we narrowed in on our skills, we've said, you know, that, that's what we want to work on versus the job because the job changes so quickly, as you mentioned, you know, if we can actually get the skills right, that actually should echo across many roles, right? Communic- you don't need to be a good communicator. Being a good communicator doesn't just lend itself to salespeople or customer service reps or whoever it is.

[00:11:17] - [Speaker 1]
Or leadership.

[00:11:18] - [Speaker 0]
Exactly. So then let's now migrate into how,

[00:11:25] - [Speaker 1]
how

[00:11:25] - [Speaker 0]
do we design programs to maximize that impact? Right. We talked about evidence informed learning design. How do we start focusing on, okay, this is the skill that we're trying to move. How are we designing using evidence to start making an impact on improving those skills, enhancing those skills?

[00:11:44] - [Speaker 1]
Well, the first thing we have to do is find out what is the barrier to that performance. And you look at performance consulting, which really comes out of an, sort of an adjacent field to instructional design. And it says, what are the barriers to people doing this? Is it a lack of knowledge and skill? Or is it, they could do it if they had the resources or could they do it if they thought that's what they had to do, but they believe they should be doing this other thing.

[00:12:12] - [Speaker 1]
There's a whole bunch of different reasons why people don't do what you're supposed to do. And only some of them are skilled and some of them can be better solved by job aids than by courses. Because you're not trying to put information ahead. You say, Hey, it can be in the world. It's something they don't do very often.

[00:12:29] - [Speaker 1]
It's really important that they do it right when they do it, but they don't do it often enough. The training is going to be gone by the time they need to do it. Let's give them a job aid just to guide them through it, little tool, support. And then when you find the right thing, that's when you move to the evidence about how do we then learn skills as opposed to how do we develop job aids, or how do we change the incentives in the organization. But when you then need to figure out, okay, it's very clear that this is a skill that people don't know when they need to know.

[00:13:04] - [Speaker 1]
So now we need to design a skill acquisition sequence. That's when you start getting in the evidence specifically about learning. And that's where you start saying, okay, the first thing we need is a very good definition of what success looks like. So what is a good objective? And for all that it's very behaviorist, I like Mager style objectives because they get down into criteria that says doing this in this context to this level of accuracy.

[00:13:32] - [Speaker 1]
Because then you can say, okay, I know what the end result is. I can test and see if they can do this now to that level of accuracy. They're good to go. You state a good objective about what it is and you know who the audience is and what their existing skill levels are. And then you can start saying, well, the most critical thing to doing that is retrieval practice.

[00:13:53] - [Speaker 1]
We need to have people practicing doing the things they'll need to be doing in the real world after the learning experience. Right? If you need to communicate, we've got to give you practice communicating before it counts, because if you get it wrong when it matters, and that, you know, there are multiple dimensions that says, well, how important is it if they get it wrong? How frequently do they perform it in the real world? How much are they coming to, you know, to the game with already?

[00:14:23] - [Speaker 1]
And all that factors into the design of your learning experience. But you start saying, what is it they need to do? We need to make sure they do the right practice. Then you assign the minimal information. It's not just content.

[00:14:36] - [Speaker 1]
Too often as in, you know, Learning Private, we talk about content. What roles in the skill development does that content play? You start understanding we need mental models that explain how the world works so we can make our decisions about what to do based upon what we know, but the consequence of doing this, we can predict if I do this, this will happen. If I do that, that'll happen. This is better than that.

[00:14:59] - [Speaker 1]
I'm doing this. But you can't know that if you don't have a model of how the world works, and then you need examples to see that model in play. And then you'd give, and we have reliable evidence that giving people a few examples, worked examples, before they actually take turn themselves. They learn better, faster. And then we give them a retrieval practice, but you have to understand all this and it has to be spaced out over time.

[00:15:26] - [Speaker 1]
Too much of what we're doing is based on an event that says, well, it's hard to get people together face to face. It's expensive. So you minimize that. But just an event or, and same with e learning, you know, you sit down, you do your half hour of e learning and you go away. Most of that's going to be gone in a day or two.

[00:15:46] - [Speaker 1]
Do you Yeah.

[00:15:48] - [Speaker 0]
And I mean, I think it's, you know, everything we're doing, we're trying to kind of shrink that investment versus payoff and, and to get that gratification as quick as we can. But, but you can't, I mean, if you're really trying to acquire a skill, it takes time, right? If it was so easy, then everybody would have every skill, right? You could just tell somebody, Hey, you've got thirty days, go learn every skill. If that's really how we thought people learned skills.

[00:16:21] - [Speaker 0]
But if we, you know, it's like that's an extreme, right? If somebody said, Oh, well, of course they can learn this skill in a three day session. I'd say, well then hell, let's invest the next sixty days to teach them the top 20 skills and we'll be good to go. We never have to train them again. And I was talking to this gentleman at Amazon and he was like, you know, I, he's like, I don't really know what the number is.

[00:16:41] - [Speaker 0]
I try to say like, you can't be working on any more than two or three skills at one time because it is such a deep thing. What, what are you like, how do you think? Because now everything is this skills conversation and everybody's creating these skills gaps and there are rakes to close it as quick as they can. How, how much can we do at once? Right?

[00:17:03] - [Speaker 0]
Like if I look at my skills gap, are, are you a proponent of, you know, let's give them a little bit of, let's put a drop in each bucket kind of along the way. Let's kind of try to master this one skill, move on to the next. What are your thoughts on how much the human brain can even kind of consume and actually make a difference?

[00:17:21] - [Speaker 1]
Yeah. And you'll love my answer is it depends. Back to those factors I was talking about how important it is, you know, you'll want to invest more. How frequently they perform it afterwards really plays a big role. Cause if it's that only happens, you know, once a week, you're going to need a lot more practice to make sure you're doing that right.

[00:17:43] - [Speaker 1]
Than if it happens every, you know, several times a day. If it happens several times a day, you may need different types of support. Togo Wanda wrote his book Checklist Manifesto about people who did several things at times a day, and they think later in the day that they'd done the step because they'd done the step earlier in the day and they hadn't done it in this instance. And so that's why he created his checklists. So there, it's a mix of tools and, and, and training.

[00:18:08] - [Speaker 1]
But to go back to your question, one of the things we need, you need to get up to a minimum level that first day of some level of the skill, but then you literally need sleep and then you need to reactivate it a couple of days later. It turns out the best time to reactivate it would be when you're just about to forget it. But that's really hard to predict, particularly at scale. So you do some good things that, by the way, the two, two, two, isn't quite right. Two days, two weeks, two months.

[00:18:39] - [Speaker 1]
No. It's more complex than that. I'm working with a startup right now. That's trying to figure out, you know, provide support for learning events to extend the learning afterwards and figuring out what works best there. But you need that half day or whatever, but you can only, to your point, you can only learn couple things.

[00:19:03] - [Speaker 1]
It helps to interleave. So learning a couple things is good. Interleaving means I study a bit of this and then I go study a bit of that. And then I come back to this a couple days later. And I come back to that a couple days.

[00:19:15] - [Speaker 1]
You're mixing things up is better than just doing it all at once. You know, the same thing, then you have a little bit less predictability about what it is you're going to face, which is a slightly more challenging retrieval task, which actually leads to better learning. You get more quickly acquiring and deeper acquisition. It strengthens the links better. So you need some spacing in your leaving.

[00:19:42] - [Speaker 1]
What matters also is the level of challenge, the desirable difficulty. Learners think if it's easy, it's good. But that turns out not to be what the evidence tells us. They need to struggle a bit. What Seymour Papert called hard fun.

[00:19:59] - [Speaker 1]
You know, when you play a game, it's not, you don't always get it right or it's not quite fun, but it can't be, you can't fail too much or it's frustrating. There's this zone and that zone changes over time. Zixam Ahalya talked about the zone for flow, and that's what games tap into, but so does learning because, Vygotsky talked about the zone of proximal development as well. You know, there's stuff that's too hard, too easy, and in between is where learning happens. And so the level of challenge, the amount of spacing, the amount of interleaving, this combines and the inherent complexity of the task.

[00:20:35] - [Speaker 1]
Is this something that has only a few factors or is something that has multiple factors? All of this comes together. So that's why, by the way, you make your first best guess and then you test and tune. And if you can't, cannot do the build it, it is good. You have to build in some testing and tuning time into your schedule And you have to find a way to make that work.

[00:21:00] - [Speaker 1]
And that's really hard for a lot of people, but if you actually care about improving people's ability to do instead of just ticking a box saying, okay, I gave them the information that's up to them. And you were hinting at that, you know, oh, well we give them three days and it's, or, you know, it's golden. Yeah.

[00:21:17] - [Speaker 0]
I want to expand on one of those things. I think it's important. I think one of the big struggles that a lot of, of LND practitioners face now is like, I'm just, I'm like trying to, you know, knock things off of my list. And I have this, this list that has no end and a budget that has a very finite end. And so, you know, you said something and obviously somebody like yourself who has such a rich history and so much knowledge in this space.

[00:21:49] - [Speaker 0]
I think it's might be easier for somebody to say, well, gosh, if I can't do all these things, then I shouldn't even start. But what I heard you say was, you know, if you don't have the means or the time, or if you feel that, you know, more research isn't going to get me a much better answer, just go with your gut. But then if you don't have a ton of evidence, go with whatever you have, Do it, you know, implement the course, whatever it is, but then go back and look for what are, you know, was I right? And if I wasn't right, then tweak. You use essentially your first launch as your means to gather more evidence.

[00:22:28] - [Speaker 0]
So you don't necessarily need to be perfect the first time every time. Get something out there. Yes. Do you know, lean prototype, get it out. But if you're not going back and saying, what am I learning from this either?

[00:22:43] - [Speaker 0]
And it's either, what am I learning from this to apply to, you know, to edit this one, but also what did I learn from this that I now need to take into my next? I think so. So, you, is that kind of what you're, you're, you're a proponent of is, is yes, you know, maybe you can't get everything, but if you don't at least take something away, if you don't build some evidence off of the program, then you've actually, you're leaving a lot of meat on the bone.

[00:23:12] - [Speaker 1]
Well, So you should be reflective practitioner, right? I'm a big fan of Megan Torrance's LAMA or Michael Allen's SAM as processes that are based on Agile and they're iterative. And, you know, BlackLine has most cases, three iterations is good enough. But absolutely what I'm advocating though, is knowing enough about the background that your first program is going to be pretty good. If it's really bad, you're going to test it and make some tweaks and test it and make some tweaks.

[00:23:44] - [Speaker 1]
It takes a lot longer. The more you know, the shorter, you know, the better your first iteration is. And then you, they have the advantage of what Michael Allen, Allen Interactions team has is they work in teams that makes life a lot easier than trying to be a solo practitioner, but even a solo practitioner, you can know a lot about, you know, using mini scenarios instead of writing knowledge test questions. Mini scenarios are retrieval practice. Knowledge test questions don't do anything.

[00:24:17] - [Speaker 1]
There's research that shows this, asking low level questions develops your ability to do low level questions. What your job requires is high level questions. If you do that, and you don't need the low level questions, just the high level questions will get you the high level abilities. But if you don't know this and it's comfortable and easy to take that knowledge in that PDF and ask some random questions about it, you'll do that. So you need, it helps to know upfront, but absolutely.

[00:24:48] - [Speaker 1]
Even if you know as much as possible, you still should bake in some time to test it, tune it, refine it. And both of those, by the way, prototype the final exam, final retrieval practice first and refine that and use that as a basis to work backwards. Cause you should figure out what your final essentially test is the way your final assessment to determine criteria. And then you work backwards to make sure that they can achieve that. But that's what they prototype first is that practice.

[00:25:23] - [Speaker 1]
And they align everything to succeed at that after they've got that right. That is a shorthand way to get, you know, to do what you said, achieve the best outcome with the least use of resources.

[00:25:39] - [Speaker 0]
So now let's, let's build off of that idea of let's get, you know, something out there, but let's get it as close to accurate as possible. I think the, that is the, claim of using AI to build out a lot of these learning designs, right. Is in the littlest amount of time possible, I'm able to put something out there. I'm able there to get content out into the world. And, and people can retrieve it in many different ways.

[00:26:10] - [Speaker 0]
We can make videos out of it. We can make infographics. We can make whatever and in the time that takes. And I was like thinking about it, that, that idea of, and I talked to a lot of people about this because I use AI pretty heavily in my, in my field. And I realized that when I use it versus when I have a, I had a college grad join and I said, listen, I'm doing an experiment.

[00:26:37] - [Speaker 0]
I know, you know, nothing about our field. I only want you using I want this to be your, your, your starting point. And I realized that the, the accuracy of what, you know, I'm using the tool, this person's using the tool. We're using the same knowledge base, the same everything. My output is, you know, 90% accurate.

[00:27:01] - [Speaker 0]
Their output was 50% accurate because they didn't know what right was, you know, you did. And so I realized, you know, like with AI, you know, as we put more out, there is a real, like, I don't know, potential problem coming of, yes, we're putting out more knowledge, but we're putting out more knowledge that is not what we intended because the people putting it out don't know the evidence. They don't know why it's doing that thing. They don't know why I said, why did it design the program this way? What are your thoughts?

[00:27:41] - [Speaker 0]
I mean, you, you've seen many evolutions throughout your forty years, of, of learning and you've attached, you know, or, or, or not attach yourself to all the buzzwords, M learning, micro learning, what was it? CBT, all the different things. What, where are you seeing, how are you seeing this idea of AI kind of, maybe everything we talked about was kind of this evidence based and the science backed. How is AI impacting that field?

[00:28:15] - [Speaker 1]
Okay. I'm going to take a brief bit to characterize the fact that I've been involved in AI relatively deeply for those decades. I was an AI and I've remained being an AI groupie. So I'm not a practitioner, but I follow what it means and I know conceptually how it works. I've played with it.

[00:28:36] - [Speaker 1]
I've programmed, so I understand it. But I've also been around several of the seminal moments. So, you know, there was the symbolic AI and I, then I was in a graduate student lab of Don Norman and the other lab leader was Dave Romohart. And Romohart is the one who, and McClelland and their grad students were the ones who recognized that there were struggles of computational models of cognition, AI, to actually capture what humans really did. And there was in cognitive science, there was this sort of post cognitive situational vision and what happened with some, with computational stuff.

[00:29:25] - [Speaker 1]
It wasn't working. They were building all these models and they were trying to get this sort of weird human behavior out of these systems. They were doing things like Douglas Hofstedter was trying out slip nets, which were slippery versions of connections and Lofty Zetta had his fuzzy logic. Rommelhart went back and fundamentally went back to what Minsky and Papert had done with perceptrons and realized that what they hadn't had was a hidden layer. And they produced the PDP books, the parallel distributed processing, which revolutionized machine learning.

[00:30:00] - [Speaker 1]
They were the ones who created what we now use as neural nets and what generative AI is built on. And I have to be clear when you say AI, most everybody today is thinking generative AI, right? And that's what you're using are these things like chat, GPT and cloud and everything else. I've seen the transitions and I worked in a lab in a, I was taught in a school of computer science and I hung out with the AI guys. And so I've just been around it a lot and built an adaptive learning system, went to AI in education.

[00:30:31] - [Speaker 1]
So I have a bit of understanding about this stuff and what generative AI I'm somewhat negative about, the hype. I have no problem with the underlying concept, except it's just doing a much better job of predicting what to say next. And when it's generating content, it's generating content out of the mass of the internet. And a bit of a worry about stolen IP, but we'll leave that aside for now. It's creating stuff that's the average in it.

[00:31:01] - [Speaker 1]
So it's average content. It's not excellent content. It's average content and it hallucinates. The way it produces things means it's going produce things that sound right, but may not be correct. They will talk about learning styles happily, for instance, until you point out that that's not a valid thing.

[00:31:22] - [Speaker 1]
They'll say, Oh, excuse me. And they're very nice. We'll say, Oh, sorry.

[00:31:24] - [Speaker 0]
I'll take that. Tell people view your agent like a golden retriever. It wants to please you more than anything in the world. It just wants to please you. And so if it doesn't have an answer, it's going to give you one because it thinks that's what you want.

[00:31:43] - [Speaker 1]
Absolutely. And it's predicated on it, but so you have to be the expert about learning design and you need to have experts on the content if you're not to validate what it says, which take makes, slows it down. Now I think it can be a really valuable partner coming up with ideas for scenarios, situations in which scenarios can happen. I don't think, you know, the evidence is doesn't build models of the world. It doesn't understand context.

[00:32:10] - [Speaker 1]
It's just highly predictive of language or video or images.

[00:32:16] - [Speaker 0]
So

[00:32:18] - [Speaker 1]
you have to figure out what are the core decisions that should And be for the then you can have it come up with ideas for nourishing. You can bet some, and it'll come up with ones you haven't thought of that are good, but it'll come up with ones you haven't thought of that are bad, and you have to be sorting it. So it's a great partner for thinking. It's just not to be trusted to do anything on its own, which is why I really worry about agentic AI, as well as we have evidence that people are been able to corrupt them and get them to do bad things. Cause they want to please, right?

[00:32:52] - [Speaker 1]
So they want to please you and they'll go off and do stuff that you've told it to do that it shouldn't do. And it's been told not to do. Asimov's three laws of robotics comes to mind. But anyways, so I worry about the IP. I worry about the environmental costs.

[00:33:10] - [Speaker 1]
I do worry about the business models. Right now the costs are supported by venture capital. When that goes away and you have to account for the environmental energy costs and the, you know, there's evidence now that smaller ones, smaller AIs purpose built are less cost environmentally and more effective at meeting needs, but that doesn't fuel the business models of the big AI engines. So I'm getting off into the weeds here. I apologize.

[00:33:39] - [Speaker 0]
No, I think it's a, I, you know, I, it's funny. So I, my, my wife is my wife does not like AI in any ways, mostly, I mean, I think part of it is just, you know, scared. But then environment is a big one for her. You know, anytime I use Clawd, she's like, well, there goes another gallon of water and you know, this, that and the other. And it's, it is true.

[00:34:02] - [Speaker 0]
It's absolutely a valid cons like, you know, as we're seeing the, the, impact on the environment we're seeing. And the interesting thing is like, I think it's, it's, it's hard for the mass to understand every tech innovation of any science. Mean, tech very loosely, right? Like even, even the printing press, right? Huge impact.

[00:34:32] - [Speaker 0]
How many more trees did we cut down to produce the paper? How much more ink did we have to create and leach into our water system to make more copies of books? How much they've shown like mass cropping has a much harsher impact on the environment than an individual person who, you know, grows corn in their backyard. All of these, and then, you know, so that's the environmental side, but then you have the social side and I think it was, I can't remember if it was in a Sapiens or if it was in a brief history of nearly everything, but they talk about how every of these major innovations are always well intended of, you know, the best example I can always give is like, you know, email. Email came out.

[00:35:17] - [Speaker 0]
I was like, Oh my gosh. I now don't have to go to the post office. I don't have to spend 30¢ for a stamp. You know, now it's whatever, 52, but I don't have to spend 30¢. I don't have to write this letter or type it or whatever it is.

[00:35:34] - [Speaker 0]
I don't have to wait five days for the answer. I can just send an email right from my machine. And we all thought this is going to be great. How much time am I going to save? But now we get 300 emails a day.

[00:35:48] - [Speaker 0]
Are we really saving more time? Is it really more efficient? Have we made our lives much better? And, and so I also think about that with AI and I was talking to somebody else about, you know, just this idea of if I can do the job of 10 with AI, yes, it's helping me make my job. Yes.

[00:36:09] - [Speaker 0]
You said it's like a good thinking partner, but now I'm going be asked to produce 10 times 10X. And so this job that used it, you know, I had eight hours in my day now that I have AI, it only takes me four hours to do that. It's only a matter of time before that four hours is going say, well now use AI, we're going to expect eight hours of AI work. And so there are like a lot of these things I think, but what I, what I hear you talking most about is just the, there is a lot of harm of not knowing what it is that you are using. And, and, and, and, and as you, since the tool is more powerful than maybe what you've used in the past, the ability to get it wrong is magnified, right?

[00:36:56] - [Speaker 0]
If any of you play golf, you use your driver, you hit a bad drive, it can go 300 yards down the fairway, or it can go 300 yards into the ocean, you know, into the pond. But if you use your putter, you're going to go 30 feet or you're going to go two inches to the right. So like, you know, a big miss. So is that kind of where you, I guess are advising people today is, is the people that are using it ought to know its limitations, know what it can do good and what it can't do so that we're not expanding and multiplying every mistake.

[00:37:32] - [Speaker 1]
Yeah. I just shared a report this morning that with a colleague that about how the organizations that are sitting with AI are very focused purposes that they're trying to use them for and using them as a general solution is ending up not achieving the outcomes they were hoping for. There's evidence as well that when you offload that cognitive processing, you're not doing that cognitive processing, you're not learning. And it's even worse for kids who are supposed to be doing that cognitive processing. When they don't do the writing themselves, they're not doing the thinking, they're not learning.

[00:38:06] - [Speaker 1]
And so you have to be very careful and everybody's going, Oh, I'm going to use AI to support learning. It's going to I was communicating yesterday with somebody on LinkedIn who was going, Oh, I've got this company and we take your content and make videos about it that are really engaging. Yeah, but it's still content dump. Where's the interaction that's supposed to Cause they were touting it for learning. It's like, that's content.

[00:38:28] - [Speaker 1]
Where's the interaction? What's going to lead to actually developing abilities? They didn't have an answer for that. It's, yeah, there are big worries you really shouldn't be throwing a technology you don't fully understand the trade offs for, to problems. You should, and the best advice I hear, from people like Marcus Bernhardt, are strategists about assets, know what you're trying to achieve.

[00:38:56] - [Speaker 1]
Do experiments, but do smart experiments, but don't, you know, and also saying don't make a relationship with a company longer than three months because the market keeps changing what matters. And so what you're using and, you know, whatever prices you're paying now may not stay stable going forward. There are lots of issues. You bring up one and I think that's apt. But, my take is use it to compliment the things you want to offload.

[00:39:31] - [Speaker 1]
You've heard the silly meme. I don't want AI doing art and making music while I'm doing the laundry. I want to reverse that. I want the AI doing the laundry. I want to be doing art and making music.

[00:39:47] - [Speaker 1]
Both of which I'm horrible at by the way. But that's not me, but the meme is apt. Think we have made a choice. We made it decades ago. I was reading Popular Science of Kid and they were talking about how we're going to work a day a week in the future because we'll use technology to make our lives easier.

[00:40:05] - [Speaker 1]
We didn't make that choice. We have to think about our choices in that respect going forward, just to your point, you know, being asked to do well, if I give you AI, you can do twice the work. So I now expect twice the work from you instead of you can work half the time.

[00:40:23] - [Speaker 0]
Half the time. Yeah. Yeah. So let's, I don't know if we'll win out with that. I don't know if you and I will win that debate.

[00:40:31] - [Speaker 1]
Well, have a choice about that too.

[00:40:33] - [Speaker 0]
Yeah. Yeah, absolutely. Well, thank you, Clark, for sharing your wisdom with us. This has been an excellent podcast. Before we wrap up any closing remarks, any points that we didn't get to you want to leave our audience with?

[00:40:50] - [Speaker 1]
Let's see. I didn't mention, you know, this evidence based stuff, it takes a while to acquire it and to put it into practice and recognize it. I'm co directing the Learning Development Accelerator Society around Evidence Based L and D. And so, if you want to have a way to accelerate your understanding in an easy and vetted way, and we very much don't have corporate sponsors and we do have an advisory board that are some of the best known translators of research to practice people like Wilt Thalhimer and Ruth Clark and Patty Shank. These are advisory board.

[00:41:29] - [Speaker 1]
These are people you should look to, Julie Dirksen. The list goes on. These are people who understand what the research says and can translate that into practice. They're people you should be paying attention to, because evidence based shouldn't be this scary, heavy topic. It should just be part of our practice.

[00:41:48] - [Speaker 0]
Yeah. Yeah. And where can people find that information, Clark?

[00:41:52] - [Speaker 1]
Ldaccelerator.com.

[00:41:54] - [Speaker 0]
Ldaccelerator.com for those that are wanting to learn more and Clark, you can find him on LinkedIn, a great follow post, really, really good stuff. Encourage you to check that out. Well, thank you so much, Clark. I appreciate you spending some time with us. I look forward to possibly doing this again soon.

[00:42:11] - [Speaker 1]
No worries. I appreciate the opportunity, Nolan. And I just wish all your listeners stay curious.

[00:42:22] - [Speaker 0]
Thank you, Clark.