Lindsay Velecina: 0:03

Welcome to the scrum.org community podcast, a podcast from the home of Scrum. In this podcast we feature professional scrum trainers and other scrum practitioners sharing their stories and experiences to help learn from the experience of others. We hope you enjoy this episode.

Patricia Kong: 0:21

Hi, everyone. Welcome to the scrum.org community podcast. My name is Patricia Kang. I'm with Scrum. org and I will be your guest host today giving Dave West a little bit of a break. So, today's discussion is about evidence based management, also known as EBM, which is framework that carpenter and I use on this podcast have been co developing this for 10 years. And the notion of EBM evidence based management and the difference in the similarities and etc, etc with objective key results OKRs, which is the model that's been really popular, especially in the management consulting group. So for today's discussion, not only do we have Kurt Bittner, who's my colleague, again, that's from the org looking at Enterprise Solutions, and he's the editor for our professional series book. We have Laurens Bonnema the Netherlands, and he is a professional scrum trainer, agile management consultant and to my delight, a graphic facilitator. Did I leave anything else out, gentleman?

Laurens Bonnema: 1:29

No, that's pretty much it. Good for me.

Patricia Kong: 1:34

All right. Well, thanks for being here. I wanted to have a short discussion about specifically EBM and OKR. So for those who aren't that familiar, evidence based management is a framework from scrambled or created by country labor. And for us, it is focused on and this is very general. But you can say that Patricia concept, this is very general, but it's focused on measure men management and empiricism toward goals. So in a nutshell, that's what it looks like and objective key results OKRs, to me is is really focused on milestones and results toward objectives. So we have an EBM guide, and we have an EBM class, a pally BM class. And a lot of the things that come up during that class had been Hey, what are what are the similarities? What's the difference? Why would we care about EBM? Well, we have OKRs that everybody seems to love. So for me what I've been hearing and what I see is that the similarities are Yeah, there. There's an objective goal orientation. There's this notion maybe of improvement, alignment, transparency, you know, evidence or measurement. I'm interested in Lorenz, what other similarities do you see in EBM and OKRs. And then maybe what would be interesting is for us to then go into the differences.

Laurens Bonnema: 3:00

Know what I see in addition to what you already mentioned, is that both methodologies, they truly encourage organizations to constantly evaluate their performance and to like adjust as they need to or as they feel they need to and to run some experiments with it and to see literally empirically look at what is happening and then adjust as they need to. They also they aim to align the efforts of the organization towards achieving specific goals, and they could be strategic or non strategic goals. And they're designed to be adaptable, right. So a goals can change objectives can change. You do that as is necessary based on the performance and, and changing circumstances. Now, that is a very obvious statement in a sense. But usually, there like an answer to hard core KPI thinking, right? We have this number, it's set in stone, we have to hit it, this is the target, and let's let's not change anything, and then both EBM and OKRs. In my opinion, they, they help you have a more human take on that.

Patricia Kong: 4:11

What about you, Kurt?

Kurt Bittner: 4:14

I see a lot of the same things with Okay, I see OKRs as a very specific, in a sense technique that you can use to apply the ideas that we've described in EBM. So in an EVM, we have three different kinds of goals or three different levels of goals. There's a strategic goal, which tends to be multi year and in, in focus, we have an intermediate goal which tends to be sort of quarterly or so you know, quarterly to half yearly in focus and we have immediate tactical goals which tend to be like, you know, if you're if you're listening to listen to me, right scrum which you probably are attends to be basically like a sprint goal, something that's very short term and focus and very focused on a specific result that the team is wanting to achieve. So with OKRs, generally OKRs are quarterly in focus doesn't have to be but typically quarterly in focus, so that aligns relatively well with intermediate goals. What, how ABM and, and OKRs reinforce each other is that OKRs don't really talk a lot about long term strategic goals. So, that part of EBM can can be sort of brought into the picture and OKRs don't talk a lot about sort of a day to day week to week level of goals, which is where the immediate tactical goals may be, can come into play. So I see the two techniques complementing each other. And okay, there's being kind of a very specific way to apply EBS, EBM EBM could could be applied in other ways. So, it's EBM is more general framework, okay, or more more specific technique.

Patricia Kong: 6:08

Alright, the general framework piece for me too, is that within EVM, and the the notion of the goals is that we have the the four key value areas, which are lenses that can help say, Hey, here's how we're going, where do we want to improve and kind of inform those goals? And I know that we're obviously very big on experimentation, empiricism. Now, I know that the timeframe for OKRs quarterly but is it is it is experimentation required with OKRs, or no,

Laurens Bonnema: 6:44

it's not required. Also, it's not necessary to do that to a quarterly rhythm. That's just the habit that has formed. It's similar to EBM in a sense, where you also don't have a prescriptive prescribed specific timeframe to do anything. There's just a certain logic to it like long term, intermediate and short term goals. And for OKRs, they tend to be, well, mostly intermediate, if you compare them to the EVM rhythm.

Patricia Kong: 7:15

So let's get into some of the juicy stuff about what's going on in the industry, and what some of your clients learn. So a lot I know a lot of people are coming to us and I know two different consultancies and research firms and saying, hey, please help me with my KPIs. And they might say this says, Please help me with my KPIs. Please help me with my metrics, please help me with my agile KPIs, agile metrics, and it kind of leads down into the path if we're guiding them into talking about okay, what are your goals? What are the different outcomes, and objectives you're looking for? What are the some of those asks that you're hearing specifically? And at least I know, in the, in the US, we're seeing some companies, not even companies, I would say even even in our industry, like, the teams, the Scrum Masters, everybody's trying to show their worth is a really bad way to see it. But they're trying to do that, because they're nervous about their jobs. And so I want to see if there's anything happening similarly, on your side.

Laurens Bonnema: 8:26

Yeah, actually, there is, there's a lot of ask for it. Because how can we prove that this thing that we're doing is actually working? So there's a lot of fat behind it. But there's also quite a genuine puzzlement just because what should we be measuring? Right? So we're currently usually measuring billable hours or, like time spent? And we know that that's not right. But what should we measure? And what do all those other terms mean? Because apparently, we haven't been able to, like, clearly define that yet. Or it's hard to find, or I honestly wouldn't know why it's that hard to find. But for me, it's always like just to a Google search away, and then I have like, this myriad of options of stuff you can measure. And I think that's the problem that a lot of people they just want to so where should I get started? What's the minimum set? What's the base set? And there I usually start with, with just really basic flow metrics, and then build it from that those aren't even tied into EBM per se or, or to OKRs it just get them into the habit of measuring something and then, okay, so now you're doing this and then Okay, let's move to a more strategic framework of measurement to balance things out. Because the thing that goes wrong with all the measures as they expect to measure only one or two things, and then be done, as opposed to truly understanding that you have to balance the metrics that you use, like you would in the quadrants of EBM To make sure that you're not only in one quadrant and therefore sub optimizing unintentionally, and possibly destroying your company in the process.

Patricia Kong: 10:10

So So I do and Kurt feel free to elaborate here, I do think that there's there's some correlation where for the for for key values, the quadrants that some of the flow metrics can fall into ability to innovate and time to market which look at the organizational capability. So, so we start with the flow metrics, or we start with something or we don't boil the ocean. And then we get, okay, this is what we can measure. It's easy to measure. Curt. Well, what have you done when people are starting to make that step? Okay, we were trying to see what we understand about ourselves essentially.

Kurt Bittner: 10:50

Well, one of the things that I think a lot of, and this ties into the flow metrics idea. So organizations moving from, let's say, a more traditional plan oriented approach, are not really used to working with feedback cycles. So they're used to developing a plan and executing to that plan. And the problem with that, and this includes with EBM, and OKRs, is that if you have if your assumptions are wrong, and you're working to the wrong set of assumptions, it could take a very long time before you realize that you know something about your assumptions around like, like a major goal. So one of the things that people need to do first is just basically start measuring more frequently. So improving what EBM would call a time to market kind of measure, but just basically do a small bit of work, you know, form some hypotheses, do some do a small bit of work, measure, whether you achieve what you wanted to achieve, and then adapt based on that, that feedback. So getting into a feedback loop is, is really important just to begin with, because until you do that, you can't really do much of anything else, you're just working potentially on wrong assumptions. And it'll take you a long time, or maybe forever, before you understand that those assumptions are wrong. But once you do that, then you're faced with the question of well, what what kind of goals should we be working toward? And this is an area where in EDM, we also talked about a couple of different kinds of measures, we talked about measures that are focused on activities. In other words, you know, you tell somebody to do something, and they do it. So that's an activity based measure. There are also output oriented measures, which are, you know, you tell somebody to produce something like produce this report, or, you know, install this package, or, you know, upgrade the, you know, the security configuration of your system or something else. So that's output. And the problem with that, really, when you start talking about what customers want is that customers don't care about what you did. And they don't care about what you produce, they care about whether, in a sense, their experience has changed. So the third set of measures, which is usually more useful for setting goals are outcome oriented measures. And in EBM, we talked about the satisfaction gap between that the customer has between where they are today, and where they would like to be. So what we find is that it's really useful to Sector goals in terms of outcomes. And that has a that has sort of a benefit to OKRs is that, you know, when when I look at examples for OKRs, in literature and papers that people have written in books, a lot of what I see are activity and output or any other measures like you know, produce, you know, install this new HR system, or, you know, or, you know, get, you know, 20,000 new customers or do something, those are all output oriented measures. And it doesn't tell you whether you're doing anything to improve customer satisfaction. So switching to an outcome oriented measure, like, you know, improve customer satisfaction by as measured by by some sort of, you could say, you know, by their usage of a particular feature, by 2% might be an example of an outcome oriented goal. So, doing, stating your OKRs in terms of outcomes, helps you become more customer focused and less internally focused, and that usually has a benefit to your overall business performance.

Laurens Bonnema: 14:50

Yeah, I agree with that completely and to talk about stuff that people have written. Jeff got health. We already discussed it a little bit in In in advance, he has his published a blog post where he promotes a pattern to, to write just better metrics that are really outcome focused and actually with a twist to the outcome should be a human action, a behavioral change something that people do not a system or something that is done. But really, when, when we do that, what will happen? Right? So what will people be doing differently if we do a great job while we measure this? And he has this really simple pattern, where he says, essentially, who does what, by how much and as long as you ensure, in his opinion, that you write down metrics in that sense, right? So and then who is really a hole, right? So it's never a system or a part of your system? It's never a piece of technology or an AI? It is really a human does something different by a certain percentage? And let's say that human is your customer, what do they do differently? What do they, what do they experience differently in the system that you're building for them? And by how much if you measure, if you create metrics that reflect that, then I think whether you're using ABM metrics, or OPR, metrics, it's then easier to, to really measure an outcome. And also, it invites more to do that. So I've been experimenting with that template. And I have to say, in all its simplicity, it's really, really good. And then of course, there, you need to be mindful that you don't write who does what, by how much, literally, right? So you don't actually create it as a template, and then have the mind filter out everything that that that you, that becomes repetitive, because then you lose focus. But essentially, those questions like who is doing this? And what is happening? And then by how much is that changing? Those three key questions, just like with the user story template, right? That, that really helps also to, to make it insightful for people to truly understand because what that's the difference if I write it down, so it's not the website loads faster? VISEART percentage? No, it's users do not experience bounces of the homepage, which is exactly the same thing. But from a different perspective.

Patricia Kong: 17:31

How long would you say, when you're working with your clients, Laurens and you're, and you're getting them into this rhythm? Now? What has been the reaction? And then how long are you spending when you're writing up those types of measurements or goals?

Laurens Bonnema: 17:48

It's actually quite fast, because I usually start with what they're measuring already. And then rephrase it like retune the metrics they're already using into better ones. So it's it looks at this is what you're measuring? Why are you doing that? What's the outcome behind that? How could you phrase this in terms of the behavioral change? And then usually, as we do that, they change the metric, they change the way that they are measuring it, but they don't experience it as a change, because it starts with something they're already measuring. And then just by asking questions, because of what's behind this, why are you measuring this? It's quite easy to help them along the path of just creating a better set of metrics. And then, in all honesty, I then use EBM, especially the quadrants to see okay, so are we in balance here? Like, are we in just one quadrant all the time? Does that make sense, given what we're doing as a business? Or does it not make sense? And usually, it does not make sense. So it's good to balance it out with a healthy set of stuff to measure in all quadrants. That's a very tool Lee kind of way to use EVM. I know. So I'm running the risk of, of the wrath of either one of you. But it's still a great place to start I find and then, because I mentioned EBM all the time, because I mentioned OKRs all the time, then they start to ask like, what's, what's the CBN thing you're talking about? And then then I usually help them to one of either the classes or just give them the white paper who read this and be inspired by this. And then usually they start to do the work themselves. I find that that's much more helpful than me telling them because if I tell them, they usually either don't get it, or there's immediate resistance like No, no, I can't really afford to like retool my whole metric set because I have to report to my boss and it has to be these numbers. But when they when they start to retool their metrics, and then are able to explain why it's a better number, that they're not reporting, why that actually ties in better with the key objective and what's behind it, then it's an improvement, and it's no longer change. Plus I then do what Kurt just suggested that everybody should do, it becomes a feedback loop, right? So it is okay. So this is just an improvement. So when can we see if there actually is an improvement. So, in about one week, two weeks, four weeks, we're going to look at the numbers and see what we should change,

Kurt Bittner: 20:19

that it brings up something really important. You mentioned something about, you know, the people feel like they need to report something to their boss. And there's an implication in that, that is worth talking about. And it has nothing to do with EBM or Okay, or specifically and more how, how measures and how goals are used. And so a lot of managers make the mistake of using OKRs. And to basically reward or punish people, they they, you know, say, Well, if you achieve this objective, then you know, you'll get a promotion or you get a raise or your group will be rated highly. And the same thing would be true of EBM use goals that way. And the problem with that is that there's a, there's a psychology that people don't, you know, the people actually being measured don't want to look bad. And so if you start using OKRs, or goals, to reward and punish, then people will adapt their behaviors, they'll they'll start gaming the system. And so it has unintended consequences. So one of the harder things that we find for people to do is to basically use the measures to learn and evolve. But don't tie performance measures measurement to that. And it's really hard to do, but but if you if you do, if you use OKRs, to sort of reward and punish people, then you're going to end up with a game system, and you won't actually the measures will cease to be useful. Because, you know, if you tell people to produce more of x, they will produce more of x, whether that's the right thing to do or not, is immaterial. And so you'll lose the ability to basically gather that kind of feedback and learn and adapt. So try to separate measurement from, from the whole question of performance. At least, you obviously want to reward people for improving customer outcomes and doing things like that. But in terms of specific activities that they might perform, or specific outputs that they might perform. You know, don't use it for that. And as much as possible, try to D, personalize and D politicize the measures. And so when, let's say the team fails to achieve an OKR, then your response really should be? Well, that's interesting. What can we learn from that? And, you know, what was the measure wrong? Was the objective from was the approach that we took, maybe not the best approach? What could we do to improve in the future, so use it as feedback to improve. But don't don't punish people for Oh, you made a bad choice about you know what you did to pursue that OKR bad that that. And because you won't get the result. In fact, you'll build a system that is almost immune to improvement if you do that. And that's not, I don't think that's the goal of anybody. In working with measures, you want to genuinely improve the system and improve the way that people work. But you can't do that if you use reward and punishment.

Laurens Bonnema: 23:41

Yes, and as much as I agree with that, and I recognize that pattern, I have the hardest time avoiding it. Because whether I teach EBM, or OKRs, or just better KPIs, people are rewarded based on those performance numbers. So for me, it was a really eye opening thing to just change the phrasing of the number, right. So instead of 80%, down in this where it's easy to either get trapped into gaming that number, really phrase it in terms of an outcome for an either end user or a client. And then when you do that, if you game it, either the client says no, that's not my experience. So it's really obvious that you've gamed it, which is bad. Or you're giving the right thing, right. So let's, let's really make that claim incredibly happy. So that number starts to move. It's really hard, even if you gave that to then get a bad result, unless it's unbalanced. And that's the the other part to this that I think is really, really important that you have to have a balanced set of metrics and not just in one nook or cranny where you have a lot of things to do. And sometimes we over focus, right. So this is the key thing we need to eliminate. And then we have eliminated that. And essentially, when we look back, and we essentially well have destroyed the whole yard, or whatever the metaphor is that you want to use there. So for me, that it's it's a key challenge that, that Kurt points to there. And the thing that for me works best to, to change that is to make sure that the numbers, the the performance indicators are written in such a way that even if you gain them, you end up with the results you want.

Patricia Kong: 25:45

Yeah, there's, I mean, there's the notion of having metric versus encounter metric. And I think what what you were talking about Laurens earlier, was really nice in terms of tying together EBM and OKRs. And using it in that totally way. But when, when I think about it, certainly, it's this notion that EBM is in OKRs is having the best evidence available data informed, how are you managing the decisions you make? And then Kurt, I really liked how you were getting into some of the harder conversations that just exist, which is like we can't do this, this is too hard. And and so lucky that both of you wrote a book, good reminder for the listeners, the professional agile leader book that that hits on some of those points. So I definitely would recommend that to people to think about some of those more human aspects and how do we how do we enable and support a team and what type of leader and environment we want to provide. So back to EBM and o KR, I think the decision or the observation that that we're making today with Lorenz is that they can coexist, and it's not an either or sort of situation. Cur you talked about a lot of things that, hey, let's watch what we can avoid. Is there anything else that you'd like to distress and, and help people with with you know, what happens after EBM OKRs? Or what else can they do to get started?

Laurens Bonnema: 27:18

Well, to my mind, combining evidence based management and, and OKRs. It's actually quite easy, right? So if you've just started with, let's say, let's get, let's say you could start with OKRs, because they're all crazy. So that's, that's your starting point. And then if you do that, you just define some specific key results for each objective. And then make sure that after that you apply UBM principles to inform your decision making and run some experiments that test strategies for achieving those key results and, and use the data for that to guide your actions and, and go from from one step to the next. Right. And then if you regularly measure your progress towards those key results, and continuously evaluate the effectiveness of the strategies that you've defined on empirical evidence, then as long as you actually do something with the evidence, and that is that is a big as long if, if the evidence suggests that the current strategies are ineffective, then of course, adapt the approach. And that could be adjusting your key results or redefining his objectives. And as long as you do that, you promote transparency and alignment by sharing those objectives by sharing the key results. And, and it's really if you get started, and make sure that you define all your metrics as outcomes, and not as features or as what's the other word, but anyway, as long as you define it as outcomes, then you're probably good. And if and if it's not good, you should change something. What do you think, Kurt?

Kurt Bittner: 28:55

Ah, agree with all that. And there's one more technique that is useful in doing that, to get that outcomes is that it basically is a variation on the five way approach didn't have to be five, but just asking why. So somebody might have an objective, that is more activity or output oriented. And so asking the question, why do we want to achieve that? If, if the answer is not to improve some kind of customer experience, then you you can essentially look at that as an opportunity to improve that objective to make it more outcome oriented. And the same thing with key results is that you know, the key result your objective might be to improve customer experience. And the key result might be to improve web page load time by you know, some percentage. But the question is, is that really going to improve customer experience? Maybe not? Well, how would we know if it improved customer experience and then use that to refine both The key results and then the the objectives themselves. So I think I think having that open conversation about why is is really important thing for leaders and teams to do to try to really understand why are we doing this? Why do we think it's going to get us closer to our the objective that we'd like to achieve the goal that we'd like to achieve? And what, what might come out of that discussion is a better way to do something. So that, you know, there's always a tendency for leaders to, even inadvertently or unconsciously dictate to teams how to do things. And so to empower those team to think, basically more holistically about how to achieve goals, encourage everyone to ask, why are we doing this? What do we think that's going to achieve? How would we know if we achieved that? And that usually leads to better objectives and key results, and leads to basically better better team performance?

Patricia Kong: 31:05

Great, I agree. 100%? I wonder what your response would be to a question that I've certainly gone is, I appreciate everything that that, you know, you guys have talked about in terms of outcomes and customers. But I'm, I'm, I'm supporting headquarters for the car manufacturer in Germany, I'm an Asia, I'm responsible for this little area. You know, my customer, essentially, is the headquarters. It's, you know, I don't even I don't interact ever with the actual buyer and drivers of the cars. How, how can they see a path?

Laurens Bonnema: 31:47

Well, I would say as long as you're looking at the humans that you're servicing, right, so that could be somebody from headquarters. So what kind of what will change for the better for them? If you improve this? That is the key question. So it doesn't always have to be an end user, it could be the user of the thing that you do. And that can be somebody at headquarters.

Kurt Bittner: 32:11

I think it's also good to, to essentially challenge the assertion that well, nothing that I do is having any impact on customers, that's, it may look that way. But let's say that, you know, in the case of a manufacturer, you're producing a particular component. Might be the airbag might be, you know, the ABS system. And they like braking system might be something else. And there's almost always a sort of a customer experience aspect to that, it may simply be the reliability of the part or the, you know, could could be something related to the affordability of the part. So I think that yes, it's usually true that you're you're serving some, in a sense, internal customer, but there's almost always an aspect of the thing that you're building that affects the end customer. And the more that you can think about what you can do to improve the experience of the customer. I think it helps you focus your objectives in a way that just looking at yourself as a cog in the machine doesn't.

Patricia Kong: 33:28

On that note, before we get into a longer conversation with customers and stakeholders, I wanted to thank you both for this discussion, obviously. Thank you, for the listeners for listening. Where is current Lawrence? Where's somewhere that people can check out some of the next things? Or some of the things that you're you're working on what's coming up next for you? Why don't we start with you, Kurt?

Kurt Bittner: 33:54

Well, this, you know, because we've been working on it together. Patricia and I and two other authors have a book on EBM coming out to look around for a book on EBM coming up from Pearson As part of the scrum.org series, and we hope that you will get a lot out of them.

Laurens Bonnema: 34:12

I'm gonna see if I can pre order that book, right after

Patricia Kong: 34:17

Laura's info you I know you're doing so much work.

Laurens Bonnema: 34:21

Yeah, currently, I'm focused a lot on Obeah for Lashley. So it's weird. It's just happened. All of a sudden, everything that I do combined into, let's do more with Obeah. So there's the stuff with metrics and EBM and OKRs. And how to tie all that together with graphic facilitation. For me, tying that together into obey rooms turned out to be the ideal thing. So that's what I'm doing a lot nowadays. And I'm having a ton of fun there. Plus, organizations do get more effective if they see their information, either on a wall or in the digital system. That doesn't really matter. But as long As I can see it, for some reason, it makes it easier to act on it. So it's better to have it nicely visualized and easy to focus on, instead of just in an Excel sheet or as a metric somewhere. And that's the thing that we tend to overlook in Lean In general, but also in, in stuff like this, that the way you visualize this information is very important, or at least I hope it is because that's what I'm doing nowadays.

Patricia Kong: 35:30

Oh, that's fascinating. I wanted to chat about that, that more especially when people started to get into the conversation about dashboards and stuff. But anyways, thank you so much, both and thank you everyone for listening today.