Lindsay Velecina: 0:00

Music. Welcome to the scrum.org community podcast, a podcast from the home of Scrum. In this podcast, we feature professional scrum trainers and other scrum practitioners sharing their stories and experiences to help learn from the experience of others. We hope you enjoy this episode.

Dave West: 0:20

Hello and welcome to the scrum.org community podcast. I'm your host, Dave. We're CEO here@scrum.org today's podcast is focused on how scrum.org used discovery in the development of our discovery and validation class. So I guess you can describe this as a podcast about drinking our own champagne or certainly learning about the thing we're learning about to build the thing we're learning about. Oh my gosh, It's turtles all the way down. Anyway, luckily I don't have to talk about it, because I don't know about you, but I got a little confused there. I'm very fortunate to have two people join me on the podcast, one person from the development team. Paul kaijan, welcome to the podcast, Paul, thank you. And I'm also really it's really unusual for this. We've got one of our stakeholders, lavanish Gautam, who was very active stakeholder, helped us test hypothesis, helped us actually talk to his customers. He's a professional scrum trainer based in in the UK, and it was very useful. So we brought lavenesh in with Paul to talk about this topic. Welcome to the podcast. Lavanesh,

Lavaneesh Gautam: 1:32

thank you, Dave. Thank you.

Dave West: 1:35

Yeah. This is, this is a kind of a bit of a mind blowing topic because we used the ideas in the class to build the class and ultimately deliver the class. So before we jump into the details of using discovery and and how that all applied, I think, Paul, it'd be really good for you to set the scene talk a little about what the class is about and who is it for.

Paul Kuijten: 2:00

Yeah. So, so basically, the class is a skills course. And skills courses@scrum.org are very much about a practical application of specific skills, and mostly associated with the accountability so far. So, so we figured out more and more that, hey, if you're a product owner and you need to be maximizing the value the team delivers, it's very important to be able to assess whether things are actually valid valuable, not just when after you build it and deliver it, but there's a ton of stuff you could do before you build and deliver it to increase the odds that you're on the right path. So, so we thought that's, that's a very important topic, and it's more attention within product ownership. So that's, that's how this one day skills class came about, which is all about, how can you practically integrate that into your daily work when you're developing products?

Dave West: 3:06

And from from I've actually been fortunate to spend some time with the class, what I'm seeing is that it's a class that's very much aimed at disability, dispelling many of the myths associated with discovery in particularly Scrum and Agile practices, actually, you know, Discovery isn't done before Scrum, it's done during Scrum and and you can do it on big things, you can do it on little things. And, you know, I think that's really interesting as well. So who's it for product owners, product managers? Is that, is that, right?

Paul Kuijten: 3:41

Yeah, so, so it's for, it's for product people, actually, yeah. So, product owners, product managers, people responsible for a product that's, that's really what it's targeted at, and this whole discovery and validation thing, we sort of, we sort of left it way too much to UX designers in a team, and I think it's way bigger than that. It's not just about AB testing user experience. So in a sense, we're taking a much broader approach to discovery and validation, which extends all the way from doing a survey to discover whether something is a good idea or not, all the way to delivering down product and collecting data based on that. That's That's

Dave West: 4:22

awesome. Thanks, Paul. So let me know you when we reached out, when the development team reached out to our community, professional scrum trainers and their customers, and said, Hey, we're building this. We think this is an idea. This is a high the ultimate hypothesis. You started getting involved quite early on the process. Talk about how, as a stakeholder, discovery and validation of this class became really valuable and useful. What did you do?

Lavaneesh Gautam: 4:49

So obviously, Dave and this idea came up. We came to know about this one day class, and which I really think that this is one area where. We haven't got many courses, my other providers as well. And this is one area where many product owners also, I think, lack as well, because they are fed product backlog items that, yeah, build this, build that, build this, build that. But what is the origination of those ideas are from? Where this data is coming. Why behind those product backlog items that I think was missing in many cases, and while working with many clients, while delivering the classes, we are getting this, validating this assumption as well, that somewhere this skill is lacking. So I really got excited in the early phase itself, and when I first run my three beta classes, I tried it with the three different kind of private clients. One was in the media and entertainment. Another was in the fashion industry. Another was in so manufacturing in itself. And the feedback which I got from all those three beta classes was great. They were and this is where I would like to add one more thing into the target audience for this class is not just the product owner or product manager. I would also add the product leaders as well. I have had few product leaders in the class who were the head of digital services and the product director, and they have given the very good feedback about this class itself. So I would say, because they are, I think the torch bearer in their organization, when they can see that, yes, these kind of skills can be helpful for the product owners, product managers and all the product people, I think they are the really good advocate for this particular thing. Yeah. So,

Dave West: 6:45

I mean, yeah. I think one of the biggest realizations I have as a kind of pseudo, I guess, leader leader, is that many of the assumptions that I'm forcing onto my team to deliver aren't necessarily as validated as I would have, as I would expect and and I think that those surprises, if we leave, you know, particularly if you leave those surprises later, can be very expensive and costly. And I, you know, I've, I've got some of those scars to prove it. So, anyway, so, but lavanish, you didn't, you know, beta class was definitely when we were talking about market fit, but solution fit, you know, you got involved in solution fit and and worked with the development team on that talk me for a little bit about that sort of taking the ideas and testing it to see if people would come. What? What ways did you validate that solution. Fair thing,

Lavaneesh Gautam: 7:42

the solution fit day was coming to our data sheet where we didn't have, I would say, the exact product built, but we had a data sheet that what do we want to talk about? So generally, in our data sheet at the scrum.org we talk about our learning objectives, the key crux of the classes in and itself. And when I started reaching out to some of the private clients, and started talking about it, guys, this is a one day class. This is something that we want to talk and teach. Would you be interested? And that gave us a very good confidence, and our assumption got validated that, yes, this is something might be really, really useful for many different kinds of client,

Paul Kuijten: 8:20

I think, I think in the, in the in the early days of solution fit in development of this, course, we were actually lavanish was more involved as a, as a customer, so to say, because so, so in the early stages we were, we were engaging with trainers a lot and trying to validate some of our assumptions as well. So, so, so it's not necessarily. It was lavanish testing out with the customers that we were doing that as well, but we were doing that with just the development team, yeah, but we were, we were viewing lavanise as a customer that we needed to run some experiments on as well. So we engaged with you on on face to faces, for instance, to collect data. Yeah, yes.

Lavaneesh Gautam: 9:01

And that's why I think I really liked about this particular way of delivery as well. Because there were loss of testing was happening. Some testing was happening at the by the development team. Some testing was happening by the PSTs with their clients as well. So this is where I would say this was a very good data guided approach that we took here, I would say,

Dave West: 9:23

and where were we wrong? You know, where did we actually make any sort of pivots or changes to the direction of the class because of that feedback? I think you know that could be quite a powerful message. Paul, do you have any examples where you

Paul Kuijten: 9:44

Yeah, there's, I think there's many examples, and some of them are related to course content, like in the early beginnings, you know, we got the general idea of what the course should be about. We got that right. But there was a lot of. Changes happening in the course where initially, based on on customer feedback. So, so, so that that's, that's one part of it. Another thing that comes to mind is, at a certain point, we had a hypothesis of PTN, can sell this, can sell this, this course. So, and there's many interesting aspects to that experiment, because it turned out, yes, the PTM engaged with us trying to sell the course. But it turns out, if you have a normally released course, it's a very different story to sell it than publish the course. So yeah, so that's but yeah, but that was whether that was like, something that we were wrong on, yeah, we were wrong on that, but it cost us the course correct during the experiment, which was interesting as well, because, because you even your experiments are loaded with some assumptions and and you still need course correct sometimes and all that stuff.

Dave West: 10:57

No, yeah. And in that particular example, Paul, we ended up adding to the website a beta program, which we'd never had, though we've always historically tested our products with beta what we've never done is explicitly, publicly on the website with marketing, scrumorg, marketing engine behind it. We've never done that before. And that was that that was interesting, particularly as we, as we continually tested the idea of solution fit and market fit, these two things that gave us particularly market fit, that gave us data for the first time, of hey, are people actually clicking on this? Are people interested in this, what kind of people? And what didn't we test? We didn't test everything, right? There was only certain things that we actually tested during this, this development. What didn't we test?

Paul Kuijten: 11:54

What didn't we test? I think all of the the things that are technology related. So, so we're, we're very well versed in courseware development. So, so there's a, there's a ton of stuff that we only have to try on. We know what format to use, we know where to put the training guidance, and there's a million things like that. So, so, so, so with regards to the structure of the solution, we didn't really have to test a lot. Other than that, I think, I think most bits tried to run some tests, at least,

Dave West: 12:27

yeah, so and also so Lebanese, from the sort of more outside perspective, what as the product developed through its sort of evolution and pivoted and made changes. Did the obviously, some of your customers were brought along this journey as well. How did you sort of manage their expectations? Or did you, or did you say, Yeah, this is, you know, for instance, there's no assessment available during the process. What, what did they you know, it was, I don't want to say it was a partially completed Solution that you got. But like, we didn't have all of the content on the website that normally we have. We didn't have the assessment on the website that we normally have. We didn't have an open that you can test your ideas before you take we didn't have any of that stuff. But your How did you manage your state, your customers? With with that, I

Lavaneesh Gautam: 13:21

think there are a couple of factors in that, Dave, so one is obviously the brand value of scrum.org that scrum.org has been able to deliver some consistent, very good, high quality courses. So whenever something is coming from then, even as a beta, people have got high confidence in that, okay? And then obviously, whenever we go reaching out to the clients, and they obviously ask this question, okay, what about the assessment? And we had to be upfront with them that we do not know at this moment. I'm talking about the very early beta phases as well, when you're not sure that whether, course, is going to live or not. We had to go upfront that guys, we think that this is very good. But again, you're not sure whether you are going to get assessment or not. So that's why what we are want is please join, see the course, and feel how does it feel like? Does it fit for your company? Does it fit for your audience and your product, people within your company, or not, an assessment. If it go live, guys, it is going to be cherry on the cake. So at least you will get the learning. So that's the message I had to pass it on to them, that guys, you are getting the message learning early. Why not become an early adopter of this learning as well? So so I had to give them a little bit of guys, you are the early adopter of this training, the first people who can attend this training and experience it. And if we go lucky, you will get the assessment opportunity as well, which I think we are going as well. Yes, well, I

Dave West: 14:55

hope so, working very diligently at the moment on trying to finish. The the last set of questions after feedback, which, of course, we also use discovery and validation with. We've now got beta customers that we could give early assessments to get feedback on questions. We obviously have the trainer community as well. So that's another example of testing that we've done, or validation, we should say over and over again. So it's going live. You may be listening to this podcast, and it's already gone live, because it's going live on September the 12th. How did we've been for a strong beta phase? Paul, how did we decide to, you know, move up the truth curve. When, when did we decide to remove, you know, or did we, did we set a date? What was the reality then?

Paul Kuijten: 15:50

I think it's, I think it's really, it's, that's an interesting subject, and you're involved in that too. Dave, I had some, I had some learnings in that decision. So it was really interesting, because we were we ran some experiments, we collected data, we were confident, but us as developers, we weren't really ready to make that call yet. And at certain point, because you, as one of our big stakeholders, we engaged with you a lot, and we were talking about this, and you said, Okay, I know enough willing to take this risk. Let's Let's go. So, so that was really interesting for me, because, you know it was, my first reaction was, oh, no, you know it's, it's gonna ignore the he's gonna ignore that. We need more data. But then my second reaction was really interesting, is which is really one of scale of things, because we're sitting down there developing that course, running the experiments on the customers, running experiments with the courseware, with the PSDs. And then the scale, you know, is different, and if you're sort of above that, looking at it and looking at the data, you know, because we were really happy that we could bring some, bring you some data every now and then, it's just another, another reference. You have, another reference, you know. So, so, so what's, what's, what's perfectly good amount of data to be able to risk, to take the risk for you, it would be very different from mine. And that's, I think that's just Yeah, it is a perfect example of the exact thing repeating in the course, yeah. And

Dave West: 17:30

I think, from my perspective, that at a certain point it was clear that we'd got solution fit is very clear. You know, the conversations with the customers, and thanks aminish for providing a lot of beta customers and the feedback that we got excellent. And even myself, I went for it, and I have to say that it has changed my view. I mean, look, we've now got betas on the website. We've now got we're now more willing@scrum.org to put things up that we don't necessarily know completely enough about that we would historically have put up. So it's, it has fundamentally changed how we we look, but at a certain point we had to make the call. And it costs. It does cost money to launch a class it. You know, there's probably a team of it. We increase the team, you know. We add more marketing, we add more biz dev. We add more sort of operations as an IT element to it and it, you know, and it's a team, you know, of six or seven people that work for about eight weeks, seven, eight weeks, to actually bring it over the line, assessments, bringing that into the tools, etc. And so that costs. There's a cost. And we, I made that decision ultimately, because all the data that was showing that we had great solution fit the biggest concern I have, I'll be honest, is the name and how we position it with our brand. And you know the it isn't a professional scrum class. It's a product class, professional product class, and and obviously it's lineages Scrum. But you could use these ideas without doing Scrum has to be said at the end of the day, the only way you take a product like that is put it into the market and see what the feedback is, and then evangelize it accordingly. Obviously, working with the Lean UX guys, Jeff and Josh, in terms of their support for it, and their feedback and the like has been, will amplify and yeah, so I made that decision. I don't know if it's the right one, but truth curves are very subjective, aren't they? Yeah,

Lavaneesh Gautam: 19:54

yeah, we will know soon, isn't it, Dave, we will know soon. No, but it is.

Dave West: 20:01

Yeah, we will know something more that's true. That is definitely true. It is it is interesting, because the the numbers of variables, you know, we try to be data rich. And I think all and Levinas, you've both taught me, this is where we're not. We don't make decisions based on data. We make decisions informed by data. And, and I think that the the data and the the feedback that we're getting points us towards making a decision, and we've made that decision, and we're launching on September the 12th, and, and I think that's really, really exciting, but yeah, the how we position it in the market is still a question that we could spend another six months wrestling with. Yeah, first

Paul Kuijten: 20:52

time I'm first thing I'm thinking when you say that is, oh my God, for the next one, we need to run some experiments. See, we can run some experiments on that as well, but it's hard. I think it's hard because it easily gets very confusing for your customers really soon. Yeah,

Dave West: 21:08

and that yes, and that's always the concern. The other thing is, you know, we see a need. And I mean, as you mentioned this at the start, we see the need for professional product ownership, professional product agile product management, whatever you want to call it, to be improved in every organization. And the problem we have is that many product owners, particularly in safe implementations, have been marginalized, being managing backlogs for teams to ensure efficient flow, doing, you know, refinement and really just help it, making sure their teams are never not busy. And this, this, this shift that we see organizations need is something much bigger than that. So we're sort of navigating the change, you know, the product operating model, or what we call the Agile product operating model, is part of that change, or sort of like the, I don't know, the the theme or the or the or the brand for that change, however, the market might not be there. So the question you always have, and this is Paul, I think you're alluding to this, alluding to this a little bit, is, are you may, you know, the data might say the market's not there yet, but to get the market there, you need the class, and you need all the other stuff that comes with the class, and you need the momentum of other things that will Take time. So, yeah, and that's what we've done.

Lavaneesh Gautam: 22:44

And I think Dave, we need more and more product leaders on board with this as well. I think they need to get on board, and they need to see that is, don't make decision based on because somebody is hippo, highest paid person opinion, or somebody who has got the highest job title, maybe gather the data, run the evidence, test your assumptions that way, I think we will be probably building the better products, products that customers love, and maybe also Bring some business to the organizations as

Dave West: 23:21

well. Yeah, and also doesn't think as motivating as delivering things that people need, and nothing as unmotivating as delivering things that nobody ever uses. I've built a lot of software that has never been used, unfortunately, but it was beautifully done, and I think that that is a huge, a huge change that the industry needs. And I'm not saying this is put in the empiricism in Scrum. You know, it's just one element of empiricism in Scrum, but it definitely made me take a step back as to how scrum.org, engages with the community and our partners and our customers, our learners, with respect to discovery and validation of learning. I mean, I've still got a lot to learn, unfortunately, but we are. We're making

Lavaneesh Gautam: 24:19

learning must learning must never stop. Dave, isn't it that that's what we the key theme of this course as well continuous learning. Learning must not stop.

Dave West: 24:29

And assumption, questioning assumptions, has to be the DNA of a good scrum team, without being annoying and without being sort of like, just shut up, just do it, you know, sort of like, and the best way you do that is in the context of a persona and a customer, right? And

Paul Kuijten: 24:51

I think, yeah, I think that the best way to not be annoying is to just, just not question the assumptions. That's the assumptions at the lowest. Possible cost. You know, that's the, that's the best way. And then you could bring some data, and every product leader is going to be happy that has more data to base his decision on her decisions. So I think it's it. It's just it makes sense across the

Lavaneesh Gautam: 25:20

board. No, actually, there is a, there is a model module in the course which talks about data doesn't judge,

Dave West: 25:29

yeah, yeah, yeah. So I must admit, I have felt that data is judging me many times the my poor decisions and my poor insights, but, but that was, that was awesome. So, you know, we've delivered the class on the September the 12th, um, hopefully you're all listening to this after September 12. So you can obviously take advantage of, is there any final words this experience, any sort of like, final words about this experience that we can share of our listeners. I don't know, Paul, if you want to start and yeah. So

Paul Kuijten: 26:07

for me, that whole developing the course, using the IDs in that course, it made me learn so much about how you could do it even better. So it's like, there's, there's, there's so much more to be had. I think we put a really good first thing. How can you go about it? But there are so many other aspects that you could have a look at and become better at, even within that discovery and validation area. So that's that's that that would be my main takeaway. And I, I felt it was a really good way to together develop something because you're not exchanging opinions, you're together trying to figure out what would work.

Dave West: 26:49

Lavinesh, what about your experience?

Lavaneesh Gautam: 26:52

My experience? Dave around, this is about the art of possible. Many people, many feel that, okay, experiments are very, very costly things, so that's why they, I think they, they afraid of this world as well. When we say, Hey, mate, I want to do an experiment. Nobody. We haven't got time and money to run an experiment, but the one of the biggest art of possible and aha moment on this class is that we are not just talking about the certain tools and techniques in a one day class itself. We are identifying what to test. We are designing an experiment. We are building an experiment, and we are running the experiment in the class, in the one day class, and that at the end of it, when people they see it, that, Oh, it's a very aha moment. And I would consider the light bulb moment for people that, yes, it is possible, it is not as difficult as many people think. Sometimes,

Dave West: 27:48

yeah, I think, I mean, that's great messages. I think that one, that learning is continuous, and that we're always challenging, our understanding of our users, the market, our ability to deliver, our ability to gain insights, what data we should be collecting, etc. I think that the the idea that things don't have to be super experiment, super expensive when you experiment, and the the word experiment is something that we need to be more willing to use when, as many of us are engineers, the idea of experimenting sounds like an affirmative to our practice. You know, it's the opposite of good engineering, right? You know the result you're not experimenting. But the reality is, in the digital age, the majority of the stuff that you're delivering will only be you. You know how it actually sits on the usage. You know you're not being it's not robots that are picking it up, or even, actually even it's robots. Your materials will vary and everything. So you have to test things, see it, Learn, Build incrementally, empirically. Is at the heart of it all. Hey. Thank you so much, gentlemen, for spending these half an hour or so with us today, with our listeners, sharing this. This is this has been very open and honest. Thank you for spending the time today and sharing a little bit about our experiences with using discovery and validation on the discovery and validation clan, and thank you for being so open and honest with our listeners. I think that is something that I certainly appreciate, and I'm sure they will as well.

Paul Kuijten: 29:33

Yeah, thank you for having me, Dave. It's been interesting.

Lavaneesh Gautam: 29:37

It is. It was, it was great fun to talk about some real stuff,

Dave West: 29:43

some real stuff. So people often ask, does scrum.org eat its own drink its own champagne, eat its own dog food? The answer is yes, as you heard today on today's scrum.org community podcast, and thank you for listening today. Everybody. If you liked what you heard, please subscribe, share with your friends, and, of course, come back and listen to some more. I'm very lucky that I get to talk to a variety of guests talking about everything in the areas of professional Scrum, product thinking, and, of course, agile. Thank you, everybody and Scrum,