Assessing the AIOps Market: Brian Steele, KedronUK

Welcome Brian.

It's great to have a chance to talk to you.

We've met a few times.

This time, I get to actually grill you on some really interesting topics.

So, really looking forward to it.

I'd like to welcome Brian Steele from Kedron UK.

Brian is a technical development lead at Kedron with an extensive background in engineering and IT management, which is perfect for the topic we're going to talk about today, which is AIOps, which is, in very short, the use of AI to simplify IT operations management.

And I want to get into that a bit.

I want to ask you also about Kedron UK, the company that you work for, is focused and specialised on the topic of connecting and visualising IT data.

So, what does Kedron do? What is the service that you guys offer?

We've been in business around about 20 years now.

And we've got a portfolio of 14 or 17 different products.

And customers come to us with monitoring challenges, and then it's up to us to kind of find the best solutions or mix of solutions to fill their needs.

So, we're a solution provider with lots of experience, and also consultancy as well.

So, it's kind of a mixture of those two things.

Okay.

So, it's basically, you know, there's some advisory, some consultancy services that you guys do.

And of course, it's, you know, ever more and more critical to have somebody who can give you a kind of unbiased look at different options in the market, you know, as opposed to talking to a salesperson.

Exactly.

We haven't got one particular product to push.

It's solutions we're looking for.

We're looking to address problems.

Okay, great.

And today, what I want to talk to you about is the topic of something new, at least to me, called AIOps.

So, using artificial intelligence in the IT operations management sphere.

There's a whole lot of buzz words out there especially with the word AI in it. But tell me a little bit about this one.

This one sounds like it's getting right to the nitty-gritty of managing the IT data set.

Can you describe a little bit what AIOps is all about?

Oh, well, yeah, you're right.

There's all kind of the different opinions on it.

But for us and our customers, what we're looking to do is to bring together all of the alerts and the critical metrics across all the different tools they got in their monitoring landscape.

And then, to help them make sense of it all.

The problem used to be years ago that customers will have gaps in their monitoring, and we would help them plug that.

These days, they've got everything covered.

Yeah.

The challenge now is quite a serious one, is that there's just too much information.

And they definitely need AI machine learning to help them make sense of it all.

Yeah, there's quite a lot.

And I guess what they've got happening now is tool sprawl you know, where you end up having so many different tools.

Any one of them goes out of date or has a little bug, and then you have a whole new set of problems.

So, it sounds like this is a way to take that all as a whole and integrate, consolidate this activity.

Yeah, it's definitely the next phase in the evolution of this space. Yeah, absolutely.

So, you actually conducted quite a lengthy analysis, quite a lengthy vendor analysis specifically in the AIOps space.

Was this something you did on request of a customer or was this part of the normal research that you do like to take a topic like AIOps and scan the market?

Right, okay.

Well, we had a solution, which was a "manager-of- managers" the precursor to AIOps, and it was continually letting us down.

We were struggling to meet the customer's...ever demanding...their demands kept increasing all the time for their requirements.

They're getting more and more sophisticated.

So, it became pretty clear after quite a short time that we needed something different.

And it was around about that time that AIOps started to make itself known.

So, we took the choice that there's just such a kind of shift in approach, and it's quite a different solution. But definitely, the kind of thing that we were looking for, it came at the right time that we thought, 'Well, we're going to dig deep and find the right product for this collection of kind of problems that we needed to solve.' We kind of built up quite a lot of baggage that we needed to get addressed.

Yeah.

It's amazing that some of these like new cutting-edge technologies that, sometimes, even the problem they're solving is something that didn't exist five years ago, you know.

Well, yeah.

As I've said, yeah, that's definitely true, you know.

Yeah.

And how did you look for vendors? And did you look at established companies or did you go out to user groups? Say, how did you actually collect your initial vendors?

It was a mixture of mainly three things I seem to remember.

So, some of them, we were aware of straightaway.

So, they were quite easy.

We've put them into the list.

And then of course, in speaking to our customers, they would always say, 'Well, what about this company and that company? We've been thinking about them.' So, we put them in the list even though we knew some of those suggestions were completely irrelevant.

But because the customers thought that it could be a solution, we felt it only fair that we threw them into the process, so that you could see the low score that we're getting to kind of demonstrate that it wasn't really kind of the solution that they thought it was going to be.

So, they got included. And then of course, we did what you have to do for due diligence, which is trawl the internet and just see who is out there raising their hands up saying, 'We can do this.'

Yeah.

So, it was a bit of those three things altogether.

And you ended up with quite a list.

I think it was some 17 or so vendors that you ended up with to start out with.

Yes, that's right.

Okay.

So, what were the first factors you looked at? Like you had a...you built a scoring process.

Like you mentioned, there were some that you put in there just to kind of rule them out basically, but do it in a transparent way.

What was the method you used at this stage?

Well, we built a list of...I think it was around about 70-ish or so questions.

And they came about again from questions that we knew we wanted answering, the requirements that we had.

And then of course, our customers over the years, we knew what they were asking for.

Some of them were quite common.

Some of them were unique or interesting and sparked our interest.

So, they went into the list.

And then, as we looked at what the features the 17 said they could do, there might be one or two vendors that were doing something that we'd never thought of, and was just like an interesting feature.

So, that would go in the list because we knew they were going to meet it because it was their feature.

But we felt it was fair to let others have a crack at that because they may be able to make that product, do that thing with a little bit of configuration. So, again, it was a mixture of things that kind of built up this list of requirements.

Yeah, it's amazing, the things you learn.

You go in with a preconceived notion like these will probably be good, these won't, and then you get surprised in the process, right?

Yeah, yeah.

Were there like any immediate or urgent needs you needed to solve or was this sort of like a future scenario type of process? So, you thought like maybe you might need these features later?

Yeah.

It was pretty much medium to long-term because it was such a kind of change in direction in terms of what AIOps could offer.

We, as I said before, we knew we were going to kind of dig deep.

So, we were...we knew it wasn't going to be a quick process.

And we just had to, you know, wait for the final results.

And then when we got the correct answer, we would have something to go back to our customers and say, 'Sorry about the big wait, but, you know, here it is.

This is the answer.'

Yeah, okay.

And then...so, part of this initial survey, you know, you ended up with this list of vendors.

You developed a subjective scoring system.

And then you also came in contact with quite a few potential customers in the market.

How did you come across them, you know, these hundreds of customers that gave you feedback?

Well, I mean, as I say, we've been in business around about 20 years because of the 14 or 17, I can't remember, different vendors that we've got.

And the consultancy that we do, quite naturally, we just come across many, many customers and their requirements all the time.

Yeah.

And very often, like I say, there has been a drift from the, 'We got gaps in our monitoring.

We can't see everything.

There's, you know, some blind spots in what we're doing, and that's hindering us in our root cause analysis.' But it kind of drifted away from that as people were plugging those gaps with our help.

And sometimes, it came to us, and this problem's already solved.

But it was drifting away from that kind of need more towards the...well, we've got all the information we need.

We just need something to help us make sense of it all.

There's just too much.

And so, that's how that came about.

Okay.

So, you had...basically on that basis, you got...well, I guess that's another source of surprises, right? You found out some things that you hadn't thought to look for.

You developed your ultimate list.

Did you actually have to go and meet all of these vendors like, you know, take hour-call meetings or whatever to do the assessment?

Yeah, absolutely.

I mean, that's why it took so long.

It was a very methodical process.

You couldn't really miss any step out.

Yeah.

To have confidence in the final answer, we have to make sure that everyone was given a fair crack of the whip, and we didn't kind of shortcut anything.

Created some interesting scenarios at work.

But I ploughed through, and we got there in the end.

But yes, there was many, many hours of interviews, tests, and reading documentation, and questioning and compiling results, and going through the statistics.

Yeah.

Yeah.

See, that's already a valuable service I can imagine for that, that you went and talked to all those salespeople, so you customer wouldn't have to.

No.

You took one for the team.

Absolutely.

Yeah.

So, what were some of the things that, if you can recall, that were initially...that led you to eliminate vendors at this stage? Like what were the first kind of obvious factors that allowed you to shorten the list?

Yeah.

Well, once we got the 70 or so questions, in fact, there's one step before that, I probably haven't mentioned yet, but we also provided...we also decided, sorry, on a weighting factor.

So, each of the questions got weighted as being 10% important or 100%.

It was a feature that was definitely required and would mark you down severely if you didn't have it.

Yeah.

So, that was needed.

So, we kind of built that waiting matrix as it were.

Yeah.

So, initially, we went through all of those questions and got them answered and are weighted.

And it was kind of pretty obvious that through the first wave of that approach, some dropped right to the bottom.

Yeah.

It was pretty clear that they were never going to catch up no matter what.

So, there was a clear cut-off point to remove them from going into the next phase.

Yeah, okay.

So, that's probably I guess in a lot of technology areas, almost the...what is it, the green checkmark and red X list, right, where you finally take that down? So, it was quite a bit.

I think if I got my notes right, you were kind of about half at that point, right, half of the vendors?

Yeah, that's about right, maybe just under.

I seem to remember something like that.

Okay.

So then, in the next stage, you went with a more conversational approach.

So, you know, you've kind of done the rigid, you know, list, the disciplined approach.

Now, you're contacting the vendors that were remaining and went for a more conversational approach.

What was that like and what did you learn from that stage?

It was interesting because there's a couple of things that came out of that.

Some of the features that vendors were saying that they could do became clear that they were kind of shoehorning their product in such a way or configuring it in a very uncomfortable way to say that they could provide that feature.

So, that became clear.

And of course, they got knocked down because we wanted features out of the box.

We didn't want something that was contrived or unnatural or uncomfortable.

Right.

Because I can see the future version of that product.

That weird configuration would somehow break or go wrong.

So, that wasn't going to work.

And then also in having the face-to-face conversations with these people, we were starting to get to know them.

And that was also a big part of our evaluation process.

Yeah, yeah.

Ultimately, you have to trust the vendor, right? Or you have to trust anyone you're doing business with.

Absolutely.

It's our reputation and our customer satisfaction that's on the line.

So, yeah, our feeling towards them was an important part of the process.

Yeah, yeah.

That's the hard thing to write down in numbers, you know.

That comes from experience.

I also like the focus on out of the box.

I know that you had worked for lots of vendors, big and small.

And there's...and I've been on the customer side, and you have that infamous Q5 problem.

'It's on the roadmap, it will be available in Q5!' It's constantly Q5.

So, that's great.

And what, you know, at this stage, O guess we didn't go into a lot of details about AIOps itself.

But, you know, what were the kind of features that were really key at this point that, you know, would matter the most when you decided to keep or eliminate a vendor?

Well, let me think.

So, we wanted a big breadth of connectors.

So, obviously, one of the big things is the ability for the product to connect to a very broad range of trusted data sources for collecting alerts, metric information.

So, yeah, we definitely wanted something that had a broad range of existing connectors.

We then wanted to see some of the big names in there.

So, we built a list I think of about 20 or 30, I can't remember, connectors that we knew generally our customers were using in terms of all the tools.

So, we wanted to see those in the list.

But then again, on top of that, we wanted to...we said to the vendors, 'Well, if we came across a connector that you haven't got for some kind of weird and wonderful trusted data source, what would be your approach?' And some would, 'Well, we can't do that.

It'll...you know, put a request in, and it might happen.' Or some would be, 'Yeah, we can do that, but we'll charge you £10,000.' And others were, 'Well, thank you very much for highlighting a gap in our product.

We'll do that free of charge.' And, you know, that's the...kind of we were looking for that.

But that was a nice answer.

Yeah.

And there were many other questions like that.

We needed systems to automatically discover the topology of the landscape they were looking after.

We didn't...some products required you to configure the topology.

And that's a nightmare.

All the solutions would integrate with APM tools.

And they would rip the topology that the APM tools discovered and use that to augment its own understanding.

And so, there are all kinds of questions around the nitty-gritty things that you'd really want to see.

Different methods.

I guess if you're using AIOps, you specifically don't want to do all this manual work.

That's (laughter).

No, no.

No, there's no point having it manually configured.

And then putting AI on top it's kind of a broken solution, yeah.

Yeah, exactly.

And since this is a new area, what were the companies themselves like? Were they a lot of like start-ups or were they established vendors, medium-sized companies?

Yeah, that's interesting.

So, I never thought about that.

There were a couple of start-ups.

And when you say start-ups, let's say two or three years.

You know, that's relatively kind of start up.

Obviously, to produce these kinds of solutions requires a lot of effort.

They're not going to do it in six months.

Right.

So, yeah, there were a couple of those.

And their attitude was fresh and that they were eager.

And that was nice.

There's a trade-off, I guess, because especially when you're in a new area, because you...the ones who probably are on top of the needs, the best are usually the youngest companies.

And of course, there's always some trade-off with that.

And the established companies will always have a solution.

But when you scratch under the covers, is it just a logo or, you know? And are they going to care about you when you call them from help, you know?

Yeah.

...about you when you're calling for help, you know?

It's interesting.

We did absolutely find that sort of more established players would sit on their laurels a little bit and think the brand would sell itself.

And you're absolutely right.

I don't want to generalise too much, but there did seem to be a kind of a tendency for they're the ones to pretend that there were certain things we could do when they couldn't, and were kind of twisting it.

They couldn't make it happen, but it was an uncomfortable solution.

Yeah.

So, there was definitely some of that going on.

Whereas refreshing, when I talked about refreshing for the start-ups, they just turned around and said, 'No, we can't do that, but what an interesting idea. Let's put it on the roadmap.' And that was a much more open and honest approach and was endearing.

Yeah.

And that is the benefit of going, you know, for the trade-off of going with a less-established company the trade-off is you can get somebody who's really on top of it and open-minded and nimble. So, you also had a lot of demos from vendors at this point.

Were those helpful? How did...what did you learn from the demo phase of it?

It was...yeah, I did have a lot of demonstrations, yeah, and there was like a two-phased demonstration.

The first one was just I'll sit back and listen, and you show me.

And you've already got my list of questions.

So, make sure you address those points when you do the demonstration.

Don't give me a vanilla demonstration, you know, hit all of the requirements.

Yeah.

And I was kind of ticking the list off as we went, making sure we didn't miss anything.

And some, I had to remind to bring it back on point.

And then they would struggle.

So, they were clearly kind of avoiding their weak areas.

And again, that helped mark them down. I don't want them to struggle.

I want them to be able to deliver what I'm asking for.

So, that again was...that aspect of the demonstration was part of the kind of scoring process that help position them in phase two.

Yeah, okay.

So, the next phase after you did your demos of course was when you did your own kind of trials, your own POCs.

You got your hands on the product.

And you had an interesting approach, which was to avoid any kind of training from the vendor.

Yeah.

Rely completely on what you could find and download and, you know, access yourself to...I guess it's to really see how the ease of use was on the product.

What was that like and what did you learn from that experience?

Yeah, I mean, that approach is something I've picked up over decades of being in IT.

I got fed up with owning products that you had to have training because it was just impossible to kind of look after the product without the training because it was so unintuitive and complex.

So, quite a long time ago, I learnt that if I couldn't get it working myself from the manuals, then something was just very wrong.

So, I used that approach here.

So, they pointed me to where the documentation was, and I would download it.

And I'd go straight to the bit that says, 'This is how you install it.' And then I will just follow those instructions and see how well it worked.

And some fared better than others.

Some were shocking.

The documentation was just terrible.

And we, again, we don't want our customers to have that experience because that will come back to us.

If we're struggling to install it when they're keen to sell to us, heavens knows what that experience is going to be like when you're on-site and, you know, you're having some challenges you didn't anticipate.

It's...we didn't want that kind of a solution.

And then there were others that just flew through, you know.

You just follow the instruction, one, two, three, and then it will just go straight by the numbers.

And within 45 minutes, you had a product that's up and running.

And then with another 10 minutes, it was connected to AWS, and it was monitoring virtual landscapes and...you get the idea.

That worked for us.

Yeah.

That's especially impressive when a young company can do that.

Because having worked in many myself, I know that sometimes, documentation is like one of the lowest priority things. You know, people are trying to get product out the door.

May not work entirely well.

So, you know, that you could actually identify vendors that were able to provide that seamlessly was quite a big win I think.

Yes.

So, how much time had passed by this point, by the time that you're actually doing the testing yourself from when you began?

Yeah, that would be approaching the kind of eight, nine-month area towards the end.

Because we only ended up with about...I think it was three or four vendors that went through that kind of final stage where we were spending the time to actually install the product and evaluate it one-on-one

Okay.

One of the aspects I think at this later stage, one of the things you looked at was how well the vendors fulfil their road map promises earlier in the process, which I guess is an advantage of such a long process that there's enough time to go back and see if they walk the walk, you know, if they meant what they said.

How did that turn out?

Yeah, that was genuinely really gratifying because, yes, you're right, it took as long as it took.

We didn't have a timeline or an expectation for it.

It was finished when it was finished.

But, yes, because it did end up taking kind of nine months, by the time we went back to some of those vendors we'd already spoken to, and in the past, they'd said, 'Oh well, we can't meet this requirement and that requirement, but it is on the road map for Q2.' We didn't realise that when we went back to them.

Well, Q2 has passed.

Let's see, you know, let's see if you're as good as your word.

Yeah.

And, yeah, the final company that we selected was one of those that surprised us very nicely in that not only had they delivered everything that they said they were going to.

So, that was a great deal of confidence that they were in control of that process, and they weren't kind of fluffing it.

They were delivering real features.

What they said, 'Oh, and by the way, we've done this, and we've done this.' So, there are like always Easter eggs in there as well, which was great.

Yeah.

That is great.

You don't get those kinds of positive surprises often in technology.

So....

No.

That is really great.

And at any point, did you ever go back and look at any other vendors you had eliminated earlier in the process after some time had passed?

No.

And that relates to your previous question because we'd seen that the gap between the ones that were right...the ones or the one that was right at the top was accelerating as we could see that they were definitely delivering then their road map and over-delivering. We could see the gap just getting bigger and bigger between the others.

And we thought, yeah, these are the guys we want to be involved with.

They're at the front and they're just accelerating.

So, it was a cursory look back over our shoulders.

It became pretty clear that there was little point wasting any more time on the others.

It just would be a waste of time.

They were getting left behind.

Yeah.

And I guess at this point you did come down to a few really good candidates.

And so then, it was really about getting down to that level.

So, you found at least one and maybe a couple of other candidates that did well.

And so, I guess at this stage, you start to look at pricing and licensing.

How did this impact your decision at this stage?

Well, pricing not purposefully.

And I told them all upfront I'm not interested at this stage.

The commercials come after we've selected the one that's the most relevant solution for the problems that we've got.

If it ended up being the most expensive one, then it ended up being the most expensive one.

But we were just trying to find the right solution at this stage.

However, licensing was important.

I did evaluate the licensing.

Because, again, some solutions have very complex licensing that you'd need a degree in spreadsheet operations.

You're going to work out exactly what you need to order.

And that again was no good.

We needed something clear and simple.

So, that licensing and the way that works was part of the process.

And again, that the...the two that were right at the top fulfilled that brief.

But to allude to that when we made the final selection and started talking about pricing, I held my breath and got a very nice surprise.

That was good.

Yeah, that was great.

Yeah.

So, in the end, you made a selection. Oh, an Important question I didn't ask was how long ago was that? When did you make that selection?

Yeah, that's about 12 months ago, that.

Twelve months ago? A good year.

So, are you satisfied? You're still working with this vendor? You've had now a good bit of time.

How's that going?

Yeah, extremely well, very happy.

I shouldn't be surprised really because as I say, we put a lot of work into it, and we got exactly what we expected.

So...but, yeah, it's gratifying to see.

It's definitely going in the right direction.

That's great.

So, the long effort and long time put into this process was worth it.

Because now, it's been 12 months and it was nine months' worth of work.

So, you've now paid off your time, effort, you know, you got all that.

Yeah, absolutely.

So, now, looking back at it, if you were to start over again on this process, what would you do differently?

I have looked back over it, and I don't think I'd do anything differently.

I'm not pretending that it's a perfect process.

I think it was as close as perfect could be for what we wanted to do.

And just having a look at it and just thinking if I was to apply this approach to some other kind of similar problems to look for, you know, another product maybe in a different space, I think the framework that we've developed would still work in a number of scenarios.

So, I think, yeah...I don't think I'd change anything, really.

Okay.

Have you used this type of approach with other technologies or other assessments?

In previous roles that I've had over the years.

But at Kedron UK, no, because this is...it's a lot of effort.

And this...the task in hand required...there was so much at stake it required the correct answer. And that we were having problems with the manager of managers that we had, and it just wasn't working out.

We just needed to press the reset button and start again.

So, on this particular occasion, this set of challenges required us to undertake this complex and time- consuming work.

Yeah, makes sense.

Even that name, 'manager of managers', I already get a picture of what happened, you know, like a problem just snowballing over time, just add another patch, another patch. And then pretty soon, you have this big thing that you...the only way to really address it is to start over again fresh.

And that sounds (overlapping conversation).

Yeah.

But the solution in a changing landscape and changing technologies, not just changing configurations of landscape, it became almost as big a task to manage the 'manager of managers'. But then, it gets...and the time you potentially saved in what it provided for you, it just wasn't working out.

The Russian doll of IT operation management.

(Laughter)

That's a good analogy.

So, that's great.

That's a really fascinating process.

I think the final question I would have for you is if I was someone who was at the beginning of this process, you know, your typical customer, what would you recommend I do to start off on this journey for the...searching for an AIOps solution?

You got to love documentation.

You got to like speaking to people. You've got to be really good at Excel spreadsheets and producing dashboards.

You just got to be tenacious and stick with it and don't be put off.

And yeah, you'll get there in the end.

It's a lot of work, but it definitely pays off.

And I guess the other option is they can get in contact with you, and you can help them in (overlapping conversation).

(Overlapping conversation) Of course, yeah.

Yeah, exactly.

That was the whole point.

Customers were banging on our door and, you know, asking us to solve these problems, and this is what we have to do.

So, yeah, sharing information, that's exactly why we did it.

Oh, this is great.

No, I really appreciate this.

I mean, obviously, some of this is like, you know, a very normal structured vendor assessment. But I think because it's in such a new area and it has such a high impact, you know, how you're managing your entire operation, this has been an especially...I think this has been especially useful process to kind of go through with you and understand.

And I think what we will do at one point is do a follow-up session where we really get into the technology of AIOps.

I think that would also be really interesting. So, something that we'll be in touch again soon for hopefully.

Look forward to it.

Yeah, thanks very much, Brian.

I appreciate it.

And we'll talk soon.

Thank you.

Bye-bye.

Bye.