A fireside chat with Matt Hand

Introductions

Chris Singlemann (CS): Hey folks, thank you all for joining us here today. We've got Matt Hand from Prelude here and we're here, you know, wrapping up the end of 2024 with a bit of a retrospective on Matt's career. We're to get into a little bit about his role at Prelude as well as what he's been doing prior to joining us here at Prelude and a little bit of forecasting. What are things going to look like over the next couple of years based on his insights and expertise coming from the industry?

So we thank you all for joining us today. If you're joining us on LinkedIn or you're joining us on Twitter, thanks for being here. We will be monitoring the chat. This is a conversation. So feel free to drop in any questions as we go along and we'll stop the conversation and of course address anything that's coming up in chat. But to get started, Matt, why don't you take a minute and introduce yourself. Let us know a little bit about yourself and where you're coming from.

Matt Hand (MH): Yeah. So hi everyone. I'm Matt Hand. I've been at Prelude for a little bit over a year, coming up on a year and a half now. I'm the Director of Security Research here, so I kind of lead the team that's focused on what things should we test, how should we defend against them, what tradecraft matters, and how do we go about targeting it. Prior to that, I spent pretty much my entire life in offensive security, so little bit over a decade. I ended my career there at SpecterOps for about four years, where I doing everything from red teaming to help build the purple team service to teaching the vulnerability research course. So, I spent a large amount of time focused on the delivery of offensive services, vulnerability research, the kind of gamut on that end. Yeah, happy to be here nd talk through kind how the year went. It's pretty interesting stuff this year.

Running red team exercises at services organizations

CS: So you talked a little bit about your background largely in red teaming operations. You did some purple team build outs as well. But a lot of that was spent in services organizations. So you spent some time in SpecterOps, Rapid7, Booz Allen Hamilton. Walk me through a little bit about that. What was your day to day looking like, especially working across such a broad spectrum of organizations on similar work? Yeah, it was, you know, a lot of the predictability in delivering services is, you kind of forecast out like what a day looks like, regardless of where you work.

MH: The pressure of performance is different and different organizations. So, you know, my last job, was, you know, engagements are scoped roughly by weeks. so, you know, you got three to six weeks to go and run an operation. And like a normal week where you're like working an engagement, it's, know, you get up, check in with clients, you sync up with the team, you figure out kind of where you want to go and then different teams organize different ways.

So the teams that I worked on, you know, we'd have some pretty hard targets. So we'd split off and say like the goal for today is to go into this thing. And it's kind of on like the individual's creativity to figure out how you want to do that. There's no really right way to red team. So given the constraints that we had, we'd go focus on that. We'd spend a good amount of time kind of documenting stuff as we go. you know, a lot of the day is kind of built around like what, what goal are you trying to achieve strategically? Like how do you move the operation forward towards whatever the objective is? But then, like what's the right way to do that?

Like doing kind of a risk calculation of OPSEC and figuring out, know, we could do, you know, persistence this way, but the risk here is that, and they have this defensive stack, so it may be risky to do this, or this tool isn't working quite as well as we wanted to and the modifications would cost too much. So maybe we could take that same approach with a different tool and a lot of stuff like that. So I spent a lot of time personally weighing the pros and cons of different approaches and doing the OPSEC risk calculation based off of a background invasion. What would be the best or at least the optimal thing to do in a suboptimal situation? Got it.

So, you're thinking about how the approach despite similar work, how the approach changes depending on the outcome that the client is looking for.

CS: What consistency are you noticing across those engagements though? Like if the approach is different, how are you seeing like similarities rise across those engagements?

MH: Yeah, everything ends up being predictable just at different levels. Again, like that's one of the reasons why I kind of decided to end my tenure in road teaming was like ultimately like things just start looking exactly the same. You walk into a company and you get access. It's like, I could tell kind of like how this is going to play out in a lot of ways. There are some companies that just like exceptionally difficult targets, which those are the really, really fun ones. But the three lines are always like you get onto some system and the workstation is pretty well defended. You know, the user has, you know, they're not a domain admin. They have, you know, some EDR running their laptops up to date.

But then like you start looking at, especially like the bigger the company is, the more interesting this gets. It's like they have some piece of vulnerable software that like, you know, it's, you know, we're not out looking for CVs or whatever. It's like, you know, something trivial, like. Dotnet remoting enabled. Like we did a one customer where it was like, we got onto an endpoint and we found that bug and it's a privilege escalation. So we go from a normal user up to system. We could just kind of roll through that.

So they all have something like that in a lot of cases or you know, they, they have some weird group membership that gives them excessive privilege over something or, kind of towards my end is like the ADCS era starts taking over. And there's, you know, some level of domain wide misconfiguration or CA level misconfiguration, you know, some delegation issue that just gives you like a wild amount of access to stuff that you otherwise wouldn't have. So you target one of those and you kind of snowball your way into: "Well, now I have permissions to do whatever I want."

The more interesting engagements for me were like, they had a really sensitive set of crown jewels. You know, it's a piece of like high frequency trading infrastructure or a sensitive like HMI for manufacturing or some type of control system. And getting to those, it's kind of a predictable path of

You jump to a system, you look what's going on on that system, you check what your user has access to, you make a decision on where you want to go and what do you want to do with that information. Make a through line of like, this user is overprivileged and this system is poorly configured or is missing something or doesn't have the level of scrutiny on it that you would need it to make sure that's in good shape. It just ends up kind of being the same thing over and over in a lot of ways. Just the way that you target it is different.

Writing Evading EDR: The Definitive Guide to Defeating Endpoint Detection Systems

CS: So while you're doing a lot of this work, you are experiencing quite a bit of a lot of different organizations, a lot of different outcomes. And you're learning about the lot of the tools that you need to go off and ultimately test to make sure they're working effectively. So much so you go off and you write a book about it. So tell us a little bit about Evading EDR and kind of the driving force behind why you decided to write that book in the first place.

MH: Yeah, it's an interesting story. Like I used to have a relatively long drive that I would do a couple times a week and started listening to like podcasts and stuff like that. then, you know, when I got started in security, there were a bunch of security podcasts that all kind of fizzled out. some friends started one. And they had a guest on named Chris. And he had a really interesting perspective.

He said, I'm going to misquote this, but the general gist was: "Blue team has a huge advantage over red team when it comes to endpoint protection because the blue team has access to all the telemetry and the red team has no idea what we can see." And part of me was like, that's kind of mean. That's not all the way true. I was like, wait a second. It's mostly true. In other words, they know more than the blue team is going to know more than the red team does, no matter what.

Exactly, they can see the raw telemetry. They're the ones writing the detections. They know what the capabilities of the systems are. And we just kind of hypothesize. Like we have this, I joke with friends, it's like, how do you know that that was evasive? Did you get caught? Like, did you see the alert? Did you see the log? She was like, well, I don't know. The blue team told me that they saw me and they didn't really say anything beyond that. was like, that's anecdotal evidence. Great. But that's not empirical evidence to say like, how was that caught? Is it possible to catch it? What can you do to get around it?

I heard that and I started kind of writing a response to that, that I was originally going to just post on Twitter and be like, wow, this is actually like a really good point. Why don't we know that? Like I know in red teaming, like vendors don't want to sell things to people who are just going to go beat up their stuff. I understand that we dealt with that, that issue specifically when building labs. But, you know, I logged in the Twitter to post kind of a response.

And this is when the whole like syswhispers thing was coming around the whole direct sys calls. So I started writing blogs that I kind of, I was going to release this kind of like a series and I up writing like 10 of them and they were really long. was like, okay, sure. I it was just to be a white paper. started writing white paper got to 200 pages and I wasn't done yet. I was like, okay, this needs to be a book.

Originally I was, was going to just kind of self publish it, but writing is hard, words are hard, English is a very unique language. I was like, man, I could use some editor help. So I just kind of approached No Starch with the issue. And the original title was Inside EDR. It was a kind of a look into what's going on beyond this. But that ended up getting published in fall 2023 after two and half-ish years of work.

But ultimately, like the reason why I wanted to do it was to just kind of like democratize access this information. So there's there's an aspect of security through obscurity that like is reasonable. We know like that's not an actual security practice. But sometimes a black box being a black box is not objectively a bad thing. But it is when you need to understand like if you're putting all of your chips into the system and saying like we just did some survey that was like a vast preponderance of people think EDR is like their single most important control. like, I personally agree with that. Like that's where breaches start. That's your most high touch attack surface outside of things that are sitting on, the internet. Like this is kind of kind of where patient zero lies in a lot of cases. And if we put all of our faith into these systems and say, this is like, this is what's going to save me come the end of the day. So I'm going to run my incidents. This is what I'm going to rely to tell me what's going on.

If that system, like we don't even know how it works. We're just kind of blindly putting faith in the marketing dollars. Sorry, Chris. Like if, like if we're just trusting the marketing, people are going to tell us a hundred percent and they got a hell of a steak dinner and a good golf game. Like that's not, that's not how we actually measure efficacy. And the way that we start measuring efficacy is they're being transparent about how things actually work so that we can scrutinize them.

That's the reason why people put crypto proofs out to the public is so that people can act like mathematicians can target these things and try to break them. We want people to break stuff and to start breaking it. You have to first understand how it works so that way you can understand when you do go and test this or there is a failure, you can understand why it might have broken and what you could do in response if anything.

Understanding your EDR tool

CS: So, you made a lot of interesting points and I want to definitely come back on the testing aspect that you spoke about, but you did mention the report that we were a part of which was saying effectively two thirds of security respondents felt like their EDR was either the most important or the second most important tool and beyond everyone else, the most effective at blocking threats, which to your point is one, believable, but two, alarming that we don't necessarily know everything there is about these tools and how they work.

So, I'm curious, you're doing this for 10 years, you're an industry leader in it. What did you learn when writing the book? Like what were some things that came out to you that maybe you weren't aware when it came to how these tools actually work and function that are relevant to a red teamer.

MH: Yeah, that's a good question. The interesting things that kind of surface around this is just it is simultaneously insanely complex, like what they're doing, while also being deceptively simple. Because on the face, like you have a hundred little sensor components. Hyperbolic, but a whole bunch of little sensor components that are listening for things that are happening. They aggregate them into this agent that does some baseline processing and then ships it up to the cloud, which into a data lake or whatever. What's interesting is that a lot of these sensor components have things that are just for myriad reasons, not enabled. So you may have the capability to do something, but it's too voluminous.

You have to consider like every single event for the most part is has to be a decision has to be made on whether or not to send it up to the data. Like that could be expensive. Turn on Procmon with no filters. And you're talking about hundreds of thousands of events per second. In some cases, there's no way you can shift that up. Just the cost of doing that. So you have to make a decision about, well, what certain things am I willing to miss in favor of having better performance on, you know, X, Y, and Z technique or whatever. Like, getting name pipe telemetry is super important for a lot of people. It's how we do like peer to peer C2 linkage. It's how we do like, you know, a whole myriad other things. But like, what about ALPC? Like, well, ALPC is kind of backbone for everything. The problem is ALPC traffic is like insanely high volume.

So if you're not like really focusing on that, just like you can't collect it. Your event budget is spent and you don't have any more bandwidth to do. You have to deal with, you're deployed in a rural village in Uganda with a 40K uplink. Those people need to be protected the same way that other people do, but are you going to bomb their entire bandwidth on ALPC events or are going to hedge on file rights and things like that? So it was interesting for me to go back.

I understand and the I can do a thing and EDR has the ability to capture these certain things about it. answering that kind of inverse of like, well, why isn't it catching this other thing that might be more effective that might be more efficient, but for some reason, they're just ignoring it like learning why they may try to do it through testing was was really interesting to for me to see and kind of enlightening to like why some ways that you have ADDR are more trivial than others and why they're so effective and long lasting.

It's because there's this structural insufficiency with these products in some cases that they just can't amend.

CS: Is there anything that like, if there's one thing that folks are taking away from the book that you want to make sure that people understand? To your point, there's a lot of these devices that exist in a black box, we don't necessarily understand them. But if you're going to enable teams to know them better, what's the one thing that they're walking away with an understanding of?

MH: Honestly, like my favorite chapter in the book is the last chapter with the stupid. don't know if anyone caught this. The Binford tools example is from the old Tim Allen show. So that's the name of the fictional tool company, the left handed screwdriver. That last chapter is really kind of like how you make all this practical. A lot of the

The first chapter is more of like an introduction of like what these things are and how they are actually structured at like a meta level. The mid sections of the book are how each component very specifically works. But the last chapter is like, how do you deal with the system in aggregate? Like that's what we actually interface with when we deal with EDRs. You know, you're not just dealing with their driver and the kernel callbacks. You're not just dealing with a mini filter or you're not just dealing with ETW.

You're dealing with a bunch of different components. that have to come into play. And the biggest thing is like, especially for like red teamers, and I guess blue teamers as well is like, consider that you have this like concept of adversary capital. Like imagine like if you're a red team or you're an investor, and you have a million threat bucks or whatever, you have to invest that wisely. And you spend your threat bucks in terms of exposing yourself, you inherit this risk of anytime you do something, it's susceptible to scrutiny.

Your return on that is access. So you have to make smart investments in terms of what trade craft you're going to employ and when you're going to try to use them under what context and how specifically you may do that to maximize your returns in terms of access. Like if you can do something at like zero risk and marginal improvement to access a great investment. If you do something that's higher risk, incredibly higher reward, like, you you make your final jump, but it's a risky one. You have to weigh that decision. Having an understanding of the full system and how it works and like really deeply understanding what each thing is going to do so you can more adequately calculate risk. Like that's all ultimately what we're trying to go after is adequately understanding risk on the offense side, defense side, understanding like where the gaps are that I need to shore up in another way.

Being able to do that calculation on the fly and eventually like build it into an intuition is an incredibly powerful skill for both offense and defense because you can kind of start sussing out where the gaps may be and where you need to focus your effort on either development or remediation.

Joining Prelude to reimagine how control validation happens

CS: Absolutely. You're talking a lot about the technical knowledge that not only an individual requires, but teams require to make sure that these tools are behaving effectively. So you're spending a lot of time in services organizations where companies are coming to you spending quite a bit of money to get the answers that you and your teams are finding from your exercises.

You're going off and you're writing a book to make the knowledge and the inner workings of those tools more democratic. What is ultimately the trigger that starts to push you towards Prelude? Cause it sounds like that's where you're working towards and the concept of like, these things are expensive to do. They're very challenging. What's pushing you towards Prelude?

MH: Yeah. It was a lot of things that, that kind of pushed me towards that for, for red teaming. Like I kind of realized at, I was at a point in my career where I'd kind of done everything that I wanted to do like I got to work with some of the smartest people in the entire universe many times over I got to work in incredible organizations I got to work with some great clients. I got to, you know, live kind of at this bleeding edge of offense for at least like the red teaming world. I really really enjoyed that time but I also saw that, you know, the the world that I grew up in where you could be a relative generalist like I remember when I was at rapid seven right like the way that they build their engagements were you do an internal pen test, external pen test, a web app assessment of mobile security assessment, physical wireless, you piece like that. And like everyone could do all of that. Like that was just expected of you.

If you put me up with like an alpha card right now, I would not know how to do a wireless. So I do not know what the state of the art is. I have never particularly been interested in identity, like at all. Like I've never really gravitated towards active directory any of the like Azure AD stuff, like kind of ironic working at a company that makes bloodhound. it's just, never really called to me. And I was getting to a point where it's like, all right, I need to just accept the fact that like specialization is a real thing. And I already naturally specialized in kind of the vulnerability research side. and then the evasion side, which is, know, it's heavy reversing background required. So we get to that and, know, I was, was looking at Red Team and I was going back to a repeat client for like the third time.

And I was reading the report after the fact, I'm like, man, we already found all this stuff. what, like there's the technical risk of like you implement a prevention, like a mitigation, you go and you close a hole or whatever. And like, that's hard, like really hard work, but we go and we do this. like, man, like, I'm still not getting caught doing any of this. What's going on. And it was around this time where we had the idea to build out the purple team.

So it was me, Jared Atkinson, Michael Barclay, who coincidentally came to Prelude and Johnny Johnson, really set out to build a service that was more focused on this atomic style testing of like, these techniques are pretty relevant, and we need to do all the research to figure out what the base conditions are. If you're familiar with some of Jared's writing, he's spent a lot. Michael's done the same on abstraction maps, this capability abstraction process. We matured there.

And we got this process down to essentially say, like, this is a technique that we particularly care about, or rather a procedure that we particularly care about for one reason or another. And we could do all of the research, we could do all of this, like, underlying work. And then what we'll do is we'll come to you, and we'll do a procedure a day, and we'll walk through, like, knowledge transfer right on the outside, which I'm still incredibly passionate about. A knowledge transfer saying, like, here's how this thing works.

Here's what's going on under the hood here, all the variations that exist for this, but like really only these handful of things are true about other. had some flaws in our approach come to find out, but you know, by and large, the idea was right. And we go out and we do these one techniques a day and it'd be, you know, the three, four of us go out on these engagements. The customer would pay, we'd spend a week with them. We do all this great work and everyone would be super stoked. And we try to give them as much as we could on the way out. But, you know, I thinking about it like, man, this is so incredibly valuable.

I really do think that the rise of purple team that we're seeing right now is a massive net positive. I still believe fully in red teaming. Red teaming is a really, really interesting way to take the totality of a system and say, all cracks considered, can I weasel my way through a series of them? It's like the block of Swiss cheese, right? Can I find the through line that will get me from beginning to end where end is your kind of crown jewels? That's an incredibly valuable thing. I don't think red teaming is ever really going to go anyway. I think it's going to shift a little bit, but the idea behind red teaming will hold for a very long time.

But I think purple teaming is starting to show like when you actually want to exercise detection and response, you have to almost treat those separately of like, you're not exercising detection when you're doing red teaming. Call a spade a spade. Like you're not doing that. You are getting caught by nature, but you're not assessing it.

The response effort is what you're trying to do is like, that's why I don't believe in like a, like an index on detection. Like come get me, kick me out of your engagement, prove that you can eradicate to me so you can walk through your response exercise. So I think red team is more of like a response exercise. And purple team is more targeting that detect aspect. And then that detection aspect, we have as an industry oriented all of our detection chips around the idea of like a technique or a procedure. That's why everybody aligns to MITRE ATT&CK.

And we do these evals every year.

You do that and like that approach is super valuable, but when you deliver it as a service, it is a high cost. it's, it's hard to find people that do this. Like the amount of reverse engineering you have to do is non-trivial. And then you have to have the consultative side to present that information in digestible way. And we go in and like, one of the things that we do for each of these engagements, like, well, we're going to go into your defensive stack. We're going to hunt for ourselves and we'll build detections for you. So you have to know all of these stacks.

So it's like: It's a really high effort where it's a very high skill for in a lot of cases and it's it's great to be able to take the skill set that we have and then pass that knowledge along so that other people don't have to do it. And I still believe in that. we do this for a week and we do five techniques or whatever and then we leave and then it's see you later. Like what happens it's why Microsoft eventually killed Red Forest is like, we'll come and do all this stuff for you.

We'll set it all up for you. And the second that we leave that as a station is completely gone, like something will change. So if you can't do this continuously, if you can't test this at scale, like there's a scaling issue with purple teaming right now. I think it's solvable, but you know, you're testing on a handful of boxes and like a lab system. It's, just not, not a good analog. All this combined is like, I don't know. Like I'm really, really passionate about endpoint security.

I think that this is something that will have a massively positive impact on security as a whole. I just don't know how we can do this five days at a time. Like I think it has to be. So, a good friend of mine actually introduced me to the founders of Prelude and talked to them. I, I don't say I didn't know much about Prelude. I'd heard about operator and what have you beforehand. but I didn't know much about Prelude and through the conversation, specifically with Spencer or CEO, it clicked for me as like, you guys are kind of doing what I want to be doing. It just, unfortunately, in some cases, this is better suited as a product.

But I never worked directly on a product, you know, that I'm the most anti product, like you walk the halls at any vendor conference, and it's like, my God. I really don't like being marketed to like that.

CS: So you're my worst nightmare.

MH: Yeah, yeah, I try not to be an asshole. You know, it you get marketed to the security guy, like I spent my entire career showing people that your stuff doesn't work the way that you think that it does. Like this is not actually helping because you didn't have it configured. You didn't prove to it, you just kind of bought it, deployed it for the most part and then hoped.

But, going to a product company is like, but like, aren't I just kind of feeding into the same things? Like, well, if you do this transparently and you do this in a way that like, look at like just the fact, like, I'm not going to pine on like, you know, take money from this person to say, yes, this is fine. You're not going to do that. It's like, this is just the facts. Like I hate to be the bearer of bad news. I want all of our tests to be a hundred percent coverage. Like that's the goal. Unfortunately, they're not.

And we kind of bear that burden of like, these tests have to be good. have to be realistic. They have to be relevant and they have to prove whether or not your control works. We hope that they do. So we're all. After, but, prelude was kind of just doing what I, I thought we should be doing from a purple teaming aspect, just in a product form. So I made the leap, right after BlackHat of 23, this was my last day. I came here and started out on working on tests.

How control validation evolves from testing to monitoring

CS: You end up coming to Prelude to work on a realistically a testing infrastructure. And you mentioned two things that I think are really relevant to what Prelude was doing at that particular time, which was continuous testing everywhere. And ultimately very similar goals to what you're describing. You want to understand like, is this control that I've spent money on and expect to be doing something for me actually working as expected.

Tell me a little bit about like kind of the goals of that and you know, how things start to evolve. Cause obviously the, you know, we're very early stage things evolve.

MH: Yeah. Yeah. I mean, small company things, things go fast. I, I love a startup environment. So that's where I feel most at home. but when I started, you know, we were really oriented around like threat intelligence and saying, you know, this threat intel report, we're CISA is kind of like the de facto standard. That's why CISA puts out a new advisory describing, you know, some ransomware campaign is using this out and the other thing. And we take from that what techniques were being used. We built tests to, to mimic those. So our tests are just go binary. it's, you know, whatever you can write and go, you can make into a test. but the goal was to take that threat intelligence and produce the test as fast as possible that, that met the intent of what the threat report was saying. so we started on that.

And I did that for a couple months before I kind of grew to this point of, you know, I'm not the world's biggest threat Intel fan, frankly, like it's, it's just reactionary in nature. It's, know, a boom happened, an investigation got worked, a summary was produced, that was correlated with other data points that was then passed through a series of organizations and presented to a customer and then downstream trickles to third tier consumers like you know, federal government and so on.

We're already kind of behind the curve here. Like when you orient just purely on threat intel, it's, you end up kind of on the back foot a little bit too far for me. And, you know, it's, it's still valuable to have all of those techniques and aggregate, but you are assuming that adversaries are not adaptive in nature, meaning that they cannot change their trade craft. They will use the same playbook forever, which SOPs are SOPs.

Like every team has them, but those are not as rigidly defined as you would believe if you live in the world of threat and tell exclusivity. So I presented this idea around like, well, we can kind of reasonably assume that these techniques will happen. And, you know, if you want to access credential material on windows, what's the way that you do that? Like the canonical examples of extraction from LSAS memory.

There's other things like DPAPI secrets, there's, know, God forbid you're touching the SAM in 2024. But you know, here we are. You have all of these different ways that you could do it. And it's like, we could still use that same research process from a prior life and taking the learnings from that and then refining that down to a much smaller set of things that we actually have to test, we could do fewer tests for techniques that matter really, really, really deeply. And we don't have to produce a million of them, we could just produce a relative sample set of things that express the variance of these things and build tests for those. So we started working on tests for those and kind of decoupled a bit more from Threat Intelligence and started saying, you know, these are the techniques that we Prelude think are, are really important and we're to build really, really solid tests for those that'll be executed through our runner.

So we started on that process, again, peeling away from threat intel and becoming more opinionated and start producing those. So I did that for a couple more months and learned a lot through doing that. We had some good adoption from customers and people were running tests. we got to see what the actual guts of these things look like. That's the ultimate reason of why I'm at Prelude is like, I'm fulfilling my red team dream of like, I actually get to break these things in real time. And I get to see what it looks like at a whole bunch of companies. And I can say like, it's not just this company is doing one thing better. It's the product is not doing this thing efficiently, or it does this thing exceedingly well.

But this thing is kind of not great at seeing all of that data showed a light on a lot of kind of weird trends not only through deployment, but the types of tests that you would run and where you would run them. It became very, very interesting.

CS: Tell us a little bit more about that. So you're saying, ultimately, it's interesting because you're following a very similar trend throughout the path of your career, making this more democratic. How do we help teams prioritize the things that are most important so we trim down the scope of the tests, the areas they have to run? What are you learning about in multiple environments? Again, you're seeing this more frequently. seeing it across a lot of different, customer environments. What are we learning about the tools that we did maybe didn't know beforehand?

MH: Hopefully I don't get crucified for this. know, we do these evaluations every year that we always get a hundred percent on, but why aren't we seeing a hundred percent? Like is something wrong with our tool? Cause like, I'm like, I'm pretty confident in the tests that we write. Like they work. Like I can see your creds going across the wire and I'm looking at them. I can see my persistence fire. Like I'm watching this happen. Like this is all happening. Why is the success rate at like 10%? Like why, why is it not a hundred? Reconciling that is a matter of fact saying like, yes, like these tests are showing what real adversary tradecraft looks like. We're seeing real results for what a real adversary would see.

10% is a weird number because if you assume that the customer has the same control, the same version, it's in the same state, it's configured the same way, it should have consistent results across everything that we test on. So if we run a thousand endpoints or whatever, we run the same test at the same company with the same thing, why are the results different? And what it showed is that they're not. These are all weirdly off. They're not all the way configured.

They're not all the way adopted, they're in weird states. Like one sensor is, you know, two versions behind and is in need of an update, but the system hasn't rebooted. So it goes into like a reduced functionality passive mode and, and now isn't, it's just not responding to test stimulus. It's just kind of there and on and doing some things, but it's not doing everything that's supposed to be doing. Sometimes, you know, it's we get deployed on a system, it's like, well, okay, let's evaluate how the EDR performed. Okay, there's nothing there. Why is there nothing there? Well, there's no EDR deployed on the system at all. that's like what, why would we test something when there's nothing to test?

These kind of, I know it's super basic and like not the sexiest thing in the world, but it's like, why are we so focused on these advanced techniques and these super, super cool tests if we're missing EDR on 20 % of our endpoints or this sensor is dead in the water and we had no idea. We kind of need to fix that before we get into these tests. It's like, okay, well, how do we go about fixing that? Can we identify that state and then...

And then kind of get into the more interesting testing bits. The other piece about this is like, one of the issues I dealt with purple team side was you have like a C2 agent. Like everybody uses these kind of like really two approaches. You use an off the shelf C2 framework, use Cobalt Strike, BRC4, Nighthawk, Mythic, whatever, and you deploy it there. I guess maybe three or you have like a custom internal thing that you use specifically for that. Previous life we used Mythic and Cobalt Strike.

I cannot in any circumstance convince a customer to go to their IT manager and say, can you deploy a Cobalt strike beacon out to like random hosts in my environment? I promise it's just for funsies. These consultants over here are going to do all these great things for us. They say there's no chance in hell. It's not happening. You can have like these two boxes and originally, I thought it was like, okay, like this is just because it's like overtly malware and that's sketchy and they don't want to deal with exemptions or whatever. It's actually like a little bit weirder than that. People are super resistant to just deploying software. There's this perceived barrier of like it costs so many social capital bucks to deploy a new piece of software because it like it can be a high friction process, especially like bigger companies.

But, to deploy software to like five endpoints, it is like seen as this Mount Everest of effort to climb. And I think that like, it's just a value equation. Like it has to be so massively valuable. If I said to you today, if you install this thing on your workstation, I will solve the local admin problem for you. People one aren't going to believe you, but two, like if they do believe you, it's not even going be a question. Like they'll just install it.

Like it just, it's a value thing. The value has to outweigh the pain of deployment. So when we have a legitimate signed, reviewed, a tested piece of software that's like 4K, like it's super, our test runner probe is like insanely small. Like it checks all of the boxes. It can't do anything malicious outside of like run tests, which kind of the value of the product. that's hard to deploy too. It's like, okay.

So a lot of people are just super agent averse for a bunch of different reasons. It could be like people just don't really see a lot of value in this testing thing. It can be, it's not quite there. It's the perception is they have to deploy on everything in order to get value from this. So the people being agent averse, like you can deploy the probe through Falcon. Now you go to Falcon Store, click it you just deploy our probe. Like it's one click easy.

It that's still just too hard in a lot of cases for people. like we have this problem of. We may not all the way be ready to test these advanced techniques and we might not even want to, in a lot of ways, like I still believe like wholly in testing, like that's why I'm here. Like you have to know beyond a shadow of a doubt with objective fact that your control is working or it's not like testing super important, but there's this precursor thing to that of like, you giving your controls a chance to work? they configured well enough to work when you need them to? Like that's also a meaningfully difficult problem to solve. So we kind of started heading down the path of, of trying to solve that in tandem with, with how do we test well.

The future of control validation

CS: Right. Cause you raised the point of, okay, deploying a test everywhere might be a level of sophistication too high for a team or simply just not something the right team department is going to want to do. But, there's also the many, many easy things that we can do in our tools that are still escaping us. So, you provided an example earlier of credential dumping. We can deploy a test for that only to discover that it still gets through, but we forgot to check to see if your credential dumping prevention policy is still on. So what does control validation start to look like over the next five years? Where do we go from here if testing is not the immediate answer for a lot of people?

MH: Yeah, I think...you kind of have to split the market in some ways. Like there are companies right now, like, you know, there's a French Canadian financial institution who I love dearly. Those guys are ready to test, like they got their house in order. Everything is where it needs to be. They have a very mature program, got a good relationship like those folks are ready to test. can't just say like, but testing is hard. So it's not worth it. It is it's very well worth it for the people ready and able to receive that information and action on it.

So, we kind of put that bucket aside of saying like there are companies that are ready to do that. The companies that want to do that need to kind of figure out like, are we ready to do this? It's, you know, are we ready for a red team? Like a lot of people just get a pen test as an analog for like, just kind of like the light version of this for testing, like making sure that your control is present and enabled and configured.

Like it's super, super basic. know, but it is astonishing how many companies have like massive gaps. And I knew this from like previous life where, you know, in a world where like device guard is really prevalent. How do you get around device guard? You find systems where device guards not enabled and you go there and you do stuff there. Like that's a real answer. Like how do you evade EDR? You go where it's not.

And you create these little pockets inside of your environment where an adversary can exist. like, we just intuitively capitalize on it. If you, if you look at someone like the opsec scripts and Cobalt trick, sometimes the first thing that you'll do is you'll list out all the process. You'll get a color code to think back like, that's a defensive product. That's, know, that's CS agent, whatever that's blah, blah, blah process over here. and it'll tip you off to like what's running, but what do you do when that, when that system doesn't come back with any results?

Throw a little party for yourself because you got an easy day and you can carry on with it. Like no more easy days for adversary. Like it's, know it's not the sexiest thing in the universe, but it's a real threat. And like, when we just start from that, like is the control there? Yeah. That's like a hard problem to answer. Like how do you go into your EDR dashboard and know what's not there? Like that's not easy. You have a whole bunch of spreadsheets and V table lookups and all this crazy stuff.

Or you can find an easier way. Is it configured is also like every EDR, least the major ones today ship with these kind of policies that go with them. Like CrowdStrike prevention policy is like 41 items or something like that. it's, you know, are toggles or ML sliders. Like a toggle is an on and off and ML sliders are range that you can configure something for. How do you know like that those settings are where they need to be and are configured appropriately and aren't changing. And that last bit is like the real big piece of this is like anyone could do a point in time assessment. I can walk into my CrowdStrike dashboard right now, go to endpoint management and policy information. And I could start looking through all my prevention policies and say like, okay, this is what I do every quarter as I go and review. And that's in fact what a lot of people do.

And then the same thing that happens that we know from like, you know, the CEO calls like I am in a board demo right now and I do not have access to SharePoint to get my stuff. And the poor kid on the other end of the help desk is like, my God, enterprise admin. I don't know what to do right now. I need to give this guy access, but whatever. And then never goes back. The same thing happens with policy of like, this is causing too many false positives for us. It's too noisy for right now. I'm going to disable it. And it's like, well, should you have like what just changed this kind of continuous monitoring? think is a big thing. That's the same thing you see in a lot of products right now of like the difference between one of the differences between Bloodhound community and enterprise is this continuous check and then be able to kind of like diff these things back and forth to watch for regression. So monitoring for regression is a big, big thing like that. The control validation side of do you know that this thing is on working configured, whatever. That's one piece of it.

Doing that continuously and watching for state transitions of like, think of like an MNA, right? Like you acquire a new company, you bridge the VPN, they join your domain. And all of a sudden you're like misconfiguration thing drops or increases 300%. Like, how are you going to know that? Like, hopefully your MNA team found it, but let's be real. they just don't find it. Exactly. So like you, go and you find these little pockets of like, oopsie. Like we just onboarded our 200 summer interns rip, but also we should probably make sure their devices are fully configured before they start really hammering away on them.

Doing that over time, it unlocks a lot of value for people. Offboarding people have lots of examples. yeah. Yeah, I mean, we saw one instance where a customer had this thing running for a while and they found this device. So it was was configured in a weird way. It's like, we off-boarded that dude like three months ago. He still has access to device and it has no EDR. Like that's a, we're going to work this as an incident now, not as a, like a misconfiguration. So inside your gut right there. Exactly. Like you find all kinds of really weird things. Like that surely can't be right. There's no way that that happened. And you're like, we just forgot about this portion of our network and that just doesn't have anything running right now.

Or like the really, really nasty one is, like CrowdStrike, MDE, they both have these concepts of like a zombie mode. In CrowdStrike, it's reduced functionality mode, which is just like a, it's just kind of like limping along. It means different things based on the operating system. Like on Windows, it can mean, you know, the sensor is out of date and Windows has progressed in like the telemetry chain. So it can't really correlate, can't make sense of what it's working with.

On Mac, you know, it doesn't have full file system access and Linux is different. MDE Defender specifically has passive mode, which is a really interesting thing. It's controlled on the registry, but it's meant to be a way to say that there's another EDR or another security product acting as the effective antivirus. So it puts Defender into the background and lets this other thing step forward.

You can install an EDR and then uninstall an EDR and it gets left in passive mode. So it's still just chilling there. So it's like you had SentinelOne, you offboard SentinelOne, you went to MDE. MDE is now still running in passive mode. Like finding that information is not easy. So also finding these like runs time state problems of we're not as effective as we could be, but otherwise it's just sitting there looking like it's on, like it's checking in, it's...apparently working, but if you didn't know about this like one weird edge condition, like everything looks fine until it's not.

And how are you going to know about that?

So finding those things, I think is another really interesting value.  

Where does BAS fit into control validation?

CS: We're coming up on the time here, but it's interesting because I wanted to double click on a couple of things that you said, which are interesting in terms of like your personal narrative. So you run these red team exercises, they're point-in-time, they're expensive, and they tend to be focused on almost like model devices. Like these are the best of the best the company is putting forward. Try and breach us.

The second piece is that concept of like scalability and can you already said it continuous where you come to prelude and the platform starts to evolve and you're starting to answer those easy things first. Like there's so much about these tools that we barely understand as it is. How can we possibly put ourselves in a position to test effectively before we know like these base principles, like is it configured?

What do these policies mean? Are they enabled? Are they right for my business? The more traditional methodologies that you're coming from, one-off pen testing, breach simulations, do those have a place moving forward? Or is this like new model of control validation kind of the path forward?

MH: Yeah, I think it's, you know, the basics are the basics for a reason. Like everybody knows brush your teeth and floss your teeth. Like, just, gotta do it.

I think the control validation piece is somewhat emerging as a more serious contender, but it fits into a larger ecosystem. I still, like I said, like I believe purple teaming is probably one of, if not the most valuable offensive oriented services that's out there today. That doesn't mean that I don't think that red teaming has a place. think red teaming is incredibly valuable for finding and evaluating cracks in the entire system and then evaluating. response efficacy.

I think pen testing is useful for when you want to keep a tight scope on, and pen testing, million different definitions, same thing with red teaming, but you know, you have the idea of like a narrowly scoped engagement, where you really want to focus on one thing, or you kind of want to just run a red team without the detection concern. You just want to find cracks without worrying about evasion. Those are all valuable. I do think Bass is in a very challenging spot right now because the there's so many open source contenders right now that present really interesting value props one because they cost zero dollars, it's just operating costs. But the the value in it is really like, what are you testing? Like a test runner is really like a means to an end. The value in all of these is like, you know, one, it becomes a little bit easier to deploy the test runner.

But the value of the content inside of it is really hard. So having a really opinionated take on what you should be testing. you know, personally, I don't think that MITRE ATT&CK bingo is doing anything but harm for our industry. So if we have, you know, tests for all 643 techniques and ATT&CK V 15.1, and surely that'll make us covered. It's, you know, it's the same deal of like, but you your eval said I'm 100% covered against credential dumping.

Why did someone just run off with the bag with handle catz this week? Like that's, thought we were protected?

Taking a more opinionated and thoughtful approach to testing, I think may get the bass market out of this valley of death right now. But you know, it's, there's just not a lot of innovation happening there right now. And I think people are starting to kind of turn and look at it as like, it's a high cost to get into because you have to deploy this test runner. You have to have a team really specialized to operate these. And then you have to have the, the balance to be able to do something with it. Like the entire purpose of BAS is to highlight an issue.

One of the most exciting things that we've done, I think is the ODP feature where we like, integrate and say like, okay, we ran a test. Like I can automatically go see, did CrowdStrike see the telemetry related to this operation? Did it detect us and attribute us correctly? Did it prevent it and demonstrably prove that it prevented us like that idea of automating this workflow is important because it is so hard to take the output of a test of saying, I just did this bad thing. Did I see it? Did I catch it? Did I do something to stop it? That's hard work that requires real person hours to do.

And, I think until we can kind of reconcile that fact with like what security teams are doing right now, I think it's going to be harder and harder for people to BAS internally. Like I need this to do my job versus this would be cool. It might make things easier to have a platform to do this. So to your point, so many things that you could be doing in advance of that type of exercise to exactly shore up the defenses. Yeah.

Is it having EDR on and configured everywhere more or less valuable than knowing that your registry run key detection works as well as you think that it does. Like they're both valuable questions to ask, but you probably need to answer one before you answer the other.

Parting thoughts

CS: Excellent. Matt, this was excellent. I really appreciate the time, you know, we get towork together, but always wonderful to get to know you more about your background. And I'll turn it back over to our audience. In there's any questions, now is the time to put them in the chat. But other than that, Matt, any final thoughts before we part ways here?

MH: Not, not particularly. You know, this has been, it's been a really interesting year for a lot of reasons, but I think kind of like looking forward to, to where we're They there's simultaneously like a lot and nothing changing. think the, you know, advancements, sorry, this might be cringe to quote a few people in the audience, but like, you can't really ignore the AI thing.

Like it's, I still think, you know, we may be over hyping it to a good amount of like trying to find, you know, any reason to just shove dot AI into our domains and make it a thing that we do. But like I use co-pilot like every day to write code. It's getting to a point where like I could say like, Hey, give me a persistence example and it can give me like complete code that compiles and works right now. And what that's telling me is that, you know,

If I'm just being able to like plain English, telling a computer, give me malware and it gives me malware. Who else could be doing that? Like, did we just lower the skill for, for adversaries to enter in? Like, can we now have a system that is facilitating the entry of lower skilled threat actors to do stuff that they still suck, but they, can play ball now. And on the other side of that, was like, I can, you know, create a really highly specific model to, you know, I've got an LLM that solves some really gnarly reverse engineering problems for me. If I could do that, like what can more advanced adversaries do? Like can they now go faster? can they get answers to harder problems more quickly than they would have others? So like, I think both ends of the like adversary capability spectrum are growing, not shrinking.

So I think it behooves us to start thinking about the world in a sense that they're going to be more adversaries capable of moving more quickly and at the high end, more advanced adversaries able to pivot on information more quickly and starting to think about like what the world might look like when, you know, how long does it take you to work in an investigation right now? it a couple of days, couple of hours? I'm sure people remember the threat report that came out of like the Russians dwell time is 19 minutes or whatever.

What does the world look like if that's like 19 seconds?

Like, is that a, is that a thing that we need to worry about? Like, I think it's something that we should be thinking about actively. So, yeah. And all the more reasons to do the easy things well and the simple things. Right. It's just, yeah, no easy wins. Right. There should just take things away. The easiest thing with the highest impact is sometimes the best going back to basics. Nobody.

Everybody want to be a bodybuilder. Nobody want to these heavy weights. Like it's just you do the basic stuff and it pays dividends.

CS: We did have a question come through specifically regarding AI. And I appreciate bringing it up. I think it would probably be sacrilege to do a retrospective in 2024 and not mention AI. So the question in the chat is, is Prelude implementing AI and its agents or its runners? Certainly we have, you know, we have AIs operationalizing threat intelligence, but open to any thoughts you have as well.

MH: Yeah, we're experimenting with it. There's some interesting developments coming up today. I'm not going to out anybody in particular, but there are a few very, very exciting companies that are just now apparently getting funding, which is really cool. In the autonomous operator space where a red team is highly specialized, it requires a lot of background knowledge of how the systems work and how the tools work and when you want to use them.

teaching models and have a series of agents that are capable of doing essentially like autonomous red team ops, they you know, varying degrees of success. It's, it's always different when you put it into a real environment, they have the problem of like, who is going to let SkyNet in their environment first. That problem, I think is, is ever present there. But I do think that not not exploring the capabilities of like, exactly what I talked about, right? Like,

Can you make an adversary, can you make it easier to operate? Can you operate faster? And can you solve the hard problems more quickly? I feel like there's different avenues to go down there. So long story short, like, yes, we're experimenting with it. You know, it's still very much, I'm in charge of research. yeah, I figure what angle I'm coming from. But it, you know, the, the initial results are promising and the fact that like, you can teach a model the context behind what output means and how to convert that into input into another tool down the line to further objectives. we're not building SkyNet. We're not going to release the ultimate worm. But it's an interesting experiment to say, can we outpace a system now if our entire workflow is human in the loop?

How do we reconcile that with a world that might be fundamentally different tomorrow?

CS: Well said. Well, that is the top of the hour. So we appreciate the folks that joined us today for coming out. Those of on LinkedIn and Twitter as well. Matt has been a wonderful guest with us today here. Obviously, he's our director of security research. So I get the privilege of working with him every day.

If you're interested in learning more about Matt, you can find him on LinkedIn and Twitter, as well as pick up a copy of his book, Evading EDR from No Start to Press. What a great stocking stuffer for the EDR-obsessed friend in your life. I'm sure we all have many of those.

MH: I'll mail it to my mom.

CS: In the meantime, you can check us out at preludesecurity.com, learn a little bit more about some of the control validation stuff that Matt alluded to today. And obviously, we'll see you in the Discord. And appreciate you all coming out today. So long. Thanks for coming, everyone.