00:00.00 dcppodcast Who always started off. 00:00.00 Jonathan Johnson I normal all I just making sure we're keeping that thing all right here we go. Um everybody thanks for ah, joining us today on episode 26 of the dcp podcast today with us. We actually have a part 2 today actually I think's the None time we've had a part 2 was with Jamie Williams so 00:00.00 Jared Atkinson You always started off dude. 00:18.30 Jonathan Johnson Realized that we ah had a lot ah to talk about the last time and we didn't talk about amazing. Maybe we talked about like a None of it. So um, start doing a part too. So um, Jamie thanks for joining us today. Yeah so right before we started. 00:29.47 Jamie Williams Thanks for having me times times too. That's awesome. 00:37.16 Jonathan Johnson Recording. We're actually already on a conversation. Um, so I guess let's just restart that conversation because I think all of us had things to say and then thanks to Luke the producer over here was like whoa whoa whoa this is good content so who wants to start that one off you were Jared. 00:49.71 Jared Atkinson Why don't you start it off because it was your topic that you were interested in I think. 00:53.28 Jonathan Johnson Okay, all right sounds good. So um I think it's fairly like public knowledge now that ah miter is performing something called MdrEval um and so my question to Jamie was um, whenever creating or going through the Mdr Evaluations what changed in terms of how the execution was going to be done. Um, and maybe what was taken from the edr e-vows that maybe didn't work previously that is now being applied to the mdr evows. 01:24.84 Jamie Williams Yeah, um, so I wish I had as I'm going to had ah as good of an answer as I this time as it the none time but I'm walking through. So basically it's a different paradigm I think we're kind of chipping at the same question of like what really is a detection. Whereas like if you're evaluating edr it's more technology focused. It comes down to a lot more like visibility and then like you know as you can imagine like you know once you kind of have the data. How do you process it? How do you display it to a user a lot of that carries over to the Mdr evaluations. But honestly. Um, you know it's such a different beast where like you know Mdr is going to have you know, ah those same challenges in terms of visibility but also like their real core like value ad is not just like you know what? they're bringing in because you know a lot of users aren't even going to see how like the sausage is being made. You know it's more you know now that all that data is available in that analytic sphere. How are they presenting it to users. So. There's so much of a different paradigm in terms of you know our previous edr valuations was just you know, really purple team like hey we'll tell you exactly what we did when you did it and let's walk through and see like what is the tool capable of. 02:30.46 Jonathan Johnson Um, yeah. 02:32.22 Jamie Williams And like let's measure almost measure the ceiling in terms of like what is theoretically possible of this capability in terms of detecting these behaviors which is that detection definition is more bringing an in and putting it somewhere on a screen where there's an extra layer here with this mdr and it's like a really interesting abstraction. 02:35.35 Jonathan Johnson There. 02:47.79 Jonathan Johnson Me. 02:50.21 Jamie Williams Where it's all of that plus like that services you know Analyst What do they choose to show a customer.. How do they present it you know and it's a little bit more of ah, kind of maybe branching off to not just like detection but like response like you know as a customer I can see those outputs and start to think you know. How? how would I act on that is that enough for me is there maybe like more insights I need is that like you know is that like too much depth. Not an up depth but also like what did you choose like I think we're just talking about like you know if you threw ah the same you know, processed tree in front of all 3 of us or all 4 of us. You know we would choose different parts that's kind of stood out to us. So like all of that kind of swirls together and like I think a much different delivery but hopefully something that I think honestly is maybe a little bit more impactful for like actual end users in terms of like you know you look at a Edr evaluation. 03:40.32 Jonathan Johnson Yep. 03:43.26 Jamie Williams Like there's obviously like a lot of criticism of like you know that config doesn't make sense to me or like you know I wouldn't do it that way whereas like a service is like a maybe a little bit more straightforward in terms of like yeah that actually like I sitting in a like sock could actually act on that or like maybe not. 03:50.32 Jonathan Johnson Um, yeah, yeah, so. 03:50.32 Jared Atkinson Um. 03:56.12 Jared Atkinson Is there is there a consideration for so mdrs are built on top of edrs but it could be plural edrs right? So like for instance, red canary when they first got started at least my understanding is that they were very much tied to carbon black. But over time they've expanded to where now they have support for. 04:09.88 Jonathan Johnson Um, yeah. 04:15.75 Jared Atkinson Probably not all I would imagine edrs. But ah, you know the most common at least and probably more than just the most common. Um, but there's there's an idea that the like so the idea is is that the mdr is an abstraction over the edr and it's an additional capability but there's some and so one of the problems is is that. 04:18.69 Jonathan Johnson Period. 04:35.13 Jared Atkinson The ceiling for an Mdr is predicated on the ceiling it. It can only be as high as the ceiling of the edr to some degree does that make sense. Yeah yeah. 04:40.43 Jamie Williams Yeah, there's cascading like decisions and stuff. Yeah I think um and that was you know I've not been as hands on with like the Mdr aspect but part of the scoping was exactly that point was you know part of when you choose an mdr is that decision of you know. 04:40.67 Jonathan Johnson Yeah, yeah, yeah. 04:47.69 Jared Atkinson Um, okay. 04:58.45 Jamie Williams Whether or not they're going to work with whatever you have like I know there's some words like B Wio Detection but there's also like you said like other services where it's like hey like as part of our agreement we bring in. You know this is what like carbon black or certain sensors or we demand that here's basically a config of like the till energy types we need. So i. 05:01.50 Jared Atkinson Sure. 05:10.36 Jared Atkinson Um, yeah I say. 05:16.65 Jamie Williams You know it's the best of my knowledge. The evaluation considered all that because it's It's basically like you know, same structure as Edr evaluations. The range is there go in and set up whatever the hell you want like whatever makes sense for you but we're most importantly, going to write that down. So like you know if you go and look and you're looking at you know vendor a versus vendor b. 05:27.33 Jared Atkinson Yeah. 05:35.64 Jamie Williams You can work that back from exactly like here's the outputs but take that down to the inputs and say well you know they were able to do this but it was with this like no additional tap where you know as a customer. Maybe I don't have that type of telemetry. So I need to maybe look at the results in a different light because you know all these different aspects might not have been possible. Um, so I think you know. 05:36.77 Jared Atkinson Um, yeah. 05:45.63 Jonathan Johnson Um, yeah. 05:54.45 Jamie Williams That was you know one of the big considerations we have is like not overprescribing but I think that is another good point where in the spirit of all these evaluations doing that side by sky side like so like scoring and like you know ranking really isn't possible. 06:06.57 Jonathan Johnson Um, yeah. 06:09.16 Jared Atkinson Um, yeah. 06:11.90 Jamie Williams Because there's just so much nuance where it's like you know, rather than figuring out who won look at everything and kind of like see yourself and like does you know from soup to nuts does that make sense for me. Um I think I mean I'm going to keep like hammering that point over and over again. But I think that's this is another really good reason. Why like you know in another context. Um, that. 06:19.70 Jonathan Johnson Um, yeah. 06:30.89 Jamie Williams Point is just kind of exaggerated and highlighted. 06:31.93 Jonathan Johnson Yeah I think ah so I think like 2 things that come to mind is like 1 thing that's interesting about the evaluations in general whether it's edr mdr 1 thing that think's unfortunate is it seems to be like a big competition obviously like inherently amount of be to how miter is like wanting it to be but. 06:32.30 Jared Atkinson Um, there's. 06:50.44 Jonathan Johnson That's what yeah, but that's a yeah I mean that's how everybody's viewing it right? like I mean like obviously right canary wants to beat that next mdr and be like we want to score higher than everybody else, but another thing that's interesting too is like say all mdrs are obtaining the same amount of data. 06:50.64 Jared Atkinson Definitely not how they wanted to be thick. 06:50.74 Jamie Williams Um, I mean it's healthy though I mean like. 07:09.16 Jonathan Johnson And same types of data from the same meteors. What's interesting me and you brought this point up when you said like you looked at a processory of an execution. We'd all pull out different things. Um I think the same applies fromdrs is because people ah create detection upon different attributes and aspects of the activity. Um, and one thing I I find interesting is like. Who's to say None detection from 1 person is better than another one based on the same action and so then that goes down into maybe at what level the abstraction does it cover does it cover like a bigger gap area but then okay man. 07:46.31 Jared Atkinson Don't take us to don't take us too far from the Mdr level. Yeah like I'm happy to talk about the the direction that you're just heading but I don't like I got more things say at the higher level None I think okay. 07:46.46 Jonathan Johnson This artist. 07:50.66 Jonathan Johnson Yeah, yeah, let let me say this last thing of thought and I think one of you guys remember it and we'll get back to it I think another thing too is like is the detection from the Mdr explicitly. Created based on how sensitive they are of the false positive false negative ratio which is then going which is then going to find how deep they can actually hit the abstraction of the action. 08:09.52 Jamie Williams Um, one Oh sorry, go ahead. 08:18.70 Jamie Williams Just quick thought there the competition. Love it Healthy like is it Even if you're just competing against yourself the pure like science of assessment of like looking at what you're doing and trying to be better I Think that's a big win for the community. So like you know you know, even if it's just kind of you know, individual vendors. 08:31.46 Jonathan Johnson Yep. 08:36.85 Jamie Williams Love it. But then to your point about like who's to say I think one of the big principles of evows is like we're just trying to be like everyone wants mighter to be like the person or like the organization that says like you know who won who didn't to the best of our ability being transparent and saying like that decision comes like. 08:54.81 Jonathan Johnson You. 08:54.99 Jamie Williams We're just being a conduit to let individual users do that themselves. So I think all of those principles exist. It's just like kind of knowing your place and saying like you know recognizing the space in terms of I can't you know anyone can't. And like I said even as a user I couldn't like go across these vendors and say like this is going to be perfect for me or this is going to solve all my problems It's always going to be that calculus of like there's pros and cons and like you said there's depth within a single detection. There's a broader picture. There's ah you know even getting to like real like practicality from an edr or like Mdr perspective. How often would the solution to most of these like detections just be quarantine the host like quarantine it and then like from there we'll figure it out so like you know beyond like you know whether or not your detection is like a certain depth or a certain like you know, tactic or technique. Really It's like. You know it comes down to There's just a bunch of tradeoffs and it's like at the end of the day as long as you kind of get to that point I don't really care how you got to that point. There's just a bunch of different ways to get there and it's really just kind of that course correction of like which one makes the most sense giving everything that kind of is at play with whatever situation I'm in. So. 10:01.12 Jonathan Johnson And um. 10:02.91 Jamie Williams Super Fuzzy gray answer but at the same time like I think that's just like a you know overwhelming respect for like the general like scope of the problem we're dealing with. 10:10.71 Jared Atkinson I have a ah quote from I feel like I quote him every single episode but I think this one I haven't said before nasim nicholas tole the author of black swan the Concerto series one of the things that he he says is that ah phenomenology is robust but theories are not. And what he means by that is the phenomenon that say lifting a hundred pounds makes you stronger lifting heavy weights makes you stronger that will always be true, right? We could observe a phenomenon and we know that that's always true. The theory for why that works why lifting heavy weights makes you stronger. 10:47.75 Jonathan Johnson Yeah. 10:47.82 Jared Atkinson Has changed numerous times over like numerous times over you know the last century or whatever right? So like now we know that it has or now we think we theorize that it has to do with ah like there's all ideas about like hypertrophy and like how to build your different like kind of workout like Johnny would be way better at that than I am but like ah. The thing that stays the same is that lifting heavy weights makes you stronger but trying to understand like what we why we thought that worked and how we thought that that actually influenced the body to make the change a hundred years ago is different was different than how it is today right in that and like today we. 11:08.72 Jonathan Johnson He. 11:24.10 Jared Atkinson You know, hundred years from now we might have a better or different understanding 100 years from today and so the thing that the the reason why that came in my mind was that what you're doing from the evilval perspective you being Mitr I guess or the people conducting the is you're providing the like the phenomenology you're saying. 11:26.78 Jonathan Johnson Yeah. 11:43.36 Jared Atkinson We did this and this was the result and what what people are getting kind of up in arms about is that all the vendors are trying to provide the theory for how you should interpret that and so nasim Nicholas Teleleb says that you should be skeptical of people that speak in theories and you should be. 11:50.29 Jamie Williams There. 12:00.51 Jared Atkinson Ah, you should be more accepting of people as speaking Phenomena So basically it's like if you're going to a vendor who has a vested interest in telling you that they're the best and then you know surprise surprise to tell you that they're the best they're probably not like a even if they are even if they honestly think that they're not a um. 12:02.77 Jamie Williams Um, that makes sense. 12:08.54 Jonathan Johnson Yeah, yeah. 12:18.62 Jonathan Johnson Yeah. 12:19.19 Jared Atkinson Uninterested party right? and so you should always be skeptical about how they came to that conclusion. 12:22.20 Jamie Williams But yeah I like that a lot too is like even a step further would be like if they're all going to kind of push their own theory like honestly like you know this whole cyber thing infosec is like an industry so obviously like I get it like when these vendors come out and they do their blog post I'm like yeah you got to. 12:34.46 Jared Atkinson Um, yeah, of course. 12:39.11 Jamie Williams You know in order to do you know the things you do you need to keep the lights on you need to keep you know keep the bills paid makes sense. But that said like you know there might be None of these theories like beyond the phenomenon that like hey that aligns with me like you know whether or not I'm a sheep I don't know but like I I get that like that kind of grooves with me. So I think like that's that's a really interesting like you know I guess analogy in terms of like you know, just basically giving vendors a canvas and you know it's kind of marketing. Whatever but like you know how often do we actually get us like insight like I've I've been a fan of like vn canary for example, for years I have no idea how it works like I've seen like you know, um, like. 13:05.32 Jonathan Johnson You know. 13:11.25 Jared Atkinson Um, sure. 13:16.72 Jamie Williams Conference talks and like individuals but like actually being able to see like okay this is how this mdr actually operates in terms of collection processing outputs phenomenal like same thing with edr like I feel like I've worked and seen a bunch of these tools for years have never actually touched them. 13:22.10 Jonathan Johnson Um, damn. 13:32.36 Jamie Williams But like the insights you get from just like sitting down and like seeing like you know under the hood. Not only like how users use it but like talking to like their users and like they're like sometimes they even brought their devs or like okay under the hood like here's how this sensor work like here's how the data comes in and like talking like you know talking through like detection logic like that like I I get. 13:48.10 Jared Atkinson Me. 13:51.75 Jamie Williams The the demand signal for like tell me how to cyber like that'll probably never go away. But that said to your point like you know, um, just baby steps towards like kind of you know over time. Maybe there is a central theory that kind of makes sense for all of us. Maybe there's not but the better we can kind of create those like. Pipe pipelines of like okay, maybe there's None common ways of going about this like you know here's how these vendors kind of stack in terms of like those. Maybe we'll call them rows or whatever. Um, pick your row like that. That's not like you know push button cyber but that's at least a little bit better than just kind of. 14:13.58 Jared Atkinson Um, yeah, yeah. 14:14.26 Jonathan Johnson A. 14:19.98 Jonathan Johnson Um, yep. 14:22.48 Jared Atkinson Um, yeah. 14:28.10 Jared Atkinson Um. 14:29.66 Jamie Williams You know what? what? but vendor kind of has the you know billboard along my path to work or whatever that might otherwise influence a decision. 14:35.30 Jonathan Johnson Um, yeah, 1 thing. 14:35.37 Jared Atkinson I I suspect I suspect that they're like there probably does exist a correct answer now like we we don't we don't know what that correct answer is we don't even know if we're measuring the correct metrics to be able to make the correct answer right? and so like. Should ah there probably is like an optimal way to I think almost certainly there has to be an optimal way to approach solving this problem but it's probably so complex that there's the chances that we know what that answer is or we'd be able to identify it today or even None ears in the future. 15:11.95 Jamie Williams Um, yeah. 15:12.36 Jared Atkinson Is extremely low and so really the way this is I mean you think about like ah capitalism I know some people may may think that capitalism's not great, but None of the benefits of capitalism is nobody's actually setting the price of things and like the the like kind of the consensus or the ah the. Collective consciousness to some degree is setting the price of things and so I think by allowing people to to choose the the path that they take ultimately we as a community are able to make conclusions draw conclusions that we couldn't we couldn't draw individually right? So there's this idea of like collective consciousness to where we are able to solve problems. As a community of people like being in a city For instance that we're not able to solve even the smartest individual would not be able to solve right? And so yeah, it's yeah, like ah ah what is it not cloud computing. But um, when you take a bunch of computers. You put them together and you make it basically have. 15:56.28 Jonathan Johnson Yeah. 15:58.14 Jamie Williams It's an organic like feedback loop effectively. 16:09.25 Jared Atkinson Way more computing capability distributed computing. Yeah, so there's distributed computing is able to solve problems that ah would not be solved like it's it has more computational power than the ah components of its parts right? So like. 16:10.73 Jamie Williams Distributed. 16:24.83 Jared Atkinson If You add a None computers together. It has more than a None computers worth of computational capability and that's kind of like to some degree. What the market is doing for us in some way. Obviously there's a limited number of options to choose from and those none of the options might be ideal who knows but that's kind of I think that's kind of what we're. What we're driving towards is. 16:43.67 Jonathan Johnson Yeah, None thing I like about the evaluation is like I mean oftentimes like we see companies getting like a pintes or red team done yearly you know, maybe like every so often to me I kind of view that as like a similar thing as to what is happening to MdrEval because the mdr eval is testing the mdrs capability of the data they are collecting um but then also holding them accountable by which like hey like we're going to run these things like it's it's just less abstracted from the control of the Mdr themselves because typically like if you want to like test optics. 17:16.44 Jamie Williams Um, that. 17:20.74 Jonathan Johnson And that comes internally. There might be some bias that applies. Well there's less bias that's applied from the evaluation coming from an external company and I see that coming here and then like if something doesn't trigger or it's not like what's interesting is like when I look at Detections you know jar and I have had this conversation so is Luke and I like. There's a difference between an an alert triggering and there's a difference between an alert triggering and action being able to take upon that alert triggering right? So it's like is the context there or does the alert even make sense to have triggered because if I look at it I'm like oh like this is weird. This looks like. Win our m right? But it's like that that doesn't like give whoever is like doing the analysis much information. Um, and so they're kind of like coolest is when our m how is a special compared to like an admin performing win or m right? and so that's that's what i. 18:03.47 Jamie Williams Yes. 18:13.88 Jamie Williams Um, yeah, and that's like in general I've said this a couple other times like as as much like you know as the public is exposed to like the backend of like evals like the you know results being released. 18:17.90 Jonathan Johnson That's what I find interesting as well. I'd say. 18:29.21 Jamie Williams I think 80% of the value is just the process because like you said it's it's a natural training exercise in terms of you know, even during the edr evaluations we did that exact same kind of experiment where we would execute something we told them exactly what we did like you know we dump credentials on this host in this method they would show us an alert and like oh here's an alert for that and we're like. 18:31.69 Jared Atkinson Um, yeah, um. 18:47.52 Jamie Williams You'd read the screen just like dumb user like that doesn't say it. You think it does like I read what you like thought this alert is showing in in terms of like the evidence and the tagging and stuff and like there was like a lot of fixes like that I think like Mdr that's going to be a really exciting one too because it's like there's so much I think. 18:58.16 Jared Atkinson The. 19:04.30 Jamie Williams You know you know if if I was like a user or like a you know a customer that would be my expectation is like hey I trust that all you're going to handle all of that for me and like you know I don't necessarily I mean I don't know how your customers work and I'm curious and if you kind of have insights here but like. 19:14.27 Jonathan Johnson In. 19:21.69 Jamie Williams Like I would assume like I'm not hands-on with like much of the data like you're basically just telling me what to do like you know and I'm basically putting all of my trust on you in terms of you know what's important. What do I do about it. You know I don't I'm assuming everything you're showing me is like you know some high-level confidence of like you know this is something that actually matters. 19:23.60 Jonathan Johnson Yeah, yeah Tracker yeah. 19:40.22 Jamie Williams Um, so I think like that that whole paradigm of like you know seeing it um is going to be really really stressed and really exaggerated. But I love the way that you know these these valuations are structured because it's not just like kind of putting you through that gauntlet. But you know before anything even hits like light of day in terms of results. 19:41.33 Jonathan Johnson Yeah. 19:58.72 Jamie Williams There's that like shot calibration that jar was mentioning like that hot wash of like okay like here's everything you sent us like you're not going to be able to change your results. But here's what actually happened um some of this you might have saw and like not cared some of this you might have saw and like you didn't know what you're looking at and some of this is like we can actually really start to understand like. 19:59.56 Jonathan Johnson Um, yeah. 20:11.77 Jonathan Johnson Yeah. 20:17.62 Jamie Williams You know here's 10 We did 10 things here's the 3 4 101115 whatever you chose to highlight and let's like really think about why? um and like said that is nothing to do with the results. But you know in terms of that bigger like you said before that competition. 20:27.38 Jonathan Johnson Yep. 20:34.50 Jonathan Johnson Yeah. 20:34.74 Jamie Williams That's that value that's being injected and like you know that hopefully is something that cascades. Not only back to that like that team but hopefully that organization that vendor and really like that's like 1 of the most interesting things about like you know Mdr is like the like I guess the exponential like impact. 20:39.51 Jonathan Johnson Yeah. 20:52.18 Jamie Williams Like you know if those lessons actually like you know, cement and materialize in that organization like how many users out there that are like in this like and real-world example of what we just did are maybe going to like take some extra value add from like you know, maybe maybe there's like some lesson learned like in terms of reporting format or structure or. Even just a feedback loop of like hey this works really? Well, we should do more of this like that I think is that big like jump like you know everyone looks at the results but like I'm a big nerd for like I go and look at like I wasn't involved with like the Ics evals. 21:23.33 Jonathan Johnson Me. 21:25.33 Jamie Williams Like going and looking at what they did and like oh cool like these behaviors are really awesome like now I know like 6 or whatever vendors have experienced this like that's awesome like the fact that they are now aware of like how their tools align with this like let's see where that goes that's ah you know something that I don't think is measurable. But. 21:32.39 Jonathan Johnson Um, yeah. 21:45.12 Jamie Williams You know is something that I think is an industry we should maybe maybe appreciate and kind of push more towards in in multiple different I guess um efforts and projects. 21:52.36 Jonathan Johnson Yeah, the the the goal of Mdr I see is like to basically weed out and funnel the bullshit is kind of how I see it is like um you get like these alerts that come through. They might be false positive. It's now the Mdr's job to like kind of sift through the weeds of it and then like. Whatever is exposed to the end customer. It's like okay this is this is bad or at least like we have pretty high confidence high level confidence that this is this is probably bad. Um, but so like that makes me think was there. Ah. 22:16.93 Jared Atkinson High level of confidence. Yeah. 22:30.62 Jonathan Johnson Kind of that kind of transitions into like the false positive false negative kind of conversation. So no okay I'll wait then I'll wait. So let me forget it. Yeah is this is this like a 40 thread tweet or type of length or. 22:34.56 Jared Atkinson Don't go too far I have a still have another question on the Mdr you know, okay, mine's kind of like mine's kind of lengthy so I don't want to like yeah, it's like a 40 thread tweet. 22:34.93 Jamie Williams I Had a question too following up. But yeah. 22:49.21 Jared Atkinson Okay, well like okay so Jamie you mentioned that the edr eval is more kind of focused I know this is generally speaking but more focused on the visibility as opposed to I forget what you said for the mdr eval but is more focused on like the how do you process and handle things it sounded like so it's more procedurally focused. Um. 22:50.00 Jonathan Johnson Yeah. 23:07.71 Jared Atkinson Not procedures in the context of Ttp but in the context of the process that you're using the? ah yeah I hate that the um, one of the things. That's oh that's started to strike me recently is that when we think about red teams red teams have a place but I think I think we have an issue to where. 23:10.69 Jamie Williams Kind of alone with collisions. 23:26.25 Jared Atkinson We're asking red teams to do too much right? And that's kind of I think that's where like something like a purple team came from is that we're trying to solve or answer questions ah using red teams but red teams are actually focused at too high of a level so it's like focused at the whole process. The whole attack chain as opposed to focused at like an atomic segment of the attack chain. Um. 23:41.12 Jamie Williams Um, yeah. 23:46.20 Jared Atkinson And one of the things that I as I'm looking into this may not be that everybody's definition of purple teaming but purple teaming is kind of the squishy thing that kind of is like a catchall.. That's what I like the definition is something like well you get red and blue to work together to make things better over time or something like that's I'm being hyperbolic but that's actually. Not that far off from what I see people say purple teams are but what I think it ultimately comes down to is we want to more granularly focus on specific components of an attack so say um, you run you like you pick like one of the things I'm interested in is picking a certain technique trying to evaluate all the different iterations of it and then evaluate. How Well we detect that and that's actually your edvals kind of include I don't know if that was your intention but that's what people are trying to glean from that in some cases. Um, and I think that's that's interesting because from a visibility perspective um visibility is automated right? And so like when we talk about being collaborative. 24:32.80 Jamie Williams Me. 24:44.80 Jared Atkinson Visibility. It's like you either saw it or you didn't and like the fact that I tell ah the edr vendor that I'm about to do this that that doesn't really like that doesn't change whether they're going to see it or not to some degree right now they like they do have some time to do adjust their detections and things like that. But when we're talking about processes. Ah, very often processes involve a lot of manual features and so like ah I was talking about air force exercises previously and I'm going to talk about it again from a different context. 1 of the things that was always interesting is during an air force exercise. You knew you were being attacked right. And so like None of the big problems with things like alert fatigue is that over time your ah level of attention dwindles right? because you don't know when the attack's going to be and everything you ever see is is not an attack and so you start to not you kind of like don't care as much even if it's not conscious, but when you go to like. An eval or ah, a training exercise that has a red team. You're like well shit we know that they're going to be attacking us in the next like they may not attack. There were days that they didn't attack us during a day but like we were on high alert the whole time because they probably should have. 25:43.50 Jamie Williams Yeah. 25:55.12 Jared Atkinson And those were actually the worst days because we were expecting something and so we just felt like idiots the whole time because we never saw anything but then they come in and they're like yeah we didn't do anything today like ah, but um, so one of the one of the issues that I see. I like I don't know if I'm asking whether or not you considered this or if maybe it's just a talking point potentially but None of the issues that I see with when you get beyond the visibility or like the binary question of did you produce an alert or not, you're you're starting to have a situation to where it's almost like Schrodinger's test to where. Like the knowledge that the test is occurring actually changes the behavior of the participants in the test and so I'm curious. Um, well a what you think about that whether that's a good proposition or not because that actually affects how we potentially will propose like purple team engagements to customers because. 26:27.41 Jamie Williams Um, yeah. 26:44.87 Jared Atkinson Like there's some sort of collaboration that ah that occurs and the collaboration actually could hinder the outcome or change change the outcome whether it hindered it or not I'm just curious if that's a something that you agree with that kind of proposition and b um, whether or not that's something that was considered or you would foresee being something that would affect the like Mdr. 26:50.24 Jamie Williams Um, yeah. 27:02.68 Jamie Williams Yeah, no, you're you're spot on and before I go any further um I was going to jellybe and jar with that I think that was like about 60 tweets maybe ah more or less. But um. 27:04.30 Jared Atkinson Eval I might be with the mdr eval completely incorrectly. 27:11.80 Jared Atkinson Okay, okay. 27:16.44 Jamie Williams Yeah, exactly that point and like I've always like anytime someone's asked me like what I think about the evaluation results like MdRER or whatever I've always said I think I've said it today like it's very much exactly to your point like the knowledge that we're in an assessment kind of puts the contextualize this towards we might be measuring the ceiling. 27:20.93 Jared Atkinson Sure yeah. 27:33.44 Jared Atkinson Um, yeah, okay, that's a good way to look at it. 27:35.18 Jamie Williams We might be measuring like best case scenario. So like yeah, so like you know visibility wise like even looking like at the edr results. It's like hey how many times are we going to actually have someone who understands how to like configure like api hooking and all these like low level stuff like probably not maybe not certain environments but like the mere knowledge that that's possible. Is valuable same thing with Mdr like you know, obviously there's going to be a little bit more of like the black box text testing because like it can't be a true purple team like hey we did this and obviously there's really no point in doing like kind of service process. It. But that said like you know seeing. 28:02.74 Jared Atkinson Sure. 28:10.73 Jamie Williams Being able to respect like okay side-by-side like this is exactly what the red team did and here's how that information was conveyed to me like you know it might not happen that doesn't translate like 1 to 1 to the real-world but also like that does give you a little bit more appreciation of not necessarily just measuring like the output. 28:20.51 Jared Atkinson Um, yeah. 28:28.13 Jamie Williams But like appreciating the process and seeing like you know you know 1 to None like okay like here's how much did they kind of align with what happened did they give me like more information less information and like most importantly, like how much you know exactly to Johnny's point earlier like as a user how much of this what I really actually need. Like you know what? and what would that and actually look like for me to actually do my job and I think that is kind of that value which is again another like you know yeat this away from a rank or a score. It's like such a fuzzy gray like thing but it's like something of value. It's just kind of hard to like. 28:56.51 Jared Atkinson Um, yeah, yeah. 29:06.26 Jamie Williams Quantify in any like reasonable way at least for now. 29:07.37 Jared Atkinson I Wonder how much the like the Median performance or even the the floor of performance correlates to the to the ceiling of performance. You know what I mean because like if let's say that we do accept the proposition that we're measuring the the ceiling because it's a kind of like open-ended. 29:16.89 Jamie Williams Um, yeah. 29:26.17 Jared Atkinson Or like a collaborative. It's more collaborative than not. We'll say um, do the do the companies that perform better along the line along the metrics that I care about should I Assume that if that's their ceiling that the the same order of companies I Guess along that metric would. Would remain at this at the floor or at the Median does that make sense. 29:46.50 Jamie Williams Yeah, and I think that's that's why like the like procurement process is like such a tricky thing like I honestly have no idea and like that's why I've like also big philosophy is like evalws is probably like the concept of e-valws whether they're nots like evals directly or like re like reproducing that. 29:53.35 Jared Atkinson Um, yeah. 30:04.47 Jared Atkinson Um, yeah, sure sure. 30:06.54 Jamie Williams It's probably just like one hop along that process of like actually like not only like finding a vendor but like you know I imagine this was a question I had for Johnny is like how often do customers kind of like prescribe things to like back to the service of like hey here's how I want things done here's my philosophy versus like. 30:18.57 Jared Atkinson Oh. 30:23.89 Jamie Williams Like are you coming in because that could 100% skew that where like you know you as a service or like have your hands tied because it's like hey like they don't want this. They don't want that they have these restrictions so I'd love like there's so many realw world and like cost is another one like there's so many like real world implications that I think like you said, kind of skews that but that said like. It's exactly to your point earlier about like the you know eventually, there's the perfect answer like like that's great like I love that. But like I think a lot of the questions that we struggle with is like what do I do in the meantime like there's so like what do I do tomorrow or what do I do next week like what? what should my 1 year goal be and like I think we always kind of say this from the attack perspective is like. 30:48.61 Jared Atkinson Um, it doesn't matter. Yeah. 31:02.92 Jamie Williams You know, let's measure where we like something like attack helps us see where we are now. But most importantly, kind of sees like like what is our ideal state look like and what are those gaps like what is everything in between like you know at least for now what I can kind of conceptualize as a good you know more reasonable solution. 31:05.73 Jared Atkinson Um, yep. 31:21.61 Jamie Williams Like at least I can somewhat kind of start to ask the right questions in terms of like where am I on that roadmap. 31:27.60 Jared Atkinson Yeah I think there's almost like an evolutionarily like if you consider the development of edr along the lines of like evolution you you could start to see that something like attack evals is almost like a selective pressure. Um, like you could see things like sysmon came out with process access events. 31:39.98 Jamie Williams And yeah. 31:46.75 Jared Atkinson Right? And that like that was explicitly to dissect Mimi cats right? and that like that was why it came out um and at the time like there was nobody in the nobody in their right mind was thinking you know would be an awesome data sources process access events and now that's like a predominant. Ah. Data source that that everybody has everybody has in their edr. It's almost like you must be this tall. The ride the ride and that that height includes process access and so it's it's almost like the Evowels. Ah there's going to be so an evallu thanks Luke. Yeah, thanks! Thanks for that. Um, that's good. It's actually pretty good. 32:21.90 Jamie Williams That's 10 out of 10 my book. That's awesome. 32:21.56 Jared Atkinson Yeah, yeah, okay, so ah, Luke you might want to um, do a pregnancy test because when I started making dad jokes like that I found out my wife was pregnant so there's you be careful the um I don't know if those's a correlation or causation. But the um god I lost my train of thought. So. But the the idea that there's going like with each iteration of the evals there's going to be like some feature that kind of gets a lot of attention and that feature like the vendors who don't have that feature will end up prioritizing that I think. 32:51.86 Jamie Williams We actually did that like there was there was definitely a maturity model in terms of like the None round was like basically like if you see a process can you tell me what that is None round was like power show and like a lot of like script I think to your point like. 33:03.85 Jared Atkinson And yeah. 33:06.84 Jamie Williams Around that same time and this is like back when like Powershell was all the hotness and like all that apc 29 stuff came out like we saw sysmon introduce like the wmi like events and like file deletion and like how much of that was probably like in relation to exactly as you said like you know Mimi cats to process access versus like you know esteete and things like that and all. Like w my event subscription relative to like what we saw like sysmon and like going forward like I think like each round like internally and that's something we like you know spend probably way too much time on is like what do we see and you know getting back to I think our previous conversation. We talked about this like what are we seeing in threat intel in terms of like. Not necessarily full coverage but like there's probably things we're seeing in trends and like from real actors in terms of like it might not be the most complex thing but like what's the problem we're seeing over and over and over again that we don't think we have like a you know as an industry. 33:49.87 Jared Atkinson Transep. 33:59.50 Jared Atkinson Adequate solution. 34:01.31 Jamie Williams A solid solution for or just like maybe a lot of people aren't taking this as serious as maybe it is so like a good example was we did ah data encrypted for impact ground which was like ransonware in wipers like not these actors aren't really displaying a lot of like really really advanced tradecraft. But that said like the evaluation or whatever like the evolution needed was like okay like it's not necessarily super complex but it is worth our time to like bring this community's attention to this just for the sake of like hey like how many times have you sat and watched like a wiper run in front of your tool. 34:30.12 Jared Atkinson Um, yep. 34:36.77 Jonathan Johnson Yeah. 34:38.25 Jamie Williams Like let's just see what that looks like and make sure like everything that could be done is actually still and you know what you what that ideal that we talked about like whatever your ideal state. Let's make sure that's real or at least try to calibrate towards like fixing whatever might need be. 34:52.17 Jonathan Johnson Yeah threat intel also is heavily dependent upon what visibility they have based off the edr that they have access to as well right? So yeah, so it's like I find interesting. It's like ah you know Intel will be like hey we're seeing this in the wild. 35:01.38 Jamie Williams I was super biased. Yeah. 35:11.63 Jonathan Johnson Um, like my next question is like what are what else are we not seeing and I think like that ah that kind of makes me think of like the conversation I think we might have had this the first part but it's basically like what is the base goal of edrs right? So like you have some edrs um that have kind of like stuck in the ruts of like we're going to do. 35:13.26 Jamie Williams Um, yeah. 35:30.00 Jonathan Johnson Going to collect this type of telemetry very very well so like um like say like process based stuff so process creation termination open access things like that. But then you start to see Edr start to expand their capabilities into like apply more advanced telemetry. Which is going to be like things like maybe like file share access or like um maybe like specific api ah calls etc. Um, and so I think like evolving what telemetry is giving us is also going to evolve what visibility we have in terms of like seeing what's out there in the wild. I think the next question is like how do we leverage if we don't know about something. How do we leverage this home entry to now know about that thing. Um. 36:12.71 Jamie Williams I think I'm not a gardener analyst but I think to your point like there is probably no so true single like objective of edr where like some of them act like exactly you said like I'm just going to put a bunch of really cool telemetry and fun analyst and let them do what they do and there's other ones that probably act closer to like an Mdr where it's like hey. 36:26.77 Jared Atkinson Oh. 36:29.85 Jonathan Johnson Um, yeah. 36:32.70 Jamie Williams We're sole like low false positive. You know we'll do response. It's basically ideally at least in their like you know approach like plug and play like put this out there. It's better than where you were and let the vendor kind of we'll do all the analytics. We'll do all the work we might miss stuff but you know we're not going to let you get completely hosed. 36:40.60 Jonathan Johnson Um, yeah. 36:52.28 Jamie Williams Um, so I think that and that's something like evows doesn't index but you know like you said it's kind of you know I don't it's probably those 3 like you know 3 plus pipelines towards what are the best approach have no idea but let's kind of see how each stacks and and courting towards like does this approach make sense relative to. 37:00.71 Jonathan Johnson Um, yeah. 37:10.71 Jamie Williams Certain things maybe certain other places it doesn't but you know at least we can do is try to enumerate understand and improve that. 37:15.39 Jared Atkinson Okay, you just made me think of like um, you know and fi like fifo or any sports video games you you create your player and you get so many points that you can assign to different attributes. So I think the thing is is like what Edr is is ah. 37:24.76 Jamie Williams Um, yeah. 37:29.40 Jonathan Johnson You can do that in other games too I know you don't play other games share but you can do those things like Sky Ram doesn't have to be a sports game. Okay. 37:33.30 Jamie Williams Um. 37:33.78 Jared Atkinson I I don't I don't care about any of those games. So the um rocket league Fifa and then I just download the new ninja there's there's a new ninja turtles game shredder's revenge in case, anybody wants to play that on playstation xbox pc anyway there? Okay so edr. 37:39.52 Jamie Williams Speed Awareness like. 37:52.30 Jared Atkinson I think the way that it works is they have a number of attributes that are common amongst all Edr and you have to have some level of attributes. But I I think of like when you watch when you play like soccer video games or you look at like analysis of soccer I'm sure this is true of every other sport. There's these things called radar charts which is like it's like a circle. Yeah okay and it has. 38:07.38 Jamie Williams Um, yeah. 38:11.42 Jared Atkinson Has the different attributes and then it basically shows like this player has a lot of speed. But maybe they don't have very good defending or whatever or this player has like great shot and passing but they're not very fast or I don't know whatever whatever it is and so I think that's what what happens with edrs or probably all of these types of categories is they choose to. Ah, some choose to put all their points in a certain category and others choose to put all their points in another category or some choose to kind of spread them out evenly and the question is is like you know what? what player fits best into your team I Guess you know like are you trying to have a fast counterattacking team or do you want somebody that has a ball possession. 38:43.42 Jamie Williams Um, yeah. 38:50.55 Jared Atkinson Kind of control team or you know, whatever and I think that's that's the hard question but you're you're kind of giving us the radar the radar graph and then our job is to choose. Do we want you know Paul Pagba or do we want Tony Crus and in midfield or whatever I don't know I don't know if those names mean anything to you guys. But. 39:06.60 Jamie Williams I think the other yeah what french and ah Jere maybe I think the other thing that often gets lost to is like we always like like I guess I always think edr is like this concept and this like capability. 39:09.67 Jared Atkinson German Yeah close enough. F. 39:22.59 Jamie Williams But like really under the hood. These are still like I mean they're security-minded companies but they're also just like software companies. So there's like bugs. There's thing like you know in the spirit of any software development. There's like principles. There's optimizations. There's all these like decisions. So I think one of the interesting things is like also teasing out like. 39:26.50 Jared Atkinson Yeah. 39:40.88 Jamie Williams Those code design and like general architectural decisions relative towards like their intended function like hey like you know in terms of storage and like cpus and like all these other like considerations like what is the impact that has on terms of like actually doing what it's designed to do so they're like definitely levels of like. You know, different points which you know to your point I don't know I wish like I wish we could just write write like a huge thesis on like everything we learned about every single vendor but that said like you know would that I don't even know that be of any value but like all of that is captured somewhere. It's just. 40:08.67 Jared Atkinson Yeah. 40:18.31 Jamie Williams You know I think that's why in in general, it's so important that you know whether it's evaluations or other do this internally do this with you know on the weekend that we keep you know I think like Johnny said earlier like that competition because like you know one of the one of the big catalysts for this evaluation process was like. Sitting you know people at might are sitting down and seeing all these reports of like you know this fit this product can do this this offer can do that and it's like what's the what's the data behind it like what's the proof and like has that actually been you know even if there is like you know these are all facts like. Let's take that and put it in an inverse environment and see what we learn and kind of like you know in in this you know and the analogy of like evolution like take a bird from like you know South America and put it in like you know Arctic. It's going to like potentially learn some stuff. There might be some evolution. Yeah. 41:02.85 Jared Atkinson Yeah, it's either going to die or it's going to adapt right? Yeah yeah. 41:07.30 Jamie Williams It's going to get better or it's going to at least change and kind of Morph and like you said like collectively that feedback is probably what pushes it more to like being more robust and strong and actually getting the job done. 41:17.77 Jared Atkinson Oh man. Yeah there's this um ad I want to go that direction. That's so going back to your question that you were asking Johnny I know you you kind of were asking Johnny but I think that I see that ah the question was about like how often do you see customers kind of like demanding features to be added ah to to? Mdr. And like I've never worked for an mdr but I know a lot of people that have and this is I won't say who the company is but they I've heard that that's actually like None of the biggest hindrances possible is a lot of companies and it's actually biased towards the bigger the company the more this problem actually exists I think um. 41:51.87 Jamie Williams Um. 41:56.21 Jared Atkinson There's a lot of companies that the minimum like they might be holding a $10000000 contract over your head and they're like in order for us to sign on you have to integrates the data from like this stupid thing that we bought five years ago and we don't want to admit that we made a mistake buying it and that the Mdr companies like yeah, we could do that. And that's going to take time because we have finite number of cycles for integration and things like that but like it actually is just not valuable right? like it is possible that in the context especially in the context of Mdr it's possible that things just aren't valuable altogether in any context, but especially once you scope down the context of the job that the Mdr is trying to perform. There are probably security tools that people have bought that that aren't that like valuable from from that perspective and so there's all this instead of um I would say that sometimes sometimes companies I think this is true of Edr companies. This is true of Mdr companies and it's the same problem. It's just a different location in the in the platform. But like what happens is you have experts at the company that are trying to build out a like path forward or kind of like ah a wish list of features and then you also have customers that are coming in and just saying like I don't care what you think is best I want that I want you to integrate this thing that we have and if you don't we're not, we're going to go to somebody else who will. And I think that that um to some degree hinders. The performance of edrs or maybe the adaptation of ed or mdrs and edrs over time to trying to solve the problem better because they're they're not sometimes they're not trying to solve the actual technical problem they're trying to solve the problem of customer retention. 43:31.65 Jamie Williams Um, and yeah. 43:32.41 Jonathan Johnson Yeah, so I don't know all the answers because I'm like I'm not Keith we came out, you know the you know or Brian Byer the you know the top dogs I'm just ah well I'm just me and so. 43:33.39 Jared Atkinson And those are those are antagonistic. 43:49.53 Jonathan Johnson what I will say is I do this is what I do see I do see um requests. So like if I like I would say I don't know a couple months ago I jumped on a call with a customer and they're like we're talking about some stuff like hey by the way before you go, we let you know like we ran this 1 thing to get information for um, a red team. Um, we got nothing from you guys. So is that like a lack of ah like a detection engineer pushing that to us or is that a detection issue and so like going back and like checking what those issues may be um, like that's good from the customer because that holds us accountable right um. Another thing I do see is like yes sometimes we might like be in talks with the customer and they're like do do you like support x thing um from what I understand at least from you know this is not an official right? Canary statement. We do hold a standard right? So it's like they have to meet these things. And collect this type of telemetry in order for us to even think about integrating them and if they pass that then it goes into another check and then and I think there's a None or 3 or 4 checks before we even like say okay thumbs up. We're going to integrate it in. Um, if they don't we just basically say like sorry like toss it to the wayside. 44:49.62 Jared Atkinson Um, yeah. 45:06.88 Jonathan Johnson Um, so there's that 1 thing that's interesting from an Mdr perspective you know because like previously coming from spectroops where I was like writing explicit detections for like 1 organization at a time so can I please move on to the false positive false negative conversation I've been wanting to like. 45:21.26 Jared Atkinson Sure, yeah, yeah, you can I think I think yeah, you've tried to go into it like 4 times now. So now you got the green light though. 45:22.91 Jamie Williams Um. 45:24.66 Jonathan Johnson Thirty forty comments yeah Yeah okay cool. Cool, cool, cool, okay, cool. So like when it's interesting there because um I find like the question is like. What says one detection is better than another detection from one mdr to another um and in my head I think it really boils down to as to how that Mdr is in terms of lean false positive sensitive or false and negative sensitive. Um, because like that is ah like. Going to like subject them into how deep they can go into an abstraction map for that specific technique. Um, ideally it's good for mdrs to be I would probably say less false positive sensitive because I think that's kind of the like model. Of the mdr is like hey we're going to take that bullshit for you so that like we have really good analysts that will be able to take the false positive stuff see that and whenever we toss into you. It's high confidence whereas like. From an organizational standpoint one by 1 they don't have that luxury of having something between the edr and them to basically funnel the bullshit I just probably should just see a false positives but like all the weird shit that happens in a day or None um, and that's where I think like Mdr comes in but then it's interesting from like. 46:36.15 Jamie Williams If. 46:50.44 Jonathan Johnson Looking at one mdr versus another.. It's like okay we detected it this way. How does you detect it and why would probably never get those answers I maybe s mighter will a bit like we probably won't um so I find that interesting because it's like that then comes down into how well we understand the technology How Well we understand the telemetry that. Portrays that technology then also how well we understand how that's being performed in the world because I think like the causation to action has an equal amount of importance to just the action itself. So How do they get to the thing that they did and why did they want to perform that thing next. 47:26.41 Jamie Williams That would have been like the most baller response to that like customer response like request you had was like yeah we identified it was a red team. So like yeah like we can show you the telemetry if you want but like yeah, we just kind of teamed it out because we figured it out. But yeah to your point. Um ever since like the None time we touched evaluations. We've like. 47:37.21 Jonathan Johnson Um, the end. 47:45.70 Jamie Williams Gotten feedback and noticed ourselves that like false positives is like the next big you know, kind of you know, pasture to explore I Honestly have no idea how to evaluate that besides just like putting a bunch of sensors on a box and doing nothing and just seeing what fires. But that said it's like even there's like a problematic. 47:57.95 Jonathan Johnson Um, yeah. 48:05.30 Jamie Williams Like you know there's that that data itself is going to be tainted in terms of like not necessarily directly actionable. So like we've you knowve I've seen some things and maybe like you know we've kind of whiteboarded some ideas I honestly have no idea what a reasonable approach to asking that. But at the same time. I think that kind of gets back to that like measuring the ceiling approach of like you know the analogy I was thinking of is like you know you read about a car and it's like none and like you know 3 seconds there's realism that like hey like I can't do that like I'm a good shifter like I would you know perhaps even drive the car into the wall like false positive like huge failure. But. 48:31.81 Jonathan Johnson Yep. 48:42.34 Jamie Williams That said like I don't the best I can do is try to act on and like at least strive for that and like obviously there's going to be a bunch of roadblocks. But um, yeah I I would love and that's something we've kind of like had ah almost like a permanent like feeler out where like if someone like. 48:57.50 Jonathan Johnson Um, in a. 49:01.40 Jamie Williams Evaluation like miters doing evaluations like we're doing them at the same time like there's an open door or like if you have an idea for like what in a false positive evaluation would look like like I think there's definitely a medium for like hey let's let's roll with it I Think the only Blocker would be just like getting people to sign up for it because like imagine like. 49:17.30 Jonathan Johnson Um, yeah. 49:20.00 Jamie Williams An Ed like from an edr perspective and or or an mdr like I can't imagine like just from a business perspective signing up for that because it's just like this is not going to look good like we might benefit from this from the outward perspective like this is going to highlight maybe some problems that. 49:27.35 Jonathan Johnson Um, yeah. 49:38.20 Jamie Williams You know, hopefully we can fix but the general like keep the lights on part of the company is not really going to appreciate our participation. 49:44.63 Jonathan Johnson Yeah, that was going to be my next question was like how you guys are like addressing the false positive thing because obviously if someone knows the evaluation is happening. They can kind of tune the game right? and so it's like we know from like this these boxes are these things. Everything's bad. 49:44.74 Jared Atkinson Um, I think what. 50:04.18 Jonathan Johnson So like it kind of takes away some of like that thinking process from the analyst of like is this good is this false positive is this like not or do we only have to worry about like tagging that to a specific thing etc like um, did how did you guys like want to address that or is there really a way to address a sense like. 50:17.83 Jamie Williams Sir. Did different approaches. So for Edr. Um, it's it's problematic like you said because basically our process was we told you what happened? Let's find what was relevant and even during like our purple team exercise like we downselected like the vendor would show us something or like oh no, no, no, that's a false positive and like. 50:23.38 Jonathan Johnson Automated user simulation I guess. 50:41.31 Jonathan Johnson Now. 50:41.77 Jamie Williams It never saw the light of day so it was very much like measuring the ceiling that said as like an astute user if you're curious like that's why we include the screenshots like you can go and look and kind of see in the background like hey they're showing this alert but you could also probably see the alertqueue in the background so you could probably piece it together. But it's not something we necessarily indexed. Mdr is a little trickier. Um, so like procedurally like the actual valuation is black box like you have no idea what actually happened when it happened where it happened like that shot calibration comes after so the results will definitely probably have like false positive like hey like you know Mdr service said this. Didn't actually happen. Um, but that said like that even that I don't think is to your point like a perfect solution like there's still I think some meat on the bone there but um, things that that's that's something too is like you know it kind of goes back to the core of like evaluations like is. 51:28.20 Jonathan Johnson Um, yeah. 51:32.12 Jared Atkinson Um, I actually think during the. 51:38.36 Jamie Williams Are we really trying to measure false positives or is the other side of just ignoring all that and kind of taking the wins is that more beneficial. But Jerry sorry you were saying something. 51:46.17 Jared Atkinson Ah, as I was going to say I think that the during an mdr eval you probably would have you would probably expect that the number of false positives is higher than the average number of false positives right? because they're on high alert so they're going to be they like if you write. 52:00.63 Jamie Williams Um, yeah, yeah. 52:03.58 Jared Atkinson If you're expecting badness to happen then you're going to classify more things as badness than otherwise right? So I I think like ah None of the cool things about the mdr eval is that like treating it like a black box is exactly how a customer expects it to be treated right? And so um. I think you do actually get a pretty good representation of like the false positives and false negatives from that perspective because like they either tell you about it or they didn't and everything else is just abstracted away behind this opaque kind of opaque interaction. Um, there is like so um. 52:34.20 Jamie Williams Yeah. 52:41.88 Jared Atkinson Yeah I don't I don't know I get I always get so ah annoyed by the false positive false false negative conversation because like I I there's this thing of like of course we should. There's a measurement of how false positive prone is one edr versus the other or 1 detection from another. Um, but there's. Also the problem of like false negatives and I don't think that any of us are answering that problem. Well either. Um and so like and to tell like from a technical perspective from like a literal ah literal perspective the way that you distinguish between None detections that have the same goal. Is you. Measure the number of false positives that that they produce and the number of false negatives that they produce and those combined are error and then you would look at their error proneness and like 1 is going to have more absolute error than the other and the one that has the most absolute error is technically worse right there there is. Of course like that's a sliding scale because you may prefer like we talk you and I have talked about this at least but like in the criminal justice system. We prefer false false negatives over false positives right? A false positive would be somebody that is innocent getting locked up and that's like in the criminal justice system is the the worst thing possible right? so um so like in certain certain scenarios you would prefer one over the other but like in an absolute term ah in absolute terms where you don't consider the context of where it's being used the better one is the one that has less error and then you could you could adjust based on that. Um. 54:06.34 Jamie Williams Yeah, but in in that like bigger context of like error measurements. Also it's like to your other point about like paths is like. Can you do that type of Calculus without having done like a full like a more broader like assessment like. Especially for evaluations. It's like there's a very much a linear path where like we're testing specific procedures and everything is kind of like contextualized around like this very very you know, kind of you know, scoped implementation of like a certain subset of Techniques. So I think that's like the other reason why like you know people in general like you know. You know coverage is such a problematic thing especially like in the context of attack because like people really want attack or just in general something to tell me like you know some binary or even just like quantitative 0 to 10 metric that lets you know like you know like you know, pretty number I know where I am but like there's just so much like. 54:59.57 Jared Atkinson Yep. 55:02.90 Jamie Williams It's just so much like data lost from just those like very very like I understand why people do that because it's like easy to work with.. It's kind of like easy to quantify and use these fun things with but there's just like so much like of a telephone game in terms of like what actually happened. And I think that kind of like loops back to like bite us in the ass and we like make we work on these assumptions and later On. We're like why didn't the thing work and you're like well because you took like a very very complex idea like a Cv and put a number on it. 55:22.39 Jared Atkinson Um, yeah. Yeah I think that's ah like what I'm what I'm this may be ah me being very optimistic but what I'm working on like the Twitter thread that I released today about rank ordering paths. That's like step None and probably a 10 step process of try or maybe more. Of trying to actually build a quantitative score of coverage right? So the idea is is that there's there's numerous categories of variables in how is this okay to go into Johnny or did you have more that because this this could go. Okay, um. 55:59.55 Jonathan Johnson Go for it Chief you know me man power if I if I think of something I'm just going to go back like I let us run it back. 56:05.71 Jared Atkinson Okay, that's fine. Yeah, so okay, so the idea is is that um, attackers only care about the the outcome right? So like they for instance, they want to and this is maybe not even this is not even there's multiple layers of outcomes right? So they want to steal intellectual property. but then if you focus at the technical layer it's like in order to steal intellectual property they need to get access to a machine which means that they need to have certain credentials right? And so like maybe None of the things that they want to do is access credentials and then so then they choose to do like Mimi Catt style credential dumping. Well, there's. A number of different paths that they can choose or variations of achieving that behavior that they can choose right and so what I'm what I'm trying there and then there's a bunch of different layers of variability. So for instance, there's variables. The thing that I'm working on now which i'm. i' kind of talking to the audience because you both know this but feel free to chime in whenever you want. But um, one of the layers of variability is the functions that they choose to call right? and so like. For instance, we see mimicats calls a function called read process memory and there's a tool called dumpert produced by outflankk in the netherlands and they call the cyscall directly. And so the idea is is that Mimecats calls read process memory which then calls another read process memory which then calls another read process memory which calls int read virtual memory which then calls the syscall and al flank was like well let's just skip all that stuff up there because there are there are or at least have been some edr vendors where their visibility is actually. Predicated on read process memory not on the the syscall right? And so outflankk would evade those those kind of naive edr approaches right? and and so but then there's like you start to map it out. It's like what are then the question is is okay well if outflank found. A different way to ah, functionally execute this behavior. What are all all of the ways that you could functionally execute this and that's where you build out this graph that I call the function call graph. So that's one layer of analysis. There's also for instance, um, the language that you use right? and so there's. There is ah telemetry or visibility into powershell code execution. That's not available for like c plus plus code right? And so the the language that you choose to execute from may be an important variable and there's also like parameter information which is like when we look at like open process for instance which is that. Precursor or like a prerequisite to actually reading memory for like the mimicat style password dumping um you could choose the granted access numbers right? So there's like a minimum set but then you could go above and beyond the minimum set. However, you want and that's a variable that you could start to do to where if people are taking a naive approach to that. 58:51.20 Jared Atkinson You could change it up just enough to kind of evade them. Um, and so and then there's probably more there's way more right? The name of the process the but there's a byte level layer of analysis to where you could just change bytes to make it different at the byte layer and one of the questions is is. Okay, well there's all these different levels of variability or all these categories of variability you could say which but like they probably don't all have the same significance right? And so ah, my assertion right now that I don't know that this is actually that well backed up to be honest, but it just anecdotally seems correct. This is like. Foundational assertion of all the stuff that I'm working on right now and it's probably like the the thing that is least concrete in my mind but None of the fundamental assertions is that the functional level is one of the ah most significant levels of variability right? more significant than say the bytes being changed. Meaning that like um if you change the function that you call that's going to have a bigger impact on somebody's ability to detect it than ah if you change an individual bite. For instance I think that's probably true I don't know if like I could produce a mathematical proof for that. But I think I think that's probably a decent assumption. Okay, so so there's there's this weird thing of like um, okay, well we could start to evaluate at least at the functional level all of the different paths that are possible to achieve some behavior and then we can start to kind of predict and ah model or represent those those paths and so that's. I don't know exactly why I started going down this path to be this this course of dialogue. Yeah. 01:00:24.54 Jamie Williams I Think it has a lot of like relative like you know Contexts like you said it's like there are some instances where adversaries or defenders make those like explicit decisions like dumpper is a good example. But I think it's also like a really good measuring stick in terms of like hey like I didn't know I was unaware of like. 01:00:34.38 Jared Atkinson Um, yeah. 01:00:43.28 Jamie Williams The decisions I made and like you know hey I'm living in this process access vein versus like I didn't even think about how that's you know my ah defensive approach was relative to the rest of this paint So like I I think it's It's a really good foundation for like you know and cost is another one. It's like hey like you know was my. 01:00:51.45 Jared Atkinson Um, yup. 01:01:00.51 Jamie Williams You know here's where I am in terms of this bigger like map and this bigger like kind of you know enumeration of the problem like here's where I am why is that was that like a naive decision or was it a cost like you know I'd love to be over here. But you know here's the considerations and why I landed I think it's really good. Kind of. 01:01:12.21 Jared Atkinson Yeah. 01:01:18.75 Jamie Williams I guess catalyst for a lot of like decision that you know either cement you know? Okay, this is you know this is the reality of you know their situation or you know hey I'm already over like hey we're already doing like api hooking or function hooking. We just hooked the wrong one like here's a better path here's a stronger path for threat Intel reasons or whatever it is. 01:01:31.51 Jared Atkinson Um, yeah, yeah, well, there's like ah there's there's an issue to where um that like John Lambert has that famous quote where he talks about famous famous at least with an info sec I guess or Microsoft fan voice I suppose but. 01:01:38.63 Jamie Williams Um. 01:01:49.95 Jared Atkinson Ah, where he says like attackers thinking grasss and defenders thinking lists as long as this happens attackers will win I think like okay so there's this this thing when dumper. So like you said dumper the people have made dumper they they thought about this explicitly right? So they chose to do that on purpose for the like specific purpose of evading detection. That's why they did that um and the interesting thing is that they made that decision because they innately understood the technology enough to wear the graph the graph that I made explicitly was in their head already. They just hadn't propositionalized it and onto paper to where like you could see it but they they saw it in their mind right? and so they. They went through the graph and they're like okay well here's all the different ways that we could do this. Let's choose the one that we think is like ah the furthest down in the in the graph like the the terminal point or whatever the terminal Node. Ah. 01:02:38.62 Jonathan Johnson Um, I just want the users to know. Also while you're talking about syscalls that cyscos are a lot easier to call in c than it is like Powershell been there. 01:02:46.71 Jared Atkinson Our show. Ah, not that you're trying to do that the um and so they made that explicitly or like we see um, kind of like the the initial process so we talked about sysmon came out with process access to detect Mimi cats and one of the things that we saw. Ah, originally was people would say say things like look for a ah, granted access rights of none but like the people that were making that recommendation at the time literally didn't know what none was signified right? They just knew that that's what they had observed like being used when mimecats was run. But like what they didn't know is that that's that's a bit field and so that means that you could turn on like you had to have those bits set but you could turn on any additional bits that you wanted and it like but then people were like okay well it needs to be like greater than none and it's like well that's not actually how bitfields work because there's. There's are there's values that are greater than none that don't actually include none and it's it's all this hot mess and so there's all all these things to where a bunch of people have hardcoded and this like have hardcoded looking for none but all the attacker has to do is like request a right that they don't actually intend to use and now they've bypass your detection right? So there's there's this. Or like just to go back just to piss the people off that ah I'm thinking about you Jonathan Ke him when I do this, he gave me feet. He's like dude I wish sometimes you could talk about something besides services. So now I got to fit services into everything the um like when you go back to services the um I don't even remember what l oh okay, so there's. Ah there's events that will say things like a service was created but like we assume that that means that if a service is created then you know the event will fire. But that's not actually a good assumption because there are ways to create a service that don't fire that event and so there's this thing to where. Ah, people will say things that is that we will take to assume meaning a but it actually means meaning b and like wherever that like the divergence between a and b that's where we that's where things go wrong, right? And so my my kind of philosophy is that it. It behooves us to understand where the divergence between a and b is because that's what attackers are doing attackers are trying to say where where is there an assumption that's being made to where what the people are assuming this means is not what it actually means and how do we get? How do we get within that that delta basically. And that's where success happens for attackers and so like ah my philosophy is is that as defenders we need to really work hard to understand and propositionalize the the assumptions that we're making that are inherent in our detection strategies I suppose. 01:05:30.88 Jamie Williams I was waiting for like the perfect moment to bring this up but like for our like very dedicated like part 2 listeners I think where cliffinger last time was like what is a Ttp detection and I think like you just like kind of drilled it in terms of like you know I think like so one of thing that floats around is like you know you know. 01:05:40.53 Jonathan Johnson I. 01:05:49.20 Jamie Williams Pyramid of pain like behavior tt-based detections but like I think there's honestly like a lot of like confusion like what that actually means like people obviously like point back to attack. But even like I think we had a conversation a couple weeks ago it's like I look at like a lot of detections and analytics that I consider like that's that's a ttp like behavior-based detection. Honestly like you look at it and it maps to like multiple techniques and I'm like that like I think people honestly, there's maybe an assumption that it's a 1 to 1 yeah, yeah, it's like people want that one to None like you know you're you're detecting like ah you know. 01:06:15.33 Jared Atkinson Behaviors and techniques are the same thing. Yeah yeah, that. 01:06:18.70 Jonathan Johnson Um, yeah. 01:06:21.94 Jared Atkinson The. 01:06:24.71 Jamie Williams Ah, you know one technique or like you know it's It's drilled in and like exactly you said I think like some of a lot of when we talk about like behaviors like it's a compound like concept like it most likely is some type of like you know execution or something else or there's just so many levels of like you know ideas. Um, I'm just curious like what does that make like any sense or like is that is it is it critical as an industry for us to kind of have an idea of what that answer is or is it kind of one of those things where exactly like many other topics like as long as we kind of you know is the good definition of a ttv-based detection like. 01:06:44.45 Jared Atkinson Um, ah both Yes yes. 01:07:02.40 Jamie Williams More of of the scoping of the detection in terms of like hey there's a concrete idea of what I'm aiming for it could be Broad. It could be Compound. It could be very atomic. It doesn't Matter. It's really more of like how the detection is created and matured versus like. Fitting up but specific mold or a very particular type. 01:07:19.75 Jonathan Johnson I think what you're like describing when it comes to like tgp stuff. Detections is like I'm going to bring this up because its second top of my head I just saw a tweet' not go to say who it was um jared to send it to you. Ah Jamie I think you've seen it but essentially it's like. 01:07:32.17 Jared Atkinson I saw. 01:07:37.36 Jonathan Johnson Keeping Detections simple and they might not all wait like there's not really a ne correlation. Okay so Hashtag triggered. Okay so like um the reason that type of like thought process in detection engineering really annoys me. Um and I guess since we're at the minute part I'll just like. Go on my little like rants that I go in this podcast and so this is probably once a little hot um Jared crack that crack that sucker open. Um and so but what what Jared was describing about like the whole dumper understanding the different processes. 01:07:57.90 Jamie Williams I think. 01:08:12.20 Jonathan Johnson Or variance that an attacker can go about with dumping Lsas um is equally as important for defenders to understand and cannot be done unless correlations are made and what I mean by that is if you quote unquote keep your detection simple and you look at the None that Jared was talking about. Um. 01:08:14.70 Jared Atkinson Oh man. 01:08:31.53 Jonathan Johnson You're going to miss like probably a lot a high level of things. Um and you might not be able to actually know what that None None of like process access actually relates to in terms of behavior because I think there's multiple like things that correlation relates to it's not just the data. But it's understanding of what the data applies to you and then what you can take in terms of action after the alert is triggered so like when it comes to like what is a tdp and what is the detection when it comes to that some technique sure are going to be able to be detected at like a very basic level. Right? So like um, let's do like let's think of something very simple. It's like say run as right if you if run as is not allowed in your organization. You could probably be like very simple and just do like run as dot ai look for that in a process command line. Great. That's very procedural base. Okay, so what? if someone like drops. A binary that does exactly what run as is doing and under the hood of technology. Okay, then you have to go a step further and you might be able to go like do something process based but you probably need to jump into something like I'm writing a blog about called log on centric data. Okay, which is going to like. Expand your capability and understanding of the process data. That's there by being able to tie it via a story right? but that that can't happen unless a correlation is made and unless the correlation is made not only in terms of data but your understanding of the actual action there in play. Defenders in my and I'll be a very honest about this I think defense currently as its state stays right now is years behind what current red team capability is the goal that we have is to change that and how we change that is to open the eyes of you. Defenders need to probably understand a technology better than the attacker who is performing forming it because like Jared said attackers typically care about the output. So. There's probably a lot of red teamers out there. That's leveraging dumper not changing anything about it and running it. Great. So like. There's probably different detection levels by which someone can go about detecting dumper or what dumper does right? So like you have like Md has a um event in there. That's like read um read memory or something like that's an Api call. But that's going to detect I believe at the kernel level right? So if someone calls. 01:10:58.52 Jared Atkinson Hold on hold on. You shouldn't assume like I know you know, but 1 of the key points is that when you None discover this you shouldn't assume that it detects from the kernel level. You should like one of the things that Johnny did is he literally reverse engineered it. So I know that not everybody could do this but this is. 01:10:59.26 Jonathan Johnson Dumper the Cis call it there. Yeah. 01:11:07.60 Jonathan Johnson Um, great point. 01:11:10.25 Jamie Williams Um, yeah. 01:11:17.30 Jared Atkinson This is the thought process you should at least ask the question of does this detect at the kernel level and one of the ways that you would evaluate that is you would run the syscall you would run something like dumpert so you don't even have to write the code yourself. You could run something like dumpert and then you. You know you know or you at least trust the people that wrote dumper and have a wiki wiki on it and it says we call the syscall and it's like okay well does when I Run Dumpers does it get captured by this by this event and if it does you could assume that it at least catches. It's at least observing below the syscall. Sorry. 01:11:34.88 Jonathan Johnson Um, yeah. 01:11:42.46 Jonathan Johnson Yeah. 01:11:49.10 Jonathan Johnson Yeah, and yeah, no, that's a good point and so like you don't even have to in terms of like testing telemetry like Jared mentioned you don't have to reverse engineer things all the time. Um, like so this kind of goes to what Jared's and I tweet was today about and um, sorry one second my I have signal up and a. 01:12:06.23 Jamie Williams Shout out to Jared for taking that down a single pool that was they will a lot laugh but still much respect. 01:12:08.79 Jonathan Johnson Ah, yeah, yeah, yeah, so so what I was saying to sorry my headphones went off because I got like a phone call on my computer for some reason and like I couldn't hear a word. It was like yeah, whatever. So so you don't have to reverse engineer the telemetry. 01:12:13.71 Jared Atkinson Yeah. 01:12:22.33 Jared Atkinson Um, you're good. 01:12:28.45 Jonathan Johnson You don't have to do that what Jared was showing within like that tweet in there is that there is a way to look at an abstraction map create a function call graph and then use an automated way to test the telemetry capability at the various levels. Little insight into how threat research is done over at red canary the team that I'm on that is a very common way by wit. That's one of the like purposes of the the test harnesses is to be able to create an abstraction map have a correlating test harness that relates to the various levels and then test our partners to imagery. And then understand okay like at what level is this happening because not everything needs to be reverse engineered. However, the amount that's happening of that inside of detection today as a whole is very small There's not many people doing that and so that means our capability is going to only stay at the highest level. And that is the quote unquote keeping it at the basics. So 1 other thing I want to say about that little rant about that tweet is understanding the basics by definition means that you understand the technology that's being leveraged. Okay. So how can you detect wmi if you don't know how wmi works. 01:13:42.62 Jared Atkinson Well, you could detect. You could detect the most superficial version of Wmi That's the problem. That's the problem. 01:13:42.79 Jamie Williams I. 01:13:47.10 Jonathan Johnson Um, yeah, yeah, yeah, Sam yeah. 01:13:47.46 Jamie Williams That was going to be my point I'm I'm a huge correlation fan but like Devil's advocate would be in terms of action ability like I said earlier like how often is you know the actual output of most of these detections just quarantine the box like really. Is the goal of a detection to be the point of you know maximum like you know, minimize false positive maximize true positive and if that like I'm really trying to find that least common denominator of like. 01:14:10.38 Jonathan Johnson The. 01:14:16.72 Jamie Williams How can I reach that as simple as possible Sometimes like you said I might need a bunch of different like you know inputs and correlation and really like piece it together but they mother me up like opportunities like because I'd run out as a good example like hey if I can kind of you know from policy or some other means like say like hey I'm going to some kind of other control. Limit run as an environment. Maybe that is simplistically is good enough to get to that point where I'm actually taking action. So It's like it's kind of that philosophical tradeoff of like exactly as you said before it's like I I don't know what's best I think both approaches kind of makes sense. 01:14:53.31 Jonathan Johnson Um, yeah. 01:14:55.41 Jamie Williams But that said like it's it's I know I I wish there was like a you know I think we really just do need like almost this like persistent like realm where we can just kind of like throw ideas and see what sticks because I guess I think both approaches kind of depending on. It's all of. 01:15:06.60 Jonathan Johnson Um, yep, yep. 01:15:07.39 Jared Atkinson Um, yeah. 01:15:13.13 Jamie Williams All of that is kind of relative towards like I saying before like what are you going to do like if you were going to like if I'm if I'm slinging alerts that you know you know static like Ioc like you know I c two addresses and stuff like that might not be great, but if I know how to act on that. 01:15:15.52 Jonathan Johnson Um, yeah. 01:15:28.82 Jamie Williams Like it could work for me like we could roll this like kind of None pass like detect strategy and maybe do something with it. There's obviously a more optimal solution. But if like honestly like you said if they're if I've I've also seen the inverse of like correlation of like there's too much correlation like it's all over the place like or like you know another one. We've seen from like evaluations was like. 01:15:29.41 Jared Atkinson Um, yeah. 01:15:31.29 Jonathan Johnson Yeah. 01:15:41.88 Jonathan Johnson Yeah. 01:15:48.67 Jamie Williams Some really really slick correlation logic where like you know had like you know, maybe 5 components and like how many detections got to that fourth and never hit the fifth and then never fired and you're like oh like you had good evidence. You just didn't really release it and like that full case didn't build up. 01:15:57.60 Jonathan Johnson What? yeah. 01:16:03.96 Jonathan Johnson Yeah. 01:16:05.73 Jamie Williams So it's like it's kind of that tradeoff and like is that I wish there was like a clear right? Answer I'm kind of in the boat of like just do whatever works for you As long as you're kind of paying the bills. 01:16:09.59 Jonathan Johnson Yeah, yeah, the the there is a lot of human error when it comes to correlation like I agree with you like there is a time and place for like the quote unquote basic when I think a basic detection this might chero some people I don't care basic detection. Equals procedural detection in my head that might not always be the case but that is a very like yeah to yeah. 01:16:28.81 Jared Atkinson Tool Tool Detect yeah tool detection or procedural detection. 01:16:31.42 Jamie Williams Yeah, like a good example is like you know, like any like you know, ah like t None three like you know os control dumping whatever like there's a None different ways to do that. But like it does. It's not a great detection but like you should probably have something for just mimecats on yeah Eexc like. 01:16:48.70 Jonathan Johnson Oh hundred percent help 01:16:48.50 Jared Atkinson Sure. Yes, yeah. 01:16:50.42 Jamie Williams That's a terrible detection but like how many people will want to get like owned by like just a off github like mime cats and it's like you. 01:16:54.64 Jonathan Johnson Yeah, exactly yet. 01:16:54.87 Jared Atkinson I won't I once had a guy we once had a customer who told us his None goal was he if if they get breached. He doesn't want to look like a jackass and that that would be the like you know, step 1 step 1 is step 1 is make sure that it's not called mimi cats at least. 01:17:04.37 Jamie Williams Um, yeah, http://evil.exe and stuff. Yeah. 01:17:04.55 Jonathan Johnson Um, yeah, yeah, yeah, exactly yeah and then like oh here's the reality too is like as you go into the like deeper ah correlation. Logic. 01:17:13.83 Jared Atkinson You know. 01:17:22.35 Jonathan Johnson There's probably I mean like I haven't really seen the I mean it's like None I guess like all right? Yeah, there's there's more light like for example, if I were to write a detection explicitly on like the kernel level api re reprocess memory. 01:17:25.44 Jared Atkinson Let me go into the behavior thing in a second after this this right this rent. 01:17:39.20 Jonathan Johnson But likelihood that's going to like trigger more quote unquote false positives is debt like way higher than if I were to just do like do the win None api version of reprocessed memory theoretically right in the way I see like correlations so there's correlations man right Jared I'm gonna say this remind me to come back to this so because there's all no repital. There's correlation can be applied I think at 3 levels None is the understanding of the analyst right? understanding the technology in my head is me on to like look at data make a correlation x is happening in my environment the next in that comes from like. Correlation done at a detection level. Okay there's also correlation at the triage slash investigation level that that can be done and that can come by which of many ways which right now is very process centric um, which isn't the only approach. Um. 01:18:20.31 Jamie Williams And. 01:18:33.33 Jonathan Johnson Which I have a blog coming out on that. But before I dive into that which is like that log oncentric stuff I'll dare go with the behavior stuff because that's a whole another round hole. Yeah. 01:18:36.97 Jared Atkinson Okay I got to go fast then if you want to talk about that the the key that you're talking about is that there's correlation that there's correlation that could happen at multiple stages in the funnel that I would say that's how I would say it um and at later stages in the funnel you may have access to court like. Correlation that wasn't available to you in earlier stages in the funnel. So um, because you've reduced the volume basically. Okay, so so Jamie was asking about behavior and I'll I'll continue on the credential dumping thing because that's my new canonical example. Um the like okay, so. 01:18:57.24 Jamie Williams There. 01:18:57.82 Jonathan Johnson Um, yeah, yeah. 01:19:14.80 Jared Atkinson We're talking about Okay well the the Attacker doesn't care how they do it in most cases they care about what they're doing right? and so um, like the attacker cares about getting access to a credential to a password. Let's say they don't care about how. They get access to the credential which means that they're they're they're willing to adjust. However, they feel they need to adjust to Evade detection or Evade ah prevention actually most attackers care seem to care Anecdotally seem to care more about prevention because ah, honestly, most of time detection doesn't. Ah, the the loop the feedback loop of detection isn't tight enough to actually stop attackers from doing things so they they really care about prevention. Um, but maybe secondarily they care about from a tradecraft perspective they care about detection I I guess and so the idea is is that it's like okay well what? what are the ways. In which somebody might get access to a password. Well, that's that's ultimately like kind of what you classify as techniques in the context of attack right? But there's um, it's like okay well I'm ah there there's something like there's numerous ways to get access to techniques I could go and read ah or to passwords I'm sorry I could go and read the passwords. From the Ntds Dot Dit I could read them from the Sam Hive I can um I can get them from ah from Lss Memory I could do None of different ways right? But what we kind of do is we so when we start building techniques so that that's kind of like a tactic credential access right? and then a technique are like. Okay, within all those things There's slightly different variations like the ntds dot did is different than lss memory which is different than the Sam hive and so we call those techniques. We kind of group things. So There's like kind of a natural grouping that occurs. Um and like we more or less are able to like literally manually categorize things into into these groups. Um, but then within the context of the technique. There's numerous ways which somebody can achieve credential dumping from Ls right? And so I think back to when I None started in infosec One of the popular things that people talked about was process injection and so one of the things that they would say would be. Ah, and this was before I even knew what Api functions were and so people would say this and I was like yeah cool I got it and and then I would go and look for this thing and when I so when I saw it. It's like it never ended up being process injection I was like what the hell but they would they would say if you see the combination of virtual alloc. Ah, ah. 01:21:41.70 Jonathan Johnson Write process memory. 01:21:43.39 Jared Atkinson What is it virtual write process memory and create remote thread. You have you have process injection right? and every time and so what they were doing was they were identifying. Ah they were identifying the api functions that ah that they that somebody would use to do process injection. However, that was. That's actually too specific right? So you can you can extrapolate from that and you can say what like when they call ah virtual alloc write process memory create remote thread. What is it that they're trying to do right? Well what they're trying to do is they're trying to allocate a buffer in memory of a process that they're injecting into. They're writing their code and then they're actually telling the processor to execute the code that they wrote and the the whole theory of this function call graph thing is that for each of let's call those actions right? allocating a buffer writing code executing code for each of those actions. There's potentially. Numerous past to achieve that action. This was actually this is something that Jamie kind of helped me realize in ah in a discussion about my Twitter thread which Luke will post in the show notes. Um, so it's like okay well the thing that actually is creating the behavior is the create remote thread call or the execution of the code. But the code has to get there like you can't you can't execute code that doesn't exist and so there's there's like a relationship between those 2 things to where writing the code is a prerequisite to executing the code and then there's a relationship between allocating a memory buffer and writing the code right? So there's there's this kind of. Chain of events that have to happen in order right? And so um, that that chain for ah allocate allocate memory write code into that allocated memory and execute the code is the behavior from my perspective. That's the behavior and the key is is that there are. Ah, for each of those actions. There's potentially numerous paths. So let's let's look at credential dumping right? So credential dumping the function call graph that I've made is for ah reading memory so I would call it a process read action right? So you're reading the memory of a process. Um, but in order to read the memory of a process. This is what Jamie pointed out is. You must acquire a handle right? And there's numerous paths to acquire a handle and I don't have a function call graph right now for that. But I would I hope to have that in the future and so there's there's numerous pass for handle acquisition. Well in order to acquire a handle. You have to. Find the ah the process identifier for the process that you want to open Ls as in this case and so that would be a prerequisite to that. So the the behavior ah technically is read process memory but you can't actually decouple that from the the prerequisites right? and so you have. 01:24:27.10 Jared Atkinson I call it process enumeration which is getting the ah the process handle or the process identifier process like process access process open process acquisite Handle acquisition. Whatever you want to call it. That's ah, represented by open process or like assist Mon Even Id 10 and ah the read right. And so one of the the interesting things is is that there exists. Um, you know a plus ah a times B Times C possible pass to achieve that Overall um, that overall relationship or that overall superpath or behavior Path. You could call it right? and so there's. 01:24:57.97 Jonathan Johnson Um, me. 01:25:02.31 Jared Atkinson For instance in my graph there's 8 paths to process read. But maybe there's 5 paths to process open I'm making 5 up I don't actually know that but that would mean that between those two there's actually 40 passs because you just multiply them together and that gets you the toll number because for all the 5 pass of open that means that you could apply that to each of the 8 passs for read. 01:25:20.24 Jonathan Johnson Yeah. 01:25:21.82 Jared Atkinson Right? And so um, and so what when we start talking about behavior I don't care how Mimi Cats does it or I do care because that's kind of what introduced the idea to me but I use that as a starting point to then enumerate all of the different ways that somebody could achieve the same outcome. Mimecats is achieving right? which is basically reading the memory of Lss to get access to credentials and so my my approach to kind of doing the behavior-based detection is to say what is the behavior process enumeration process access process read. 01:25:44.41 Jonathan Johnson Um, yeah. 01:25:57.10 Jared Atkinson What are all of the different ways that somebody could achieve that at least to my knowledge right? So like my process My function call graph is probably not complete. It's like ah the best I have um. 01:25:59.22 Jonathan Johnson Um, yeah. 01:26:07.60 Jared Atkinson And I get I'm able to build that based on things like dumper dumper introduces something new to me or out mini dump introduced a new path to me that I didn't know previously right? So I could just take different iterations of tools that attackers created and I could plug them into my graph and then I could extend the graph and give me more more knowledge than I had previously. And then it's like okay, well now the question is is how can I you know there's let's say none different paths of just process open and process access. But then there's also process enumeration and we can't decouple that so there's maybe like a none different paths or 80 pass or what? Ah who knows None 01:26:38.50 Jonathan Johnson Yeah, what I think what you're describing too is like identifying what the actual behavior of the attack is like. For example, like you're talking about like reading a process memory. So if someone calls many dump write dump. But it eventually calls that function call. However, None thing you're not touching upon and I know this is purpose. Purposeful because like we've talked about this is like obtaining the handle to the process by which you want to read from I mean that same thing applies to um, process injection. You have to obtain a handle to that process now that might be like we've thought about this like the pre-action or like to the attack or whatever it is. But the reason why sometimes. 01:27:06.38 Jared Atkinson Um, yep. 01:27:14.93 Jonathan Johnson It's focused upon detection at that level is because maybe telemetry doesn't apply as well to the actual action being the reading of the process memory. 01:27:18.55 Jared Atkinson Yeah, yeah, yeah, okay so so the behavior for credential dumping that we are actually interested in is the read right? So where when somebody reads the memory because that's so actually the the interesting thing is is that you can't do the read without doing the. 01:27:21.41 Jamie Williams Um, exactly. 01:27:37.49 Jared Atkinson Access right? There's um, but just because you did the access doesn't mean that you're going to do the read so. There's like an end There's like a weird relationship to where ah you call it necessary. There's necessity and sufficiency I guess necessary but not sufficient and so the idea is is that um. The process access is necessary but not sufficient for the process read meaning that in order to do the process read. So but the the inverse is true, right? So the process read is sufficient for but not necessary for the process process access and so ideally. Your detection would be catered towards the process read because that's actually the true behavior but like Johnny was just mentioning. You may not actually have visibility into the true behavior and so I call this ah sir like I call this in medicine when they start looking at like trying to do diagnosis. They call it like surrogates. Um. Like surrogate markers or so surrogate metrics and so what they do is they they don't have the ability to measure the exact thing that they're interested in so what they do is they do a blood test and they say oh well your um I don't know what's a good your blood sugar is high so that means that you may have some issue with diabetes or something like that which is. They would get a better so signal for that if they or a more direct signal if they tested your I don't know what is what is it your pancreas that would be affected by diabetes so that that's like what you're doing is you're using a surrogate which is like it's more accessible but it's not as good right? And so there's actually the delta between ah process access and. Process read is actually false positives right? So that that there's false positives that are inherent and using that surrogate method. Um, yeah, don't know. 01:29:14.79 Jamie Williams That we touched on this last time too. That's like interesting like getting back to like the pyramid of pain I think that's why that like conceptually is maybe hard to grasp because like you look at the pyramid of pain it like it gets more narrow but like you said like even when you're getting into that ttp behavior realm. It's not like a 1 to 1 kind of like oh here's like. 01:29:23.69 Jared Atkinson Um, yeah. 01:29:33.80 Jamie Williams Ah, solution to this thing. There's like all these paths and the other interesting thing is like exactly so like surrogates. It's like now that there's like that pragmatic like overlay on this. It's like hey like you know the functions like you know we have that kind of base. But then as we kind of work with like available technology. It's like okay like. 01:29:34.28 Jared Atkinson Yeah. 01:29:50.71 Jamie Williams The way this is actually going to be presented to me like obviously like we would love to just like stab. Someone's pancreas and like okay cool like you know I can do readings on the organ I'm going to have to like abstract away in terms of like hey like you know how often are we actually going to be doing api hooking versus like maybe like Mod loads like hey like I can just like. 01:29:54.99 Jared Atkinson Um, yeah, yeah. 01:30:08.41 Jamie Williams It's not as like Chris but like I can make use almost like analytic logic is like to do that analytic leap of like hey like you know I can maybe abstract to like this dlo loading into this process is maybe a surrogate for certain function calls or like hey like a good one would be um. 01:30:20.63 Jared Atkinson Um, yeah. 01:30:26.43 Jamie Williams Everyone' obsess with like that that pro like Proc dump like you know, hey like you know I can watch a file being dumped a disk like lss dot dump like I have no idea how the handle or like the read or any of that worked but like I have this surrogate where like you know within some level of like you know, certainty like this is a good artifact of that. So like. 01:30:35.70 Jared Atkinson Um, yeah. 01:30:45.96 Jamie Williams There's multiple layers but I think that is like a thing I've kind of been circling around is like you know what? what does the tip of that pyramid really look like in terms of like you know the you know lsat dump versus dlo model loads versus those Api calls like. Are all of those equivalent or like you know in that realm of like behavior-based detections like is there a scale or another metric within it that we can actually use to kind of start pushing. You know, even further. 01:31:00.68 Jared Atkinson Um, yeah, see ah set. Almost certainly there is I have no idea how to how to glean it but that that goes into the the layers of variability right? So like. For instance, the Module load and the file the file creation For instance, those are um. Those are layers of variability that exist in a different plane or a different category than the function call Graph I Guess um and I would say that they're less significant meaning that it's like um, it's. 01:31:31.30 Jamie Williams Um, yeah. 01:31:41.81 Jamie Williams They're less true. Yeah. 01:31:43.18 Jared Atkinson They're less. They're less accurate of the behavior that you're interested in. they're they're more they're accurate of a subset of the behavior right? And so yeah, that's but yeah I mean if that's what you got? That's what you got but like what I'm trying to do is figure out a way. Um, ultimately this goes back to the like. Can we get a None or something in between for how well we cover this. 01:31:58.55 Jamie Williams Um, yeah. 01:32:02.40 Jared Atkinson What I'm trying to do is trying to say hey if and maybe you you actually reverse the way that I'm approaching. It is through testing. Um, and so you reverse engineer the answer based on your ability to have visibility into different tests that kind of tests the boundaries of what's possible. Um. You're kind of like fuzzing you're fuzzing the detection to some degree because a lot of times we don't actually have the detection logic and so you can't you can't just like statically analyze the detection because it you don't have it. Um and so what ah like what I'm trying to do is trying to say um, okay, well. Here's all the different possibilities along this dimension of analysis. The function, the functional dimension but there's other dimensions that we could eventually add into it as well. Right now I'm actually not even looking at the function like the behavior I'm looking at um, the the terminology that we've been using internally at spectroops is. You have a behavior which is the combination of actions. So those 3 actions that I laid out process enumeration process access and process read. Ah the combination of those is the behavior but then you have actions which are the like the process enumeration process access process read. Um, and you build function call graphs for each of the actions right? but you just know that those things are intertwined and that you need to represent all the variability across all 3 actions or all x actions depending on however, many actions there there happen to be there happen to be 3 in both the examples that I gave but it's not It's not just always 3 it could be None for instance. Um, and so yeah, what I'm trying to do is say like okay well if we're if we're testing you and you have visibility into like the top None um. Most significant paths right? or most dissimilar like right now I'm looking at similarity or dissimilarity of pass if you could detect if you detect if you have visibility into the top None most dissimilar paths then that means that you have ah point seven coverage out of on a scale of 0 to None um. But if you detect the top None then maybe you have point None or something like that because the the idea is is that ah, the none path is more significant. The the reason why I'm rank ordering them based on similarity is that the none None paths are the most significant and then after that each one you lose you lose significance to where the none path. Probably doesn't give you all that much to be honest, there's like less bang for your buck I guess yeah, all right? Ah John I mean I got I got more time I don't know Jamie if you have more time but Johnny wanted to talk about his was this the ah the process versus log on session thing. Johnny. 01:34:22.83 Jamie Williams The economies of scale and such here. 01:34:36.59 Jonathan Johnson Yeah, can you my camera looks frozen on my end is it frozen on your end. Oh oh no R I being piece all right cool. Ah hey producer going needs you to produce that to where I don't look an idiot for. 01:34:39.80 Jamie Williams And you're still. 01:34:41.46 Jared Atkinson You're frozen. You kind of look pissed off. Actually it's okay, we don't We don't need to see you. 01:34:53.50 Jamie Williams Um, just put like a smiley emoji like over here. 01:34:54.95 Jonathan Johnson I Don't even know how long I was going. 01:34:55.17 dcppodcast Um, my capabilities are vast. They do not extend to making Johnny not look like an idiot. 01:34:58.29 Jonathan Johnson Or are they are they so vast that ah means you can't even get a camera to work for yourself over there. Okay anyway, so um. 01:35:05.61 Jared Atkinson Oh that's. 01:35:06.66 dcppodcast Um, let he who is has a working camera cast. The None stone Jonathan Johnson 01:35:11.30 Jonathan Johnson Also have I also have a I also have a a working house right now. How's that in Texas going for you all right moving on. So um, ah he owns a house that's not finished building yet so he lives on the street. Um, probably why doesn't him a camera right now. Um. 01:35:17.39 dcppodcast I own mine. 01:35:19.42 Jared Atkinson Um, yeah, Luke owns his house so there's there's that yeah. 01:35:30.80 dcppodcast Sold it to pay for 1 2 by 4 01:35:32.43 Jonathan Johnson I'll go into this quick because I know ah. 01:35:33.31 Jared Atkinson Lu actually in a Starbucks right now because prewi. 01:35:37.70 Jonathan Johnson But um, Luke said 5 minutes 10 minutes ago left on so I'm just going to kind of briefly talk about this? Um, and then we'll kind of like just kind of end it I guess um after everybody's responses. Yeah, so when I saw a correlation. Um, one thing that I have found to be interesting. That's not often talked about like today correlation from an investigation and triage standpoint um is vastly reliant on process centric data and that's not that it's a bad thing. Um, but it's not that it's the only thing and some recent research that I've leveraged um to actually like see attacker activity and link it all together is through log on session centric data and essentially um, that is you have a user that is and I have a blog coming out I think next week of the week after talking about this. Um, and I've submitted some talks whether they get accepted or not if they don't get accepted then I'll just use this as a platform to shell all this work. Um, but essentially um, if you have a user that's logged in say attacker a and they log in. They do like a run as ah Jared and then um, they somehow like obtain access to Jamie all in the same box and they execute as as those users you can and I know you can do this because I've done it like you can actually it's in say ah, an alert triggered for Jamie. You can actually use log on session centric data to walk back. Jamie's log on session so the activities that he did during that log on session figure out who logged him on and then walk back to Jared what Jared did within that log on session and then walk back to the attacker. 01:37:20.30 Jamie Williams Um, is that based on sorry is that like I know you did some like token research is like just to correlate that like token theft and all that does trigger like a log on correct. Okay, nice. 01:37:22.29 Jonathan Johnson So you're. Um, yes, yes, yeah, this is this is all derived from like the same research A yeah um. 01:37:31.97 Jared Atkinson Ah, token theft token theft doesn't ah doesn't have to trigger a log on so there's there are ways to like generate a token by logging somebody on that's like that type 9 log on. But if you you could do like git system and it doesn't. 01:37:46.82 Jonathan Johnson Yeah, yeah, because well that's because a log on session already exists for the token you're stealing. Yeah yeah, that's. 01:37:51.54 Jared Atkinson Trigger a log on I don't think Yeah, yeah yeah I just click. Yeah, if you do a yeah a run as yeah. 01:37:55.53 Jamie Williams But if you do like a run as or something. It's probably going to generate like a yeah. 01:38:01.64 Jonathan Johnson That's using the log on you that's using the I think it's like log on user api underneath the hood to log that user on and then you can actually like there's many apis like um man I can't think of at the top of my head. It's actually implementing the test harness. Um that I have out there. But essentially you log on a user and the um. Return of that Api is actually the token and so you just take that token and apply it to your current thread. But yeah, so like all this is derived from that. Um, um, same same research. Really. Um, because I I felt like there was more juice to be squeezed and whenever like I look at an action and I'm like man like I'm not really seeing a lot of telemetry for this my next kind of thought was okay, let me look at the activity around it maybe look at like some of the other things. That could be applied and one of those things that I saw was like there's not many people leveraging um log on session data now there's a lot of companies right now that are saying they do quote unquote identity I think that's a different type of thing today and that's for another podcast but um. Yeah, so that's essentially like what this is what that research is about um because it might not be the best solution. We don't know that yet because not enough people have tried it. But it's at least a new solution that challenges the process session centric data. And but all our process centric data but it's also one of those things I keep mind like yes with the log on stuff. The the point is to look at the processes that were applied during that log on session. So it's another way to expand your understanding or look at those um processes that were ran by. Yeah. 01:39:36.72 Jared Atkinson So so. 01:39:37.51 Jamie Williams That's super interesting too is like it also like getting back to your like layers of correlation like I imagine like now like like that's a really good model too because like I imagine you're working through it from like ah like an investigation abstraction then it like it solidifies and goes into like more of a static um detection. 01:39:47.49 Jonathan Johnson Um, yep. 01:39:52.29 Jared Atkinson Yeah. 01:39:52.60 Jonathan Johnson Absolutely. 01:39:54.38 Jamie Williams But that said like a good use case for that would just be like how many users log on and immediately start like hammering like Powershell like that's probably super weird. Um that in itself could be probably something you could probably deploy easily and get some pretty solid results. 01:40:00.88 Jonathan Johnson Yeah, well yeah, whenever I first started detection. Um, first like at the beginning and I didn't understand why until now I always use either like process goodids from cysmon to be like. The glue that I use to bring multiple processes together for context or I used log on Ids and log on guids back then I didn't have the understanding that I have now on how and why leveraging those are so powerful I shared this with. So Jared actually picked this up after that access token blog that I wrote a while back. He hit me up on the cellie as the kids would say and we were talking about it and he goes like I think this is what you're kind of originally getting at. But yeah, so like I was essentially releasing that research so I could tee up this research and it sounds like I mean Jared sounds pretty excited about it too. 01:40:54.18 Jared Atkinson Yeah I think there's ah okay so one of the problems that we that we have is like we tend to see some malicious activity right? like so we have an initial detection. This is your point Jamie we have initial detection. We say what process made that detection right. 01:40:56.12 Jonathan Johnson Um, and but yeah. 01:41:13.92 Jared Atkinson Um, and so then we correlate that and then we might say okay what network connections or what other processes did that process create but the the problem is is that in scenarios like injection. For instance, it's not actually the like the process isn't the rights. Entity to be paying attention to right because it's actually like when you inject you don't you inject into a process but your code execution isn't the process. It's the thread right? And so what's more important is what did that thread do well when you do things like token theft or injection or whatever. The the thread often has its own represent. Ah its own log on session that's associated with it and so what you could do is you could start to say Ah, what did this log on session do and like so it's it's not as good to say what did this user do because a user can have numerous log on sessions that are some are legitimate and some are malicious. Um. 01:42:08.38 Jonathan Johnson Yeah, like fun fact, like to that point sorry to kick off like. For example, if you have a local administrator to log on. They actually have two separate log on sessions and that's because they have a token applied to each log on session. That's a high elevated context and like a medium elevated context when they switch between those. 01:42:10.24 Jared Atkinson The profit. 01:42:25.26 Jonathan Johnson Actually switch between the log on sessions of execution. 01:42:27.70 Jared Atkinson And so yeah, and so ah, what? like for instance, ah, a good lateral movement detection at least anecdotally that that I've that I've played around with is you could look for for instance network log on sessions right? So look for any network log on sessions which means that somebody used something like wmi or ah win Rm or something to connect and ah, there's lots of network log on sessions but the vast majority of them are using things like um, ah like the Ipc dollar share or they're like ah mounting mountain mounting file file shares and things like that so they're doing innocuous things that are just for authentication purposes. But what you could do is you could actually correlate the log on session and you could you could query what processes ah belong to that log on session and the vast majority of like if if all I'm doing is authenticating for like a file share access I'm not creating a process that doesn't happen and so what you could do is you can look for. Network logons that have that have child processes and those are going to be ah, somebody doing some sort of remote like remote execution. It could be something like your nessa scanner or something like that. So it could be It could be legit in the context of like intended functionality. 01:43:26.74 Jamie Williams Um, the night are. 01:43:41.49 Jared Atkinson But it will be somebody running code from None system to another which is a good way to narrow in focus I guess so there's there's certain things that you could use the log on session for that would change the paradigm of how we actually like view the relationship between actions because I think that the. Process is not a perfect representation of the of the action and then you could actually tie parent log on sessions to child log on sessions in the same kind of manner and so you could see like hey you know this this log on session created a new log on session because they logged on some different user or whatever and so we could track what they did and that kind of thing. So. 01:44:15.77 Jonathan Johnson Yeah. 01:44:17.68 Jared Atkinson Just a sifma bank explore. 01:44:18.75 Jamie Williams Aligns with like some research like we were exploring too is like you know exactly to your point like how much of our detection Realm is like focused on process like why is that is it is it because it's the best data source or is it like cost effective like this is really easy and like very mature and like I think another like thing we. 01:44:24.31 Jonathan Johnson And. 01:44:35.90 Jamie Williams Probably should explore is like what's 2 What's 3 like log on sessions really good one because like you know it's pretty like common and almost like organic for most oses just to do that. So like now that we kind of have that and it's like tangible and it's like in everyone's hands like it's probably in our best interest interest to like exploit it and optimize it. 01:44:44.45 Jared Atkinson Um, it's. 01:44:52.81 Jonathan Johnson Um, different. 01:44:54.76 Jamie Williams But like same thing like you know what's next like file registry network connection like and kind of work our way through like that entire stack and figure out like okay like you know, let's now that we kind of have these things like what do we actually do with it and how do we correlate it back with like everything we've already built. Um. 01:45:09.13 Jared Atkinson I think I think that the reason why we use processes is because processes are you know from the abstract perspective where code executes and so like the None people start looking at processes because they're like that's where all the bad things happen. 01:45:09.35 Jonathan Johnson Um, yeah I. 01:45:19.50 Jonathan Johnson Get. 01:45:25.48 Jared Atkinson And then it's it's kind of like a ah geocentrism versus heliocentrism thing to where ah we were in the process-centric box and like in order to break out of that box. It actually is like quite difficult because all of our tools have been developed to facilitate a process-centric view of the world and so what we're doing what Johnny is trying to do I guess is. Ah, facilitates a different perspective but like None of the one of the difficulties of that is like very few of the tools are actually written in such a way that they facilitate that and so you are. You're actually like fighting upstream to some degree to even try to do that in the none place. 01:45:46.27 Jamie Williams Um, yeah. 01:45:55.91 Jonathan Johnson You? yeah, what's interesting is like I think like the original goal of like Lunar Lake registry process and foll events was who initially like who performed x action and can we determine that that like person was compromised. So we know this is good or bad and I think like due to the lack of either research or telemetry and insight into that we've pigeonhoed ourselves into being very process centric and so it's like okay so now since we might have able to know that can we look at the process execution and determine good or bad here. Um, and so it's like it's almost like the because we don't have telemetry into like reprocess memory. We go a step further or step up in the stream and look at open process I think the same thing was done here. However, now um, there are good. There's like a really good amount of insight and I think vendors have like realized the insight and the value in that. So they started exposing that telemetry better. Yeah. 01:46:53.19 Jamie Williams Like technical debt. Basically we're kind of digging our way. Yeah,, that's that and it's I think it's also like it's it's kind of interesting too because you apply that to a separate domain like thinking like Cloud like identity management is like all the hotness in Cloud for exactly that reason like conceptually the idea of like process monitoring doesn't necessarily exist. So It's kind of like you know're we're loading on a bunch of like you know, assumptions and a bunch of like you know putting our eggs in this basket of like you know log on and identity management where like the same thing. It's like you know is that the best approach and you know in general like what's the exploitation. Maybe those like you know philosophies kind of converge. 01:47:13.70 Jared Atkinson Yep. 01:47:32.92 Jamie Williams But at the same time. Yeah I think you're you're poking at a really good you know question not only like merging it back to process but like agnostic of process like what could you? What could you do like you know that'd be a really cool experiment or like given a single data source like what is what is the realm of like detecting environment look like from the concept of just log on sessions like I imagine. 01:47:38.59 Jared Atkinson Um, f. 01:47:49.79 Jared Atkinson Um, yeah. 01:47:52.44 Jamie Williams Be kind of cool just to see like how many like truly mature detections could be made just from like a single perspective. 01:47:55.27 Jonathan Johnson Um, yeah. 01:47:56.73 Jared Atkinson Yeah, it's like um, it's It's not necessarily just using log on sessions. But it's using log on sessions as the plane at which you pivot you know what? I mean because you could you could kind of like it's multidimensional right? So you could pivot along different planes and like. 01:48:06.73 Jamie Williams Um, yeah. 01:48:06.74 Jonathan Johnson Yeah, yeah. 01:48:14.35 Jared Atkinson From the from the yeah from the process. Yeah from the process plane. There's only certain pivots pivots you can make and the the assertion is is that from the log on session plane you can make different types of pivots which might glean more information. But I think I think to your point Jamie is like we're making. 01:48:15.32 Jonathan Johnson It's un pigeon holding ourselves from the the parent processed child relationship. 01:48:16.70 Jamie Williams If just. 01:48:33.24 Jared Atkinson Okay, so I think the the problem is is that in the cloud sphere we kind of know that we're making some assumptions right now but 10 years from now. It will be more of a it's all this is how we've always done it type thing and so I think what happened with the process thing is we made assumptions and over time that just became the way that we always did it. 01:48:43.27 Jamie Williams Yeah, yeah. 01:48:44.90 Jonathan Johnson The. 01:48:52.25 Jared Atkinson And so we we forgot that there were assumptions. Maybe we never knew in the None place but um, because people make assumptions subconsciously all the time right? But um, eventually we just lost all focus on the idea that assumptions even exist and like I actually almost went into this earlier but not to not to Derail. There's this.. There's this problem with like to some degree.. There's an this is the evolutionary thing right? So um, our visibility think of like think of human visibility. There's actually this theory called Snake Detection Theory which is how ah human vision vision. The human visual sense. Has developed as a byproduct of our coevolution with snakes and so like Snakes snakes were always our like major predator. Um, according to this theory anyway and ah and so the idea is is that snakes have lots of camouflage and so what what ended up happening is people that had poor poorer vision because like humans. Among Primates humans have like a very high level of visual acuity relative to like our body size and our head size and things like that. So a very large portion of our brain is actually dedicated to our our visual sense. Um, which like why like we know that dogs for instance have way better sense of smell than humans do Well We have a way better sense of vision than dogs do right. And that's ah attributed based on this theory to to our development with snakes and snakes being predators because people that didn't see snakes got eaten and the people that did see Snakes didn't get eaten and so there's actually this this ah kind of the theory goes that we prioritize movement over static. 01:50:24.76 Jamie Williams Um. 01:50:26.31 Jared Atkinson Things that move over things that are static because there's so much information to take in that we can't process it all right? and so we we actually like when you pay attention to something you're actually selecting subconsciously often. The things that you will actually process right? And what we do is we we prioritize movements over over static things. And so one of the big problems I think is that the same type of evolution. You could say happens with visibility along the the cyber realm because eds but and other like indr and all that kind of stuff that is our those are our senses in the cyber realm you could you could liken it to to site and so one of the big problems is is that. I don't think that in the cyber realm our visual sense in the form of Edr is subject to evolution like or selection you would say um in the same way that our ancestors were subject to selection because like. Bad ideas. Don't necessarily die because it's actually market forces. It's not literal performance because we don't actually know how to measure measure performance and so like yeah death's not the same thing and so ah and so I think that there's this thing to where ah we made the assumption like initially you know. 01:51:25.27 Jamie Williams We don't have a diverse environment. Yeah. 01:51:38.97 Jared Atkinson You could say evolution made the assumption. Ah, the assumption of what the optimal like visual acuity for human the human species was but over time that got the bad assumption got weeded out into a good assumption is the the general idea and so the the problem is is that we made an assumption. But it's never been weeded out and so like eventually like we don't even think about what what the best type of vision is to have really like we don't really think about that as humans because like that's all been taken care of for us over you know millions of years or whatever. Um, but in in infosec. Ah, we don't think about it because we've just forgotten. But there's no actual pressure or force that's actually helping us make that better over time. So like you have to take an my my perception is that perspective is that you have to take an active role in trying to make that better because the passive role doesn't work the same way as it does for. Like biological in the biological realm you could say. 01:52:34.12 Jonathan Johnson Yeah, we got to ah wrap this up I think because I think Luke's got to head out here a little bit. So do I we're going along. Um I will say last thing I'll say before we wrap up is um that your conversation about movement. 01:52:39.97 Jared Atkinson Options guy. 01:52:50.90 Jonathan Johnson Jared reminded me of a quote that ding jo washington said that's one of my favorite quotes don't confuse movement with progress boom. 01:52:56.47 Jamie Williams Ah, data and. 01:52:56.62 Jared Atkinson Okay, boom all right. That's good cool. Thanks Jamie appreciate you part 2 well sounds like we need part three so work on. Yeah. 01:53:06.70 Jonathan Johnson Yeah, Jamie thanks so much man. 01:53:09.14 Jamie Williams But thanks for having me this is a blast. Can't think of a better way to spend a Friday afternoon. 01:53:14.19 Jonathan Johnson Um, ah. 01:53:14.66 Jared Atkinson It do all right? cool. Well thanks, everybody who made it this far. Thanks for listening and Johnny and Luke go do whatever it is that you got to do and I'm going to get back to work I suppose.