Skip to main content
LawHub
Sign In
About LawHub Advantage Thinking About a Career in Law Applying to Law School Thriving in Law School and Beyond Listen to I Am The Law Podcast Trends Sign In
Search

Goodbye World

Mar 9, 2025
Listen to this episode

We are digging into a superpower inside your Linux Kernel. How eBPF works, and how anyone can take advantage of it.

Sponsored By:

Support LINUX Unplugged

Links:

Transcript

WEBVTT 00:00:00.005 --> 00:00:04.085 Enhanced BPF is another hotness of Linux for the last couple of years. 00:00:04.945 --> 00:00:09.105 And when the patches were first added to Linux, the lead developer, 00:00:09.205 --> 00:00:13.265 Alexei Storitov, said this allows you to do crazy things. 00:00:13.765 --> 00:00:17.485 This is normally the words you don't tell Linus when you want him to accept 00:00:17.485 --> 00:00:20.905 patches into the kernel, but fortunately the patches were accepted. 00:00:21.585 --> 00:00:25.645 Enhanced BPF puts a virtual machine in the kernel that we can program from user space. 00:00:36.845 --> 00:00:41.605 Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. 00:00:41.745 --> 00:00:42.565 My name is Wes. 00:00:42.745 --> 00:00:43.465 And my name is Brent. 00:00:44.125 --> 00:00:48.425 Well, hello, gentlemen. Today, we're digging into a superpower that is inside 00:00:48.425 --> 00:00:50.085 all of our Linux kernels. 00:00:50.305 --> 00:00:54.945 We're going to talk about how eBPF works and how anyone can take advantage of it. 00:00:55.065 --> 00:00:57.365 Then we're going to round out the show with some great feedback, 00:00:57.365 --> 00:01:02.145 some picks, and a lot more. It is a special out-of-time episode. 00:01:02.145 --> 00:01:06.525 As you listen to this right now, we are at Planet Nix and Scale. 00:01:06.765 --> 00:01:08.505 So this is an episode in between. 00:01:08.965 --> 00:01:12.245 You're going to get all our Planet Nix and Scale coverage coming soon, 00:01:12.245 --> 00:01:17.105 but we wanted to take a moment in between episodes and do something kind of 00:01:17.105 --> 00:01:19.265 fun and really dig in and get technical. 00:01:20.085 --> 00:01:23.305 But first, I want to say a big good morning to our friends at TailScale. 00:01:23.945 --> 00:01:28.505 TailScale.com slash unplugged. They are the easiest way to connect your devices 00:01:28.505 --> 00:01:31.325 and services to each other wherever they are. 00:01:31.525 --> 00:01:35.705 It is modern networking. It's a flat mesh network that is protected by... 00:01:35.705 --> 00:01:36.565 Waggaard. 00:01:36.765 --> 00:01:42.725 That's right. And it is so fast, so quick to get going, and it gives you superpowers. 00:01:43.025 --> 00:01:46.145 Not only do you get a flat mesh network across complex networks, 00:01:46.245 --> 00:01:49.465 so maybe you have multiple data centers or VPSs, or you've got mobile devices, 00:01:49.525 --> 00:01:53.745 or you've got double carrier grade NAT, it'll smooth all of that out. 00:01:54.025 --> 00:01:57.945 That's fantastic. But then there's also a whole suite of tools that make it 00:01:57.945 --> 00:01:58.965 really convenient to use. 00:01:59.105 --> 00:02:02.505 Sort of like AirDrop for your entire Tailnet, including like your Android device 00:02:02.505 --> 00:02:04.265 and your Linux device, so you can send files around. 00:02:04.465 --> 00:02:07.485 They will manage your SSH keys through your Tailnet for you, 00:02:07.485 --> 00:02:10.285 so you can just log into all your individual devices. You don't have to manually 00:02:10.285 --> 00:02:11.925 copy keys around like an animal. 00:02:12.405 --> 00:02:16.585 And they also offer more advanced features, so you can set up ACLs and really 00:02:16.585 --> 00:02:19.525 manage the system and lock down only certain things to certain people. 00:02:19.765 --> 00:02:23.265 And when you try it now, when you go to tailscale.com slash unplugged, 00:02:23.365 --> 00:02:26.125 you get it for free up to 100 devices and three users. 00:02:26.765 --> 00:02:30.405 No credit card required. I mean, you can really cook with 100 devices, 00:02:30.405 --> 00:02:33.545 and then maybe you'll discover it's great to bring to work too. 00:02:33.665 --> 00:02:36.185 Thousands of companies like Instacart, Hugging Face, Duolingo, 00:02:36.325 --> 00:02:40.385 Jupyter Broadcasting, and many others use Tailscale, and we love it. 00:02:40.565 --> 00:02:43.545 Try one for yourself. Go get yourself a little Tailscale right now. 00:02:43.725 --> 00:02:45.985 You're going to love the way it tastes, and you're going to love how easy it 00:02:45.985 --> 00:02:49.545 is to get going. If you've got five minutes, you'll probably get it running on three devices. 00:02:49.785 --> 00:02:55.465 I have no inbound ports on any of my firewalls. Talescale.com slash unplugged. 00:02:58.187 --> 00:03:02.207 Well, like I mentioned, we are at Planet Nix and Scale right now. 00:03:02.907 --> 00:03:09.627 But we did think it was sort of a perfect out-of-time episode because we really 00:03:09.627 --> 00:03:15.147 first got excited about eBPF at Scale back in 2019? 00:03:15.707 --> 00:03:17.207 Yeah, a million years ago. 00:03:17.627 --> 00:03:25.327 And it was a great presentation that really just brought home to us how powerful this was going to be. 00:03:25.587 --> 00:03:28.947 Yeah, you heard from the man himself, Brendan Gregg. observability, 00:03:29.187 --> 00:03:33.367 performance, tracing, guru, eBPF was still kind of new then. 00:03:33.607 --> 00:03:38.087 He's kind of well-known. You've maybe seen his famous video online using D-Trace 00:03:38.087 --> 00:03:40.667 to show how hard drives don't like it when you yell at them. 00:03:41.567 --> 00:03:46.207 So, you know, has deep insight into this area and was, even in 2019 and earlier, 00:03:46.447 --> 00:03:48.947 was already getting excited about eBPF. 00:03:49.067 --> 00:03:55.407 So it's kind of neat to look back now as like a whole giant marketplace of eBPF-based, 00:03:56.147 --> 00:03:57.387 observability tools now exist. 00:03:57.387 --> 00:04:01.367 And the name is sort of a misnomer, right? Because it sounds like a packet filter. 00:04:01.927 --> 00:04:04.927 And so you think, well, how? Okay, what is this a firewall, guys? 00:04:05.167 --> 00:04:08.527 No, no, no. That's why we wanted to play that intro clip. It is so much more. 00:04:08.967 --> 00:04:13.887 It is really a VM inside the kernel that can run simple code that you can create 00:04:13.887 --> 00:04:15.047 and craft that is protected. 00:04:15.867 --> 00:04:18.747 And so since this is a prerecord, we're not going to have your boost this week, 00:04:18.807 --> 00:04:21.947 but we do want to know if you like this type of deep dive into this particular topic. 00:04:22.207 --> 00:04:24.607 So boost those in and we'll bank them for when we come back. 00:04:25.127 --> 00:04:28.067 But let's talk about the extended Berkeley packet filter. 00:04:28.427 --> 00:04:32.207 Yeah. So it does do some networking stuff still to this day. 00:04:32.527 --> 00:04:39.167 But it did, as you say, start out as a packet filter introduced in 1992 to efficiently 00:04:39.167 --> 00:04:42.927 filter network packets in BSD operating systems. and. 00:04:42.927 --> 00:04:46.387 Pf sense and other firewall products like open sense of you know they've been 00:04:46.387 --> 00:04:51.367 using bpf as part of their core product i that's one of the reasons i was an early pf sense user. 00:04:51.367 --> 00:04:54.927 And you know before too long within the same decade it made its way over to 00:04:54.927 --> 00:04:58.407 linux in the form of tcp dump and already you're seeing this thing right where 00:04:58.407 --> 00:05:03.307 you can kind of use user space to help better observe what's going on on your system, 00:05:04.185 --> 00:05:07.545 And so if you've ever written sort of an expression to filter things or look 00:05:07.545 --> 00:05:11.205 at packets using TCP dump, well, you're using a language that then gets compiled 00:05:11.205 --> 00:05:14.205 down to BPF bytecode and executed. 00:05:14.565 --> 00:05:17.645 That bytecode right there is kind of the magic, right? Because it turns out 00:05:17.645 --> 00:05:21.025 that this thing is essentially capable of running this bytecode. 00:05:21.085 --> 00:05:22.105 So it's not just a packet filter. 00:05:22.565 --> 00:05:25.645 Yeah, and that's like how the implementation works. As it started out, 00:05:25.745 --> 00:05:27.265 it was a very simple virtual machine. 00:05:27.445 --> 00:05:32.045 And don't think virtual machine like QEMU necessarily are like simulating a full computer. 00:05:32.045 --> 00:05:35.665 the point is it's like a very limited restricted bytecode 00:05:35.665 --> 00:05:39.245 that can only do certain things relevant to at first filtering packets 00:05:39.245 --> 00:05:42.245 and that lets you make sure that like it's not going to do anything crazy it 00:05:42.245 --> 00:05:45.165 can't go into infinite loops all kinds of other nice things and optimize 00:05:45.165 --> 00:05:47.965 it and then be able to load it 00:05:47.965 --> 00:05:50.885 in and have you know you run the program it supplies 00:05:50.885 --> 00:05:53.785 the packet and anything else that you need is input to it 00:05:53.785 --> 00:05:56.645 and then the machine executes and ultimately that's how you tell like 00:05:56.645 --> 00:06:00.145 do i accept the packet or do i drop the packet uh but 00:06:00.145 --> 00:06:03.665 at first it was a you know a very limited 00:06:03.665 --> 00:06:06.845 i think i had like a two two registers to use super limited thing 00:06:06.845 --> 00:06:14.685 but ebpf extended bpf was introduced in linux 318 this was like 2014 so bpf 00:06:14.685 --> 00:06:18.905 had been around for a while there'd been various developments but ebpf really 00:06:18.905 --> 00:06:23.265 kicked things off in the linux side of things bpf hadn't you know caught up 00:06:23.265 --> 00:06:26.625 with the times in some ways it was still 32 bit it had some of those register limitations. 00:06:27.185 --> 00:06:30.145 So they upgraded to 64 bit registers added more 00:06:30.145 --> 00:06:33.165 instructions they added the verifier to 00:06:33.165 --> 00:06:36.285 the kernel which is a big part of it that lets you analyze bpf programs 00:06:36.285 --> 00:06:40.145 to make sure they're safe no infinite loops they don't do invalid memory access 00:06:40.145 --> 00:06:43.785 there's a checker process right because it's like you're loading something from 00:06:43.785 --> 00:06:48.085 user space into the kernel that's a big security concern so you you want to 00:06:48.085 --> 00:06:51.505 make sure that you have and just be on security operationally too right you 00:06:51.505 --> 00:06:53.225 don't want it to be able to crash the kernel so. 00:06:53.225 --> 00:06:57.285 We actually started to see this stuff land you know progressively in linux 3 00:06:57.285 --> 00:07:02.525 18 and beyond so this is actually like you said 2014 that's a this has been landing for a while. 00:07:02.525 --> 00:07:07.405 Yeah definitely and then so that was kind of like the raw stuff and then 2018 00:07:07.405 --> 00:07:12.605 and beyond people started adding more tools like the bcc but the bpf compiler 00:07:12.605 --> 00:07:18.225 collection bpf trace which we'll talk about too uh the company psyllium and 00:07:18.225 --> 00:07:20.845 their their products have a bunch of like ebpf stuff for, 00:07:22.105 --> 00:07:26.405 Kubernetes offerings plus we got better like compiler support there's something 00:07:26.405 --> 00:07:31.265 called Kori or compile once run everywhere so like compilers have better support 00:07:31.265 --> 00:07:34.645 to be able to make you know you can compile your BPF program and not have to 00:07:34.645 --> 00:07:37.585 worry about as much necessarily depending on what you're doing with it about 00:07:37.585 --> 00:07:41.765 how compatible it'll be with the kernel where you compiled it versus where you're running it so. 00:07:41.765 --> 00:07:46.365 The idea being like it should be compatible across multiple versions of Linux 00:07:46.365 --> 00:07:48.585 as long as they have this correct implementation. 00:07:48.585 --> 00:07:56.305 Right okay This has been so successful that newer versions of Dtrace on Linux, 00:07:58.081 --> 00:08:02.061 They're basically just some extra user space stuff that uses eBPF and other 00:08:02.061 --> 00:08:03.321 kernel primitives under the hood. 00:08:03.501 --> 00:08:04.841 Really? Yeah. Huh. 00:08:05.081 --> 00:08:07.481 And eBPF was also ported to Windows. 00:08:07.821 --> 00:08:11.501 Yeah, I heard about this. They're getting a lot of good stuff over there. 00:08:11.681 --> 00:08:16.401 One important thing with the extended part is they were also able to make the 00:08:16.401 --> 00:08:21.081 instruction set be more sympathetic to modern hardware, and they implemented 00:08:21.081 --> 00:08:22.681 just-in-time compilation as well. 00:08:22.721 --> 00:08:25.901 So that meant eBPF can be really fast. 00:08:26.461 --> 00:08:29.281 They've also made it so that there's relatively stable APIs we've kind of been 00:08:29.281 --> 00:08:33.921 talking about, so that as kernels change that, you can use eBPF to hook into internals. 00:08:34.361 --> 00:08:37.501 If you do that, there's no guarantees, right? But that's kind of on the tin. 00:08:37.841 --> 00:08:41.461 But there is some stable interfaces that you can use, which is nice. 00:08:42.901 --> 00:08:46.641 So the other important point to know is there's kind of various things it can do. 00:08:47.461 --> 00:08:52.221 There's XDP, which is Express Data Path, which intercepts packets basically 00:08:52.221 --> 00:08:54.781 at like the earliest point that you can. 00:08:54.781 --> 00:08:59.761 you have limitations on what you can do but it can be super low level and performant 00:08:59.761 --> 00:09:03.041 and so you can see this sometimes maybe unlike responding to ddos attacks where 00:09:03.041 --> 00:09:06.801 you're getting flooded with traffic that you have you know maybe you can identify 00:09:06.801 --> 00:09:10.461 various ways that you can program in here so you could essentially. 00:09:10.461 --> 00:09:14.401 Ideally catch it before it begins to overwhelm the system because you're catching 00:09:14.401 --> 00:09:16.281 it much further in like the driver stack. 00:09:16.281 --> 00:09:19.881 Yeah that's the idea and it doesn't do as much as like the very general and 00:09:19.881 --> 00:09:23.901 powerful but you know full-featured linux kernel networking stack you can kind 00:09:23.901 --> 00:09:26.381 of just be like, well, no, if it looks like this at all, just cut it off immediately. 00:09:26.561 --> 00:09:29.581 Yeah, if I'm getting DDoSed, I don't want the network stack trying to figure 00:09:29.581 --> 00:09:31.861 out what to do with all this traffic because that's what's going to take me down. 00:09:31.981 --> 00:09:34.701 Yeah, one way to say it is limited context but maximum performance. 00:09:35.641 --> 00:09:36.421 I like that. 00:09:37.081 --> 00:09:39.981 Okay, so then you can also do various types of K-probes. 00:09:40.221 --> 00:09:40.601 I'm sorry? 00:09:40.961 --> 00:09:46.281 Yeah, K-probes. And this is dynamic instrumentation. That's why it sounds quite probing. 00:09:46.601 --> 00:09:49.041 You can hook almost any kernel function. 00:09:50.861 --> 00:09:54.421 but you don't there's no like clear definition of what you're going to get you 00:09:54.421 --> 00:09:57.081 have to go look at the function you're hooking into it's all going to be dependent 00:09:57.081 --> 00:10:01.381 on that but no kernel internals any kernel function almost i'm sure there are 00:10:01.381 --> 00:10:03.561 some limitations that's pretty a lot of them yeah i. 00:10:03.561 --> 00:10:08.501 Mean that could be like from you know keyboard input to network traffic to disk 00:10:08.501 --> 00:10:10.381 io i mean that's there's all kinds of things. 00:10:10.381 --> 00:10:13.721 There so that's where a lot of some of the power comes from yeah but that may 00:10:13.721 --> 00:10:16.381 or may not be stable there's no guarantee about it being stable across kernel 00:10:16.381 --> 00:10:18.761 versions, can change at any time. People can update the signatures. 00:10:19.241 --> 00:10:22.541 That's one of the things that's been happening in the Rust discussion is the 00:10:22.541 --> 00:10:26.361 kernel, many developers expect to be able to make a change like that and because 00:10:26.361 --> 00:10:29.761 they have one big code base, they can update it everywhere and be able to do 00:10:29.761 --> 00:10:32.721 a refactoring like that. And, but the other one, 00:10:33.548 --> 00:10:39.408 Trace points. This is an important one. Trace points. Predefined stable instrumentation. 00:10:39.688 --> 00:10:39.868 Ah. 00:10:40.048 --> 00:10:44.968 Low overhead, but only available where explicitly added by kernel developers. 00:10:45.248 --> 00:10:49.968 So, okay. So a trace point would be like a spot you hook in and start getting 00:10:49.968 --> 00:10:51.588 metrics or information out of. 00:10:51.888 --> 00:10:56.108 But the developer of the subsystem has to implicitly support that. Yeah. 00:10:56.168 --> 00:10:58.748 You have to add that in versus totally dynamic with a K-probe. 00:10:58.848 --> 00:10:58.988 Okay. 00:10:59.288 --> 00:11:02.988 But the upside is you get a structured data specific to each trace point, 00:11:03.048 --> 00:11:05.008 right? So the trace point tells you what it is. 00:11:05.168 --> 00:11:05.968 Yeah, yeah, okay. 00:11:06.128 --> 00:11:10.368 And they're maintained across kernel versions, so you can kind of rely on them for more long-term use. 00:11:10.748 --> 00:11:13.708 I have a feeling this might be relevant later. Okay, good to know. 00:11:14.128 --> 00:11:18.488 Yeah, I think that's probably like the quick, high-level version of what eBPF 00:11:18.488 --> 00:11:19.808 is and kind of what it can do. 00:11:19.948 --> 00:11:26.488 And it is so, so, so simple, but yet so powerful is what I love about it. 00:11:26.568 --> 00:11:31.588 And we have a couple of examples in the show, and I think some hands-on stuff 00:11:31.588 --> 00:11:32.508 that people could take away. 00:11:33.128 --> 00:11:35.948 And maybe we should start with BCC tools themselves. 00:11:36.388 --> 00:11:39.188 Yeah, because we've both had a chance to play with those. 00:11:39.328 --> 00:11:42.648 Yeah, I was, you know, I was joking around with Wes and be like, 00:11:42.708 --> 00:11:48.168 it'd be kind of great to know where in the system when I open up this directory 00:11:48.168 --> 00:11:49.468 that's a fuse mounted directory, 00:11:49.688 --> 00:11:54.568 like where is the actual delay happening and what part of the system is sitting around waiting. 00:11:54.568 --> 00:11:59.108 And through this process of trying to get to that, I came across tools that 00:11:59.108 --> 00:12:03.508 let me look at my disk IO analysis or let me look at the network traffic and 00:12:03.508 --> 00:12:06.688 really kind of using these different tools, putting them together. 00:12:06.688 --> 00:12:11.728 You can actually start to get a really good picture of where the delay was happening in the system. 00:12:12.523 --> 00:12:15.623 And, you know, for me, it was like, this is going to be amazing. 00:12:16.043 --> 00:12:21.603 I had discovered by running it on my desktop, just as when I was experimenting 00:12:21.603 --> 00:12:25.263 with this stuff, that, oh, yeah, I've still got this errands app that we talked 00:12:25.263 --> 00:12:27.243 about on the show that I'm not currently using like today. 00:12:27.283 --> 00:12:31.123 But I guess when I close it, it doesn't actually close. It doesn't leave an icon. 00:12:31.303 --> 00:12:36.863 I had no idea it was running. But using these tools, I started seeing this application 00:12:36.863 --> 00:12:40.483 that was hitting my disk every so often. And I'm like, I don't recognize this. 00:12:40.483 --> 00:12:42.963 and I discovered I actually had that running the entire time. 00:12:43.103 --> 00:12:44.063 And it was kind of useful. 00:12:44.303 --> 00:12:48.983 Yeah, it kind of skips through a lot of the boundaries and other limitations 00:12:48.983 --> 00:12:51.723 that can sometimes pop up when you're trying to look at your system. 00:12:51.983 --> 00:12:53.743 So it can be surprisingly insightful. 00:12:54.403 --> 00:12:57.843 They do have a nice tutorial, which we'll link. They also, I like that they 00:12:57.843 --> 00:12:59.303 have a set of things to run first. 00:12:59.443 --> 00:13:02.883 Like before you go to these tools, make sure you check all like the regular 00:13:02.883 --> 00:13:06.403 system monitoring tools first because we'll see as a theme, like, 00:13:06.903 --> 00:13:09.823 you know, your H-tops and B-tops and all kinds of things. 00:13:10.483 --> 00:13:14.483 kind of get a broad look and you could see some specifics whereas you could 00:13:14.483 --> 00:13:18.683 you can make broad eppf programs but by default they're going to be a lot more 00:13:18.683 --> 00:13:21.043 specific when you're looking at like one thing. 00:13:21.043 --> 00:13:21.743 Like disk. 00:13:21.743 --> 00:13:23.743 Latency or something specific to the file system. 00:13:23.743 --> 00:13:29.003 Or networking yeah yeah that's a good point um all right so we could talk about 00:13:29.003 --> 00:13:33.663 some of the commands we tried i know that uh file top and tcp top we played 00:13:33.663 --> 00:13:39.363 around with um i did not get a chance to play around with ext4 slower and XFS 00:13:39.363 --> 00:13:41.643 slower, but this is an interesting way you can use it too. 00:13:41.883 --> 00:13:45.223 Uh-huh, yeah. It'll tell you when, you know, there's things happening in your 00:13:45.223 --> 00:13:49.143 file system that are causing latency more than, like, a programmable amount 00:13:49.143 --> 00:13:50.603 you can pass on the command line. 00:13:51.363 --> 00:13:56.503 Or they've got ZFS dist, which traces ZFS reads, writes, opens, 00:13:56.683 --> 00:14:00.383 fsyncs, and then summarizes the latency of all that in a histogram for you. 00:14:00.503 --> 00:14:01.363 That's really cool. 00:14:01.823 --> 00:14:07.943 Yeah, right? And then two really useful basic ones are exec snoop and open snoop. 00:14:08.799 --> 00:14:14.099 Yeah, OpenSnoop, isn't that kind of like, oh, there's like a Mac app that monitors 00:14:14.099 --> 00:14:16.679 traffic? Is that what it was? OpenSnoop something else? 00:14:16.859 --> 00:14:19.319 Yeah, no. So OpenSnoop, you're thinking of, there's a Linux one, 00:14:19.399 --> 00:14:21.459 I think, right? OpenSnitch? Yeah, just one thing. 00:14:22.039 --> 00:14:23.379 Snitch. Snitch. That's, yes. 00:14:24.759 --> 00:14:26.479 OpenSnoop tracks file opens. 00:14:26.679 --> 00:14:26.879 Okay. 00:14:27.059 --> 00:14:32.079 So you can run this and it hooks into the kernel via BPF. And then anything 00:14:32.079 --> 00:14:35.259 that's running on your system that opens files, it'll just print out in your 00:14:35.259 --> 00:14:36.499 console right in front of you. 00:14:36.519 --> 00:14:38.199 I did play with this one. I did play with this OpenSnoop one. 00:14:38.279 --> 00:14:41.359 You're right. Yeah, that's really cool because I remember I launched, 00:14:41.479 --> 00:14:44.659 you could launch particular things and then just watch all the crap that happens on your system. 00:14:44.759 --> 00:14:47.719 Right. And especially, you know, you can filter it, you can have it look at 00:14:47.719 --> 00:14:50.419 specific processes or you could, you know, grab output or whatever. 00:14:50.859 --> 00:14:54.499 So you can filter on a busy system, but I think it's especially useful on a 00:14:54.499 --> 00:14:58.139 system you think should be, you know, relatively quiet and just as a way to 00:14:58.139 --> 00:15:00.659 see, like, what is actually happening in the background? 00:15:01.879 --> 00:15:07.179 And for short-lived processes that might be hard to see in a summary-type program 00:15:07.179 --> 00:15:09.999 that are still doing things, it'll still print in OpenSnoop. 00:15:10.499 --> 00:15:13.719 That's where ExecSnoop does a similar thing, but for process opens. 00:15:14.299 --> 00:15:17.279 So again, for things where it's hard to see in a general tool, 00:15:17.379 --> 00:15:21.099 but you really want to see at a nitty-gritty level what is happening in being 00:15:21.099 --> 00:15:24.459 Exec on this box, ExecSnoop is handy for that. 00:15:25.059 --> 00:15:30.899 I have a sense Exec Snoop has come up actually in real life for you as a handy tool. 00:15:31.059 --> 00:15:31.139 Yeah. 00:15:31.219 --> 00:15:32.719 Do you want to share a war story with us? 00:15:32.719 --> 00:15:38.579 I do. So a few years ago, I was adminning a VPS box. I wasn't totally in charge 00:15:38.579 --> 00:15:42.759 of, but had some, you know, administration over. And... 00:15:44.289 --> 00:15:48.289 Things were seemingly broken. I'd noticed some messed up statistics first, 00:15:48.409 --> 00:15:49.629 right? The metrics were a little odd. 00:15:49.869 --> 00:15:53.889 And I went on and I started just kind of doing regular updates and poking around at the system. 00:15:54.249 --> 00:15:57.749 And I noticed that I wasn't able to actually update the initRAMFS, 00:15:58.069 --> 00:16:01.529 right? I was doing all the, it was an Ubuntu box. I had everything going. 00:16:01.689 --> 00:16:08.689 But there was some like dynamic library linking problem that was happening after 00:16:08.689 --> 00:16:11.529 I started digging into the output and tried rebuilding it way too many times. 00:16:11.529 --> 00:16:13.529 And I'm like, what, why? Why can't I get it? 00:16:13.529 --> 00:16:15.549 Yeah, well, I can't get my initRAMFS updated. 00:16:15.789 --> 00:16:18.149 The kernel's here. Everything else in the update's fine. 00:16:18.229 --> 00:16:20.589 But you can't boot in that new kernel if you can't get that updated. 00:16:20.809 --> 00:16:26.349 Yeah. So this was maybe closer to when I really started playing with these tools 00:16:26.349 --> 00:16:28.889 a little bit more seriously and having them installed on more boxes. 00:16:29.189 --> 00:16:30.589 And so they were already available. 00:16:31.689 --> 00:16:37.049 And I used OpenSnoop because I wanted to see exactly what was being opened and 00:16:37.049 --> 00:16:39.649 touched by the update initRAMFS process. 00:16:39.889 --> 00:16:40.149 Right. 00:16:40.149 --> 00:16:43.769 And then that led me to look at some weird file paths. 00:16:44.209 --> 00:16:51.609 And I also started, then I did exec snoop, and that showed similar file paths running on the system. 00:16:51.829 --> 00:16:53.549 Oh, uh-oh. 00:16:53.729 --> 00:16:57.809 And then I was able to figure out that there was, in fact, a crypto miner running on the box. 00:16:57.909 --> 00:16:58.329 Oh, no. 00:16:58.529 --> 00:17:05.649 But it had loaded a custom kernel module to hide itself from tools like PS and top and htop. 00:17:05.749 --> 00:17:06.429 That's pretty slick. 00:17:06.549 --> 00:17:07.689 But not from exec snoop. 00:17:08.209 --> 00:17:11.809 Clever. So that's why it was breaking when you were trying to update in that in an FS. 00:17:12.029 --> 00:17:15.789 Yes. Something it did to the system. I never quite figured out exactly what 00:17:15.789 --> 00:17:19.529 it had changed, but it had messed with some of the files on the system in ways 00:17:19.529 --> 00:17:21.769 that meant that the linking was no longer cleanly happening. 00:17:21.929 --> 00:17:21.989 Yeah. 00:17:22.844 --> 00:17:26.724 So that's a good way to stack those two tools. 00:17:26.884 --> 00:17:31.004 You know, OpenSnoop and ExecSnoop are really a couple of nice tools you can stack together. 00:17:31.404 --> 00:17:36.044 And there's a whole bunch of the, this is part of BCC, the BPF compiler collection, 00:17:36.784 --> 00:17:38.264 which includes a whole bunch of other stuff. 00:17:38.344 --> 00:17:42.784 But these are the tools that they ship by default, which leverages the framework 00:17:42.784 --> 00:17:45.984 they've built to implement these via eBPF. 00:17:46.084 --> 00:17:49.044 And so there's like, yeah, all kinds of stuff, file system specific stuff. 00:17:49.224 --> 00:17:51.804 They've got things for, you know, looking at network connections, 00:17:51.964 --> 00:17:55.244 TCP connections. They've got stuff for monitoring databases. It's broad. 00:17:55.664 --> 00:17:59.524 I don't know if it's necessarily best practice, but you could say that with 00:17:59.524 --> 00:18:03.704 these tools, you could be fairly confident that you had cleaned up the crypto 00:18:03.704 --> 00:18:04.844 miner and the infection. 00:18:05.624 --> 00:18:09.984 You know, because you can actually watch at a much more intimate level what's 00:18:09.984 --> 00:18:10.804 happening at the system. 00:18:10.944 --> 00:18:11.904 Definitely useful. Yeah. 00:18:12.124 --> 00:18:14.884 I'm not saying it's probably a best practice, you know, but probably just wipe 00:18:14.884 --> 00:18:18.804 the box. Yeah, but it is sort of nice if for some reason you don't have that 00:18:18.804 --> 00:18:22.744 option, you can use these tools to kind of confirm that stuff isn't coming back after a reboot. 00:18:23.004 --> 00:18:24.684 Right. And the more, you know, depending 00:18:24.684 --> 00:18:28.704 on, but you can see exactly where they're hooking and modify the, 00:18:28.844 --> 00:18:33.104 because a lot of this is implemented via a combo of C because that's the part 00:18:33.104 --> 00:18:36.944 that gets compiled and like it's like a limited subset of C that does the BPF 00:18:36.944 --> 00:18:40.144 stuff that gets converted into a loadable thing for the kernel. 00:18:40.144 --> 00:18:45.784 but there's a bunch of python utilities around it to wrap it so you could fork 00:18:45.784 --> 00:18:50.724 that copy it modify it if you need it even more or to try to observe specific 00:18:50.724 --> 00:18:54.724 things once you identified like a particular problem or threat now. 00:18:54.724 --> 00:18:55.584 That's really going deep, 00:18:59.061 --> 00:19:06.241 onepassword.com slash unplugged. That's the number one password.com slash unplugged, all lowercase. 00:19:06.421 --> 00:19:10.241 And you're going to want to go there because this is something that if I had, 00:19:10.321 --> 00:19:14.621 when I still worked in IT, I think it would have sustained me for many, many more years. 00:19:14.781 --> 00:19:19.421 The reality is your end users don't, and I mean, without exception, 00:19:19.781 --> 00:19:22.601 work on only company-owned devices, applications, and services. 00:19:22.981 --> 00:19:27.441 Maybe you get lucky and they mostly do, but I don't find that to be the reality today. 00:19:27.861 --> 00:19:31.941 And so the next question becomes, how do you actually keep your company's data 00:19:31.941 --> 00:19:35.641 safe when it's sitting on all of these unmanaged apps and devices? 00:19:36.681 --> 00:19:40.241 Well, that's where 1Password has an answer. It's extended access management. 00:19:40.461 --> 00:19:45.101 1Password extended access management helps you secure every sign-on for every 00:19:45.101 --> 00:19:49.961 app on every device because it solves the problems that your traditional IAMs 00:19:49.961 --> 00:19:51.901 and MDMs just can't touch. 00:19:52.241 --> 00:19:57.721 And it's the first security solution that brings unmanaged devices and applications 00:19:57.721 --> 00:20:01.701 and identities, like even vendor, under your control. 00:20:02.161 --> 00:20:05.661 It ensures that every credential is strong and protected, every device is known 00:20:05.661 --> 00:20:08.261 and healthy, and every app is visible. 00:20:08.921 --> 00:20:12.821 This is some powerful stuff, and it's available for companies that use Okta 00:20:12.821 --> 00:20:16.241 or Microsoft Entra, and it's in beta for Google Workspace customers too. 00:20:17.041 --> 00:20:20.561 OnePassword changed the game for password management, and now they're taking 00:20:20.561 --> 00:20:24.921 everything they've learned there and expanding it to the login level and the application level. 00:20:25.161 --> 00:20:28.741 You know what a difference it makes when people have proper password management. 00:20:28.921 --> 00:20:32.241 Now let's have proper login management and authorization. 00:20:32.921 --> 00:20:38.001 1Password also has regular third-party audits and the industry's largest bug 00:20:38.001 --> 00:20:39.981 bounty. They exceed the standards set by others. 00:20:40.321 --> 00:20:44.501 They really do. So go secure every app, every device, and every identity. 00:20:44.681 --> 00:20:49.521 Even the unmanaged ones. You just go to 1Password.com slash unplugged. 00:20:49.641 --> 00:20:52.321 All over case. that's onepassword.com. 00:20:52.321 --> 00:20:54.061 Slash unplugged, 00:20:56.501 --> 00:21:02.261 now Wes it sounds like BCC is an abstraction to EBPF what's going on under the hood here. 00:21:02.261 --> 00:21:05.061 Yeah so there are tools that you 00:21:05.061 --> 00:21:09.221 can just run like we've been talking about but how did those tools come to be 00:21:09.221 --> 00:21:13.921 well that's where some of those abstractions and a framework comes in that allows 00:21:13.921 --> 00:21:19.561 basically embedding C code directly within Python scripts and then And BZC's 00:21:19.561 --> 00:21:23.441 tools also sort of handle making sure you've got the whole tool chain available. 00:21:23.441 --> 00:21:28.181 So it uses LLVM and Clang to compile things. It handles verification of stuff. 00:21:28.361 --> 00:21:31.501 It handles loading it into the kernel for you and attaching it to the right 00:21:31.501 --> 00:21:35.341 hooks, basically through like Python method calls instead of you having to run 00:21:35.341 --> 00:21:37.381 the right commands in the bash shell. 00:21:37.981 --> 00:21:41.121 So you kind of get like a unified approach where you can write kernel level 00:21:41.121 --> 00:21:44.381 programs without necessarily having to deal with like all the other stuff. 00:21:44.381 --> 00:21:48.101 You know I was thinking gosh this really seems like going way beyond what my 00:21:48.101 --> 00:21:50.381 skill set would be able to manage but then I thought, 00:21:51.299 --> 00:21:56.959 python code i bet an llm would get me 90 of the way there nice these days and 00:21:56.959 --> 00:21:58.239 then i could probably just finish it off. 00:21:58.239 --> 00:22:01.059 And there's a fair amount of like existing tools that you 00:22:01.059 --> 00:22:03.759 can either feed into an llm and ask about or 00:22:03.759 --> 00:22:07.979 modify or try and you know hack on yourself yeah and so here's a simple example 00:22:07.979 --> 00:22:12.879 that we can play with it uses xdp okay and so there's a simple c program that 00:22:12.879 --> 00:22:17.459 has a function called xdp drop all okay uh and that's going to attach to the 00:22:17.459 --> 00:22:21.479 xdp hook and so we get like a little data summary of the packet info, 00:22:21.639 --> 00:22:22.619 which we're not going to care about. 00:22:22.859 --> 00:22:26.839 All we're going to do is return XDP drop, which is a magic value that tells 00:22:26.839 --> 00:22:28.839 the kernel, hey, just drop this. 00:22:29.079 --> 00:22:29.479 Just drop all. 00:22:29.479 --> 00:22:33.019 So it's going to drop everything. And then in the Python, that's it C-wise. That's all we do. 00:22:33.299 --> 00:22:36.039 And then in the Python world, there's a little setup to make, 00:22:36.119 --> 00:22:37.579 hey, I want a new BPF thing or whatever. 00:22:37.939 --> 00:22:41.159 And then we tell it to load in our little blob of C. 00:22:41.679 --> 00:22:44.679 And then we attach it to the interface we want. That's two lines of Python. 00:22:45.959 --> 00:22:49.279 And then it prints a nice little message and starts working. 00:22:49.839 --> 00:22:53.139 And what it should do is drop all network traffic? 00:22:53.359 --> 00:22:55.559 Yeah, so we specified a specific interface. 00:22:56.079 --> 00:22:58.079 Yeah, so you say on this interface, just drop everything. 00:22:58.159 --> 00:23:00.779 And that's because we attached it to that specific interface. 00:23:01.199 --> 00:23:04.899 And this is like a quick kill switch. So what we're going to do as a test here 00:23:04.899 --> 00:23:08.459 is I've set up a ping with an audible sound so we can see when we're getting a result. 00:23:09.279 --> 00:23:13.699 So we can see we're pinging the box right now. And then at some point, Wes is going to kill. 00:23:13.939 --> 00:23:17.719 He's actually going to run that kill script, and it's going to drop all traffic. 00:23:17.899 --> 00:23:19.619 and we'll hear it drop off. Whenever you're ready, Mr. Payne. 00:23:20.259 --> 00:23:22.479 Three, two, one. 00:23:24.039 --> 00:23:30.019 There it goes. Yeah, it takes almost no time at all. It happens almost immediately. 00:23:30.399 --> 00:23:35.159 Now, and of course, in my regular SSH session, I can't stop it, 00:23:35.239 --> 00:23:39.299 but I do have a sneaky console here, so we should be able to take a peek. 00:23:39.299 --> 00:23:40.719 Wes always has a sneaky console. 00:23:41.219 --> 00:23:43.159 All right, I'm leaving the ping going, so if it resumes. 00:23:43.359 --> 00:23:45.459 We should hear it. I'm hitting Control-C now. 00:23:45.839 --> 00:23:48.399 Okay, out there it goes. Look at that. 00:23:49.039 --> 00:23:53.279 Yeah. So like that wasn't that much work. This is just an Ubuntu 24.04 box. 00:23:53.459 --> 00:23:56.679 So like, you know, all the stuff you needed was in the repos already. 00:23:58.385 --> 00:24:02.425 I'm thinking, you know, like that's just a it's it's not a total practical example, 00:24:02.545 --> 00:24:08.185 but it's a quick example of the power you have there and you're executing that from user space. 00:24:08.405 --> 00:24:12.725 Yeah, you do need, you know, root permissions to be able to load it into the kernel. 00:24:12.905 --> 00:24:16.185 But then you're executing things inside the kernel that just immediately cutting 00:24:16.185 --> 00:24:17.625 off traffic to that interface. 00:24:17.805 --> 00:24:22.145 Yes. And I didn't have to build a per kernel kernel module with the right headers 00:24:22.145 --> 00:24:26.425 and like have to worry about messing up my implementation in a way that's going to crash the box. 00:24:26.425 --> 00:24:29.945 You didn't even have to create like a IP tables like that rejects traffic because 00:24:29.945 --> 00:24:33.085 that would actually be much further down in the processing stack if you're using 00:24:33.085 --> 00:24:33.925 something like IP tables. 00:24:34.045 --> 00:24:37.225 Yeah, that's cool. That's fun. 00:24:37.545 --> 00:24:40.485 So in reality, you'd want, right, you would do some sort of filtering where 00:24:40.485 --> 00:24:44.445 you would look at the data structure you get from XDP to figure out like, 00:24:44.745 --> 00:24:47.205 oh, is this one I want to block or let it keep going through processing? 00:24:47.525 --> 00:24:50.965 But yeah, this is your basic. It's not hello world. It's goodbye world. 00:24:51.105 --> 00:24:52.985 Yeah. It's goodbye world. Exactly. 00:24:52.985 --> 00:24:57.945 But the main point to show that was that this BCC framework is one way if you 00:24:57.945 --> 00:24:59.245 want to start developing custom tools. 00:24:59.365 --> 00:25:01.705 But you can get even more ad hoc than that. 00:25:01.785 --> 00:25:02.125 All right. 00:25:02.285 --> 00:25:03.425 Because there's BPF trace. 00:25:03.745 --> 00:25:05.565 Yeah. I want to talk about this. 00:25:05.745 --> 00:25:09.845 And this is basically sort of like the closest thing to D-Trace for Linux if 00:25:09.845 --> 00:25:13.065 you don't want to use the Oracle tool, which I believe Gentoo now has. 00:25:13.565 --> 00:25:19.245 Oh, all right. So BPF trace is sort of what I know of D-Trace is like this ultimate 00:25:19.245 --> 00:25:22.465 tool when you're debugging or trying to figure out where your system has gone sideways. 00:25:22.465 --> 00:25:25.325 again like i was talking about like why is this one fuse directory taking so 00:25:25.325 --> 00:25:28.565 long to open and so i imagine this is a similar type of. 00:25:28.565 --> 00:25:32.085 Yeah one way to think about is you basically get like a nice targeted little 00:25:32.085 --> 00:25:39.045 scripting language to write tracing programs oh that use the trace points in the kernel. 00:25:39.045 --> 00:25:41.925 Wow okay all right. 00:25:41.925 --> 00:25:44.525 And there are some other options there's a program called ply that i 00:25:44.525 --> 00:25:47.305 haven't tried but looks also nice um but bpf trace 00:25:47.305 --> 00:25:50.085 is quite popular uh so we talked about right 00:25:50.085 --> 00:25:53.185 there's different there's xdp there's the k prob stuff these are the trace points 00:25:53.185 --> 00:25:56.885 which is the the easy stable ones because they're defined and they give you 00:25:56.885 --> 00:26:03.385 a known data structure so you can do bpf trace dash l or the bcc tools has tplist 00:26:03.385 --> 00:26:06.905 which is also just lists all the trace points it also shows you can do other 00:26:06.905 --> 00:26:09.845 user space stuff but we're not going to talk about that today so. 00:26:09.845 --> 00:26:13.405 There is so tplist i guess that makes it a lot easier if tplist is going to 00:26:13.405 --> 00:26:17.985 give you all of the kernel trace points so you know what you have to work with that's really useful. 00:26:17.985 --> 00:26:21.065 Right And it tells you the shape of them, too. You get the sort of data structure. 00:26:21.185 --> 00:26:21.425 Okay. 00:26:21.985 --> 00:26:22.265 Um... 00:26:23.624 --> 00:26:25.164 Let me pull it up here. 00:26:26.464 --> 00:26:27.484 He's got it there on the machine. 00:26:28.004 --> 00:26:32.784 Yeah, so here's sort of like, here's some stuff about block dirty buffer. 00:26:32.984 --> 00:26:36.104 And it tells you you get a dev device, you get a sector and a size. 00:26:36.284 --> 00:26:39.224 And so you know those are the things you can work with. And then it has a name 00:26:39.224 --> 00:26:41.884 and you just tell it that you want to trace that thing. 00:26:42.884 --> 00:26:47.404 So this came up because, you know, we talked a little bit about BCC having file system specific tools. 00:26:47.584 --> 00:26:51.464 Like, oh, I want to see watch for slow operations from ButterFS, say, right? 00:26:51.564 --> 00:26:51.684 Right. 00:26:52.044 --> 00:26:56.624 Well, they haven't yet. I suppose I should step up or someone should step up 00:26:56.624 --> 00:27:00.204 and add these, but there isn't yet a BcacheFS version of these tools, right? 00:27:00.324 --> 00:27:01.264 There we go. That'd be cool. 00:27:01.424 --> 00:27:05.064 Obviously, probably what you want to do is just, you know, fork the existing 00:27:05.064 --> 00:27:08.404 ButterFS one and modify it and make it work right for BcacheFS. 00:27:08.624 --> 00:27:08.744 Sure. 00:27:08.944 --> 00:27:13.284 But if you want to be ad hoc about it, BPF Trace can handle this too. 00:27:13.324 --> 00:27:18.204 So I did TP list and then grepped that for things that said BcacheFS. 00:27:18.204 --> 00:27:22.244 So you got all of the kernel interfaces regarding trace points, 00:27:22.444 --> 00:27:24.584 as I should say, trace points regarding BcacheFS. 00:27:25.144 --> 00:27:28.544 So here's just like a list of some of what the trace points look like. 00:27:29.004 --> 00:27:33.364 I see. Okay. So now, okay, so what I'm seeing here on Wes's screen is like an 00:27:33.364 --> 00:27:38.624 output, BcacheFS colon, and I can get data update, rebalance, extent. 00:27:38.884 --> 00:27:41.944 There's a lot of different options here. Essentially, just what is the file system up to? 00:27:41.944 --> 00:27:46.644 And so these would all be things that Kent or team have put in explicitly so 00:27:46.644 --> 00:27:51.744 that there's a way to like easily and with low overhead watch these things. 00:27:51.964 --> 00:27:54.784 So since these trace points already exist, you can use... 00:27:56.807 --> 00:28:00.107 bcc to write your own kind of monitoring. 00:28:00.107 --> 00:28:02.367 Yeah or bpf trace which is right. 00:28:02.367 --> 00:28:03.187 Yeah thank you. 00:28:03.187 --> 00:28:06.307 Um there's a lot of terms there's a lot of bpf terms 00:28:06.307 --> 00:28:09.707 so here's one that stood out uh copy gc 00:28:09.707 --> 00:28:12.467 wait and bcashfs is a copy on write 00:28:12.467 --> 00:28:15.327 file system and it does this bucket based allocation and so 00:28:15.327 --> 00:28:18.047 it has what's called a copying garbage collector so as 00:28:18.047 --> 00:28:20.867 you're copying files it'll handle moving things around so 00:28:20.867 --> 00:28:24.007 that it can then like get good bucket allocation and 00:28:24.007 --> 00:28:28.167 defragment things on the fly that kind of thing so copy 00:28:28.167 --> 00:28:30.927 gc wait is a trace point that tells you when 00:28:30.927 --> 00:28:33.967 you're when bcachefs is waiting for copy gc 00:28:33.967 --> 00:28:36.787 to complete so it can be a reason that your 00:28:36.787 --> 00:28:39.707 file system is being slow or not responding especially if you're moving or copying 00:28:39.707 --> 00:28:46.227 files around so you can see on your screen uh just a little tiny script uh sudo 00:28:46.227 --> 00:28:51.187 bpf trace dash e and then you pass it a little string and we tell it we want 00:28:51.187 --> 00:28:56.267 to use the trace point bcachefs copy gc wait and then we have a little script here that, 00:28:56.947 --> 00:29:01.107 one of the inputs is the device and then it has little stuff that extracts the 00:29:01.107 --> 00:29:04.567 major and minor number from that you don't have to do that and then all it does 00:29:04.567 --> 00:29:09.987 is print what device we're waiting on and how much we're waiting yeah and so 00:29:09.987 --> 00:29:12.047 it's you know i don't know 10 lines of code. 00:29:12.047 --> 00:29:16.747 And you get a nice kind of structured output it's easy to human readable it 00:29:16.747 --> 00:29:19.967 tells you the total flushes the total buffers flushed you get you know the total 00:29:19.967 --> 00:29:22.227 output whatever whatever the stat is you're watching. 00:29:22.227 --> 00:29:26.727 And then so on my system i'm just doing it on you know one rudifest disc so 00:29:26.727 --> 00:29:29.407 i knew which disc it would be it was just kind of interesting to see like i 00:29:29.407 --> 00:29:32.807 went around and i dd'd some big files and copied them around and i could see 00:29:32.807 --> 00:29:36.347 like oh yeah right the operation completes and then immediately the print tells 00:29:36.347 --> 00:29:38.347 me that oh we waited that much and. 00:29:38.347 --> 00:29:41.487 The practical use here is you know in a rate array, 00:29:42.398 --> 00:29:47.298 It might be useful to discover that one of your disks is significantly underperforming the others. 00:29:47.458 --> 00:29:49.578 It could be indicative of a larger problem. It could be indicative of why you're 00:29:49.578 --> 00:29:50.498 having performance issues. 00:29:51.238 --> 00:29:54.518 So, like, it turns out that like this, there's actually probably a fair amount 00:29:54.518 --> 00:29:57.938 of situations where single trace points, just on their own, might be something 00:29:57.938 --> 00:29:58.838 you want to look at, right? 00:29:58.918 --> 00:30:03.098 So other ones for BcacheFS, write buffer flush. That's an important event. 00:30:03.358 --> 00:30:07.118 Or journal writes. You might be wanting to know stuff about how the journal works. 00:30:07.778 --> 00:30:10.678 but you can also use the scripting language and the 00:30:10.678 --> 00:30:13.778 fact that eBPF supports basic data structures like maps 00:30:13.778 --> 00:30:17.078 and histograms to make more 00:30:17.078 --> 00:30:20.658 complicated combined programs so i had an llm i 00:30:20.658 --> 00:30:24.038 fed it the output of that tplist stuff that told it the available trace all 00:30:24.038 --> 00:30:27.118 the traces you had and what the structures looked like so you had to write the 00:30:27.118 --> 00:30:34.158 right code that's clever and uh bpf trace makes it easy to have stuff that runs 00:30:34.158 --> 00:30:38.138 at an interval right so you can set up all the traces And then every five seconds, 00:30:38.478 --> 00:30:40.138 it can print out a little summary that it's done. 00:30:40.238 --> 00:30:44.978 And so each time it traces, it can update a variable, and then it can calculate latencies or deltas. 00:30:45.838 --> 00:30:50.018 So this is a quick one that it did that looks at that right buffer flush trace 00:30:50.018 --> 00:30:54.938 point, and it samples it over every five seconds, and then it tells you how many flushes happened, 00:30:55.538 --> 00:30:59.478 how many buffers were flushed, how many were skipped, how many were fast path 00:30:59.478 --> 00:31:02.778 flushes, and then the total amount of data that was flushed. 00:31:02.778 --> 00:31:06.898 So just like a quick way to get, you know, an accurate little like every five 00:31:06.898 --> 00:31:07.818 second little printout. 00:31:07.978 --> 00:31:09.158 And what's great is like, as 00:31:09.158 --> 00:31:13.078 far as I know, this is really putting no measurable strain on the system. 00:31:13.178 --> 00:31:14.918 It's one of the more performant ways to do it. Yeah. 00:31:14.998 --> 00:31:18.798 So you can really get deep insights without some of the overhead you sometimes 00:31:18.798 --> 00:31:19.838 get by that kind of monitoring. 00:31:20.677 --> 00:31:24.537 So then to kind of further stress just how far this whole having an AI help 00:31:24.537 --> 00:31:27.137 me out would go, I had an idea. 00:31:27.457 --> 00:31:30.257 I did ultimately tone it back a bit, but basically like, what if I wanted to 00:31:30.257 --> 00:31:33.097 monitor some of the mesh network traffic that was going on? 00:31:33.157 --> 00:31:35.877 There's a lot of options for that, but can this do this too? 00:31:36.337 --> 00:31:40.717 And so it built me a little script that you give it an interface name, 00:31:40.857 --> 00:31:42.377 like tailscale0, say, right? 00:31:42.377 --> 00:31:46.317 and then it'll do a similar thing where every five seconds it'll just look at 00:31:46.317 --> 00:31:51.097 that interface and it'll count sent packets sent bytes receive packets receive 00:31:51.097 --> 00:31:57.277 bytes and it'll do a send latency histogram for you on the sent packets yeah. 00:31:57.277 --> 00:32:01.477 So you get this you actually i mean it's not a gui but it's it's it's a bar 00:32:01.477 --> 00:32:03.037 graph on the command line. 00:32:03.037 --> 00:32:05.997 Yeah right and so the combination of the built-ins ebpf and 00:32:05.997 --> 00:32:08.877 then the built-ins into bpf trace which 00:32:08.877 --> 00:32:11.937 has some of this stuff to do nice histogram display and stuff from 00:32:11.937 --> 00:32:16.397 their print command um it's you know this is a little bit longer some of it's 00:32:16.397 --> 00:32:19.677 kind of because there's a print statement for each thing we're printing it's 00:32:19.677 --> 00:32:22.957 like five different things we're measuring right so it takes up some space on 00:32:22.957 --> 00:32:28.337 the screen but like it's a single page of code here so it's not a it's not crazy 00:32:28.337 --> 00:32:29.757 to start trying to understand it. 00:32:29.757 --> 00:32:33.397 Right i've never even seen some of this stuff but i under i could like the trace 00:32:33.397 --> 00:32:37.857 points they all make sense they're all it's just real plain english really it's 00:32:37.857 --> 00:32:40.357 really easy and then you have the print and the time interval. 00:32:40.357 --> 00:32:45.697 And so under the hood, it is tracing something called NetDevStartTransmit. 00:32:45.837 --> 00:32:46.097 Okay. 00:32:46.317 --> 00:32:50.357 So then it filters, it gets data, it filters on interface name from that data, 00:32:50.397 --> 00:32:53.457 and then it has a little counter it's keeping, so it increments the counter. 00:32:53.697 --> 00:32:58.997 And then it has a thread ID that you can use, and so it gets the current timestamp 00:32:58.997 --> 00:33:03.857 in nanoseconds and stores that in a map based on its thread ID. 00:33:04.437 --> 00:33:08.797 And then it also does a trace point for sent packets with NetDevTransmit, 00:33:08.797 --> 00:33:11.777 and then it grabs the start time if it exists for 00:33:11.777 --> 00:33:14.757 its thread id and then it can compute how 00:33:14.757 --> 00:33:17.557 long it took to send from that and then that's where it can use 00:33:17.557 --> 00:33:20.837 the histogram stuff to print out a nice little thing now that it's tracking 00:33:20.837 --> 00:33:27.637 latency and then it has another uh trace point to use uh net if receive skb 00:33:27.637 --> 00:33:31.557 which tracks incoming packets and then it just has a little bit of code here 00:33:31.557 --> 00:33:34.417 to kind of tie it together every five seconds and do the printout and then it 00:33:34.417 --> 00:33:37.117 clears all the counters so that it can do the next cycle. 00:33:39.294 --> 00:33:43.394 That's a neat little magic, like, superpower, a pocket of power that's in the 00:33:43.394 --> 00:33:44.474 Linux kernel there for this. 00:33:44.614 --> 00:33:48.314 Now, it does mean, right, like, you can't use it if you don't have trace points you care about. 00:33:48.374 --> 00:33:51.554 And so you have to start understanding some kernel internals and, 00:33:51.634 --> 00:33:53.494 like, what trace points might matter and what they mean. 00:33:54.134 --> 00:33:59.234 But if you have a specific problem or a mystery on your system that you're trying 00:33:59.234 --> 00:34:02.594 to look into and you're willing to try to chase down hypotheses using these 00:34:02.594 --> 00:34:04.134 tools, I think you could get pretty far. 00:34:04.274 --> 00:34:08.014 Yeah, that's for sure. And then, you know, there are actual complete products 00:34:08.014 --> 00:34:09.574 out there that are dipping into this stuff. 00:34:10.654 --> 00:34:15.674 I was reading online that there's actually several Kubernetes products that 00:34:15.674 --> 00:34:18.734 are tapping into eBPF to do things under the hood. 00:34:19.234 --> 00:34:24.634 Also, Falco, which is a real-time security monitoring application you can run on Linux. 00:34:24.714 --> 00:34:28.354 It says, Falco uses eBPF to monitor system calls and network events directly 00:34:28.354 --> 00:34:32.714 in the kernel, enabling rapid detection of anomalous behavior with low latency. 00:34:33.174 --> 00:34:36.854 This kernel-level approach is faster and less resource-intensive than the user 00:34:36.854 --> 00:34:37.814 space monitoring tools. 00:34:38.094 --> 00:34:41.734 There's also user space tracing you can do. So if you set it up right, 00:34:41.814 --> 00:34:45.754 you can trace JVM events or Ruby things or Python stuff. 00:34:46.394 --> 00:34:50.034 And so you have products now, too, that will use things like open telemetry 00:34:50.034 --> 00:34:51.274 or other tracing standards, 00:34:51.274 --> 00:34:55.194 and you can have a trace then that can have both the kernel side via the ePPF 00:34:55.194 --> 00:35:00.694 stuff and the user space stuff without necessarily having to do as much custom 00:35:00.694 --> 00:35:05.774 observability implementation in the code base because they can kind of do more dynamic. 00:35:07.114 --> 00:35:09.854 And you might be able to then see stuff, right, where like, oh, 00:35:10.054 --> 00:35:12.654 there was a problem on the kernel layer, and then that's why we started seeing 00:35:12.654 --> 00:35:14.574 increased latency in the application layer. 00:35:16.174 --> 00:35:20.594 But you can use that. There's a lot of open source products or open core things, 00:35:20.634 --> 00:35:24.154 and there's just a lot of sort of primitives in one layer above primitives that 00:35:24.154 --> 00:35:27.214 are probably already on your kernel. 00:35:27.534 --> 00:35:31.234 I wonder if it's possible people out there listening have already been using 00:35:31.234 --> 00:35:33.694 eBPF for a while and this is all old news to them. 00:35:34.094 --> 00:35:35.574 Who didn't tell us? Like, have 00:35:35.574 --> 00:35:38.234 you used it in what applications and what utilities has it been for you? 00:35:38.274 --> 00:35:42.794 I mean, we've gone through a few examples here, but like Wes's story with the 00:35:42.794 --> 00:35:48.354 VPS and my story with tracking down like a background desktop application, 00:35:48.814 --> 00:35:52.074 there's just these also very practical day-to-day uses for this stuff. 00:35:52.074 --> 00:35:55.014 that's nothing more than just using some of the pre-existing tools that you 00:35:55.014 --> 00:35:57.934 just run and tap into this, and you don't have to write any kind of scripting code. 00:35:58.394 --> 00:36:03.094 Yeah, I mean, it's been around for long enough now that it's well-packaged in most distros. 00:36:03.634 --> 00:36:06.634 Can you use eBPF to figure out why I have so many tabs open? 00:36:08.774 --> 00:36:11.554 Well, maybe we can add a trace point to Firefox to keep track at least. 00:36:15.247 --> 00:36:20.067 Jupiterbroadcasting.com slash river river is the most trusted spot in the u.s 00:36:20.067 --> 00:36:26.487 for individuals and businesses to buy send receive bitcoin and they make it 00:36:26.487 --> 00:36:30.427 easy in three simple steps jupiterbroadcasting.com slash river hey. 00:36:30.427 --> 00:36:33.587 I just got set up with them and it was in fact very easy. 00:36:33.587 --> 00:36:36.427 I think their best low-key feature actually 00:36:36.427 --> 00:36:39.507 has to do with cash just as an aside so bitcoin is 00:36:39.507 --> 00:36:42.247 dipping as we record and if oh man this is 00:36:42.247 --> 00:36:45.467 this was the moment so they have a 3.8 percent 00:36:45.467 --> 00:36:48.307 interest cash savings account and they pay the interest 00:36:48.307 --> 00:36:51.587 out in sats so you put your cash in there and it's 00:36:51.587 --> 00:36:55.407 fdic insured 3.8 percent interest and 00:36:55.407 --> 00:36:58.407 then when the when the price dips on bitcoin you can use that balance 00:36:58.407 --> 00:37:03.187 to smash buy and so you can you can essentially dca with that 3.8 percent interest 00:37:03.187 --> 00:37:08.187 and then you can smash buy when the price dips it's just brilliant and river 00:37:08.187 --> 00:37:13.227 is a really great company i had a call with their community management person 00:37:13.227 --> 00:37:15.607 and was really impressed with what they said. 00:37:15.927 --> 00:37:19.367 And if you're in Canada and you're looking for another trusted source to get 00:37:19.367 --> 00:37:22.127 sats, bitcoinwell.com slash Jupiter. 00:37:22.367 --> 00:37:24.907 There you go. That's our tip. 00:37:28.187 --> 00:37:32.907 Well, since this week we are hoarding our boost for next week's episode, 00:37:32.907 --> 00:37:36.407 we decided to do a little dive into the old mailbag. 00:37:36.787 --> 00:37:40.267 Zach sent us a little note here. I'm behind on listening to episodes. 00:37:40.447 --> 00:37:45.287 But the NixOS episode from last month, Linux Unplugged 5.9.8, 00:37:45.567 --> 00:37:47.567 was really interesting. 00:37:47.747 --> 00:37:51.047 I wanted to throw out some thoughts responding to a comment that was sent in 00:37:51.047 --> 00:37:57.907 about wanting a NixOS-like system, but being able to use the old traditional system admin tools. 00:37:58.487 --> 00:38:02.347 For context, I've tried out NixOS, but quickly got to the point where I would 00:38:02.347 --> 00:38:04.887 need to dive in and figure out those flakes. 00:38:05.147 --> 00:38:08.327 At that point, it just became one more thing on that list to do. 00:38:08.327 --> 00:38:11.147 Since then, I've gotten into bootable containers. 00:38:11.447 --> 00:38:15.107 I've specifically been using the Universal Blue ecosystem with Bootsy, 00:38:15.347 --> 00:38:20.207 but have wanted to try out Vanilla OS and Common Arch that also use OCI images 00:38:20.207 --> 00:38:21.787 for delivery and customization. 00:38:22.407 --> 00:38:27.347 Listening to that NixOS episode, I felt like many of those talking points that 00:38:27.347 --> 00:38:32.267 were given in favor of NixOS were also very in line with the benefits that I've 00:38:32.267 --> 00:38:36.527 seen in using OCI images to build out customized OS images. 00:38:37.818 --> 00:38:41.298 I don't use any of the headliner distros like aurora or 00:38:41.298 --> 00:38:44.618 bluefin although i have tried a few and overall they were quite 00:38:44.618 --> 00:38:47.378 well put together but i've taken their more 00:38:47.378 --> 00:38:50.718 low-level image builds and layered my own customizations 00:38:50.718 --> 00:38:55.878 on top of them it's been a fantastic way to manage everything as when i need 00:38:55.878 --> 00:38:59.518 a new server or desktop all i have to do is install my image and everything's 00:38:59.518 --> 00:39:05.098 ready to go i will add the caveat that there may be some learning if you haven't 00:39:05.098 --> 00:39:07.938 already been familiar with building OCI containers, 00:39:08.238 --> 00:39:13.378 but I live in a world where professionally and for hobbies, that's what I'm doing. 00:39:13.578 --> 00:39:15.918 So that wasn't a huge burden for me to learn. 00:39:16.458 --> 00:39:20.158 Thanks for the show. Really have been enjoying it and wanted to send this little 00:39:20.158 --> 00:39:23.938 message in just in case someone else might find some use. 00:39:24.478 --> 00:39:27.738 Well, cool. Thank you for sharing. We love success stories like this. 00:39:27.838 --> 00:39:29.998 I wonder if you have any of it on GitHub or anything too. 00:39:30.138 --> 00:39:33.038 If people are curious. Good question. I really 00:39:33.038 --> 00:39:35.738 feel like this nails something that we've been feeling and chatting a 00:39:35.738 --> 00:39:40.978 lot about behind the scenes is uh the really 00:39:40.978 --> 00:39:43.978 really brilliant thing that the universal blue folks 00:39:43.978 --> 00:39:50.818 have tapped into is this existing knowledge set around how oci images work and 00:39:50.818 --> 00:39:55.478 how you can layer things and really tapped into the whole devops workflow that 00:39:55.478 --> 00:39:59.678 people live every day and now you can use it that skill set that you've learned 00:39:59.678 --> 00:40:03.638 and you can use it to customize your Linux desktop. I mean, that is just really powerful. 00:40:04.438 --> 00:40:07.538 It's a different approach than Nix, right? Whereas Nix, you're building it from 00:40:07.538 --> 00:40:12.418 the ground up. You would use Nix to maybe generate those OCI images or something like that. 00:40:12.598 --> 00:40:16.318 But both, like he says, are essentially accomplishing similar goals. 00:40:16.598 --> 00:40:22.378 Yeah, and a lot of the immutability side of things and the image-based approach, right? 00:40:22.498 --> 00:40:27.438 Like starting to think of your system as a cohesive whole that you deliver in 00:40:27.438 --> 00:40:31.238 those coherent units rather than ad hoc imperative updates. 00:40:31.378 --> 00:40:35.258 Now, admittedly, I don't use them as much, but Brent, when I think of an image-based 00:40:35.258 --> 00:40:37.458 system, I don't necessarily think of flexibility. 00:40:38.118 --> 00:40:42.178 Yeah, I was kind of wondering about this. I've had a few people tell me, 00:40:42.258 --> 00:40:43.878 oh, yeah, images is totally the way to go. 00:40:44.358 --> 00:40:47.558 Of course, we've been playing with Nix and NixOS at the same time, 00:40:47.558 --> 00:40:50.598 and the thing I keep tripping on with images is... 00:40:51.779 --> 00:40:55.419 You build them once and then you use them many times. But the way, 00:40:55.599 --> 00:41:00.379 Chris, I think you and I have been using at least NixOS on the desktop is your 00:41:00.379 --> 00:41:02.599 Nix config is like constantly evolving. 00:41:03.059 --> 00:41:06.379 Therefore, the next time you go to install it somewhere else, 00:41:06.519 --> 00:41:08.699 it's always that new updated copy. 00:41:08.819 --> 00:41:12.279 And I would imagine one of the downsides with building an image is, 00:41:12.679 --> 00:41:14.159 well, you have that intermediate step. 00:41:14.279 --> 00:41:18.039 You have to go ahead and build your image every time you want that newest, freshest version. 00:41:18.179 --> 00:41:20.979 Am I understanding that correctly? Is that a proper downside here? 00:41:20.979 --> 00:41:25.439 I know some of the silver blue types, right, they're still using a lot of RPM 00:41:25.439 --> 00:41:28.079 OS tree, which has ways to apply things live. 00:41:28.519 --> 00:41:31.399 So it might depend a little bit on how exactly what you're putting in those 00:41:31.399 --> 00:41:34.499 images, or maybe you have like a dynamic switch image thing. 00:41:34.659 --> 00:41:35.719 But yeah, I do wonder about that. 00:41:35.839 --> 00:41:37.479 And then you also got to remember, it depends on what you're doing. 00:41:37.699 --> 00:41:40.419 You know, it might just be maybe a lot of your applications are flat packs and 00:41:40.419 --> 00:41:42.279 you're installing those and updating those separately anyways. 00:41:43.119 --> 00:41:47.919 Or you're using, you know, various containers or other things for your dynamic environments. 00:41:48.479 --> 00:41:51.619 I'll take Liam's email here. He says, good day, gents. In the most recent LUP, 00:41:51.719 --> 00:41:54.419 you mentioned not having received much feedback about multi-monitor setup. 00:41:54.499 --> 00:41:55.539 So he sent us a picture of his. 00:41:55.779 --> 00:42:00.199 I've got a laptop screen at 1080 at 144 hertz. That's my primary display, 00:42:00.359 --> 00:42:01.239 you know, for things like Skype. 00:42:01.599 --> 00:42:04.639 Then I have two externals, a 32-inch 2K. 00:42:05.399 --> 00:42:08.979 I do think that's the sweet spot, he says, at 60 hertz. And then my rightmost 00:42:08.979 --> 00:42:10.159 monitor is in portrait mode. 00:42:10.519 --> 00:42:15.139 That's where I can see more lines of code. All of this works without issue on XFCE. 00:42:15.539 --> 00:42:19.079 In case you do navigate to the image, yeah, that's a treadmill desk. 00:42:19.079 --> 00:42:22.119 I'm on my second one since the start of the pandemic this. 00:42:22.119 --> 00:42:23.099 Is a nice setup. 00:42:23.099 --> 00:42:27.759 Listeners since Sidebyte, Lunduk and Last Days and Early Tech Snap member since 00:42:27.759 --> 00:42:33.539 2022 annual member since December that's awesome thank you very much Liam and 00:42:33.539 --> 00:42:37.799 I like your setup I am as you boys know big fan of the vertical monitor, 00:42:38.962 --> 00:42:44.422 For our show docs or, you know, like reviewing a config file or just having a terminal up there. 00:42:45.382 --> 00:42:46.102 Journal D. 00:42:46.822 --> 00:42:50.322 Yep. The vertical monitor is a productivity hack. 00:42:50.362 --> 00:42:54.762 If you can afford the luxury of, because you're not, it's not great. 00:42:54.762 --> 00:42:58.922 But one of the other things I will use the vertical monitor for is I stack two chats. 00:42:59.242 --> 00:43:01.682 Like if I have two different work chats or something, I'll stack them on the 00:43:01.682 --> 00:43:03.382 vertical monitor. Works really great. 00:43:03.542 --> 00:43:07.902 Right. Yeah. It doesn't fit every activity. So it's sort of a less generally useful screen at times. 00:43:08.062 --> 00:43:12.502 Yeah. Yeah. It is more limited, so it's more of a luxury. I do have to admit that. 00:43:12.682 --> 00:43:18.502 I have a bit of a question here for us. Did we ever divulge what our current setups are? 00:43:18.602 --> 00:43:22.042 Chris, I know you've gone into details, but I don't think Wes and I have actually shared that. 00:43:22.422 --> 00:43:25.242 Wes, I bet you kind of move around depending on, because you're mostly using laptops. 00:43:25.622 --> 00:43:31.662 I do a lot of laptop. I've got a dual monitor set up in the sort of main office-y 00:43:31.662 --> 00:43:38.442 bit, and then I have just a single where I do laptop with one screen in my living room sort of desk. 00:43:38.442 --> 00:43:40.482 You got a monitor at the living? That's nice. 00:43:40.762 --> 00:43:43.842 That's for the like casual work where I want to do stuff, but with the TV on. 00:43:43.942 --> 00:43:46.162 Yeah. Okay, Brent, what's your setup? 00:43:46.622 --> 00:43:49.882 Well, I don't actually do dual monitor anymore. 00:43:51.062 --> 00:43:55.222 I am now doing the tri-monitor thing. Chris, I know you're like a quad monitor 00:43:55.222 --> 00:44:01.222 guy, but I did steal your little tip. You've been trying to get me to use this for years. 00:44:01.282 --> 00:44:06.362 I have same as Liam here on the right-hand side, a vertical monitor. 00:44:06.522 --> 00:44:08.802 It's just an old monitor I've had forever. it doesn't, 00:44:10.018 --> 00:44:12.958 matter though because it's just chats or like currently 00:44:12.958 --> 00:44:15.778 i have our episode recorder there i 00:44:15.778 --> 00:44:18.678 have the window on which we're connected together 00:44:18.678 --> 00:44:21.458 and then i also have the jb chat even though we're not doing 00:44:21.458 --> 00:44:24.098 it live for some reason it just makes me feel better and it's 00:44:24.098 --> 00:44:28.258 perfect for that and so i've got three monitors running usually for me off a 00:44:28.258 --> 00:44:34.478 laptop and got this big massive 20 well i say massive although liam beat me 00:44:34.478 --> 00:44:39.258 on this one but it's this 27 inch like 4k dell monitor right as my main laptop 00:44:39.258 --> 00:44:42.798 as my main monitor in front of me and that's been a pretty sweet setup. 00:44:42.798 --> 00:44:44.538 I'm on to him. 00:44:44.538 --> 00:44:45.178 He asked. 00:44:45.178 --> 00:44:46.478 This question because he wanted to. 00:44:46.478 --> 00:44:49.298 Answer exactly and i'm here for yeah 00:44:49.298 --> 00:44:51.978 i love it um you know i've been i know i mentioned this 00:44:51.978 --> 00:44:54.798 recently on the show but i have i definitely have always 00:44:54.798 --> 00:44:57.718 been a multi-monitor person for many many many years now but i 00:44:57.718 --> 00:45:01.338 have been enjoying the single extra extra 00:45:01.338 --> 00:45:04.418 ridiculously wide screen it's actually a curved wide 00:45:04.418 --> 00:45:10.258 screen at home and uh and i use gnome there as well and i i really like i mean 00:45:10.258 --> 00:45:14.718 you can you can lay out three full windows side by side on the screen it's just 00:45:14.718 --> 00:45:17.638 really you get a lot of room and when you start getting that kind of room you 00:45:17.638 --> 00:45:23.938 don't really need a second screen as much and i will say it is a much easier way to do desktop linux. 00:45:23.938 --> 00:45:27.178 I do think that'll be in one of my future monitor purchases for sure. 00:45:27.178 --> 00:45:30.698 If you go with like an intel or an amd gpu right now maybe one day in the video 00:45:30.698 --> 00:45:35.378 But Intel and AMD and a single monitor, I promise you, your life will be easier 00:45:35.378 --> 00:45:38.538 than if you try to be a quad monitor guy. It's not an easy path. 00:45:38.718 --> 00:45:41.318 But what if I want to chain DisplayPort together? 00:45:41.638 --> 00:45:47.758 Yeah. Okay. Yeah. Oh, man. And, like, I don't know what my deal is because I've 00:45:47.758 --> 00:45:50.298 only been using computers for, like, 30-plus years. 00:45:50.698 --> 00:45:56.638 But, like, I still have major strugs trying to make sure that the correct monitor 00:45:56.638 --> 00:45:59.158 is the default, like, bioscreen monitor. 00:45:59.178 --> 00:45:59.718 Oh, gosh. 00:46:00.738 --> 00:46:03.478 And I don't know maybe this isn't how it works, 00:46:05.158 --> 00:46:10.418 hand to the mixer if it doesn't change i get it just set up right and then like 00:46:10.418 --> 00:46:13.958 a couple of days go by and the next time i turn the machine on like the different 00:46:13.958 --> 00:46:18.238 monitors lighting up and i've gone through and i've i i thought i did the proper 00:46:18.238 --> 00:46:22.418 ordering you know in the plugs and it changes i swear that happens i don't know 00:46:22.418 --> 00:46:23.518 maybe maybe i'm making it up. 00:46:23.518 --> 00:46:26.638 You need to like use uuids for your monitor setup. 00:46:26.638 --> 00:46:31.518 Yeah well it's at the bios level like you know props to plasma by the time i 00:46:31.518 --> 00:46:32.838 get to Plasma, it's always fine. 00:46:34.018 --> 00:46:37.198 And I've been actually having really good success with locking my screens. 00:46:37.198 --> 00:46:41.838 They go to sleep. They wake back up. Everything's fine. All the orientations are correct. 00:46:42.458 --> 00:46:48.378 And when it boots, it always nails it. So Plasma has been just doing great with my multi-moder setup. 00:46:48.638 --> 00:46:52.078 But my BIOS, right? Or like when the system's booting, the console where you're 00:46:52.078 --> 00:46:57.858 just seeing the output, like that is a moving target for some reason. 00:46:57.898 --> 00:47:02.918 And I don't think that's technically how this works but maybe it's me you know 00:47:02.918 --> 00:47:05.058 moving monitors around i don't know you. 00:47:05.058 --> 00:47:06.438 Could take the next year off. 00:47:06.438 --> 00:47:06.938 And just. 00:47:06.938 --> 00:47:08.258 Rewrite your bios. 00:47:08.258 --> 00:47:12.778 Oh i thought maybe i'd do a study where i just empirically like write every 00:47:12.778 --> 00:47:17.238 move i make down okay this one's in this plug boot yeah you write that step one i. 00:47:17.238 --> 00:47:20.278 Think maybe you should study the cameras that you have in the studio because 00:47:20.278 --> 00:47:23.678 every time i show up i tend to shuffle all your cables around for your monitors. 00:47:23.678 --> 00:47:26.518 That is true that is true, 00:47:28.852 --> 00:47:33.372 Okay, well, the pick this week has to be something that gets you up and running 00:47:33.372 --> 00:47:35.692 with BPF right away, right? 00:47:35.832 --> 00:47:38.332 Like, we don't want to sit here and talk about eBPF this whole episode and not 00:47:38.332 --> 00:47:39.792 give you a great tool to check out. 00:47:40.052 --> 00:47:46.072 And this week, it's Network Top. It helps you monitor traffic from your network using BPF. 00:47:46.232 --> 00:47:49.832 Yeah, we should be clear. This one is classic BPF, but it's still handy. 00:47:49.832 --> 00:47:52.012 And that means it is broadly useful, too. 00:47:52.172 --> 00:47:52.352 Yeah. 00:47:52.692 --> 00:47:53.792 It's built in Rust. 00:47:54.192 --> 00:47:54.872 Oh, it is. 00:47:54.952 --> 00:47:58.652 Yeah, so it's super performant. It's easy to use. 00:47:58.812 --> 00:48:05.292 And it means you get to use like the TCP dump style syntax in a nice little Tui. 00:48:05.512 --> 00:48:08.692 And so I sent you an example. What do you think of it? 00:48:09.012 --> 00:48:12.052 Well, first of all, it's very readable, right? Because it is, 00:48:12.132 --> 00:48:15.492 like you said, you're getting charts in a terminal UI. 00:48:16.492 --> 00:48:20.832 And I mean, it's immediately understandable. I'm trying to figure out what you're 00:48:20.832 --> 00:48:22.452 doing here because I see ginormous, 00:48:22.612 --> 00:48:28.112 like you have almost no traffic and then it jumps up to almost 26.3 megabytes 00:48:28.112 --> 00:48:34.032 a second and it hovers there for about 40 seconds and then it drops down to absolutely nothing. 00:48:34.032 --> 00:48:37.652 Yeah so you'll you see how there's okay there's an input section at the top 00:48:37.652 --> 00:48:41.852 and then under that there's rules yeah you'll see what it says all on the left 00:48:41.852 --> 00:48:45.132 and then to the right it says host and then a specific ip address. 00:48:45.132 --> 00:48:47.872 So you're you're looking at all traffic from just this host. 00:48:47.872 --> 00:48:52.172 Yes um and so those are two separate rules so basically it's it's looking at 00:48:52.172 --> 00:48:57.032 one interface and just watching it totally by default and then you can add bpf 00:48:57.032 --> 00:49:00.252 rules that it'll add as additional things that it'll monitor at the same time 00:49:00.252 --> 00:49:05.832 and then you can use the arrow keys to toggle between the views for different uh rules okay and, 00:49:07.244 --> 00:49:11.704 Um, there was no traffic because, uh, I SSH to another machine here at the studio. 00:49:12.044 --> 00:49:14.784 We hadn't really been talking, right? I exchanged a few packets for that, but that was all. 00:49:14.984 --> 00:49:20.724 Um, and then I did a net cap back to my laptop, just reading from random. 00:49:21.324 --> 00:49:24.904 Uh, and so that was the big traffic spike. And then I killed it just to watch it all drop off. 00:49:25.344 --> 00:49:29.144 But you know, like it could, you can do, you can do like specific ports that you want to watch. 00:49:29.264 --> 00:49:33.444 You can do specific IPs, host names, TCP states, like whatever you can do with 00:49:33.444 --> 00:49:35.924 TCP dump. Basically you can put those rules in here. 00:49:35.924 --> 00:49:40.444 And you're getting a command line TUI, a terminal user interface. 00:49:40.724 --> 00:49:44.024 So it's really easy to understand. Like, these rules Wes are talking about are 00:49:44.024 --> 00:49:47.624 clearly delineated in the UI. It's really simple syntax. 00:49:47.904 --> 00:49:49.804 So maybe there's something you're trying to hunt for. You find it here, 00:49:49.824 --> 00:49:53.404 and then you can go do the TCP dump to capture the traffic and do some more analysis. 00:49:53.604 --> 00:49:57.824 And it is MIT licensed, so it is free to use. 00:49:58.244 --> 00:50:03.344 And then there is a bonus pick, BPF Tune. 00:50:03.504 --> 00:50:06.944 Yeah, well, you kind of hinted at this a bit. And I happen to know you turned 00:50:06.944 --> 00:50:08.744 it on for at least one of your systems. 00:50:08.744 --> 00:50:10.324 I have it running on my home workstation. 00:50:10.444 --> 00:50:15.764 Despite the fact that this is indeed a GPL 2.0 with Linux Syscall Note licensed 00:50:15.764 --> 00:50:17.584 Oracle open source project. 00:50:18.484 --> 00:50:24.304 BPF Tune aims to provide lightweight, always-on auto-tuning of system behavior. 00:50:24.564 --> 00:50:28.484 The key benefits it provides is by using BPF observability features, 00:50:28.484 --> 00:50:32.784 it continuously monitors and adjusts system behavior. because we can observe 00:50:32.784 --> 00:50:34.344 the system behavior at a fine grain. 00:50:35.084 --> 00:50:40.324 We can then tune at a finer grain too. So like individual socket policies, 00:50:40.804 --> 00:50:42.164 individual device policies. 00:50:42.444 --> 00:50:46.224 Yeah, right. So think of all those things like CTL can tweak. 00:50:46.764 --> 00:50:51.564 It's using BPF hooks to watch your system and then automatically go tweak those. 00:50:51.764 --> 00:50:55.244 Yeah. Now I haven't really noticed a difference, but I've only had it on for a couple of days. 00:50:55.404 --> 00:50:58.744 But the idea is brilliant. It's just so brilliant. 00:50:58.884 --> 00:51:01.604 Like here's the system. I'm monitoring. OK, now I'll just go make adjustments 00:51:01.604 --> 00:51:05.444 here to make essentially maybe like ease pressure on the system or something like that. 00:51:06.062 --> 00:51:08.742 I don't know if anybody has experience with this, because I just started, 00:51:08.942 --> 00:51:10.862 but it seems like a fantastic idea. 00:51:11.602 --> 00:51:14.442 BPF tune. Chris will have a link to these in the show notes. 00:51:16.042 --> 00:51:20.342 And in Nix, it's like just one, it's like, you know, service enable BPF auto 00:51:20.342 --> 00:51:21.642 tune. And then you just, you're good. 00:51:23.502 --> 00:51:26.702 There's more you can do, but that's, it's really simple. So I was like, 00:51:26.742 --> 00:51:27.802 all right, I'm just going to turn this on. 00:51:27.942 --> 00:51:31.622 And I just love the concept of the system auto tuning itself. 00:51:31.622 --> 00:51:35.482 Yeah, and they kind of make the case here too, if you're doing the cattle not 00:51:35.482 --> 00:51:40.282 pets approach and how many of your systems ever really even have a human person 00:51:40.282 --> 00:51:41.922 who might be on it who could tune it. 00:51:42.302 --> 00:51:44.902 Okay, maybe you can hand-tune the database, but you're not going to hand-tune 00:51:44.902 --> 00:51:46.622 all your dynamic web workers. 00:51:46.802 --> 00:51:48.162 This could be great for my Odroid. 00:51:48.322 --> 00:51:52.182 But if some of them are longer lived, this king can make sure they're running all right. 00:51:52.322 --> 00:51:55.782 I should put this on my Odroid. I really should. Little Odroid's doing a lot 00:51:55.782 --> 00:51:59.282 of work and I never really check in on it. Just sitting there being a little yeoman about it. 00:51:59.702 --> 00:52:03.862 Well, we will have links to this and everything else. There's a lot of links this week. 00:52:04.882 --> 00:52:11.042 Linuxunplugged.com slash 605. And remember, we want to know if you enjoy these deep dives. 00:52:11.202 --> 00:52:13.942 We can get into the weeds here a bit too much, but if this is the kind of stuff 00:52:13.942 --> 00:52:18.182 you like, let us know because there's, it's honestly for us as creators, 00:52:18.342 --> 00:52:22.062 it's a little, creators, it's a little scary to do topics like this. 00:52:22.162 --> 00:52:24.422 I know that sounds stupid, but it is. It's just a little scary. 00:52:24.722 --> 00:52:26.462 So we always like to know what your thoughts are. 00:52:27.202 --> 00:52:29.222 We'll be back at our regular live time. 00:52:30.022 --> 00:52:33.802 Tuesdays as in Sunday Sunday at 12 p.m. Pacific, 3 p.m. Eastern. 00:52:38.642 --> 00:52:43.962 Now if you want more show remember our members get the full bootleg which is 00:52:43.962 --> 00:52:48.342 clocking in at like an hour and 20 minutes or something right now and this is a short one, 00:52:49.162 --> 00:52:54.142 and of course you can get details of that at linuxunplugged.com Thank you so 00:52:54.142 --> 00:52:57.842 much for joining us on this week's episode of the Unplugged program we just 00:52:57.842 --> 00:53:00.922 really appreciate your time for listening if you want to share it with somebody 00:53:00.922 --> 00:53:04.242 we always like that too word of mouth is the best advertising for a podcast 00:53:04.242 --> 00:53:05.402 we always appreciate that, 00:53:05.862 --> 00:53:09.722 thank you so much for being here and we'll see you right back here next Tuesday 00:53:09.722 --> 00:53:12.722 as in Sunday which isn't Tuesday at all.
Previous episode Next episode

Related episodes

Search

Search

Listen on

Apple Podcasts Spotify RSS feed

Tags

Business Partner