diff --git a/proposals/20240507-modern-bpf-kernel-side-filters.md b/proposals/20240507-modern-bpf-kernel-side-filters.md new file mode 100644 index 0000000000..0e420af1b8 --- /dev/null +++ b/proposals/20240507-modern-bpf-kernel-side-filters.md @@ -0,0 +1,104 @@ +# Adding Kernel-side filtering to the Modern BPF driver +## Summary + +For deployments of Falco on larger instances, we need to be able to cut down on the number of noisy events coming from the modern-bpf driver source as the rules engine being single-threaded has become a bottleneck. + +There exists the feature to toggle individual syscalls, but more granularity here would allow for more control - still monitoring all relevant syscalls while filtering out sources of concentrated noise. + +## Motivation + +There are users of Falco deploying to hosts with 128+ CPUs where syscall throughput is too high for the system to keep up. These hosts are especially susceptible to bursty workloads capable of causing a large percentage of event drops. + +Here, being able to filter out the noisy/bursty workloads individually is preferable to disabling the collection of entire syscalls. + +## Goals + +- Define a flexible config schema for these filters +- Implement the filters to be as performant as possible without introducing new means of circumventing detection + +## Non-Goals + +- Design/implement filters for other driver types + +## The Plan + +### Review Filter Schema v1 + +The initial implementation of the filter config schema is as follows: + +``` +filters: + - syscall: 257 # syscall number for `openat` on `x86_64` + arg: 1 # file path + prefixes: ["/proc", "/sys/fs/cgroup"] +``` + +Each entry in the YAML list contains: +- The syscall number for the target architecture. +- The arg number for the syscall. This is expected to be a `char *`. +- The prefixes to match and filter against inside the kernel hook. + +The diffs for this initial implementation can be found [here](https://github.com/falcosecurity/falco/compare/master...HubSpot:falco:master) for Falco and [here](https://github.com/falcosecurity/libs/compare/master...HubSpot:falcosecurity-libs:master) for falcosecurity-libs. +#### Performance + +We were able to deploy this patch on our infrastructure and collect the following data. All of this is aggregated over a 4 day window. + +Event rate (`scap.evts_rate_sec`: +``` +count 1022425.000000 (number of metric samples) +mean 78169.513609 +std 56479.322696 +min 0.000000 +50% 71634.525317 +90% 139310.344902 +99% 260659.990578 +99.9% 565055.582845 +99.99% 734383.889699 +99.999% 779872.462636 +max 824091.991662 +``` + +Event drop rate (`scap.n_drops_perc`): +Vanilla Falco `0.36.2`: +``` +count 1022425.000000 (number of metric samples) +mean 0.282009 +std 3.849407 +min 0.000000 +50% 0.000000 +90% 0.000016 +99% 1.781471 +99.9% 77.758152 +99.99% 94.549621 +99.999% 97.712998 +max 99.290935 +``` +Falco `0.37.0` with our patch filter filtering out 6 commonly seen path prefixes for file open events: +``` +count 104192.000000 (number of metric samples) +mean 0.472717 +std 2.702165 +min 0.000000 +50% 0.000000 +90% 0.593645 +99% 9.937187 +99.9% 46.086308 +99.99% 71.471973 +99.999% 76.030033 +max 76.692583 +``` + +To summarize, it looks like we get drastically better worst-case performance at the cost of worse 90-99th performance. + +This doesn't factor in other changes in Falco made between `0.36.2` and `0.37.0`, but it's likely not too significant looking at the changelog. + +### Make necessary improvements + +To make this feature production ready, we should include more guardrails to improve the UX when the feature is misused. It would also be preferable to specify syscalls by name instead of by syscall number in the config. + +As for performance, it may be worth looking into how we can optimize this further. For example, fully qualified filenames into a map for a constant time check in the kernel hook would be preferable for performance, but a lot harder for users to maintain than supplying prefixes. + +### Next Steps + +1. Gather feedback on the current implementation in regard to how we can improve filtering performance while maintaining usability +2. Improve the UX of this feature and add documentation for how to use it