After an exciting start with the first day of
the eBPF Summit, the second and also final day of the summit this year
was kicked off with another warm welcome from Thomas Graf where he reminded attendees:
Bees are nice. Be a bee.
Thomas set the stage for an incredible list of presenters for today’s keynote
talks, but not before announcing the winner of yesterday’s trivia question
Which mountain is behind Thomas in the video background? The correct answer was
provided by Slack user Bala and they are receiving a cute plush bee as a reward.
Some of the Slack user poll results were also presented and it was great to see
that a lot of participants were completely new to eBPF. Naming suggestions for
the eBPF mascot that we collected from the prior day also found a clear winner:
Full length video of the eBPF Summit, day two:
Safe Programs, the Foundation of BPF by Alexei Starovoitov (Facebook)
Alexei Starovoitov from Facebook kicked off the keynotes with his talk titled “Safe programs. The foundation of BPF.” Alexei is the co-creator and co-maintainer
of eBPF. Alexei’s talk was focused on using eBPF for the purpose of extending
and programming the kernel – and how it was a much safer approach than with
kernel modules. Thanks to the verifier, programs are checked for safety, ensuring
they will not crash the kernel and bring the system to a halt. When comparing
compilation versus reliance on the eBPF verifier, Alexei provided examples where
a compiler will assume safety and the verifier ensures safety – resulting in
safer programs. He pointed out, however, that interactions between the verifier
and LLVM will vary resulting in differences on how to approach a solution.
BPF Type Format (BTF) further enhances the kernel, providing a new level of safety for
eBPF based C programs. What’s next? Look out for new features in development
including eBPF linker, eBPF libraries, and eBPF dynamic linking extensions.
Alexei closed with a quote:
Maximum attention to safety in all aspects of BPF programming that’s what makes it unique and that’s why BPF is an undoubted choice today for kernel extensions and kernel programming.
Kernel Tracing in Production with Falco by Kris Nóva (Falco)
Following Alexei, Kris Nóva from Falco quickly stole the stage – or rather the
presentation screen, to present her own screen sporting a retro hacker theme.
Kris is a maintainer of the Falco project.
Her talk title changed multiple times throughout the weeks leading up to the summit,
finally landing on “How do we instrument the kernel without a kernel module in GKE?”
Falco uses kernel instrumentation to trace what’s happening in the kernel and
can alert based on specific rules. To do this, a kernel module loaded on a host
provides the necessary access. But what about on GKE where there is no support
for loading kernel modules on a host? This is where eBPF proved to be a natural
solution. Falco developers wrote an emulation device using eBPF and were able to
replace the kernel module. With an eBPF based engine now able to run in place
of the Linux kernel module, the other parts of Falco were able to plug in easily
without much change. Kris went on to show some examples of Falco in action using
a kernel module, eBPF with clang 9 and clang 10 – the main purpose of the examples
being to demonstrate the ease of moving between the different implementations and
how they all work to alert about suspicious activity.
Performance Wins with BPF: Getting Started by Brendan Gregg (Netflix)
After the break, Brendan Gregg from Netflix presented on “Performance wins with
BPF: Getting started”. Brendan’s primary goal is to help people find performance
wins with eBPF quickly and easily. Many articles about eBPF aren’t for beginners,
but Brendan believes everyone can find small wins with eBPF easily. We can do this
by thinking like sysadmins, not like programmers. Brendan highlighted some
tools we can explore that are available today, such as execsnoop to find periodic
processes, opensnoop to find misconfigurations or files not found, tcplife to
inspect TCP session details, ext4slower to find slow I/O performance or biosnoop
for finding block I/O performance bottlenecks. These tools are just a few examples
of easily accessible programs that can help find unusual activity for some quick
wins. Beyond these examples, there is a huge selection of tools included in the
bcc and bpftrace toolkits which can be easily leveraged for even more gains.
In a majority of cases, existing tools can solve the problem – but when they can’t
then it’s time to think like a programmer and build the solution. If you are just
getting started today, Brendan suggests to start with bpftrace because it is concise
and behaves like pseudo-code. If looking into bcc implementations, Brendan first
recommends to check out libbpf-based tools for an implementation instead. Thanks to
libbpf, Brendan was able to build opensnoop resulting in just a 151kb large binary.
Brendan believes the future of eBPF is in development in the form of GUIs which
surface some of the same information, but uses eBPF in the underlying system.
He believes that the ability to surface this data to operators or users, with no
knowledge of the intermediate system relying on eBPF, will open up new avenues.
In closing, Brendan encourages everyone to think like a sysadmin, install bcc and
bpftrace tools, run them and get some wins. To dive even deeper into eBPF, you
should pick up a copy of Brendan’s books, “BPF Performance Tools” and “Systems
Performance 2nd edition”.
Kubernetes Network Policy Logging with eBPF by Zang Li (Google)
Zang Li from Google presented next on “Kubernetes Network Policy Logging with
eBPF.” Zang works on GKE and Anthos and is also a Cilium core project member. She started
with an explanation of how Kubernetes network policy is implemented along with
an example policy and scenario. She went on to highlight how differences in
implementation from one CNI provider to the next directly impacts how policies
are implemented, whether that is via iptables, eBPF, Open vSwitch or some other
implementation. In addition to enforcement, she added that security-conscious
customers will want to be able to log events related to policy decisions.
For Google, it was important this happened with minimal impact to the data path.
During their design phase, they quickly realized that iptables would not scale
to satisfy their complex networking requirements. Cilium provided the flexibility
and programmability they needed. In addition, its implementation of eBPF-based
network policies meant that Google could remove all reliance on iptables from
the data path. These factors ultimately led Google to select Cilium as the CNI
provider for GKE and Anthos.
Zang went on to explain how Google has extended, and makes use of, the Cilium
monitor infrastructure to extract and log useful data about network connections. To
provide visibility and observability into network connectivity, Google uses
Cilium monitor, which runs as a user space application and is able to access data
made available by an eBPF program pushing data into a perf ring buffer. This
buffer can contain any information they might need such as event type, source,
policy verdict, direction and more. Google further optimized this process by only
generating policy events on new connections, not on every packet observed, allowing
for a detailed policy logging engine.
Thanks to the partnership between Google and Cilium users can leverage this
functionality on Google Kubernetes Engine
The Future of eBPF-based Networking and Security by Thomas Graf (Isovalent)
Thomas Graf, CTO and co-founder of Isovalent, closed the keynote talks with “The Future of eBPF-based Networking and Security.” Before we can understand why
eBPF is the future, we need to briefly look at history. In the 1990s, networking
was almost entirely physical. It was also the time of dial-up modems. In 1999,
iptables was created as a successor for ipchains. In 2003, VLANs came to be, the Xen
hypervisor was born and EMC bought VMWare. KVM was merged into the Linux kernel in 2007, and the era of
virtualization was upon us. However, in the case of networking – almost nothing
had changed. In 2009, the first release of Open vSwitch started the network
virtualization era. In 2010, OpenStack was born. 2013 brought us Docker, which
directly inherited networking from the virtualization layers and containers were
initially treated like miniature VMs. In 2014, Kubernetes was born which
deliberately did not make many networking implementation assumption but was
initially still heavily based on iptables, a system that literally was designed
15 years prior when modems ruled.
In that same year, 2014, eBPF was initially introduced to the Linux kernel.
2015 saw Linux networking become programmable and in 2016 XDP was merged into
the kernel – a source of many load balancing solutions which are built on eBPF.
In 2016, Cilium project was first announced, designed entirely from scratch and
based on eBPF as its foundation. The transition was complete. We had moved from
an era first built on top of physical servers, to an era based on virtual
machines where software defined networking was first born, finally to today
where the networking and security layers are built specifically for
cloud-native workloads. Google’s adoption of the eBPF-based Cilium project for
its flagship Kubernetes products GKE and Anthos are great examples on how eBPF
plays a central role in networking and security for Kubernetes and cloud-native
What’s Next? Thomas believes the industry will start to focus on edge load
balancing with eBPF and XDP. He sees continuing opportunities to improve
application awareness by further improving pod-to-pod and socket-to-socket
networking. Similarities between API and system calls will lead to a more
unified approach to network and system
security, with less separation of powers between policy, enforcement and alerting.
Finally, he sees an opportunity for additional support to bridge physical and
virtualized or containerized workloads, as well as eBPF-based service meshes
being implemented more in the kernel.
For the Cilium project, it has been an exciting trip over the past 4 years. It’s
almost certain to become more exciting as eBPF opens up additional possibilities,
not just for networking but for all forms of observability, security, tracing –
and so much more.
After another break, we returned for another incredible round of lightning talks,
all of which were just 5 minutes in length but were absolutely full of amazing
- Giulia Frascaria (Vrije Universiteit Amsterdam) – Can eBPF Save us from the Data Deluge? A Case for File Filtering in eBPF.
- Timo Reimann (Digital Ocean) – From Managed Kubernetes to App Platform: 1.5 Years of Cilium Usage at DigitalOcean
- Zain Asgar (Pixie) – Debugging Go in Prod with eBPF
- Martynas Pumputis (Isovalent) – North-South Load Balancing of Kubernetes Services with eBPF/XDP
- Luan Guimarães (Wildlife Studios) – Global Gaming Infrastructure with Cilium
- Ramiro Berrelleza (Okteto) – The Tale of Smokey and the Crypto Bandits
- Vlad Ungureanu (Palantir) – Past, Present, and Future of Cilium and Hubble at Palantir
- Itay Shakury (Aqua Security) – Tracing and Detecting Malware using eBPF
- Natalia Reka Ivanko (Isovalent) – Identity Aware Threat Detection and Network Monitoring by using eBPF
- William Findlay (Carleton University) – bpfbox: Simple Precise Process Confinement with eBPF and KRSI
- Lorenz Bauer (Cloudflare) – How to Ship BPF with your Go Project
- Yutaro Hayakawa (LINE) – eBPF at LINE’s Private Cloud
- Dinesh Venkatesan (Microsoft) – Building a Behavioral Knowledge Graph using eBPF
- Andrey Ignatov (Facebook) – Containers and BPF: twagent story
We hope you had as much fun attending the eBPF Summit as it was for us to host this event.
We’d like to give a huge thanks to everyone who attended, all of our speakers and
the amazing team of people who worked in the background to support the event to
make it a success.