mohitmishra786/kernel-internals

Kernel Internals

Purpose

Guide agents through Linux kernel internals: the CFS and EEVDF schedulers, runqueues and vruntime, the buddy allocator and SLUB, vmalloc vs kmalloc, VFS dentry/inode/file objects, page cache and readahead, memory zones, OOM killer heuristics, and /proc/meminfo interpretation.

When to Use

• Diagnosing scheduling latency or unfair CPU distribution
• Understanding kmalloc failures vs vmalloc for large kernel allocations
• Analyzing page cache behavior and readahead effectiveness
• Interpreting /proc/meminfo during memory pressure
• Debugging OOM killer victim selection
• Reading kernel source in kernel/sched/, mm/, or fs/

Workflow

1. Scheduler — CFS and EEVDF

Linux uses EEVDF (Earliest Eligible Virtual Deadline First) for CFS-class scheduling. EEVDF was merged as an option in 6.6 (2023); the gradual CFS→EEVDF transition completed in 6.12 (Nov 2024). On older 6.6–6.11 kernels, verify which scheduler is active. Core concepts (runqueues, vruntime/lag) carry over:

Per-CPU runqueue (struct rq)
├── cfs_rq — fair-class tasks sorted by vruntime/deadline
├── rt_rq  — real-time tasks (FIFO/RR)
└── dl_rq  — SCHED_DEADLINE tasks

Key metrics:

# Task scheduling info
chrt -p <pid>
cat /proc/<pid>/sched

# Runqueue latency (scheduler debugging)
cat /sys/kernel/debug/sched/debug  # requires debugfs

# Trace scheduler events
perf sched record -a -- sleep 5
perf sched latency

vruntime — virtual runtime tracking CPU time consumed; lower vruntime = more eligible for CPU.

# CFS tunables
sysctl kernel.sched_latency_ns
sysctl kernel.sched_min_granularity_ns
sysctl kernel.sched_wakeup_granularity_ns

2. EEVDF specifics

EEVDF picks the task with the earliest eligible virtual deadline, improving latency fairness for short time-slice tasks. Tasks with positive lag are eligible; the scheduler selects the earliest virtual deadline among eligible tasks.

# Check kernel version (EEVDF transition complete in 6.12+)
uname -r

# Scheduler documentation
# docs.kernel.org/scheduler/sched-eevdf.html

3. Memory subsystem — buddy allocator

Physical pages allocated in power-of-two order (order 0 = 4KB, order 1 = 8KB, ...).

ZONE_DMA / ZONE_DMA32 / ZONE_NORMAL / ZONE_MOVABLE
└── free_area[MAX_ORDER] — buddy lists per order

# Buddy allocator stats
cat /proc/buddyinfo

# Per-zone page counts
cat /proc/zoneinfo | head -80

4. SLUB allocator

Default kmalloc backend — per-CPU caches (slabs) for common sizes.

# SLUB debug (requires CONFIG_SLUB_DEBUG)
cat /sys/kernel/slab/*/objects 2>/dev/null | head

# kmalloc size classes visible in /proc/slabinfo
cat /proc/slabinfo | head -20

| API | Use when | |-----|----------| | kmalloc(size, GFP_KERNEL) | ≤ ~128KB (arch-dependent), physically contiguous | | kzalloc(size, flags) | Zeroed kmalloc | | vmalloc(size) | Large, virtually contiguous (may be physically fragmented) | | get_free_pages() | Direct page allocation |

// Kernel module allocation example
void *buf = kmalloc(4096, GFP_KERNEL);
if (!buf)
    return -ENOMEM;
kfree(buf);

5. Highmem

On 32-bit or specific configs, ZONE_HIGHMEM holds pages not permanently mapped in kernel virtual address space. On 64-bit x86/arm64, essentially all RAM is in ZONE_NORMAL.

grep -i highmem /proc/zoneinfo

6. VFS layer

Path lookup: /home/user/file.txt
├── dentry cache (dcache) — directory entry tree
├── inode — metadata (permissions, size, ops)
└── file — per-open-file state (offset, flags)

# Mounted filesystems
cat /proc/mounts

# Inode/dentry cache pressure
sysctl vm.vfs_cache_pressure   # higher = reclaim caches sooner

# File descriptor usage
ls /proc/<pid>/fd | wc -l

7. Page cache and readahead

# Drop caches (testing only — destructive to perf)
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

# Readahead tuning
blockdev --getra /dev/sda
blockdev --setra 4096 /dev/sda

# Per-file cache status
cat /proc/<pid>/smaps_rollup

Page cache pages appear as Cached in meminfo. Dirty pages await writeback.

8. /proc/meminfo interpretation

cat /proc/meminfo

| Field | Meaning | |-------|---------| | MemTotal | Total usable RAM | | MemFree | Completely unused pages | | MemAvailable | Estimate of allocatable memory (includes reclaimable cache) | | Cached | Page cache + tmpfs | | Buffers | Block device metadata cache | | SwapTotal / SwapFree | Swap space | | Dirty | Pages pending writeback | | AnonPages | Anonymous (heap/stack) pages | | Slab | Kernel object cache | | SReclaimable | Reclaimable slab | | SUnreclaim | Kernel structures (not easily reclaimed) |

Memory pressure diagnosis
├── MemAvailable low + Cached high → page cache reclaim candidate
├── AnonPages high + SwapFree low → OOM risk
├── Slab huge → kernel object leak; check /proc/slabinfo
└── Dirty high → slow writeback; check I/O scheduler

9. OOM killer

# OOM score per process (higher = more likely victim)
cat /proc/<pid>/oom_score
cat /proc/<pid>/oom_score_adj   # -1000 to 1000, admin adjustment

# OOM events in kernel log
dmesg | grep -i "out of memory"
journalctl -k | grep -i oom

OOM killer selects based on oom_score considering memory usage, child processes, and oom_score_adj. Protect critical daemons:

# Protect process from OOM (requires root)
echo -1000 | sudo tee /proc/<pid>/oom_score_adj

Common Problems

| Symptom | Cause | Fix | |---------|-------|-----| | kmalloc: allocation failed | Fragmentation or size too large | Use vmalloc; reduce pressure; GFP_ATOMIC only when needed | | High iowait, low MemAvailable | Page cache thrashing | Increase RAM; tune vfs_cache_pressure | | OOM kills wrong process | High memory user with low adj | Set oom_score_adj; use cgroups memory.max | | Scheduling unfairness | RT tasks starving CFS | Check chrt; isolate CPUs with isolcpus | | SLUB corruption | Use-after-free in module | Enable KASAN; audit with slub_debug | | Slow file reads | Readahead too small | Increase readahead; check backing device |

Related Skills

• skills/kernel/device-drivers — char/platform drivers using these subsystems
• skills/kernel/kernel-debugging — ftrace, kgdb for internals investigation
• skills/low-level-programming/linux-kernel-modules — LKM development basics
• skills/kernel/kernel-testing — KUnit for subsystem unit tests
• skills/profilers/linux-perf — perf sched and memory profiling
• skills/observability/ebpf — trace scheduler and memory events from userspace