Magic Numbers in Linux: A Deep Systems Guide

A low-level, systems-oriented guide to understanding how Linux determines file types using magic numbers — going beyond basic definitions into kernel vs. user-space behavior, libmagic internals, binary layouts, and security implications.

What magic numbers really are

Magic numbers are constant byte sequences embedded into binary data structures to provide early identification hints for file formats, executables, disk layouts, and network protocols. They act as a weak contract between producers and consumers of binary data.

Key properties:

• Positioned at well-known offsets
• Endian-aware when numeric
• Designed for fast rejection of invalid data

Why file extensions fail

Linux always trusts content over name.

Magic numbers vs MIME types

file --mime-type bridges both worlds.

Kernel vs. user-space detection

Kernel:

• Does not rely on file magic
• Executes ELF only if loaders exist
• Uses shebang (#!) interception

User space:

• file, desktops, scanners
• Powered by libmagic

libmagic architecture

Execution pipeline:

open() → read() → pattern engine
     → indirect offsets
     → strength scoring
     → description synthesis

Capabilities: conditional rules, nested patterns, offset arithmetic.

File header layouts

Example (PNG):

Offset  Size  Meaning
0x00    8     Signature
0x08    4     IHDR length

Magic numbers rarely act alone — they validate structure presence, not full correctness.

Common magic numbers (extended)

Notice: not all magic appears at offset 0.

ELF internals

Beyond the magic bytes: e_ident[EI_CLASS], e_ident[EI_DATA], program headers.

readelf -h file

Executable detection & shebang

#! works via kernel loader redirection:

#! /usr/bin/python3

Kernel steps: reads the first 2 bytes, parses the interpreter path, re-execs.

Disk structures as magic numbers

MBR: offset 510 → 0x55AA.

GPT: primary header at LBA 1, magic = EFI PART.

Disk formats are file formats.

Byte-level inspection

xxd -l 64 file
hexdump -C file
od -An -tx1 file

Writing file detectors

Python:

magic = f.read(4)

C:

memcmp(buf, "\x7FELF", 4)

Avoid trusting magic alone.

Advanced magic files

Features: indirect offsets, numeric comparisons, strength modifiers.

0  string  ELF
>4 byte =2 64-bit ELF

Security implications

Threats: polyglot files, content spoofing, malformed headers.

Mitigations: full structure validation, size & bounds checking, sandboxing.

Reverse engineering

Magic numbers help identify packed binaries, firmware, and unknown dumps — paired with binwalk, strings, radare2.

Cross-platform notes

Multi-format binaries exist.

Performance considerations

Early short-circuit, read minimal bytes, cache detection results — used heavily by antivirus engines.

Practical labs

Beginner → dump headers. Intermediate → parse MBR. Advanced → implement a detector. Expert → fuzz malformed headers.

Further reading

man magic, libmagic source, the OSDev Wiki, the ELF specification.

This guide is aimed at Linux system programmers, reverse engineers, security researchers, and low-level developers. Understanding magic numbers means understanding how systems recognize reality from raw bytes.