carm benchmark

Run micro-benchmarks to measure peak arithmetic performance and memory bandwidth across cache levels. This is the primary subcommand for constructing the Cache-Aware Roofline Model.

Run carm benchmark --help for a complete listing of all arguments, types, and defaults.

Basic Usage

# Full roofline benchmark (all available ISAs)
carm benchmark --test roofline

# Thorough exploration of the parameter space
# Automatically tests all parameter combinations
carm benchmark --test roofline \
    --isa x86 x86_avx2 x86_avx512 \
    --data-type f32 f64 \
    --ld-st-ratio 2:1 1:1 1:0 0:1 \
    --threads 1 4 8 16 \
    --instruction add fma div

# Arithmetic-only test with specific data types
carm benchmark --test arithmetic --data-type f32 f64 --instruction add fma

# Memory bandwidth test targeting L1 only
carm benchmark --test memory --mem-target L1 --ld-st-ratio 2:1

# Dry run (generate code only, inspect generated assembly)
carm benchmark --test arithmetic --dry-run

# Generate a TOML config template for custom cache topology
carm benchmark --emit-config my_topology.toml

Test Types

Test	Use this to…
roofline	Default. Get the full picture: peak compute + bandwidth for each cache level in one run, constructing the CARM.
arithmetic	Measure GFLOP/s for specific instruction/data-type combinations. Useful when you only care about peak performance.
memory	Measure bandwidth for specific cache levels. Useful when you only care about memory bandwidth.
memory_sweep	Profile bandwidth across a range of buffer sizes to observe the full memory bandwidth curve.
mixed	Measure arithmetic performance and memory bandwidth at selected arithmetic intensities.

Arguments by Category

Which ISAs and hardware to target (--isa, --vector-length, --vector-lmul, --frequency)

By default the tool auto-detects all available ISAs on your system. Use --isa to restrict testing to a specific ISA (e.g. x86_avx2), which is useful when cross-compiling or when you only care about one vector extension. --vector-length overrides the detected vector register length for RVV/SVE. --vector-lmul sets the RISC-V LMUL group multiplier. --frequency overrides the detected frequency, make sure you use the correct value to avoid incorrect results. --set-frequency applies the aforementioned frequency (x86 only, requires root).

Custom cache topology (--topology-config, --emit-config)

The tool auto-detects your cache hierarchy from sysfs. If that fails or you want to model a different system, use --emit-config to dump a template TOML file, edit it with your cache parameters, and pass it via --topology-config.

What to measure (--test, --data-type, --instruction, --mem-target, --ld-st-ratio, --arith-mem-ratio, --ai-range, --ai-points, --mem-test-sizes, --num-ops)

These options control the test performed on the system. The main selection is --test, described above. Within a test you can narrow further: restrict data types with --data-type, pick arithmetic instructions with --instruction, choose memory levels with --mem-target, and control the load-to-store ratio with --ld-st-ratio. For the mixed test, use --ai-range MIN MAX to sweep an arithmetic-intensity range, and use --ai-points COUNT to set how many points to generate. Points are logarithmically spaced. Use --arith-mem-ratio ARITH:MEM to set the arithmetic-instruction to memory-instruction ratio for the repeating balanced block. This should match the microarchitecture’s ports, e.g., 2:3 for a microarchitecture with 2 arithmetic ports, 2 load ports and 1 store port. --mem-test-sizes lets you set exact array sizes per cache level instead of auto-sizing. --num-ops sets the arithmetic-test loop work.

How long to run (--test-time)

Each micro-benchmark loops until --test-time seconds elapse. The default (25s) gives reliable results on most systems. You can reduce this for a quicker benchmark, but accuracy may suffer. However, even 1 second is enough for a rough comparison between machines/configurations.

Threading (--threads, --interleaved)

Benchmark with multiple threads by passing one or more thread counts, e.g. --threads 1 4 8. The tool will run the full suite at each count. --interleaved changes thread pinning for NUMA systems where core indices are interleaved across nodes (e.g. node 0 has cores 0, 2, 4…).

Output and verbosity (--verbose, --name, --dry-run, --output-dir, --output-fmt, --keep-artifacts)

Results land in a directory under --output-dir (default: platform user data dir). --output-fmt controls which formats are written (table, plot, csv, jsonl). --name sets a machine name instead of the auto-generated one (derived from CPU model). --dry-run skips compilation and execution, useful for inspecting generated assembly. --keep-artifacts preserves the temporary source/binaries for debugging. Use --verbose/-v [0-4] for more detail (-v 4 for full debug).

Cross-compilation and simulation (--compiler, --sim-cmd)

Under development

Target a non-native architecture by setting --compiler to a cross-compiler (e.g. riscv64-linux-gnu-gcc) and optionally --sim-cmd to run the binary under a simulator. Use {binary} as a placeholder in the sim command, e.g. --sim-cmd "qemu-riscv64 {binary}" or --sim-cmd "sde -mix -- {binary}".

Output Files

Results are written to the output directory in a subdirectory named after the run. Typical outputs:

• Roofline plot (PNG): visual roofline model with measured points
• Arithmetic performance plot: GFLOP/s across operations and data types
• Memory bandwidth plot: bandwidth per cache level
• CSV/JSONL: raw data consumed by carm gui for roofline visualization