CodeBlueprint · Cloud/Edge, robotics, distributed platforms

Cross-Compiling C++ with Bazel: Platforms, Toolchains and the Exec/Target Split

Mon, 19 Oct 2026 00:00:00 GMT

Alamedin gorge. The tent goes where the ground is flat, not where the view is. Nothing required those to be the same place.

As far as I’m concerned, Bazel is the best build tool you can pick today, as long as it supports your language. A real dependency graph, hermetic toolchains, a remote cache that actually works, one command to build and test everything. And yet every time I bring it up with embedded people, the first question is the same: fine, but what about cross-compilation?

Fair question. Most of us grew up with a vendor SDK that hijacks the shell through an environment-setup script, or with the “just build it on the board” approach because it looks simpler. It isn’t simpler. A Raspberry Pi 5 is a decent target and a lousy build machine, and nothing about an ARM CPU requires the compiler to run on it.

So here’s a small and fairly well-known example, done the way I think it should be done: C++23 built on an x86_64 Linux host for an aarch64 Linux device. The device is a Pi 5 because that’s the board sitting on my desk. Swap in any ARM64 Linux board and the mechanics stay the same. The code lives in pi-bazel¹, and the whole cross setup is a 137-line MODULE.bazel, a 23-line .bazelrc and two platform() declarations. No hand-written cc_toolchain_config, no select() anywhere in the tree, no #ifdef on architecture in the sources.

Platforms Instead of `--cpu`

If you touched Bazel C++ cross builds a few years ago, you remember --cpu=aarch64 plus --crosstool_top. The CPU name was a lookup key into a crosstool file, the host compiler was a second lookup through --host_cpu, and anything that wasn’t a C++ compiler had no mechanism at all. Forget it. Bazel 9 still accepts --cpu, but nothing in this repo reads it.

The current model is constraint matching². A platform is a set of constraint values (CPU, OS, whatever axis you define). A toolchain declares two sets of constraints: exec_compatible_with says where the tool runs, target_compatible_with says what it produces output for. Every build has one target platform, set with --platforms, and a list of execution platforms where actions may run. For each toolchain type a rule needs, Bazel walks the registered toolchains and takes the first one whose target side fits the target platform and whose exec side fits some available execution platform³.

And that’s the whole trick. Cross-compilation stops being a special mode: it’s simply the case where the matched execution platform and the target platform differ.

Here’s every platform the project defines:

bazel/platforms/BUILD.bazel:1-17

package(default_visibility = ["//visibility:public"])

platform(
    name = "host",
    constraint_values = [
        "@platforms//cpu:x86_64",
        "@platforms//os:linux",
    ],
)

platform(
    name = "pi5",
    constraint_values = [
        "@platforms//cpu:aarch64",
        "@platforms//os:linux",
    ],
)

Two constraints each, all from @platforms. You might expect a Cortex-A76 constraint on pi5, but @platforms//cpu only knows cortex-r52 and cortex-r82 among the cortex-* values, so A76 tuning ends up as a compiler flag, attached to the Pi’s toolchain in MODULE.bazel (next section). A project-local constraint_setting for the microarchitecture would earn its place the day two aarch64 toolchains have to be told apart during resolution. With one target board, it’s ceremony.

Strictly speaking, :host is optional, since Bazel’s default host platform already carries x86_64 and linux. I keep it anyway because it gives CI a label to pin and gives the execution platform list a first entry I control (more on that later).

Note the bottom box. Both builds run every action on the same x86_64 machine; only the target side of the match moves.

One Config per Target

Picking the target is a named config in .bazelrc. bazel build --config=host //... is the host build, for the workstation; bazel build --config=pi //... is the Pi build. Each config is nothing but a platform choice, and this is every per-target line in the file:

.bazelrc:14-16

build:host --platforms=//bazel/platforms:host

build:pi --platforms=//bazel/platforms:pi5

The whole .bazelrc is 23 lines, and the rest is lockfile strictness, three output-directory flags covered near the end, a CI config, test output settings and a user.bazelrc import. The A76 tuning used to live here too, as two build:pi --copt lines. That had a real gap: the tuning was tied to a config name, not to the target, so a plain bazel build --platforms=//bazel/platforms:pi5 produced an untuned binary without a word of complaint. The flags now sit on the toolchain itself, keyed to the Pi’s target pair:

MODULE.bazel:77-84

    # @platforms has no cortex-a76 constraint value, so A76 tuning can only be a
    # compiler flag rather than a platform property. Keyed to the target pair
    # rather than set under build:pi, so the tuning follows the toolchain: a
    # plain --platforms=//bazel/platforms:pi5 build gets it too.
    extra_compile_flags = {"linux-aarch64": [
        "-mcpu=cortex-a76",
        "-mtune=cortex-a76",
    ]},

aquery on that bare --platforms build found zero tuning flags before the move and two after. Keying by pair also keeps the flags out of everything else. The host compile action carries no cortex-a76 flag, and neither will a tool that runs during the build (a code generator, say), because those get built for the exec platform, which is x86_64-linux.

There’s no --extra_toolchains line, and that’s worth a detour, because an earlier version of this repo had one. The host toolchain was registered globally with register_toolchains("@llvm_host//:all") and the cross one was selected by label, on the theory that registering both would leave resolution order to arbitrate between them. The theory was right. The premise was wrong.

Ask toolchains_llvm what @llvm_pi actually generates and you get five:

$ bazel query '@llvm_pi//:all' | grep cc-toolchain-
@llvm_pi//:cc-toolchain-aarch64-linux
@llvm_pi//:cc-toolchain-riscv32-none
@llvm_pi//:cc-toolchain-riscv64-none
@llvm_pi//:cc-toolchain-x86_64-linux
@llvm_pi//:cc-toolchain-x86_64-none

That’s the aarch64-linux pair, the native x86_64-linux pair, and three bare-metal pairs that need no sysroot. Now run the same query against the host-only toolchain the repo used to have (git checkout f550049 first, since it no longer exists on main) and you get exactly those five minus aarch64-linux. toolchains_llvm emits a cross pair only when it has a sysroot for it, and the host toolchain had none, so it could never have shadowed the Pi. @llvm_pi is a strict superset: diff the two generated BUILD files at that commit and you get 144 added lines, nothing removed and nothing changed.

Which inverts the fix. Registering both would put two x86_64-linux candidates in front of every host build, with order picking the winner, so I registered @llvm_pi//:all alone and deleted the host toolchain entirely:

MODULE.bazel:134-137

# Registered globally rather than selected by label under build:pi. Nothing
# arbitrates: aarch64-linux is the only candidate for //bazel/platforms:pi5,
# x86_64-linux the only one for :host.
register_toolchains("@llvm_pi//:all")

That’s the whole mechanism, and it’s why .bazelrc above has nothing to say about toolchains any more: it picks a platform and the registration does the rest. One registration serves both configs, and the --extra_toolchains line goes away along with nineteen lines of MODULE.bazel. I compared the link command lines before and after under both configs: flag for flag identical, down to the sysroot path and the order of the archives. The only thing that changes is which repo’s cc_wrapper.sh runs, and the two are byte-identical.

The :all in that line isn’t special syntax. It’s the ordinary “everything in this package” pattern, and @llvm_pi holds 120 targets: filegroups, the cc_toolchain rules, wrappers for clangd and clang-format, and so on. register_toolchains silently skips everything that isn’t a toolchain() and keeps the five that are. I’d rather have that than a hand-written list of labels. A toolchains_llvm upgrade that adds or renames a target would leave an explicit list quietly incomplete, while :all just picks up whatever got generated.

Registering a toolchain doesn’t mean it gets used, though. All five land in the candidate list, and resolution throws out the ones whose constraints don’t fit. --toolchain_resolution_debug shows it happening (repo prefixes shortened, most rejections cut):

$ bazel cquery --config=pi \
    --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' \
    //src/hello:hello

ToolchainResolution:   Toolchain @@…llvm_pi//:cc-toolchain-aarch64-linux (resolves to
  @@…llvm_pi//:cc-clang-aarch64-linux) is compatible with target platform, searching
  for execution platforms:
ToolchainResolution:   Rejected toolchain @@…llvm_pi//:cc-toolchain-x86_64-linux
  (resolves to @@…llvm_pi//:cc-clang-x86_64-linux) ; mismatching values: x86_64
ToolchainResolution:   Rejected toolchain @@…local_config_cc_toolchains//:cc-toolchain-k8
  (resolves to @@…local_config_cc//:cc-compiler-k8) ; mismatching values: x86_64

Every line names two targets because cc-toolchain-* is the toolchain() wrapper that gets registered, and cc-clang-* is the actual cc_toolchain behind it. The last line isn’t mine at all: rules_cc (and apple_support) register their own autodetected toolchains, and resolution walks past those too. If a rerun prints nothing, that’s the analysis cache. The debug output only appears when resolution actually runs.

There’s one cost. With @llvm_host gone, host builds come out of @llvm_pi too, from its x86_64-linux toolchain. So every setting in the llvm.toolchain(name = "llvm_pi") block also applies to host builds, unless it’s keyed to a specific target pair. stdlib is the one that matters here. It’s keyed to linux-aarch64, so only Pi builds get libc++. Change the key to "" (an easy mistake when the block is literally named llvm_pi) and host builds quietly switch away from builtin-libc++ as well. A separate host toolchain made that mistake impossible. Now the only guard is a comment above that line in MODULE.bazel.

Where the Toolchain Comes From

Writing a cc_toolchain_config by hand is a waste of a weekend. toolchains_llvm⁴ generates the toolchain, its features and its flag sets from a few tags in MODULE.bazel, so the interesting work is in the parts it can’t decide for you.

The first one surprised me. An LLVM release tarball is two different things glued together. LLVM-23.1.0-Linux-X64 carries clang, clang++ and ld.lld as x86_64 binaries, which every build needs to run, plus x86_64 headers and lib/x86_64-unknown-linux-gnu/libc++.a, which the host build links and the Pi build can’t use. The aarch64 archives only exist in LLVM-23.1.0-Linux-ARM64, whose binaries can’t execute on my workstation at all. So the Pi toolchain, @llvm_pi’s aarch64-linux entry, is assembled from two distributions:

MODULE.bazel:74-115, with the two flag attributes elided:

llvm.toolchain(
    name = "llvm_pi",
    cxx_standard = {"": "c++23"},
    # extra_compile_flags = {...}, extra_link_flags = {...}
    llvm_version = _LLVM_VERSION,
    # Keyed to the aarch64 target rather than "" because this toolchain also
    # serves host builds: an empty key would move them off builtin-libc++ too,
    # with nothing in a --config=host command line pointing at the cause.
    stdlib = {"linux-aarch64": "libc++"},
)
llvm.toolchain_root(
    name = "llvm_pi",
    label = "@llvm_x64//:BUILD",
)
llvm.target_toolchain_root(
    name = "llvm_pi",
    label = "@llvm_arm64//:BUILD",
    targets = ["linux-aarch64"],
)
llvm.sysroot(
    name = "llvm_pi",
    label = "@pi_sysroot//sysroot",
    targets = ["linux-aarch64"],
)

toolchain_root is the distribution whose binaries run. target_toolchain_root is the one you compile against and link with. Both tags read only the package path out of the label, which explains the odd-looking @llvm_x64//:BUILD. (Both trees are fetched by sha256 through a small repository rule of mine. It exists for a linker problem with my workstation’s system libraries that has nothing to do with cross-compilation, so I’ll skip it here.)

@llvm_arm64 supplies more than the aarch64 runtime archives (libc++.a, libc++abi.a, libunwind.a, compiler-rt). Run aquery on a Pi compile action and you’ll see -resource-dir pointing at @llvm_arm64/lib/clang/23, -nostdinc++, and two -cxx-isystem paths into @llvm_arm64/include. So the target-side C++ headers and the clang resource directory come from the ARM64 tree too. For the Pi build, the X64 tree contributes executables and nothing else. The host build gets everything from the X64 tree, because x86_64-linux has no target_toolchain_root and falls back to toolchain_root.

One thing I didn’t expect: pointing toolchain_root at a tree you’ve already fetched also skips toolchains_llvm’s own LLVM download. The generated @llvm_pi repo is 39 KB of BUILD files and a cc_wrapper.sh, not a third copy of a 12 GB distribution.

The Sysroot

The C side (glibc headers, libc.so.6, libm, libpthread) comes from a sysroot. The obvious way to bring one in is http_archive plus a filegroup with a glob(). Don’t. On the sysroot used here, that glob produces a warning naming 6,749 files, and every one of them becomes a separate input symlinked into every compile and link sandbox.

toolchains_llvm ships its own sysroot repository rule, and it declares the extracted tree as a single source directory (srcs = ["."]). That’s one input in the action graph and one symlink per sandbox instead of thousands. It also extracts with a hermetic bsdtar instead of whatever tar the build machine has.

For now the sysroot itself is borrowed. Chromium builds Debian sysroots for its own cross-compiles and publishes them as immutable tarballs, and toolchains_llvm’s own tests use the same ones, so it was the quickest way to get a working one. Mine is their Debian stretch arm64 image, with glibc 2.24. The Pi runs 2.41 and glibc is forward-compatible in that direction, so the artifacts carry a lower symbol floor than intended rather than a broken one. A Debian trixie snapshot replaces it next, and the label @pi_sysroot//sysroot won’t move when that happens.

This is what reaches the linker. I pulled it with bazel aquery --config=pi 'mnemonic("CppLink", //src/hello:hello)' and trimmed the hardening flags:

exec external/toolchains_llvm++llvm+llvm_pi/bin/cc_wrapper.sh \
  -o bazel-out/pi5-fastbuild/bin/src/hello/hello \
  --target=aarch64-unknown-linux-gnu -fuse-ld=lld \
  -resource-dir external/+llvm_distribution+llvm_arm64/lib/clang/23 \
  -rtlib=compiler-rt -l:libunwind.a -lpthread -ldl \
  bazel-out/pi5-fastbuild/bin/src/hello/_objs/hello/main.pic.o \
  -Wl,--start-lib bazel-out/.../hello_lib/hello.pic.o -Wl,--end-lib \
  --sysroot=external/+sysroot+pi_sysroot/sysroot/ \
  -Lexternal/+llvm_distribution+llvm_arm64/lib \
  -Lexternal/+llvm_distribution+llvm_arm64/lib/aarch64-unknown-linux-gnu \
  -l:libc++.a -l:libc++abi.a

The detail I care about most is that --sysroot is workspace-relative. Nothing in the command line knows where my checkout lives, so the action key doesn’t either. There’s no shared cache here yet, but once there is, a CI runner and my workstation will compute the same key for this link even though the checkouts live at different paths. An absolute path would have made every machine’s keys unique. (Why that output path says pi5-fastbuild gets a section near the end.)

If I had to recommend one Bazel habit for toolchain work, it’s this one: when something looks wrong, read the action with aquery⁵ instead of guessing. Every finding in the next section came out of a command line or an input list.

Defaults That Fail Quietly

libstdc++ through the back door

toolchains_llvm controls the C++ runtime through a stdlib attribute, and the default is builtin-libc++: static libc++ and libc++abi, plus compiler-rt and libunwind. Sounds exactly like what I want. But when exec and target architectures differ, the toolchain config rewrites that default:

toolchain/cc_toolchain_config.bzl:437-438 (toolchains_llvm 1.9.0)

    if stdlib == "builtin-libc++" and is_xcompile and not is_darwin_exec_and_target:
        stdlib = "stdc++"

Leave stdlib alone and the Pi build takes its C++ standard library from the sysroot. With the stretch sysroot this project uses today, you find out immediately, because nothing compiles:

./src/hello/hello.h:3:10: fatal error: 'expected' file not found
    3 | #include <expected>
      |          ^~~~~~~~~~

aquery shows why. The compile action gets -stdlib=libstdc++ and no -nostdinc++, so <expected> is looked up in external/+sysroot+pi_sysroot/sysroot/usr/include/c++/6.3.0, the libstdc++ headers that came inside the sysroot tarball. The build machine’s own /usr/include plays no part. Debian stretch ships GCC 6.3, and its libstdc++ has no <expected>, <flat_map> or <print>. With the libc++ key set, the same action gets -nostdinc++ and two -cxx-isystem paths, and the headers come from @llvm_arm64/include/c++/v1 instead.

A failing compile is the lucky outcome. With a sysroot whose libstdc++ is new enough, the build goes green and nothing tells you that the binary now loads libstdc++.so.6 from the Pi at runtime, or that hello_test on the workstation exercises libc++ while the Pi runs libstdc++. The neighbouring branch, an empty stdlib on a cross build, at least prints a warning. This one prints nothing.

The fix is stdlib = {"linux-aarch64": "libc++"}, keyed by the target pair. An empty key would apply to every pair, including x86_64-linux, and drag the host build off builtin-libc++ as well.

Except the libc++ value isn’t builtin-libc++ minus the cross rewrite. It emits -l:libc++.a -l:libc++abi.a and nothing more, so the first link died like this:

ld.lld: error: undefined symbol: __aarch64_ldadd8_acq_rel
ld.lld: error: undefined symbol: __aarch64_ldadd8_relax
ld.lld: error: undefined symbol: __aarch64_cas8_acq_rel

Those are AArch64’s outline-atomics helpers, small functions the compiler calls for atomic operations. libc++.a and libc++abi.a both use them, and they live in compiler-rt, which the libc++ mode never links. It skips the unwinder, pthread and dl as well, all of which builtin-libc++ brings along for free. Four flags on the toolchain, keyed to the Pi’s pair again, put all of that back:

MODULE.bazel:85-95

    # The libc++ stdlib branch emits only -l:libc++.a -l:libc++abi.a, unlike
    # builtin-libc++: AArch64 outline-atomics live in compiler-rt, and libunwind,
    # pthread and dl are otherwise supplied for free. extra_link_flags appends to
    # the toolchain's own link_flags, so these land before the object files,
    # where builtin-libc++ puts them on the host.
    extra_link_flags = {"linux-aarch64": [
        "-rtlib=compiler-rt",
        "-l:libunwind.a",
        "-lpthread",
        "-ldl",
    ]},

-rtlib=compiler-rt brings in the atomics. -l:libunwind.a names the static archive on purpose: -unwindlib=libunwind looks equivalent, but it links the shared libunwind.so and puts a runtime dependency back on the Pi. -lpthread and -ldl resolve against the sysroot’s glibc.

The attribute name matters. toolchains_llvm has two families of flag attributes, both keyed by target pair. compile_flags, link_flags and link_libs replace the toolchain’s computed defaults, so using them means restating everything the toolchain already emits. Their extra_* twins append after the defaults, which is what you want almost every time.

Don’t count on upstream to catch a regression here. toolchains_llvm 1.9.0’s CI tests every stdlib value, but only in native builds⁶. Its one cross-compile test leaves stdlib unset⁷, so it takes the silent libstdc++ path described above. Nothing upstream builds cross with libc++, so in this repo the ABI check further down is the only guard on that combination.

A file nobody declared

The second quiet failure looked like a broken include path and wasn’t one:

fatal error: '__config_site' file not found

The -cxx-isystem path pointed exactly where __config_site lives in the ARM64 tree. The file simply wasn’t in the sandbox. Each compile action’s inputs come from a generated filegroup:

toolchain/internal/configure.bzl:706-715 (toolchains_llvm 1.9.0)

filegroup(
    name = "compiler-components-{suffix}",
    srcs = [
        ":cxx_builtin_include_files-{suffix}",
        ":sysroot-components-{suffix}",
        "{llvm_dist_label_prefix}extra_config_site",
        "{toolchain_root}:clang",
        {extra_compiler_files}
    ],
)

{llvm_dist_label_prefix} is the exec distribution. With one distribution serving both sides that’s correct. With X64 as exec and ARM64 as target, it declares the x86_64 __config_site and nobody declares the aarch64 copy that the include path actually resolves to. The template quietly assumes exec and target are the same tree.

The fix is one tag:

MODULE.bazel:127-131

llvm.extra_compiler_files(
    name = "llvm_pi",
    label = "@llvm_arm64//:extra_config_site",
    targets = ["linux-aarch64"],
)

This is the pattern to recognise, because any cross setup built from two trees will hit it again: a sandboxed action fails not because a path is wrong, but because a correct path was never declared as an input. The error talks about a missing file. The actual bug is a missing declaration. extra_compiler_files and its twin extra_linker_files are the escape hatches, and aquery’s input list tells you which one you need.

An Execution Platform Nothing Runs On

This block looks wrong, and most reviewers would recommend deleting it:

MODULE.bazel:15-21

# Test-toolchain resolution needs a registered execution platform matching the
# target platform before it consults a target's exec_compatible_with, so :pi5
# must be registered even though no action ever runs on aarch64 hardware.
register_execution_platforms(
    "//bazel/platforms:host",
    "//bazel/platforms:pi5",
)

No action in this project ever executes on aarch64. Remove :pi5 anyway, and bazel test --config=pi //... fails at analysis:

ERROR: .../bazel/deploy/BUILD.bazel:5:8: While resolving toolchains for target
//bazel/deploy:device_smoke (d9c8cc0): No matching toolchains found for types:
  @@bazel_tools//tools/test:default_test_toolchain_type: By default, tests are
  executed on the first execution platform that matches all constraints specified
  by the target platform. Either register the target platform (or a platform with
  at least the same constraints) as an execution platform, or override the default
  behavior by registering a custom toolchain for
  @bazel_tools//tools/test:default_test_toolchain_type.

To its credit, Bazel 9 says exactly what to do. What’s surprising is the reason. A test whose target platform is aarch64 needs an aarch64 execution platform to exist, even when its own exec_compatible_with sends every action to x86_64. Registering :pi5 satisfies that check, and nothing ever runs on it. The other way out, a custom toolchain for default_test_toolchain_type, is more machinery for the same result. Restore the line and the suite goes green again.

Order matters here too. :host is listed first, so every action that doesn’t constrain its own exec platform lands on the build machine. In a --config=pi build the compile, link and genrule actions all report Execution platform: //bazel/platforms:host, which is what you want.

Tests That Know Which Side They’re On

target_compatible_with is how one bazel test //... stays correct under both configs:

Target	`target_compatible_with`	`--config=host`	`--config=pi`
`//src/hello:hello_test`	x86_64	runs	skipped
`//bazel/deploy:cross_artifact_abi_check`	aarch64	skipped	runs, no Pi needed
`//bazel/deploy:device_smoke`	aarch64	skipped	runs, needs the Pi

Incompatible tests are skipped, not failed, and that cuts both ways. Incompatibility is transitive, data edges included: a CPU constraint on hello_lib would make hello incompatible in the Pi build, device_smoke with it, and the run would report green with the device test silently skipped. That’s why hello_lib stays unconstrained, with a comment saying so.

The two tests in bazel/deploy check an aarch64 binary from the x86_64 build machine, and Bazel can express that directly:

bazel/deploy/BUILD.bazel:41-56

sh_test(
    name = "cross_artifact_abi_check",
    size = "small",
    srcs = ["cross_artifact_abi_check.sh"],
    args = ["$(rlocationpath @llvm_x64//:readelf)"],
    data = [
        "//src/hello",
        "@llvm_x64//:readelf",
    ],
    exec_compatible_with = [
        "@platforms//cpu:x86_64",
        "@platforms//os:linux",
    ],
    target_compatible_with = ["@platforms//cpu:aarch64"],
    deps = ["@rules_shell//shell/runfiles"],
)

target_compatible_with puts the test in the Pi build, so data brings in //src/hello as the aarch64 binary. exec_compatible_with keeps the test itself, the cross_artifact_abi_check.sh script, on the x86_64 build machine. @llvm_x64//:readelf sits in data too and survives the aarch64 configuration only because it’s a filegroup around a prebuilt binary; a cc_binary in the same slot would come out as aarch64 and fail to start. With that readelf, the script checks the two things all the toolchain work was for: the artifact is AArch64, and its NEEDED list has libc.so.6 but no libstdc++, libc++ or libgcc_s.

device_smoke has the same shape, plus scp and ssh to prove the binary actually runs on the Pi. env_inherit forwards HOME, SSH_AUTH_SOCK and PI_SSH_HOST to this one test only, and the external tag keeps its result out of the cache.

One Output Directory for Two Targets

Until recently, both configs wrote to the same place. bazel aquery on the link action reported bazel-out/k8-fastbuild/bin/src/hello/hello under --config=host and --config=pi alike, so bazel-bin/src/hello/hello was whichever build ran last. I hit it while checking something else: an x86-64 ELF sitting at the path an aarch64 build had produced earlier in the same session. It’s also why the ABI check lives inside the build graph and not as a file bazel-bin/... line in the README.

k8 is Bazel’s legacy CPU name for x86_64, and despite the look, no relation to k8s. It’s named after AMD’s K8, the Athlon 64 and Opteron family that shipped the first x86-64 chips, and it came over from Blaze, Google’s internal build system, together with piii for 32-bit x86. rules_cc still carries the mapping⁸:

return "k8" if arch in ["amd64", "x86_64", "x64"] else "piii"

The output directory is named <cpu>-<compilation_mode>, and <cpu> comes from the deprecated --cpu flag. Nothing here sets it, so Bazel falls back to the host’s legacy name and every build gets k8, the Pi build included. That’s also why bazel config shows cpu: k8 for the Pi configuration right next to platforms: //bazel/platforms:pi5. It looks contradictory, but nothing reads that field, and $(TARGET_CPU) correctly says aarch64. The CPU value was never wrong. The target platform just never reached the path.

Three lines in .bazelrc fix it:

.bazelrc:10-12

common --experimental_platform_in_output_dir=yes
common --experimental_override_platform_cpu_name=//bazel/platforms:host=host
common --experimental_override_platform_cpu_name=//bazel/platforms:pi5=pi5

The first puts a platform name in the output path instead of the CPU. The other two pick those names explicitly instead of leaving them to an inference scheme Bazel documents as experimental. Outputs now land in bazel-out/host-fastbuild and bazel-out/pi5-fastbuild and coexist on disk. The lines use common rather than build so test, cquery and aquery all agree on the paths. Both flags are still experimental, so a future Bazel may rename them.

Two things stay as they were. bazel-bin is a single symlink and still follows the last build, so a stable path means asking bazel cquery --output=files. And switching configs still discards the analysis cache, because --platforms is a configuration input no matter where the outputs land.

Next Board, Next Steps

Moving this to a different aarch64 Linux board is a short edit: a new platform() (possibly with the same two constraints), different -mcpu/-mtune in extra_compile_flags, a different sysroot. A different architecture, say armv7 or riscv64, adds a matching target_toolchain_root distribution and a new stdlib key. src/ doesn’t change in either case, which was the point of keeping every per-target difference in the toolchain declaration.

For this project, the next step is replacing the stretch sysroot with a Debian trixie snapshot that matches the Pi’s actual glibc. After that, GStreamer and friends go in through third_party/ as C libraries. That last part matters more than it sounds: the static libc++ decision only holds as long as no C++ library from the sysroot ever lands on the link line. Then a remote cache, because ~/.cache/bazel on my workstation is already at 62 GB.

Context enrichment for multi-agent code review

Tue, 29 Sep 2026 00:00:00 GMT

Kok-Moinok lake, eleven kilometres and 1,100 metres above the trailhead. No preview on the way up: you get the whole thing at the end or nothing.

When a model gets something wrong, we reach for the same words: it hallucinated, it ignored the instructions, it isn’t smart enough yet. Sometimes that’s true. More often, in my experience, the model did a decent job with what it was given, and what it was given was thin.

Think about what makes a developer good on a project they’ve been on for a while. It’s rarely typing speed. It’s the context they carry around: which approaches were tried and dropped, which subsystem already solves a similar problem, and above all why each decision was made. None of that is in the diff. It lives in their heads, in the docs, in old tickets, in a Slack thread nobody bookmarked.

A model can use that context too, to a degree, if we hand it over. The question is how: which inputs, in what shape, and how much of them.

Here’s the review that made me take that question seriously.

Two reviews, one document

A few weeks ago a design MR proposed a request/mailbox protocol for feature control. My review setup went through it (three Claude reviewers and a Codex cross-model pass, plus adversarial verification for anything only one agent raised¹) while I read the same document myself.

The panel returned 35 findings. Claim atomicity, a deny list that fails open, stale sessions, a camera daemon with no idle mode. Every one of them was correct.

I returned six Criticals, and none of them was on the panel’s list.

One pair shows the gap. The panel found that the camera daemon has no idle mode, so “idle until requested” can’t be implemented: add an idle mode. I found that “idle until requested” is state control, which this mechanism shouldn’t touch at all: delete the requirement. Same line of the design, opposite fixes.

Fixing all 35 would have produced a self-consistent design of the wrong system. I posted the six and held the rest back, since most of them die once the frame changes.

35 findings inside the frame, 6 Criticals aimed at the frame itself. The bold pair is the same line of the design.

What I had that they didn’t

Look at where my six came from.

Two of them pointed at work the roadmap already owned: a milestone planning the dependency graph that multi-hop requests need, and an epic scoping the audit that dynamic stopping depends on. One pointed at OTA, which already splits config ownership the way this design was trying to reinvent. And I’d already told the team in Slack that this could be simple, and that we already do something simpler. No reviewer prompt was ever going to include that thread.

The panel got the document, the diff and the code, with no browsing allowed. Not the repo’s docs, where almost every decision on this project is written down. A reviewer can’t cite what it was never given.

Missing inputs aren’t the only cause. Role bias works against premise attacks too, and so does the way consensus treats a lone dissent. Those get their own post. This one is about the inputs.

So give them more context

The fix follows from that picture: hand every reviewer the inputs my Criticals came from, or as many of them as exist in writing. On this project that’s most of them. Since mid-September the MR review gives every agent three more inputs.

The first, and the one that matters most, is a reference table over those docs. projctl, my open-source CLI for this workflow, has a search docs command: offline BM25 over the repo’s docs root and its planning/ tree, with every matched Markdown section sorted into a tier such as docs, roadmap, failure, alternative or constraint. The top 40 go into the prompt as rows of score, tier, repo, path and heading, with no body text at all.

A row costs a line and a body costs a page, so reviewers get the rows and open the bodies they want.

The second is the ticket. On a bug-fix MR the same week, the review command called projctl load issue to find the epic folder, kept that one line and dropped the description, the one part that explained the change. An intentional change came back as a defect. Now the description travels verbatim from the same call.

The third comes from the same bug-fix MR, where one finding rested on a 3 MiB log record that exists only in a test fixture. The instruction is one sentence: before judging whether something can happen, find what registers the changed code to run, what launches it and what CI selects. It names no paths, because those differ per repo.

Codex gets all three through its review-request document, since that’s the only thing it reads.

Everything that was written down now reaches every prompt. The Slack thread still doesn’t.

Two details matter as much as the search itself. The table opens with its query, its flag state and N of M shown, so a table cut at 40 rows never reads as full coverage. And the prompt frames everything as inputs, not gates:

platforms/claude/commands/review-mr.md:121

a hit in the failure or alternative tier is prior art to weigh, not a directive, and a reviewer handed unframed prior art treats it as authority and stops reasoning instead.

Measured against my six Criticals, that covers most of the ground. The two roadmap ones now have a tier of their own, and precedent like OTA arrives through the docs. The Slack thread stays out, and so does anything else nobody wrote down.

Three agents, one opinion

My consensus protocol has a shortcut. A finding raised by only one agent goes to adversarial verification, where another agent tries to refute it. A finding raised by two or more skips that step, on the theory that independent readers agreeing is evidence.

The readers aren’t independent, though. Every agent gets the same inputs, so whatever is missing from them, all three miss together, and their agreement waves the mistake past the one step built to catch it.

On the bug-fix MR, four findings were coherent against the diff and false against the deployment. Two questions from me killed them: “is it real or theoretical?” and “check the ticket description and the MR text”. Every one of the four had been raised by more than one agent, so none of them ever reached verification.

Three agents, one input, one opinion counted three times.

Richer inputs shrink what’s missing, but every agent still gets the same richer set. So agreement should count as a prior at most: a finding that claims something can happen at runtime needs verifying however many agents raised it.

Reading the proposal last

Inputs are the cheap fix, and they leave the order untouched: every reviewer still reads the proposal first, and whatever it reads first becomes its frame.

Next is a counter-designer: an agent that gets only the problem statement, the roadmap and an instruction to diff against the nearest in-repo precedent, writes the minimal design it would build, and only then reads the proposal. The calibration set already exists: the design MR from the top of this post. If it finds even one of my six Criticals without me in the loop, that’s the next post.

Code paths touched

projctl/handlers/docs_search.py: the BM25 search and the reference table it renders
platforms/claude/commands/review-mr.md, Step 3b: the three inputs every MR reviewer now gets

Making GenAI code review actually useful. sysdev.me, February 2026. ↩

I cancelled my JetBrains subscription

Mon, 07 Sep 2026 00:00:00 GMT

Sulutor Lakes. The only gate up here is the weather, and it's the only one that actually gets enforced.

Today I cancelled my JetBrains subscription. Cancelling subscriptions is a popular trend right now, except the ones people announce dropping are Claude, ChatGPT, whichever model annoyed them this week. I went the other way. The models stay. The IDE goes. For a line item on a credit card statement it feels more symbolic than it should.

I have always been a Vim person. That never changed, and it isn’t changing now. But I have been paying JetBrains since 2012, because for a long time Vim lost on two things. Navigation was the real gap: before language servers you had ctags, which is a text index pretending to understand code. No types, no scopes, five candidates for every common name. The other gap was visual. An IDE highlights faster and shows more, errors and types painted onto the code before you ask for them. Language servers closed the first gap. The second one JetBrains still wins, and the rest of this post is about why that no longer buys anything.

Somewhere along the way the reason changed. The products stopped being the thing I was buying and started being the thing I was supporting. If you want a good product to keep existing, the most honest thing you can do is pay for it, so I kept paying. Fourteen years of that.

What is actually on my screen in 2026

tmux for splits, Vim in most of them. Claude Code and Codex for writing code. Typora for reading and writing documents. Sublime Text and Sublime Merge for the cases where Vim is inconvenient, which in practice means big merge requests with a lot of changes, where I want a fast visual diff and nothing else.

There is no IDE in that list, and nothing lighter took its slot. The function it performed left my hands entirely.

The IDE is very good at something I no longer do

Completion, refactoring, jump-to-definition, inline inspections: the whole apparatus exists to make one person produce and navigate code by hand, quickly. It’s excellent apparatus. Thirty years of work went into it and it shows.

I don’t do that anymore. Writing code by hand in 2026 is an inefficient use of a senior engineer’s day. Less politely: it’s stupid. My time goes into deciding what should be built, putting a fence around it, and checking what came back against the fence. None of that happens in a buffer.

Nothing is wrong with JetBrains. Their tools are as good as they ever were. The category is what expired.

The obvious objection is the AI-native editors, Cursor and its relatives, and I don’t think they change the argument. Putting a model next to the buffer gives you a faster buffer. The buffer is not where my day goes. They are still code-centric tools; they just generate the code instead of autocompleting it.

What I would pay for and nobody sells

Here is where my money would go if the product existed.

My workflow with LLMs is a state machine. Research, design, design review, implementation, code review, verification, and back around. I drew the whole thing out earlier¹, so I won’t repeat it here. What matters is the shape: states, transitions, entry conditions, and gates that a transition is not allowed to skip.

Nothing on my screen understands any of that. tmux shows panes. Claude Code shows a conversation, a long one. Sublime Merge shows a diff. Three views of three artifacts, and no view of the process that produced them.

I want the opposite. Show me which state I’m in and which gate is holding. What evidence closed the last one? How much code did the last transition produce, against how much was declared up front? Then the part no tool even attempts: what is not allowed to happen next. And let me tune agent behaviour in one place, visually, instead of editing a scattered tree of markdown files and hoping the model reads the one I meant.

To be clear about the shape: this is not a CI harness. Orchestrators like Bernstein² take a goal in one end and hand back a verified merge at the other, and that’s useful, but I’m inside the loop at every transition, and the tool has to be too. Kiro³ got closer by putting requirements, design and tasks into the interaction itself, then wrapped it around a code editor and left the gates as conventions.

Non-determinism needs rules, not vibes

Underneath all of that is one problem: predictability.

I wrote last month about a feature I’d sized at roughly 200 lines coming back as 8,391 added⁴. Nothing went off the rails. Every guard defended against something real, every review finding was correct, and the total was still absurd. The fix I landed on was making each design option declare what it costs, so the expensive option has to justify itself against a number instead of against adjectives.

That’s a rule. Right now it’s enforced by being written as prose in a template that I then read carefully, which is the weakest enforcement mechanism available to a human being.

What I want is somewhere to declare rules like that and have something check them. The volume of code a transition may produce is bounded, and the bound is declared before the transition starts. Verification runs before a state can be exited, no exception for changes somebody labelled trivial. A finding doesn’t close without evidence attached. Implementation doesn’t start without an approved design review. I have every one of those rules today, written down. Nothing checks them.

This problem has been solved before, for a different flavour of non-determinism. P⁵, which started at Microsoft and now lives at AWS, is a language where you model a system as communicating state machines and hand it to a checker that explores message interleavings and failures against the correctness specifications you wrote. It’s not academic furniture: P shipped the USB 3.0 drivers in Windows 8.1 and Windows Phone, and teams across AWS use it on S3, EBS, DynamoDB, MemoryDB, Aurora, EC2 and IoT to reason about whether their designs are correct⁶. The non-determinism there comes from interleaving and node failure. Mine comes from a sampling distribution. Different source, same shape: a system whose behaviours you cannot enumerate by hand, with properties you still need to hold.

I am not asking for a model checker over agents. That isn’t a coherent thing to want: the state space is natural language, there is nothing to exhaustively explore. What I want is P’s posture without P’s rigour. An explicit state machine instead of an implied one. Specifications of what must hold, written where a program can read them, and monitors that shout when a run violates one. P calls those spec monitors and runs them twice: against the model during verification, and afterwards against production logs, through a tool called PObserve⁶. The second half is the part I could use tomorrow.

Call it a P for LLM workflows. Looser than the original, but with rules that are real.

So I’m building a bad version of it myself

Since it doesn’t exist, I’ve been inventing it, badly, for about a year. A command per phase. An agent per role. A contract file every model reads on entry. YAML state that survives task switching. Consensus review with an inclusion threshold⁷, because a single agent’s opinion isn’t worth acting on. Fix loops with iteration caps. It works well enough that I stopped considering the alternative⁸.

It is also, honestly described, a state machine implemented as prose and handed to a language model with a polite request that it comply. The rules are suggestions, the gates are conventions, and nothing anywhere in the system can say no.

I know exactly how weak that is, because I went looking. Three separate files in my configuration asserted something about review history that was flatly untrue, and had been for months⁴. Nothing caught it, because nothing in the system is capable of catching anything. A tool with a real notion of state would have failed on the first transition that depended on it.

Where this lands

The IDE industry spent three decades optimising the production of code, and did an outstanding job. Then the bottleneck moved somewhere else, and the whole category is still standing where the bottleneck used to be.

I don’t miss the IDE. I miss having somewhere to put the rules. Whoever ships that gets my next fourteen years.

Over-engineering used to be bounded by boredom

Mon, 10 Aug 2026 00:00:00 GMT

Belogorka waterfall, Sokuluk gorge. Two hours from "the office" and nothing there needs a daemon.

A feature I’d sized at roughly 200 lines came back as 8,391 added and 692 removed, 3,431 of those lines being tests.

The requirement was one sentence. A CI job needs one of the two physical devices wired to the test rig, so: enumerate what’s attached, pick one that’s online, hand it back at teardown. What shipped was a forked daemon holding a flock for the lifetime of the job, PID identity verification against /proc start times, a SIGTERM to SIGKILL termination ladder, orphan sweeps, a readiness pipe, file-descriptor relocation, stdio detach, signal reset, and the tests to keep all of that honest.

Nothing there is an agent going off the rails. Every review finding was correct, and every guard defends against something that can really happen. The design is defensible option by option, which is the problem: the failure isn’t visible anywhere except in the total. One incident, examined afterward against a census of my own configuration, so n is one.

The column that got deleted

Think about picking between two designs in 2019. One is 150 lines with a known weakness. The other is a thousand lines with a background process and doesn’t have that weakness. Part of what you weighed, without noticing you were weighing it, was three weeks of your life, a chunk of that spent on file-descriptor semantics, and a pager going off at 2am when the reaper missed an orphan.

Nobody wrote that down. No design template has a field for implementation cost, because the field was redundant: the person choosing was the person paying, and the bill arrived on its own.

We weren’t disciplined, to be clear. We shipped mountains of over-built code, usually because something was interesting or looked good on a résumé, and plenty of it is still running. But over-engineering had a ceiling, and the ceiling was how much tedium one person would absorb before going to find the cheaper option. Low ceiling. Effective. It’s now zero.

The nearest human analogue is a very smart junior who wants everything perfect, and the analogy breaks in the place that matters. Two years on the rig teaches a junior which failures actually arrive, and the perfectionism decays into judgment on its own. There’s no trajectory here. Correct a person once and it sticks; correct this and it holds until the session ends, which is why every correction I’ve made ends up as a file on disk rather than a habit.

Here’s what that costs in practice. A review finding objected to a device-health step of mine: if we fail to restore a device, assume the device is at fault. The objection was that restoring also fails on a corrupt package, a full runner disk, or a dead adb server, none of which say anything about the board, so a healthy device gets quarantined. All true. And a colleague who’s run that rig for a year answers “sure, twice in eighteen months, both times obvious in the log” and it’s over in four seconds. Frequency is what seasoning consists of. Without it, every true-but-negligible statement arrives at the weight of a real one, and enumerating them costs nothing, so you get the complete taxonomy rather than the three entries a bored human stops at.

Where it shows

The design document enumerated seven options and rejected six. Each rejection is defensible on its own terms. One is load-bearing: a simpler lockfile scheme was thrown out to avoid a stale-holder identity problem, and the winning design then spent 550 lines solving the identical problem after relocating it from a lockfile to a daemon PID. The defect class didn’t go away, it got a bigger budget.

That section runs 1,200 words across seven options and contains no number of any kind. Not lines, not files, not new-mechanism versus reuse. My template’s version is four lines long: option name, pros, cons, decision.

So the agent wasn’t choosing badly. It had nothing to compare on. Robustness was on the page in words, cost wasn’t on the page at all, and one option got rejected on a diagnostics-quality argument that appears nowhere in the requirements section, which makes it unanswerable precisely because nothing can check it.

An eighth option never made the page. The runner already runs exactly as many concurrent jobs as there are devices, because the chosen design leans on that for throttling, and it exposes a stable slot index per job.

serials = sorted(online_serials)
device = serials[int(os.environ["CI_CONCURRENT_ID"])]

Distinct devices by construction. No lock, no daemon, no PID identity, nothing to reap. It has a real weakness: unplug a device while the concurrency setting stays put and two jobs index the same serial. Kernel-enforced correctness under fleet drift, against 970 lines and a background process. That’s a one-sentence question with an obvious asker and an obvious answerer, and it was never asked, because the number that makes it obvious was never written down.

The same thing at review time

Seven review passes on the design, producing 84 findings, exiting on “accepted after pass six without re-review.” Round four alone returned 9 high-severity and 14 medium, all new rather than unresolved.

That doesn’t happen with human reviewers, and not because human reviewers are better calibrated. It doesn’t happen because round four is when somebody says enough. Fatigue is a convergence mechanism. Ugly, unprincipled, effective.

Tirelessness isn’t the whole of it either. A tireless human with perfect recall would still converge, because they’d remember what they’d already accepted and stop reopening it. My fix loop retains exactly one thing between passes: the iteration counter. The review document is overwritten each pass, findings and severity IDs and all. Three separate files in my configuration claim git history preserves prior reviews, and all three are false, because that directory is gitignored and never had a file tracked. So each pass isn’t continuing the review, it’s drawing a fresh sample against a target that got bigger since last time. I did have stall detection, and it keys on the same root cause appearing unresolved three passes running, which is exactly the shape this doesn’t have. Growth looks like progress from every angle that predicate can see.

Round four’s headline was that mutation testing killed seven guards and the tests survived. They asserted things that could not fail. The honest reading is that the primitive had grown too intricate to test truthfully and should be replaced. The loop’s terminal states are approved and changes-requested, so the only expressible reading is insufficient coverage. More tests. Of the 58 entries in my regression ledger, 50 originated as findings raised and fixed inside that loop, which means most of the permanent test surface guards defects that never existed outside the branch.

The obvious diagnosis is wrong

Nothing deletes, sure. But that’s a constant, not a variable. We didn’t delete documentation before either, or dead code, or tests guarding behaviour nobody had shipped in years. If the bloat is new, the cause has to be something that changed, and deletion reluctance didn’t change. It decides whether the result is permanent. It doesn’t decide the volume.

Putting the price tag back

One fix does most of the work: every option in a design carries two declared lines, what it costs in files and new mechanisms and test cases, and which stated requirement it fails, or “none.” Then the cheap option wins by default and the expensive one is admissible only by naming a requirement the cheap one misses. In my design document that addition measures 31 words as a single table for all seven options.

It has a hole, and the hole is the same missing signal one level up. If enumerating failure modes is free and every enumerated mode lands in the requirements section formatted like a requirement, the override is always available. No back-filling needed, no bad faith. So a requirement can only override cost if its provenance is checkable by someone other than its author: it traces to the ticket, to a dated incident, or to an external authority like a protocol or a platform limit. Anything produced during analysis and never observed gets recorded and can’t override. The axis isn’t observed versus derived, since a constraint read off a spec is never observed and is genuinely hard. It’s checkable by someone else, or not.

None of which enforces anything. The requirements and the options get written in the same pass by the same agent, document order isn’t authoring order, and I can’t prove which came first. What the cost line buys is that the trade sits somewhere a human comparing document against ticket has a chance of catching it.

What I actually miss is cheaper than any of this. I miss knowing, while reading option A, that picking it meant I’d be the one debugging fd inheritance on a Tuesday night. That did more design review than seven passes did.

Vibe Coding experiment

Wed, 25 Mar 2026 00:00:00 GMT

A bit more interesting data on vibe-coding vs structured assistant-coding.

Yesterday I ran an experiment. Had a task that will never touch production (local utility), and I’d been reading yet another piece about some engineer cranking out 10 MRs per day with AI. Good enough reason to actually try the dump-everything-at-once approach.

So I described the full scope to Claude upfront and let it run. Two things stood out. It took longer and produced more frustrating dead ends than my usual flow (detailed design first, then feature-by-feature implementation¹). And it burned roughly twice the tokens compared to the structured approach. March 24 alone hit $35 after a session involving all three models, versus $10 on a more focused day. If you want to track your own numbers, npx ccusage@latest does the job.

I’ve seen the “feed it the whole task at once” question pop up a lot lately. Based on this, my answer is: don’t. At least not yet. The model doesn’t have the context to make the right trade-offs upfront, and you end up steering it through corrections that a proper design phase would have avoided entirely. And if you think vibe-coded output is going into a proper review pipeline, that’s a separate problem².

Vibe-coding makes for great demos. In practice, it’s just paying more for worse results.

Making GenAI code review actually useful

Thu, 19 Feb 2026 00:00:00 GMT

Alamedin Gorge is one of my favorite places for a short weekend walk.

I ran a single Claude reviewer agent against an approximately 1000-line C++ change last month (yes, it’s too big for a normal workflow, but we are in the GenAI era and norms are different)—some pretty important video pipeline updates, but nothing exotic. The review came back with 14 findings. Three were real. The rest included a phantom race condition in code that runs on a single thread, two style nitpicks elevated to High severity, and a complaint about missing error handling on a function that already returns std::expected. The signal-to-noise ratio was bad enough that I almost closed the tab.

This is the dirty secret of GenAI-assisted code review. The models are good enough to spot real issues, sometimes ones a tired human would miss. But they also hallucinate problems with enough confidence to waste your time, and the false positives are not random. They cluster around the same blind spots every run because they come from the same weights, the same training distribution, the same biases baked into one model family.

I wrote about the false positive problem briefly in my earlier piece on development processes in the GenAI era¹, but I didn’t have a concrete solution at the time. Now I do, and it’s been running in my workflow for a few weeks. The short version: stop asking one agent. Ask three, and only keep what two of them agree on.

Why single-agent review fails predictably

The intuition is that running a model three times should give you roughly the same output three times. That’s wrong, and understanding why is the key to making this work.

LLM generation is not deterministic in practice (even at temperature zero, batching and floating-point ordering introduce variation). Each independent run samples a different path through the model’s latent space. One run might fixate on thread safety and miss an error-handling gap. Another might catch the error handling gap and hallucinate a performance concern. The overlap between runs is where the model is confident and correct. The divergence is where it’s guessing.

Single-agent review gives you no way to distinguish between these. Every finding arrives with the same authoritative tone. A real race condition and a phantom one look identical in the output. So you end up doing the thing GenAI was supposed to save you from: reading every finding carefully, mentally re-deriving whether it’s valid, and throwing away half of them. At that point you’re doing a code review with extra steps.

The problem compounds over time. Engineers who get burned by false positives stop trusting the review output. They start skimming or skipping it entirely. And then the real findings, the ones the model actually caught, get ignored too. Trust erosion is the failure mode, not any single bad finding.

The 2/3 inclusion rule

The fix is conceptually simple. Run three independent reviewer agents against the same change. Each agent gets the same input: the diff, the design document (if one exists), and the review checklist. They don’t see each other’s output. Three separate invocations, three separate generation paths.

Then apply one rule: a finding only makes it into the final review if at least two out of three agents flagged it independently. Everything else gets dropped.

The 2/3 inclusion threshold is the noise filter. It works because the false positives from independent runs are mostly uncorrelated, they come from different hallucination paths, so they rarely overlap. Real issues, on the other hand, tend to be visible from multiple reasoning paths because they’re grounded in actual code problems, not model artifacts.

In practice, this cuts the finding count roughly in half while keeping nearly all the real issues. The phantom race condition from my cache invalidation review? Only one agent flagged it. Gone. The actual missing bounds check on an index parameter? All three caught it.

Severity consensus

Filtering findings is only half the problem. The other half is severity. One agent calls something Critical, another says Medium, the third says High. Which one do you trust?

I use a simple rule. If two or three agents agree on severity, that’s the severity. If all three disagree (say Critical, High, and Medium), take the middle one. This isn’t sophisticated. It doesn’t need to be. The goal is to prevent a single agent’s tendency to catastrophize from dominating the output.

The resolution rules:

Agent votes	Severity used
2/2 or 3/3 agree	Agreed severity
2 agree, 1 differs	Majority severity
All 3 differ	Middle severity (Critical > High > Medium > Low)

The alternative is letting the highest severity win, which is what most people default to. That’s a mistake. It biases every review toward alarm, which feeds right back into the trust erosion problem. If every finding is Critical, nothing is.

Cross-model verification

The consensus protocol handles intra-model noise well. But there’s a second failure mode it doesn’t address: systematic blind spots shared across all three runs because they come from the same model family.

Claude (to pick the obvious example, since that’s what I use for review agents) has specific patterns it over-indexes on and specific patterns it consistently misses. Three Claude runs will share those biases. The 2/3 rule filters random noise, not systematic noise.

The obvious next step is a cross-model pass. After the Claude consensus, run the same review through a different model family (OpenAI’s Codex, in my case) as an independent verifier. Codex sees the same diff and checklist but produces its own findings without seeing the Claude output. Then cross-aggregate: findings that both families flag get marked as corroborated, Claude-only findings stay as-is, and Codex-only findings get included but labeled clearly so you know they didn’t survive the primary consensus.

The corroborated findings are almost always real. A finding that survives the 2/3 Claude consensus and independent Codex verification has been a genuine issue every time I’ve checked. That part works.

What I’m less sure about is the right protocol around Codex-only findings. Right now I include them with a label and review them manually, which is fine at low volume. But the question I keep circling back to is whether the right move is to feed Codex-only findings back to Claude for re-verification. That creates a tighter loop: Codex catches something Claude missed, Claude gets a second chance to evaluate it with the specific finding in front of it. The risk is that Claude just agrees because the finding is now in the prompt, which would defeat the purpose of independent verification.

The other option I haven’t tried yet is running multiple Codex instances with the same 2/3 consensus approach, then cross-aggregating two independent consensus outputs. That’s more compute, but it would give the cross-model step the same noise filtering the primary review gets. Right now, a single Codex run is a single opinion, and I already know why single opinions are unreliable (that’s the whole premise of this article).

I don’t have a settled answer here. The single Codex pass is good enough to catch systematic Claude blind spots, and the corroboration signal is strong. But the handling of Codex-only findings feels like it needs another iteration. If you’re implementing something similar, this is the part I’d expect to change first.

What the agents should not flag

The protocol is only as good as the instructions each agent receives. One thing I learned early: you have to explicitly tell the reviewer agents what to ignore, not just what to look for.

Without explicit exclusions, GenAI reviewers love to flag pre-existing issues that have nothing to do with the change under review. They’ll also flag style preferences as if they were correctness problems, and raise speculative bugs that depend on specific inputs without any evidence those inputs are reachable. A senior engineer would never raise these in a review. The model will, confidently, every time.

My reviewer agents get explicit instructions to skip: pre-existing issues not introduced by the current change, subjective style preferences, potential bugs without clear evidence of reachability, and nitpicks a senior engineer would dismiss. This doesn’t eliminate all noise (that’s what the 2/3 rule is for), but it reduces the volume each individual agent produces, which makes the consensus step cleaner.

For what it’s worth, I run this full protocol (three Opus reviewer agents, Codex cross-model pass, plus separate agents for research, implementation, and debugging) on a $100/month Claude Pro plan. After a full working week, my usage sits around 20 to 30 percent. The consensus approach sounds expensive when you describe it in the abstract. In practice, if you keep your changesets small and your review inputs focused, it barely dents the budget.

What this doesn’t solve

The consensus protocol is a noise filter. It makes GenAI review signal usable. It does not make GenAI review authoritative.

Formal correctness guarantees still come from static analysis, testing, and runtime instrumentation. The consensus review is a complement to those, not a replacement. It catches the category of issues that are visible in code but not caught by automated tooling: design adherence gaps, missing error handling strategies, observability blind spots, and subtle concurrency concerns that a linter won’t flag.

And it still requires a human at the end. The engineer reviews the filtered findings and makes the final call. The protocol reduces the volume of findings to something a human can actually process without fatigue, which is the whole point. GenAI review that produces 14 findings is a chore. GenAI review that produces 5 high-confidence findings, with the noise already removed, is a tool.

Development processes in the GenAI era. sysdev.me, January 14, 2026. ↩

IPO Season and the Death of Software Engineering (Again)

Thu, 12 Feb 2026 00:00:00 GMT

Our Perseids campsite at Song Kol. No cell signal, no Blind, no LinkedIn panic.

I check Blind the way some people check horoscopes: every morning, expecting drama, occasionally finding truth. The Economist I read for the opposite reason, it’s calm, structured, backed by actual data. These two don’t usually agree on much. But over the past couple of months, they’ve converged on the same narrative, and that’s when I start paying attention.

December 2025: the quiet IPO prep

Most people missed the December signal. No press conference, no earnings call. Just a Schumpeter column in The Economist¹ noting that SpaceX, OpenAI, and Anthropic are all circling public listings. Anthropic hired Wilson Sonsini, the firm that took Google and LinkedIn public. Valuation tripled in six months to $183 billion². Revenue reportedly went up ninefold in a year³.

None of this was loud. But if you’ve watched companies prepare for IPOs before, you recognize the choreography. Legal team locked in. Valuation inflating fast. Private rounds getting so large they start to look absurd. OpenAI’s last was $40 billion, bigger than most IPOs in history.

The Economist asked the right question: will more capital trump more scrutiny? Public markets are a different audience than a handful of VCs fighting for allocation. They want a story that justifies tenfold valuation growth in two years. That story showed up in January.

January 2026: the bold claim

Dario Amodei sat down with Zanny Minton Beddoes at the World Economic Forum in Davos and said what every IPO roadshow needs to hear⁴. “We might be six to twelve months away from when the model is doing most, maybe all of what SWEs do end to end.” He added that engineers within Anthropic already don’t write code anymore, they just let the model generate it and then edit the output.

That’s a hell of a statement from the CEO of a company actively preparing for one of the largest IPOs in American history.

I’ve watched this exact script play out before. At Motional, the AV pitch was always the same: full autonomy is right around the corner. The only difference between then and now is the buzzword. Back then the investor story was “human drivers are obsolete.” Today it’s “human engineers are obsolete.” Both claims share one feature: they’re calibrated for capital markets, not engineering reality. At Motional, we believed the timeline enough to staff for it. Then we cut 40 percent of engineering when reality didn’t cooperate.

Saying “AI is a useful tool that makes engineers 30 percent more productive” doesn’t get you to a $350 billion valuation. Saying “software engineering is about to be obsolete” does. It implies total market capture, minimal future labor costs, exponential returns. Whether it’s true is almost beside the point for the roadshow.

February 2026: the panic

And then, right on schedule, LinkedIn lit up. “We’re all dead.” “Software engineering is over.” “Time to learn plumbing.” The usual cycle. Every major tech publication ran a variation of the story. Zoho’s founder Sridhar Vembu posted on X advising developers to consider alternative livelihoods. I’ve seen this exact reaction pattern play out at least three times in my career (yeah, it looks like I’m old :-D), each time with a different buzzword. Cloud was going to kill ops. Low-code was going to kill developers. Autonomous vehicles were going to wipe out the entire transportation workforce within five years. The panic always sounds the same, and it always serves someone’s fundraising timeline.

While LinkedIn was running eulogies, Blind had something more interesting. A post about OpenAI’s hiring freeze⁵: rumors of drastically reduced headcount, people stuck in the reference stage, roles going quiet. Not the dramatic AI-replaces-everyone narrative. Just the mundane, familiar silence of a company that stopped calling candidates back.

Sam Altman had said at a developer town hall in late January that OpenAI plans to “dramatically slow down” hiring⁶. AI lets them do more with fewer people, he explained. Careful phrasing: not a freeze, not a layoff, just “hiring more slowly.” Blind didn’t buy the distinction.

So one CEO says engineering is becoming obsolete. The other says he needs fewer engineers. And both companies are hemorrhaging money. OpenAI is projected to lose around $12 billion this year. The Economist noted they expect to burn through another $115 billion before profitability around 2030¹. Anthropic might break even a couple of years earlier. Might.

If your AI is truly replacing engineers, your costs should be going down. They aren’t.

What the data says

I covered this in detail in my earlier piece on what actually drove the tech layoffs⁷, and the short version hasn’t changed: even after all the cuts, Meta and Google employ 50 to 65 percent more people than they did in 2019. Spending per engineer went up, not down. The pandemic hiring bubble created the correction. AI was the convenient scapegoat.

Ask yourself: if you were preparing to take a company public at a $350 billion valuation, would you keep hiring at pandemic-era rates? Of course not. You’d slow down, tighten the numbers, show Wall Street that you run a disciplined operation. That’s what’s happening. The AI angle is a press release. The hiring freeze is an accounting decision.

Where this actually lands

I use AI tools every day. Genuinely useful. Claude Code alone is reportedly a $1 billion a year product for Anthropic³, and I understand why.

That said. I once spent six hours debugging a video pipeline that produced garbage output on a single device out of twelve. The root cause: a thermal throttle changed the encoder bitrate, and a downstream component written by a different team assumed constant bitrate and silently dropped frames. One-line fix. Six hours of staring at thermal logs and corrupt H.264 frames to get there. Try putting that in a prompt when you don’t know the problem yet.

“AI writes boilerplate faster” is real. “Engineering is obsolete” is an IPO story. The AV industry promised my car would drive itself by now. Still holding the steering wheel.

Development processes in the GenAI era

Wed, 14 Jan 2026 00:00:00 GMT

The current debate around GenAI and C++ is a good illustration of the real problem. Many engineers report that models are worse than juniors. Others report dramatic speedups on the same language and problem space. Both observations are correct.

The difference is not the model. It is the absence or presence of the state.

Most GenAI usage today is stateless. A model is dropped into an editor with a partial view of the codebase, no durable memory, no record of prior decisions, no history of failed attempts, and no awareness of long-running context. In that mode, the model behaves exactly like an amnesic junior engineer. It repeats mistakes, ignores constraints, and proposes changes without understanding downstream consequences.

When engineers conclude that “AI is not there yet for C++”, they are often reacting to this stateless setup.

At the same time, GenAI does not elevate engineering skill. It does not turn a junior into a senior. What it does is amplify the level at which an engineer already operates. A senior engineer using GenAI effectively becomes a faster senior, and a junior becomes a faster junior. Judgment is not transferred, and the gap does not close automatically.

These two facts are tightly coupled. In stateless, unstructured usage, GenAI amplifies noise. In a stateful, constrained workflow with explicit ownership and review, it amplifies competence.

This is why reported productivity gains vary so widely. Claims of 200–300% speedup are achievable, but only locally and only within the bounds of the user’s existing competence. Drafting, exploration, task decomposition, and mechanical transformation accelerate sharply. End-to-end throughput increases are lower because planning, integration, validation, and responsibility remain human-bound.

The question, then, is not whether GenAI is “good enough”. The question is what kind of system you embed it into.

Note

Everything I’ll explain below is only applicable to the Stateful GenAI setup.

Stateless vs Stateful

What changed and what did not

One fundamental shift is that everyone becomes a TPM, at least a little. GenAI moves the bottleneck from typing to coordination. Breaking work into coherent tasks, sequencing dependencies, tracking risk, and explaining intent clearly now dominate more of the day. Engineers with prior TPM experience adapt faster. Others find it uncomfortable at first, but the skill is learnable.

There is a failure mode here. When coordination skills are weak, work fragments into tickets that lose architectural coherence, or decisions get made implicitly inside prompts instead of explicitly in reviews. That is why process matters more as velocity increases, not less.

Another thing that did not change is the nature of GenAI itself. These systems are powerful pattern matchers, not reasoning engines. They accelerate drafting and exploration, but they do not replace engineering judgment. Humans remain the source of truth for architecture, correctness, performance, concurrency guarantees, undefined behavior, and production readiness. There are entire classes of problems where trusting model output is actively dangerous, especially in low-level or highly concurrent systems.

Process matters more under higher speed

GenAI allows more change in less time. Without discipline, that extra change turns into larger merge requests, unclear ownership, brittle integration, and review fatigue. GenAI amplifies whatever process you already have, good or bad.

A practical end-to-end flow that works well looks roughly like this: planning, architectural research and documentation, implementation, integration testing, and documentation updates. Every stage includes a mandatory human review. GenAI can assist, but it cannot be the approver.

Ownership is explicit. The engineer who initiated the change remains accountable throughout. GenAI is treated like a junior contributor that can draft and suggest, not like an authority. Peer review at the end is mandatory, but the author is always the first human gatekeeper.

Review gates are not overhead. They are the control mechanism that prevents increased speed from degrading correctness.

Code review and validation

Manual review quality degrades as the change size increases. Unfortunately, GenAI assistance does not remove this limitation; changes must remain small and coherent to remain reviewable under higher throughput. GenAI-based review tools generate useful signals but also a high rate of false positives in complex systems code. Review output must be treated as hypotheses. Formal guarantees still come from static analysis, testing, and runtime instrumentation.

Cross-model review reduces correlated failure modes. Different models tend to fail differently and can unblock each other. Human reviewers arbitrate conflicts and retain decision authority.

Failure mode without control

Without structure, GenAI introduces a reinforcing failure loop. Increased change volume leads to unfamiliar code and larger diffs. Review fatigue follows, then shallow validation, latent defects, incidents, and loss of trust. Velocity declines as rework dominates.

Process discipline exists to break this loop.

Traceability, task switching, and state

Higher iteration speed increases task switching. This is not a focus problem. It is a structural consequence of discovering constraints late. The same pattern exists without GenAI. When a hard dependency is discovered during implementation, scope is reduced, new tasks are created, work is paused, and attention shifts elsewhere. The difference is not the mechanism, but the frequency. GenAI accelerates the path to constraint discovery, which makes these transitions happen earlier and more often.

A common pattern is partial implementation followed by discovery of a hard dependency. Work pauses while the dependency is resolved. The primary cost is not the interruption, but the effort required to resume.

A persisting feature or epic state in a small, human-readable file externalizes working memory. The state captures what was discovered, why work stopped, which assumptions changed, and what the next action is. This reduces reload cost and enables both humans and models to resume work deterministically.

This mechanism is particularly effective across weekends and public holidays, where implicit context is most likely to be lost.

Multi-agent workflows

Single-agent GenAI workflows collapse architectural, implementation, and operational concerns into a single output. This does not scale.

From an LLM perspective, all decisions in a single-agent setup are derived from the same prompt, context window, and latent state. Assumptions are shared across concerns and are not independently challenged. As scope grows, the model resolves competing objectives implicitly within one generation path, which causes errors to become correlated. Design mistakes propagate directly into code and operational logic, and incorrect assumptions are rarely revisited once embedded in the output.

To scale safely, independent reasoning paths are required. At a minimum, separate agents should reason independently about structure, correctness, and operability. Disagreement between agents is expected and desirable. Humans arbitrate conflicts and retain ownership.

YAML-based epic and feature state

Persisting state requires a concrete, durable representation. In practice, YAML works well for this role.

Epic- and feature-level YAML files act as shared execution state for humans and GenAI agents. They capture intent, scope, constraints, and current understanding in a form that is diffable, reviewable, and version-controlled. Unlike conversational context, this state survives task switching, interruptions, and tool changes.

These files intentionally describe what needs to be achieved rather than how it should be implemented. Outcome-focused descriptions and acceptance criteria prevent premature design decisions from being embedded into state and allow independent reasoning by both humans and agents.

Validation at the state boundary keeps the structure intact as the iteration speed increases. Required sections, label constraints, and basic consistency checks ensure that the state remains predictable and reviewable over time.

YAML state does not replace issue trackers. Trackers remain the system of record for execution and discussion. The YAML files represent the current shared understanding of the work and serve as a stable input contract for GenAI usage.

Example

A concrete example of this approach, including templates and validation rules, is covered in a separate article.

Cross-model consistency and shared state

Using multiple models introduces inconsistency unless entry conditions are controlled. A single repository-level agent contract, such as AGENT.md or CLAUDE.md, provides a stable starting point for all models. In practice, this file is shared across tools, often via symlinks, to ensure that different models operate under the same constraints.

The agent contract defines global invariants: coding standards, validation requirements, review expectations, and links to authoritative documentation. It does not duplicate domain knowledge. Its role is to fix assumptions at the boundary and make them explicit.

On its own, this is not sufficient. The agent contract establishes how models should behave, but not what they are working on. That role is filled by explicit task and execution state, represented in YAML at the epic and feature level. Together, the agent contract and YAML state form a minimal, durable memory for GenAI-assisted workflows.

This combination addresses the primary failure mode of GenAI systems. Statelessness. Models no longer infer intent from partial diffs or conversational history. They enter with fixed rules and consume explicit state. Memory becomes an artifact, not an assumption.

Costs and constraints

GenAI-assisted development introduces costs that follow directly from its operating model. Effective usage typically involves multiple models, trading latency and cost for correctness and review quality. Inference cost is usually small relative to senior engineer time but becomes a predictable infrastructure expense at scale. The relevant metric is cost per accepted change.

The dominant non-monetary cost is cognitive. GenAI replaces typing with validation. Engineers spend more time reading diffs, checking assumptions, resolving contradictions, and restoring context. This resembles continuous code review and leads to faster fatigue if unmanaged.

There is also a process cost. Large gains require explicit state, strict ownership, small changesets, role separation, and shared contracts. These controls add upfront overhead but prevent regression in correctness and operability.

GenAI increases the rate at which change is produced. It does not reduce system complexity, validation requirements, or engineering responsibility.

Sustained gains require explicit state, clear ownership, disciplined review, role separation, and controlled entry points. Without these, GenAI accelerates failure as efficiently as it accelerates delivery.

Used as part of a system, GenAI is a force multiplier. Used in isolation, it is noise.

Warning

GenAI rapidly evolves. Meaning, this info is valid as of Jan 2026 and may no longer be valid.

Today I learned... git shallow

Wed, 31 Dec 2025 00:00:00 GMT

Sometimes you need to understand why something exists, and instead, you’re staring at a mystery. It feels like magic for a moment. But there is no magic in IT. There is always a reason, and usually it’s painfully concrete.

Today I learned that if git blame suddenly claims I wrote the entire million-line project, it might be lying 🙂

I ran into a situation where my local git blame attributed every line to a single recent commit, while GitLab showed the correct historical authors. At first glance, it looked like history had been rewritten, which is odd and incorrect.

The first real clue was in the blame output itself, the caret prefix before the commit hash:

^4e8fb9eb2 (Alex Stavonin …) #include <cassert>

Google told me that ^ means “boundary commit”, meaning Git had reached the end of the history it could see. A quick check confirmed it:

ls .git/shallow
git log --oneline | wc -l

Only a couple of commits were visible. The repo was a shallow clone.

In a shallow clone, git blame can’t walk past the depth limit, so it assigns all lines to the oldest commit it knows about. The real authors live before that boundary. GitLab has the full history, so its blame view stays correct.

The fix was simple:

git fetch --unshallow

That pulls the missing history, removes .git/shallow, and turns the repo into a normal clone. After that, git blame immediately showed the correct commits and authors, with no boundary markers.

And Happy New Year!

Dealing with ThreadSanitizer Fails on Startup

Fri, 05 Dec 2025 00:00:00 GMT

Usually, you need just a few lines to initialize TSan in your project: you compile with the sanitizer flags, run the tests, and get a clear report of which threads touched which memory locations. On a modern Linux system, that simple expectation can fail in a very non-obvious way.

FATAL: ThreadSanitizer: unexpected memory mapping 0x...

In my case, I attached TSan to a not-so-young C++ codebase and immediately encountered a fatal runtime error from the sanitizer, long before any of the project’s code executed. No race report, no helpful stack trace, just a hard abort complaining about an “unexpected memory mapping.”

If you can upgrade your toolchain to LLVM 18.1 or newer, this problem effectively disappears, because newer TSan builds know how to recover from the incompatible memory layout. Suppose you are pinned to an older LLVM (by CI images, production constraints, or corporate distro policy). In that case, you are in the same situation I was: you have to understand what the sanitizer is trying to do with the address space, and work around the failure mode yourself.

The process terminates in its initialization phase, long before your program’s logic executes. The failure stems from changes to Linux’s memory-layout randomization and from assumptions baked into TSan’s address-space model. At the end, it was not as complicated¹ an issue as I initially thought, and it was interesting to dig a bit.

A Minimal Possible Example

To keep the demonstration as small as possible, let’s make a deliberately naive example: multiple threads increment a shared global integer with no synchronization of any kind. We ignore atomics, mutexes, and every correctness rule you would normally apply.

int shared_counter = 0;

void increment_counter(int thread_id, int iterations) {
    for (int i = 0; i < iterations; ++i) {
        int old_value = shared_counter;
        shared_counter = old_value + 1;
    }
}

Any sanitizer-enabled build should report concurrent accesses to shared_counter, show at least two stacks, and explain that the update is non-atomic. Compiling with Clang and ThreadSanitizer is straightforward.

Note

And one note upfront: if you build this locally with an LLVM toolchain newer than 18.1, you may not see the failure at all. Modern TSan versions recover automatically by re-executing the process with a compatible memory layout, so the crash only appears when using older LLVM releases. For experiments I made a project with LLCV-14 installed that uses Docker to provide builds with reproducible issues.

clang++ -std=c++17 -g -Wall \
    -fsanitize=thread -fno-omit-frame-pointer \
    -o simple_test simple_test.cpp

On older Linux distributions, you get a standard TSan report. In newer environments, TSan may crash before producing any output.

Will non-PIE Binaries Help?

Typically, GCC/LLVM generates Position Independent Executables (PIE) by default. From the loader’s perspective, a PIE binary behaves like a shared object: it can be mapped at any randomized base address. A non-PIE binary (ET_EXEC) has a static image layout, and the loader only applies relocations around fixed segments. TSan’s shadow memory model behaves differently depending on which form you use.

First of all, we need to build two variants:

# PIE build
clang++ -std=c++17 -g -Wall \
    -fsanitize=thread -o simple_test_pie simple_test.cpp

# non-PIE build
clang++ -std=c++17 -g -Wall \
    -fsanitize=thread -fno-PIE -no-pie \
    -o simple_test_nopie simple_test.cpp

On systems with high memory randomization, the PIE variant usually fails first because its relocation window is more likely to overlap with TSan’s expected shadow ranges. The non-PIE build can also fail, but slightly less frequently. So, this is also not a solution, even though I initially thought it was the cause.

What Actually Goes Wrong: TSan’s Shadow Memory Layout

ThreadSanitizer maps every real address to a shadow address. It doesn’t do this on demand for each page. It wants large, fixed chunks of the address space reserved up front for its metadata: access histories, timestamps, and everything else it needs to track races. The layout for x86_64 was designed at a time when Linux used much lower ASLR ranges, and the sanitizer relied on that space being predictable. You can check how much randomness your kernel uses with:

sud o cat /proc/sys/vm/mmap_rnd_bits

Typical value on modern Linux is 32. Older kernels stayed around 28 to 30. Bumping it to 32 simply widens the area where the kernel can drop anonymous mappings, shared libraries, JIT code, or loader segments. With that extra room, the chance that one of those mappings lands in the region TSan wants for its shadow space goes up noticeably.

When that happens, the sequence is boringly predictable:

The loader picks a random high address for a library or for the binary itself.
TSan starts up and asks for its shadow memory slice.
That slice is already occupied.
Older TSan doesn’t try to recover. It just aborts immediately.

Disabling ASLR for the Test Process

It’s possible to disable ASLR for the whole system, but it does not look like a good idea. It will weaken the security posture of the host, which is usually unacceptable. Fortunately, Linux allows individual processes to run with ASLR off via the ADDR_NO_RANDOMIZE personality flag. The simplest interface is setarch:

setarch $(uname -m) -R ./simple_test_nopie

The command launches the child process with randomized layout disabled while leaving the system configuration intact. With deterministic placement restored, TSan can reserve its shadow memory and run normally. This was sufficient for me for local development and CI pipelines that run dynamic analysis tools.

But What is about Docker?

Here comes the tricky part. Containers don’t get their own ASLR policy; they just reuse whatever the host is doing. And most CI images run with Docker’s default seccomp profile, which blocks a bunch of syscalls, including personality(). Since setarch depends on that syscall, it simply can’t flip the ASLR flag inside the container, so the call fails with an error.

setarch: failed to set personality to x86_64: Operation not permitted

Relaxing the container’s seccomp configuration solves the problem:

docker run --security-opt seccomp=unconfined ...

Once the sandbox permits personality(), you can run your sanitizer-instrumented binary via setarch -R inside the container with the same behavior you observe on a host.

If you want a more selective policy, you can create a custom seccomp file that mirrors Docker’s defaults but allows personality. This avoids granting full syscall access to the container while still enabling TSan.

Why Upgrading the Toolchain Solves the Problem

LLVM 18.1 changes how TSan handles bad luck with the address layout. When it sees that the current mappings collide with what it needs, it doesn’t give up anymore. Instead, it quietly re-executes the process with ASLR turned off just for that run. The sanitizer gets a layout it can work with, and from the outside the program simply starts normally.

With LLVM 18.1 or newer, you don’t need setarch, you don’t need to relax seccomp in Docker, and you don’t need to switch to non-PIE builds. The newer runtime handles the mismatch on its own and avoids the crashes that show up on older toolchains.

Original issue discussion on GitHub: Thread Sanitizer FATAL error on kernel version 6.6.6-x. ↩

Cross-Compiling Rust for Raspberry Pi

Thu, 27 Nov 2025 00:00:00 GMT

I just started a new embedded pet project on the Raspberry Pi, and I expect it’ll be a pretty big one, so I’ve been thinking about the technology from the beginning. The overall goal is to create a glass-to-glass video pipeline example. Let’s see how it’s going. For now, I’m using a USB V4L2 camera while waiting for the native Pi modules to arrive, but it’s enough to sketch the capture loop and start testing the build pipeline. The application itself is minimal—open /dev/video0, request YUYV at 1280x720, set up MMAP buffers, and iterate over frames—but the real challenge occurs when v4l triggers bindgen, and the build must cross-compile cleanly for aarch64.

The language choice immediately becomes part of the equation right away. Go is my favorite and, usually, is not considered as an option by many embedded developers. But it’s a good choice for small embedded utilities because its cross-compilation story is nearly effortless. Need an ARM binary? One command and you have it!

GOOS=linux GOARCH=arm64 go build

But Go’s garbage collector isn’t something I want to use in a timing-sensitive video-processing capture loop. In theory, C++ seems like an alternative, but cross-compiling C++ code for any embedded platform is often a nightmare with toolchains, sysroots, and inconsistent environments. Frankly speaking, I’m not ready to do it just for fun; it’s OK only if I’m getting paid for it. Rust falls into a more practical middle ground. Its build system is not as elegant as Go’s, but it sidesteps the C++ toolchain maze and provides predictable behavior without GC pauses. It looks like a win-win for the Pi video-capture codebase.

The plan is straightforward: develop on fast x86 hardware, reliably ship aarch64 binaries, and avoid relying on the Pi itself for builds.

Info

If you’re not interested in the reasoning behind the project but just want a reference, I’ve created a small repository for you. It includes the capture loop, cross-compilation setup, CI pipeline, and all supporting configurations, presented exactly as they are today.

It’s the straightforward, no-commentary version of this article.

Test App: V4L Camera Capture

To start, I needed something as simple as possible while still being close to real life, so I chose a minimal Rust application that reads frames from a USB V4L2 camera. The goal wasn’t to build a complete pipeline from the beginning. I first need to have minimal, but stable CI and cross-compilation in place, and this small capture loop is good enough to identify any toolchain problems in the early stages.

flowchart LR

    subgraph HW[Hardware]
        CAM[USB Camera]
        DEV[//dev\/video0//]
    end

    subgraph SW[Software Stack]
        API[V4L2 Kernel API]
        CRT[v4l crate]
        APP[Test App]
    end

    CAM --> DEV --> API --> CRT --> APP

let dev = v4l::Device::new(0)?;

let mut fmt = dev.format()?;
fmt.width = 1280;
fmt.height = 720;
fmt.fourcc = v4l::FourCC::new(b"YUYV");

let _ = dev.set_format(&fmt)?;
let mut stream =
    v4l::io::mmap::Stream::with_buffers(&dev, v4l::buffer::Type::VideoCapture, 4)?;
while let Ok((buf, meta)) = stream.next() {
    println!("size={} seq={} ts={}", buf.len(), meta.sequence, meta.timestamp);
}

And I was right - you can’t just take it and cross-compile it without any problems! Even this tiny example should have some issues during cross-compilation: the v4l crate, bindgen, and the need for the correct libclang inside the build environment.

Why Not Just gcc-aarch64-linux-gnu?

You can easily wire the toolchain manually. It usually starts like this:

sudo apt install gcc-aarch64-linux-gnu
rustup target add aarch64-unknown-linux-gnu

Then you point Cargo at the cross-linker:

# .cargo/config.toml

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

At this point, you can:


cargo build --target aarch64-unknown-linux-gnu

It is working at the moment I’m writing this text. I did test it. But it may also cause issues in the future. The linker might be fine, but bindgen now requires a compatible libclang, along with the correct headers and sysroot. If any of those are missing or mismatched, the build collapses into the usual cascade of libclang errors — exactly the experience that pushed me toward a dedicated cross-image instead.

By contrast, pushing everything into a dedicated cross-image turns the toolchain into a versioned artifact. The linker, sysroot, libclang, and all the supporting packages live in a single Docker image that I can rebuild, tag, and publish. The Rust project only needs a Cross.toml pointing at ghcr.io/astavonin/cross-aarch64:llvm12 (yep, not a standard one from ghcr.io/cross-rs/, I’ll explain it later), and both local builds and CI reuse precisely the same environment. That is the main reason I didn’t stop at gcc-aarch64-linux-gnu and went straight to a containerized approach.

Building a Custom cross Image

The natural question is, why not just use the stock ghcr.io/cross-rs/aarch64-unknown-linux-gnu image? It works for many projects, and for simple crates, it’s perfectly fine. The problem shows up the moment the build pulls in v4l, because that drags in bindgen, and bindgen depends on libclang. The default cross image ships an older libclang that doesn’t play well with some of the functions bindgen expects. That’s how you end up with the familiar error:

A libclang function was called that is not supported
called function = clang_getTranslationUnitTargetInfo

I started from the official cross-rs base:

FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:latest

This image is based on Ubuntu 16.04 Xenial, which becomes a critical part in components selection. Xenial is old, and the available LLVM packages for it are limited. If you look at: https://apt.llvm.org/xenial/dists/, the highest LLVM/Clang version published for Xenial is 12. That’s the ceiling. Anything newer doesn’t exist for this base distribution; we can use PPA if needed, but 12 is enough for my case.

So the choice of Clang wasn’t aesthetic — the platform dictated it. LLVM 12 is the newest supported toolchain you can install inside Xenial without rebuilding LLVM from source, and it’s new enough to keep bindgen happy for v4l.

The Dockerfile installs the LLVM 12 family and exposes it to bindgen:

ENV LIBCLANG_PATH=/usr/lib/llvm-12/lib
ENV CLANG_PATH=/usr/bin/clang-12
ENV LLVM_CONFIG_PATH=/usr/bin/llvm-config-12

Configuring Rust Projects with `Cross.toml`

Once the custom image is built, wiring it into a Rust project is trivial. This is the part I wanted to keep as boring as possible. All the configuration lives in a small Cross.toml file at the root of the repo. In that file, I simply tell Cross which Docker image to use for aarch64 builds, so both local builds and CI run against the same toolchain.

A minimal configuration looks like this:

[target.aarch64-unknown-linux-gnu]
image = "ghcr.io/astavonin/cross-aarch64:llvm12"

[build.env]
passthrough = ["RUST_BACKTRACE"]

That’s all you need. Now the actual build becomes:

cross build --release --target aarch64-unknown-linux-gnu

And under the hood, cross will:

Pull or reuse your custom image with the correct libclang.
Mount the Rust project inside the container.
Invoke cargo build using the aarch64 toolchain baked into the image.

The important part here is that the toolchain is no longer spread across multiple machines or CI runners. Everything that matters — linker, sysroot, libclang, headers — travels together inside a versioned container.

CI Strategy: Fast x86 Builds + On-Demand ARM Artifacts

Since this is a personal project and I am using the free GitHub plan, I need to minimize the time spent by the runners. Building natively for x86 on GitHub Actions provides quick feedback from unit tests and potentially integration tests. This approach avoids the costs associated with container startup and does not involve the aarch64 toolchain at all. This setup is sufficient for feature branches and routine development.

gitGraph LR:
    commit id: "base"

    branch feature
    checkout feature
    commit id: "FB1" tag: "Commit"
    commit id: "FB2" tag: "x86 Build"
    commit id: "FB3" tag: "Tests"
    commit id: "FB4" tag: "CI Result"

    checkout main
    commit id: "MB1" tag: "Commit"
    commit id: "MB2" tag: "x86 Build"
    commit id: "MB3" tag: "aarch64 Cross Build"
    commit id: "MB4" tag: "Upload ARM Artifact"
    commit id: "MB5" tag: "Tests"
    commit id: "MB6" tag: "CI Result"

Cross-compiling for ARM comes with overhead: container startup, toolchain load, and bindgen doing its thing. It’s fine when you need an artifact, but wasteful for every feature branch. To avoid chewing through CI time and monthly limits, I only run aarch64 builds on main. Sure, will never do it for the production environment, but it looks like a perfect idea for this case.

This split keeps the feedback loop tight while still producing reliable ARM binaries when I actually need them. It also avoids depending on native ARM runners, which are either slower or require extra infrastructure. Everything runs on GitHub’s standard x86 machines, and the custom cross-image ensures the ARM build behaves exactly the same every time.

With the toolchain in place and the CI behaving predictably, it’s time to move on to the parts of the project that actually matter. The tiny capture app still lacks many functionalities. Next comes testing with the native Pi camera modules, measuring queue depth, watching how the system behaves under thermal load, and shaping the first real version of the video pipeline.

What Actually Drove the Tech Layoffs

Thu, 20 Nov 2025 00:00:00 GMT

A 10-minute ride and you have such a view from Chon Aryk hills.

I truly enjoy reading Blind and Levels. There is so much internal drama, messy details, and unexpected insights that you almost do not need reality shows anymore. And if you ever feel bored, you can always drop a mildly toxic comment into a thread and watch the whole thing ignite. It fits the overall style of Blind a little too well, but that is part of the fun. And considering that mix of casual toxicity and surprisingly rational takes you see there, you would expect people to look at layoffs with a bit more perspective. But when the topic comes up, the conversation usually drifts to the same explanation. People blame AI. People say their jobs vanished because a model wrote some code. And while I understand the frustration, the logic never sits right with me. Nobody complained during the hiring boom of 2020 and 2021, when companies doubled their headcount like it was nothing. That part gets forgotten. Now that the correction is here, many want a simple villain. AI fits the story, but it does not fit the data.

I have seen this pattern before. Back at Motional we were deep in the autonomous vehicle race. That industry grew on promises we could not keep, and considering what I know about the AV domain today, those promises were never possible in the first place. For years the message to investors was that full self driving was right around the corner. We hired fast and built big. Then reality caught up. Cruise paused its operations. Argo AI shut down. Uber ATG was sold. Zoox cut teams after delays. Pony.ai struggled with regulations. And inside Motional we cut about 40 percent of engineering in a single move, after several smaller rounds. It was painful, but it made sense once you looked past the hype. We hired for a future that had not arrived.

Big tech did something similar. When the pandemic hit and the world moved online, demand exploded. Companies assumed that surge was permanent. They hired for a future where remote work and digital services would keep climbing forever. Then the world shifted back to normal, the economy cooled, and interest rates went up. Suddenly those oversized teams were hard to justify. The layoffs that followed were not about engineers losing relevance. They were about companies correcting unrealistic growth curves.

Headcount is still way above pre pandemic levels

Even after layoffs, Meta, Google, and Netflix remain much larger than they were before COVID. The numbers make this very clear.

Company	Employees 2019	Employees 2024	Growth
Meta	44,942	74,067	+65 percent
Google (Alphabet)	118,899	183,323	+54 percent

If AI replaced engineers at scale, we would see headcounts dropping below 2019 levels. That did not happen. Tech firms grew fast, then trimmed the excess.

Companies spend more money per engineer than before

Now look at what each company spends on its engineering and research efforts. To compare across years, I used a simple method. Assume that about thirty percent of the workforce is engineering and use R&D or a similar cost as a proxy for engineering investment. It is not perfect, but it shows direction.

Company	Year	Estimated Engineers	Investment (R&D or FCF)	Investment per Engineer
Meta	2019	~13,483	~$10.3B	~$760k
Meta	2024	~22,220	~$43.9B	~$1.98M
Google	2019	~35,670	~$30.97B	~$870k
Google	2024	~54,997	~$72.76B	~$1.32M

So companies are spending more money per engineer than before. Not less. That does not sound like replacement. It sounds like deeper investment. Fewer generalists. More specialized people. More infrastructure. More complex systems. More AI operations. More cost per seat. This is not what a company does when it is trying to remove engineers from the equation. This is what a company does when it is trying to level up.

Why the engineer layoffs feel confusing

The AV industry taught me that hype can inflate teams far beyond what reality can support. When the correction comes, people feel blindsided because they only remember the hiring wave, not the assumptions that created it. The same thing is happening now. Engineers see cuts, but they do not see the earlier hiring spike that caused the imbalance. They see AI headlines, but they do not see the rising investment per engineer. They feel the pain, but they do not see the context.

If you work in tech today, your job is not evaporating. The shape of the work is just shifting. The companies are cutting the soft edges, the duplicated roles, the nice-to-have teams. They are keeping the people who understand systems, scale, and actual engineering. That is not a philosophical trend. It is just how budgets behave when the hype dies down.

And yes, layoffs are rough. I lived through enough of them to know how it feels from the inside. But pointing at AI as the main cause is an easy way to avoid looking at the real picture. The hiring boom was unrealistic from day one, and the correction was coming no matter what models we built. The only surprise is that it took this long.

Sources

Identifying Video Streams Using RTP Header Extensions

Sun, 09 Nov 2025 00:00:00 GMT

Birch Grove near Bishkek. The grove is incredibly popular in the autumn.

When a system manages dozens of cameras or edge devices, packets alone don’t tell you much. An IP and port might change, SSRCs can roll over, and NATs tend to shuffle everything just enough to break simple assumptions. Yet every media packet still needs a clear identity — not for transport, but for logic.

There are many ways to attach that identity: control channels, per-session negotiation, external registries. But the most simple one is already part of RTP itself — the header extension defined by RFC 8285.

How it works

RTP was designed to be extensible. After the fixed header and payload, packets can carry short metadata blocks called header extensions. Each extension has a small numeric ID and a URI describing its purpose.

That makes it an ideal place for lightweight metadata such as a unique stream identifier. The ID repeats on every packet and travels through NATs, relays, and tunnels with zero signaling overhead. It’s simple, deterministic, and standardized.

Quite often you need t o create an additional plugin for GStreamr, but, furtunatly, not in this case. GStreamer already exposes the right API hooks through gstrtpbuffer.h and gstrtphdrext.h. One pad probe is enough to inject an identifier before the payload leaves the payloader.

#include <gst/gst.h>
#include <gst/rtp/gstrtpbuffer.h>
#include <gst/rtp/gstrtphdrext.h>

static GstPadProbeReturn add_ext_probe(GstPad *pad, GstPadProbeInfo *info, gpointer user_data)
{
    GstBuffer *buf = GST_PAD_PROBE_INFO_BUFFER(info);
    if (!buf)
        return GST_PAD_PROBE_PASS;

    buf = gst_buffer_make_writable(buf);

    GstRTPBuffer rtp = GST_RTP_BUFFER_INIT;
    if (gst_rtp_buffer_map(buf, GST_MAP_WRITE, &rtp))
    {
        guint8 data[4] = {0x12, 0x34, 0x56, 0x78}; // Stream ID
        gst_rtp_buffer_add_extension_onebyte_header(&rtp, 1, data, sizeof(data));
        gst_rtp_buffer_unmap(&rtp);
    }

    GST_PAD_PROBE_INFO_DATA(info) = buf;
    return GST_PAD_PROBE_PASS;
}

The sender can advertise the mapping:

extmap-1=urn:example:stream-id

and the receiver can decode it using the same API. No external signaling, no synchronization delay. Each packet holds data to tell who it belongs to.

Why it matters

Media servers often depend on connection state to classify incoming data. When a camera reconnects, the backend has to guess whether this new socket belongs to the same stream or not. That guess introduces delay, synchronization issues, and sometimes packet loss. With an identity embedded inside RTP itself, the backend doesn’t need to guess. The stream is self-describing. Relays and recorders can demultiplex by ID, loggers can tag frames consistently, and analytics services can correlate metadata without waiting for control messages.

Performance and compliance

The extension adds roughly five bytes per packet — one-byte header, one-byte ID, and the payload itself. For H.264 or H.265 streams, that’s less than 0.1 % bandwidth overhead.

Because it’s part of RFC 8285, it’s backward-compatible: receivers that don’t recognize the extension simply skip it. No negotiation cost, no version mismatch.

Verification

You can use any tool you like to capture traffick and check for RTP Header Extension in captured traffic. It will contain:

RTP Header Extension
  One-Byte Header
  ID: 1
  Data: 12 34 56 78

The complete example

You can find example here. It was tested using Ubuntu 24.04 and GStreamer ≥1.18.

Multilingual benchmarking project => Bazel for advanced engineering

Fri, 01 Aug 2025 00:00:00 GMT

In her one year, Molly saw many more exciting places than I did until I was about 28. She does pretty well :-D

When working on performance experiments across C++ and Go, you obviously need a multilingual project structure. There were two paths forward: create separate build systems under a shared repository, or consolidate everything under a single, coherent framework. Bazel made that decision easy.

Using Bazel to unify builds isn’t just convenient—it should be the default choice for any serious engineering effort that involves multiple languages. It eliminates the friction of managing isolated tools, brings deterministic builds, and handles dependencies, benchmarking, and cross-language coordination with minimal ceremony.

Here’s why Bazel makes sense for performance-critical, multilingual projects like this one—no fragile tooling, no redundant setups, just clean integration that scales.

Info

The project I’m talking about is available here.

Structuring a Multilingual Bazel Project

I do not need any complicated structures here, as this repo is only about performance testing on multiple languages. For now, it’s only Go and C++, but because of Bazel, it would be pretty easy to add additional language support.

perf-tests
├── daily-temps
│   ├── cpp
│   │   ├── BUILD.bazel
│   │   ├── daily_temperatures.cpp
│   │   ├── daily_temperatures.h
│   │   └── daily_temperatures_benchmark.cpp
│   └── go
│       ├── BUILD.bazel
│       ├── daily_temperatures.go
│       └── daily_temperatures_test.go
├── MODULE.bazel
└── BUILD.bazel

Structure is also standard for Bazel and covers all my needs. Each test has a separate language-specific folder. Considering the goal of comparing implementation on different languages, this is precisely what I need and is fully covered by Bazel out of the box.

The Modern Bazel Modules Approach

Bazel used to rely on WORKSPACE files for dependency and toolchain management, but that approach quickly turns into a mess—too much boilerplate, too little structure. MODULE.bazel is the modern replacement, and it actually gets the job done. It lets you declare dependencies with proper versioning, avoids the usual transitive hell, and makes projects easier to share and compose. Instead of treating every repo like a silo, modules help build real ecosystems.

A typical module setup for a Go + C++ performance project may look like this:

module(
    name = "perf_tests",
    version = "0.1.0",
)

bazel_dep(name = "rules_go", version = "0.55.1")
bazel_dep(name = "googletest", version = "1.14.0.bcr.1")
bazel_dep(name = "google_benchmark", version = "1.9.2")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.23.1")

It’s pretty evident that you have a much clearer separation between configuration and implementation with modules compared with the old WORKSPACE files approach. I love how it makes dependencies explicit and maintainable!

Bazel’s Advantage over CMake and Go Build Systems

Bazel handles multiple languages out of the box. Trying to do the same with CMake and Go’s native tooling is either painful or flat-out unworkable. CMake is great (primarily for historical reasons) for C++, but anything beyond that usually means wrestling with clunky extensions. Go’s build system works well—for Go—but it has zero awareness of external toolchains or how to wire them together. There’s no clean way to coordinate builds across both without gluing everything together by hand.

This is where Bazel actually delivers. It builds C++ and Go in the same context, tracks dependencies across both, and rebuilds only what’s necessary—no wrappers, no hacks. Outputs are reproducible regardless of language, which means CI doesn’t have to guess, and toolchain drift isn’t a thing.

Joining Multilingual Performance Tests with Bazel Tags

Running benchmarks across languages is usually messy—different tools, different assumptions, no common entry point. Bazel simplifies this by letting you tag benchmark targets regardless of language. With consistent tags in place, you can query and run all benchmarks from a single script without needing language-specific logic. You can write it up pretty quickly:

Tagging Benchmark Targets

C++ benchmark (daily-temps/cpp/BUILD.bazel):

cc_binary(
    name = "daily_temperatures_benchmark",
    srcs = ["daily_temperatures_benchmark.cpp"],
    deps = [
        ":daily_temperatures",
        "@google_benchmark//:benchmark_main",
    ],
    tags = ["benchmark"],
)

Go benchmark (daily-temps/go/BUILD.bazel):

go_test(
    name = "daily_temperatures_test",
    srcs = ["daily_temperatures_test.go"],
    embed = [":daily_temperatures"],
    importpath = "perf-tests/daily-temps/go",
    tags = ["benchmark"],
)

Running All Benchmarks via Bash

With all benchmarks tagged consistently, a simple Bash script can query and execute benchmarks for multiple languages at once:

#!/bin/bash
set -euo pipefail

echo "Executing Go benchmarks..."
bazel test --test_output=streamed \
  --test_arg=-test.bench=. \
  --test_arg=-test.benchmem \
  --test_tag_filters=benchmark \
  --cache_test_results=no \
  //...

echo "Executing C++ benchmarks..."
cpp_benchmarks=$(bazel query 'attr(tags, "benchmark", kind(cc_binary, //...))')

for bench in $cpp_benchmarks; do
  echo "Running $bench"
  bazel run -c opt "$bench"
done

It might look crude, but using tags with a Bash runner is hard to beat for flexibility and portability. Bazel doesn’t natively support “run all benchmarks,” and this approach gives you exactly that—without reinventing anything. It scales across languages, plays well with CI, and stays out of the way.

Bazel handles complex multilingual builds in a way that legacy tools like CMake or language-specific systems simply don’t. With modules, tagging, and minimal scripting, even mixed-language benchmarks become easy to manage. For engineers who care about repeatability, clean integration, and performance visibility across stacks, Bazel isn’t just an option—it’s the default.

Functional Programming style in C++

Sat, 21 Jun 2025 00:00:00 GMT

Teskey-Torpok Pass is a lovely pass leading you to Song-Kol Lake, Naryn region.

I’ve been on a bit of a Leetcode streak lately, poking at problems from companies I secretly admire. To keep things interesting (and to avoid nodding off in front of the screen), I challenged myself to solve the same task three ways: good old C++, its shiny modern C++20 cousin, and Elixir. It turns out that staring at a problem through a functional programming lens is like putting on X-ray specs—you see the same lines of code, but suddenly, there’s a weird beauty in that filter chain.<

My first pass was as traditional as it gets—a simple C++ class with an array and a loop. No magic here, just pushing timestamps and scanning them one by one. The implementation is pretty naive, but considering the constraints provided by the Design a hit counter challenge, even an unscalable approach is acceptable. So, we will push all new timestamps into the ~~std::queue~~ std::vector and then simply count.

class HitCounter {
    std::vector&lt;int> timestamps_;
    static constexpr int HITS_PERIOD = 300;
public:
    HitCounter() {
        timestamps_.reserve(300);
    }

    void hit(int timestamp) {
        timestamps_.push_back(timestamp);
    }

    int getHits(int timestamp) {
        int res = 0;
        int delta = timestamp - HITS_PERIOD;

        for(auto it=timestamps_.begin(); it &lt;= timestamps_.end(); it++) {
            if(*it > timestamp) {
                break;
            }
            if(*it > delta) {
                res++;
            }
        }
        return res;
    }
};

And, yes, this solution is completely compliant with the requirements and passes all tests. But it’s hideous, isn’t it?

Next up was the Elixir version, where I leaned on an Agent to hold my list of timestamps. Elixir doesn’t let you mutate global state willy-nilly, so I spun up a small process to mind that list for me. Here’s the gist:

defmodule HitCounter do
  @hits_period 300
  @agent_name __MODULE__

  @spec init_() :: :ok | {:error, any}
  def init_() do
    if Process.whereis(@agent_name) do
      Agent.stop(@agent_name)
    end

    Agent.start_link(fn -> [] end, name: @agent_name)
    |> case do
      {:ok, _pid} -> :ok
      error -> error
    end
  end

  @spec hit(timestamp :: integer) :: :ok
  def hit(timestamp) when is_integer(timestamp) do
    Agent.update(@agent_name, fn timestamps -> timestamps ++ [timestamp] end)
  end

  @spec get_hits(timestamp :: integer) :: integer
  def get_hits(timestamp) when is_integer(timestamp) do
    lower_bound = timestamp - @hits_period

    Agent.get(@agent_name, fn timestamps ->
      timestamps
      |> Stream.take_while(fn ts -> ts <= timestamp end)
      |> Stream.filter(fn ts -> ts > lower_bound end)
      |> Enum.count()
    end)
  end
end

Why an Agent? Because each test run in Leetcode’s harness executes sequentially without dropping your module’s state. If you don’t stop the Agent on init_/0, you end up with phantom hits from previous tests—state leaks faster than coffee through a paper filter. I guarantee a clean slate every time by explicitly stopping any existing Agent before starting fresh.

Finally, I couldn’t resist a modern C++ take using C++20 ranges and views. No loops, no manual counters—just a pipeline that reads like a little poem:

int getHits(int timestamp) {
    int delta = timestamp - HITS_PERIOD;

    return std::ranges::distance(
        timestamps_
        | std::views::take_while([=](int ts) {
            return ts <= timestamp; // stop processing beyond this point
        })
        | std::views::filter([=](int ts) {
            return ts > delta;
        })
    );
}

It’s concise, lazy, and stops early once hits exceed the current timestamp—exactly mirroring our original loop logic. Feels a bit like writing haiku in code: short lines, precise meaning, and a dash of Zen.

Mixing paradigms this way has been surprisingly refreshing. You refactor the same logic through different lenses and discover nuances you’d never notice if you stuck to one language. Plus, it’s way more entertaining than another unadorned C++ loop.

Fixing C++ lacks switches on strings

Mon, 17 Feb 2025 00:00:00 GMT

Birch Grove is just 40 minutes from Bishkek. Although I was concerned about the strong, foggy weather, it was an amazing opportunity for photography!

C++ is a powerful language, and I genuinely love it, but sometimes, even in modern versions, it lacks some surprisingly simple features. One such missing feature is switch on std::string. You’d think that by now, we could use a switch statement on strings just like we do with integers or enums—after all, Go has it! But no, C++ keeps us on our toes.

Why Doesn’t C++ Support switch on strings? Because “you only pay for what you use,” which is the standard C++ mantra. The switch statement in C++ relies on integral types. Under the hood, it works by converting the case values into jump table indices for efficient execution. But std::string (or even std::string_view) is not an integral type—it’s a more complex data structure. That’s why you can’t simply do:

switch (msg->get_value<std::string>()) { // Nope, not possible :-(
    case "topology":
        // Handle network topology
        break;
    case "broadcast":
        // Handle network broadcast
        break;
}

So… how do we work around this? We need a way to turn strings into something switch can handle—like an integer. And that’s where hashing comes in. If we can map strings to unique integer values at compile time, we can switch on those instead. To do so, we need a user-defined literal operator and compile-time hash function. Something like this:

constexpr uint64_t hash(std::string_view str) {
    uint64_t hash = 0;
    for (char chr : str) {
        hash = (hash * 131) + chr;
    }
    return hash;
}

constexpr uint64_t operator"" _hash(const char* str, size_t len) {
    return hash(std::string_view(str, len));
}

And, no magic, we have switch on strings!

switch (hash(msg->get_value<mf::data_type>())) {
    case "topology"_hash: {
        // Handle topology case
        break;
    }
    case "broadcast"_hash: {
        // Handle broadcast case
        break;
    }
    default:
        // Handle unknown case
        break;
}

There are some downsides to using this approach, specifically regarding hash collisions. Hashing introduces a small, but non-zero, risk of collisions. The proposed hash function, which multiplies the current hash by 131 and adds each character, is sufficiently effective for this use case. It provides a fast, compile-time method to map short, distinct string literals to unique integer values.

The choice of 131 as a multiplier contributes to achieving a reasonable distribution of values, thus reducing the likelihood of collisions, especially for a limited set of keywords like “topology,” “broadcast,” and others. The number 131 is commonly used in simple hash functions because it is a prime number. This helps to spread out the hash values more evenly across the available space, minimizing clustering and improving distribution properties. Additionally, it is small enough to maintain fast calculations while ensuring sufficient entropy accumulation over short strings.

Since the hash is computed at compile-time (using constexpr), there is no runtime overhead, and the resulting integer comparisons in switch statements remain O(1). This makes this approach both efficient and practical for situations involving a limited number of predefined strings. Furthermore, the risk of hash collisions is exceedingly low for a small number of strings. Using the Birthday Problem approximation, we can estimate that even with 1,000 unique strings, the collision probability remains below 0.000003%, making it practically irrelevant for most real-world applications where a switch statement would be used.

It’s ridiculous that in 2025, we still have to hack our way around something as basic as switch on strings in C++. But on the bright side, this was a perfect excuse to use a user-defined literal (operator"") finally! It’s one of those C++ features you rarely use, but it feels like a wizard when you do.

AI, Software Engineering, and the Evolution of Code Generation

Wed, 05 Feb 2025 00:00:00 GMT

From a weekend trip to a nearby gorge. Why is it here? Because I love the mountains of Kyrgyzstan!

Mark Zuckerberg recently made a bold statement: AI will soon take over the work of mid-level engineers (Forbes). While this may sound like another tech CEO hyping AI, my latest experience with OpenAI’s o3-mini-high model suggests he might not be too far off.

Thanks to DeepSeek, OpenAI was compelled to make o3-mini-high available in a regular ChatGPT subscription instead of locking it behind a steep $200 paywall. I would never pay original $200 for a model, but since I already have the regular ChatGPT subscription, it was an obvious choice to try it out. With this in mind, I decided to experiment: Could o3-mini-high generate a functional Go codebase for my GFSM library?

The experiment

For context, GFSM is my Go Finite State Machine library, and I needed a new generator to extract and save state machines in formats like PlantUML and Mermaid. Writing such a generator requires a solid understanding of Go’s Abstract Syntax Tree (AST) package, something I hadn’t used in years. Instead of writing the code myself, I handed most of the heavy lifting to o3-mini-high. The result? Almost all the code for the generator was AI-generated, with minimal manual adjustments. You can check out the generated code in this GitHub commit.

Code Quality: Mid-Level Engineer Competency (But Better Documentation)

The AI-generated code was not just functional—it worked on the first attempt! Unfortunately, it still needed some refinements:

Improving logging and error handling
Refactoring functions for better readability and maintainability

Overall, the output quality was comparable to that of a solid mid-level engineer—not perfect, but good enough. But here’s the kicker: Mid-level engineers usually suck at writing documentation (if they write any at all), while the documentation generated by o3-mini-high was actually pretty nice—clear, structured, and covering all the necessary details.

My productivity gains may not be impressive, but they are reasonably good for a first attempt:

Time spent “coding” with ChatGPT: 3 hours<
The time it would have taken manually (my estimation): 6+ hours

The biggest time-saver? AI handled the AST package interactions, which would have required me to refresh my knowledge.

The Future of Software Engineering

What does this mean for engineers? I see this as an evolutionary shift rather than a revolutionary one—similar to previous transitions from Assembly to C or from C to Java. AI isn’t outright replacing engineers; it’s redefining what it means to be one.

The reality is that “boot camp engineers” and all other types of IT specialists without fundamental education are likely obsolete. Learning how to piece together APIs or follow tutorials won’t be enough when AI can do it better and faster, with approximately the same number of mistakes. Instead, fundamental Computer Science education—actual deep IT knowledge—becomes more critical than ever.

Why? Because understanding how and why a system works will distinguish those who can drive AI-assisted development from those who are replaced by it. Engineers with a strong foundation in algorithms, data structures, OS internals, networking, cybersecurity, and distributed systems will remain invaluable.

The AI-driven migration process will not be fast or simple. Even at first glance, there are a significant number of challenges ahead.

The crucial part of the AI migration challenge is system verification. If AI is going to generate more and more of our code, who is going to verify that it actually works? This is where Software Development Engineers in Test (SDET), or rather their next evolution—Software Verification Engineers (SVE), become indispensable. SVE will focus not just on functional correctness but also on formal verification, property-based testing, and system reliability. AI can generate code, but it doesn’t inherently know if it’s correct, secure, or efficient—that’s where human engineers must step in.

Another major challenge? AI-generated code will include security vulnerabilities. It’s not a question of if but when. AI models don’t “understand” security—they just generate plausible-looking code. This means:

Secure coding practices must be enforced at the architecture level.
Penetration testing and threat modeling will be more crucial than ever.
Automated security analysis tools will need to evolve alongside AI-generated code.

Security engineers will no longer be an “optional” hire for big enterprises. Every AI-assisted development team will need a strong cybersecurity presence, or they risk shipping highly vulnerable software.

And the ultimate pain in the ass? AI suggests solutions for AI-generated code. This is where I start crying :-D Right now, AI is great at creating “reasonable-looking” code but lacks true reasoning about correctness and intent. This means:

AI suggests code that works—but may be subtly wrong.
The engineer spots a bug and asks AI for a fix.
AI proposes a fix—but does it actually solve the problem, or does it introduce new ones?
Repeat until frustration kicks in.

This loop is already frustrating enough when debugging human-written code. With AI generating both the problem and the solution, we might end up in an infinite AI-fix cycle, in which engineers waste more time validating AI’s reasoning than fixing it manually.

How do I see my future? It’s perfectly incredible! With AI tools, I can do more, build faster, and focus on higher-level problem-solving instead of getting bogged down in boilerplate coding. AI is an amplifier, not a replacement—it extends my capabilities rather than diminishing my value.

And as for job security? I’m not worried in the slightest. With my specialization and experience in system architecture, distributed computing, and software engineering at scale, I will always have a place in this industry. If anything, AI clears the way for me to work on more interesting, complex problems—things AI alone can’t solve.

How to compile C++ in 2025. Bazel or CMake?

Mon, 20 Jan 2025 00:00:00 GMT

Today, we’re examining two modern build systems for C++: CMake, the industry favorite, and Bazel, a powerful alternative. While CMake is often the default choice, I believe that approach warrants a bit more scrutiny—after all, we’re focusing on modern tools here (yep, not counting Make, right?). To explore this, I’ve created a practical demo project showcasing how both systems manage a real-world scenario.

Using the maelstrom-challenges project as a starting point, I’ve extracted a C++ library called maelstrom-node. This library has been set up to work seamlessly with both Bazel and CMake, giving us a hands-on comparison of their approaches, strengths, and quirks.

The Project Structure

Here’s what the final directory layout for maelstrom-node looks like:

➜  maelstrom-node git:(main) tree
.
├── BUILD.bazel
├── CMakeLists.txt
├── CMakePresets.json
├── LICENSE
├── MODULE.bazel
├── MODULE.bazel.lock
├── README.md
├── include
│   └── maelstrom-node
│       ├── handler.hpp
│       ├── message.hpp
│       └── node.hpp
├── src
│   ├── CMakeLists.txt
│   ├── message.cpp
│   └── node.cpp
├── vcpkg-configuration.json
└── vcpkg.json

With this structure in place, the library could seamlessly integrate with both Bazel and CMake while maintaining clarity and modularity.

Building with Bazel

Bazel provided a relatively straightforward approach to compile maelstrom-node. The key was to define two critical files: MODULE.bazel and BUILD.bazel.

MODULE.bazel

The MODULE.bazel file in Bazel serves as a central place to define project metadata and manage dependencies. In this file, we use bazel_dep for some dependencies like spdlog and nlohmann_json, but not for rules_boost. Why the difference? It all comes down to how these dependencies are distributed and managed in Bazel’s ecosystem.

module(
    name = "maelstrom_node",
    version = "1.0.0",
    compatibility_level = 1,
)

# Dependencies

# Adding spdlog and nlohmann_json via Bazel Dependency Archive
bazel_dep(name = "spdlog", version = "1.15.0.bcr.3")
bazel_dep(name = "nlohmann_json", version = "3.11.3.bcr.1")

# Adding rules_boost manually due to its unavailability in Bazel Central Registry
bazel_dep(name = "rules_boost", repo_name = "com_github_nelhage_rules_boost")
archive_override(
    module_name = "rules_boost",
    integrity = "sha256-ZLcmvYKc2FqgLvR96ApPXxp8+sXKqhBlCK66PY/uFIo=",
    strip_prefix = "rules_boost-e3adfd8d6733c914d2b91a65fb7175af09602281",
    urls = ["https://github.com/nelhage/rules_boost/archive/e3adfd8d6733c914d2b91a65fb7175af09602281.tar.gz"],
)

The bazel_dep function simplifies dependency management by fetching dependencies from the Bazel Central Registry (BCR)—a curated collection of Bazel-compatible packages. Dependencies like spdlog and nlohmann_json are available in the BCR, which makes adding them straightforward and efficient. Using BCR ensures version consistency and eliminates the need for custom configuration.

Why Not Use `bazel_dep` for `rules_boost`?

While popular, the rules_boost library is not often updated in the Bazel Central Registry, and while we have 1.86 release publicly available, BCR includes only 1.83. That’s why, we may need to configure its source using archive_override manually. This involves specifying details like the URL of the archive, the integrity hash, and the strip prefix.

Using bazel_dep wherever possible simplifies project configuration by leveraging the Bazel Central Registry, ensuring that dependencies are well-maintained and easy to integrate. However, when libraries like rules_boost aren’t available in the registry, Bazel’s manual configuration options, like archive_override, allow us to include them. This dual approach showcases Bazel’s flexibility and ability to adapt to different project needs.

BUILD.bazel

We cannot proceed without BUILD.bazel, as it serves as a location to store the actual app or library information.

cc_library(
    name = "maelstrom_node",
    srcs = [
        "src/message.cpp",
        "src/node.cpp",
    ],
    hdrs = glob(["include/maelstrom-node/**/*.hpp"]),
    copts = [
        "-std=c++23",
    ],
    includes = ["include"],
    visibility = ["//visibility:public"],
    deps = [
        "@boost//:algorithm",
        "@boost//:asio",
        "@nlohmann_json//:json",
        "@spdlog",
    ],
)

The includes = ["include"] directive ensures Bazel can resolve headers using shorter paths like #include "maelstrom-node/message.hpp" instead of full paths like #include "include/maelstrom-node/message.hpp".

To build the library, simply run:

bazel build //...

Building with CMake

Adapting maelstrom-node for CMake brought its own set of challenges and solutions, exposing the limitations of this ancient technology. Despite its widespread use, even a straightforward migration effort highlighted CMake’s inherent bottlenecks and inefficiencies, reinforcing the idea that it might be time to consider moving on from it entirely.

Managing Includes

This is the first challenge you’ll face when trying to move from Bazel to CMake. CMake assumes #include paths are relative to the include directory, so you write:

#include "maelstrom-node/message.hpp"

Bazel, on the other hand, often requires full paths from the project root unless explicitly configured using the includes property in the cc_library rule:

#include "include/maelstrom-node/message.hpp"

While CMake’s shorter paths may appear cleaner within the source code and more natural for C++ developers, they abstract away the file hierarchy, making it harder to understand the project structure at a glance. As Bazel requires, full paths provide better context by explicitly showing where a file is located in the directory tree, making navigation and maintenance easier, especially in larger projects.

Setting Up CMake and VCPKG

Using vcpkg as the package manager is one of the ways to simplify dependency management if you are a CMake fun. After installing vcpkg, I initialized the project and added the necessary ports. The following command will create vcpkg-configuration.json and vcpkg.json files:

vcpkg new --application
vcpkg add port boost-algorithm
vcpkg add port boost-asio
vcpkg add port nlohmann-json

Another option for CMake dependency management is Conan. However, based on my experience, it’s even more complex than VCPKG, so I won’t discuss it in this post.

Making a CMakeLists.txt

Managing dependencies was relatively streamlined using the vcpkg package manager. The toolchain file was included dynamically to ensure compatibility, allowing for flexibility when configuring the build environment. This setup reduces manual configuration efforts while ensuring that dependencies are correctly integrated:

if(NOT DEFINED CMAKE_TOOLCHAIN_FILE)
    set(CMAKE_TOOLCHAIN_FILE "${CMAKE_SOURCE_DIR}/vcpkg/scripts/buildsystems/vcpkg.cmake"
        CACHE STRING "Vcpkg toolchain file")
endif()

The main part of the CMakeLists.txt file is standard for CMake projects. The maelstrom_node library is defined as a static library with its core functionality implemented in message.cpp and node.cpp. Dependencies such as Boost, nlohmann_json, and spdlog are configured using find_package, ensuring they are available during the build process.

add_library(maelstrom_node STATIC
    src/message.cpp
    src/node.cpp
)

find_package(Boost REQUIRED NO_MODULE)
find_package(nlohmann_json REQUIRED)
find_package(spdlog REQUIRED)

target_include_directories(maelstrom_node PUBLIC
    ${CMAKE_SOURCE_DIR}/include
    ${Boost_INCLUDE_DIRS}
)

target_link_libraries(maelstrom_node PUBLIC
    nlohmann_json::nlohmann_json
    spdlog::spdlog
    Boost::headers
)

The find_package command works seamlessly in this setup because the vcpkg toolchain file is integrated into the project configuration. This file, specified as CMAKE_TOOLCHAIN_FILE, ensures that CMake knows the paths to all dependencies installed via vcpkg. When find_package is called, it searches these paths for the necessary configuration files, allowing dependencies like Boost, nlohmann_json, and spdlog to be located and linked automatically. This integration eliminates manual dependency management and ensures consistency across environments.

Finally, the installation section of the file ensures that the library can be packaged and reused in other projects. This includes specifying where the compiled library should be installed and ensuring that headers are copied to the appropriate directories:

install(TARGETS maelstrom_node
    EXPORT maelstrom_nodeConfig
    ARCHIVE DESTINATION lib
    LIBRARY DESTINATION lib
    RUNTIME DESTINATION bin
)

install(DIRECTORY ${CMAKE_SOURCE_DIR}/include/maelstrom-node
    DESTINATION include/maelstrom-node
)

This setup ensures the maelstrom-node library is compiled correctly and packaged in a way that makes it easy to integrate into other projects. While CMake offers much flexibility, its verbosity and reliance on external tools like vcpkg highlight some of its limitations in managing dependencies efficiently. CMake provides multiple approaches to configure and build the project. Ideally, you could use a preset configuration for simplicity, such as (based on the official documentation):

cmake --preset=vcpkg
cmake --build build

This approach is clean and convenient, leveraging a preset defined in CMakePresets.json. However, due to a defect in Visual Studio Code’s CMake Tools, environment variables like $VCPKG_ROOT cannot be directly used in CMakePresets.json. As a result, the preset may fail to locate the toolchain file correctly.

To address this issue, the following manual configuration proves reliable despite its ugliness:

cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake
cmake --build build

Both methods ultimately lead to the same result, but the manual approach is necessary in environments where preset limitations or toolchain file paths are not resolved automatically. This highlights a minor inconvenience in the CMake ecosystem that can complicate workflows when integrating with tools like vcpkg.

Let’s integrate it!

Let’s integrate maelstrom-node into our application! This is the final step to bring everything together, and it’s where the differences between Bazel and CMake really stand out in terms of setup, dependency management, and usability. Here’s how each approach handles this critical task.

Bazel Integration

In Bazel, the integration is streamlined by declaring the maelstrom-node library as a dependency using bazel_dep and providing a git_override to fetch the library from its repository:

bazel_dep(name = "maelstrom_node", version = "1.0.0")
git_override(
    module_name = "maelstrom_node",
    commit = "004b1d793427838db32d5e175f1f474bec260766",
    remote = "https://github.com/astavonin/maelstrom-node.git",
)

The library is then added as a dependency in the build rule for the application:

cc_binary(
    name = "echo-cpp",
    srcs = [
        "main.cpp",
    ],
    copts = [
        "-std=c++23",
    ],
    visibility = ["//visibility:public"],
    deps = [
        "@maelstrom_node",
        "@spdlog",
    ],
)

Bazel’s approach to dependencies is more declarative and centralized. Dependencies are fetched and managed globally at the repository level, ensuring consistency across the project. The use of bazel_dep and overrides simplifies versioning, and Bazel’s caching mechanism avoids redundant downloads and builds, leading to faster iterations.

The compilation process is still very straightforward for the final app:

bazel build //...

CMake Integration

The CMakeLists.txt file for the integration is relatively standard, with one notable addition: the use of FetchContent to fetch and include the maelstrom-node library directly from its Git repository. This setup is straightforward, as shown below.

include(FetchContent)

FetchContent_Declare(
    maelstrom_node
    GIT_REPOSITORY https://github.com/astavonin/maelstrom-node.git
    GIT_TAG v1.0.0
)

FetchContent_MakeAvailable(maelstrom_node)

This snippet looks pretty and allows the library to be built alongside the application. But, it’s not without its drawbacks. Unlike Bazel, CMake relies heavily on external tools like vcpkg for dependency management. Unfortunately, vcpkg requires a complete list of dependencies to be declared for every project (is it a defect? It looks so to me.). In this case, both echo-cpp and maelstrom-node must redundantly list their dependencies in echo-cpp’s vcpkg.conf file. This duplication can be tedious and error-prone, especially in larger, more complex projects.

Another significant limitation is CMake’s lack of integration with other ecosystems like Rust and Cargo. If you already have existing integration tests written in Rust, as I have, CMake cannot reuse them natively. This inability to leverage cross-language infrastructure is one of CMake’s most significant downsides and a stark contrast to Bazel, which excels in mixed-language project support.

While this integration approach works, it highlights the challenges of relying on CMake for projects beyond simple use cases, especially when dealing with multiple dependencies or mixed-language ecosystems.

Managing Multi-Language Projects with Bazel

Tue, 14 Jan 2025 00:00:00 GMT

In today’s software development landscape, it’s rare to encounter a project built with just one programming language or platform. Modern applications often require integrating multiple technologies to meet diverse requirements. This complexity is both a challenge and an opportunity, demanding robust tools to manage dependencies, builds, and integrations seamlessly. Bazel, a powerful build system, is one such tool that has proven invaluable for multi-language projects.

Recently, I decided to extend my Maelstrom challenges with a C++-based test to explore how Bazel can simplify managing multi-language dependencies and streamline development workflows.

Why Bazel for Multi-Language Projects?

Bazel’s design philosophy emphasizes performance and scalability, making it an excellent choice for projects that involve multiple languages. With its support for Bazel modules, adding dependencies is as simple as declaring them in a MODULE.bazel file. For example, integrating the popular logging library spdlog is straightforward:

bazel_dep(name = "spdlog", version = "1.15.0.bcr.3")

However, not every library supports Bazel modules natively, which can complicate things. A case in point is BOOST, a widely used C++ library. Adding BOOST required a bit of additional setup, fortunately, we have rules_boost project:

bazel_dep(name = "rules_boost", repo_name = "com_github_nelhage_rules_boost")

archive_override(
    module_name = "rules_boost",
    integrity = "sha256-ZLcmvYKc2FqgLvR96ApPXxp8+sXKqhBlCK66PY/uFIo=",
    strip_prefix = "rules_boost-e3adfd8d6733c914d2b91a65fb7175af09602281",
    urls = ["https://github.com/nelhage/rules_boost/archive/e3adfd8d6733c914d2b91a65fb7175af09602281.tar.gz"],
)

non_module_boost_repositories = use_extension("@com_github_nelhage_rules_boost//:boost/repositories.bzl", "non_module_dependencies")
use_repo(
    non_module_boost_repositories,
    "boost",
)

While the setup for BOOST is less elegant than for spdlog, Bazel’s flexibility ensures that even complex dependencies can be integrated efficiently.

Another critical aspect of C++ development is code navigation. Generating a compile_commands.json file makes life easier for developers using tools like Vim or Emacs. Thankfully, Bazel has a powerful solution: Hedron Compile Commands Extractor.

It’s pretty easy to set it up:

1. Add the dependency to your MODULE.bazel file:

bazel_dep(name = "hedron_compile_commands", dev_dependency = True)

git_override(
    module_name = "hedron_compile_commands",
    commit = "4f28899228fb3ad0126897876f147ca15026151e",
    remote = "https://github.com/hedronvision/bazel-compile-commands-extractor.git",
)

2. Generate the compile_commands.json file by running:

bazel run @hedron_compile_commands//:refresh_all

This simple addition dramatically improves the developer experience, making it easy to navigate and understand the C++ codebase. What I love even more there is NeoVim’s lsp-config provides excellent navigation and auto-completion with the generated compile_commands.json.

Building C++ Support in Maelstrom challenges

The process of adding C++ support to the Maelstrom challenges project involved several key changes. In this commit, I outlined the required modifications to the MODULE.bazel file to set up the necessary dependencies. This provided the foundation for implementing a basic Maelstrom protocol in C++.

The implementation itself, along with a simple echo app as a proof of concept, can be seen in this commit. This example showcases how Bazel simplifies managing dependencies and building multi-language projects while allowing for robust testing and experimentation.

Bazel and Rust: A Perfect Match for Scalable Development

Sun, 29 Dec 2024 00:00:00 GMT

Bazel never fails to impress, and its support for Rust demonstrates its versatility and commitment to modern development. Two distinct dependency management modes—Cargo—based and pure Bazel—allow developers to tailor workflows to their projects’ needs. This adaptability is particularly valuable for integrating Rust applications into monorepos or scaling complex systems. I decided to explore how Bazel supports Rust, including managing dependencies, migrating from Cargo.toml to BUILD.bazel, and streamlining integration testing.

Harnessing Cargo-Based Dependency Management

Bazel’s ability to integrate with Cargo, Rust’s native package manager, is a standout feature. This approach preserves compatibility with the Rust ecosystem while allowing projects to benefit from Bazel’s powerful build features. By using rules_rust, a Bazel module can seamlessly import dependencies defined in Cargo.toml and Cargo.lock into its build graph. It was pretty easy to feed dependency information from Cargo to Bazel. The only tricky part is Cargo workspaces. If you use a Cargo workspace like I do, you should list all Cargo.toml files under the manifests section in MODULE.bazel file.

crate = use_extension("@rules_rust//crate_universe:extensions.bzl", "crate")
crate.from_cargo(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    manifests = [
        "//:Cargo.toml",
        "//:echo/Cargo.toml",
        "//:tests/Cargo.toml",
    ],
)
use_repo(crate, "crate_index")

This setup allows Bazel to parse and manage dependencies from multiple Cargo.toml files. The crate_universe extension ensures that Bazel respects the dependency versions specified in the lockfile, providing reproducible builds. This approach is particularly practical in projects with nested crates or submodules, as Bazel automatically consolidates them into a unified build graph.

Integrating with Cargo in this way provides the best of both worlds. Developers can continue using Rust’s native tooling for development while leveraging Bazel for its scalability and advanced dependency management. Although this dual-system approach adds some complexity, the flexibility it offers is invaluable for scaling projects and ensuring compatibility with the broader Rust ecosystem.

Migrating from `Cargo.toml` to `BUILD.bazel`

The next step after making a proper MODULE.bazel file is to create BUILD.bazel. It’s basically a translation of all component-level Cargo.toml files into BUILD.bazel one-by-one.

[package]
name = "echo"
version = "0.1.0"
edition = "2021"

[[bin]]
path = "src/main.rs"
name = "echo"

[dependencies]
async-trait = "0.1.83"
maelstrom-node = "0.1.6"

In Bazel, each bin section from Cargo.toml should be translated into a rust_binary definition in BUILD.bazel. Another important part here is @crate_index which refer to crate.from_cargo(name = "crate_index", ...) command from the MODULE.bazel described above.

load("@rules_rust//rust:defs.bzl", "rust_binary")

rust_binary(
    name = "echo",
    srcs = ["src/main.rs"],
    proc_macro_deps = [
        "@crate_index//:async-trait",
    ],
    deps = [
        "@crate_index//:maelstrom-node",
    ],
)

This shift ensures that Bazel can handle the entire build lifecycle, from compiling dependencies to linking final binaries. While Cargo.toml defines dependencies and build metadata, BUILD.bazel brings this into Bazel’s optimized dependency graph. This transition is seamless for developers familiar with both systems and ensures scalability as the project grows.

Simplifying Integration Testing with Bazel

Integration testing is another area where Bazel simplifies workflows compared to Cargo. In Rust’s native system, it’s not straightforward to ensure that an application is built before a test requiring that application is executed. With Bazel, this process becomes effortless.

Consider a BUILD.bazel setup for integration tests:

load("@rules_rust//rust:defs.bzl", "rust_library", "rust_test")

rust_library(
    name = "utils",
    srcs = [
        "utils/lib.rs",
        "utils/paths.rs",
        "utils/runner.rs",
    ],
)

rust_test(
    name = "test_echo",
    size = "small",
    srcs = ["echo/test_echo.rs"],
    data = [
        ":maelstrom/lib/maelstrom.jar",
    ],
    deps = [
        ":utils",
        "//echo",
    ],
)

This configuration defines a rust_test that explicitly depends on the //echo binary. Bazel ensures that the binary is built before the test is executed, avoiding any manual coordination. Shared utilities used during the tests are included in a rust_library, promoting code reuse. Additionally, test resources like :maelstrom/lib/maelstrom.jar are specified as data, ensuring they are available during execution.

It took some Googling to figure out how to access external resources like maelstrom.jar for the Rust integration test. However, it’s simpler than it appears at first. We basically need to retrieve the value from the RUNFILES_DIR environment variable, which points to a temporary Bazel-managed directory, and then add the "_main" folder on top of it.

pub fn bazel_runfiles_dir() -> PathBuf {
    PathBuf::from(env::var_os("RUNFILES_DIR").unwrap()).join("_main")
}

pub fn maelstrom_dir() -> PathBuf {
    bazel_runfiles_dir()
        .join("tests")
        .join("maelstrom")
        .join("lib")
}

Bazel’s approach eliminates the need for custom scripts or workarounds, streamlining the test lifecycle and ensuring reliable, reproducible results. This simplicity becomes increasingly important as projects grow, where manual processes can hinder productivity and introduce inconsistencies.

Returning to Rust: A Journey Through Tooling, Performance

Tue, 10 Dec 2024 00:00:00 GMT

When I started tackling the Maelstrom challenges, my initial thought was to use C++. It’s a language I know inside out, and its performance is hard to beat. However, as I contemplated setting up the project, I realized I couldn’t justify fighting with the C++ pipeline for free. Crafting a proper CMake or Bazel configuration might be worthwhile for large-scale projects or when compensated, but for personal experiments? It’s an unnecessary headache.

Why Go is My Default Choice

For most non-performance critical scenarios, Go is my default, no-brainer choice. It has a clean build system, excellent tooling, and a developer experience that doesn’t make me dread the setup process. Go’s simplicity allows me (and any level team) to focus on solving the problem rather than wrestling with the environment. Yet, this time, I decided to take a different path.

Revisiting Rust

My relationship with Rust has been a journey of highs and lows. I was initially drawn to it during its early days, around Rust 0.1, when the language was brimming with potential. What captivated me back then was its promise to be a kind of “Erlang on steroids”—a language that combined Erlang’s lightweight concurrency model and fault-tolerant programming paradigms with the performance and control of a systems language.

Unfortunately, with the release of Rust 1.0, much of that original vision shifted. Green threads—a key feature that enabled lightweight concurrency and aligned closely with the “Erlang on steroids” dream—were abandoned. The decision to remove them in favor of direct integration with the OS threading model made sense for speeding up the 1.0 release. Still, it marked a significant departure from the language’s earlier aspirations.

Adding to the change was a complete overhaul of Rust’s memory model, which moved toward its now-iconic ownership system. While the new model introduced unparalleled safety and performance guarantees, it also made the language more complex and less approachable, particularly for those who were excited about its initial simplicity.

These shifts left me feeling that Rust had transformed into “just another overcomplicated C++-like language,” albeit with better safety features and tooling. Over time, I moved away from Rust, drawn instead to more straightforward and developer-friendly languages for most of my work.

But now, years later, I’m giving Rust another shot while having plenty of free time. Its strong presence in emerging fields like Web3 has made it impossible to ignore. The language has clearly matured, and while its evolution may not have aligned with my initial hopes, I’m curious to explore how its modern ecosystem can contribute to my work. Rust may no longer be the “Erlang on steroids” I once envisioned, but it’s carving out a unique space for itself, and I’m eager to see how it fits into my technical journey moving forward.

Comparing Tooling and Developer Experience

My return to Rust brought some pleasant surprises—and a few frustrations. Here’s a breakdown of my observations:

Cargo vs. Build Systems

Cargo, Rust’s build system and package manager, is undeniably better than CMake—but then again, that’s not a high bar. Let’s be honest: CMake is an exercise of frustration for most developers.
Comparing Cargo to Bazel: Cargo feels more straightforward and approachable, but it requires more power and flexibility than Bazel. In that sense, Bazel is still the gold standard for me in large, complex projects.
Go’s build system, however, remains my favorite. Its simplicity and convention-over-configuration philosophy are unmatched.

IDEs and Editor Support

Tools like RustRover have evolved to deliver an exceptional development experience, far surpassing the likes of CLion or most other C++ IDEs. The level of integration with Cargo, code analysis, and developer assistance is simply stellar.

Compiler Diagnostics

One of Rust’s standout features is its compiler diagnostics. The clarity and detail in Rust’s error messages are leagues ahead of what you get with Clang or GCC. It’s an area where Rust truly shines, making debugging and iteration smoother. However, it’s not all perfect.

When it comes to borrowing-related errors, the story changes. These errors often feel nonintuitive, even for developers familiar with Rust’s ownership model. Despite the compiler’s best efforts to provide explanations, deciphering and resolving these errors frequently requires external help. You’ll find yourself Googling extensively or turning to tools like ChatGPT (model 4.0 preferred) to figure out what’s going wrong and how to fix it. This aspect of the Rust learning curve remains a significant hurdle for newcomers and a source of frustration even for experienced developers.

While the error diagnostics are still better than those offered by C++, the borrowing system’s complexity can sometimes undermine the excellent developer experience.

Reflections and Future Plans

Sometimes, stepping back into an ecosystem with a fresh perspective can reveal how much has changed—for better or worse. Rust is no longer just the “overcomplicated” C++ sibling I left behind. It’s still a language with comparable complexity to C++ (which means it’s overcomplicated) but with a critical difference: compile-time sanitizers you need in C++ baked into its Rust core. This feature transforms those complexities into tools that actively guide you toward safer, more robust code. Last but not least, as I’d mentioned above, I do not want to setup a proper C++ pipeline either :-D

Minimal CI for Go library with GitHub actions

Thu, 05 Dec 2024 00:00:00 GMT

Continuous Integration (CI) has become an essential part of modern software development, and for good reason. It ensures code quality, speeds up development, and catches potential issues early. However, you can get started without an elaborate CI setup. Even a minimal CI setup can significantly improve your workflow. Here’s why every project should have at least minimal CI and how to implement it effectively using GitHub Actions.

What Constitutes Minimal CI?

For a project to benefit from CI without excessive complexity, it should include the following essential components:

Project Compilation: Verify that the codebase is always in a buildable state.
Unit Test Execution: Ensure the core functionality works as expected.
Static Code Analysis: Catch bugs and enforce coding standards before they become an issue.

Why GitHub Actions?

GitHub Actions offers a simple yet powerful CI/CD platform that integrates seamlessly with GitHub repositories. Its ease of use makes it accessible even for developers without extensive DevOps experience. In fact, with just a few workflows, you can have a robust CI pipeline for small projects.

For example, for my small open-source Go library GFSM, three workflows—build, static-check, and publish—are sufficient.

Setting Up Your Workflows

Info

A fully functional GitHub Actions-based CI example from GFSM.

1. Build Workflow

The build workflow ensures your code compiles and passes tests on every <inl_code>push</inl_code> or <inl_code>pull_request</inl_code> to any branch:

on: ["push", "pull_request"]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: prepare-env
        uses: ./.github/env
      - name: Build
        run: go build -v ./...
      - name: Test
        run: go test -v ./...

2. Static Code Analysis

Static analysis helps maintain code quality by catching potential issues early. The best Go linter at the moment - staticcheck, provides an excellent integration with GitHub actions out of the box. Like the build workflow, this workflow runs on every push or pull_request:

jobs:
  ci:
    name: "Run CI"
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 1
      - name: prepare-env
        uses: ./.github/env
      - uses: dominikh/staticcheck-action@v1
        with:
          version: "latest"

3. Publish Workflow

Publishing a new version to pkg.go.dev ensures that users always have access to the latest updates. The tricky part is triggering the pkg.go.dev proxy to update its cache for new versions. On comparison with build and static-check workflows, we should trigger new version publication only on a new tag push in a semantic versioning format. Here’s how you can set up this workflow to track new tags and ping the proxy:

on:
  push:
    tags:
      - "v[0-9]+.[0-9]+.[0-9]+"

jobs:
  publish:
    name: "pkg.go.dev publishing"
    runs-on: ubuntu-latest
    steps:
      - name: Publishing new version
        run: |
          curl https://sum.golang.org/lookup/github.com/astavonin/gfsm@${{ github.ref_name }}

Reusable Composite Actions

Both the build and static-check workflows require the same environment setup. To avoid duplicating this logic, use GitHub Actions’ composite feature to create a reusable setup action:

runs:
  using: composite
  steps:
    - name: Setup Go
      uses: actions/setup-go@v5
      with:
        go-version: 1.23.x
    - name: Generate
      shell: bash
      run: |
        go get golang.org/x/tools/cmd/stringer@latest
        go install golang.org/x/tools/cmd/stringer@latest
        go generate ./...

This composite action ensures consistency and simplifies workflow maintenance.

Looking Ahead

Go currently relies on the tools.go file for dependencies, which I find a too ugly approach and prefer to have some extra go get/install/generate in my environment setup yaml file. But Go 1.24 promises a much cleaner approach with its track tool dependencies in go.mod (#48429) feature. Once available, it will streamline dependency management much more excellently.

Final Thoughts

Implementing minimal CI with GitHub Actions requires minimal effort but yields significant benefits. By automating builds, tests, and static analysis, you can ensure your project maintains high quality and is always ready for deployment. For small projects, this setup is a no-brainer, and GitHub Actions makes it easy to get started.

GFSM: A Simple and Fast Finite State Machine for Go

Mon, 25 Nov 2024 00:00:00 GMT

Design patterns are not widely used in Go, as they can often lead to unnecessary complexity in the codebase. However, the Finite State Machine (FSM) is an exception that proves to be incredibly useful. When I set out to design GFSM, I aimed to create a fast and straightforward FSM implementation for Go.

I initially sought a quick and straightforward FSM solution for Go, but I couldn’t find anything that met my needs. Drawing inspiration from the speed-focused and minimalistic principles of C++ while remaining true to Go’s idioms, I developed GFSM to fill this gap. The outcome is GFSM—a library that distinguishes itself from alternatives like looplab/fsm by prioritizing speed and simplicity.

Whether orchestrating microservices, handling distributed systems, or designing embedded systems, GFSM brings the reliability and efficiency needed to keep things running smoothly.

Real-World Use Case: Two-Phase Commit Protocol

One of the best showcases for FSMs is the Two-Phase Commit protocol (TPC), widely used in distributed systems to ensure atomic transactions. Let’s explore how GFSM can model this protocol step by step.

1. Define Your State Machine

To build a TPC protocol FSM, we start with a basic state machine:

flowchart TD;
    Init-->Wait;
    Wait-->Abort;
    Wait-->Commit;

This state machine has four states:

Init: The initialization phase.
Wait: Waiting for votes.
Abort: Aborting the transaction.
Commit: Committing the transaction.

2. Enumerate All Possible States

GFSM allows any comparable type as state identifiers, but integer-based enumerations keep things clean and fast. For TPC, here’s how the states are defined:

type State int
const (
    Init State = iota
    Wait
    Abort
    Commit
)

3. Define State Behaviors

Each state in GFSM is represented by an action handler implementing the StateAction interface:

type StateAction[StateIdentifier comparable] interface {
    OnEnter(smCtx StateMachineContext)
    OnExit(smCtx StateMachineContext)
    Execute(smCtx StateMachineContext, eventCtx EventContext) StateIdentifier
}

OnEnter: Called when entering the state.
OnExit: Called when leaving the state.
Execute: Handles events and determines the next state.

For example, the initState implementation might look like this:

func (s *initState) Execute(smCtx gfsm.StateMachineContext, eventCtx gfsm.EventContext) State {
    cCtx := smCtx.(*coordinatorContext)
    req, ok := eventCtx.(commitRequest)
    if !ok {
        // Handle invalid event
        return Init
    }
    // Transition logic
    return Wait
}

4. Build the State Machine

Using the StateMachineBuilder, you can define the states, their transitions, and the associated context:

sm := gfsm.NewBuilder[State]().
    SetDefaultState(Init).
    SetSmContext(&coordinatorContext{partCnt: 3}).
    RegisterState(Init, &initState{}, []State{Wait}).
    RegisterState(Wait, &waitState{}, []State{Abort, Commit}).
    RegisterState(Abort, &responseState{
        keepResp: Abort,
    }, []State{Init}).
    RegisterState(Commit, &responseState{
        keepResp: Commit,
    }, []State{Init}).
    Build()

Here, coordinatorContext holds transaction details, such as the number of participants and the commit ID.

5. Run the State Machine

Once constructed, the FSM is ready to process events. Start the state machine, handle events, and gracefully terminate it:

sm.Start()
defer sm.Stop()

err := sm.ProcessEvent(commitRequest{"commit_1"})
if err != nil {
    // Handle error
}

During event processing, the FSM delegates logic to the current state’s Execute method, determining whether to transition or remain in the current state.

Learn More

Ready to integrate GFSM into your project? Check out the documentation on pkg.go.dev and explore the examples for hands-on guidance.

Reflections on Reading “Introduction to Reliable and Secure Distributed Programming”

Mon, 18 Nov 2024 00:00:00 GMT

I’ve had Introduction to Reliable and Secure Distributed Programming sitting on my bookshelf for years, silently whispering, “Read me when you have time.” Of course, “later” always seemed like the right time. As software developers, we often prioritize practical, hands-on books that help solve immediate problems—topics like Kubernetes, Kafka or mastering another layer of C++ intricacies. But in hindsight, neglecting foundational theory is a mistake.

I began my journey with distributed systems over a decade ago. Back then, I wish someone had handed me this book and insisted I dive in immediately. It’s the kind of resource that can help set the foundation for anyone venturing into the complexities of distributed computing. Instead, I learned through trial, error, and practical exposure, which, while valuable, left gaps that only became apparent when I finally picked up this book.

The book doesn’t hold back—it dives deeply into the theoretical underpinnings of distributed systems. Despite my experience, I found several topics both fascinating and challenging. The discussions on randomized consensus with coin and hierarchical consensus stood out to me, offering insights that are as practical as they are thought-provoking. These are concepts that, while grounded in theory, can influence how we design systems in real-world applications. The chapter on randomized consensus with coin was particularly eye-opening. This approach leverages randomness to break symmetry among processes, a fundamental challenge in achieving consensus in distributed systems. By introducing a probabilistic element—like flipping a virtual “coin”—systems can progress even in scenarios where deterministic methods fail due to asynchronous environments or faults. The beauty of this method lies in its simplicity and elegance; it transforms a seemingly insurmountable problem into something tractable by embracing uncertainty. While not always the fastest approach, it offers a practical fallback in scenarios where other algorithms may struggle.

The adorable excuse.

**Hierarchical consensus** was another standout topic for me. This technique structures the consensus process across multiple layers or hierarchies, allowing for scalability and fault tolerance in large distributed systems. Instead of forcing a single, global agreement across all nodes, hierarchical consensus breaks the problem into smaller, manageable sub-problems, which are then aggregated into a system-wide decision. This approach is particularly relevant in modern architectures, where systems span multiple data centers or cloud regions. The ability to balance local decisions with global consistency is critical for maintaining performance and reliability at scale, and this chapter provided a clear framework for tackling that challenge.

Reading the book took me about three weeks longer than I’d planned. I’ll admit, I had an adorable excuse for the delay—Molly, my now-5-month-old German Shepherd puppy, decided that my reading sessions were the perfect time to demand attention. Progress was slower but enjoyable between her antics and the material’s complexity.

For anyone serious about distributed systems, I highly recommend carving out time for this book. It may not solve your immediate programming challenges, but it will fundamentally enhance how you think about distributed systems’ reliability, security, and design. And if you have a Molly of your own, well, consider it a bonus challenge.

Sabbatical and Distributed Systems

Mon, 30 Sep 2024 00:00:00 GMT

View from the rooftop of the apartment, where we settled down. I love running up on the left side of the river—quite a nice trail run.

Working at a unicorn company for four years was an exhilarating experience, though incredibly exhausting. Tackling significant projects that few have ever attempted means Google can’t offer much help. Surrounded by highly talented colleagues, many of whom you may never encounter again in one place, all while facing a relentless pace and high expectations from leadership, it eventually takes a toll. After a while, you just need to pause, rest, and reflect. That’s why I left Motional—to take several months to recharge, spend time with family, and reconnect with my hobbies.

The irony is, after more than 20 years in IT, I’ve become somewhat of a workaholic, so proper rest eludes me. Just last weekend, I found myself speaking at the DevFest ’24 conference, sharing insights on best practices in project development with the local IT community. The event exceeded all my expectations. It’s been incredibly gratifying to see how much Kyrgyzstan’s IT sector has grown since I left in 2003. I sincerely hope this growth continues and that, eventually, talented engineers will want to stay here or return to the country instead of trying themselves outside, as I did. However, I’m also acutely aware that I’m now overqualified for most roles in the Kyrgyz job market, which is a bit bittersweet. So, my journey will likely inevitably take me elsewhere after my sabbatical.

It’s funny (or maybe a bit sad, depending on how you look at it), but my workaholic side insists that taking it easy isn’t enough. Now, with plenty of free time, I’ve decided to tackle something I’ve always wanted to dive deeper into: Distributed Systems theory. Yes, the theory itself—not just the practical implementation that most developers focus on. So, I have a Plan. I had previously watched Martin Kleppmann’s Distributed Systems lecture series, but this time, I’m approaching it with more focus, adding the course’s accompanying textbook into the mix. A quick review shows that it covers what I’m looking for, though it doesn’t dive quite deep enough into the theoretical aspects.

At the same time, I’ve had Introduction to Reliable and Secure Distributed Programming by Christian Cachin and co-authors sitting on my bookshelf for a while, and that’s where I’ll be focusing my efforts. I suspect the Distributed Systems lecture series is grounded in much of this book’s theory. It may not be the most leisurely read on the art of distributed systems, but who ever said learning something worthwhile should be easy?

However, even theoretical study requires some hands-on practice. That’s where the fantastic Maelstrom tool from jepsen.io comes in. Maelstrom offers a series of challenges, starting with something simple like an Echo and progressing to more complex tasks like Raft. Since Maelstrom is a beautifully designed Clojure-based app with well-defined protocols for each challenge, adding more challenges should be both easy and fun.

Bishkek - Osh road. Kyrgyz mountains are magnificent at any time.

That said, most of this work revolves around Centralized Distributed Systems, which makes sense given their dominance. Naturally, I’m also curious about a solid theoretical resource on Decentralized Distributed Systems. The best I’ve found so far is the Decentralized Thoughts post series. It’s relatively easy to follow and seems to cover many important aspects of decentralized distributed systems theory. I’m unsure if a better resource exists, but it looks like a great starting point.

Ultimately, this sabbatical has given me the time to reconnect with my passion for learning and exploring the theoretical depths of distributed systems. Whether centralized or decentralized, each layer of complexity opens up new ways to think about how we build and maintain reliable systems. As I continue down this path, I’m excited to see where these studies will lead, and I hope that by sharing my journey, I can inspire others to dive deeper into the technical topics they’ve always wanted to explore.

TIL: how to debug randomly hanging Python applications

Wed, 29 Jun 2022 00:00:00 GMT

Usually, if a Python-based application hangs, you either read logs or grab one of the PBD-based solutions, attaching to the application, and uses the Python console for investigation. The approach is straightforward; for example, you installed <inl_code>pdb-attach</inl_code>, and add a few lines to your application:

import pdb_attach
pdb_attach.listen(50000)

and expect that “magic” will just works:

> python -m pdb_attach <PID> 50000
(Pdb) YOU HAVE PDB SESSION HERE

But sometimes, magic is broken, and my theory is (I didn’t search for proof) that this is due to GIL. So, sometimes, no PDB prompt after you have attached to the application with PDB. In my case, the application hang in the multiprocessing.Process call where I used a gRPC server. The gRPC server didn’t react to the termination request, the process cannot stop, and like aggravating circumstances, all these are a part of PyTest that hang 1 of 20 executions.

This is a general PDB-based debuggers issue, which means all other tools like pyrasite-shell and PyTest PBD integration also don’t work. The only option here is GDB for Python, which is surprisingly amazing! First of all, you need to install Python extension for GDB.

sudo apt-get install python3.9-dbg

Then you can connect to your Python application which is a regular Python process with GDB, and explore the call-stack!

> gdb

(GDB) attach <PID>
(GDB) py-bt

If you use not APT-based Linux, search for proper instruction here.

CodeBlueprint · Cloud/Edge, robotics, distributed platforms

Cross-Compiling C++ with Bazel: Platforms, Toolchains and the Exec/Target Split

Platforms Instead of --cpu

One Config per Target

Where the Toolchain Comes From

The Sysroot

Defaults That Fail Quietly

libstdc++ through the back door

A file nobody declared

An Execution Platform Nothing Runs On

Tests That Know Which Side They’re On

One Output Directory for Two Targets

Next Board, Next Steps

Footnotes

Context enrichment for multi-agent code review

Two reviews, one document

What I had that they didn’t

So give them more context

Three agents, one opinion

Reading the proposal last

Code paths touched

Footnotes

I cancelled my JetBrains subscription

What is actually on my screen in 2026

The IDE is very good at something I no longer do

What I would pay for and nobody sells

Non-determinism needs rules, not vibes

So I’m building a bad version of it myself

Where this lands

Footnotes

Over-engineering used to be bounded by boredom

The column that got deleted

Where it shows

The same thing at review time

The obvious diagnosis is wrong

Putting the price tag back

Vibe Coding experiment

Footnotes

Making GenAI code review actually useful

Why single-agent review fails predictably

The 2/3 inclusion rule

Severity consensus

Cross-model verification

What the agents should not flag

What this doesn’t solve

Footnotes

IPO Season and the Death of Software Engineering (Again)

December 2025: the quiet IPO prep

January 2026: the bold claim

February 2026: the panic

Also February 2026: the Blind reality

What the data says

Where this actually lands

Footnotes

Development processes in the GenAI era

Stateless vs Stateful

What changed and what did not

Process matters more under higher speed

Code review and validation

Traceability, task switching, and state

Multi-agent workflows

YAML-based epic and feature state

Cross-model consistency and shared state

Costs and constraints

Today I learned... git shallow

Dealing with ThreadSanitizer Fails on Startup

A Minimal Possible Example

Will non-PIE Binaries Help?

What Actually Goes Wrong: TSan’s Shadow Memory Layout

Disabling ASLR for the Test Process

But What is about Docker?

Why Upgrading the Toolchain Solves the Problem

Footnotes

Cross-Compiling Rust for Raspberry Pi

Test App: V4L Camera Capture

Why Not Just gcc-aarch64-linux-gnu?

Building a Custom cross Image

Configuring Rust Projects with Cross.toml

CI Strategy: Fast x86 Builds + On-Demand ARM Artifacts

What Actually Drove the Tech Layoffs

Platforms Instead of `--cpu`

Configuring Rust Projects with `Cross.toml`

Why Not Use `bazel_dep` for `rules_boost`?

Migrating from `Cargo.toml` to `BUILD.bazel`