I've been using Nix for "embedded" systems for few years now and it works fine. I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Like the article points out, the software stacks people use for embedded devices are the same as people use non-embedded use; Rust, Go, NodeJS, and sure still C++. The only real difference with embedded devices is non-OS components like the bootloader (u-boot, EDK2) and customizing the device tree. (And of course firmware flashing). Fundamentally those are all just packages that I can describe in Nix. I don't need a separate tool just because the board is small.
IMO the embedded space, especially in the US, is already pretty Niche. Most companies either just ship the vendors BSP example (Ubuntu/Debian/Yocto) and pay very little attention to the detail or re-useablity. Once you vendor declares the BSP EOL you are stuck unless you re-implement it yourself.
Using Nix (or Guix) has the massive advantage of a large and active community that isn't fractured like the Yoctoo/Buildroot community. (By fractured I mean there may by many, many people using those tools at $DAYJOB but due to vendor BSP customization they likely share much less with the upstream community maintained sources).
I share some of the same observations that seem to have motivated this project, but my solution was to just use Alpine on the same architecture as the target (possibly via qemu) and then export a subset of it to a filesystem image. I implemented it as a perl module collection with no dependencies other than core perl modules, and then run the export from within the image itself. Among other things, it lets me use strace to find the runtime library dependencies of the things that need to be in the image.
Modern SBCs are just normal computers and not "embedded" in the traditional sense. You can generally just use Debian, and spend time on the actual project, instead of wrestling with the system
For more mainstream SBCs, totally agree. However a lot of the niche industry boards (especially new ones) only support Yocto out of the gate. Debian might be "supported" by the SoM/CPU vendor but manually reapplying kernel patches and praying it boots is a PITA.
I agree. In the past, I have successfully used Debian and Alpine for embedded. Never needed to compile OS kernels or standard DLLs, other people already did and published in these package repositories.
I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.
Adding new targets is deceptively easy, just copy an existing template and substitute your values.
The problems building with Linux and GNU environments exist because they were terribly designed with assumptions like
- You're building on the same native system as GNU and Linux packages, you install them globally in the same places that servers and desktops use
- Your C, C++ compiler and entire toolchain and other binary utilities with the kernel is a one single unit that you can only change one part at a time
- You use the same up to date headers with glibc, gcc and Linux kernel
- You're building software in the same universe of all the other packages, especially gcc libraries (libgcc_s, libstdc++), glibc (especially bad since ld-linux.so is part of it)
- The build system only uses standard paths
The reason Yocto is so complicated is that developing in a Linux environment actually sucks when you're not writing web-oriented or server / VM software. Yocto fixes it. It introduces a good set of abstractions that work around terrible design decisions that were made in overall Linux ecosystem. There are a lot because the OS design is fundamentally broken, especially with C-based toolchains which is 99.999% of the ecosystem. Current C toolchains including MSVC strongly ties OS with the C's internal types and bad decisions of 70s.
As always all articles whose title asks a question are answered with NO, 99% time. By taking away the cross-compiling abilities and the workarounds doesn't fix the brokenness of Linux and overall FOSS ecosystem.
If you're looking for how a better embedded environment looks like, look at Rust toolchains. For Linux take a look at musl-libc based ones (you 100% need a systemd distro to get away from nss complexities that musl introduces). Or even better take a look at relibc. There are barely any assumptions about the target filesystem and tooling in Rust toolchains, unlike C/C++/Make toolchains. There is redox OS but it is still in slow development and they stuck with Make, which I think was a bad decision. Android uses its own build tooling but cannot run away from C/C++ tooling unless Google revives Fuschia.
I haven't used it, but there's also Avocado OS [1]. Myself, I use Yocto for our embedded build. It's not perfect but for our application, which involves building a bunch of software that uses both Swift and Flutter for an embedded audio product running on a RPi CM4, it works just fine. Shout out to Mender for A/B updates.
I've used buildroot extensively, and an occasional yockto.
My impression in recent years is that these image cross build environments are just not as frequently needed as they were back in the day of their invention.
My most recent embedded linux environments were just embedded archlinux.
No need to cross build an image, just install and run the minimized linux environment right on the target.
Of course, a big part of the need for these cross-tools is that it seems most modern embedded linux developers are running windoze on their development workstations 8-/
Are you proposing compiling on the target? For a vast number of embedded systems that is not only impossible (insufficient disk and memory) but also incredibly slow.
At the end of the day you need some cross compilation just for board bring up.
If you're playing with some platform for which this has already been done, then sure, but that's not really the "normal" way of doing embedded.
Embedded just means ARM 99% of the time and it's cheaper and easier to use native ARM servers (AWS has them cheap) than to make 100% of software cross compile. Some parts of the firmware might need to be cross compiled but those projects are designed to cross compile.
I’ve been experimenting with what a next-generation embedded Linux build system might look like: native builds on the target architecture, modern language package managers as first-class citizens, and AI as a primary interface to the system.
Instead of cross-compiling with a large meta-layer stack, the tool builds kernel, rootfs, and applications together using one engine, with a CLI, TUI, and AI assistant talking to the same core. All you need is the tool, Docker, and Git — no global SDKs or hidden state.
It’s pre-1.0 and rough around the edges; I’m sharing it early to get feedback from people who live in Yocto/OpenEmbedded, Buildroot, Nix, etc. I’d love to hear where this breaks on your boards, what workflows feel wrong, and whether the “native builds + AI-aware build graph” direction seems promising.
Armbian and Qemu worked well for me when I needed to compile packages for an orange pi without enough RAM to actually run cargo build. Built the image on an emulator with more RAM then the target system, dropped the customized image on the SD card and booted right into the entry script.
I wish you luck! The pain points you identified are definitely real and solving them would be valuable.
The workflow for user space can definitely improve some of this pain but I feel like a large portion of any embedded Linux development effort still ends up in the weeds for boot related items (secure boot, proper updates, nuanced kernel patches, bootloaders, device trees, and supporting machine variants, etc). Solving those to make them easy is a hard problem for sure.
I've been using Nix for "embedded" systems for few years now and it works fine. I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Like the article points out, the software stacks people use for embedded devices are the same as people use non-embedded use; Rust, Go, NodeJS, and sure still C++. The only real difference with embedded devices is non-OS components like the bootloader (u-boot, EDK2) and customizing the device tree. (And of course firmware flashing). Fundamentally those are all just packages that I can describe in Nix. I don't need a separate tool just because the board is small.
IMO the embedded space, especially in the US, is already pretty Niche. Most companies either just ship the vendors BSP example (Ubuntu/Debian/Yocto) and pay very little attention to the detail or re-useablity. Once you vendor declares the BSP EOL you are stuck unless you re-implement it yourself.
Using Nix (or Guix) has the massive advantage of a large and active community that isn't fractured like the Yoctoo/Buildroot community. (By fractured I mean there may by many, many people using those tools at $DAYJOB but due to vendor BSP customization they likely share much less with the upstream community maintained sources).
I share some of the same observations that seem to have motivated this project, but my solution was to just use Alpine on the same architecture as the target (possibly via qemu) and then export a subset of it to a filesystem image. I implemented it as a perl module collection with no dependencies other than core perl modules, and then run the export from within the image itself. Among other things, it lets me use strace to find the runtime library dependencies of the things that need to be in the image.
https://metacpan.org/pod/Sys::Export
Modern SBCs are just normal computers and not "embedded" in the traditional sense. You can generally just use Debian, and spend time on the actual project, instead of wrestling with the system
For more mainstream SBCs, totally agree. However a lot of the niche industry boards (especially new ones) only support Yocto out of the gate. Debian might be "supported" by the SoM/CPU vendor but manually reapplying kernel patches and praying it boots is a PITA.
I agree. In the past, I have successfully used Debian and Alpine for embedded. Never needed to compile OS kernels or standard DLLs, other people already did and published in these package repositories.
I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.
Adding new targets is deceptively easy, just copy an existing template and substitute your values.
https://codeberg.org/guix/guix/src/branch/master/gnu/system/...
https://codeberg.org/guix/guix/src/branch/master/gnu/bootloa...
Does guix have any advantage here over nix?
I didn't know Nix had support for embedded systems. Where are the images defined?
Lisp
The problems building with Linux and GNU environments exist because they were terribly designed with assumptions like
- You're building on the same native system as GNU and Linux packages, you install them globally in the same places that servers and desktops use
- Your C, C++ compiler and entire toolchain and other binary utilities with the kernel is a one single unit that you can only change one part at a time
- You use the same up to date headers with glibc, gcc and Linux kernel
- You're building software in the same universe of all the other packages, especially gcc libraries (libgcc_s, libstdc++), glibc (especially bad since ld-linux.so is part of it)
- The build system only uses standard paths
The reason Yocto is so complicated is that developing in a Linux environment actually sucks when you're not writing web-oriented or server / VM software. Yocto fixes it. It introduces a good set of abstractions that work around terrible design decisions that were made in overall Linux ecosystem. There are a lot because the OS design is fundamentally broken, especially with C-based toolchains which is 99.999% of the ecosystem. Current C toolchains including MSVC strongly ties OS with the C's internal types and bad decisions of 70s.
As always all articles whose title asks a question are answered with NO, 99% time. By taking away the cross-compiling abilities and the workarounds doesn't fix the brokenness of Linux and overall FOSS ecosystem.
If you're looking for how a better embedded environment looks like, look at Rust toolchains. For Linux take a look at musl-libc based ones (you 100% need a systemd distro to get away from nss complexities that musl introduces). Or even better take a look at relibc. There are barely any assumptions about the target filesystem and tooling in Rust toolchains, unlike C/C++/Make toolchains. There is redox OS but it is still in slow development and they stuck with Make, which I think was a bad decision. Android uses its own build tooling but cannot run away from C/C++ tooling unless Google revives Fuschia.
I haven't used it, but there's also Avocado OS [1]. Myself, I use Yocto for our embedded build. It's not perfect but for our application, which involves building a bunch of software that uses both Swift and Flutter for an embedded audio product running on a RPi CM4, it works just fine. Shout out to Mender for A/B updates.
[1] https://www.peridio.com/avocado-os
I've used buildroot extensively, and an occasional yockto.
My impression in recent years is that these image cross build environments are just not as frequently needed as they were back in the day of their invention.
My most recent embedded linux environments were just embedded archlinux.
No need to cross build an image, just install and run the minimized linux environment right on the target.
Of course, a big part of the need for these cross-tools is that it seems most modern embedded linux developers are running windoze on their development workstations 8-/
Are you proposing compiling on the target? For a vast number of embedded systems that is not only impossible (insufficient disk and memory) but also incredibly slow.
At the end of the day you need some cross compilation just for board bring up.
If you're playing with some platform for which this has already been done, then sure, but that's not really the "normal" way of doing embedded.
Embedded just means ARM 99% of the time and it's cheaper and easier to use native ARM servers (AWS has them cheap) than to make 100% of software cross compile. Some parts of the firmware might need to be cross compiled but those projects are designed to cross compile.
I’ve been experimenting with what a next-generation embedded Linux build system might look like: native builds on the target architecture, modern language package managers as first-class citizens, and AI as a primary interface to the system.
Instead of cross-compiling with a large meta-layer stack, the tool builds kernel, rootfs, and applications together using one engine, with a CLI, TUI, and AI assistant talking to the same core. All you need is the tool, Docker, and Git — no global SDKs or hidden state.
It’s pre-1.0 and rough around the edges; I’m sharing it early to get feedback from people who live in Yocto/OpenEmbedded, Buildroot, Nix, etc. I’d love to hear where this breaks on your boards, what workflows feel wrong, and whether the “native builds + AI-aware build graph” direction seems promising.
Armbian and Qemu worked well for me when I needed to compile packages for an orange pi without enough RAM to actually run cargo build. Built the image on an emulator with more RAM then the target system, dropped the customized image on the SD card and booted right into the entry script.
I wish you luck! The pain points you identified are definitely real and solving them would be valuable.
The workflow for user space can definitely improve some of this pain but I feel like a large portion of any embedded Linux development effort still ends up in the weeds for boot related items (secure boot, proper updates, nuanced kernel patches, bootloaders, device trees, and supporting machine variants, etc). Solving those to make them easy is a hard problem for sure.
Native builds on target can be very slow?