I switched over to NixOS around a month ago from Ubuntu and it's been just a dream. I expected there to be some friction with installing things that aren't already built for NixOS, but honestly it's been easier with LLM+NixOS than it was with Ubuntu.
It's really hard to tell what the benefits of Incus over Ganeti are. The Incus site doesn't really lead with features or benefits that I can see. About the only things I could figure out are that Incus can manage containers and can manage instance creation through cloning a base volume, where ganeti (afaik) always creates volume and populates it. Well, and Incus seems to be MUCH more popular than Ganeti.
I searched the documentation but it wasn't really clear what its live migration and ZFS migration story is, but when I asked Claude to research it, it tells me that it supports live migration via ZFS snapshot replication, which is exactly what I'm looking for. I implemented a ganeti storage driver that does the same thing and am just getting ready to start testing it, but if Incus supports it I might look at moving that direction.
CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.
CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...
Doing things via the CLI and editing config files doesn't automatically give you reproducability or audit trail. Last time I check etckeeper doesn't work with proxmox either (due to the proxmox FUSE mount in /etc).
I'm a big LXD and now Incus fan. But I went with NixOS rather than IncusOS for my latest build because I prefer the LTS linux kernel over the mainline kernel.
NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.
> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
I don't if you know this, but there was a collection for LXD. Last time I checked, it didn't seem very popular so not maintained very well but it did work. Maybe that could be used for inspiration.
I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.
Another proxmox-esque project I've been watching is Sylve, a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop.
One of the developers building Sylve gave a talk last year [2].
> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm.
Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.
This seems very cool and I will probably try it, but I think I’m missing something. I run Proxmox so that I can have multiple VMs running on my NUC. This doesn’t really solve that right? I cant spin up a windows 11 vm one weekend for a random experiment.
Incus is roughly a frontend for qemu, so you can launch an empty VM and use the ISO to install the OS. You don't have to use a preconfigured base image.
VM support has been there for quite a while now and works nicely. I think that's what they're referring to. It started out with only system containers and then gained support for VMs and now there's work to support launching OCI images directly.
I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API (https://pve.proxmox.com/pve-docs/api-viewer/index.html) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory that takes some really somple config on disk, bakes a fresh qcow image, pushes it to proxmox and then configures networking and boots it. It's extremely rudimentary but has been working well for me.
I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo.
I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.
>The Agentic Multiplier
Oh man, you buried the lead there.
I switched over to NixOS around a month ago from Ubuntu and it's been just a dream. I expected there to be some friction with installing things that aren't already built for NixOS, but honestly it's been easier with LLM+NixOS than it was with Ubuntu.
It's really hard to tell what the benefits of Incus over Ganeti are. The Incus site doesn't really lead with features or benefits that I can see. About the only things I could figure out are that Incus can manage containers and can manage instance creation through cloning a base volume, where ganeti (afaik) always creates volume and populates it. Well, and Incus seems to be MUCH more popular than Ganeti.
I searched the documentation but it wasn't really clear what its live migration and ZFS migration story is, but when I asked Claude to research it, it tells me that it supports live migration via ZFS snapshot replication, which is exactly what I'm looking for. I implemented a ganeti storage driver that does the same thing and am just getting ready to start testing it, but if Incus supports it I might look at moving that direction.
Anyone use Incus live migration with ZFS?
I get the advantage of fully declarative systems such as Nix, but everything you do in proxmox via GUI can be done via CLI: https://pve.proxmox.com/pve-docs/#_command_line_interface
CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.
CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...
Doing things via the CLI and editing config files doesn't automatically give you reproducability or audit trail. Last time I check etckeeper doesn't work with proxmox either (due to the proxmox FUSE mount in /etc).
The amount of b.s. in this article is staggering.
I'm also considering migrating from Proxmox to Incus, but I'd look into IncusOS rather than having to manage the host OS myself.
I'm a big LXD and now Incus fan. But I went with NixOS rather than IncusOS for my latest build because I prefer the LTS linux kernel over the mainline kernel.
I've been using incus for a while now and actually run it on a side project in production for the better part of a year. Rock solid performance.
NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.
> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
Just use Terraform against Proxmox and don't leave mysterious things nobody else knows how to work with.
What do you mean exactly? You have a nix workstation and a physically separate PVE server, and configure the second from the first?
Incus is great. I've been trying to revive an unmaintained ansible collection to manage incus resources https://github.com/sbstp/ansible-collection-incus
I don't if you know this, but there was a collection for LXD. Last time I checked, it didn't seem very popular so not maintained very well but it did work. Maybe that could be used for inspiration.
I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.
Another proxmox-esque project I've been watching is Sylve, a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop.
One of the developers building Sylve gave a talk last year [2].
[1]: https://sylve.io/
[2]: https://youtu.be/wo4oD5UON30
> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm.
Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.
This seems very cool and I will probably try it, but I think I’m missing something. I run Proxmox so that I can have multiple VMs running on my NUC. This doesn’t really solve that right? I cant spin up a windows 11 vm one weekend for a random experiment.
It sounds like all his containers are Linux, so that's why Incus is such a good fit for him. For your use case, yeah, proxmox is likely a better fit.
This small project makes running Windows on Incus a breeze: https://github.com/antifob/incus-windows
Incus is roughly a frontend for qemu, so you can launch an empty VM and use the ISO to install the OS. You don't have to use a preconfigured base image.
I think incus can be a frontend for qemu, but it's primary mode of operating is to run containers. It's a fork of LXD.
VM support has been there for quite a while now and works nicely. I think that's what they're referring to. It started out with only system containers and then gained support for VMs and now there's work to support launching OCI images directly.
I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API (https://pve.proxmox.com/pve-docs/api-viewer/index.html) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory that takes some really somple config on disk, bakes a fresh qcow image, pushes it to proxmox and then configures networking and boots it. It's extremely rudimentary but has been working well for me.
I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo.
I yanked vmfactory out and into a standalone repo if anyone is interested: https://github.com/whalesalad/vmfactory
https://github.com/Telmate/terraform-provider-proxmox ...
I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.