Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.
I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.
That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.
Exactly this. I was thinking of making a similar comment but you made it far better than I could.
Number of binaries is kind of a meaningless metric, especially for a system that historically follows the UNIX philosphy of each program doing one thing.
Sure, a shell is complicated and a potential risk, and perhaps it's a good idea to exclude from the base system in this context.
But I'd rather have ls, tr and wc on my system than some bespoke, all-encompassing API service that has been far less battle tested providing similar functionality.
And like you rightly pointed out, these new binaries all contain their own list of dependencies which are pulled in at build time and need to be taken into scope as well.
That's not to say Talos or its approach doesn't hold merit, but I think it's a little disengenious to simply point at the number of binaries.
I agree number of binaries is an arbitrary metric but also an indicator that things work differently with Talos. You have to use the declarative API for management which some people could see as a bad thing.
I’d also like to point out that the system API is designed to be extendable and adaptable to different operating systems. We’d love for more vendors to create adapters/shims to get the benefits of API managed Linux
SystemD bashing aside :p Talos is pretty awesome for setting up clusters. At home I just run talos with matchbox for PXE bootstrapping it works like a charm. Been really easy to maintain too. I normally just update matchbox and then reset a machine at a time with talos ctl for a clean install. It's something very reassuring with completely reset your machines so you know you could reinstall or replace them easily.
Granted just used in a home setting running smaller workloads for backups, private projects, git etc.
The /sbin/init binary is hard linked to /sbin/dashboard, /sbin/poweroff, /sbin/shutdown, and /sbin/wrapperd. While this technically is 5 files, it’s a single file hard linked 4 times to provide convenience commands.
Err, that's definitely 1 file with 5 directory entries, not 5 files.
Hard links only work in the same filesystem. If your system only has one volume with one filesystem (e.g. a PC) that's fine, but not a very portable option for servers.
Only if its the last hard link. If nothing else, its a wee bit of insurance from deletion since no single link removal should remove the file.
Anecdote, eons ago, we had a problem where the vendor needed to log in to the machine with the intent that they were going to upload some utilities, fix a problem, and then delete them.
Before I let them in, I set up a script that constantly scanned the directory tree they were in, and hard linked everything so I could look at what they were using later.
That's a problem with directory hardlinks: a recursive deletion will wipe the directory (in the same way that truncate will destroy a file regardless of how many hardlinks it has). But we don't usually allow directory hardlinks these days.
that isn't generally how hardlinks work: you need to delete all instances of it to delete the file, as opposed to having one 'real' file like with symlinks.
There's a lot of benefits actually, as symbolic links often need to be handled as a completely different type of file with different semantics which often leads to bugs. Hard links are always better when you know you are dealing with a single static host & filesystem. Symbolic links in such a case are only better for indicating quickly to the user which files are linked to which others.
Big fan of Talos, have used it in some homelab + cloud clusters over the years, currently powers all my self-hosting. The `talosctl` command is great, and any time you need to do node-level debugging, there's always something like node-shell [1].
I wonder how a really slimmed down distro like Alpine would compare here, particularly in terms of image size.
It offers most of the standard Linux utilities we know and love, but most of them are actually just symlinks to Busybox, which is ~900K on my (ARM64) system. That's less than a hello world in Go, for a program that can replace most common Linux utilities in daily usage.
We operate talos and alpine based nodes, many thousands of them. The build chain for alpine is many orders of magnitude more complex than the build pipe for our talos image modifications. Alpine is really not made for doing a lot of “host” tasks and needs much coercion to get it to be capable of running something like k3s, and much more complex to get kubeadm clusters running on it. In the end the complexity is required for flexibility, alpine nodes can be modified on a whim, talos is R/O and ephemeral, but more secure.
The benefit of Talos isn't low binary count but that _can_ reduce the amount of maintenance required.
The benefit is declarative API driven management. You spend less time automating a system to a desired state in a similar vein Kubernetes provides a declarative API.
What does any of this have to do with my question? You just restated a fact, that busybox is a binary with a lot of symlinks, but why? So what? Yes, also the sky is blue and water is wet. It's like you didn't read the comment you responded to. It wasn't even about the benefit of Talos.
It's disingenuous to say that /sbin/init (machined/main.go) is less than 400 lines of code. Sure, that file is. What about all of the in-tree modules that are being imported? A super lazy summing of Go lines in the master branch of the repo:
$ find . -name *.go | xargs wc -l | tail -1
354085 total
Heck, there are almost 100k lines under internal/app!
Kind of like TextAdept claiming that text editor is "just 2K lines of C/C++ and 4K lines of Lua", when it's in fact wrapping a well known third party text editor widget.
At least here it makes more sense, since it's not like "our init system is just 200 lines" when those lines are wrapping a third party init system library.
It's more like "our init system logic is just 200 lines, not including third party dependencies". That's legit, provided that those deps are stuff like parsers, some library for dealing with strings or running processes, and things like that.
I know there are a lot more lines and I didn’t count any of the imports from systemd either. 300 loc (machined) vs 3000 loc (systemd) was the closest comparison I could think of without crawling all imports and deps.
Would be happy to update with a different comparison you think is more fair.
There's work in a new kublet replacement that moves things that would normally go into daemonset into systemd (or something like systemd).
There's also a neat feature of podman that runs pods as systemd units, which is a nice intermediate step between a more traditional pet server and a full kubernetes cluster.
it could have been said without the x lines of code comment.
"lines of code" is so often used as a "disparagement" about software rather than a metric for understandability.
Something along the lines of "...not needing a general purpose init system that integrates with logging, network and mounting, when all we are running is Kubernetes."
The second I learned how to write systems unit files was the second I evicted initv/rc.d scripts from my mind. Well, okay, from my search terms at least ;-)
It can even kind of replace cron with timers, and no more mucking with grub. Also, true parallel init tasks. Love it.
Many of the binaries you see in the demo video are showing processes running from inside containers. There will be a lot more processes once you start pulling containers and starting containerized services.
Notably the kubelet is also missing from the list because it's not built into the OS but pulled as needed from the correct version of Kubernetes requested.
Bottlerocket runs systemd and also runs 2 versions of containerd. One for the system and one for workloads. This (in theory) hardens the OS more, but in practice makes things extremely annoying to manage because you have to get a shell on the host to access the API.
disclaimer, I used to work at AWS on EKS and closely with the Bottlerocket team.
In retrospect, it would've saved a lot of trouble and misunderstandings if systemd had called the init daemon "systemd-init" to make it clear that not literally everything that is under the umbrella is part of the init daemon.
It's not the other way around though, which is a very important distinction. You don't need to use systemd-networkd or systemd-resolved or any manner of other things just to use the init daemon. The init daemon itself is extremely useful, and there are many machines that use the init daemon without most of the other services under the umbrella.
It makes sense that a lot of the other services in systemd depend on the init daemon, it provides a lot of baseline services and features that are used for the rest of it. As a matter of fact, I don't even know what other init daemon I would choose if I wanted similar features around system daemon management, as there's a lot in the surface area that is genuinely useful. Honestly, there's a lot of useful stuff for handling secrets, handling UNIX domain sockets, temporary files, sandboxing apps, setting resource limits, managing unit lifecycles, etc. There are a few features I find somewhat more dubious (personally I'm not sold on DynamicUsers) but by and large I actually like a lot of the surface area systemd's init daemon provides and if I were to use something else I'd want something in a similar ballpark.
20 years ago, I used to make custom Linux distros for fun. Floppy distros, CDROM distros, RAM-resident distros, network-boot distros. In a few of them, I custom-made my own binary that was both the init system, and a few applications, stripped it down, and shipped just that as the distro (basically just a few files and my static binary).
A lot of people downloaded them, and it was great fun - to start. Problem is when you want to do more things. You have to start finding workarounds to bolt-on additional tools, or maybe you just throw one or two extra tools in there by default. Over time you find more and more missing things or incompatibilities with other systems, which make it harder to cover more use cases. And finally you realize that "the tiniest system" is a lot more effort than it's worth, and what you really want is "a slim yet compatible system". The system you end up with is a lot fatter, but a lot less headache.
(The security benefits of fewer files are overblown, too. If you audit and harden the system, it doesn't matter how many binaries you have, because the attack vectors they use will be mitigated)
How is that different than adding binaries, in the context of this comment where the point was "In the end you end up just adding everything back in that you originally took out, because managing all the little weird different subsets is not worth the benefit."
In the context of that assertion, adding or subtracting "extensions" and adding or subtracting binaries are equivalent. Both are adding or subtracting "piece of code that provides fuction".
They are similar but extensions don't have to be binaries. It can also be files you need to be available before Kubernetes starts.
Talos is purpose built to run Kubernetes workloads and not general purpose Linux. Hopefully, you don't have to add _everything_ back to the OS, but we know some things cannot run as a container or Kubernetes workload.
Extensions are required for specialized hardware (eg network, GPUs) and is the closest thing to a "package manager" available in Talos. Extensions can be binaries but don't have to be. We have a lot of common extensions provided and maintained by us but anyone can create extensions as needed.
One nice thing about extensions is they get layered and you don't have to pre-build an artifact like you do with other Linux distros with something like packer. factory.talos.dev will let you pick you extensions and get an artifact no packer/bash/config management required.
Like the other commenter said, this is the kind of thing custom distros always have to do, that lead to either reinventing or just going back to a big fat distro. I've gone around this wheel a couple times. It's a nice idea to have something custom-tailored, but a tailored suit only fits one person, and is expensive.
You can still build a system to manage Kubernetes nodes without making your own distro. Even a heavily modified stock distro gains you benefits from basing off someone else's work. You can reuse the solutions they've made, and contribute back your customizations for your specific use cases.
That's how today's distro installers/package managers/etc came to include all the functionality they have. You couldn't eject a CDROM from a Busybox system, until one weird kid in high school decided to try to use Busybox to make a CD-bootable RAM-resident distro, found out it had no 'eject' command, and then sent a patch in to Busybox to add it. Now everyone can use that command, and that functionality is still there 20 years later.
It's also a lot harder for users to use proprietary solutions than ones they're familiar with. Your OS has no shell or console, only an API? So if there's a problem, how do I drop in with gdb, strace, tcpdump, and the entire suite of Linux debugging tools, to try and quickly diagnose and then patch an issue? I'm sure you've created some way to do it, but now I need to go find out how to do it, and probably use whatever stock tools are there, which may have their own quirks or incompatibilities.
But I get that a corporation's interest is mostly in "get something working now" as opposed to "get something working that will be better in the long run". DIY/NIH often becomes the engineering department's watchwords, and a custom distro is one of those eventualities.
We run many talos clusters at Civo, and they are far easier to manage than the other cluster types that use a standard Linux distribution stripped and stuffed with what we need, and the custom image build process is easy to get running in CI, all in all talos is wonderful both as tenant nodes and our region supercluster nodes, and much simpler process to add/remove nodes to the pool and do a few other k8s-centric tasks like etcd snapshot backups and pre-configuration of our regions before we have kit on the ground.
MX Linux would be that "slim-yet-compatible" system. It can load itself completely into RAM, and has many distro-specific components such as Frugal Install and Remastering, yet also lets you install standard Debian packages.
With Kubernetes you can schedule workloads in a number of different ways. Let’s say you insisted on having a shell and package manager. Run your favorite distro’s container as a DaemonSet. With the proper mounts and permissions you can do a lot. In other words use Kubernetes to do the things need to do. Then what role does the OS really play? Well in the case of Talos it’s only there to run Kubernetes.
You literally do not need more than the most basic of host-OS facilities to run any Linux workload you want in a pod, the OS just needs to run containerd and have a kernel, talos adds the management plane to that mix to make it useable, but the userland provided by the oci image will not see any difference at all than if it were running on Ubuntu; it will see the kernel.
You use machineConfigs that are used to provision the base OS and configure it, and the clusterConfig is used to bootstrap k8s on those machines. You can make subtypes and super types, you can have different networking setups, whatever you like, just apply and the OS is driven to state, then k8s is brought up from there. You are presented a kubeconfig after. Changes are done via application of updated machineConfig. Works great in practices and if you write an operator you can manage the config generation via k8s manifests and get wild with it.
I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.
That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.
[1]: https://github.com/siderolabs/talos/blob/main/internal/app/m...
[2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list