I am a huge fan of SmartOS. Back in the 2010s (around 2012), I was advocating its use in production at a small startup I worked. The SunOS kernel, ZFS, zero install, immutable core, convenient way to manage containers and VMs together - all of this looked great on paper, especially containers.
In reality, I ended up running almost everything in VMs. The only thing worked well natively was nginx. MongoDB, Mysql, even our php backend (some libraries) had issues, unfortunately.
A year ago, I considered SmartOS again as a home lab driver, and no success again, Linux just has better support: drivers, pci passthrough, etc... and now with containers+vm through Proxmox or anything else. You can even run a k8s+kubevirt with zfs practically out of the box as a complete overkill though.
> SmartOS is a "live OS", it is always booted via PXE, ISO, or USB Key and runs entirely from memory, allowing the local disks to be used entirely for hosting virtual machines without wasting disks for the root OS.
Does anyone know if something like this is possible with Proxmox? I've got three servers I'm thinking of setting up as a small cluster and would like to boot them from a single image instead of manually setting PVE on each. Ansible or salt is an option but that tends to degrade over time.
It's close, but there are some missing pieces I think. The way it manages storage pools would fit your use case, if you import a zpool, for example, it will scan the datasets and can figure out what zvols should be attached to which VMs..
but there's also VM config info under `/etc/pve` or something similar. I'm pretty sure that's some kind of FUSE filesystem, it's supposed to be synchronized between cluster members.. you might be able to host that externally somehow. But that'll probably take some effort.
You'll also need to figure out how to configure `/etc/network/interfaces` on boot for your network config. But that's doable.
It depends on what "this" you meant, but in general the ways of netbooting an OS are many and varied. You'd have to declare what kind of root device you ultimately want, such as root on iSCSI.
Personally, I feel that "smartOS does not support booting from a local block device like a normal, sane operating system" might be a drawback and is a peculiar thing to brag about.
What I'm looking to achieve are three identical proxmox host boxes. As soon as you finish the install you now have three snowflakes no matter how hard you try.
In the case of smartOS (which I've never used) it would seem like that is achieved in the design because the USB isn't changing. Reboot and you are back to a clean slate.
Isn't this how game arcades boot machines? They all netboot from a single image for the game you have selected? That is what it seems smartOS is doing but maybe I'm missing the point.
Home stuff was the last holdout for me, but even that has been replaced by Proxmox these days. I used SmartOS for a solid 7-8 years, though, and like it for most of that time.
I couldn't point to any one single major reason that prompted the switch - just lots of small annoyances stemming from the world expecting you to be running Linux instead of Solaris, and once you move away from zones, you lose one of the most compelling reasons for being on SmartOS
I have a personal box I keep updated running some utility zones and a couple VMs. I enjoy the tooling very much but it's so niche that I'm wary of using it for Customer engagements.
I never used Solaris in my real life but I can understand the appeal for people who did.
It was acquired by Samsung, which is notoriously bad at open source. But the reason why it quietly faded into the background wasn't that. It was that Joyent's ex Sun people had an annoying elitism that made them not care about working with the community.
I’m confused by the wording “without wasting disks for the base OS” - I wouldn’t normally consider this a “waste”, would anyone else? There are big downsides to running off of a USB key all the time unless I’m missing something
> This architecture has a variety of advantages including increased security, no need for patching, fast upgrades and recovery.
SmartOS was developed by Joyent for their cloud computing product, it's primary use case isn't desktop computing. I think the advantages mentioned above were probably a bigger factor than the disk space. I would also guess that PXE would be the standard way to boot in a datacenter, not USB.
I was intrigued by the idea that in the Manta object store you could schedule computations on the storage nodes. However I am not sure how much improvement that brings in practice. Any practical experience with this?
bcantrill gave a great talk many years ago about compute-data locality. would be nice to know if those ideas panned out for some customers, but it seems the world has by-and-large continued to schlep data back and forth.
it's too bad too. The concepts behind Manta were such a great idea. I still want tools that combine traditional unix pipes with services that can map-reduce over a big farm of hyperconverged compute/storage. I'm somewhat surprised that the kubernetes/cncf-adjacent world didn't reinvent it.
I did use it on a project, it was meh, alright? In the end the main cost of our processing wasn’t storage latency but code, and this quite arcane scheduler was a barrier too much for most of our team.
I believe it was removed shortly after i left the project..
wait this seems totally awesome? i hadnt remembered until reading the comments now that this was a joyent thing, and that somehow it has largely disappeared despite seeming like an awesome way to do all sorts of things.
In reality, I ended up running almost everything in VMs. The only thing worked well natively was nginx. MongoDB, Mysql, even our php backend (some libraries) had issues, unfortunately.
A year ago, I considered SmartOS again as a home lab driver, and no success again, Linux just has better support: drivers, pci passthrough, etc... and now with containers+vm through Proxmox or anything else. You can even run a k8s+kubevirt with zfs practically out of the box as a complete overkill though.
Code + issues are active under https://github.com/TritonDataCenter (smartos-live, illumos-joyent, triton, etc.), and docs are at https://docs.smartos.org/.
SmartOS is released every two weeks, and Triton is released every 8 weeks -- see https://www.tritondatacenter.com/downloads
And Triton object storage will have S3 support in the next release!
[edit: removed semicolon from link!]
Does anyone know if something like this is possible with Proxmox? I've got three servers I'm thinking of setting up as a small cluster and would like to boot them from a single image instead of manually setting PVE on each. Ansible or salt is an option but that tends to degrade over time.
but there's also VM config info under `/etc/pve` or something similar. I'm pretty sure that's some kind of FUSE filesystem, it's supposed to be synchronized between cluster members.. you might be able to host that externally somehow. But that'll probably take some effort.
You'll also need to figure out how to configure `/etc/network/interfaces` on boot for your network config. But that's doable.
Would be pretty neat.
Personally, I feel that "smartOS does not support booting from a local block device like a normal, sane operating system" might be a drawback and is a peculiar thing to brag about.
In the case of smartOS (which I've never used) it would seem like that is achieved in the design because the USB isn't changing. Reboot and you are back to a clean slate.
Isn't this how game arcades boot machines? They all netboot from a single image for the game you have selected? That is what it seems smartOS is doing but maybe I'm missing the point.
https://blog.kail.io/pxe-booting-on-proxmox.html
But why bother? A read-only disk image would be simpler.
Doesn't linux have that as well? https://www.kernel.org/doc/html/next/filesystems/smb/ksmbd.h...
Joyent, the company behind SmartOS, was since acquired, and I don’t usually see anyone talking about SmartOS nowadays.
Is anyone on HN using SmartOS these days?
I couldn't point to any one single major reason that prompted the switch - just lots of small annoyances stemming from the world expecting you to be running Linux instead of Solaris, and once you move away from zones, you lose one of the most compelling reasons for being on SmartOS
Are there any workloads (other than as a VM host) that run on SunOS derived OSes?
But that is the same for most server images nowdays.
What in portend is that Oxide upstreams all their work so 'traditional' users should get benefit from it too.
I never used Solaris in my real life but I can understand the appeal for people who did.
https://oxide.computer/blog/engineering-culture
[1] https://www.tritondatacenter.com/blog/a-new-chapter-begins-f...
SmartOS was developed by Joyent for their cloud computing product, it's primary use case isn't desktop computing. I think the advantages mentioned above were probably a bigger factor than the disk space. I would also guess that PXE would be the standard way to boot in a datacenter, not USB.
https://apidocs.tritondatacenter.com/manta/index.html
it's too bad too. The concepts behind Manta were such a great idea. I still want tools that combine traditional unix pipes with services that can map-reduce over a big farm of hyperconverged compute/storage. I'm somewhat surprised that the kubernetes/cncf-adjacent world didn't reinvent it.
I believe it was removed shortly after i left the project..
judging by https://doc.qubes-os.org/en/latest/_images/qubes-trust-level... it looks very linux-centered.