How I Virtualized My Workplace's Baremetal Servers

It is kinda rude to talk about your friend's home when you visit them.

But hey, is it the same principle for company? I guess not.

When I entered my workplace's data center, I came in awe. There're a lot of servers, stacking each other with beautiful blinking leds and sound of the fans. Until I came into realization -- that many servers only handed by two persons?

Like wow, I have stacked jobdesks already; network, servers, and cybersecurity. Having to handle this many servers gonna cut a lot of time, even maybe two persons were not enough!


 So I said hell no, I googled a lot and do a lot of research to solve this problem. I found lots of ways to minimize my workload, but in a nutshell it's divided into 2 big categories, hardware and software.

At that time, optimizing software was really confusing for me. Well, I didn't have a lot of fundamentals knowledge to begin with. All I have in my arsenal pretty much self-taught thanks to Google.

So I chose to optimize the hardware. How to reduce hardware but keep anything inside it? Yep, virtualization.

 

Eliminating old thoughts

Of course at first my idea was kinda being softly rejected. It's not without a reason, because my company's system already ran from a long time ago and maybe have a 'if it's not broken don't fix it'. So baremetal always the first option to go, without ever considering moving to virtualized servers.

 

But I pushed my will to them and it luckily it worked. Some servers started to move from baremetal to virtualized.

At first only one, then came two, and until today -- already around 60% already moved to virtualized.

Why only 60%? It's complicated. Some servers attached to hardware (like usb key for licensing, and I can't (yet) mapped it to virtual), and the other just they are my co-worker's responsibility. Which means, it's up to him to create the server in virtualized environment or baremetal, and yes he is more old-styled than me so it's inevitable. But that's fine.


Technology used

Before going prod of course I tried some; Xen Server, VMWare, Virtualbox, and Proxmox. I listed all the benefits and downsides then compare all of them.

I won't share the comparison here because it's already 4 years ago -- which can be pretty outdated.

But at that time, Xen server was pretty strange (and clunky), VMWare was too costly, Virtualbox was a bit buggy, so finally we used Proxmox.

At first I only installed Proxmox on default and without any funny configurations. But years after years studying later I installed it with a bit more configuration like using ZFS and clustering. Will write about this later.


How it happened

At that time I moved my servers using Clonezilla or simply reinstalling them from scratch. Yes, I didn't know any fancy tools like Helm or Ansible to automate them, and what made it worse was that our system built without that.

There're just simply pdf lying around our NAS, filled with instructions to reinstall them. And of course, it's not updated.

So it's up to me to build the automation first, to install into VM or just went without that. It's kinda overwhelming for me, and I think it's more important to made a change first, optimize can be done later -- one step each -- so I used either Clonezilla or reinstall from scratch.

It's kinda messy, some errors here and there, but that really taught me a lot about every servers. There's always a good thing to pick up isn't it?

The benefits

There're just many benefits other than reducing hardware count to maintenance, like:

  • Centralized dashboard to control many servers at once, even Proxmox gave a nice performance dashboard also
  • Easy networking for the VMs (just create a trunk on the switch and set the tag on the Proxmox, voila easily created servers with different VLANs)
  • Downtime was a lot reduced (because hard-resetting VMs won't need to enter the slow baremetal BIOS phase (thanks to the RAID cards and other peripheral)). Even faster because of the reduced harddisk size for clean aftercrash
  • Easier backup and restore -- I think this is the biggest benefits. Cloning servers now as easy as clicking a button.
  • Etc etc, which made me fully become a fan of virtualized servers.


 

The downsides

Maybe for me it's only the reduced performance. Of course it's faster to go baremetal, until now I tried to research how to minimize this. But even though the reduced performance are not that big, it's enough to become a reason for our team to decide some servers just can't go virtualized yet.


How it is today

Today it's pretty much being a new standard and normal thing to create a server on virtual, not on a baremetal anymore. Hell, how the table flipped.

Our datacenter's rack usage also reduced, the amount of servers increasing a lot this 4 years, but the total of the machine are almost the same.

It felt nice to reduce the cost also electricity, which can be pretty nice for environment also.


 

What's next

The next step in my mind is to do all over again but this time it's containerization. I already have a lot in my mind, but just doesn't really have a lot of time to do the research. Finger crossed it will be the next big project that I will do.