How I Enhanced My Workplace's System Backup to The Next Level
Backups are old friends for us system administrators. They're always there, ready to help us on a bad time.
Just like security, sometimes backups are treated as something that's not important. Some sysadmins even think that simply just adding RAID to the server is enough to prevent any data lost due to hardware failure. Some even don't do backups at all (trust me, I know some sysadmin and some company did that).
That's a big misconseptions. In lots of case, we need to restore the backup NOT because of hardware error or some astronomical disaster -- BUT human error. Human are prone to do something silly, like accidentaly removing a file in the server -- thanks to SSH and rm command.
So at a fine day, I discovered the awfulness of our workplace's backup system -- and thought; This needs to change.
The old method
My workplace's backup is kinda vanilla. It's just a script that compressed some folders, and then copy them to the NAS.
Just in case you don't know, NAS is Network Attached Storage. It is basically a disk that can be accessed over network. It's a separated machine.
Our web servers contained a lot of things from all user around our company. They're mainly text data, but sometimes pictures and files too. So imagine the script compressed 1TB files into 500GB, then copied to the NAS yesterday. Today, the script will be running again, and adding 500GB again to the NAS.
The NAS storage will eventually ran out of free space at best after creating maybe a week of backup. Then a user came out to me, asked 'can you give me the backup of my file two weeks ago? I didn't realize I corrupted the docx file before I went to vacation two weeks ago'.
Kaboom, no solution.
The realization
That problem made me realize that the majority of users didn't touch their file often (well, maybe me too). They usually changed it, and only realized that it was wrong after some milenia. Or maybe, the changes are just pretty rare.
So it came into my brain that the current backup system only kept the copy of their file -- and ironicaly they are in vain, because changes are not made in a short timeline. After googled a little, I found the word; It's deduplication.
There're just deduplication anywhere. A single mp4 file duplicated into same 7 files (a week) and they didn't serve any purpose at all; just eating resources of my NAS like cancer.
So, how to solve this?
Snapshots
Why remake a house if you only need to remove one room. Why replacing the whole file, if only the last change destroyed it?
That's where snapshots came in.
Luckily, my NAS have the capability of creating snapshots automatically every some time -- depends on the setting. It only capture the delta or the difference between the last snapshot-ed file and the current file.
So if I had a 1GB video, edited it into 1.2GB video, the system will only read the change -- the 0.2GB. Therefore, the backup will took less -- much less storage, than copying them whole (1GB + 1.2GB vs 1.2GB).
It excited me, but snap back to reality; the partition of the NAS was incorrectly created, therefore there's no space left for the snapshots.
So I reformatted it, of course there was a big downtime, but after years later I am truly glad I did that.
Incremental Backups
How about the standard-native-linux servers that doesn't have snapshot feature? Actually, while things like btrfs-snapshot exist, sometimes it's not a go-to solution. Especially in the old school servers that are just 'dont touch it or it will go broke and all hell will loose'.
So we need a simple application that can detect the delta / incremental change between yesterday and today. No need to reinvent the whell, it is already there. The mighty Restic.
But out of the box, Restic are not enough. I need to define things in my old backup script like when to backup, where to backup, mounting NAS, how many retentions, and send the telegram notification when the script goes wrong.
So I made the bash script to automate them here. Of course, it's also vanila, I planned to make it more complicated and with feature later.
Happy ending
After changing all the backup methods in all servers, I went from only keeping a week of backup into 365 days (a year) AND with reduced storage used.
This was a groundbreaking for us, and quickly become a new standard.
So it came in my mind; The wheels are already out there, we didn't have to do the hard work reinventing the wheel, just use brains and lots of nice tools and lots of our heavy burden are lifted. But a lot of workers just too narrow minded and too busy with their hard work (that can be made easier). Pretty ironic.
Work smart,don't work hard.