Over the years, I have used a few different backup solutions. Each solution had it’s own perks and it’s own downsides. I started off with Rsync, followed by back blaze, Unitrends, Veeam, carbonite, back to Veeam, then Veeam again. So, I have a large amount of Veeam experience. Each backup solution gave me an understanding, along with each environment that I like to pass down. I’m going to be focusing on Veeam Backups the most.
Standards
3-2-1-1-0
Veeam is unique in how it holds it’s standards for backups. If you have ever worked with Backups you know the 3-2-1 or the 3-2-1-0 options. Veeam uses the 3-2-1-1-0 options. It’s known as the Veeam Golden Backup Rule.
- 3: Maintain at least 3 copies of your data
- 2: Store the backups on 2 different medias
- 1: Store at least 1 of these copies at an offsite location
- 1: Store at least 1 of these copies offline
- 0: Be sure to have a verified backup copy without errors.
Tiering
Having your servers organized into tiers allows you to know what is important. For example, While working with Unitrends, we had 7 tiers. Each tier was assigned it’s own Recovery point Objective (RPO). The tier 7 was the highest RPO while 1 was the lowest. The core apps were set to 1. For that company it was the AD server, a core app server and it’s database server, along with a Custom script server. They kept those at the lowest because everything else could be built back on top of it. Thier highest was remote desktop terminal servers from remote locations.
The Recovery Point Objective (RPO) defines the maximum amount of data that an organization can afford to lose in the event of a disaster. It is typically measured in time, such as minutes or hours, and indicates how often data should be backed up or replicated. For example, if an organization has an RPO of 1 hour, it means that in the event of a failure, the organization can tolerate losing up to one hour’s worth of data.
– Geek For Geeks
Tiering is also effected by the Recovery time Objective (RTO). Basicly, how long does it take to recover. One of the Veeam company’s I worked with the remote locations were more important than the main location because of the security. However, it took a long time to pass over 5 TB over 4G cellular signals. So we had to build out the solution based on the RTO of that location. So things get unique.
Levels
I have seen different reasons to do different things. From data corruption to ransomware, backups helps protect each level. Lets look at each one with the 3-2-1-1-0.
Data Corruption
So, imagine you have a system engineer who is great with many things but sucks at DBA stuff. Let’s be real here, no engineer knows everything. Surprisingly, I’m good with Meraki, Fortinet, and untangled firewalls, but I suck at sonic wall and cisco firewalls. This engineer gets a ticket about clearing out the logs for a server. Being very surface level, the engineer uses their knowledge and goes in and removes the logs. Wait, those logs were tranaction logs for a SQL database. Well, backups are needed at this point. This is where Disk backups comes into play. Think of the 2 in our 32110. Lots of companies have a disk backup and a cloud backup. Some have tape backups. For data corruption, restoring from the disk, is very important because the RTO needs are high. So, restoring from a tape (300mbps at max) isn’t really logical for something here and now. These kinds of backups are normally within a hour/day kind of deal. Going into the backups and restoring the server from an hour before the error, we should be good to go.
Environmental
You are at home, and a bad storm comes through. You go to check out the datacenter and, a small car has became one with your server racks. This is where the off sites come into play. We are assuming you have enough hardware else where to bring the system backup and running. If not, then it’s time to play musical chairs with your tier levels. Good luck. I have been there. This becomes a complicated, what is the company willing to not have right now. How long will it take to get new hardware? Those things should be discussed before hand. Companies like Unitrends offers who data stacks, but others like carbonite, Veeam, etc don’t. When we used Unitrends, we did a proof of concept for the board members. Unitrends flew out a datacenter to our DR site. It was mind boggling. So, if you have a DR site, yay, if not, environmental is by far the scariest.
Cyber
Ransomware… You get to work and you can’t log in because the system is encrypted. This is where the Offline backup comes into play. Along with the does it work. Being offline, the backup can’t be effected. One of the sites I worked with at my MSP time used tape. They pushed things to tape and pulled those tapes every week. This saved them when they did get ransomware. It took them a week to recovery as they had to find it, but that tape stack was wormed and ready to go. They used Veeam Backups to make these tapes happen.
Reality
Now we have established some of the best ideas to follow, lets look at reality. Each solution looks different for each company. Veeam Backups are my preferred backups because I know them the best. The reality is, some company work better with something as simple as PowerShell and resync. I’m going to give some examples I have seen over the years. AKA story time.
Solar Company
I worked with a solar panel company. They used next Cloud to do all of their management along with a open-source email system that all was hosted in their cloud server. Thier provider provided a barebone backup system once a week. They only needed something locally. So, what we did was setup a rsync. It wasn’t anything special, but it was what they needed. The barebone backup from their provider provided them what they needed if they had a ransom ware. He wanted to be able to go to a folder on his synology and get the files from the day before. It worked, and he was happy. Was it ideal, nope, was it a Veeam Backups 3-2-1-1-0, not really. But he was ok with being behind a week. He also printed a lot of his stuff too for legal purposes.
Bank
This one was the Unitrends masterpiece. They had the 3-2-1-1-0 down pat. They had local data backing up every hour on the hour to the disk, and having Fulls each day to the same disk. Then it went up to Unitrends every hour as well for the off site. Unitrends did the rest for them. Unitrends handled all of the other items. They even went as far as spined up 10% of each tier we had and we would test it. Along with that, we had sentinel one on all the computers. So when ransom ware hit Sentinel One stopped it from spreading. That one server was restored from a disk backup.
Fuel Company
The fuel company was a unique store. We used Veeam Backups for this one. We had the Veeam system backup each remote site to a local Synology. Then the local Synology moved the backups to the main office. Once at the main office, the main office Synology moved it to the back blaze cloud. We did it like this because of the throttled 4g/5g networks. Taking it to the Synology, we had the backups compressed, and de-dupped. So we had less data to bring back. So, we got a weekly syn full each week to a cloud setting that was protected. Once a year, we took a 8tb hard drive, and loaded active fulls. That hard drive was placed in a lock box at the bank.
Manufacturing
This was the worst setup we had. It was server by server, no tiers, no idea where things were. They did this because each department maintained their own servers. The politics ate away at this company. So we did our best. We had each department doing it’s own thing, but we made sure we had backups at least once a week. But the company didn’t want to spend money on a secondary backup nor a cyber protection.
Education
A friend of mine told me about how they did things. They had 200tb of data (active fulls) but only 100tb of storage. But they had tapes. Lots of tapes. So, what they did is staged backups. They kept their tiers 1 and 2 on the disk, and kept enough space for 3 and 4 to load to disk then load to tape. Their tier 3 and 4 was roughly 140tb of space by itself. Thus, they broke it up. Is it the best way to do it, no, but is it a way to keep them safe from ransomware? Yes. The org had to understand it will take time to restore. But that was ok. It was a Veeam Backups kind of setup too. Thus, they couldn’t do the 3, but they got the 2 1 1 and 0 with the structure.
Reality checks
If you don’t have space, if you don’t have company approval, if you don’t have xyz. It becomes a game to make it happen. How do you fit everything into the package you need. It takes time and thought. The hardest to work with is if the company as a whole doesn’t care. When I was at the manufacturing, the feeling was, ok, I will just rebuild it. Which never happened. Each puzzle will take time to figure out. So don’t get frustrated, instead enjoy the puzzle.
What can we learn as a person
Everything worth while takes time and effort. More you put into it, the more it shows. Sometimes, we just need to breath. You got this, keep up the good work. Don’t stop when you get close to the end. A great person once told me, 83% is where most people reach and then we switch to something else. Lets make it to 99%.