Hello, my name is Jim Russ. I am an Enterprise Solution Architect and Vice President of Enterprise Technology at Nth Generation. I’ve been working in IT in one form or another for almost 40 years. Throughout my time in IT I have seen many things. Whether it be standing in a data center during an earthquake and getting out just before the entire data center was flattened, to helping a company recover from a major flood, and many other natural or man-made disasters.
The most important lesson I’ve learned in my years in IT, is that protecting company data should be the number one priority. Data is the life blood of an organization. When you lose data or lose access to data, you can lose money, and in some cases, people may lose their lives.
With that knowledge I have made it a matter of importance in my life to study the best ways to provide business continuance, high availability, and in worse-case scenarios, disaster recovery. I’ve presented at several technology conferences over the years due to my past experiences in this area. Typically, the topic has been “Business Continuance and Disaster Recovery”.
I like to start these discussions by asking the audience, “How many of you have a Disaster Recovery (DR) Plan?” Early on, surprisingly, I would get very few hands raised. Then I would follow up with the question, “How many of you backup your data?” Most of the audience would raise their hands in reply. Then I would ask, “Why are you backing up your data if you do not have a plan?”
Protecting data is important, but what happens when you can’t access your data? Imagine, the scenario of working for days or weeks on a very important proposal. It is late in the day on the last day before the document is due and you are working late into the night. You are just about finished, then you get a call from your significant other that you need to be home early. You quickly save your work to the network share before you leave, log-off of your computer and head home. You plan to review and make final edits in the morning before submitting the document.
The next morning you arrive early wanting to get a head start on the project. You turn on your computer and get a message saying, “Your credentials have expired. Contact the IT department”. YOU ARE the “IT Department”! You start panicking and as you look through your systems, you start getting messages from the rest of the company. Employees start asking, “Where is my data? The systems won’t let me in!” Sadly, you were hit with a virus, and your data is no longer there.
For the short-term, panicking, screaming at the moon, and pulling your hair out seems like a good temporary solution; but once you get that out of your system, now is when the real work begins. I do not have time to discuss the detailed forensics for this scenario, but this is why a DR plan was created. If you were doing local high availability clusters, all that data most likely is compromised. Even remote replication may not be reliable. There needs to be a separation of the data from the production version: a periodic point in time copy, snapshot, or air gap from the primary data.
Many times, recovery can be as simple as mounting a snapshot, and drag and drop. But in more critical infections, all systems are inaccessible. That is when we need to refer to our backups for recovery.
Backing up the data is important as long as the data being backed up is secure and is the pertinent data that is needed to run your business. The DR plan should include the prioritization of the mission critical applications for your business. It should also include a list of contingency applications that are required to get the business in operation. Most importantly, when doing backups, the “3-2-1 rule” should be followed: 3 - copies of the data, the primary copy is production data, 2 - the secondary copy can be a snapshot or a backup on a local disk (not on the same storage array as the primary data), and 1 - the third copy needs to be on a different media type and needs to be offsite.
I’ve seen many instances when the backup server was also compromised. That is why having a media separation in place is critical. You can rebuild the backup server and then begin the restoration process from your offsite media.
It's crucial for businesses to create and regularly update a DR plan. Will it be easy? Not likely, but depending on your plan and your available resources, recovery from a disaster is doable. It can be done!
If you need more assistance on building out a recovery/DR plan to protect your company from ransomware or other attacks, feel free to reach out to Nth Generation. We will be happy to help!
Jim Russ Vice President, Enterprise Technology