Some years ago, I was working as the Communications Manager for a website editor. Our product helped people who had little to no understanding of programming, HTML or even computers, to build their websites. It wasn’t revolutionary, but it was close. We gained leads and clients very quickly. The product launched in 2009, and by 2011, we had yearly growth of 200% for 2 years in a row.
The secret to our success? Our product was simple and buzzing. It helped businesses be online, at a time when our target audience, mostly SMBs and auto entrepreneurs, started to realize that it was important. They were simply lacking the know-how, and that’s where we came in!
We were running our whole infrastructure from a little server room on premise. We lovingly called it the shed. It was a dark, remote and noisy little room, inhabited by old computers with beeping lights and clicking sounds. Needless to say, we were happy our clients couldn’t see it!
That’s how it all started. As our launch gained traction, we were running out of options and had to upgrade to a rack in a data center. The data center was a 3-hour drive from where we were based. So we took the trip and built our shiny new rack, made all the required settings and migrated data from our shed to the data center. At this point, we were hosting about 500,000 websites, from businesses all over the world. From Canada, France and Morocco, all the way to Australia. Things were getting serious. It was 2010.
Because our product was easy to access and manage, we created a little bit of a wave in the site- builder industry. From 2010 to 2011, we were targeted by multiple DDoS (Distributed Denial of Service) attacks. We never knew the perpetrator, but we knew our 500,000+ clients were down, and they were letting us know! As the Communications Manager, it was my job to try to explain to a lot of SMBs that their websites and e-shops — the business core for many — weren’t available because of something caused by a third-party. Not an easy task, especially when we were responsible for putting their brands out there. They trusted us with their data and were expecting us to be up and running with a decent availability! They didn’t really care that it wasn’t our fault. Our infrastructure wasn’t redundant, so we couldn’t simply switch the load. A DDoS attack meant we were dead in the water until we found a solution or the attacker stopped. The first few times, we were down for a day or so. We also got help from the data center IT guys to help us find solutions. At the end, we were pretty bummed out about the whole situation. But we thought, ‘hey, those servers are brand new. We’re probably doing something wrong with them! Let’s keep pushing through.’
In late 2011, we came to work one morning to flooded inboxes and angry phone calls. We were down again. No traces of attacks, no quick explanation. After hours on the phone, trying to reach someone at the data center, we finally discovered why… and we weren’t pleased! Apparently, the janitor’s shift at the data center coincided with our downtime. Further investigation revealed … that our power cord had been pulled out of the electrical outlet. It seems a bit funny now, but at the time, we were furious and our clients were outraged. Of course, we never told them that the downtime was due to an inattentive janitor who accidentally unplugged the cord from our rack, but the end result was the same: We were down and our business’s image was suffering.
By 2012, we were hosting close to 1.5 million websites. Our servers weren’t capable anymore and we could see that our systems were stretched and in the red most of the time. My IT Director looked more ghoulish every day. Stress was keeping him awake at night and he was dreadfully waiting for the “Server failed” email and text alerts. The service was up and running, but we had a few downtimes related to failures. In 3 years, it was our 2nd time shopping for machines. We knew we needed something that could handle our previous growth rate. That was important information we didn’t have in 2010. We never expected to grow that fast, but during 2012 our growth actually slowed down. Our product was great and our tech support was above the industry average, but we had some frustrating downtime that was hard to recover from. Our goals with the new machines? Think long term; adapt quickly; save a few bucks if possible and deliver the best service we could.
Our IT Director locked himself in a room and started making calls, designing infrastructure, asking for quotes and researching CPUs, RAM, VPN and everything else. After all, we were pressed for time! After a few days, the boxes started coming in. We got $250,000 worth of machines delivered to our office that week. Everything was planned and strategized, and we didn’t forget redundancy this time! Boxes were piled up in a corner, and we started planning a 6-hour round- trip to spend a day building a rack in our data center.
Over the course of the next weeks and months, subscriptions started to slow down and eventually our growth ground to a halt. We had a lot of clients in France, but when an auto entrepreneur subsidy expired, many SMBs and one-man shops started closing down. People were cancelling their websites and new clients were coming in at a slower rate than ever before. The data center trip was pushed back to try and evaluate what marketing could do to improve the situation. Programmers were reassigned to develop shiny new features in an attempt to gain new clients and weren’t available to travel and unbox our new acquisition. Dust started to pile up on the server boxes, while our systems were in dire need of attention.
I can’t say why, but the boxes were never unpacked. The new servers were never installed. Our IT team never went to our data center and never installed the new rack. One morning, my ghoulish IT Director came into work chirpier than ever. He seemed alive for the first time in weeks, even confident. When I asked him what was up, he told me that we shouldn’t have to manage crises and prolonged, hard-to-explain downtime ever again. That morning, he found a Cloud server solution that was tailored to our needs.
Yes, it came a little too late for us from the accountant’s point of view, but the solution was perfect. We could have redundancy, disaster recovery strategies, and frequent online backups and adapt our infrastructure to our current demand without losing a day traveling. Scalability was way easier than shopping for servers every two years or spending too much on high-power machines we didn’t need right away. I say it came a little too late, because subscription rates were still falling and we had a quarter of a million dollars’ worth of machinery rotting away in a corner. You might think that servers have a high resale value, but you’re wrong. When I left that company in 2013, we were still stuck with the boxes. I can only assume that these new servers were sold at a loss or given to start-ups so that they can build their first hybrid cloud solutions!
Your company might not be hosting thousands of websites, but I’m sure you’re hosting data or running a few business applications that you or your clients rely on. I can speak from experience: You don’t want to manage the crisis that follows by being down and unable to deliver. Not only does it tarnish the reputation you’ve established with your clients, but it also results in lost time and poor credibility. There’s no disaster recovery plan for that!