Disaster Recovery: Bucharest, Romania
Power
These facilities are protected against power outages by being equipped with UPS (uninterruptible power supply) that is set for minimum 15 minutes backup at maximum load. They’re far from capacity at this point so the backup is more like 90 minutes. Their generator kicks in at a maximum delay of 2 minutes, although (usually more like 10 seconds). It is always online and preheated. Fuel capacity is 1000 liters, and depending on the load it can last for 10 to 100 hours (per current load they estimate it can last at least 40 hours). There is a company that they have contracted that can send a tanker truck in a maximum 6 hours. As a backup, they can also use the diesel from a gas station that is close to their location.
For their Electricity and cooling infrastructure, they have a 4 hour response time for all alerts and malfunctions. Major replacement parts are kept in stock as cold spares. Everything they have in place is redundant at least N+1.
Fire
Fire suppression and security, they have monitoring and alarming in place (< 10 min intervention), respecting all the local fire protection standards, that are apparently very restrictive there. Fire suppression is activated manually, because false positives are very expensive.
Network
They have their own 2x 10G dark fiber ring to the local carrier hotel where they connect to some providers and the local exchange. Also, they have direct links from other providers directly in the data-center. The wires, routers, switches and so on are fully redundant and resilient. Although everything has failed at least once, the whole thing never went down.
Natural Disaster
Earthquakes are a potential problem in Romania. They don’t have raised floors (always an issue in quakes). All the racks have special grounding systems to protect against such problems. Other than this they have everything insured, for natural disasters or other exceptional things that can occur over time.
Estimated Restore Times For Full Data Loss
Because each server (even at the same location) can vary in disk usage, it’s unfortunately not possible to give exact estimates on how long a full restore will take. From experience however, we have found that most servers can be restored within 8 to 16 hours.
For information regarding restoring VPS services, please click here.
