I got this email, did anyone notice the server going down?
Quote:
Dear Customer,
We would like to address the power loss which occurred in our Virginia
Datacenter on Wednesday, June 13th. We are still investigating the
root cause, but in the interest of full disclosure, here are the facts
as we know them today. A more complete post-mortem will be sent to you
as soon as possible.
- A massive wind and hail storm struck the mid-Atlantic region of the
United States including Northern Virginia yesterday afternoon.
- Our internal monitoring system alerted us that the local power grid
dropped at approximately 4:01 pm EDT.
- We have three generators on site:
- - The first generator failed to start up correctly. We are not sure
why and our engineers are investigating further.
- - The second generator started as expected.
- - The third generator started initially but soon failed for an
unknown reason. We are investigating root cause here as well.
- Because we had only partial generator power, not all the Power
Distribution Units (PDUs) were receiving power.
- Power was restored to approximately 50% of our servers in a matter
of minutes. The remaining ~50% required further attention from our
engineers.
- All power was restored at approximately 6:15pm EDT although there
were isolated power fluctuations over the next two hours.
All available technicians were on-site throughout this incident and
have remained in the datacenter over the last 24 hours to ensure all
servers came back online.
Our engineers are still investigating the root cause(s) and we expect
to have more details shortly. We will send a follow up email to you as
quickly as possible. In the meantime, we encourage you to visit
forums for updates. Of course, you can call us
anytime at 9999999999 and press 1 to speak with our 24x7 customer
service team.
In addition to a root cause analysis, we will also outline our plans
to improve power stability and reduce this risk in the future.
We sincerely apologize for this disruption to your business. Our
datacenter infrastructure is designed to withstand acts of nature.
However, there was an obvious failure yesterday for which we take full
responsibility.
Thank you for your patience and we will be in touch soon with more
details.
Respectfully,
We would like to address the power loss which occurred in our Virginia
Datacenter on Wednesday, June 13th. We are still investigating the
root cause, but in the interest of full disclosure, here are the facts
as we know them today. A more complete post-mortem will be sent to you
as soon as possible.
- A massive wind and hail storm struck the mid-Atlantic region of the
United States including Northern Virginia yesterday afternoon.
- Our internal monitoring system alerted us that the local power grid
dropped at approximately 4:01 pm EDT.
- We have three generators on site:
- - The first generator failed to start up correctly. We are not sure
why and our engineers are investigating further.
- - The second generator started as expected.
- - The third generator started initially but soon failed for an
unknown reason. We are investigating root cause here as well.
- Because we had only partial generator power, not all the Power
Distribution Units (PDUs) were receiving power.
- Power was restored to approximately 50% of our servers in a matter
of minutes. The remaining ~50% required further attention from our
engineers.
- All power was restored at approximately 6:15pm EDT although there
were isolated power fluctuations over the next two hours.
All available technicians were on-site throughout this incident and
have remained in the datacenter over the last 24 hours to ensure all
servers came back online.
Our engineers are still investigating the root cause(s) and we expect
to have more details shortly. We will send a follow up email to you as
quickly as possible. In the meantime, we encourage you to visit
forums for updates. Of course, you can call us
anytime at 9999999999 and press 1 to speak with our 24x7 customer
service team.
In addition to a root cause analysis, we will also outline our plans
to improve power stability and reduce this risk in the future.
We sincerely apologize for this disruption to your business. Our
datacenter infrastructure is designed to withstand acts of nature.
However, there was an obvious failure yesterday for which we take full
responsibility.
Thank you for your patience and we will be in touch soon with more
details.
Respectfully,