EasyBib Downtime [Resolved]

UPDATE STATUS [Resolved]

2:20 pm UPDATE: EasyBib’s main website is back online as of 2:20pm EDT. However, citations and lists still do not load.

2:36 pm UPDATE: We’re still bringing the citation data back online. You can format new citations, but currently cannot retrieve existing ones or stored lists. We will update this post as we progress further.

3:55 pm UPDATE: We are still working to restore saved citations (all citations are safe; we are just trying to get our servers that hold the data back online).

4:45 pm UPDATE: We are still working to restore saved citations.

1:22AM 4/22/11 UPDATE: citations are still not restored, but our hoster is working on bringing them up. We know that this has been an awful day for our users, and we recognize that many, many of you have projects, dissertations, and more stored on our systems that you have been unable to retrieve. We’re not proud of it, and we hope by tomorrow morning we’ll be back in action. We’ll update you as soon as we can on as we move things forward. This isn’t an abnegation of responsibility; we’re working 100% to get your data back online. Thank you for your patience. 

11:22AM 4/22/11 UPDATE: You can now create and save new citations and projects. Additionally, projects made BEFORE November 2010 are also available. Projects made after November 2010 up to now are still being brought online. ETA will be today. However, if you have an emergency bibliography to create, we recommend you create a new project to re-enter your citations. For premium users, the Notebook is still inaccessible.

4:35PM 4/22/11 RESOLUTION: We’re happy to report we’re back online with all citation, notebook, and project data. Thank you for your patience. We apologize to the many, many students we’ve let down in the middle of dissertations and midterms; a whole scale meltdown of Amazon Web Services has never really happened. Now that we know we should expect the worst, we will prepare for it in the future.

  • Note: for the curious, this is a partial list of all sites affected by this outage, in addition to EasyBib: http://ec2disabled.com/
  • Users interested in the technical reasons for failure can check out the posts on the AWS status page (click More in the Amazon Elastic Compute Cloud (N. Virginia) row): http://status.aws.amazon.com

——-

Hey everyone,

As you’re aware, EasyBib is currently down. Our hosting provider, Amazon Web Services, had issues starting at 3AM EDT this morning.

Here is their status. We are working to get our site back up and running despite the issues they are continuing to experience: 

Via http://status.aws.amazon.com/ :

  • 1:41 AM PDT We are currently investigating latency and error rates with EBS volumes and connectivity issues reaching EC2 instances in the US-EAST-1 region.
  • 2:18 AM PDT We can confirm connectivity errors impacting EC2 instances and increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region. Increased error rates are affecting EBS CreateVolume API calls. We continue to work towards resolution.
  • 2:49 AM PDT We are continuing to see connectivity errors impacting EC2 instances, increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region, and increased error rates affecting EBS CreateVolume API calls. We are also experiencing delayed launches for EBS backed EC2 instances in affected availability zones in the US-EAST-1 region. We continue to work towards resolution.
  • 3:20 AM PDT Delayed EC2 instance launches and EBS API error rates are recovering. We’re continuing to work towards full resolution.
  • 4:09 AM PDT EBS volume latency and API errors have recovered in one of the two impacted Availability Zones in US-EAST-1. We are continuing to work to resolve the issues in the second impacted Availability Zone. The errors, which started at 12:55AM PDT, began recovering at 2:55am PDT
  • 5:02 AM PDT Latency has recovered for a portion of the impacted EBS volumes. We are continuing to work to resolve the remaining issues with EBS volume latency and error rates in a single Availability Zone. 
  • 6:09 AM PDT EBS API errors and volume latencies in the affected availability zone remain. We are continuing to work towards resolution.
  • 6:59 AM PDT There has been a moderate increase in error rates for CreateVolume. This may impact the launch of new EBS-backed EC2 instances in multiple availability zones in the US-EAST-1 region. Launches of instance store AMIs are currently unaffected. We are continuing to work on resolving this issue.
  • 7:40 AM PDT In addition to the EBS volume latencies, EBS-backed instances in the US-EAST-1 region are failing at a high rate. This is due to a high error rate for creating new volumes in this region.8:54 AM PDT We’d like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it’s difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We’re starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.
  • 10:26 AM PDT We have made significant progress in stabilizing the affected EBS control plane service. EC2 API calls that do not involve EBS resources in the affected Availability Zone are now seeing significantly reduced failures and latency and are continuing to recover. We have also brought additional capacity online in the affected Availability Zone and stuck EBS volumes (those that were being remirrored) are beginning to recover. We cannot yet estimate when these volumes will be completely recovered, but we will provide an estimate as soon as we have sufficient data to estimate the recovery. We have all available resources working to restore full service functionality as soon as possible. We will continue to provide updates when we have them.
  • 11:09 AM PDT A number of people have asked us for an ETA on when we’ll be fully recovered. We deeply understand why this is important and promise to share this information as soon as we have an estimate that we believe is close to accurate. Our high-level ballpark right now is that the ETA is a few hours. We can assure you that all-hands are on deck to recover as quickly as possible. We will update the community as we have more information.
  • Later status updates can be found at http://status.aws.amazon.com


%d bloggers like this: