While AWS burns....
-
@dcon said in While AWS burns....:
Looks like it was due to a typo. Oops.
Straight from the guilty's mouth: https://aws.amazon.com/message/41926/
At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart.
While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected.
Basically it took them 5 hours to restart S3, and the joking comparisons to the GitLab issue were far more on point than expected.
-
@dcon said in While AWS burns....:
Looks like it was due to a typo. Oops.
So in other words, the cloud wasn't really cloud-like in some areas?
Stupid icicles!
-
@Tsaukpaetra said in While AWS burns....:
@dcon said in While AWS burns....:
Looks like it was due to a typo. Oops.
So in other words, the cloud wasn't really cloud-like in some areas?
Stupid icicles!
The cloud got too thick and someone cut it with a knife.
-
@pydsigner Cloudy with a chance of fuck ups.
-
@loopback0 said in While AWS burns....:
@pydsigner Cloudy with a chance of fuck ups.
Cloudy with a chance of E_WEATHER_TRACKER_DEPENDENT_ON_WEATHER