Resolved -
All services have been restored. We will continue to monitor to ensure stability. We sincerely apologize for the service disruption.
Please limit publishing to Syndicate to time sensitive documents, as there may be more traffic than normal for the rest of the day as people get caught up.
Oct 20, 19:34 UTC
Update -
Syndicate is back up. Create is available for authoring but preview. publishing is still offline. We're working to fully restore services.
Oct 20, 18:19 UTC
Update -
Here's the latest from AWS. It appears that they are making progress but Syndicate and Create are still being impacted.
"Oct 20 10:38 AM PDT Our mitigations to resolve launch failures for new EC2 instances are progressing and the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside."
Oct 20, 17:47 UTC
Update -
Our team has implemented a descriptive error page for Syndicate. Here is the latest from AWS
Oct 20 10:03 AM PDT We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services. Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely. We will provide an update by 10:45 AM PDT.
Oct 20, 17:14 UTC
Update -
Latest from AWS
Oct 20 9:13 AM PDT We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
Oct 20, 16:15 UTC
Update -
Latest from AWS
Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations.
Oct 20, 15:52 UTC
Update -
Latest from AWS
Oct 20 8:04 AM PDT We continue to investigate the root cause for the network connectivity issues that are impacting AWS services such as DynamoDB, SQS, and Amazon Connect in the US-EAST-1 Region. We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations.
Oct 20, 15:09 UTC
Update -
Latest from AWS
Oct 20 7:29 AM PDT We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause.
Oct 20, 14:42 UTC
Update -
The latest from AWS
Oct 20 7:14 AM PDT We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information.
Oct 20, 14:24 UTC
Update -
We are now seeing issues with US-hosted Syndicate instances as well.
Oct 20, 14:22 UTC
Identified -
The issues were triggered by an outage at the Amazon Web Services (AWS) data center in Virginia. That is where we host-US customers and beta instances for all customers. It seems to be limited to Create for most US-Hosted customers. We are aware of two customers, in special environments, where their Syndicate instances that are also impacted. We don't expect that other Syndicate instances should be affected.
Our team is monitoring the AWS status and working through any issue they have control of as well. Here is the latest update from Amazon Web Services.
"[06:42 AM PDT] We have applied multiple mitigations across multiple Availability Zones (AZs) in US-EAST-1 and are still experiencing elevated errors for new EC2 instance launches. We are rate limiting new instance launches to aid recovery. We will provide an update at 7:30 AM PDT or sooner if we have additional information."
Oct 20, 14:01 UTC
Investigating -
We are currently experiencing a service disruption impacting all customer who are not EU-hosted. Our engineering team is investigating the issue and working on restoring services as quickly as possible. Please continue to check the Status Page for the latest information. We sincerely apologize for the inconvenience.
Oct 20, 13:26 UTC