Services appear to be restored. We will continue to monitor to ensure stability. We sincerely apologize for the service disruption and resulting inconvenience.
Amazon Web Services message:
[RESOLVED] Increased API Error Rates and Latency [11:21 AM PDT] Starting at 7:34 AM PDT, we experienced increased error rates and latencies for the Systems Manager Parameter Store in the [region removed]. Customers using the [system removed] for the storage of configuration were affected by the event as they were unable to retrieve the data. Customers using [system removed] from Lambda functions also experienced increased failure rates. The issue was resolved at 10:45 AM PDT, when error rates and latencies returned to normal levels. The issue has been resolved and the service is operating normally.
Posted Sep 09, 2022 - 18:21 UTC
Update
Latest update from Amazon Web Services:
[11:02 AM PDT] We are seeing recovery for the issue causing increased API error rates and latencies for the Systems Manager Parameter Store in the [region removed]. Customers using the [system removed] for the storage of configuration should be seeing recovery at this stage. We continue to monitor the subsystem that was experiencing resource contention, but expect error rates and latencies to remain at normal levels.
Posted Sep 09, 2022 - 18:09 UTC
Update
No change to report. Latest update from Amazon Web Services
[10:33 AM PDT] We continue to work on the issue causing increased API error rates and latencies for the [system removed] in the [region removed]. Customers using the [system removed] for the storage of configuration may be affected by the event as they are unable to retrieve the data.... We are making progress towards resolution but continue to see elevated error rates for the affected subsystem. We do not have an ETA on recovery at this stage and will keep you updated on our progress.
Posted Sep 09, 2022 - 17:59 UTC
Update
We are seeing minor improvements depending on the instance but issues remain. Publishing and some Syndicate instances are still down.
Lastest from AWS [09:50 AM PDT] We continue to work on the issue causing increased API error rates and latencies for the [system removed] in the [region removed]. Customers using the [system removed] for the storage of configurations may be affected by the event as they are unable to retrieve the data. Customers invoking Lambda functions that use Environment Variables will also experience increased error rates and latencies. The subsystem responsible for the Parameter Store is experiencing resource contention, which is leading to the increased error rates and latencies. We continue to work toward resolving the issue, but do not have an ETA on recovery at this stage. We will keep you updated on our progress.
Posted Sep 09, 2022 - 17:18 UTC
Update
We are still seeing issues across our US-hosted customers but it seems that some Syndicate tenants are functioning but with degraded performance. Authors with active sessions in Create appear to be able to create content, but those logging in are getting errors.
Latest update from AWS [09:29 AM PDT] We can confirm increased API error rates and latencies for the [system removed] in the [region removed]. Customers using the [system removed] for the storage of data may be affected by the event as they are unable to retrieve the data. We have identified the subsystem where the errors are occurring and are working to resolve the issue. At this stage, we expect recovery to take more than an hour but will keep you updated on our progress.
Posted Sep 09, 2022 - 16:44 UTC
Identified
We have identified the issue as being with Amazon Web Services. We’re working hard to restore services as quickly as possible. Please continue to check the Status Page for the latest information.
Posted Sep 09, 2022 - 16:02 UTC
Update
We are continuing to investigate this issue.
Posted Sep 09, 2022 - 15:52 UTC
Update
We are continuing to investigate this issue.
Posted Sep 09, 2022 - 15:51 UTC
Investigating
We are currently experiencing a service disruption. Our engineering team is investigating the issue and working on restoring services as quickly as possible. Please continue to check the Status Page for the latest information. We sincerely apologize for the inconvenience.
Posted Sep 09, 2022 - 15:44 UTC
This incident affected: Syndicate - US Hosting (Syndicate (CDS), SCORM Player (CloudPlayer), Learning Analytics) and Create - US Hosting (Web Publisher, Print Publisher, Studio Publishing Service, XML Publisher).