We have fixed another rare pagination bug on our side, and expecting huge advance in rewards generation. However, Ascendex API still having issues with Websocket connections stability, which means that we're still expecting some data delay.
Posted Jan 20, 2022 - 09:05 PST
Update
More websocket, and API issues are being observed on the Ascnedex user order fetcher. These are also related to issues on the Ascendex server side, and will have a resolution soon.
Posted Jan 19, 2022 - 09:17 PST
Update
Our fetcher is lagging due to receiving timeout and server errors from Ascendex. We have confirmed with the Ascendex team that these errors are a result of server overload, and they will have a resolution soon. The issue only affect mostly users in account group 0, but the entire data pipeline is lagged waiting for these users.
We will post update as soon as we have them. In the meantime, rewards are expected to lag frequently. Please reach out to our support team if you experienced missing rewards.
Posted Jan 17, 2022 - 22:16 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 17, 2022 - 17:31 PST
Identified
Ascendex API seems to be having server issues. We are working with the Ascendex team to resolve the issue.
Posted Jan 16, 2022 - 01:37 PST
Investigating
Current Ascendex outage is caused by bad error reporting from the Ascendex API. The issue is caused by Ascendex returning valid status code on invalid API key, which our server treat it as internal error and continue to retry, and halting the data progression. The frequent reward delay this two weeks is caused by multiple issues: 1. Server error from Ascendex API, they restarted their server which causes WS disconnection to spike 2. Bad API key are returned with 200 status code, which our fetcher treat it as internal error, and we retry infinitely
The offending users have been manually disabled while we work on implementing a fix.
Posted Jan 15, 2022 - 22:12 PST
Identified
The issue is caused by Ascendex returning valid status code on invalid API key, which our server treat it as internal error and continue to retry, and halting the data progression. We will be making a fix, no data is lost.
Posted Jan 15, 2022 - 21:37 PST
Update
This issue is happening again. Currently investigating.
Posted Jan 15, 2022 - 18:24 PST
Update
It looks like the OOV aggregators have not done any work in the past 30 minutes. I am going to restart the OOV aggregators now.
Posted Jan 15, 2022 - 13:16 PST
Investigating
We are currently investigating this issue.
Posted Jan 15, 2022 - 05:49 PST
This incident affected: Data fetchers (Ascendex Data Fetcher).