System Status
Rewards tracking is lagging for Ascendex
Monitoring - A fix has been implemented and we are monitoring the results.
Jan 22, 21:39 PST
Update - We are continuing to work on a fix for this issue.
Jan 22, 20:58 PST
Update - We are continuing to work on a fix for this issue.
Jan 22, 20:08 PST
Update - We are still experiencing delays with AscendEX rewards. Investigations show that one of the collectors has no recent data. We are working on this internally and will provide an update soon as possible.
Jan 22, 20:06 PST
Identified - This issue is due to the ascendex price recorder encountering errors. We are working on a fix now.
Jan 21, 09:33 PST
Gate.io Order Tracking & Rewards are delayed
Update - We are continuing to investigate this issue.
Jan 22, 20:08 PST
Investigating - We have received user reports of Gate.io Rewards & Orders being delayed. We are currently reviewing the cause of this issue.
Jan 22, 20:07 PST
Inflated open order volume
Identified - We have identified possible causes and will be delivering fixes soon.
Nov 8, 17:33 PST
Update - We are continuing to investigate this issue.
Sep 24, 15:13 PDT
Investigating - We have identified inflated open order volume in some markets including PHB-TUSD-Binance, while we are investigating for the root cause, the max order life time will be set to 1 hour to reduce the inflation.
Jun 30, 01:32 PDT

About This Site

Welcome to the status page for Hummingbot Miner, our liquidity mining platform. Here, we post notices about system downtime, exchange API outages, and our issues that affect our user and their bots.

Uptime over the past 90 days. View historical uptime.
Miner Web App ? Operational
90 days ago
99.94 % uptime
Today
Rewards Engine ? Degraded Performance
90 days ago
99.99 % uptime
Today
Price recorders Operational
90 days ago
99.99 % uptime
Today
Binance mid price recorder Operational
90 days ago
99.96 % uptime
Today
Ascendex mid price recorder Operational
90 days ago
100.0 % uptime
Today
Kucoin mid price recorder Operational
90 days ago
100.0 % uptime
Today
Gateio mid price recorder Operational
90 days ago
100.0 % uptime
Today
Data fetchers Degraded Performance
90 days ago
98.86 % uptime
Today
Binance Data Fetcher ? Degraded Performance
90 days ago
100.0 % uptime
Today
Ascendex Data Fetcher Degraded Performance
90 days ago
97.45 % uptime
Today
Gate.io Data Fetcher ? Degraded Performance
90 days ago
100.0 % uptime
Today
KuCoin Data Fetcher Operational
90 days ago
97.99 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Past Incidents
Jan 23, 2022

No incidents reported today.

Jan 22, 2022

Unresolved incidents: Gate.io Order Tracking & Rewards are delayed, Rewards tracking is lagging for Ascendex.

Jan 21, 2022
Binance Rewards are not showing
Resolved - The error was caused by PIVX/ETH pair removed from trading on Binance side. Mid price recorder wasn't able to handle this. We've deployed newer version of the component which should be able to self-recover from such issues.
Jan 21, 02:09 PST
Update - We are continuing to investigate this issue.
Jan 21, 02:07 PST
Investigating - From our monitoring, we have observed that Binance rewards are not being displayed (orders are still being tracked). Initial investigation shows an error with the recorder having an invalid timestamp price map. Our teams are now engaged in resolving this. No ETA for the fix and will provide an update accordingly.
Jan 20, 22:57 PST
Ascendex reward tracking is delayed
Resolved - All issues has been resolved.
Jan 21, 00:59 PST
Update - We have fixed another rare pagination bug on our side, and expecting huge advance in rewards generation. However, Ascendex API still having issues with Websocket connections stability, which means that we're still expecting some data delay.
Jan 20, 09:05 PST
Update - More websocket, and API issues are being observed on the Ascnedex user order fetcher. These are also related to issues on the Ascendex server side, and will have a resolution soon.
Jan 19, 09:17 PST
Update - Our fetcher is lagging due to receiving timeout and server errors from Ascendex.
We have confirmed with the Ascendex team that these errors are a result of server overload, and they will have a resolution soon.
The issue only affect mostly users in account group 0, but the entire data pipeline is lagged waiting for these users.

We will post update as soon as we have them. In the meantime, rewards are expected to lag frequently.
Please reach out to our support team if you experienced missing rewards.
Jan 17, 22:16 PST
Monitoring - A fix has been implemented and we are monitoring the results.
Jan 17, 17:31 PST
Identified - Ascendex API seems to be having server issues. We are working with the Ascendex team to resolve the issue.
Jan 16, 01:37 PST
Investigating - Current Ascendex outage is caused by bad error reporting from the Ascendex API.
The issue is caused by Ascendex returning valid status code on invalid API key, which our server treat it as internal error and continue to retry, and halting the data progression.
The frequent reward delay this two weeks is caused by multiple issues:
1. Server error from Ascendex API, they restarted their server which causes WS disconnection to spike
2. Bad API key are returned with 200 status code, which our fetcher treat it as internal error, and we retry infinitely

The offending users have been manually disabled while we work on implementing a fix.
Jan 15, 22:12 PST
Identified - The issue is caused by Ascendex returning valid status code on invalid API key, which our server treat it as internal error and continue to retry, and halting the data progression.
We will be making a fix, no data is lost.
Jan 15, 21:37 PST
Update - This issue is happening again. Currently investigating.
Jan 15, 18:24 PST
Update - It looks like the OOV aggregators have not done any work in the past 30 minutes. I am going to restart the OOV aggregators now.
Jan 15, 13:16 PST
Investigating - We are currently investigating this issue.
Jan 15, 05:49 PST
Jan 20, 2022
No data showing in Rewards tab and Dashboard in Miner
Resolved - A fix has been deployed to handle 502 errors observed on the Miner Web App. No further issues observed since then and the teams will monitor its performance.
Jan 20, 23:07 PST
Monitoring - Rewards and Dashboard is now reflecting data in the Miner App. We are monitoring the frontend performance to verify if issue is now resolved.
Jan 17, 21:19 PST
Identified - Have identified an issue with the account permission, causing performance issues. Service is now restarted and currently monitoring the performance.
Jan 17, 21:15 PST
Investigating - We are seeing a re-occurrence of the UI issues observed in miner, wherein the Rewards and Dashboard is not reflecting any data. Teams are informed and engaged on resolving this issue. Rest assured that we aim to fix this asap. We will continue to post updates.
Jan 17, 18:01 PST
Kucoin serializer lags behind real time
Resolved - History fetch was completed and the rewards have been reflected on the dashboard. No further issues observed.
Jan 20, 23:02 PST
Monitoring - The fetchers have completed the history fetch and rewards should reflect momentarily.
Jan 20, 19:31 PST
Investigating - The Kucoin fetchers were doing a history fetch, and have since finished. The serializers will catch up soon, and rewards will follow soon after.
Jan 20, 19:11 PST
Jan 19, 2022
Jan 18, 2022

No incidents reported.

Jan 17, 2022
Ascendex User Order Fetchers have stopped fetching
Resolved - This issue has been resolved.
Jan 17, 15:40 PST
Monitoring - Monitoring in progress.
Jan 17, 11:58 PST
Update - All fetchers were restarted an hour ago, and have not found data since then. Since the restart took place, we have seen an uptick in failed API requests, and timeout errors. This is due to a history fetch taking place for all users. The issue will resolve itself over time.
Jan 17, 11:58 PST
Investigating - There are other ascendex issues at the moment being resolved, so this could potentially be related. Investigating now.
Jan 17, 11:49 PST
Jan 16, 2022
Jan 15, 2022
Jan 14, 2022

No incidents reported.

Jan 13, 2022
Datadog high number of indexed logs
Resolved - This incident has been resolved.
Jan 13, 17:18 PST
Investigating - We are currently investigating this issue.
Jan 13, 11:34 PST
Jan 12, 2022

No incidents reported.

Jan 11, 2022
Delayed rewards for Kucoin
Resolved - This incident has been resolved.
Jan 11, 05:33 PST
Monitoring - Issue should be resolved. We are monitoring any further events from our end.
Jan 10, 06:50 PST
Update - We are continuing to investigate this issue.
Jan 9, 18:38 PST
Investigating - Kucoin serializer and fetchers are behind real time, possibly due to key-based rate limits being hit.
Jan 9, 18:33 PST
Delayed rewards for Ascendex
Resolved - Timeout error issues self-resolved.
Jan 11, 05:31 PST
Update - Seeing again timeout issues
Jan 10, 11:41 PST
Monitoring - Timeout error issues self-resolved.
Jan 10, 06:31 PST
Investigating - We're getting timeout errors from Ascendex API during data fetch for the several clients.
Jan 10, 05:14 PST
Jan 10, 2022
Jan 9, 2022
← Incident History Powered by Atlassian Statuspage