Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DmanV3 script not cleaning up all pending orders after close on HyperLiquid #7032

Open
tijsk opened this issue May 19, 2024 · 10 comments
Open

DmanV3 script not cleaning up all pending orders after close on HyperLiquid #7032

tijsk opened this issue May 19, 2024 · 10 comments
Labels
bug hyperliquid_perpetual

Comments

@tijsk
Copy link

tijsk commented May 19, 2024 •
edited

Describe the bug

This is tested on the 1.28 development using HyperLiquid

While having the script running I've noticed that some orders of the previous executor are still present on the exchange. See attached screens of Hummingbot status and exchange graph.

When these "ghost orders" are hit and filled they are not taken into account by the Hummingbot script. Resulting in a difference in active amount and finally in a partial closure with a left over active order.

I'm not sure what the philosophy behind the scripts is, but it looks like it is only taking in account the actions from the script itself. Now that this bug is active it's not correcting itself with the actual data from the exchange. In my opinion the script should be based on the actual values on the exchange instead of its own interpretation of it.
Now the script has a 'filled_amount_quote' of 321 USD and the exchange has a "Position Value" of 451 USD.

Steps to reproduce

  1. create script and controller config using dman_v3 with 3 controllers (not sure whether this amount could be related)
  2. let it run for a while
  3. notice leftover orders still in the system of Hummingbot and exchange

Release version

dev-1.28.0

Type of installation

Docker

Attach required files

image
image
@tijsk tijsk added the bug label May 19, 2024
@nikspz
Copy link
Contributor

nikspz commented May 20, 2024

hi @tijsk Could you please provide config of your controller / controllers?

@tijsk
Copy link
Author

tijsk commented May 20, 2024

conf_directional_trading.dman_v3-BTC.txt

@nikspz
Copy link
Contributor

nikspz commented May 20, 2024

conf_directional_trading.dman_v3-BTC.txt

Thanks,
Could you also send log file here to see client behavior?

@tijsk
Copy link
Author

tijsk commented May 20, 2024

logs_conf_v2_with_controllers_2024-05-18.txt
logs_conf_v2_with_controllers.txt

@tijsk
Copy link
Author

tijsk commented May 20, 2024

@nikspz Thanks for looking into this. Could this be because of a too low 'cooldown_time' setting? The default is 300, I've set it to 60.

@tijsk
Copy link
Author

tijsk commented May 20, 2024

@nikspz Thanks for looking into this. Could this be because of a too low 'cooldown_time' setting? The default is 300, I've set it to 60.

Wasn't fixed by changing this setting. Now trying to 'bb_long_threshold' to 0.9 instead of 1.0

@tijsk
Copy link
Author

tijsk commented May 20, 2024

@nikspz Thanks for looking into this. Could this be because of a too low 'cooldown_time' setting? The default is 300, I've set it to 60.

Wasn't fixed by changing this setting. Now trying to 'bb_long_threshold' to 0.9 instead of 1.0

Isn't the solution either. So I atm I'm out of idea's. Let me know if I can do something else.

@nikspz
Copy link
Contributor

nikspz commented May 20, 2024 •
edited

Okay will review logs later this week.
From initial check you have 50 order levels, could be related to this. Since Hyperliquid is DEX exchange, could be hard to handle many order levels requests. That's my personal assumption.

logs_conf_v2_with_controllers_2024-05-18.txt logs_conf_v2_with_controllers.txt

issues from the logs:

1.Failed to cancel order

2024-05-18 10:52:46,720 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - ERROR - Failed to cancel order 0x45aab825cb869797f1590bcfb7956660
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 520, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 540, in _execute_order_cancel_and_process_update
    cancelled = await self._place_cancel(order.client_order_id, order)
  File "/home/hummingbot/hummingbot/connector/derivative/hyperliquid_perpetual/hyperliquid_perpetual_derivative.py", line 277, in _place_cancel
    raise IOError(f'{cancel_result["response"]["data"]["statuses"][0]["error"]}')
TypeError: string indices must be integers
  1. Server disconnected error
2024-05-18 17:18:01,378 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - NETWORK - Unexpected error while fetching last fee payment for ETH-USD.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/perpetual_derivative_py_base.py", line 403, in _update_funding_payment
    timestamp, funding_rate, payment_amount = await self._fetch_last_fee_payment(trading_pair=trading_pair)
  File "/home/hummingbot/hummingbot/connector/derivative/hyperliquid_perpetual/hyperliquid_perpetual_derivative.py", line 790, in _fetch_last_fee_payment
    funding_info_response = await self._api_post(path_url=CONSTANTS.GET_LAST_FUNDING_RATE_PATH_URL,
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 867, in _api_post
    return await self._api_request(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 906, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
    response = await self.execute_request_and_get_response(
  File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 94, in execute_request_and_get_response
    response = await self.call(request=request, timeout=timeout)
  File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 108, in call
    resp = await wait_for(self._connection.call(request), timeout)
  File "/opt/conda/envs/hummingbot/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/hummingbot/hummingbot/core/web_assistant/connections/rest_connection.py", line 10, in call
    aiohttp_resp = await self._client_session.request(
  File "/opt/conda/envs/hummingbot/lib/python3.10/site-packages/aiohttp/client.py", line 608, in _request
    await resp.start(conn)
  File "/opt/conda/envs/hummingbot/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 976, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/opt/conda/envs/hummingbot/lib/python3.10/site-packages/aiohttp/streams.py", line 640, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

@tijsk
Copy link
Author

tijsk commented May 25, 2024

Hi @nikspz,
I've been testing this with fewer orders and the error is still present. When using 50 order for 3 pairs they are created about 1 every second. As far as I can tell from the API rate docs the cancel order rate has a bigger limit than the other types of calls.

@tijsk
Copy link
Author

tijsk commented May 28, 2024

When stopping and exiting the instance and running it again these errors occur:

2024-05-28 21:03:11,250 - 15 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2024-05-28 21:03:12,001 - 15 - hummingbot.strategy.script_strategy_base - WARNING - hyperliquid_perpetual is not ready. Please wait...
2024-05-28 21:03:12,149 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - INFO - Leverage for BTC-USD successfully set to 30.
2024-05-28 21:03:12,150 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-05-28 21:03:12,469 - 15 - hummingbot.data_feed.candles_feed.binance_perpetual_candles.binance_perpetual_candles - INFO - Subscribed to public klines...
2024-05-28 21:03:12,470 - 15 - hummingbot.data_feed.candles_feed.binance_perpetual_candles.binance_perpetual_candles - INFO - Subscribed to public klines...
2024-05-28 21:03:12,470 - 15 - hummingbot.data_feed.candles_feed.binance_perpetual_candles.binance_perpetual_candles - INFO - Subscribed to public klines...
2024-05-28 21:03:12,471 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_api_order_book_data_source.HyperliquidPerpetualAPIOrderBookDataSource - INFO - Subscribed to public order book, trade channels...
2024-05-28 21:03:12,471 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_api_order_book_data_source.HyperliquidPerpetualAPIOrderBookDataSource - INFO - Subscribed to public order book, trade channels...
2024-05-28 21:03:12,472 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_api_order_book_data_source.HyperliquidPerpetualAPIOrderBookDataSource - INFO - Subscribed to public order book, trade channels...
2024-05-28 21:03:12,473 - 15 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for BTC-USD. 1/3 completed.
2024-05-28 21:03:12,808 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - INFO - Leverage for SOL-USD successfully set to 20.
2024-05-28 21:03:12,966 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - INFO - Leverage for ETH-USD successfully set to 25.
2024-05-28 21:03:12,967 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_user_stream_data_source.HyperliquidPerpetualUserStreamDataSource - INFO - Subscribed to private order and trades changes channels...
2024-05-28 21:03:13,121 - 15 - hummingbot.strategy.script_strategy_base - WARNING - hyperliquid_perpetual is not ready. Please wait...
2024-05-28 21:03:13,485 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930193.0, "order_id": "0x84a025370f037c981623341166046d09", "exchange_order_id": "24209867839", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:13,485 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0x84a025370f037c981623341166046d09.
2024-05-28 21:03:13,809 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930193.0, "order_id": "0xf012e87c690268a53e84a364bf138252", "exchange_order_id": "24209871687", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:13,809 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0xf012e87c690268a53e84a364bf138252.
2024-05-28 21:03:13,966 - 15 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for ETH-USD. 2/3 completed.
2024-05-28 21:03:14,128 - 15 - hummingbot.strategy.script_strategy_base - WARNING - hyperliquid_perpetual is not ready. Please wait...
2024-05-28 21:03:14,133 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930194.0, "order_id": "0x1f1b7f14e3c190664e8b9b53e341234e", "exchange_order_id": "24209873994", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:14,134 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0x1f1b7f14e3c190664e8b9b53e341234e.
2024-05-28 21:03:14,475 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930194.0, "order_id": "0x13dac90c72dcb4c0c47aea128d2549bf", "exchange_order_id": "24209876719", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:14,475 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0x13dac90c72dcb4c0c47aea128d2549bf.
2024-05-28 21:03:14,810 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930194.0, "order_id": "0xb433158fe9d04e82023f206d40fb15b8", "exchange_order_id": "24209879097", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:14,810 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0xb433158fe9d04e82023f206d40fb15b8.
2024-05-28 21:03:15,130 - 15 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2024-05-28 21:03:15,494 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930195.0, "order_id": "0x96592e49ea6b9b28f6c64a65ed4f1edf", "exchange_order_id": "24209880077", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:15,494 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0x96592e49ea6b9b28f6c64a65ed4f1edf.
2024-05-28 21:03:15,498 - 15 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for SOL-USD. 3/3 completed.
2024-05-28 21:03:15,828 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0xca265c4b79096d09b87908f4677da088 (order not found)
2024-05-28 21:03:15,834 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930195.0, "order_id": "0x33d6dd34a8c03369dc9bb8b286cd9c42", "exchange_order_id": "24209883266", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:15,834 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0x33d6dd34a8c03369dc9bb8b286cd9c42.
2024-05-28 21:03:15,996 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0xc6a2f3522ceafe546144107b4b1ecdf7 (order not found)
2024-05-28 21:03:16,161 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0x67b114d8fd2bd2ada835f86fe43abae6 (order not found)
2024-05-28 21:03:16,161 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0x33d6dd34a8c03369dc9bb8b286cd9c42 (order not found)
2024-05-28 21:03:16,162 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0x96592e49ea6b9b28f6c64a65ed4f1edf (order not found)
2024-05-28 21:03:16,170 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1716930196.0, "order_id": "0xca265c4b79096d09b87908f4677da088", "exchange_order_id": "24209886144", "event_name": "OrderCancelledEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:16,170 - 15 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 0xca265c4b79096d09b87908f4677da088.
2024-05-28 21:03:16,833 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0xc6a2f3522ceafe546144107b4b1ecdf7 (order not found)
2024-05-28 21:03:16,833 - 15 - hummingbot.connector.client_order_tracker - WARNING - The order 0xc6a2f3522ceafe546144107b4b1ecdf7(24209888806) will be considered lost. Please check its status in the exchange.
2024-05-28 21:03:16,840 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": NaN, "order_id": "0xc6a2f3522ceafe546144107b4b1ecdf7", "order_type": "OrderType.LIMIT", "event_name": "MarketOrderFailureEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:16,840 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 0xc6a2f3522ceafe546144107b4b1ecdf7 has failed. Order Update: OrderUpdate(trading_pair='SOL-USD', update_timestamp=nan, new_state=<OrderState.FAILED: 6>, client_order_id='0xc6a2f3522ceafe546144107b4b1ecdf7', exchange_order_id=None, misc_updates=None)
2024-05-28 21:03:16,840 - 15 - hummingbot.connector.derivative.hyperliquid_perpetual.hyperliquid_perpetual_derivative.HyperliquidPerpetualDerivative - WARNING - Failed to cancel order 0x67b114d8fd2bd2ada835f86fe43abae6 (order not found)
2024-05-28 21:03:16,841 - 15 - hummingbot.connector.client_order_tracker - WARNING - The order 0x67b114d8fd2bd2ada835f86fe43abae6(24209900287) will be considered lost. Please check its status in the exchange.
2024-05-28 21:03:16,846 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": NaN, "order_id": "0x67b114d8fd2bd2ada835f86fe43abae6", "order_type": "OrderType.LIMIT", "event_name": "MarketOrderFailureEvent", "event_source": "hyperliquid_perpetual"}
2024-05-28 21:03:16,846 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 0x67b114d8fd2bd2ada835f86fe43abae6 has failed. Order Update: OrderUpdate(trading_pair='SOL-USD', update_timestamp=nan, new_state=<OrderState.FAILED: 6>, client_order_id='0x67b114d8fd2bd2ada835f86fe43abae6', exchange_order_id=None, misc_updates=None)

After deleting the database file this the script is starting again. Maybe this helps in finding the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug hyperliquid_perpetual
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tijsk @rapcmia @nikspz