DataCoord is not ready, error=service not ready: Abnormal #32459
Replies: 3 comments 2 replies
-
Similar to this issue: #31858 |
Beta Was this translation helpful? Give feedback.
-
@yhmo Hi, I encountered this error again.
Could you please review these logs? Currently, I am using SSD disk. The system load is low, insert ~ 3 records/s and a query ~ 5 queries/s. I believe my etcd cluster performance is adequate to handle the requests. Here are the etcd benchmark:
milvus.py
When I restart the cluster, Milvus works again, so I think there is a problem with the services. |
Beta Was this translation helpful? Give feedback.
-
watch timeout: "timeout and stop timer: wait for channel ACK timeout" you need to offer log from datanode and check why channel watch is timeout |
Beta Was this translation helpful? Give feedback.
-
My deployment:
During operation, I encountered several warning logs indicating issues with the DataCoord service. Initially, there were warning logs related to session revocation failures and context deadline exceeded errors.
After that, rootcoord, querycoord, datacoord,... restarts, all services run but datacoord outputs many errors
Here are all log files at the error time
datacoord.log
querycoord.log
datanode.log
indexnode.log
Beta Was this translation helpful? Give feedback.
All reactions