kong lua-resty-lock lock timeout 500 error {"message":"An unexpected error occurred"}

[dubuqingfeng]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.0.0

Current Behavior

Few requests：
{"message":"An unexpected error occurred"}
status: 500

Expected Behavior

Normal

Steps To Reproduce

Few requests

500 log:

remote_addr: 10.1.33.25
realip_remote_addr: 10.1.33.25
upstream_addr: -
body_bytes_sent: 42
time_local: 27/Apr/2024:01:06:47
request_method: POST
upstream_host_name: HOST_NAME
http_host: fxxx.com
request_uri: /rpc
http_user_agent: okhttp/4.9.3
remote_user: -
upstream_status: -
request_time: 5.017
rpc_method:
request_length: 35
http_referer: -
msec: 1714180007.468
http_x_forwarded_for: -
servicename: service_name
upstream_response_time: -
request_body_head:
server_protocol: HTTP/2.0
status: 500

Anything else?

After modifying the code, print relevant logs:

failed the initial dns/balancer resolve for 'UPSTREAM_HOST' with: failed to get from node cache: could not acquire callback lock: timeout

Depoly: Hybrid mode

https://github.com/openresty/lua-resty-lock

timeout Specifies the maximal waiting time (in seconds) for the lock method calls on the current object instance. You can specify up to 0.001 seconds. Default to 5 (seconds). This option value cannot be bigger than exptime. This timeout is to prevent a lock method call from waiting forever. You can specify 0 to make the lock method return immediately without waiting if it cannot acquire the lock right away.

#12811

KONG_MEM_CACHE_SIZE=128m，This value has not been changed，so maybe 128 m

The code involved in 500 is below：

https://github.com/Kong/kong/blob/3.0.0/kong/runloop/handler.lua#L1011

local function balancer_execute(ctx)
  local balancer_data = ctx.balancer_data
  local ok, err, errcode = balancer.execute(balancer_data, ctx)
  if not ok and errcode == 500 then
    err = "failed the initial dns/balancer resolve for '" ..
          balancer_data.host .. "' with: " .. tostring(err)
  end
  return ok, err, errcode
end

[chronolaw]

Kong 3.0 is EOL, could you try the latest 3.6 or 3.4 LTS? thanks.

[dubuqingfeng]

It can be confirmed that this is a problem, but there is no way to confirm whether 3.6 has fixed this, because our upgrade process is quite long. Can you explain the reason for this problem

[github-actions]

This issue is marked as stale because it has been open for 14 days with no activity.