Nil entity returned when using dao select statement on Kong 3.3 Postgres

[evelynchua5772]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong 3.3

Current Behavior

After upgrading plugins from Kong 2.2.1 to Kong 3.3, under the following conditions:

1. I have a sample piece of lua code in a custom plugin that utilizes the kong cache function

local credential_cache_key = kong.db.oauth2_credentials:cache_key(client_id)
        client, err = kong.cache:get(credential_cache_key, nil,
            load_oauth2_credential_by_client_id,
            client_id)

where the callback function to retrieve the entity to be loaded into the cache is a dao select function

local function load_oauth2_credential_by_client_id(client_id)
    local credential, err = kong.db.oauth2_credentials:select_by_client_id(client_id)
    if err then
        return nil, err
    end

    return credential
end

2. Kong is deployed with the custom plugin in Kubernetes with more than one instance.

More often than not, after the initial start up and during the plugin execution, kong will hit the following error

[error] 1261#0: *4864420 [kong] init.lua:359 [partner-custom-oauth2] /opt/kong/plugins/partner-custom-oauth2/access.lua:1071: attempt to index local 'client' (a nil value)

indicating that the value in the cache/entity returned from the select statement is nil, even though a manual db query indicates that the entity exists but ws_id is null.

After a few kong restarts, it is noticed that the error goes away and when checking the oauth2_credentials table the ws_id is populated.

When turning on OpenTelemetry logs, it is noted that the dao select statement queries ws_id as well, and having ws_id as null could then be an issue

SELECT
  id,
  EXTRACT(EPOCH FROM created_at AT TIME ZONE 'UTC') AS created_at,
  expires_in,
  access_token,
  ws_id,
  FLOOR(EXTRACT(EPOCH FROM (ttl AT TIME ZONE 'UTC' - CURRENT_TIMESTAMP AT TIME ZONE 'UTC'))) AS ttl
.... (columns scrubbed to remove sensitive information)
FROM oauth2_tokens
WHERE
  access_token = 'jkCFYnyzN2gumQI9lbKgYTuHE63XNXDO'
  AND (ttl IS NULL OR ttl >= CURRENT_TIMESTAMP AT TIME ZONE 'UTC')
  AND (ws_id = '6c431e2e-020c-4e4c-af58-a25f21b2d5de')
LIMIT 1;

Anyone has had a similar issue or know the cause? What causes the ws_id to be null in the first place

Expected Behavior

No response

Steps To Reproduce

No response

Anything else?

No response

[chobits]

@bungle could you give some advice to this issue?

also related to #12597

[samugi]

Hello @evelynchua5772, just to check whether my understanding is correct: has the issue stopped occurring once the oauth2_tokens rows finished updating to include the workspace id? I am still looking into this but I believe, if that is the case, this might be expected and the result of migrating from an older version that did not have the ws_id field.