Can't start postgresql 16 over tls #12702

michael-todorovic · 2024-03-07T16:03:50Z

Is there an existing issue for this?

I have searched the existing issues

Kong version (`$ kong version`)

3.6.1

Current Behavior

I'm currently migrating from pg 13 to 16. Kong 3.6.1 (docker image) worked well on pg13 but when I try to start or just run a kong migrations list on pg16 (bypassing pgbouncer to reduce potential issues), I get:

2024/03/07 15:39:21 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/03/07 15:39:21 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/03/07 15:39:21 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default.c2024/03/07 15:39:21 [warn] 1298#0: *2 [lua] nginx.lua:300: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
Error: 
/usr/local/share/lua/5.1/pgmoon/init.lua:398: attempt to index local 'ssl' (a nil value)
stack traceback:
	/usr/local/share/lua/5.1/pgmoon/init.lua:398: in function 'auth'
	/usr/local/share/lua/5.1/pgmoon/init.lua:268: in function 'connect'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:215: in function 'connect'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:546: in function 'query'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:296: in function 'init'
	/usr/local/share/lua/5.1/kong/db/init.lua:144: in function 'init_connector'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
	(command line -e):7: in function 'inline_gen'
	init_worker_by_lua(nginx.conf:136):44: in function <init_worker_by_lua(nginx.conf:136):43>
	[C]: in function 'xpcall'
	init_worker_by_lua(nginx.conf:136):52: in function <init_worker_by_lua(nginx.conf:136):50>
rt) and key (/usr/local/kong/ssl/admin-gui-kong-default.key) for listener
2024/03/07 15:39:21 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.key) for listener
2024/03/07 15:39:21 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined

I'm using postgres 16.2-1.pgdg120+2 on Debian 12. The TLS config itself is ok:

╰ sslscan --starttls-psql master-postgres.domain.com:5431
Version: 2.0.7
OpenSSL 3.0.11 19 Sep 2023

Connected to 10.154.192.7

Testing SSL server master-postgres.domain.com on port 5431 using SNI name master-postgres.domain.com

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   enabled

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
OpenSSL version does not support compression
Rebuild with zlib1g-dev package for zlib support

  Heartbleed:
TLSv1.3 not vulnerable to heartbleed
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve P-256 DHE 256
Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve P-256 DHE 256
Accepted  TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve P-256 DHE 256
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-GCM-SHA384     DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-CHACHA20-POLY1305     DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM8           DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM            DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-ARIA256-GCM-SHA384      Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-ARIA256-GCM-SHA384    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-GCM-SHA256     DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM8           DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM            DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-ARIA128-GCM-SHA256      Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-ARIA128-GCM-SHA256    DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA384       Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA256         DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CAMELLIA256-SHA384  Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA256    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA256         DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-CAMELLIA128-SHA256  Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA256    DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA       DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA       DHE 2048 bits
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384            
Accepted  TLSv1.2  256 bits  AES256-CCM8                  
Accepted  TLSv1.2  256 bits  AES256-CCM                   
Accepted  TLSv1.2  256 bits  ARIA256-GCM-SHA384           
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256            
Accepted  TLSv1.2  128 bits  AES128-CCM8                  
Accepted  TLSv1.2  128 bits  AES128-CCM                   
Accepted  TLSv1.2  128 bits  ARIA128-GCM-SHA256           
Accepted  TLSv1.2  256 bits  AES256-SHA256                
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA256           
Accepted  TLSv1.2  128 bits  AES128-SHA256                
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA256           
Accepted  TLSv1.2  256 bits  AES256-SHA                   
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA              
Accepted  TLSv1.2  128 bits  AES128-SHA                   
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA              

  Server Key Exchange Group(s):
TLSv1.3  128 bits  secp256r1 (NIST P-256)
TLSv1.2  128 bits  secp256r1 (NIST P-256)

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  patroni-main-xxxx.domain.com
Altnames: DNS:master-postgres.domain.com, DNS:replica-postgres.domain.Com, DNS:patroni-main-xxxx.domain.com
Issuer:   R3

Not valid before: Mar  7 08:54:32 2024 GMT
Not valid after:  Jun  5 08:54:31 2024 GMT

I checked, just in case, the docker image supports those TLS versions+ciphers.
I tried:

an old 3.5.0 on pg16: worked ok!
a newer 3.6.0 on pg16: failed
latest 3.6.1 on pg16: failed

On each try, this was the same pg16 cluster of course 😄
I tried to open pgmoon lua code but can't really understand what can be an issue nor how to get more details.
I also checked what went at the network level with tcpdump/wireshark but nothing was really showing.
On postgres side, I don't have any logs about the failure either.

Do you have an idea how to make progress debugging this issue?
Thanks for your help!

Expected Behavior

No response

Steps To Reproduce

I run:

docker run --name kong --rm -e KONG_ADMIN_ACCESS_LOG="/dev/stdout json_admin" -e KONG_ADMIN_LISTEN=0.0.0.0:8001 -e KONG_DATABASE=postgres -e KONG_PG_DATABASE=kong361 -e KONG_PG_HOST=master-postgres.domain.com -e KONG_PG_USER=kong_user -e KONG_PLUGINS=bundled,jwt-consumer-to-users -e KONG_PROXY_ACCESS_LOG="/dev/stdout json_proxy" -e KONG_PROXY_ERROR_LOG=/dev/stdout -e KONG_PROXY_LISTEN=0.0.0.0:8000 -e KONG_PG_SSL="on" -e KONG_PG_PORT=5431 -e KONG_PG_PASSWORD=xxx -e KONG_LOG_LEVEL=debug kong:3.6.1 kong migrations list -vv
We're using patroni 3.2.2 + postgres 16.2 on Debian 12. Here's the generated postgres config:

root@patroni-main-xxx:~# cat /var/lib/postgresql/16/main/data/postgresql.conf
# Do not edit this file manually!
# It will be overwritten by Patroni!
include 'postgresql.base.conf'

auto_explain.log_analyze = 'True'
auto_explain.log_format = 'json'
auto_explain.log_min_duration = '-1'
auto_explain.log_nested_statements = 'True'
auto_explain.log_parameter_max_length = '0'
cluster_name = 'int-patroni-kong-g2'
effective_cache_size = '1957MB'
hot_standby = 'on'
listen_addresses = '0.0.0.0'
log_connections = 'False'
log_destination = 'jsonlog'
log_directory = '/var/log/patroni'
log_disconnections = 'True'
log_duration = 'True'
log_filename = 'postgresql'
log_hostname = 'True'
log_lock_waits = 'True'
log_min_duration_statement = '0'
log_min_error_statement = 'DEBUG1'
log_min_messages = 'DEBUG1'
log_replication_commands = 'True'
log_statement = 'mod'
log_temp_files = '204800'
log_timezone = 'Etc/UTC'
logging_collector = 'True'
maintenance_work_mem = '391MB'
max_connections = '100'
max_locks_per_transaction = '64'
max_prepared_transactions = '0'
max_replication_slots = '10'
max_standby_archive_delay = '30s'
max_standby_streaming_delay = '30s'
max_wal_senders = '10'
max_worker_processes = '8'
port = '5431'
shared_buffers = '978MB'
shared_preload_libraries = 'auto_explain,pg_stat_statements'
ssl = 'on'
ssl_cert_file = '/etc/company/certs/patroni-main-xxx.domain.com_crt.pem'
ssl_key_file = '/etc/company/certs/patroni-main-xxx.domain.com_key.pem'
track_commit_timestamp = 'off'
unix_socket_group = 'psql-socket'
unix_socket_permissions = '504'
wal_keep_size = '128MB'
wal_level = 'replica'
wal_log_hints = 'on'
hba_file = '/var/lib/postgresql/16/main/data/pg_hba.conf'
ident_file = '/var/lib/postgresql/16/main/data/pg_ident.conf'

# recovery.conf
recovery_target = ''
recovery_target_lsn = ''
recovery_target_name = ''
recovery_target_time = ''
recovery_target_timeline = 'latest'
recovery_target_xid = ''

Anything else?

No response

The text was updated successfully, but these errors were encountered:

sgrzemski · 2024-03-10T12:59:45Z

I bet that's because of the 0.8.25 to 1.2.0 upgrade of resty-openssl mentioned here: https://docs.konghq.com/gateway/changelog/#3600 and introduced here: #12265.
That's a bummer. It's sad to see, but Kong's stability with the new releases is surprisingly low and the introduced versions are full of smaller bugs. E.g. I have to stay on 3.4.2, because 3.5.X does not tolerate custom logging format and I cannot upgrade to 3.6.1, because of the SSL issue.

walter-bd · 2024-03-15T13:22:08Z

I have the same problem but with postgres 15

bungle · 2024-03-18T08:23:08Z

Yes, in 3.6.0 there was a couple of big bumps on dependencies. OpenSSL and OpenResty. We need to check what is going on. Thank you for reporting, unfortunately we didn't catch this.

bungle · 2024-04-15T07:56:19Z

Just link some context here:
https://github.com/Kong/pgmoon/blob/v1.16.2/pgmoon/init.lua#L397-L398

local ssl = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`
local server_cert = ssl:get_peer_certificate()

@michael-todorovic, could you modify that line (in most probably /usr/local/share/lua/5.1/pgmoon/init.lua):

local ssl = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`

to:

local ssl, err = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`
if err then
  error(err)
end

And report back?

bungle · 2024-04-15T08:02:59Z

Also @fffonion could you take a look at it. Is the auxilary module compiled with CE? Is it a requirement now? Is there difference in EE shipped pgmoon regards to this?

pmorelli92 · 2024-04-19T11:01:00Z

Any updates on this? I got the same, working on 3.5 and stopped with 3.6 and 3.6.1. Leaving my trace just in case :)

KONG_PG_DATABASE=xxx \
KONG_PG_HOST=xxxxx \
KONG_PG_PASSWORD=xxxx \
KONG_PG_PORT=5432 \
KONG_PG_SSL=on \
KONG_PG_SSL_REQUIRED=on \
KONG_PG_SSL_VERSION=tlsv1_3 \
KONG_PG_USER=kong kong migrations up --v

On 3.5:

2024/04/19 10:53:19 [verbose] Kong: 3.5.0
2024/04/19 10:53:19 [verbose] no config file found at /etc/kong/kong.conf
2024/04/19 10:53:19 [verbose] no config file found at /etc/kong.conf
2024/04/19 10:53:19 [verbose] no config file, skip loading
2024/04/19 10:53:19 [verbose] prefix in use: /usr/local/kong
2024/04/19 10:53:19 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/04/19 10:53:19 [verbose] SSL enabled on proxy, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating proxy SSL certificate (/usr/local/kong/ssl/kong-default.crt) and key (/usr/local/kong/ssl/kong-default.key) for listener
2024/04/19 10:53:19 [verbose] generating proxy SSL certificate (/usr/local/kong/ssl/kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/kong-default-ecdsa.key) for listener
2024/04/19 10:53:19 [verbose] SSL enabled on admin, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating admin SSL certificate (/usr/local/kong/ssl/admin-kong-default.crt) and key (/usr/local/kong/ssl/admin-kong-default.key) for listener
2024/04/19 10:53:19 [verbose] generating admin SSL certificate (/usr/local/kong/ssl/admin-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-kong-default-ecdsa.key) for listener
2024/04/19 10:53:19 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default.key) for listener
2024/04/19 10:53:20 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.key) for listener
2024/04/19 10:53:20 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
2024/04/19 10:53:20 [warn] 15#0: *2 [lua] nginx.lua:261: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
2024/04/19 10:53:20 [verbose] retrieving database schema state...
2024/04/19 10:53:20 [verbose] schema state retrieved
2024/04/19 10:53:20 [verbose] retrieving database schema state...
2024/04/19 10:53:20 [verbose] schema state retrieved
2024/04/19 10:53:20 [info] Database is already up-to-date

On 3.6.1:

2024/04/19 10:51:50 [verbose] Kong: 3.6.1
2024/04/19 10:51:50 [verbose] no config file found at /etc/kong/kong.conf
2024/04/19 10:51:50 [verbose] no config file found at /etc/kong.conf
2024/04/19 10:51:50 [verbose] no config file, skip loading
2024/04/19 10:51:50 [verbose] prefix in use: /usr/local/kong
2024/04/19 10:51:50 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/04/19 10:51:50 [verbose] SSL enabled on proxy, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default.crt
2024/04/19 10:51:50 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] SSL enabled on admin, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt
2024/04/19 10:51:50 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default.crt
2024/04/19 10:51:50 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
2024/04/19 10:51:50 [warn] 64#0: *2 [lua] nginx.lua:300: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
Error: 
/usr/local/share/lua/5.1/pgmoon/init.lua:398: attempt to index local 'ssl' (a nil value)
stack traceback:
	/usr/local/share/lua/5.1/pgmoon/init.lua:398: in function 'auth'
	/usr/local/share/lua/5.1/pgmoon/init.lua:268: in function 'connect'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:215: in function 'connect'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:546: in function 'query'
	.../share/lua/5.1/kong/db/strategies/postgres/connector.lua:296: in function 'init'
	/usr/local/share/lua/5.1/kong/db/init.lua:144: in function 'init_connector'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
	(command line -e):7: in function 'inline_gen'
	init_worker_by_lua(nginx.conf:170):44: in function <init_worker_by_lua(nginx.conf:170):43>
	[C]: in function 'xpcall'
	init_worker_by_lua(nginx.conf:170):52: in function <init_worker_by_lua(nginx.conf:170):50>

fffonion · 2024-04-23T08:20:24Z

We are working on fix right now : ) Thanks for you patience.

weberpatr · 2024-05-06T10:02:18Z

We are working on fix right now : ) Thanks for you patience.

Alerady some updates on this topic?
We're currently running into the same issue after updating to 3.6.x

chronolaw added the core/db/migrations label Mar 8, 2024

bungle added the task/needs-investigation Requires investigation and reproduction before classifying it as a bug or not. label Mar 18, 2024

bungle assigned fffonion Apr 15, 2024

chronolaw added the pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... label Apr 15, 2024

chronolaw removed the pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... label Apr 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't start postgresql 16 over tls #12702

Can't start postgresql 16 over tls #12702

michael-todorovic commented Mar 7, 2024 •

edited

sgrzemski commented Mar 10, 2024

walter-bd commented Mar 15, 2024

bungle commented Mar 18, 2024 •

edited

bungle commented Apr 15, 2024 •

edited

bungle commented Apr 15, 2024

pmorelli92 commented Apr 19, 2024

fffonion commented Apr 23, 2024

weberpatr commented May 6, 2024

Can't start postgresql 16 over tls #12702

Can't start postgresql 16 over tls #12702

Comments

michael-todorovic commented Mar 7, 2024 • edited

Is there an existing issue for this?

Kong version ($ kong version)

Current Behavior

Expected Behavior

Steps To Reproduce

Anything else?

sgrzemski commented Mar 10, 2024

walter-bd commented Mar 15, 2024

bungle commented Mar 18, 2024 • edited

bungle commented Apr 15, 2024 • edited

bungle commented Apr 15, 2024

pmorelli92 commented Apr 19, 2024

fffonion commented Apr 23, 2024

weberpatr commented May 6, 2024

michael-todorovic commented Mar 7, 2024 •

edited

Kong version (`$ kong version`)

bungle commented Mar 18, 2024 •

edited

bungle commented Apr 15, 2024 •

edited