Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with permissions during the deploy when using a legacy herokuish version #6813

Open
dbagaric opened this issue Apr 9, 2024 · 2 comments

Comments

@dbagaric
Copy link

dbagaric commented Apr 9, 2024

Description of problem

Getting error on deployment:
`remote: =====> Start healthcheck output                                                                
remote:        state=restarting                                                                        
remote: =====> End healthcheck output                                                                  
remote: 53b6fdd76fa5095386013d0697133526ef0eb5de8bf5f0d3f83034f48da5225a        
remote:  !     Could not start due to 1 failed checks (web.1)        
remote: =====> Start of profimont container output (web.1)        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote: =====> End of profimont container output (web.1)        
remote: =====> Start of  container output (.)        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote:        setuidgid: fatal: unable to setgid: permission denied        
remote: =====> End of  container output (.)        
remote: parallel: This job failed:        
remote: /var/lib/dokku/plugins/available/scheduler-docker-local/bin/scheduler-deploy-process-container profimont herokuish dokku/profimont:latest latest web 1 1 `

Steps to reproduce

Deploying the app using:
git push dokku master
It is legacy project and using gliderlabs/herokuish:v0.3.24

dokku report $APP_NAME

-----> uname: Linux apps 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                      total        used        free      shared  buff/cache   available
       Mem:           15988        3551         700          11       11735       12087
       Swap:              0           0           0
-----> disk utilization: 
       Filesystem      Size  Used Avail Use% Mounted on
       /dev/vda1       310G   44G  267G  15% /
       /dev/vda15      105M  6.1M   99M   6% /boot/efi
-----> disk inode utilization: 
       Filesystem     Inodes IUsed IFree IUse% Mounted on
       /dev/vda1         40M  2.4M   37M    7% /
       /dev/vda15          0     0     0     - /boot/efi
-----> docker version: 
       Client: Docker Engine - Community
        Version:           26.0.0
        API version:       1.45
        Go version:        go1.21.8
        Git commit:        2ae903e
        Built:             Wed Mar 20 15:17:48 2024
        OS/Arch:           linux/amd64
        Context:           default
       
       Server: Docker Engine - Community
        Engine:
         Version:          26.0.0
         API version:      1.45 (minimum version 1.24)
         Go version:       go1.21.8
         Git commit:       8b79278
         Built:            Wed Mar 20 15:17:48 2024
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.28
         GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
        runc:
         Version:          1.1.12
         GitCommit:        v1.1.12-0-g51d5e94
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client: Docker Engine - Community
        Version:    26.0.0
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.13.1
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.25.0
           Path:     /usr/libexec/docker/cli-plugins/docker-compose
       
       Server:
        Containers: 18
         Running: 18
         Paused: 0
         Stopped: 0
        Images: 35
        Server Version: 26.0.0
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
        Swarm: inactive
        Runtimes: runc io.containerd.runc.v2
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
        runc version: v1.1.12-0-g51d5e94
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 5.15.0-101-generic
        Operating System: Ubuntu 22.04.4 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 8
        Total Memory: 15.61GiB
        Name: apps
        ID: 4d71e495-6acf-431c-a61a-3b05c11e2367
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 134
         Goroutines: 117
         System Time: 2024-04-09T18:15:08.028497932Z
         EventsListeners: 1
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> herokuish version: 
       herokuish: v0.9.1
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v267
         heroku-buildpack-nodejs    v240
         heroku-buildpack-clojure   v91
         heroku-buildpack-python    v246
         heroku-buildpack-java      v73
         heroku-buildpack-gradle    v39
         heroku-buildpack-scala     v98
         heroku-buildpack-play      v26
         heroku-buildpack-php       v248
         heroku-buildpack-go        v189
         heroku-buildpack-nginx     v25
         buildpack-null             v3
-----> dokku version: dokku version 0.34.4
-----> dokku-event-listener version: 0.15.0build+5268732                                                             
-----> dokku-update version: dokku-update 0.8.0
-----> docker-container-healthchecker version: 0.9.0                                                                           
-----> docker-image-labeler version: 0.6.1build+c6e15a9                                                              
-----> git version: git version 2.34.1
-----> lambda-builder version:        0.6.0                                                                           
-----> netrc version: 0.8.0build+0751c1b
 !     pack binary is not available
-----> plugn version: plugn: 0.13.0build+fd5297a
-----> sigil version: 0.10.1build+e443be0
-----> sshcommand version: sshcommand 0.17.1
-----> dokku plugins: 
         00_dokku-standard    0.34.4 enabled    dokku core standard plugin
         20_events            0.34.4 enabled    dokku core events logging plugin
         app-json             0.34.4 enabled    dokku core app-json plugin
         apps                 0.34.4 enabled    dokku core apps plugin
         builder              0.34.4 enabled    dokku core builder plugin
         builder-dockerfile   0.34.4 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.34.4 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.34.4 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.34.4 enabled    dokku core builder-nixpacks plugin
         builder-null         0.34.4 enabled    dokku core builder-null plugin
         builder-pack         0.34.4 enabled    dokku core builder-pack plugin
         buildpacks           0.34.4 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.34.4 enabled    dokku core caddy-vhosts plugin
         certs                0.34.4 enabled    dokku core certificate management plugin
         checks               0.34.4 enabled    dokku core checks plugin
         common               0.34.4 enabled    dokku core common plugin
         config               0.34.4 enabled    dokku core config plugin
         cron                 0.34.4 enabled    dokku core cron plugin
         docker-options       0.34.4 enabled    dokku core docker-options plugin
         domains              0.34.4 enabled    dokku core domains plugin
         enter                0.34.4 enabled    dokku core enter plugin
         git                  0.34.4 enabled    dokku core git plugin
         haproxy-vhosts       0.34.4 enabled    dokku core haproxy-vhosts plugin
         http-auth            0.10.0 enabled    HTTP authentication for apps
         letsencrypt          0.20.3 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.34.4 enabled    dokku core logs plugin
         maintenance          0.8.0 enabled    Maintenance mode for apps
         mongo                1.36.9 enabled    dokku mongo service plugin
         mysql                1.38.1 enabled    dokku mysql service plugin
         network              0.34.4 enabled    dokku core network plugin
         nginx-vhosts         0.34.4 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.34.4 enabled    dokku core openresty-vhosts plugin
         plugin               0.34.4 enabled    dokku core plugin plugin
         ports                0.34.4 enabled    dokku core ports plugin
         postgres             1.36.4 enabled    dokku postgres service plugin
         proxy                0.34.4 enabled    dokku core proxy plugin
         ps                   0.34.4 enabled    dokku core ps plugin
         redirect             0.9.1 enabled    Plugin for managing application redirects
         redis                1.37.4 enabled    dokku redis service plugin
         registry             0.34.4 enabled    dokku core registry plugin
         repo                 0.34.4 enabled    dokku core repo plugin
         resource             0.34.4 enabled    dokku core resource plugin
         run                  0.34.4 enabled    dokku core run plugin
         scheduler            0.34.4 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.34.4 enabled    dokku core scheduler-docker-local plugin
         scheduler-k3s        0.34.4 enabled    dokku core scheduler-k3s plugin
         scheduler-null       0.34.4 enabled    dokku core scheduler-null plugin
         shell                0.34.4 enabled    dokku core shell plugin
         slack                0.4.0 enabled    Slack notifier on deploy
         ssh-keys             0.34.4 enabled    dokku core ssh-keys plugin
         storage              0.34.4 enabled    dokku core storage plugin
         trace                0.34.4 enabled    dokku core trace plugin
         traefik-vhosts       0.34.4 enabled    dokku core traefik-vhosts plugin
=====> profimont app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> profimont app information
       App created at:                1704891243
       App deploy source:             git-push
       App deploy source metadata:    c839abfbe43cb1f0975578e11a30a54c8e1ea959
       App dir:                       /home/dokku/profimont
       App locked:                    false
=====> profimont builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> profimont builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> profimont builder-herokuish information
       Builder herokuish computed allowed: true                     
       Builder herokuish global allowed: true                     
       Builder herokuish allowed:                              
=====> profimont builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml               
       Builder lambda global lambdayml path: lambda.yml               
       Builder lambda lambdayml path:                          
=====> profimont builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml            
       Builder nixpacks global nixpackstoml path: nixpacks.toml            
       Builder nixpacks nixpackstoml path:                          
       Builder nixpacks computed no cache: false                    
       Builder nixpacks global no cache: false                    
       Builder nixpacks no cache:                              
=====> profimont builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> profimont buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:v0.3.24
       Buildpacks global stack:       
       Buildpacks list:               https://github.com/dbagaric/heroku-buildpack-ruby
       Buildpacks stack:              gliderlabs/herokuish:v0.3.24
=====> profimont caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.8
       Caddy letsencrypt email:                                
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR                    
       Caddy polling interval:        5s                       
       Caddy tls internal:            false                    
=====> profimont ssl information
       Ssl dir:                       /home/dokku/profimont/tls
       Ssl enabled:                   true                     
       Ssl hostnames:                 rn.profimont.com         
       Ssl expires at:                Jul  6 19:23:35 2024 GMT 
       Ssl issuer:                    C = US, O = Let's Encrypt, CN = R3
       Ssl starts at:                 Apr  7 19:23:36 2024 GMT 
       Ssl subject:                   subject=CN = rn.profimont.com
       Ssl verified:                  verified by a certificate authority
=====> profimont checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
       Checks computed wait to retire: 60                       
       Checks global wait to retire:  60                       
       Checks wait to retire:                                  
=====> profimont docker options information
       Docker options build:                                   
       Docker options deploy:         --restart=on-failure:10  
       Docker options run:                                     
=====> profimont domains information
       Domains app enabled:           true                     
       Domains app vhosts:            rn.profimont.com         
       Domains global enabled:        true                     
       Domains global vhosts:         beap.pro                 
=====> profimont git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       c839abfbe43cb1f0975578e11a30a54c8e1ea959
       Git source image:                                       
       Git last updated at:           1704956173               
=====> profimont haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.4.0  
       Haproxy letsencrypt email:                              
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR                    
=====> profimont http-auth information
       Http auth enabled:             false                    
       Http auth allowed ips:                                  
       Http auth users:                                        
=====> profimont letsencrypt information
       Letsencrypt active:            true                     
       Letsencrypt autorenew:         true                     
       Letsencrypt computed dns provider:                          
       Letsencrypt global dns provider:                          
       Letsencrypt dns provider:                               
       Letsencrypt computed email:    services@beap.co         
       Letsencrypt global email:                               
       Letsencrypt email:             services@beap.co         
       Letsencrypt expiration:        1720293815               
       Letsencrypt computed graceperiod: 2592000                  
       Letsencrypt global graceperiod:                          
       Letsencrypt graceperiod:                                
       Letsencrypt computed lego docker args:                          
       Letsencrypt global lego docker args:                          
       Letsencrypt lego docker args:                           
       Letsencrypt computed server:   https://acme-v02.api.letsencrypt.org/directory
       Letsencrypt global server:                              
       Letsencrypt server:                                     
=====> profimont logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector global image:      timberio/vector:0.36.1-debian
       Logs vector sink:              
=====> profimont maintenance information
       Maintenance enabled:           false                    
=====> profimont network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.7:5000
=====> profimont nginx information
       Nginx access log format:                                
       Nginx computed access log format:                          
       Nginx global access log format:                          
       Nginx access log path:                                  
       Nginx computed access log path: /var/log/nginx/profimont-access.log
       Nginx global access log path:  /var/log/nginx/profimont-access.log
       Nginx bind address ipv4:                                
       Nginx computed bind address ipv4:                          
       Nginx global bind address ipv4:                          
       Nginx bind address ipv6:                                
       Nginx computed bind address ipv6: ::                       
       Nginx global bind address ipv6: ::                       
       Nginx client max body size:                             
       Nginx computed client max body size: 1m                       
       Nginx global client max body size: 1m                       
       Nginx disable custom config:                            
       Nginx computed disable custom config: false                    
       Nginx global disable custom config: false                    
       Nginx error log path:                                   
       Nginx computed error log path: /var/log/nginx/profimont-error.log
       Nginx global error log path:   /var/log/nginx/profimont-error.log
       Nginx hsts include subdomains:                          
       Nginx computed hsts include subdomains: true                     
       Nginx global hsts include subdomains: true                     
       Nginx hsts max age:                                     
       Nginx computed hsts max age:   15724800                 
       Nginx global hsts max age:     15724800                 
       Nginx hsts preload:                                     
       Nginx computed hsts preload:   false                    
       Nginx global hsts preload:     false                    
       Nginx hsts:                                             
       Nginx computed hsts:           true                     
       Nginx global hsts:             true                     
       Nginx last visited at:                                  
       Nginx nginx conf sigil path:                            
       Nginx computed nginx conf sigil path: nginx.conf.sigil         
       Nginx global nginx conf sigil path: nginx.conf.sigil         
       Nginx proxy buffer size:                                
       Nginx computed proxy buffer size: 4k                       
       Nginx global proxy buffer size: 4k                       
       Nginx proxy buffering:                                  
       Nginx computed proxy buffering: on                       
       Nginx global proxy buffering:  on                       
       Nginx proxy buffers:                                    
       Nginx computed proxy buffers:  8 4k                     
       Nginx global proxy buffers:    8 4k                     
       Nginx proxy busy buffers size:                          
       Nginx computed proxy busy buffers size: 8k                       
       Nginx global proxy busy buffers size: 8k                       
       Nginx proxy read timeout:                               
       Nginx computed proxy read timeout: 60s                      
       Nginx global proxy read timeout: 60s                      
       Nginx underscore in headers:                            
       Nginx computed underscore in headers: off                      
       Nginx global underscore in headers: off                      
       Nginx x forwarded for value:                            
       Nginx computed x forwarded for value: $remote_addr             
       Nginx global x forwarded for value: $remote_addr             
       Nginx x forwarded port value:                           
       Nginx computed x forwarded port value: $server_port             
       Nginx global x forwarded port value: $server_port             
       Nginx x forwarded proto value:                          
       Nginx computed x forwarded proto value: $scheme                  
       Nginx global x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
       Nginx computed x forwarded ssl:                          
       Nginx global x forwarded ssl:                           
=====> profimont openresty information
       Openresty access log format:                            
       Openresty access log path:     /var/log/nginx/profimont-access.log
       Openresty allowed letsencrypt domains func base64: cmV0dXJuIHRydWUK         
       Openresty bind address ipv4:                            
       Openresty bind address ipv6:   ::                       
       Openresty client max body size:                          
       Openresty error log path:      /var/log/nginx/profimont-error.log
       Openresty global hsts:         true                     
       Openresty computed hsts:       true                     
       Openresty hsts:                                         
       Openresty hsts include subdomains: true                     
       Openresty hsts max age:        15724800                 
       Openresty hsts preload:        false                    
       Openresty image:               dokku/openresty-docker-proxy:0.7.0
       Openresty letsencrypt email:                            
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4k                       
       Openresty proxy buffering:     on                       
       Openresty proxy buffers:       8 4k                     
       Openresty proxy busy buffers size: 8k                       
       Openresty proxy read timeout:  60s                      
       Openresty underscore in headers: off                      
       Openresty x forwarded for value: $remote_addr             
       Openresty x forwarded port value: $server_port             
       Openresty x forwarded proto value: $scheme                  
       Openresty x forwarded ssl:                              
=====> profimont ports information
       Ports map:                     
       Ports map detected:            http:80:5000 https:443:5000
=====> profimont proxy information
       Proxy computed type:           nginx
       Proxy enabled:                 true
       Proxy global type:             nginx
       Proxy type:                    
=====> profimont ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 2be2d797716)
=====> profimont registry information
       Registry computed image repo:        dokku/profimont
       Registry computed push on release:   false
       Registry computed server:            
       Registry global image repo template: 
       Registry global push on release:     
       Registry global server:              
       Registry image repo:                 
       Registry push extra tags:            
       Registry push on release:            
       Registry server:                     
       Registry tag version:                
=====> profimont resource information
=====> profimont scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> profimont scheduler-docker-local information
       Scheduler docker local init process: true                     
       Scheduler docker local parallel schedule count:                          
=====> profimont scheduler-k3s information
       Scheduler k3s computed deploy timeout:       300s
       Scheduler k3s computed image pull secrets:   
       Scheduler k3s computed letsencrypt server:   prod
       Scheduler k3s computed namespace:            default
       Scheduler k3s computed rollback on failure:  false
       Scheduler k3s deploy timeout:                
       Scheduler k3s global deploy timeout:         300s
       Scheduler k3s global image pull secrets:     
       Scheduler k3s global ingress class:          nginx
       Scheduler k3s global kube context:           
       Scheduler k3s global kubeconfig path:        /etc/rancher/k3s/k3s.yaml
       Scheduler k3s global letsencrypt email prod: 
       Scheduler k3s global letsencrypt email stag: 
       Scheduler k3s global letsencrypt server:     prod
       Scheduler k3s global namespace:              default
       Scheduler k3s global network interface:      eth0
       Scheduler k3s global rollback on failure:    false
       Scheduler k3s image pull secrets:            
       Scheduler k3s letsencrypt server:            
       Scheduler k3s namespace:                     
       Scheduler k3s rollback on failure:           
=====> profimont storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:                                     
=====> profimont traefik information
       Traefik api enabled:           false                    
       Traefik api vhost:             traefik.dokku.me         
       Traefik basic auth password:                            
       Traefik basic auth username:                            
       Traefik dashboard enabled:     false                    
       Traefik image:                 traefik:2.11.0           
       Traefik letsencrypt email:                              
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR 

Additional information

No response

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

@josegonzalez josegonzalez changed the title Issue with permissions during the deploy "setuidgid: fatal: unable to setgid: permission denied " Issue with permissions during the deploy when using a legacy herokuish version Apr 9, 2024
@danielschnee
Copy link

Having the same with herokuish:latest-22

remote: =====> Start of elephouse release task (e162d57ea) output        
remote:  !     /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote: =====> End of elephouse release task (e162d57ea) output        
remote: =====> Processing deployment checks        
remote:  !     No healthchecks found in app.json for web process type                                  
remote:        No web healthchecks found in app.json. Simple container checks will be performed.        
remote:        For more efficient zero downtime deployments, add healthchecks to your app.json. See https://dokku.com/docs/deployment/zero-downtime-deploys/ for examples        
remote: -----> Deploying elephouse via the docker-local scheduler...        
remote: -----> Deploying web (count=1)        
remote:        Attempting pre-flight checks (web.1)        
remote: -----> Executing 2 healthchecks                                                                
remote:        Running healthcheck name='default' type='uptime' uptime=10                              
remote:        Running healthcheck name='port listening check' attempts=3 port=5000 retries=2 timeout=5 type='listening' wait=5        
remote:  !     Failure in name='default': container has restarted 7 times                              
remote: =====> Start healthcheck output                                                                
remote:        state=restarting                                                                        
remote: =====> End healthcheck output                                                                  
remote:  !     Failure in name='port listening check': container state is not running                  
remote: c36d98e4df985a8b686785574a82efade2ce21d152be93897395fe019a3f9af2        
remote:  !     Could not start due to 1 failed checks (web.1)        
remote: =====> Start of elephouse container output (web.1)        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote: =====> End of elephouse container output (web.1)        
remote: =====> Start of  container output (.)        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote:        /app/.profile.d/100-php.sh: line 19: /app/.heroku/php/etc/php/php-cli.ini: Permission denied        
remote: =====> End of  container output (.)        
remote: parallel: This job failed:  

dokku report

-----> uname: Linux projects 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                     total        used        free      shared  buff/cache   available
       Mem:           1923         481         435          32        1006        1218
       Swap:             0           0           0
-----> disk utilization: 
       Filesystem      Size  Used Avail Use% Mounted on
       /dev/sda1        38G   31G  5.0G  87% /
       /dev/sda15      253M  6.1M  247M   3% /boot/efi
-----> disk inode utilization: 
       Filesystem     Inodes IUsed IFree IUse% Mounted on
       /dev/sda1        2.4M  990K  1.4M   42% /
       /dev/sda15          0     0     0     - /boot/efi
-----> docker version: 
       Client: Docker Engine - Community
        Version:           26.0.0
        API version:       1.45
        Go version:        go1.21.8
        Git commit:        2ae903e
        Built:             Wed Mar 20 15:17:51 2024
        OS/Arch:           linux/amd64
        Context:           default
       
       Server: Docker Engine - Community
        Engine:
         Version:          26.0.0
         API version:      1.45 (minimum version 1.24)
         Go version:       go1.21.8
         Git commit:       8b79278
         Built:            Wed Mar 20 15:17:51 2024
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.28
         GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
        runc:
         Version:          1.1.12
         GitCommit:        v1.1.12-0-g51d5e94
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client: Docker Engine - Community
WARNING: No swap limit support
        Version:    26.0.0
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.13.1
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.25.0
           Path:     /usr/libexec/docker/cli-plugins/docker-compose
         scan: Docker Scan (Docker Inc.)
           Version:  v0.23.0
           Path:     /usr/libexec/docker/cli-plugins/docker-scan
       
       Server:
        Containers: 7
         Running: 5
         Paused: 0
         Stopped: 2
        Images: 36
        Server Version: 26.0.0
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Cgroup Version: 1
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
        runc version: v1.1.12-0-g51d5e94
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
        Kernel Version: 5.4.0-176-generic
        Operating System: Ubuntu 20.04.6 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 1.878GiB
        Name: projects
        ID: 6K26:HLSL:IASM:6XN5:JTON:B3TH:GDSA:UOB7:HTGL:5AGS:G7T5:C7ZU
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 56
         Goroutines: 65
         System Time: 2024-04-11T07:25:55.218527738Z
         EventsListeners: 1
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> herokuish version: 
       herokuish: v0.9.1
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v267
         heroku-buildpack-nodejs    v240
         heroku-buildpack-clojure   v91
         heroku-buildpack-python    v246
         heroku-buildpack-java      v73
         heroku-buildpack-gradle    v39
         heroku-buildpack-scala     v98
         heroku-buildpack-play      v26
         heroku-buildpack-php       v248
         heroku-buildpack-go        v189
         heroku-buildpack-nginx     v25
         buildpack-null             v3
-----> dokku version: dokku version 0.34.4
-----> dokku-event-listener version: 0.15.0build+5268732                                                             
-----> dokku-update version: dokku-update 0.8.0
-----> docker-container-healthchecker version: 0.9.0                                                                           
-----> docker-image-labeler version: 0.6.1build+c6e15a9                                                              
-----> git version: git version 2.25.1
-----> lambda-builder version:        0.6.0                                                                           
-----> netrc version: 0.8.0build+0751c1b
 !     pack binary is not available
-----> plugn version: plugn: 0.13.0build+fd5297a
-----> sigil version: 0.10.1build+e443be0
-----> sshcommand version: sshcommand 0.17.1
-----> dokku plugins: 
         00_dokku-standard    0.34.4 enabled    dokku core standard plugin
         20_events            0.34.4 enabled    dokku core events logging plugin
         app-json             0.34.4 enabled    dokku core app-json plugin
         apps                 0.34.4 enabled    dokku core apps plugin
         builder              0.34.4 enabled    dokku core builder plugin
         builder-dockerfile   0.34.4 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.34.4 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.34.4 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.34.4 enabled    dokku core builder-nixpacks plugin
         builder-null         0.34.4 enabled    dokku core builder-null plugin
         builder-pack         0.34.4 enabled    dokku core builder-pack plugin
         buildpacks           0.34.4 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.34.4 enabled    dokku core caddy-vhosts plugin
         certs                0.34.4 enabled    dokku core certificate management plugin
         checks               0.34.4 enabled    dokku core checks plugin
         common               0.34.4 enabled    dokku core common plugin
         config               0.34.4 enabled    dokku core config plugin
         cron                 0.34.4 enabled    dokku core cron plugin
         docker-options       0.34.4 enabled    dokku core docker-options plugin
         domains              0.34.4 enabled    dokku core domains plugin
         enter                0.34.4 enabled    dokku core enter plugin
         git                  0.34.4 enabled    dokku core git plugin
         haproxy-vhosts       0.34.4 enabled    dokku core haproxy-vhosts plugin
         http-auth            0.10.0 enabled    HTTP authentication for apps
         letsencrypt          0.20.3 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.34.4 enabled    dokku core logs plugin
         mysql                1.38.1 enabled    dokku mysql service plugin
         network              0.34.4 enabled    dokku core network plugin
         nginx-vhosts         0.34.4 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.34.4 enabled    dokku core openresty-vhosts plugin
         plugin               0.34.4 enabled    dokku core plugin plugin
         ports                0.34.4 enabled    dokku core ports plugin
         postgres             1.36.4 enabled    dokku postgres service plugin
         proxy                0.34.4 enabled    dokku core proxy plugin
         ps                   0.34.4 enabled    dokku core ps plugin
         redirect             0.9.1 enabled    Plugin for managing application redirects
         registry             0.34.4 enabled    dokku core registry plugin
         repo                 0.34.4 enabled    dokku core repo plugin
         resource             0.34.4 enabled    dokku core resource plugin
         run                  0.34.4 enabled    dokku core run plugin
         scheduler            0.34.4 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.34.4 enabled    dokku core scheduler-docker-local plugin
         scheduler-k3s        0.34.4 enabled    dokku core scheduler-k3s plugin
         scheduler-null       0.34.4 enabled    dokku core scheduler-null plugin
         shell                0.34.4 enabled    dokku core shell plugin
         ssh-keys             0.34.4 enabled    dokku core ssh-keys plugin
         storage              0.34.4 enabled    dokku core storage plugin
         trace                0.34.4 enabled    dokku core trace plugin
         traefik-vhosts       0.34.4 enabled    dokku core traefik-vhosts plugin
=====> elephouse app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> elephouse app information
       App created at:                1712779228
       App deploy source:             
       App deploy source metadata:    
       App dir:                       /home/dokku/elephouse
       App locked:                    false
=====> elephouse builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> elephouse builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> elephouse builder-herokuish information
       Builder herokuish computed allowed: true                     
       Builder herokuish global allowed: true                     
       Builder herokuish allowed:                              
=====> elephouse builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml               
       Builder lambda global lambdayml path: lambda.yml               
       Builder lambda lambdayml path:                          
=====> elephouse builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml            
       Builder nixpacks global nixpackstoml path: nixpacks.toml            
       Builder nixpacks nixpackstoml path:                          
       Builder nixpacks computed no cache: false                    
       Builder nixpacks global no cache: false                    
       Builder nixpacks no cache:                              
=====> elephouse builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> elephouse buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-22
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> elephouse caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.8
       Caddy letsencrypt email:                                
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR                    
       Caddy polling interval:        5s                       
       Caddy tls internal:            false                    
=====> elephouse ssl information
       Ssl dir:                       /home/dokku/elephouse/tls
       Ssl enabled:                   false                    
       Ssl hostnames:                                          
       Ssl expires at:                                         
       Ssl issuer:                                             
       Ssl starts at:                                          
       Ssl subject:                                            
       Ssl verified:                                           
=====> elephouse checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
       Checks computed wait to retire: 60                       
       Checks global wait to retire:  60                       
       Checks wait to retire:                                  
=====> elephouse docker options information
       Docker options build:          --link dokku.postgres.elephouse-db:dokku-postgres-elephouse-db 
       Docker options deploy:         --link dokku.postgres.elephouse-db:dokku-postgres-elephouse-db --restart=on-failure:10 
       Docker options run:            --link dokku.postgres.elephouse-db:dokku-postgres-elephouse-db 
=====> elephouse domains information
       Domains app enabled:           true                     
       Domains app vhosts:            elephouse.de             
       Domains global enabled:        true                     
       Domains global vhosts:         services.snowprojects.de 
=====> elephouse git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       HEAD                     
       Git source image:                                       
       Git last updated at:                                    
=====> elephouse haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.4.0  
       Haproxy letsencrypt email:                              
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR                    
=====> elephouse http-auth information
       Http auth enabled:             false                    
       Http auth allowed ips:                                  
       Http auth users:                                        
Can't open /home/dokku/elephouse/tls/server.crt for reading, No such file or directory
140659767883072:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/home/dokku/elephouse/tls/server.crt','r')
140659767883072:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load certificate
=====> elephouse letsencrypt information
       Letsencrypt active:            false                    
       Letsencrypt autorenew:         true                     
       Letsencrypt computed dns provider:                          
       Letsencrypt global dns provider:                          
       Letsencrypt dns provider:                               
       Letsencrypt computed email:    letsencrypt@snowprojects.de
       Letsencrypt global email:      letsencrypt@snowprojects.de
       Letsencrypt email:                                      
       Letsencrypt expiration:        1712793600               
       Letsencrypt computed graceperiod: 2592000                  
       Letsencrypt global graceperiod:                          
       Letsencrypt graceperiod:                                
       Letsencrypt computed lego docker args:                          
       Letsencrypt global lego docker args:                          
       Letsencrypt lego docker args:                           
       Letsencrypt computed server:   https://acme-v02.api.letsencrypt.org/directory
       Letsencrypt global server:                              
       Letsencrypt server:                                     
=====> elephouse logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector global image:      timberio/vector:0.36.1-debian
       Logs vector sink:              
=====> elephouse network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                
=====> elephouse nginx information
       Nginx access log format:                                
       Nginx computed access log format:                          
       Nginx global access log format:                          
       Nginx access log path:                                  
       Nginx computed access log path: /var/log/nginx/elephouse-access.log
       Nginx global access log path:  /var/log/nginx/elephouse-access.log
       Nginx bind address ipv4:                                
       Nginx computed bind address ipv4:                          
       Nginx global bind address ipv4:                          
       Nginx bind address ipv6:                                
       Nginx computed bind address ipv6: ::                       
       Nginx global bind address ipv6: ::                       
       Nginx client max body size:                             
       Nginx computed client max body size: 1m                       
       Nginx global client max body size: 1m                       
       Nginx disable custom config:                            
       Nginx computed disable custom config: false                    
       Nginx global disable custom config: false                    
       Nginx error log path:                                   
       Nginx computed error log path: /var/log/nginx/elephouse-error.log
       Nginx global error log path:   /var/log/nginx/elephouse-error.log
       Nginx hsts include subdomains:                          
       Nginx computed hsts include subdomains: true                     
       Nginx global hsts include subdomains: true                     
       Nginx hsts max age:                                     
       Nginx computed hsts max age:   15724800                 
       Nginx global hsts max age:     15724800                 
       Nginx hsts preload:                                     
       Nginx computed hsts preload:   false                    
       Nginx global hsts preload:     false                    
       Nginx hsts:                                             
       Nginx computed hsts:           true                     
       Nginx global hsts:             true                     
       Nginx last visited at:                                  
       Nginx nginx conf sigil path:                            
       Nginx computed nginx conf sigil path: nginx.conf.sigil         
       Nginx global nginx conf sigil path: nginx.conf.sigil         
       Nginx proxy buffer size:                                
       Nginx computed proxy buffer size: 4k                       
       Nginx global proxy buffer size: 4k                       
       Nginx proxy buffering:                                  
       Nginx computed proxy buffering: on                       
       Nginx global proxy buffering:  on                       
       Nginx proxy buffers:                                    
       Nginx computed proxy buffers:  8 4k                     
       Nginx global proxy buffers:    8 4k                     
       Nginx proxy busy buffers size:                          
       Nginx computed proxy busy buffers size: 8k                       
       Nginx global proxy busy buffers size: 8k                       
       Nginx proxy read timeout:                               
       Nginx computed proxy read timeout: 60s                      
       Nginx global proxy read timeout: 60s                      
       Nginx underscore in headers:                            
       Nginx computed underscore in headers: off                      
       Nginx global underscore in headers: off                      
       Nginx x forwarded for value:                            
       Nginx computed x forwarded for value: $remote_addr             
       Nginx global x forwarded for value: $remote_addr             
       Nginx x forwarded port value:                           
       Nginx computed x forwarded port value: $server_port             
       Nginx global x forwarded port value: $server_port             
       Nginx x forwarded proto value:                          
       Nginx computed x forwarded proto value: $scheme                  
       Nginx global x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
       Nginx computed x forwarded ssl:                          
       Nginx global x forwarded ssl:                           
=====> elephouse openresty information
       Openresty access log format:                            
       Openresty access log path:     /var/log/nginx/elephouse-access.log
       Openresty allowed letsencrypt domains func base64: cmV0dXJuIHRydWUK         
       Openresty bind address ipv4:                            
       Openresty bind address ipv6:   ::                       
       Openresty client max body size:                          
       Openresty error log path:      /var/log/nginx/elephouse-error.log
       Openresty global hsts:         true                     
       Openresty computed hsts:       true                     
       Openresty hsts:                                         
       Openresty hsts include subdomains: true                     
       Openresty hsts max age:        15724800                 
       Openresty hsts preload:        false                    
       Openresty image:               dokku/openresty-docker-proxy:0.7.0
       Openresty letsencrypt email:                            
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4k                       
       Openresty proxy buffering:     on                       
       Openresty proxy buffers:       8 4k                     
       Openresty proxy busy buffers size: 8k                       
       Openresty proxy read timeout:  60s                      
       Openresty underscore in headers: off                      
       Openresty x forwarded for value: $remote_addr             
       Openresty x forwarded port value: $server_port             
       Openresty x forwarded proto value: $scheme                  
       Openresty x forwarded ssl:                              
=====> elephouse ports information
       Ports map:                     
       Ports map detected:            http:80:5000
=====> elephouse proxy information
       Proxy computed type:           nginx
       Proxy enabled:                 true
       Proxy global type:             nginx
       Proxy type:                    
=====> elephouse ps information
       Deployed:                      false
       Processes:                     0
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       false
=====> elephouse registry information
       Registry computed image repo:        dokku/elephouse
       Registry computed push on release:   false
       Registry computed server:            
       Registry global image repo template: 
       Registry global push on release:     
       Registry global server:              
       Registry image repo:                 
       Registry push extra tags:            
       Registry push on release:            
       Registry server:                     
       Registry tag version:                
=====> elephouse resource information
=====> elephouse scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> elephouse scheduler-docker-local information
       Scheduler docker local init process: true                     
       Scheduler docker local parallel schedule count:                          
=====> elephouse scheduler-k3s information
       Scheduler k3s computed deploy timeout:       300s
       Scheduler k3s computed image pull secrets:   
       Scheduler k3s computed letsencrypt server:   prod
       Scheduler k3s computed namespace:            default
       Scheduler k3s computed rollback on failure:  false
       Scheduler k3s deploy timeout:                
       Scheduler k3s global deploy timeout:         300s
       Scheduler k3s global image pull secrets:     
       Scheduler k3s global ingress class:          nginx
       Scheduler k3s global kube context:           
       Scheduler k3s global kubeconfig path:        /etc/rancher/k3s/k3s.yaml
       Scheduler k3s global letsencrypt email prod: 
       Scheduler k3s global letsencrypt email stag: 
       Scheduler k3s global letsencrypt server:     prod
       Scheduler k3s global namespace:              default
       Scheduler k3s global network interface:      eth0
       Scheduler k3s global rollback on failure:    false
       Scheduler k3s image pull secrets:            
       Scheduler k3s letsencrypt server:            
       Scheduler k3s namespace:                     
       Scheduler k3s rollback on failure:           
=====> elephouse storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:                                     
=====> elephouse traefik information
       Traefik api enabled:           false                    
       Traefik api vhost:             traefik.dokku.me         
       Traefik basic auth password:                            
       Traefik basic auth username:                            
       Traefik dashboard enabled:     false                    
       Traefik image:                 traefik:2.11.0           
       Traefik letsencrypt email:                              
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR                    

@josegonzalez
Copy link
Member

@danielschnee mind jumping on our discord/slack for your particular issue? This is specifically for cases where the herokuish version is configured to be a (much) older version than the latest that should work fine, whereas your case seems a bit different since you're actually using the latest herokuish release (and latest dokku release).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants