Switch from vanilla Caddy to lucaslorentz/caddy-docker-proxy

[benpsnyder]

Problem

Caddy is amazing! Your current docker-compose.yml requires a custom mathesar/mathesar-caddy:latest image. This can be eliminated!

Proposed solution

Leverage the lucaslorentz/caddy-docker-proxy image and abandon the custom mathesar/mathesar-caddy:latest image. All we have to do is add labels to the mathesar/mathesar-prod:latest service definition.

Additional context

The implementation will look something like this. I will post back when I confirm what works for me.

version: '3'

services:
  ############################
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # this volume is needed to keep the certificates
      # otherwise, new ones will be re-issued upon restart
      - caddy_data:/data
    deploy:
      labels: # Global options
        caddy.email: systems@mydomain.com
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      restart_policy:
        condition: any
      resources:
        reservations:
          cpus: '0.2'
          memory: 200M
  mathesar:
    image: mathesar/mathesar-prod:latest
    expose:
      - 8000
    labels:
      caddy: msar.mydomain.com
      caddy.reverse_proxy: '{{upstreams 8000}}'
      caddy.encode: 'zstd gzip'
      caddy_0.handle_path: '/media/*'
      caddy_0.handle_path.@downloads.query: 'dl=*'
      caddy_0.handle_path.header_@downloads: 'Content-disposition "attachment; filename={query.dl}"'
      caddy_0.handle_path.file_server: ''
      caddy_0.handle_path.file_server.precompressed: 'br zstd gzip'
      caddy_0.handle_path.file_server.root: 'msar_media:/code/media'
      caddy_1.handle_path: '/static/*'
      caddy_1.handle_path.file_server: ''
      caddy_1.handle_path.file_server.precompressed: 'br zstd gzip'
      caddy_1.handle_path.file_server.root: 'msar_static:/code/static'
networks:
  caddy:
    driver: overlay
    attachable: true
volumes:
  caddy_data: {}
  msar_media: {}
  msar_static: {}

[seancolsen]

Thanks for suggesting this @benpsnyder. I'm tagging two of our maintainers who I think might have more thoughts/opinions/ideas about this...

@pavish and @Anish9901 can you respond here with your thoughts on this proposal?

[benpsnyder]

Here is my working configuration with environment variables (just replaced my company with ACME)

Please note I deploy with docker swarm init on a single host

version: '3'

services:
  ############################
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # this volume is needed to keep the certificates
      # otherwise, new ones will be re-issued upon restart
      - caddy_data:/data
    deploy:
      labels: # Global options
        caddy.email: systems@acme.com
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      restart_policy:
        condition: any
      resources:
        reservations:
          cpus: '0.2'
          memory: 200M
  ############################
  whoami:
    image: containous/whoami
    networks:
      - caddy
    expose:
      - 80
    deploy:
      replicas: 1
      restart_policy:
        condition: any
    labels:
      caddy: whoami.${ACME_DEPLOYMENT_DOMAIN_PARENT}
      caddy.reverse_proxy: '{{upstreams}}'
  mathesar:
    image: mathesar/mathesar-prod:latest
    environment:
      # First we load the variables configured above.
      # You can generate one at https://djecrety.ir/ or by running:
      SECRET_KEY: ${ACME_MSAR_SECRET_KEY}
      DOMAIN_NAME: ${ACME_MSAR_DOMAIN_NAME}
      POSTGRES_DB: ${ACME_MSAR_DB_NAME}
      POSTGRES_USER: ${ACME_MSAR_DB_USER}
      POSTGRES_PASSWORD: ${ACME_MSAR_DB_PASS}
      POSTGRES_HOST: ${ACME_MSAR_DB_HOST}
      POSTGRES_PORT: ${ACME_MSAR_DB_PORT}
      DJANGO_SETTINGS_MODULE: config.settings.production

      # We set ALLOWED_HOSTS to * (allow all hosts) by default here since we are
      # relying on caddy to manage which domains could access the mathesar web
      # service.  If you do not want to use caddy add the domain(s) that you
      # want to ALLOWED_HOSTS. Doing so will restrict traffic from all other
      # domains.
      ALLOWED_HOSTS: ${ALLOWED_HOSTS:-*}
      # WARNING: MATHESAR_DATABASES is deprecated, and will be removed in a future release.
      MATHESAR_DATABASES: ${MATHESAR_DATABASES:-}
    volumes:
      - msar_static:/code/static
      - msar_media:/code/.media
    healthcheck:
      test: curl -f http://mathesar:8000
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 5s
    # If using caddy, expose the internal port 8000 only to other containers and
    # not the docker host.
    networks:
      - caddy
    expose:
      - 8000
    labels:
      caddy: 'https://${ACME_DEPLOYMENT_SUBDOMAIN_MSAR}.${ACME_DEPLOYMENT_DOMAIN_PARENT}'
      caddy.reverse_proxy: '{{upstreams http 8000}}'
      caddy.encode: 'zstd gzip'
      caddy_0.handle_path: '/media/*'
      caddy_0.handle_path.@downloads.query: 'dl=*'
      caddy_0.handle_path.header_@downloads: 'Content-disposition "attachment; filename={query.dl}"'
      caddy_0.handle_path.file_server: ''
      caddy_0.handle_path.file_server.precompressed: 'br zstd gzip'
      caddy_0.handle_path.file_server.root: '/code/.media/'
      caddy_1.handle_path: '/static/*'
      caddy_1.handle_path.file_server: ''
      caddy_1.handle_path.file_server.precompressed: 'br zstd gzip'
      caddy_1.handle_path.file_server.root: '/code/static/'
networks:
  caddy:
    driver: overlay
    attachable: true
volumes:
  caddy_data: {}
  msar_media: {}
  msar_static: {}

[Anish9901]

Thanks for suggesting this @benpsnyder! I'll admit that this does look very cool, and I'm glad you got it working on your setup.

The thing is, we don't really expect our caddy configuration to change that often, nor do we expect our users to fiddle around with caddy's settings if we can provide something that works out of the box.

We expect docker-compose.yml to serve as a flexible blueprint, allowing users to customize it for deploying Mathesar on alternate setups like e.g. k8s. So, I don't think it's advantageous for us to use the suggested image instead of our own.

[benpsnyder]

Thanks for suggesting this @benpsnyder! I'll admit that this does look very cool, and I'm glad you got it working on your setup.

The thing is, we don't really expect our caddy configuration to change that often, nor do we expect our users to fiddle around with caddy's settings if we can provide something that works out of the box.

We expect docker-compose.yml to serve as a flexible blueprint, allowing users to customize it for deploying Mathesar on alternate setups like e.g. k8s. So, I don't think it's advantageous for us to use the suggested image instead of our own.

One of the chief reasons for suggesting this change is k8s and keeping total number of containers to a minimum 😎

Imagine if the majority of projects shipped an extra container just for the http ingress ... we'd have a bit of chaos with so much extra overhead.

I suggest we arrive at a solution using labels 🏷️ as this reduces overhead. Using this variant of a caddy image makes it incredibly easy to have one ingress container with labels on many apps and supporting infrastructure containers in one compose

If be happy to supply a traefik approach too, also with labels 🏷️, if the desired outcome is maximum k8s support

[Anish9901]

You won't even have to use Caddy when using Ingress! You'd directly point Ingress to the port(8000) which is exposed by mathesar/mathesar-prod:latest

Something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mathesar-ingress
spec:
  defaultBackend:
    service:
      name: mathesar
      port:
        number: 8000

EDIT:
@benpsnyder I did some more research, and It seems that you were talking about using docker swarm's ingress instead of k8s in which case using the suggested image would make sense if the goal is to reduce the number of containers. It seems that swarm's ingress behaves differently to the one used by k8s, in the sense that using caddy with k8s' ingress won't generate automatic SSL certificates, however, It seems that's not the case while using swarm's ingress.

Nonetheless, it looks like traefik can provide SSL certificates for both docker swarm's as well as k8s' ingress, thanks for suggesting this btw! It would be great if you could provide a config for traefik since you've already expressed interest in doing so.

[amca01]

Can you use this approach with normal, off the shelf, Caddy, and with an appropriate Caddyfile? This is what would work for me and my current setup. Many thanks!

[Anish9901]

Can you use this approach with normal, off the shelf, Caddy, and with an appropriate Caddyfile? This is what would work for me and my current setup.

Yes you most definitely can @amca01, here is the link to our Caddyfile.

Here is an example configuration using the Caddyfile mount in docker compose:

caddy-reverse-proxy:
    image: caddy:<version>
    # This service needs the config variables defined above.
    environment: *config
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - <path to Caddyfile on your machine>:/etc/caddy/Caddyfile # Your Caddyfile mount.
      - ./msar/media:/code/media
      - ./msar/static:/code/static
      - ./msar/caddy:/data

[amca01]

Thank you very much, @Anish9901. I do everything with docker compose files, and at the moment I'm using a plain "vanilla" Caddy, defined as

 caddy:
    container_name: caddy
    image: caddy:2.7.6
    restart: always
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    networks:
      - caddy_net
    volumes:
      - /home/amca/Docker/Caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/amca/Docker/Caddy/www:/srv
      - caddy_data:/data
      - caddy_config:/config

Do I have to add the Mathesar volumes as well to this Caddy block? I already, as you see, have a file structure which includes a Caddy subdirectory in my Docker directory. Do I need separate directories for Mathesar? Currently my Caddy data is kept in the bind mount "caddy_data"; would it be confusing to have two directories both pointing to Caddy's "/data"?

[Anish9901]

Thanks for providing this snippet @amca01, since you already have a caddy_data directory, you won't need to add ./msar/caddy:/data.

You'd, however, need to add the following as volumes:

• ./msar/media:/code/media (stores user uploaded datafiles(.csv/.tsv) to Mathesar)
• ./msar/static:/code/static (stores static files for Mathesar)

Please do note that, the paths mentioned above are relative and you may need to change ./msar/media & ./msar/static if you are running your caddy docker-compose file from a different directory to the one that was used to run mathesar's docker-compose file.

[amca01]

That makes a lot of sense, and again, many thanks @Anish9901. I'll give this a go tomorrow; right now (in Australia) it's bed time!

[amca01]

The current trouble is setting the POSTGRES_HOST environment variable. What it is supposed to be? Both the values of mathesar_db and my VPS hostname produce errors shown in docker logs mathesar_service. With value mathesar_db the error is

django.db.utils.OperationalError: could not translate host name "mathesar_db" to address: Name or service not known

I see from the example given by @benpsnyder above, that the value is to be set to ${ACME_MSAR_DB_HOST} - whatever that is!

Of course, the problem could be my setup; here are the relevant lines from my docker compose file:

  mathesar_service:
    container_name: mathesar_service
    image: mathesar/mathesar-prod:latest
    environment:
      SECRET_KEY: ${SECRET_KEY}
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_PORT: ${POSTGRES_PORT}
      POSTGRES_HOST: ${POSTGRES_HOST}
      DJANGO_SETTINGS_MODULE: config.settings.production
      ALLOWED_HOSTS: "*"
    entrypoint: ./run.sh
    volumes:
      - ./msar/static:/code/static
      - ./msar/media:/code/media
    depends_on:
      mathesar_db:
        condition: service_healthy
    healthcheck:
      test: curl -f http://localhost:8000
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 5s
    # If using caddy, expose the internal port 8000 only to other containers and
    # not the docker host.
    networks:
      - caddy_net
    expose:
      - "8000"

  mathesar_db:
    image: postgres:13
    container_name: mathesar_db
    # This service needs the config variables defined above.
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_PORT: ${POSTGRES_PORT}
    # Expose the internal port 5432 only to other containers and not 
    # the underlying host.
    expose:
      - "5432"
    volumes:
      - ./msar/pgdata:/var/lib/postgresql/data
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 5s
      timeout: 1s
      retries: 30
      start_period: 5s

and from my .env file (slightly redacted):

SECRET_KEY=<randomly generated>
DOMAIN_NAME=mathesar.mysite.net
POSTGRES_PASSWORD=mathesar
POSTGRES_DB=mathesar_django
POSTGRES_USER=mathesar
POSTGRES_PORT=5432
POSTGRES_HOST=mathesar_db

All of my files are in the same directory, and I know that Caddy is set up correctly as it works with other services. Should I be adding the mathesar_db container to the caddy_net network as well? I didn't think I needed to as there is not an outside connection to the database. But maybe a docker network is needed? Anyway, if there are any glaringly obvious errors in my files, I'd be glad to know of them! Many thanks.
Alasdair

[Anish9901]

@amca01 Since you are explicitly adding a network(caddy_net) for the mathesar_service container, it will be isolated from the db container and hence the error.

You can fix this by creating another network and adding it to the mathesar_service and mathesar_db containers.

Here is a similar setup that I've got working, please notice the networks section in the following snippets:

docker-compose.yml

service:
    container_name: mathesar_service
    image: mathesar/mathesar-prod:latest
    environment: 
      # First we load the variables configured above.
      <<: *config

      DJANGO_SETTINGS_MODULE: config.settings.production

      # We set ALLOWED_HOSTS to * (allow all hosts) by default here since we are
      # relying on caddy to manage which domains could access the mathesar web
      # service.  If you do not want to use caddy add the domain(s) that you
      # want to ALLOWED_HOSTS. Doing so will restrict traffic from all other
      # domains.
      ALLOWED_HOSTS: ${ALLOWED_HOSTS:-*}

      # WARNING: MATHESAR_DATABASES is deprecated, and will be removed in a future release.
      MATHESAR_DATABASES: ${MATHESAR_DATABASES:-}
    volumes:
      - ./msar/static:/code/static
      - ./msar/media:/code/media
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: curl -f http://localhost:8000
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 5s
    # If using caddy, expose the internal port 8000 only to other containers and
    # not the docker host.
    networks:
      - caddy_net_
      - mathesar_default
    expose:
      - "8000"
    # Uncomment the following if not using caddy
    # ports:
    #   - ${HOST_PORT:-8000}:8000

  #-----------------------------------------------------------------------------
  # PostgreSQL Database
  #
  # This service provides a Postgres database instance for holding both internal
  # Mathesar data, as well as user data if desired, using the official
  # PostgreSQL image hosted on Docker Hub
  #
  # As configured, this service exposes Postgres' default port (5432) to other
  # services, allowing the Mathesar web sevice to connect to it.
  #
  db:
    image: postgres:13
    container_name: mathesar_db
    # This service needs the config variables defined above.
    environment: *config
    # Expose the internal port 5432 only to other containers and not 
    # the underlying host.
    expose:
      - "5432"
    volumes:
      - ./msar/pgdata:/var/lib/postgresql/data
    networks:
      - mathesar_default
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 5s
      timeout: 1s
      retries: 30
      start_period: 5s
networks:
    caddy_net_:
      name: caddy_net
      external: true
    mathesar_default:
      name: mathesar_default
      external: true

caddy.yml

services:
  caddy_msar:
    container_name: caddy
    image: caddy:2.7.6
    # This service needs the config variables defined above.
    environment:
      DOMAIN_NAME: ${DOMAIN_NAME:-http://localhost}
    ports:
      - "80:80"
      - "443:443"
    networks:
      - caddy_net_
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./msar/media:/code/media
      - ./msar/static:/code/static
      - ./msar/caddy:/data
networks:
    caddy_net_:
      name: caddy_net
      external: true

[amca01]

Thank you so much, @Anish9901 - that works just as it should! Brilliant. I'm utterly delighted now to be able to run mathesar concurrently with my other services.

[Anish9901]

I'm glad it worked out @amca01 :)