in vhost.d nginx config for your container have something like

        error_page 503 = @custom_error;
        location @custom_error {
                internal;
                return 503 'please check back in a few moments';
        }

not 100% if its 503 or some other error you have to cover.

Great, very useful... But how do I get it so the error appears immediately in the browser. If the site isn't up and this gets triggered, then it seems the browser will always display this first:

Attackers might be trying to steal your information from [domain] (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID

Then you have to click twice to proceed.
Maybe you can't do custom errors when there are no vhosts up?

@dm17 I dont think you can do anything with that. Browser remembers that site is being loaded over HTTPS and if your site is down/certificate is invalid, user will get this error from browser.

Note that you might want to use the proxy_intercept_errors on; directive to intercept and redirect proxied error responses to nginx for processing with the error_page directive in order to prevent information exposure through error messages.