A fix for running newsblur behind an NGINX proxy

[xrd]

I want to run newsblur behind an nginx proxy. I've been struggling to do this.

It endlessly redirects when I hit the site in browser, incorrectly sending me to the http site again, which loops. But, curl from the docker host machine (hitting localhost) works on http and returns the content. Nginx does not work from a remote proxy machine.

The simple fix is to specify localhost in the proxy header:

       proxy_set_header Host 'localhost';

But, I suspect this is not the correct way to do it. I wanted to file this bug to further discussion.

Here is what I did:

1. Modify the docker-compose.yml to use the non-arm image processor. I'm using an amd64 machine (not macos).
2. Setup a custom domain using bash ./utils/custom_domain.sh mycustomdomainzzzzzzzz.com script.
3. Remove the SSL certs and configuration inside haproxy so it does not restart and only listens on non-https port. This means the macos-specific tools (like /usr/bin/security ) are no longer needed in generating certs.
4. Modify docker-compose.yml to use HTTP port at 10080 rather than 80 (avoiding conflicts with other services).
5. Turn off the SSL redirect inside the haproxy

I suspect there is something (perhaps in the python code?) that determines the server is answering on http (which is correct because I'm pointing to the http service) and then assumes (wrongly in this case) that it should issue a redirect to HTTPS, but it isn't doing that correctly. IMHO.

When I use curl -vvv http://mycustomdomainzzzzzzzzzz.com/ (from the proxy machine) it always issues a 302 back to the http service, not HTTPS.

curl -vvvv http://mycustomdomainzzzzzzzzzz.com/
...
< HTTP/1.1 302 Found
< Server: nginx/1.18.0 (Ubuntu)
< Date: Mon, 16 Oct 2023 13:26:12 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< location: http://mycustomdomainzzzzzzzzzz.com/

But, hitting the server from the docker host using localhost does respond correctly with the NewsBlur HTML.

$  curl -vvvv http://localhost:10080 | head -n 40
*   Trying 127.0.0.1:10080...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 10080 (#0)
> GET / HTTP/1.1
> Host: localhost:10080
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: gunicorn
< date: Mon, 16 Oct 2023 13:49:13 GMT
< content-type: text/html; charset=utf-8
< expires: Mon, 16 Oct 2023 13:49:13 GMT
< cache-control: max-age=0, no-cache, no-store, must-revalidate, private
< vary: Authorization, Cookie, Accept-Encoding
< x-gunicorn-server: nblocalhost
< x-marge: Get ready, skanks! It's time for the truth train!
< content-length: 72830
< strict-transport-security: max-age=0; includeSubDomains
< 
{ [55230 bytes data]
...

Interesting headers in there...

Here is the nginx configuration:

map $http_upgrade $connection_upgrade {  
   default      keep-alive;  
   'websocket'  upgrade;  
   ''           close;  
}  
  
server {  
       server_name  http://mycustomdomainzzzzzzzzzz.com;
       listen 80;  
       listen [::]:80;  
       listen 443 ssl;  
       listen [::]:443 ssl;  
       root /var/www/html;  
       ssl_certificate     /etc/letsencrypt/live/http://mycustomdomainzzzzzzzzzz.com/fullchain.pem;  
       ssl_certificate_key     /etc/letsencrypt/live/http://mycustomdomainzzzzzzzzzz.com/privkey.pem;
       ssl_protocols TLSv1.2 TLSv1.3;  
       # Add index.php to the list if you are using PHP  
       index index.html index.htm index.nginx-debian.html;  

       proxy_ssl_verify off;  
       proxy_set_header Host 'localhost';

       location /.well-known/acme-challenge/ {
         default_type "text/plain";
           # rewrite /.well-known/acme-challenge/(.*) /$1 break;
             root /var/www/letsencrypt/.well-known/acme-challenge/;
       }             

       location / {  
             proxy_pass http://100.64.0.8:10080;
             proxy_http_version 1.1;  
       }  


}  

Here is the diff from NewsBlur:master.

diff --git a/Makefile b/Makefile
index 57642ffde..e0e5be031 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
-SHELL := /bin/bash
+SHELL := bash
 CURRENT_UID := $(shell id -u)
 CURRENT_GID := $(shell id -g)
-newsblur := $(shell gtimeout 2s docker ps -qf "name=newsblur_web")
+newsblur := $(shell timeout 2s docker ps -qf "name=newsblur_web")
 
 .PHONY: node
 
@@ -83,7 +83,7 @@ keys:
        openssl req -new -nodes -newkey rsa:2048 -keyout config/certificates/localhost.key -out config/certificates/localhost.csr -subj "/C=US/ST=YourState/L=YourCity/O=Example-Certificates/CN=localhost"
        openssl x509 -req -sha256 -days 1024 -in config/certificates/localhost.csr -CA config/certificates/RootCA.pem -CAkey config/certificates/RootCA.key -CAcreateserial -out config/certificates/localhost.crt
        cat config/certificates/localhost.crt config/certificates/localhost.key > config/certificates/localhost.pem
-       sudo /usr/bin/security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./config/certificates/RootCA.crt
+       # sudo /usr/bin/security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./config/certificates/RootCA.crt
 
 # Doesn't work yet
 mkcert:
diff --git a/config/fixtures/bootstrap.json b/config/fixtures/bootstrap.json
index 591ea4d90..37963aaeb 100644
--- a/config/fixtures/bootstrap.json
+++ b/config/fixtures/bootstrap.json
@@ -3,7 +3,7 @@
         "pk": 1,
         "model": "sites.site",
         "fields": {
-            "domain": "localhost",
+            "domain": "mycustomdomainzzzzzzzzzz.com",
             "name": "NewsBlur"
         }
     },    
diff --git a/docker-compose.yml b/docker-compose.yml
index 0482e9835..605d316fa 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,7 +2,7 @@ version: '2'
 services:
 
   newsblur_web:
-    hostname: nb.com
+    hostname: nb.com  
     container_name: newsblur_web
     image: newsblur/newsblur_${NEWSBLUR_BASE:-python3}:latest
     # build: 
@@ -53,8 +53,8 @@ services:
   
   imageproxy:
     container_name: imageproxy
-    # image: ghcr.io/willnorris/imageproxy:latest # Enable if you don't need arm64 and want the original imageproxy
-    image: yusukeito/imageproxy:v0.11.2 # Enable if you want arm64 
+    image: ghcr.io/willnorris/imageproxy:latest # Enable if you don't need arm64 and want the original imageproxy
+    # image: yusukeito/imageproxy:v0.11.2 # Enable if you want arm64 
     user: "${CURRENT_UID}:${CURRENT_GID}"
     entrypoint: /app/imageproxy -addr 0.0.0.0:8088 -cache /tmp/imageproxy -verbose
     restart: unless-stopped
@@ -68,7 +68,7 @@ services:
     image: nginx:1.19.6
     restart: unless-stopped
     ports:
-      - 81:81
+      - 10081:81
     depends_on:
       - newsblur_web
       - newsblur_node
@@ -169,9 +169,9 @@ services:
       - db_elasticsearch
       - db_mongo
     ports:
-      - 80:80
-      - 443:443
-      - 1936:1936
+      - 10080:80
+      # - 10443:443
+      - 11936:1936
     volumes:
       - ./docker/haproxy/haproxy.docker-compose.cfg:/usr/local/etc/haproxy/haproxy.cfg
       - ${PWD}:/srv/newsblur
diff --git a/docker/haproxy/haproxy.docker-compose.cfg b/docker/haproxy/haproxy.docker-compose.cfg
index 3bdbf221c..0e3b03c27 100644
--- a/docker/haproxy/haproxy.docker-compose.cfg
+++ b/docker/haproxy/haproxy.docker-compose.cfg
@@ -1,11 +1,11 @@
 global
     maxconn 100000
     daemon
-    ca-base /srv/newsblur/config/certificates
-    crt-base /srv/newsblur/config/certificates
+#    ca-base /srv/newsblur/config/certificates
+#    crt-base /srv/newsblur/config/certificates
     tune.bufsize 32000
     tune.maxrewrite 8196
-    tune.ssl.default-dh-param 2048
+#    tune.ssl.default-dh-param 2048
     log 127.0.0.1 local0 notice
     # log 127.0.0.1 local1 info
 
@@ -32,13 +32,13 @@ defaults
 
 frontend public
     bind :80
-    bind :443 ssl crt /srv/newsblur/config/certificates/localhost.pem #ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES128-SHA:AES256-SHA256:AES256-SHA no-sslv3
+#    bind :443 ssl crt /srv/newsblur/config/certificates/localhost.pem #ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES128-SHA:AES256-SHA256:AES256-SHA no-sslv3
     http-response add-header Strict-Transport-Security max-age=0;\ includeSubDomains
     option http-server-close
 
     # Redirect all HTTP traffic to HTTPS
-    acl is_root path /
-    redirect scheme https if is_root !{ ssl_fc }
+    # acl is_root path /
+    # redirect scheme https if is_root !{ ssl_fc }
     
     acl gunicorn_dead nbsrv(gunicorn) lt 1
     acl nginx_dead nbsrv(nginx) lt 1
@@ -126,7 +126,7 @@ backend elasticsearch
     server db_elasticsearch db_elasticsearch:9200 check inter 2000ms
 
 listen stats
-bind :1936 ssl crt /srv/newsblur/config/certificates/localhost.pem
+# bind :1936 ssl crt /srv/newsblur/config/certificates/localhost.pem
 stats enable
 stats hide-version
 stats realm Haproxy\ Statistics
diff --git a/newsblur_web/docker_local_settings.py b/newsblur_web/docker_local_settings.py
index 31e353f54..b4f006fab 100644
--- a/newsblur_web/docker_local_settings.py
+++ b/newsblur_web/docker_local_settings.py
@@ -11,9 +11,9 @@ ADMINS                = (
 
 SERVER_EMAIL          = 'server@newsblur.com'
 HELLO_EMAIL           = 'hello@newsblur.com'
-NEWSBLUR_URL          = 'https://localhost'
+NEWSBLUR_URL          = 'https://mycustomdomainzzzzzzzzzz.com'
 PUSH_DOMAIN           = 'localhost'
-SESSION_COOKIE_DOMAIN = 'localhost'
+SESSION_COOKIE_DOMAIN = 'mycustomdomainzzzzzzzzzz.com'
 
 # ===================
 # = Global Settings =
@@ -180,10 +180,10 @@ DO_TOKEN_LOG = '0000000000000000000000000000000000000000000000000000000000000000
 DO_TOKEN_FABRIC = '0000000000000000000000000000000000000000000000000000000000000000'
 
 SERVER_NAME = "nblocalhost"
-NEWSBLUR_URL = os.getenv("NEWSBLUR_URL", "https://localhost")
+NEWSBLUR_URL = os.getenv("NEWSBLUR_URL", "https://mycustomdomainzzzzzzzzzz.com")
 
-if NEWSBLUR_URL == 'https://localhost':
-    SESSION_COOKIE_DOMAIN = "localhost"
+if NEWSBLUR_URL == 'https://mycustomdomainzzzzzzzzzz.com':
+    SESSION_COOKIE_DOMAIN = "mycustomdomainzzzzzzzzzz.com"
 
 SESSION_ENGINE = 'redis_sessions.session'
 

[TobiaszCudnik]

I got to the same point and it seems like newsblur_web is doing 302 http://DOMAIN instead of simply serving the content.

Have you managed to find the solution? Im really starting to doubt if newsblur is the right choice after seeing how the config, deployment and IoC is handled. Its actually surprising it works at all...

[xrd]

Well for me that proxy fix (noted above) works great. I'm not happy with having to do it that way, but it's working great. And, newsblur is fantastic in so many ways with so much functionality that it feels like a little thing.

[samuelclay]

Are you hitting this line:

NewsBlur/docker/haproxy/haproxy.docker-compose.cfg

Lines 39 to 41 in ed1a2c4

	# Redirect all HTTP traffic to HTTPS
	acl is_root path /
	redirect scheme https if is_root !{ ssl_fc }

[xrd]

I'm assuming yes