NPE in JavaHttpClient

[raymanoz]

I was attempting to get an API Token for Graph API, via a proxy, using the JavaHttpClient. Unfortunately, the password for the account had changed, so the password I was supplying was incorrect.

This caused a 407, but most importantly, the body of the response was null.

According to the javadoc for java.net.http.HttpResponse, it is quite possible for body() to return null. This needs to be considered & handled in JavaHttpClient.invoke(Request)

[s4nchez]

@raymanoz I'm struggling to reproduce this in the http4k tests (server sending no body seems fine). Do you happen to have a stack trace for it, or are able to provide an example of client + server that shows that behaviour?

[raymanoz]

Hi Ivan, I had trouble actually writing a test in http4k to reproduce it. I'll dig through my code, and see what I can find. Cheers, Raymond

…

On Thu, 28 Jan 2021 at 20:48, Ivan Sanchez ***@***.***> wrote: @raymanoz <https://github.com/raymanoz> I'm struggling to reproduce this in the http4k tests (server sending no body seems fine). Do you happen to have a stack trace for it, or are able to provide an example of client + server that shows that behaviour? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#566 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFA3O3JJH5DLK3TDIIA2LS4E6HBANCNFSM4WWQBUUQ> .

[raymanoz]

I managed to recreate the stack trace (below).

java.lang.NullPointerException: null
                at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
                at org.http4k.client.JavaHttpClient.invoke(JavaHttpClient.kt:36)
                at org.http4k.client.JavaHttpClient.invoke(JavaHttpClient.kt:28)
                at org.http4k.filter.ClientFilters$CustomBasicAuth$invoke$1$1.invoke(ClientFilters.kt:96)
                at org.http4k.filter.ClientFilters$CustomBasicAuth$invoke$1$1.invoke(ClientFilters.kt:94)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt:15)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt)
                at **************.filter.AccessTokenFilter.requestNewToken(AccessTokenFilter.kt:43)
                at **************.filter.AccessTokenFilter.token(AccessTokenFilter.kt:37)
                at **************.filter.AccessTokenFilter.invoke(AccessTokenFilter.kt:30)
                at **************.filter.AccessTokenFilter.invoke(AccessTokenFilter.kt:22)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt:15)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt)
                at **************.AzureServiceServerKt.downloadAndStoreImages(AzureServiceServer.kt:81)
                at **************.AzureServiceServerKt.access$downloadAndStoreImages(AzureServiceServer.kt:1)
                at **************.AzureServiceServerKt$cacheImages$1.invoke(AzureServiceServer.kt:67)
                at **************.AzureServiceServerKt$cacheImages$1.invoke(AzureServiceServer.kt)
                at org.http4k.filter.ServerFilters$CatchLensFailure$2$1.invoke(ServerFilters.kt:239)
                at org.http4k.filter.ServerFilters$CatchLensFailure$2$1.invoke(ServerFilters.kt:49)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt:15)
                at org.http4k.core.Http4kKt$then$2.invoke(Http4k.kt)
                at org.http4k.routing.TemplateRouter$match$1.invoke(Router.kt:112)
                at org.http4k.routing.TemplateRouter$match$1.invoke(Router.kt:108)
                at org.http4k.routing.RouterMatch$MatchingHandler.invoke(Router.kt:39)
                at org.http4k.routing.RouterMatch$MatchingHandler.invoke(Router.kt:38)
                at org.http4k.routing.RouterBasedHttpHandler.invoke(RouterBasedHttpHandler.kt:24)
                at org.http4k.routing.RouterBasedHttpHandler.invoke(RouterBasedHttpHandler.kt:13)
                at org.http4k.servlet.Http4kServletAdapter.handle(Http4kServletAdapter.kt:19)
                at org.http4k.servlet.HttpHandlerServlet.service(HttpHandlerServlet.kt:14)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
                at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
                at org.eclipse.jetty.server.Server.handle(Server.java:516)
                at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
                at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
                at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
                at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
                at java.base/java.lang.Thread.run(Thread.java:834)

[raymanoz]

I think this is caused by something to do with (from java.net.http.HttpResponse):

    /**
     * Returns the body. Depending on the type of {@code T}, the returned body
     * may represent the body after it was read (such as {@code byte[]}, or
     * {@code String}, or {@code Path}) or it may represent an object with
     * which the body is read, such as an {@link java.io.InputStream}.
     *
     * <p> If this {@code HttpResponse} was returned from an invocation of
     * {@link #previousResponse()} then this method returns {@code null}
     *
     * @return the body
     */
    public T body();

[raymanoz]

I suspect something like this would expose the error, but I cannot set body to null (doesn't compile):

fun main() {
    val proxy = { _: Request -> Response(PROXY_AUTHENTICATION_REQUIRED).body(null) } //compile error
    proxy.asServer(SunHttp(9000)).start()

    val proxyAuthedClient = ClientFilters.ProxyBasicAuth(Credentials("thehammer", "ninjasrule")).
        then(JavaHttpClient())
    proxyAuthedClient(Request(Method.GET, "http://localhost:9000/"))
}

[s4nchez]

In http4k, there's no such thing as a null body. I've played using SunHttp directly and not sending a body..

fun createSimpleServer() {
    val httpServer = HttpServer.create(InetSocketAddress(9000), 1000)
    httpServer.createContext("/") { exchange ->
        exchange.sendResponseHeaders(407, -1)
        exchange.responseBody.close()
    }
    httpServer.executor = Executors.newWorkStealingPool()
    httpServer.start()
}

And it also doesn't trigger the NPE behaviour :(

Which version of http4k was that on? Maybe is something we've already fixed?

[raymanoz]

I was on 3.278.0, so I just tried upgrading to 4.3.2.2. Unfortunately, I am still getting the NPE in the same spot.

Line 37, JavaHttpClient.kt, the body() (which is ultimately jdk.internal.net.http.HttpResponseImpl.body(), line 112) returns null, which causes ByteBuffer.wrap to throw an NPE.

Happy to pair on it, if we can line up a time.

[s4nchez]

A few things that may help:

• Are you using the default internal java client or a custom one? The docs mention redirection/authentication but we try to turn all those features off by default.
• Are you able to capture the raw responses (HTTP message payloads) that the server sends that are triggering this behaviour?

We could easily just put a guard in the body() call but would be nice to put together an example.

[raymanoz]

I am using the default internal java client.

Here is the "raw response" as output by coreResponse().toMessage():

HTTP/1.1 407
cache-control: no-cache
connection: close
content-length: 971
content-type: text/html; charset=utf-8
pragma: no-cache
proxy-authenticate: NEGOTIATE
proxy-authenticate: NTLM
proxy-authenticate: BASIC realm="IWA_Realm"
proxy-connection: close

[s4nchez]

Did the server actually send a body? Your example has content-length: 971 which is suspicious. Here's how I'm testing

fun main() {
    val response = Response.parse(
"""HTTP/1.1 407 ignored
cache-control: no-cache
connection: close
content-length: 0
content-type: text/html; charset=utf-8
pragma: no-cache
proxy-authenticate: NEGOTIATE
proxy-authenticate: NTLM
proxy-authenticate: BASIC realm="IWA_Realm"
proxy-connection: close

""".replace("\n", "\r\n"))

    val server = { _: Request -> response }.asServer(SunHttp(8000))

    server.start()

    val client = JavaHttpClient()

    println(client(Request(Method.GET, "http://localhost:${server.port()}")))

    server.stop()
}

This times out because the client keeps waiting for bytes. Without the content-length header or set to zero it works (no NPE yet)

[raymanoz]

Well, when I did a toMessage(), no body was shown, so I guess not. I had also noticed it had a content-length with no body.

I recreated the request, and tried it in curl to see how it handled it. The result is below (with redacted bits). You'll notice that there is a content-length and no body in this too. But, it does say "Ignore 921 bytes of response-body".

$ curl -v -k --location 'https://login.microsoftonline.com/(redacted)/oauth2/v2.0/token' \
--header 'Proxy-Authorization: Basic (redacted)' --header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_id=(redacted)' --data-urlencode 'client_secret=(redacted)' \
--data-urlencode 'scope=https://graph.microsoft.com/.default' --data-urlencode 'grant_type=client_credentials'
* Uses proxy env variable https_proxy == 'http://(redacted):89'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.149.50:89...
* Connected to (redacted proxy domain) (192.168.149.50) port 89 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to login.microsoftonline.com:443
> CONNECT login.microsoftonline.com:443 HTTP/1.1
> Host: login.microsoftonline.com:443
> User-Agent: curl/7.73.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NEGOTIATE
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: BASIC realm="IWA_Realm"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: close
< Connection: close
< Content-Length: 921
<
* Ignore 921 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 407 from proxy after CONNECT

[daviddenton]

I wonder if there isn't a test for "wrong content length" in our contract there should be... (If we can make it work!)