Incomplete downloads are not detected

[memo33]

When the transmission is interrupted while downloading a file without checksums, the partial file ends up in the cache. Coursier does not detect that the download is incomplete. This can happen due to network outages or server timeouts, for example.

Steps to reproduce locally

• host a file using jwebserver,
• stop the server while the download is in progress,
• the partial file is now in the file cache -- the .part suffix is removed from the file name.

(Tested with coursier-cache_2.13:2.1.2 and OpenJDK 20.0.2 targeting Java 8.)

Solutions

1. A basic fix would consist of validating the file size in Downloader#Blocking.doDownload based on the content-length of the connection, so that at least the error does not go unnoticed.

2. A more advanced fix would try to resume the partial download.

There seems to be some code in place for partial downloads already. To which extent is this supported?

There is also some code for checking the file size that produces a WrongLength artifact error, but it does not regulary seem to be invoked, except in case the file already exists and is locked.

[memo33]

Regarding the code for resuming downloads that is already implemented:
It seems that if a .part file ends up in the cache, Coursier tries to resume the download but appends the entire file length to it if the server does not support byte serving, so that the resulting file is too long.

Two sections of code that look suspicious to me:

coursier/modules/cache/jvm/src/main/scala/coursier/cache/CacheUrl.scala

Lines 217 to 225 in ef2b102

	val startOver = {
	val isPartial = conn0.getResponseCode == partialContentResponseCode ||
	conn0.getResponseCode == invalidPartialContentResponseCode
	isPartial && {
	val hasMatchingHeader = Option(conn0.getHeaderField("Content-Range"))
	.exists(_.startsWith(s"bytes $alreadyDownloaded-"))
	!hasMatchingHeader
	}
	}

Here, the condition startOver = isPartial && !hasMatchingHeader seems not general enough. If the client sends a request for a partial download, but the server responds with a code different from 206/416, the download needs to start-over as well (or it needs to be treated as non-partial). In other words, isPartial needs to account for the intent of the request, rather than looking at the response code.

coursier/modules/cache/jvm/src/main/scala/coursier/cache/CacheUrl.scala

Lines 341 to 350 in ef2b102

	val rangeResOpt0 = rangeResOpt(conn, alreadyDownloaded)
	rangeResOpt0 match {
	case Some(true) =>
	closeConn(conn)
	Left(args.copy(alreadyDownloaded = 0L))
	case _ =>
	val partialDownload = rangeResOpt0.nonEmpty
	val redirectOpt =
	redirect(url0, conn, followHttpToHttpsRedirections, followHttpsToHttpRedirections)

Here Coursier checks whether the opened connection refers to a partial download and then proceeds to follow redirects. Ideally, this check should be done after following redirects though.