When there is a missing jar in Maven's local repo, it will fail.

[panbingkun]

sbt/sbt#7506

When we use the local Maven repo as the first level cache of SBT, if there is a corrupted jar in this repo, it will directly fail.
When there is a corrupted jar in Maven's local repo, It should not fail, but rather automatically skip this cache.

[eed3si9n]

FYI here's full stack trace from OP's https://github.com/panbingkun/spark/actions/runs/8105142421/job/22153031672:

[info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 
[error] 	at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
[error] 	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
[error] 	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 	at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:590)
[error] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error] 	at scala.util.Success.map(Try.scala:213)
[error] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] (mllib-local / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (mllib-local / mimaPreviousClassfiles) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (sql-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (streaming-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (connect-client-jvm / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar: not found: /root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar
[error] file:/root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar: not found: /root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar
[error] file:/root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar: not found: /root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar
[error] Total time: 299 s (04:59), completed Mar 1, 2024, 1:42:34 AM

I wonder if file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/ contains POM but not JAR files?

[panbingkun]

FYI here's full stack trace from OP's https://github.com/panbingkun/spark/actions/runs/8105142421/job/22153031672:

[info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 
[error] 	at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82)
[error] 	at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
[error] 	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
[error] 	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 	at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:590)
[error] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error] 	at scala.util.Success.map(Try.scala:213)
[error] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] (mllib-local / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (mllib-local / mimaPreviousClassfiles) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (sql-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (streaming-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (connect-client-jvm / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar: not found: /root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar
[error] file:/root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar: not found: /root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar
[error] file:/root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar: not found: /root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar
[error] Total time: 299 s (04:59), completed Mar 1, 2024, 1:42:34 AM

I wonder if file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/ contains POM but not JAR files?

Yea, that's right, only pom, not jar.
Here is a record of my previous testing, eg:

[He-Pin]

it would be greate if we can delete the jars with command

[panbingkun]

@lefou @ghostbuster91 Sorry to bother you, do we have a way to bypass this case? Except for disable maven's local repo.

[panbingkun]

Because I found #2929, it seems that our scene is very similar.
Thanks.

[lefou]

@lefou @ghostbuster91 Sorry to bother you, do we have a way to bypass this case? Except for disable maven's local repo.

I don't know. I think the proper solution would require a change in coursier. At least I didn't found anything in the docs, that could help here.

Some developer note: Maven does use different implementations for Remote Repositories and Local Repositories. The local one stores extra information like lastest update attempts and other metadata and its core property is to store partial/incomplete dependency trees. Coursier has probably only one implementation for remote repositories and isn't able to detect a partial Maven repository state. So we either need to implement some local repository support or improve the existing implementation to better handle local repos. If you plan to rely on a Maven local repository with coursier, you better get prepared to contribute.

[panbingkun]

@lefou @ghostbuster91 Sorry to bother you, do we have a way to bypass this case? Except for disable maven's local repo.

I don't know. I think the proper solution would require a change in coursier. At least I didn't found anything in the docs, that could help here.

Some developer note: Maven does use different implementations for Remote Repositories and Local Repositories. The local one stores extra information like lastest update attempts and other metadata and its core property is to store partial/incomplete dependency trees. Coursier has probably only one implementation for remote repositories and isn't able to detect a partial Maven repository state. So we either need to implement some local repository support or improve the existing implementation to better handle local repos. If you plan to rely on a Maven local repository with coursier, you better get prepared to contribute.

Thank you for your patient and detailed explanation.
Additionally, I have another background to add about this case,
When we used sbt 1.9.3, it worked very well and the Courier version we used was as follows

• "io.get-coursier" %% "lm-coursier-shaded" % "2.1.0"
• "io.get-coursier" %% "coursier" % "2.1.2"

But when we use SBT greater than 1.9.3, it will encounter the above issues, such as sbt 1.9.4, using the following Courier version:

• "io.get-coursier" %% "lm-coursier-shaded" % "2.1.1"
• "io.get-coursier" %% "coursier" % "2.1.6"

I speculate that Courier has a regression in behaviour since 2.1.3 or 2.1.4 or 2.1.5 or 2.1.6.

[alexarchambault]

I'd be tempted to close this as "won't fix". ~/.m2/repository use has always been strongly discouraged with coursier (#1832 (comment), #667 (comment), but I'm pretty sure this has been mentioned in other places).

coursier has always been assuming that the repository that contains metadata (like POM files) also contains the JARs. That allows to limit network requests (single request to download the JAR), but also to know for sure that a JAR doesn't exist (checking if we get 404 is the only way to know that a dependency doesn't have a JAR), rather than check all repositories for dependencies that don't have a JAR.

I'm not sure breaking that assumption and issuing more network request for ~/.m2/repository support is worth the effort. Plus there are ways to use another directory to publish things to, and using that directory as a repository (with a URI like file:///…) from coursier is fine.