Download url location for artifacts appears to not always be correct

[fkz]

I'm trying to use this project to build an Android app, which I read is not yet supported; but my build already fails in the download step.
I don't understand gradle things very well, so I may be misunderstanding some things, but what I found is the following:

My verification-metadata.xml contains the following:

      <component group="androidx.compose.animation" name="animation-core-android" version="1.7.8">
         <artifact name="animation-core-android-1.7.8.module">
            <sha256 value="3ce792d09bcf03a1cc1d55f1424822f0961ec9bb019450d2f036c7af1205a10c" origin="Generated by Gradle"/>
         </artifact>
         <artifact name="animation-core-release.aar">
            <sha256 value="8237488870c73e9b66f7ecab5ec1855b0cafc023b0a937f3f7c65602e2c8c7b2" origin="Generated by Gradle"/>
         </artifact>
      </component>

Now buildGradleApplication tries to download the file from https://maven.google.com/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar, but cannot find it:

nix log

nix log /nix/store/1fk4s0c35jc6m1mmk8kbra34ny9z9jbr-animation-core-release.aar.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/1fk4s0c35jc6m1mmk8kbra34ny9z9jbr-animation-core-release.aar.drv^*'
Downloading animation-core-release.aar from https://plugins.gradle.org/m2/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar
error checking the existence of https://plugins.gradle.org/m2/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar:
curl: (22) The requested URL returned error: 404
Downloading animation-core-release.aar from https://repo1.maven.org/maven2/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar
error checking the existence of https://repo1.maven.org/maven2/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar:
curl: (22) The requested URL returned error: 404
Downloading animation-core-release.aar from https://maven.google.com/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar
error checking the existence of https://maven.google.com/androidx/compose/animation/animation-core-android/1.7.8/animation-core-release.aar:
curl: (22) The requested URL returned error: 404
File animation-core-release.aar was not found with hash sha256-gjdIiHDHPptm9+yrXsGFWwyvwCOwqTfz98ZWAuLIx7I= on any of the given urls

After some digging, I figured out that the animation-core-android-1.7.8.module file mentioned in the lockfile can be downloaded at https://maven.google.com/androidx/compose/animation/animation-core-android/1.7.8/animation-core-android-1.7.8.module is a json file that contains metadata about the package; and there's information about the files with a name and a url attribute:

cat animation-core-android-1.7.8.module | jq '.variants | .[] | .files | .[] | select(.name == "animation-core-release.aar")'
{
  "name": "animation-core-release.aar",
  "url": "animation-core-android-1.7.8.aar",
  "size": 1473986,
  "sha512": "6468b1cef3c92e2814a53ff439a1c25e5eb12a60dd27d8b667bf21ed48a17301e01e81f720a9ea7509ce62562688be0f0af9f201c97e25a3ece3dc50c98f75ff",
  "sha256": "8237488870c73e9b66f7ecab5ec1855b0cafc023b0a937f3f7c65602e2c8c7b2",
  "sha1": "9f78d4f11c4b89774095e314ded5fed47babf8f5",
  "md5": "a67f0d0327d26d9d02efb83dd736ac4e"
}
{
  "name": "animation-core-release.aar",
  "url": "animation-core-android-1.7.8.aar",
  "size": 1473986,
  "sha512": "6468b1cef3c92e2814a53ff439a1c25e5eb12a60dd27d8b667bf21ed48a17301e01e81f720a9ea7509ce62562688be0f0af9f201c97e25a3ece3dc50c98f75ff",
  "sha256": "8237488870c73e9b66f7ecab5ec1855b0cafc023b0a937f3f7c65602e2c8c7b2",
  "sha1": "9f78d4f11c4b89774095e314ded5fed47babf8f5",
  "md5": "a67f0d0327d26d9d02efb83dd736ac4e"
}

so the animation-core-release.aar name is mapped to animation-core-android-1.7.8.aar url and indeed https://maven.google.com/androidx/compose/animation/animation-core-android/1.7.8/animation-core-android-1.7.8.aar exists.

Now I wonder if this translation could be done by buildGradlePackage (if that is even what is supposed to happen, I'm not sure how the semantics is supposed to be).

[raphiz]

buildGradleApplication is (currently) unable to extract the declared repositories from your Gradle build (see also #7) and only checks maven central and plugins.gradle.org to download artifacts - but you can add arbitrary maven repositories including maven.google.com as follows:

buildGradleApplication {
  repositories = ["https://maven.google.com" "https://plugins.gradle.org/m2/" "https://repo1.maven.org/maven2/"],
  # ...
}

Hope this helps :)

[fkz]

Yes I already added that line; but the problem is that the name and the url appear to not be the same, and the mapping defined in the .module file (as already explained in my original message). I should have mentioned, that I added the repositories line though.

I'm trying to extend the logic to work in this case as well, see proof of concept here: fkz@6522ae1. This appears to have fixed this issue; but other things didn't work yet in my somewhat complicated setup.

[Paradyx]

I'm running in the same or similar issue with the com.github.ajalt.clikt:clikt-core-jvm dependency.

The verification-metadata.xml contains the following lines:

<component group="com.github.ajalt.clikt" name="clikt-core-jvm" version="5.0.3">
         <artifact name="clikt-core-jvm-5.0.3.module">
            <sha256 value="745b599b0f4aa00cf7015633e36f4a56f25d18bcb6f7686dcab3b1a11419dcf1" origin="Generated by Gradle"/>
         </artifact>
         <artifact name="clikt-jvm.jar">
            <sha256 value="69cba94a2291f74b78c88239cd8b67eaf3f08e0dad297749abe6477b3e9b81b1" origin="Generated by Gradle"/>
         </artifact>
      </component>

The pre-fetching tries to download the jar from the (heuristically guessed) URL https://repo1.maven.org/maven2/com/github/ajalt/clikt/clikt-core-jvm/5.0.3/clikt-jvm.jar which does not exist. In fact, the corresponding module declares the jar to be located at https://repo1.maven.org/maven2/com/github/ajalt/clikt/clikt-core-jvm/5.0.3/clikt-core-jvm-5.0.3.jar

[raphiz]

hm, interesting... So the Gradle Module Metadata allows the definition of both a name and a url for artifact files - and if they differ, we have this problem. I don't know why that should be a good idea and why this is a "feature" of Gradle... 🤷 But here we are...

The workaround I see right now is that we could adapt the download heuristics to also attempt to fetch the *.module file, parse it and looking for the url of the file that has the requested name.

To do the same thing as Gradle does, we should attempt in any case to download the Module Metadata first even if for the majority of packages, the filename is equal to the url. But I'm not sure if that is worth the extra roundtrip. But before implementing this, it's probably a good idea to read the Gradle Module Metadata spec and Gradle Docs in detail.

I'm not sure if I find the time to fix this soon. But if you open a PR, I will review and merge it!

[Paradyx]

FWIW: I went through all (about 20) broken dependencies and replaced the artifact name in the verification-metadata.xml by the name in the URI declared by the module and the pre-fetching seemed to work fine now.

<component group="com.github.ajalt.clikt" name="clikt-core-jvm" version="5.0.3">
         <artifact name="clikt-core-jvm-5.0.3.module">
            <sha256 value="745b599b0f4aa00cf7015633e36f4a56f25d18bcb6f7686dcab3b1a11419dcf1" origin="Generated by Gradle"/>
         </artifact>
         <artifact name="clikt-core-jvm-5.0.3.jar">
            <sha256 value="69cba94a2291f74b78c88239cd8b67eaf3f08e0dad297749abe6477b3e9b81b1" origin="Generated by Gradle"/>
         </artifact>
</component>

Without having read the docs, the name probably indicates the name that the JAR should take locally after the download and the url is how it is called in the repository.

I'm just a mere beginner with Nix. Creating a PR by my-self feels very ambitious. I'll see what I can do. Perhaps based on the prior work of @fkz I'll manage to build something.

(Apart from that, your project works great. It does exactly what I expect from a Gradle-Nix-integration. Good Job 👍)

[fkz]

Nice, that there is some attention here :). I checked my old branch again, updated it with main and tried it out once more on a project of mine.

@Paradyx Maybe you want to try out if it also works for you? You should be able to do so by using

  build-gradle-application.url = "github:fkz/buildGradleApplication/module";

Not sure if I should already open a PR with that code 🤔 . I have not checked it against the spec, the code currently does the following:

• if there is no .module file it does the same as before
• if there is a .module file it downloads that first (as it's in its own derivation, that happens only once). Then it searches for files which use the same hash as mentioned in the verification.xml file. If it finds a match, it uses that url instead.
• if no such match is found, it falls back to using the name as url, as before

Diff

[Paradyx]

I tested @fkz version and it works for me. Would you mind opening a Draft PR? I would like to discuss some parts of the code with you, if you don't mind.

@raphiz, I see that you are ignoring a bunch of -metadata artifacts, which is probably OK to do. But could this bug be the reason for your decision to ignore metadata, and should we perhaps rethink that heuristic? I ask, because a lot of my broken dependencies were metadata.

Nameley,

<          <artifact name="clikt-metadata.jar">
<          <artifact name="colormath-metadata.jar">
<          <artifact name="mordant-omnibus-metadata.jar">
<          <artifact name="mordant-metadata.jar">
<          <artifact name="yamlkt-metadata-0.13.0-all.jar">
---
>          <artifact name="clikt-core-5.0.3.jar">
>          <artifact name="colormath-3.6.0.jar">
>          <artifact name="mordant-3.0.1.jar">
>          <artifact name="mordant-core-3.0.1.jar">
>          <artifact name="yamlkt-0.13.0-all.jar">

[fkz]

@Paradyx thank you I opened the PR at #33.

[fkz]

Fixed as #33 is merged now. Followup improvements will be discussed in #36.