Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error #01: Needed resources are used by other tasks. #1276

Open
ishaulov opened this issue Apr 18, 2024 · 7 comments · May be fixed by #1271
Open

Error #01: Needed resources are used by other tasks. #1276

ishaulov opened this issue Apr 18, 2024 · 7 comments · May be fixed by #1271

Comments

@ishaulov
Copy link

ishaulov commented Apr 18, 2024 •
edited

Hi! Need help with publishing packages!

Packages are assembled on Jenkins, and then they are further uploaded and published to the repository using the aptly api:

curl -f -X POST -F file=@"libson-1.11.6248_amd64.deb" http://repos.local/deb/api/files/uploaded-files?_async=true -u repouser:repopassword 
curl -f -X POST http://repos.local/deb/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./debrepo?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-20.04?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./mint?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-22.04??_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword

The problem is that there may be parallel builds, downloads, and publications at the same time, and I get the following error:
aptly — "Error #1: Needed resources are used by other tasks."
curl — (22) The requested URL returned error: 409 Conflict


{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"info","remote":"172.27.0.4","method":"POST","path":"/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true","protocol":"HTTP/1.0","code":"202","latency":"166.954µs","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00"}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./debrepo?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.423243ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-20.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.386ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./mint?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.258301ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-22.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.272133ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

Have I tried using this version:
#1125 (comment)
and using ?_async=true, but the error persists.

I tried to build aptly from the master branch, errors persist.
What else can I do, or maybe I'm doing something wrong?
@neolynx
Copy link
Member

neolynx commented Apr 18, 2024

Hi, thanks for reporting !

yes, this might be fixed with #1125, which is currently being rebased on master.

the ?_async=true will make the API calls return a task_id. the caller should wait for the task to complete, before issuing a next api call. this can be done with GET /tasks/:id/wait
maybe you could try this ?

@ishaulov
Copy link
Author

Thanks for the reply!

Yes, I can. But let me clarify: do I understand correctly that I need to write a shell handler that will process each curl request, take the ID and put it into GET /tasks/:ID/wait ?

@neolynx
Copy link
Member

neolynx commented Apr 18, 2024

yes, something like this:
curl http://repos.local/api/tasks/1/wait

@ishaulov
Copy link
Author

ishaulov commented May 21, 2024 •
edited

Hello, @neolynx!

I apologize for the delay in replying, it took a while to check everything out.

I've added /tasks/:ID/wait in my shell script, but I still get this error when building in parallel on Jenkins:

[GIN] 2024/05/21 - 13:41:39 | 200 |   83.949261ms |      172.19.0.2 | GET      "/api/tasks/9/wait"
2024/05/21 13:41:39 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:39 | 202 |    2.403509ms |      172.19.0.2 | PUT      "/api/publish/:./ubuntu-20.04?_async=true"
[GIN] 2024/05/21 - 13:41:39 | 200 |   87.470772ms |      172.19.0.2 | GET      "/api/tasks/10/wait"
[GIN] 2024/05/21 - 13:41:47 | 200 |  310.719834ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 202 |      115.25µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6316_amd64.deb?_async=true"
[GIN] 2024/05/21 - 13:41:47 | 200 |  307.632802ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 409 |     116.276µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6317_amd64.deb?_async=true"
Error #01: Needed resources are used by other tasks.

Aptly version 1.5.0+ds1-1+b4 .

This is stdout of shell script with set -x:

+ echo 'Moving deb package to repository...'
Moving deb package to repository...
++ curl -f -X POST 'http://repos.local/deb/api/repos/libs/file/uploaded-files//libs-generator.87.6317_amd64.deb?_async=true' -u repo-api:aGF#EQLr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    53    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 409

And do I understand correctly that I can't get ID and STATE on this request?

+ echo 'Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb'
Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb
++ curl -s -f -X POST -F file=@libsdeb///libs-generator.87.6317_amd64.deb 'http://repos.local/deb/api/files/uploaded-files?_async=true'
+ upload_response='["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ '[' 0 -ne 0 ']'
+ echo '["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ json_pp
[
   "uploaded-files//libs-generator.87.6317_amd64.deb"
]

And is it correct that on GET /tasks/:ID/wait I should get STATE=2? Is there any actual documentation on this function somewhere?

@neolynx
Copy link
Member

neolynx commented Jun 5, 2024

Hi !

I looked in the problem and I think aptly does not support parallel operation, i.e. different clients sending requests at the same time. For this reason in the molior project there is a queue before all aptly operations, making in work with parallel request.
The background API also does not solve the problem, is is merely allowing to do other things while waiting for aptly to complete, the requests still must be sent sequentially.

There are ideas to introduce a mutex to wait until the resources are available, but I am not sure this is safe and requests will be executed in order (imagine creating, deleting and recreating same repo. those should be executed in order as they arrive). So probably aptly should queue requests and process them in order, which whould be a bigger implementation.

In the meanwhile, I there a way to wait for all jenkins tasks to complete, and then do all aptly operations in one go, not using async ?

@neolynx
Copy link
Member

neolynx commented Jun 8, 2024 •
edited

I implemented a queue in #1271, this should allow to publish from different clients instead of returning 409.

Something like this should work for all commands which support async:

task_id=`curl -f -X PUT http://localhost:3142/api/publish/bookworm_12.5-archs-copy_repos_test_1.0-copy2/stable?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"reposign@molior", "PassphraseFile": "/var/lib/aptly/gpg-pass"}}'  | jq .ID`

curl http://repos.local/api/tasks/$task_id/wait

Could you give this a try?

Regarding your questions:

  • I will check if the file upload API also support async
  • yes, STATE=2 is SUCCEEDED (see task/task.go:39)
  • all of this should be documented indeed, if you would like to help with this, PRs to https://github.com/aptly-dev/www.aptly.info are very welcome :)

The question now is, how to deal with non async requests. Maybe they should go through the queue as well and additionally just wait for the task to finish before returning 200 ?

This was referenced Jun 8, 2024
Closed
Open
@neolynx
Copy link
Member

neolynx commented Jun 8, 2024

The normal non-async requests are not also queued in #1271, and also file uploads should work concurrently now.

Could you please give it a try?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix/publish concurrency aptly-dev/aptly
3 participants
@neolynx @ishaulov and others