Object does not exist using s3 api

[piyush-bhanushali1]

Describe the bug
I'm using mc client to access seaweedfs bucket from the terminal. I can list the bucket objects but while copying them to local or accessing them using java code i'm getting Object does not exist. How can i fix this?

System Setup

• seaweedfs deployed in kubernets using helm chart with kubernetes version 1.28.
• seaweedfs version 3.57.
• running given command :- mc cp -r demoseed/databyte/ seedminio/test/
• demoseed is alias of demo env and seedminio is alias of dev env seaweedfs. i have access in both cluster.

Expected behavior

• can copy files from one seaweedfs to another.

Error logs
mc: Failed to copy [http://localhost:8222/databyte/Logging/resourceFiles/db.properties.](http://localhost:8222/databyte/Logging/resourceFiles/db.properties%60.) Object does not exist
mc: Failed to copy [http://localhost:8222/databyte/Logging/resourceFiles/dbfConfig.properties.](http://localhost:8222/databyte/Logging/resourceFiles/dbfConfig.properties%60.) Object does not exist
mc: Failed to copy [http://localhost:8222/databyte/Logging/resourceFiles/jgraphdb.properties.](http://localhost:8222/databyte/Logging/resourceFiles/jgraphdb.properties%60.) Object does not exist
mc: Failed to copy [http://localhost:8222/databyte/Logging/resourceFiles.zip.](http://localhost:8222/databyte/Logging/resourceFiles.zip%60.) Object does not exist

Additional context
Same things happens while i'm using java code to access same file which is uploaded to that bucket.

[chrislusf]

works for me:

chrislu/tmp ()  > AWS_SECRET_ACCESS_KEY=admin_secret AWS_ACCESS_KEY_ID=admin_access aws --endpoint-url http://localhost:8333 s3 cp s3://test/abc/value_builder.go .
download: s3://test/abc/value_builder.go to ./value_builder.go

[piyush-bhanushali1]

Hi @chrislusf ,

My scenario is in test/abc/ i have multiple jar around 550-600 i'm fetching them via mc (minio) client. when i'm running mc cp command at them time for some of the jar it is giving me object does not exist and jar size is some where around 300-500 kb.

I don't have an AWS environment; I'm running on my own on-premises Kubernetes environment.

[vishesh-bootnext]

Hello @chrislusf,

I hope you're having a great day! Firstly, I wanted to express my gratitude for developing such an incredible tool. It has been immensely helpful in my work.

I'd like to bring to your attention an issue that I'm encountering, which is similar to one raised by @piyush-bhanushali1. Whenever I attempt to copy data using the MinIO client (mc), I consistently encounter an error stating "Object Does Not Exist," despite the object being visible in the filer UI and listed in the S3 bucket.

Moreover, I've identified two additional issues:

1. Upon restarting the filer server, the entire metadata is lost. Subsequently, neither the files are visible in the filer UI nor in the S3 bucket list.
2. When fetching a file using the S3 API, I encounter the "Object Does Not Exist" error. Strangely, even when copying data from one bucket to another internally, the issue persists.
   Given these challenges, I'm exploring potential solutions. Do you know of any tools or methods to seamlessly migrate the entire dataset to another empty server? It's worth mentioning that I'm operating within a Kubernetes environment and have installed SeaweedFS using a Helm chart. While I'm not utilizing any Persistent Volume Claims (PVC) for the filer, they are in use for the volume servers. I am using 1 master, 1 s3, 3 filer-servers and 3 volume-servers.

I've exhausted various admin commands and attempted solutions such as backup restoration and "mc cp," but unfortunately, none have yielded success.

Your guidance and insights would be greatly appreciated in resolving these issues. Thank you for your attention to this matter.

Warm regards,
Vishesh

[Jackbennett]

You should be able to see in the CSI node logs the s3 request for the file and the mounted path backend by a <volume_id>,<needle> that's giving the 404. Is this the expected volume when you look up the bound PVC name to the bucket on the volume server?

We have in the past quite happily moved files with minio client talking to seaweedfs, I just wanted to add a positive datapoint.

I am seeing this behaviour since I ran the disk space to zero, that's my fault and something else I don't want to hijack this issue about.

[zuzuviewer]

I met this too. Some object return 404 because filer id not found

[bntv-config]

The issue seems to be frequent when using seaweedfs with kubernates . I am also facing it . Any permanent solution for this @chrislusf

[vishesh-bootnext]

Hi @chrislusf,

I'd like to provide a more detailed overview of the issue we're experiencing, along with some additional context and the filer logs for your reference. Both @piyush-bhanushali1 and I are part of the same team and have encountered similar challenges.

Here's a breakdown of the problem:

Accessing Files and Copying with mc Client:

When attempting to access files from the filer or copying files between servers using the mc client (MinIO client), we consistently encounter errors.
Despite the filer UI indicating that the file is present, we still encounter these errors.
Filer Logs:

Below are the logs from the filer server, providing insight into the errors encountered:

E0517 06:21:27.503448 stream.go:97 operation LookupFileId 188,3271665c5b56 failed, err: urls not found
E0517 06:21:27.503506 filer_server_handlers_read.go:259 failed to stream content /buckets/nst-fm/template/ExportAlarmLibraryTemplate.xlsx: operation LookupFileId 188,3271665c5b56 failed, err: urls not found
E0517 06:21:27.503533 common.go:297 processRangeRequest: operation LookupFileId 188,3271665c5b56 failed, err: urls not found
I0517 06:25:48.510151 filechunk_manifest.go:197 read http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286 failed, err: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 06:25:48.510173 filechunk_manifest.go:203 retry reading in 1s
I0517 06:25:49.511228 filechunk_manifest.go:197 read http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286 failed, err: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 06:25:49.511251 filechunk_manifest.go:203 retry reading in 1.5s
I0517 06:25:51.012108 filechunk_manifest.go:197 read http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286 failed, err: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 06:25:51.012131 filechunk_manifest.go:203 retry reading in 2.25s
I0517 06:25:53.263030 filechunk_manifest.go:197 read http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286 failed, err: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 06:25:53.263057 filechunk_manifest.go:203 retry reading in 3.375s
I0517 06:25:56.639527 filechunk_manifest.go:197 read http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286 failed, err: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 06:25:56.639550 filechunk_manifest.go:203 retry reading in 5.0625s
E0517 06:26:01.702667 filer_server_handlers_read.go:259 failed to stream content /buckets/nst-fm/template/ImportAlarmLibrary.xlsx: read chunk: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
E0517 06:26:01.702708 common.go:297 processRangeRequest: read chunk: http://seaweedfs-volume-0.seaweedfs-volume.swf:8080/190,32753482b286?readDeleted=true: 404 Not Found
I0517 15:06:14.991789 auth_credentials_subscribe.go:85 updated bucketMetadata /buckets/name:"testbucket" is_directory:true attributes:{mtime:1715958374 file_mode:2147484159 crtime:1715958374}

Another filer Logs while accessing the Files from UI are:

E0517 16:30:30.313735 filer_server_handlers_read.go:259 failed to stream content /buckets/bntvpm/PM/requestnodewithfile/20230804_060701_00032_qxqun_f97a121c-093e-4036-8bb5-c149855763f7: operation LookupFileId 508,0be389debeb199 failed, err: urls not found
E0517 16:30:30.313760 common.go:341 processRangeRequest range[0]: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="20230804_060701_00032_qxqun_f97a121c-093e-4036-8bb5-c149855763f7"] Content-Length:[6] Content-Range:[bytes 490-495/496] Content-Type:[application/octet-stream] Etag:["f2aba708eb30f181e09c0b0428292edd"] Last-Modified:[Fri, 26 Apr 2024 15:19:47 GMT] Server:[SeaweedFS Filer 30GB 3.57] X-Amz-Meta-Cache:[no=cache]] err: operation LookupFileId 508,0be389debeb199 failed, err: urls not found
2024/05/17 16:30:30 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:342)
E0517 16:30:36.145808 stream.go:97 operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
E0517 16:30:36.145874 filer_server_handlers_read.go:259 failed to stream content /buckets/bntvpm/PM/requestnodewithfile/20230804_060719_00086_qxqun_8d80696d-7dfe-4493-86d5-7c82b7238a39: operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
E0517 16:30:36.145916 common.go:341 processRangeRequest range[0]: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="20230804_060719_00086_qxqun_8d80696d-7dfe-4493-86d5-7c82b7238a39"] Content-Length:[31] Content-Range:[bytes 490-520/521] Content-Type:[application/octet-stream] Etag:["6a6d8a0370e366e8d69e56f68dbf78c3"] Last-Modified:[Fri, 26 Apr 2024 15:19:47 GMT] Server:[SeaweedFS Filer 30GB 3.57] X-Amz-Meta-Cache:[no=cache]] err: operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
2024/05/17 16:30:36 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:342)
E0517 16:30:43.815691 stream.go:97 operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
E0517 16:30:43.815756 filer_server_handlers_read.go:259 failed to stream content /buckets/bntvpm/PM/requestnodewithfile/20230804_060719_00086_qxqun_8d80696d-7dfe-4493-86d5-7c82b7238a39: operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
E0517 16:30:43.815796 common.go:341 processRangeRequest range[0]: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="20230804_060719_00086_qxqun_8d80696d-7dfe-4493-86d5-7c82b7238a39"] Content-Length:[31] Content-Range:[bytes 490-520/521] Content-Type:[application/octet-stream] Etag:["6a6d8a0370e366e8d69e56f68dbf78c3"] Last-Modified:[Fri, 26 Apr 2024 15:19:47 GMT] Server:[SeaweedFS Filer 30GB 3.57] X-Amz-Meta-Cache:[no=cache]] err: operation LookupFileId 507,0be37878e61e54 failed, err: urls not found
2024/05/17 16:30:43 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:342)
E0517 16:30:55.806201 stream.go:97 operation LookupFileId 508,0be389debeb199 failed, err: urls not found
E0517 16:30:55.806245 filer_server_handlers_read.go:259 failed to stream content /buckets/bntvpm/PM/requestnodewithfile/20230804_060701_00032_qxqun_f97a121c-093e-4036-8bb5-c149855763f7: operation LookupFileId 508,0be389debeb199 failed, err: urls not found
E0517 16:30:55.806272 common.go:341 processRangeRequest range[0]: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="20230804_060701_00032_qxqun_f97a121c-093e-4036-8bb5-c149855763f7"] Content-Length:[6] Content-Range:[bytes 490-495/496] Content-Type:[application/octet-stream] Etag:["f2aba708eb30f181e09c0b0428292edd"] Last-Modified:[Fri, 26 Apr 2024 15:19:47 GMT] Server:[SeaweedFS Filer 30GB 3.57] X-Amz-Meta-Cache:[no=cache]] err: operation LookupFileId 508,0be389debeb199 failed, err: urls not found
2024/05/17 16:30:55 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:342)
E0517 16:32:42.668837 stream.go:97 operation LookupFileId 401,0b32a306d28537 failed, err: urls not found
E0517 16:32:42.668882 filer_server_handlers_read.go:259 failed to stream content /buckets/document-mgnt/uploads/documents/11454/smc_outdoor2.png: operation LookupFileId 401,0b32a306d28537 failed, err: urls not found
E0517 16:32:42.668903 common.go:297 processRangeRequest: operation LookupFileId 401,0b32a306d28537 failed, err: urls not found
E0517 16:34:01.716059 stream.go:97 operation LookupFileId 401,0b32a306d28537 failed, err: urls not found
E0517 16:34:01.716103 filer_server_handlers_read.go:259 failed to stream content /buckets/document-mgnt/uploads/documents/11454/smc_outdoor2.png: operation LookupFileId 401,0b32a306d28537 failed, err: urls not found
E0517 16:34:01.716125 common.go:297 processRangeRequest: operation LookupFileId 401,0b32a306d28537 failed, err: urls not found

Your assistance in resolving these issues would be greatly appreciated.

Warm regards,
Vishesh

[vishesh-bootnext]

@chrislusf. Is there any update on this common issue? Please revert.

[chrislusf]

which commit to revert?

[vishesh-bootnext]

#5567 (comment)
#5567 (comment)

[chrislusf]

Need to be able to reproduce the issue.

[vishesh-bootnext]

Thanks for the update Chris. Will wait for your response. Let us know once you are able to reproduce the issue or seek any help from me.

Appreciate your time and help.

[chrislusf]

I am not able to reproduce this issue.

[zuzuviewer]

Maybe should add startup parameter "-dir=/data"。default it's tmp directory。
The doc add this is better for user.

[zuzuviewer]

Maybe should add startup parameter "-dir=/data"。default it's tmp directory。
The doc add this is better for user.