[fix] weed fix built an index with the expected size smaller than the actual size

[kmlebedev]

Describe the bug
ls

-rw-r--r-- 1 seaweed seaweed 16M Apr 18 13:33 /data/bucket_231.dat

volume.list

volume id:231 size:16047320 collection:\"bucket\" file_count:4 delete_count:2 deleted_byte_count:16046990 replica_placement:100 version:3 ttl:261 compact_revision:149 modified_at_second:1713447196

log:

E0418 15:10:28.454905 volume_grpc_vacuum.go:65 failed compact volume 231: volume 231 unexpected new data size: 88 does not match size of content minus deleted: 128
I0418 15:10:28.455663 volume_vacuum.go:174 Cleaning up volume 231 vacuuming...

weed fix built an index with the expected size smaller than the actual size, because of this volume switches to read-only mode

after weed fix
volume.list

volume id:231 size:16047320 collection:\"bucket\" file_count:2 read_only:true replica_placement:100 version:3 ttl:261 compact_revision:149 modified_at_second:1713447196

log

I0418 17:23:53.324309 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 17:23:53.324343 volume_checking.go:121 data file /data/bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 17:23:53.324351 volume_loading.go:128 volumeDataIntegrityChecking failed data file /data/bucket_231.dat actual 16047320 bytesexpected 200 bytes
I0418 17:23:53.324366 needle_map_sorted_file.go:24 Start to Generate /data/bucket_231.sdx from /data/bucket_231.idx
I0418 17:23:53.324783 needle_map_sorted_file.go:26 Finished Generating /data/bucket_231.sdx from /data/bucket_231.idx
I0418 17:23:53.324819 disk_location.go:182 data file /data/bucket_231.dat, replication=100 v=3 size=16047320 ttl=1M
I0418 17:23:53.324826 store.go:471 mount volume 231

see idx:

./see_idx -volumeId 231    
key:fc138c121e6e6b6c offset:1 size:82(82 B)
key:fc138c121e7c6fef offset:15 size:46(46 B)

see dat

./see_dat -volumeId 231
I0419 00:11:56.464368 volume_loading.go:91 readSuperBlock volume 231 version 3
I0419 00:11:56.467121 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.05674303 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected
I0419 00:11:56.467135 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 120 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595410049 +0500 +05 name -protected
I0419 00:11:56.474538 see_dat.go:36 231,fc138c121ecd24bdca1a808d offset 200 size 8023495(7.65 MiB) cookie ca1a808d appendedAt 2024-04-17 20:22:04.112901039 +0500 +05 name projects-non_protected
I0419 00:11:56.480959 see_dat.go:36 231,fc138c121ed0f05ca5ff77a6 offset 8023728 size 8023495(7.65 MiB) cookie a5ff77a6 appendedAt 2024-04-18 18:32:45.305981473 +0500 +05 name projects-non_protected
I0419 00:11:56.480976 see_dat.go:36 231,fc138c121ecd24bd00000000 offset 16047256 size 0(0 B) cookie 00000000 appendedAt 2024-04-18 18:32:46.253828824 +0500 +05 name 
I0419 00:11:56.480981 see_dat.go:36 231,fc138c121ed0f05c00000000 offset 16047288 size 0(0 B) cookie 00000000 appendedAt 2024-04-18 18:33:16.590179944 +0500 +05 name 

hexdump -n 200 -C /tmp/231.dat
00000000  03 64 01 05 00 95 00 00  ad 4c 01 b3 fc 13 8c 12  |.d.......L......|
00000010  1e 6e 6b 6c 00 00 00 52  00 00 00 16 50 4b 05 06  |.nkl...R....PK..|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 1e 2e 73 65 63 75  69 72 74 79 2d 61 6e 73  |....secuirty-ans|
00000040  69 62 6c 65 2d 67 61 6c  61 78 79 2d 72 65 6e 6f  |ible-galaxy-reno|
00000050  76 61 74 65 2d 6e 6f 6e  5f 70 72 6f 74 65 63 74  |vate-non_protect|
00000060  65 64 00 00 65 f9 5c f8  01 05 e2 4f 23 93 17 be  |ed..e.\....O#...|
00000070  20 e5 db ed 84 76 00 00  ef b2 96 42 fc 13 8c 12  | ....v.....B....|
00000080  1e 7c 6f ef 00 00 00 2e  00 00 00 16 50 4b 05 06  |.|o.........PK..|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 1e 0a 2d 70 72 6f  74 65 63 74 65 64 00 00  |....-protected..|
000000b0  65 fd 48 41 01 05 e2 4f  23 93 17 bf 0a 7e 70 9a  |e.HA...O#....~p.|
000000c0  86 81 00 00 00 2e 00 00                           |........|
000000c8

ls cpd

-rw-r--r-- 1 seaweed    seaweed     88 Apr 18 18:53 /data/bucket/231.cpd

see cpd

I0419 00:22:54.506440 volume_loading.go:91 readSuperBlock volume 231 version 3
I0419 00:22:54.508873 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 8 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595410049 +0500 +05 name -protected

hexdump -C /tmp/231.cpd
00000000  03 64 01 05 00 96 00 00  ef b2 96 42 fc 13 8c 12  |.d.........B....|
00000010  1e 7c 6f ef 00 00 00 2e  00 00 00 16 50 4b 05 06  |.|o.........PK..|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 1e 0a 2d 70 72 6f  74 65 63 74 65 64 00 00  |....-protected..|
00000040  65 fd 48 41 01 05 e2 4f  23 93 17 bf 0a 7e 70 9a  |e.HA...O#....~p.|
00000050  86 81 00 00 00 2e 00 00                           |........|
00000058

see cpx:

key:fc138c121e7c6fef offset:1 size:46(46 B)

System Setup

weed version
version 8000GB 3.65 0edc5aad9a489d0c99213a45e27654703fa5da70 linux amd64

Expected behavior
weed fix it is necessary to create an index at which compact will not cause errors.

[kmlebedev]

I remember the problem when the deleted file is counted twice, so we see the file_count field as 4, although in fact there are 2.
It turns out that there are no current files in the dat file.

[kmlebedev]

Also tried to do compact manually

For method=0, it compacts based on the .dat file, works if .idx file is corrupted.
For method=1, it compacts based on the .idx file, works if deletion happened but not written to .dat files.

weed compact -collection bucket -volumeId 231 -dir=/data
I0418 18:53:14.938912 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 18:53:14.939216 volume_checking.go:121 data file bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 18:53:14.939332 volume_loading.go:128 volumeDataIntegrityChecking failed data file bucket_231.dat actual 16047320 bytes expected 200 bytes
I0418 18:53:14.939837 volume_loading.go:91 readSuperBlock volume 231 version 3

weed compact -collection bucket -volumeId 231 -method=1 -dir=/data
I0418 18:53:56.382445 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 18:53:56.382699 volume_checking.go:121 data file bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 18:53:56.382835 volume_loading.go:128 volumeDataIntegrityChecking failed data file bucket_231.dat actual 16047320 bytes expected 200 bytes
F0418 18:53:56.385220 compact.go:54 Compact Volume [ERROR] volume 231 unexpected new data size: 88 does not match size of content minus deleted: 128
goroutine 1 [running]:
github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)
	/github/workspace/weed/glog/glog.go:768 +0x85
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x4027380, 0x3, 0xc0005edd50, {0x32de15a?, 0xc000da9d28?}, 0x1?, 0x0)
	/github/workspace/weed/glog/glog.go:719 +0x38a
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printf(0x4027380, 0x3, {0x24aa0bb, 0x1a}, {0xc000da9d28, 0x1, 0x1})
	/github/workspace/weed/glog/glog.go:657 +0x10a
github.com/seaweedfs/seaweedfs/weed/glog.Fatalf(...)
	/github/workspace/weed/glog/glog.go:1154
github.com/seaweedfs/seaweedfs/weed/command.runCompact(0x4008738?, {0xc0000500a0?, 0x6?, 0x6?})
	/github/workspace/weed/command/compact.go:54 +0x209
main.main()
	/github/workspace/weed/weed.go:95 +0x3d9

[kmlebedev]

@chrislusf It looks like, judging by the index, after weed fix there should be two files left, but in fact after weed compact there remains one file(-protected) and the second(secuirty-ansible-galaxy-renovate-non_protected) one was unexpectedly deleted:

I0419 00:11:56.467121 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.05674303 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected

[kmlebedev]

display additional TTL values is 1M

./see_dat  -volumeId 231
I0427 18:13:32.053612 volume_loading.go:91 readSuperBlock volume 231 version 3
I0427 18:13:32.055783 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.057043092 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected flags 0000001e ttl 1M
I0427 18:13:32.055811 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 120 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595009609 +0500 +05 name -protected flags 0000001e ttl 1M

accordingly, the compactor deletes files in accordance with TTL, and the indexer does not take into account TTL

[chrislusf]

Compaction would not run on TTL volumes. We should add some logic to disable it by default in weed compact also.