Moments: Don't back up albums in worker to avoid disk activity #4237
Comments
|
Are you sure that the YAML files are updated every 15 minutes (edit: or whatever you have configured as worker interval)? Is there any other activity, like changes to the index or faces, that could trigger an update? According to this, the YAML files / Moments should only be updated if photoprism/internal/workers/meta.go Lines 50 to 51 in 38a1616 While this background worker does perform a query to determine if files need to be checked (e.g. the generated title is updated if people have recently been recognised in an image), this should only cause brief activity in the form of a database query and other steps, like updating Moment albums, should be skipped except every 3 hours (as currently implemented, it can of course be further improved): photoprism/internal/entity/photo.go Line 29 in 38a1616 |
|
In log I can see just the repeated sequence attached in fist comment. It seems there is no other update or background processing. Photo set were stable a few days without adding new photos. |
|
I wonder if the background activity really depends on the YAML backup files being enabled, or if the worker is constantly checking the index for changes anyway and the main increase in power consumption is due to the magnetic disks spinning up? We could of course check the existing backup files to see if the data has changed, but then the files would have to be read first, which might be enough to wake the disks up - so this wouldn't solve your particular problem. |
|
I am looking into code now. moments.go:269 // Make sure that the albums have been backed up before, otherwise back up all albums.
if fs.PathExists(filepath.Join(w.conf.AlbumsPath(), entity.AlbumManual)) &&
fs.PathExists(filepath.Join(w.conf.AlbumsPath(), entity.AlbumMonth)) {
// Skip.
} else if count, err := BackupAlbums(w.conf.AlbumsPath(), false); err != nil {
log.Errorf("moments: %s (backup albums)", err.Error())
} else if count > 0 {
log.Debugf("moments: %d albums saved as yaml files", count)
}There is no time constraint. Each moments generation updates yaml files? |
Yes, increases of power consuption are caused by disks spinning up. I think if there is nothing changed at all, it could skip backup phase. |
|
These are dynamic search filters generated based on the currently indexed files, and at the moment there is no cache or timestamp indicating when the files were last backed up. However, as mentioned above, the code should not be executed every 15 minutes (or hourly), but only every 3 hours if photoprism/internal/workers/meta.go Lines 122 to 134 in 38a1616 |
lastzero
changed the title
|
Really interesting. I have dropped database and used default wake up time and now it runs each 10 minutes. It is probably caused by OR (cell_id = 'zz' AND photo_lat <> 0)Some photos have that values that never changes. |
|
Maybe there could be counter that counts failed attemtps of coordinates translation. |
|
Thanks, that's really helpful! Never thought of this and we have SSDs in our severs, so you don't hear the disks either way. |
|
Also photos do not have location in exif. |
So fix could help also for SSD wearing a very tiny little bit :) |
|
Seems possible to remove "(cell_id = 'zz' AND photo_lat <> 0)" all together. I would think this was meant to resolve coordinates in case a request failed. However, the metadata will be periodically checked/updated either way. So the worst thing that could happen is that the location information is added with a longer delay (if available). |
Signed-off-by: Michael Mayer <michael@photoprism.app>
Signed-off-by: Michael Mayer <michael@photoprism.app>
Signed-off-by: Michael Mayer <michael@photoprism.app>
|
It would be great if you could test these changes with trace log mode enabled: An updated development preview build will be available for this soon. Thank you very much! Edit: Should location information be missing, you can now run |
lastzero
added
please-test
This should effectively prevent all related disk activity. Signed-off-by: Michael Mayer <michael@photoprism.app>
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero
changed the title
|
I decided to remove the album backups completely from the Moments Worker because even checking if the directory already exists could cause disk activity in your case. |
|
Our release notes have been updated to mention this change: |
|
Testing preview over night. |
|
Disks are still sleeping. 👍 |
In the next step, this worker can be configured to automatically create index and/or album backups at regular intervals. Signed-off-by: Michael Mayer <michael@photoprism.app>
Signed-off-by: Michael Mayer <michael@photoprism.app>
|
@sgflt After further testing, we found that the timestamp of the background worker was reset after each run, as a new instance was created every time. Also, the album/moment update methods did not check if the data had changed. All this together explains why the YAML backup files were constantly updated in the background. Both should be fixed now. Thank you for bringing this to our attention! |
Signed-off-by: Michael Mayer <michael@photoprism.app>
graciousgrey
added
tested
1. What is not working as documented?
I was curious why disk storage was spinning up each hour.
It corelated to PHOTOPRISM_WAKEUP_INTERVAL: 3600
2. How can we reproduce it?
Steps to reproduce the behavior:
Check history of power usage.
3. What behavior do you expect?
Backup to yaml only if there is change in metadata.
6. Which software versions do you use?
(a) PhotoPrism x86_64 & Build 240420-ef5f14bc4
(b) Database Type MariaDB
(c) Operating System: Fedora 40, Podman 5.0.2, SELinux
7. On what kind of device is PhotoPrism installed?
(a) Device / Processor Type: Intel Celeron N5105
(b) Physical Memory & Swap Space in GB 16 GiB
(c) Storage Type: HDD main data, SSD storage/cache
(d) Anything else that might be helpful to know?
After setting PHOTOPRISM_DISABLE_BACKUPS: "true" are wake ups gone.
The text was updated successfully, but these errors were encountered: