Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0 Byte media files saved while network or disk error #257

Open
yfdyh000 opened this issue Jul 2, 2022 · 2 comments
Open

0 Byte media files saved while network or disk error #257

yfdyh000 opened this issue Jul 2, 2022 · 2 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@yfdyh000
Copy link
Collaborator

yfdyh000 commented Jul 2, 2022

Describe the bug
I haven't checked further, but I'm experiencing this problem since a long time ago to the latest v2.7.0 version.
In simple terms, I see some pictures or videos in downloaded files are 0 bytes and their modification time is related to network outage or lack of disk space. These files are not retried for download, they are logged to the index as if the download was successful.

To Reproduce
Steps to reproduce the behavior (not exact):
a. Make sure the TumblThree uses a proxy server (e.g. via proxy agent), start download until the media file is downloading, and then stopping the proxy agent, check the downloaded media files. I'm not sure this has an effect, it might work fine if a retry is requested, but significantly if a proxy is necessary for the connection.
b. Do like above, but stop your network (e.g. network adapter in Device Manager or router).
c. Similar approach but set the download directory to a space-starved partition, and downloads until the out-of-disk error occurs.

Expected behavior
a. No 0-byte file is saved when downloading error occurs. Or write files with .tmp, and change the name when the download is complete, that is, safe write.
b. At least, don't save it to the downloaded index.
c. Allow to check and redownload for the downloaded file is 0 byte.
d. Check for free disk space earlier instead of throw error when unable to write.

Screenshots
N/A

Desktop (please complete the following information):

  • TumblThree version: v2.7.0 and earlier.
  • OS: Win10
  • Browser N/A
  • Version N/A

Additional context
This happens for Tumblr and Twitter files. Sorry in advance if this is a known bug.

@thomas694
Copy link
Contributor

Thanks for reporting a bug.
When it run out of disk space here, the crawlers were stopped (due to errors) and TumblThree "waited". Then after creating some free space again, it was possible to close the app and it saved some changed blog files. As far as I remember I didn't see 0 byte files.
As to proxy usage I cannot say much, as I don't use one.

But further investigations could be made when there is time.

@thomas694 thomas694 added bug Something isn't working help wanted Extra attention is needed labels Jul 3, 2022
@yfdyh000
Copy link
Collaborator Author

yfdyh000 commented Jul 3, 2022

I've noticed that even manually stopping the crawler task, it also produces incomplete or empty files and doesn't resume the downloaded files later.

In the DownloadFileWithResumeAsync, the TODO says it should be rewritten.
File.Delete(destinationPath); may be needed while catch (WebException webException) if resume not work.
Incomplete files seem likely to trigger the ProgressSkipFile.

I was expecting an action/option to redownload incomplete files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants