Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Bug: retry scrape will lost POST requestData #794

Merged
merged 1 commit into from
Mar 25, 2024

Conversation

Shinku-Chen
Copy link

@Shinku-Chen Shinku-Chen commented Nov 29, 2023

when retry scrape
requestData will loss in http.NewRequest

so Seek requestData before scrape.NewRequest

req.ContentLength  always 0 and

if req.GetBody != nil && req.ContentLength == 0 {
			req.Body = NoBody
			req.GetBody = func() (io.ReadCloser, error) { return NoBody, nil }
		}

@Shinku-Chen Shinku-Chen changed the title Fix post retry loss data, Seek requestData before do scrape Fix Bug: retry post will loss data, so Seek requestData before scrape.NewRequest Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry post will loss data, so Seek requestData before scrape.NewRequest Fix Bug: retry scrape will loss requestData , so Seek requestData before scrape.NewRequest Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss requestData , so Seek requestData before scrape.NewRequest Fix Bug: retry scrape will loss requestData Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss requestData Fix Bug: retry scrape will loss POST requestData Dec 25, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss POST requestData Fix Bug: retry scrape will lost POST requestData Dec 25, 2023
@Shinku-Chen
Copy link
Author

@WGH-

Copy link
Collaborator

@WGH- WGH- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good idea to limit the API to seekable bodies.

However, we might try to rewind the body if it happens to implement io.Seeker. And it might be a good idea to make Retry retry an error if doesn't.

@Shinku-Chen
Copy link
Author

是个好主意
现在已经按照需求修改了代码
请帮忙检查一下 感谢

That's a great idea.
The code has now been modified according to the requirements.
Please help me check it out, thank you.

colly.go Outdated Show resolved Hide resolved
@WGH-
Copy link
Collaborator

WGH- commented Mar 25, 2024

Otherwise, looks good

When calling Request.Retry, and request has a body (e.g. POST form or file
upload), try to rewind it.  If it's not seekable, return
ErrRetryBodyUnSeekable error.
@WGH- WGH- merged commit 4ccfe78 into gocolly:master Mar 25, 2024
7 checks passed
@WGH-
Copy link
Collaborator

WGH- commented Mar 25, 2024

And thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants