Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default #6347

Merged
merged 4 commits into from
May 10, 2024
Merged

Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default #6347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
180bc9b
Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default
aisha-partha May 8, 2024
3590a1f
Closes #6342. Update documentation on METAREFRESH_IGNORE_TAGS default…
aisha-partha May 8, 2024
2e13a9b
Update METAREFRESH_IGNORE_TAGS default value in documentation
aisha-partha May 8, 2024
fe163d9
Fix typo
Gallaecio May 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/topics/downloader-middleware.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,10 @@ Meta tags within these tags are ignored.
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
``['script', 'noscript']`` to ``[]``.

.. versionchanged:: VERSION
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
``[]`` to ``['noscript']``.

.. setting:: METAREFRESH_MAXDELAY

METAREFRESH_MAXDELAY
Expand Down
2 changes: 1 addition & 1 deletion scrapy/settings/default_settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@
MEMUSAGE_WARNING_MB = 0

METAREFRESH_ENABLED = True
METAREFRESH_IGNORE_TAGS = []
METAREFRESH_IGNORE_TAGS = ["noscript"]
METAREFRESH_MAXDELAY = 100

NEWSPIDER_MODULE = ""
Expand Down
5 changes: 2 additions & 3 deletions tests/test_downloadermiddleware_redirect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,9 +395,8 @@ def test_ignore_tags_default(self):
"""content="0;URL='http://example.org/newpage'"></noscript>"""
)
rsp = HtmlResponse(req.url, body=body.encode())
req2 = self.mw.process_response(req, rsp, self.spider)
assert isinstance(req2, Request)
self.assertEqual(req2.url, "http://example.org/newpage")
response = self.mw.process_response(req, rsp, self.spider)
assert isinstance(response, Response)

def test_ignore_tags_1_x_list(self):
"""Test that Scrapy 1.x behavior remains possible"""
Expand Down