Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add support for custom exporter class #6273

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

add support for custom exporter class #6273

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d1a0bed
support deferred returns from export_item
guillermo-bondonno Mar 22, 2024
bb8ff74
undo FeedExporter changes because we get can achieve the same only fr…
guillermo-bondonno Mar 22, 2024
202f4ab
allow custom finish exporting to be async
guillermo-bondonno May 16, 2024
eb7d6f6
add tests for async finish_exporting and update docs
guillermo-bondonno May 17, 2024
3438432
fix naming of finish_exporting_deferred
guillermo-bondonno May 17, 2024
13a5c3e
solve conflicts
guillermo-bondonno May 17, 2024
32750db
ignore typing check in close_slot
guillermo-bondonno May 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 9 additions & 3 deletions scrapy/extensions/feedexport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,9 @@ def _get_exporter(self, file, format, *args, **kwargs):

def finish_exporting(self):
if self._exporting:
self.exporter.finish_exporting()
result_or_coro = self.exporter.finish_exporting()
self._exporting = False
return result_or_coro


_FeedSlot = create_deprecated_class(
Expand Down Expand Up @@ -481,11 +482,11 @@ def get_file(slot_):

if slot.itemcount:
# Normal case
slot.finish_exporting()
finish_exporting_maybe_deferred = defer.maybeDeferred(slot.finish_exporting)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd remove maybe from the name, it's an actual deferred.

elif slot.store_empty and slot.batch_id == 1:
# Need to store the empty file
slot.start_exporting()
slot.finish_exporting()
finish_exporting_maybe_deferred = defer.maybeDeferred(slot.finish_exporting)
else:
# In this case, the file is not stored, so no processing is required.
return None
Expand All @@ -507,6 +508,11 @@ def get_file(slot_):
)
d.addBoth(lambda _: self._pending_deferreds.remove(d))

self._pending_deferreds.append(finish_exporting_maybe_deferred)
finish_exporting_maybe_deferred.addBoth(
lambda _: self._pending_deferreds.remove(finish_exporting_maybe_deferred)
)

return d

def _handle_store_error(self, f, logmsg, spider, slot_type):
Expand Down