Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Uniform MapCompose-like processor #37

Open
Chratho opened this issue Feb 10, 2016 · 0 comments
Open

Feature Request: Uniform MapCompose-like processor #37

Chratho opened this issue Feb 10, 2016 · 0 comments
Labels
enhancement New feature or request

Comments

@Chratho
Copy link

Chratho commented Feb 10, 2016

It took me some time to figure out that MapCompose flattens iterables returned by the specified functions. Even though I then I realized that this behavior is well documented, I still wonder: Why is that? To be honest, it feels only appropriate for highly-specific use-cases. It works fine for unary values, but if people start using it on multi-valued fields (e.g., tuples) the results might be counter-intuitive.

The following code is a slightly adapted version of MapCompose, mostly just using append instead of += for value-concatenation:

class ListCompose(object):

def __init__(self, *functions, **default_loader_context):
    self.functions = functions
    self.default_loader_context = default_loader_context

def __call__(self, values, loader_context=None):
    if loader_context:
        context = MergeDict(loader_context, self.default_loader_context)
    else:
        context = self.default_loader_context
    wrapped_funcs = [wrap_loader_context(f, context) for f in self.functions]
    for func in wrapped_funcs:
        next_values = []
        for v in values:
            next_values.append(func(v))
        values = next_values
    return values

Since nearly my all fields are initially created by add_xpath (an approach where Compose turns out to be pointless), I ended up using ListCompose a lot (just like MapCompose it works for unary fields as well

@Gallaecio Gallaecio transferred this issue from scrapy/scrapy Oct 30, 2020
@Gallaecio Gallaecio added the enhancement New feature or request label Feb 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants