Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize fork #15

Open
stoney95 opened this issue Nov 14, 2021 · 0 comments
Open

Parallelize fork #15

stoney95 opened this issue Nov 14, 2021 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@stoney95
Copy link
Owner

Context

The package is providing the fork function. This function takes multiple functions and forwards the intermediate result to every of these functions. By this multiple independet branches are created that all consume the result of the previous pipeline step.

Description

Currently the fork function is executing the branches sequentially. It would be nice to have a version that is parallelizing these steps.

Hints

  • A new module should provide a parallel version of fork. The new function should be imported by from pypely.parallel import parallel_fork
  • It needs to guaranteed that the order of the result is kept. The order the functions are given in needs to reflect the order of their outputs.
  • Take care about variable references. It might be reasonable to create a copy of the intermediate result.
@stoney95 stoney95 added enhancement New feature or request good first issue Good for newcomers labels Nov 14, 2021
@stoney95 stoney95 pinned this issue Nov 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant