feat: RND-48: Parallel sync runtime #99
Conversation
niklub
changed the title
| elif self.concurrency > 1: | ||
| # run batch processing each row in a parallel way, using a fixed number of CPUs | ||
| logger.info(f"Running batch processing in parallel using {self.concurrency} CPUs") | ||
| pandarallel.initialize(nb_workers=self.concurrency, progress_bar=self.verbose) |
There was a problem hiding this comment.
Would still like to get a comment here explaining the doubled memory footprint, but not PR-blocking worthy
|
nbd if we don't have numbers for this, but would be nice to see what the speedup is here, if you have even some rough/anecdotal tests |
| """ | ||
|
|
||
| verbose: bool = False | ||
| batch_size: Optional[int] = None | ||
| concurrency: Optional[int] = Field(default=-1, alias='concurrent_clients') |
There was a problem hiding this comment.
Maybe a nitpick, but calling this var "concurrency" for both async and sync runtimes is possibly confusing, for async runtimes this is the level of concurrency and for sync runtimes it's the level of parallelism. Technically what we usually call parallelism in python is "concurrency plus parallelism", while what we call concurrency is "concurrency without parallelism". I can't think of a better name though, I guess it's okay 🙂
There was a problem hiding this comment.
yeah, I'd agree with your descriptions, and while I agree that we typically mean something more specific when we say "concurrency", imo it's fair enough to use "concurrency" as the umbrella term (since it technically is, and I also can't think of a better umbrella term 🙂 )
Nikolai, if you have a better term up your sleeve, fantastic, but I'm personally good with "concurrency" too, fwiw
concurrencyparameter top-level forRuntimebase class