Complex pipelines, hierarchical pipelines, and "PipelineRunner" as a component #7623
Unanswered
mikebellerU
asked this question in
Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi -- I've tried a lot of LLM orchestration tools out there and haystack seems really promising. Thanks
for making it available!
I am currently building complex RAG pipelines -- often involving multiple routes and parallel
execution. I have built one out with haystack, and I find that the pipelines are getting too
complex. I'd like to break them down into testable 'sub-pipelines' and run the smaller
ones from a higher-level pipeline. This would keep the pieces testable, and help make
the whole pipeline more understandable.
I noticed there doesn't seem to be
any PipelineRunner component, which might enable this. Is there a reason for that?
I'm suggesting something
conceptually like this. (Non working pseudo code:):
Does anyone know -- Is this sort of hierarchical pipeline concept something being done already,
or perhaps on the roadmap?
Note: I can imagine also that a "ParallelPipelineRunner" would be an awesome next step after that, which
could allow reusing a sub-pipeline but running in parallel (using ThreadPoolExecutor for example) or on a cluster.
Beta Was this translation helpful? Give feedback.
All reactions