Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge responses from InferenceGraph Sequence node steps #3639

Open
asd981256 opened this issue Apr 27, 2024 · 2 comments · May be fixed by #3692
Open

Merge responses from InferenceGraph Sequence node steps #3639

asd981256 opened this issue Apr 27, 2024 · 2 comments · May be fixed by #3692

Comments

@asd981256
Copy link

asd981256 commented Apr 27, 2024

/kind feature
I want sequence node to merge the responses from steps(could select which steps).

But I can't find a way to do it.
according to document and cmd/router/main.go
Ensemble: routes the request to multiple models and then merge the responses

and in main.go, Ensemble Node use same input in gorountines


output, statusCode, err := executeStep(step, graph, input, headers)

and I observe that sequence node only return result from final step, then check main.go.


for i := range currentNode.Steps {
	...
	responseBytes, statusCode, err = executeStep(step, graph, request, headers)
	...

}

return responseBytes, statusCode, nil

I suppose there is no such feature. (or is there any workaround?)

Describe the solution you'd like
specify responses to be merged in yaml file of InferenceGraph sequence node.

add a field in InferenceStep
beMerged: bool (optional)(if not explictly set true, response from this step is not merged)
and let Sequence Node collects these responses and merge them maybe like Ensemble Node
{${STEP_NAME}: ${RESPONSE_FROM_STEP}, ...}

Example:
if some step response is wanted

{
	"stepName2" : {"prediction" : "2"}
	"finalStepName" : {"prediction" : "3"}
}

stepName could be empty, then use index(which start from 0) instead

{
	"1": {"prediction": "2"}
	"2" {"prediction": "3"}
}

if no other step response wanted, maintain same structure

{
	"prediction": "3"
}

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Links to the design documents:
[Optional, start with the short-form RFC template to outline your ideas and get early feedback.]
[Required, use the longer-form design doc template to specify and discuss your design in more detail]

@asd981256
Copy link
Author

/assign @asd981256

@yuzisun
Copy link
Member

yuzisun commented May 18, 2024

@asd981256 Just curious about the use case of merging the steps from sequence node

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants