Why is the completion not resolved in the promise ?

[mspronesti]

Hello and thanks a lot for this project !
I was just wondering why you don't resolve the data obeject in the promise here. In fact, I'd find it very useful to assign the completion result to a variable, which is not so convenient with a callback (in my humble opinion).

[hlhr202]

Hi, @mspronesti

The reason for using callback here is to stream tokens one by one so that you can display it in realtime(once the backend has inferred the next token) instead of waiting the accumulation of tokens all together which may cost a lot of time. Generally, users would like to see the results directly even they came out in sequence, but users will get frustrated if they wait the results for over several seconds(in some slow machine it get even worse). Thats why we choose to implement the streaming like this.

The promise here is to notify the caller that all the inferencing progress has just completed. it is not used for accumulating tokens in sentences.

[mspronesti]

That would help, thanks :)
However I still miss what difference would it make in terms of delay to resolve the vector of tokens (in the .End branch of the switch) instead of the boolean value. Isn't that the exact same streaming approach as before ?

[hlhr202]

emm i think i misunderstand your question from beginning. what you were asking is to keep the streaming but return the token strings instead of boolean, right? alright it really make sense and i may switch to this implementation in the future.

[mspronesti]

Yep, that's it. Thanks for taking it into considerations :)

[hlhr202]

https://github.com/Atome-FE/llama-node/pull/39/files#diff-d50d95d4d2e1defa9457fce8ea6f9d0672c5da30e533e069a998a9ce0f542ef2R72
will introduce new promise resolve/reject type like this