You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using this package in Node.js environment and call cohere.chatStream to generate long Chinese texts. However, The replacement character (�) appears in random places in 'stream-end' event. The following code may convert incomplete UTF-8 chunks, which are yielded in parts by stream, into strings.
While Latin Basic characters require only 1 byte in UTF-8, other characters, such as CJK characters, need more bytes to encode. This means there's a chance that a character could be split across chunks.
The text was updated successfully, but these errors were encountered:
I'm using this package in Node.js environment and call
cohere.chatStream
to generate long Chinese texts. However, The replacement character (�) appears in random places in'stream-end'
event. The following code may convert incomplete UTF-8 chunks, which are yielded in parts bystream
, into strings.cohere-typescript/src/core/streaming-fetcher/Stream.ts
Lines 29 to 32 in 40c146c
While Latin Basic characters require only 1 byte in UTF-8, other characters, such as CJK characters, need more bytes to encode. This means there's a chance that a character could be split across chunks.
The text was updated successfully, but these errors were encountered: