Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EscapeString #440

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

EscapeString #440

wants to merge 11 commits into from

Conversation

rhinogo
Copy link

@rhinogo rhinogo commented Jul 14, 2023

The official Python package, when converting objects to JSON strings, will convert Chinese and other text into Unicode encoding, and can effectively avoid sensitive words in responses. This modification is to enable the Golang package to have this capability.

@rhinogo rhinogo closed this Jul 14, 2023
@rhinogo rhinogo reopened this Jul 14, 2023
@rhinogo rhinogo closed this Jul 14, 2023
@rhinogo rhinogo reopened this Jul 14, 2023
@rhinogo rhinogo closed this Jul 14, 2023
@rhinogo rhinogo reopened this Jul 14, 2023
@rhinogo rhinogo closed this Jul 14, 2023
@rhinogo rhinogo reopened this Jul 14, 2023
Role string `json:"role"`
Content string `json:"content"`
Role string `json:"role"`
Content EscapeString `json:"content"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you adopt a method that doesn't involve breaking changes?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not very familiar with other implementation methods.

@sashabaranov
Copy link
Owner

Hey, thank you for the PR! Could you please provide an illustration where default Go's string handling doesn't work? And what could be our alternatives here?

@codingknees
Copy link

when there are Emoji in string, Marshal would fail:

json: error calling MarshalJSON for type openai.EscapeString: invalid character 'U' in string escape code

for example, 😗 will be quoted to ASCII \U0001f617 , but it is not valid in json (encoding/json/scanner.go:353).
Because of the \U

@codingknees
Copy link

when there are Emoji in string, Marshal would fail:

json: error calling MarshalJSON for type openai.EscapeString: invalid character 'U' in string escape code

for example, 😗 will be quoted to ASCII \U0001f617 , but it is not valid in json (encoding/json/scanner.go:353). Because of the \U

I find that in python, json.dumps encode that Emoji to \ud83d\ude17. So strange

@codingknees
Copy link

hack as
` EscapeString string

func (esc EscapeString) MarshalJSON() ([]byte, error) {
x := strings.ReplaceAll(strconv.QuoteToASCII(string(esc)), "\U", "\u")
return []byte(x), nil
}`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants