Issues with control characters.

[JoelKronander]

Is your feature request related to a problem? Please describe.
I have a weirdish use case, where one of the fields of the pydantic model represents code.
The code is often returned with a bunch of invalid json characters in it, like control characters (\u0000-\u001F).

This makes instructor fail on errors like this:
File "/opt/homebrew/lib/python3.11/site-packages/pydantic/main.py", line 530, in model_validate_json return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pydantic_core._pydantic_core.ValidationError: 1 validation error for RustCode Invalid JSON: control character (\u0000-\u001F) found while parsing a string at line 4 column 0 [type=json_invalid, input_value='\n{\n"generated_code": "...xample_output": "37"\n}', input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/json_invalid joelkronander@MacBook-Pro-5 swissknife %

Describe the solution you'd like
Maybe one could handle cases like this with some form of "pre-validators" that could for example run byte64 encoding on those non-json compatible strings? Not sure how it would fit in exactly.

Additional context
Instructor is nice.

[jxnl]

interesting this might be more of a pydantic issue than instructor. maybe it should come from a different strict mode?
I'll see if i can pass the modes across json and pydantic

[jxnl]

@JoelKronander Let see if this work, can you verify that control characters work with strict mode is turned off?

[jwmarshall]

@jxnl I was not able to get this working without removing the newlines manually first.

[jxnl]

this is super dumb but sometimes you gotta remind it to escape new lines in the prompt, acuse the control characters are an issue unless you use the latest models.

[jwmarshall]

Indeed. My best fix was the following prompt addition with gpt-4-turbo-preview YMMV.

You will escape all control characters and newlines, this is extremely important and necessary to producing valid output.