[Core feature] Support for Native JSON types in flyte

[kumare3]

Motivation: Why do you think this is important?

Currently flyte supports JSON through protobuf Struct. This causes a lot of problems for the users, especially when using ints as Structs only store numbers - which are double values.

Goal: What should the final outcome look like, ideally?

Support a new FlyteIDL.LiteralType called LiteralType.JSON. So users can simply use any object that dumps to a JSON using binary / bytearray as the transport layer.

It should be possible to also have customer json serializers and deserializers for the data like JsonPickle etc

Describe alternatives you've considered

New literalType called LiteralType.JSON
This maps to a Literal of type binary (wire format)
Optionally the metadata / schema can be extracted and stored

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

To support native JSON types in Flyte, you propose introducing a new LiteralType.JSON in FlyteIDL, using binary/bytearray for transport, and allowing custom JSON serializers/deserializers. This approach aims to address issues with current JSON handling via protobuf Struct, particularly with integer representation.

Would you like more detail on how to implement this feature or any potential challenges?

Sources:

• GitHub Issue #4505
• GitHub Issue #1282
• Discuss Flyte

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[kumare3]

Related to #4505

[MortalHappiness]

#take

[thomasjpfan]

From #4505 (comment), I see the follow Python types that can use Literal.JSON:

1. dicts -> JsonPickle
2. dataclasses -> Use Mashumuro's encoders/decorders or the object's to_json + from_json (These methods are defined with DataClassJSONMixin)

@kumare3 Are you considering other Python types that would use Literal.JSON?

[kumare3]

I am saying wholesale migration to json for these unsupported types from struct

[wild-endeavor]

#5337