Type is not JSON serializable: Timestamp

[kiwamizamurai]

dlt version

0.4.10

Describe the problem

I cannot do elt from mongodb to bigquery, especially ingestion from mongo

Actually, this problem resides within the dependent package orjson
and there is an opening issue. ijl/orjson#442

Error message

<class 'TypeError'>
Type is not JSON serializable: Timestamp

 {'allUsers': [{'db': 'admin', 'user': 'hogehoge'}],
  'appName': 'MongoDB Compass',
  'client': 'xxxxxxxxxxx',
  'command': {'$clusterTime': {'clusterTime': Timestamp(1686813362, 110),
                               'signature': {'hash': b'xxxxxxxx',
                                             'keyId': xxxxxxxxxxxx}},
              '$db': 'test',
              'aggregate': 'fugafuga',
              'cursor': {},

TypeError: Timestamp(1686813362, 110) is not JSON serializable

https://mongodb.github.io/node-mongodb-native/4.0/interfaces/clustertime.html

Expected behavior

no error occur and the elt finishes successfully

Steps to reproduce

working on it

Operating system

macOS

Runtime environment

Local

Python version

3.11

dlt data source

mongo

dlt destination

Google BigQuery

Other deployment details

Referred this article

• https://www.holistics.io/blog/modeling-unstructured-data-mongodb/

Additional information

No response

[sultaniman]

Hey @kiwamizamurai thanks for feedback. I am wondering if you are using our mongodb source if so then you can adjust it and make explicit conversion like?

def convert_mongo_objs(value: Any) -> Any:
    if isinstance(value, (ObjectId, Decimal128)):
        return str(value)
    if isinstance(value, _datetime.datetime):
        return ensure_pendulum_datetime(value)
    return value