Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DateRange Timestamp conversion on windows fails with timestamps close to EPOCH #261

Open
bdemirtas opened this issue Apr 10, 2024 · 0 comments

Comments

@bdemirtas
Copy link

bdemirtas commented Apr 10, 2024

Expected Behavior

When you use DateRange with starting date before 1970 (EPOCH) it raise OSError.
OSError: [Errno 22] Invalid argument
Linking the bug ticket from Python
https://bugs.python.org/issue37527

Current Behavior

Currently it works as intended for any non Windows OS . The work around is to provide the datetime with a timezone utc.

Steps to Reproduce (for bugs)

This code will fail and raise OSError on windows.

testDataSpec = (
    dg.DataGenerator( spark, name="test_data_set1", rows=1000 partitions=4)
    .withColumn(
        "purchase_date",
        "date",
        data_range=dg.DateRange("1910-10-01 00:00:00", "1950-10-06 11:55:00", "days=3"),
        random=True,
    )
)

Context

Your Environment

  • dbldatagen version used:
  • Databricks Runtime version:
  • Cloud environment used:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant