Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pyspark & ibis capture to SDK #876

Closed
wants to merge 3 commits into from
Closed

Add pyspark & ibis capture to SDK #876

wants to merge 3 commits into from

Conversation

skrawcz
Copy link
Collaborator

@skrawcz skrawcz commented May 2, 2024

adds sdk capture of some basic ibis and pyspark info.

Changes

  • sdk

How I tested this

  • locally

Notes

  • need to create tests

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

These are lazily done - so best we can do is get some
schema and maybe a query plan in the case of pyspark.
Because some dependencies might not exist -- because they
are lazily loaded. This guards against that.
@skrawcz skrawcz changed the title Add pyspark ibis Add pyspark & ibis capture to SDK May 2, 2024
@zilto
Copy link
Collaborator

zilto commented May 2, 2024

Ibis 9.0 added a .describe() method (equivalent to pd.DataFrame().describe())

@skrawcz
Copy link
Collaborator Author

skrawcz commented May 2, 2024 via email

@skrawcz skrawcz marked this pull request as ready for review May 5, 2024 22:46
@skrawcz skrawcz marked this pull request as draft May 5, 2024 22:58
@skrawcz
Copy link
Collaborator Author

skrawcz commented May 5, 2024

Just need to add some tests.

@skrawcz
Copy link
Collaborator Author

skrawcz commented May 13, 2024

closing in support of #895

@skrawcz skrawcz closed this May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants