Maximum data limit?

[vpv97]

What is the maximum number of rows/columns that pygwalker can handle? I have a pretty big machine so that shouldn't be a bottleneck.

[Asm-Def]

It'll be interesting to optimize the performance.

For now, all the data are printed into the browser and processed in memory, which is limited by the browser process

[EdwardMarsDS]

I tried using it on a subset of my data that was 21k rows and 420 columns... this is what I got. All in my 32 gb ram laptop...

[Asm-Def]

Hello, I recently made a version that supports loading data dynamically, and while it's not robust enough, it makes better use of the memory the browser process allows. Please check it out at #104 or https://github.com/Kanaries/pygwalker/releases/tag/0.1.8.dev0

[bikram20]

@Asm-Def I tried your patch. But I still get the same issues. Running Jupyter Lab (Edge browser) on Mac with 16GB RAM.

Dataframe is too large for ipynb files. Only 20129 sample items are printed to the file.
Dynamically loading into the frontend... 20129/832387

[longxiaofei]

Hi, after dynamically loading end, can you see all the data on the page render by pygwalker?

[longxiaofei]

I also found that there is a problem with this feature recently, and this feature will be fixed in the next version.

[bikram20]

I did not. Thank you for the closure. BTW, Pygwalker is a great product!

[longxiaofei]

I have fixed this bug in a new PR, but the new release is a few days later.

[longxiaofei]

fixed in version 0.1.11, thanks for your feedback. ♪(･ω･)ﾉ

[longxiaofei]

Hi, in next version, we will support computing with duckdb.

In the POC test, we can use datas(10G+, 10 columns, 200W+ row), the result can be obtained within 1s.