Integration with GeoArrow and Lonboard for large-scale interactive visualization/animation

[kylebarron]

👋 This is a really cool library, and I wonder if there might be some mutual benefit to integrating our work!

In the quickstart you note

for quick interactive representations with few elements, the Leaflet widget is a good option

ipyleaflet indeed is not very performant. I've been working on an open source library called Lonboard which connects to deck.gl. By using Arrow and Parquet efficiently, it can render ~3 million coordinates in an interactive map from Python in a few seconds.

deck.gl was originally built by Uber and since they also wanted to visualize large numbers of moving objects, the JS library includes a TripsLayer that animates moving objects (example, note that reducing the "trail" slider on the right makes each trajectory clearer).

I've started but haven't yet finished a PR that connects this to Python. That PR's description has a screen recording of an example animation from ship AIS data, but I think air traffic data would make a really compelling demo. With that PR you should be able to render trajectories containing 3 million coordinates, and ideally be able to set a different color per coordinate based on some attribute like current speed.

Under the hood, Lonboard works by converting data to the GeoArrow data format, which for an array of LineStrings is essentially raw arrays of coordinates for the entire column plus an offset array to determine at which indices each LineString starts and ends. So what I'd need to implement is the adapter from a Traffic object to a pyarrow.Table with timestamps at the same level of nesting as the LineString column.

Side note: I see in https://github.com/xoolive/rs1090 one of the goals is "exporting trajectory data to cross-platform formats such as JSON or parquet". You might be interested in using (Geo)Arrow for this, which would allow for sharing large data between Rust and Python without copies. I'm building a Rust-Python (and Rust-Wasm) library for GeoArrow that benefits from being able to move Arrow data between Rust and Python for free.

Do you think this kind of integration makes sense? Is this interesting? If so, is there a data source to play with that contains multiple flights at the same time in the same airspace? I'm thinking something like a 30-minute window of flights in congested airspace like NYC would make for a really cool visualization. I haven't yet figured out a good way to access data like that though.

[xoolive]

Hello @kylebarron

I am quite interested with your project, looks really cool!

So there's a reason why I am not using the geopandas from the start. That's a question I asked myself early on and the thing is that:

• most analyses we do are actually not related with the positional information,
• geopandas does not extend well when you want to attach more features beyond the lat/lon/alt data.

So in traffic paradigm, a trajectory is not a line in a dataframe but a sub-dataframe.

I have some functions to iterate over flights, and the summary() maps flights to a row with attributes passed in parameters. So I tried the following code (which could be a start for what you need?) and it obviously needs some fine-tuning. :) This is data for Zurich airport which is already integrated in the library, but we can build a new dataset with New York at some point for your demo if you want. That's 2 months of data but you can reduce the scope easily.

Does it help?

Side note about Rust: I am interested too! For now, I am basically serializing vectors of JSON through pickle when passing decoded messages from Rust to Python, but it's not ideal... (I am new to Rust, this is my first project!) Considering my comment above I am not sure it would fit well though, but will have a look. About what I read here wouldn't it be a smarter move to stick to a full Rust (more simple) implementation like https://github.com/busstoptaktik/geodesy/ ?

# %%

import geopandas as gpd
from traffic.data.datasets import landing_zurich_2019

summary = (
    landing_zurich_2019.resample("10s")
    .summary(["start", "stop", "callsign", "icao24", "linestring"])
    .eval(desc="", max_workers=4)
)
gdf = gpd.GeoDataFrame(summary.rename(columns=dict(linestring="geometry")))

# %%
from lonboard import viz

viz(gdf)

[kylebarron]

in the browser, it's super smooth

Ok, so at least we've confirmed that in some setup it works well. Hopefully that means the issue is outside of my GeoArrow glue and instead in some browser webview interactions.

For me, on a Mac M2, it's just as fast in VSCode as it is in Chrome.

You already know the option in vscode maybe? I am never starting a jupyter server actually... 😰

Vscode will automatically start a jupyter process under the hood for you

[xoolive]

yes it's probably on me, but if it's good I get through everything before others do, at least I can process most of the stuff under the hood then (that's how traffic was born)

vscode

I usually use the # %% nodes to run an ipykernel. I would like very much that it works as well on my Linux too!

[kylebarron]

I usually use the # %% nodes to run an ipykernel.

You don't actually have to define those blocks manually. See the gif in https://github.com/kylebarron/vscode-jupyter-python

[xoolive]

🤯

[xoolive]

@kylebarron so in Ubuntu, vscode through snap doesn't get access to gpu... but with the .deb file it does... mystery solved.

now i can start wrapping things 👍