What is the vision for godot-python with GDExtension?

[konsuko]

Hi there,

Really happy to find that this exists and is being converted to work with the upcoming 4.0 release :)
I love Godot and GDScript, but very often I find myself needing various utility libraries and the Python ecosystem is a great place to find them, which is a primary motivation to use Python with Godot in the future.

Seeing as GDExtension changes the way native extensions work quite a bit, I was wondering:

How will godot-python work in the future?

Right now, I was under the impression that this was a project that works like a module, allowing Python script files to work equivalently to GDScript?

From what I gathered about GDExtension, this might not work anymore? I thought that GDExtension would let us compile any language that can interface with the Godot C API/Bindings straight to native dynamic libraries, then allow us to use the Nodes declared in those libraries straight from the Godot Editor?

Anyways, since I'd love to try this out once it is reworked, I'd love to hear what direction you are taking this in.

EDIT: I'd also be super interested in how this compares to GDScript in terms of performance. Is it way slower? Same speed? ... faster?

Pinging @touilleMan just in case :)

[konsuko]

I have done some more thinking about what I would want out of using Python with Godot in the future.

For me, the biggest strength of Python is in its massive collection of accessible libraries. It's the only language (maybe other than JS) where I can find a library for anything and get it running, quickly.
One scenario I can think of is e.g. generating PDF files or parsing PSD files from Python scripts, which I would love to be able to do directly in my Godot app instead of externally.
For a lot of use cases, there are no GDScript libraries, and that's where Python comes in.

A major roadblock I see with this, though, is the fact that exporting the Godot app to all supported platforms needs to be possible when using Python as well. If this is not possible, it kind of defeats the purpose of using Python with Godot in the first place.

What I mean is: We would need to be able to compile our games with Python to ALL platforms that Godot supports: Desktop, Mobile, Web and also Consoles (in the future).

I am not sure if this is a possibility, but it would be incredibly important if this library's goal is to become production-ready for serious projects, which I hope it attempts to become.

[touilleMan]

Hi @konsuko

Godot 3 extension system work with 2 APIs:

• GDNative: api to load a 3rd party library, and to call Godot from this library
• PluginScript: api to expose a new language to Godot

So basically in Godot-Python with Godot3 , GDScript code calling into a Python class goes through PluginScript, where Python code calling Godot goes through GDNative.

From what I gathered about GDExtension, this might not work anymore?

GDExtension is basically GDNative+PluginScript on steroid, so nothing is going to change (when GDExtension reached master, it didn't support what PluginScript do, but it has been fixed since then ^^)

I thought that GDExtension would let us compile any language that can interface with the Godot C API/Bindings straight to native dynamic libraries, then allow us to use the Nodes declared in those libraries straight from the Godot Editor?

That's true, which means you can use GDExtension to create a new class that inherits ScriptLanguageExtension, and now you've added support for a new script language ! That's what Godot-Python is doing.

But ! There is another goal I'd like to provide: the ability to use Cython to code GDExtension directly. This way you could start writting regular Python code for fast and easy dev, then later on rewrite some parts in Cython and get maximum performances just like if you code were written in C \o/

For me, the biggest strength of Python is in its massive collection of accessible libraries.

Totally agree ;-)

is the fact that exporting the Godot app to all supported platforms

Yes, Python's poor support of non-desktop platform is definitely an issue !
But on the bright side, this issue is well understood and there is some work (for instance https://pyodide.org/en/stable/) to improve that.

This is however outside of the scope of Godot-Python (I already have plenty of work for desktop !)

[konsuko]

@touilleMan Thanks so much for your in-depth reply, I really appreciate it! :)

Wow, this is very exciting for the future of Godot (and IMO Python as well).
Compiling Cython for performance gains also seems like a great idea.

I feel like – while I do like GDScript overall – Python's strengths almost perfectly complement Godot's weaknesses:

Godot has a brilliant, joyful workflow for creating GUIs, a superb editor, animation systems and has great cross-platform exports.
Python, on the other hand, lacks all that but offers basically everything else: Libraries for every single little utility thing you could think of.

I dream of making not only games, but non-game application software in Godot.
For the ideas I currently have, like I already stated: I need specific, niche libraries that I don't want (or can not) write myself.
I have looked into using C#, Go, Rust, Nim and Zig with GDExtension, but I found that none of them had the libraries that I needed (or at least not in a high enough quality).

None, except for Python.

Python's library ecosystem feels like playing with Legos sometimes (when it works, at least :D )
I feel that it fits perfectly with the way Godot feels, as well – after all, GDScript also feels a lot like Python.

Anyway, really hopeful and happy about the progress you seem to be making lately.
Maybe I can help somehow, once 4.0 is out: I am using macOS primarily, so I can help test stuff on that.
I also have console dev kits, so I could in theory try to get stuff running on that as well, although that's way out of my skill set currently.

I really hope, though, that web export is not entirely out of scope after 4.0, since that would be super important for me personally. Pyodide seems very promising indeed, maybe that could be used?

[touilleMan]

Maybe I can help somehow, once 4.0 is out: I am using macOS primarily, so I can help test stuff on that.

That would be very welcome ^^

I really hope, though, that web export is not entirely out of scope after 4.0

The issue with web support (not even considering the implementation complexity 😄 ) is that it will only support pure-python code.
This is also an issue with Android/iOS given the native modules must be cross-compiled (and Python is famously bad at doing that...)

On web it is even more complex than on native platform given compiling to web is a more exotic task.

And as you said, the big selling point in Python is the ecosystem, so having Python without the ability to use numpy/pytorch/pynacl/cryptography etc. is a bit sad :'(

[konsuko]

I just took a look at the pyodide docs and on the first page it says

Many packages with C extensions have also been ported for use with Pyodide. These include many general-purpose packages such as regex, pyyaml, lxml and scientific Python packages including numpy, pandas, scipy, matplotlib, and scikit-learn.

It looks like there are significant efforts to port the most popular C extensions to WASM as well, so maybe not all hope is lost :P
But yes, I agree: It's probably never going to be possible to run all Python modules in the browser, but that's true for all other languages that compile to WASM as well, no?

Anyway, I am looking for web export mostly for small game jam games/microgames, so maybe it's best to just stay in GDScript for those anyway.

Maybe the best approach is to not use Python scripting for 100% of the Godot game logic anyway, but really mix and match it with GDScript and aim for great interoperability. Was that a use case you aimed for?

EDIT: @touilleMan I am not super knowledgeable in how Python works cross-platform, yet, so I have another question:
Does compiling Cython scripts to C potentially allow for interop with other C APIs, such as the ones used by Console SDKs? I tried to grok how that all works but it seems really complicated.