LuaJIT

[LosFaul]

Is your feature request related to a problem? Please describe.

There are many things that are not possible todo or cause performance problems because standart Lua is too slow.

Describe the solution you'd like

Yes, you can try to optimice things as much as you can, but everything has it's limits.

I would say we first try to gather examples were LuaJIT, would bring us better performance.

Here are also a few examples were LuaJIT profits over standart Lua: http://luajit.org/performance_x86.html

[MegadreamsBE]

Here is an example of where I would benefit from LuaJIT:

Sphene (a SCM virtual machine in MTA with the purpose of building a Co-Op experience on top of the singleplayer missions) only really uses MTA functions to read the files and for the implementation of the actual game logic.

All the other logic (which is a lot of code, easily ran over hundreds of time per second) is just regular Lua that actually would see a nice performance boost if LuaJIT is available, this includes the logic that properly parses the parameters, handles if-statements, etc. This can finally help us get closer to getting Sphene to maintain a stable framerate which it currently is still struggling with.

[patrikjuvonen]

Here is our original LuaJIT branch as reference: https://github.com/multitheftauto/mtasa-blue/commits/LuaJIT

[LosFaul]

Simulations on cars

• complex aerodynamics: downforce, custom airresitance
• tempratures for different car parts e.g. engine, breaks, tires, cooling system, weather
• engine simulation (*1): rpm, power output
• gearbox simulation (*1): power translation from engine for each gear
• ...

In general any kind of simulation would profit from LuaJIT.
Also when simulation code is faster executed, you could save CPU time which means you could also make more features without killing player fps.

Another point would be all kinds of processing strings in pure Lua.

Note: I will edit my message when i find more

(*1) requires fix of setAnalogControlState: #567

[AlexTMjugador]

All the other logic (which is a lot of code, easily ran over hundreds of time per second) is just regular Lua that actually would see a nice performance boost if LuaJIT is available, this includes the logic that properly parses the parameters, handles if-statements, etc. This can finally help us get closer to getting Sphene to maintain a stable framerate which it currently is still struggling with.

In the meantime, on the server side, it's possible to use modules to interface Lua code with native C/C++ code, which can result in a nice performance boost, and the ability to communicate more easily with other languages and/or the underlying operating system. For a project like that, I guess you run most player-independent or synchronization logic in the server, so it could benefit from it. But nobody uses modules :(

Back on topic, I think LuaJIT is a good choice, but I'm concerned that it may generate some inconvenience for server owners that use compiled scripts, because the bytecode generated with the standard Lua interpreter isn't compatible with LuaJIT's. I think that any change in the Lua VM which affects bytecode should be carefully done, with these pros and cons in mind.

[MegadreamsBE]

For a project like that, I guess you run most player-independent or synchronization logic in the server, so it could benefit from it. But nobody uses modules :(

Most of Sphene actually runs on the client (especially SCM interpretation) and even synchronization will mostly not be done on the server as it adds a lot of unnecessary overhead (including the fact that the server has no knowledge about the map). Naturally sending the information between the clients still would go through the server and could be moved into a module if we'd add extra checks to validate this data.

[WADmitry]

LuaJIT has a huge overhead when switches context (call Lua code from C-side), AFAIK.
Also it has limit 1GB per process (IDK, whether it was fixed)

[blockspacer]

cling based C++ as scripting language / hot code reload
Why? Able to run C++ script in runtime or compile it for max speed ( as in example https://github.com/derofim/cling-cmake )

HOT code reload
possible approaches:

store app state
fix cling undo for files
https://root-forum.cern.ch/t/loading-unloading-class-as-interpreted-macro-in-cling-multiple-times/32976/2

execute cling code to change callbacks & variables
nested cling::Interpreter with multiple cling::MetaProcessor
IDK how to do it, but you can create child cling::Interpreter

[se16n]

Hi,
I see there was some efforts made to integrate LuaJIT.
Could you please tell what was the issues faced? As far as i understand LuaJIT can be used as drop-in replacement of Lua 5.1, at least it claims to be API+ABI compatible.

@patrikjuvonen

[patrikjuvonen]

I have no idea.

@Cazomino05 worked on LuaJIT, perhaps he might remember what the status was at the time on the attempt.

[Pirulax]

Theres a waay bigger context switch penalty with JIT than with regular Lua, thus having LuaJIT would only benefit if you dont use MTA functions a lot, which isnt feasible that much(bc of dx rendering).

Edit(2020.12.28): This comment is based on some random guy's comment over at discord, we really should do some tests, with actual gamemodes. IIRC theres a working LuaJIT MTA implementation already, made by a Hungarian guy.

[Kenix157]

Please checkout this
https://github.com/iponweb/luavela

Implementation of Lua 5.1, originally a fork of LuaJIT 2.0

[Pirulax]

A discussion I had with sbx over at discord.

Tldr: Sbx suggested we create a transpiler which would transpile Lua 5.1 native bytecode to LuaJIT 5.1. This should be implemented on MTA server, as all and every bytecode needs to be signed before its ran.
I disagreed with him, because I think people should just recompile their scripts, and creating such a transpiler would be another thing we'd need to maintain for no actual benefit IMHO.

[LosFaul]

Made some research about the Memory limit in LuaJIT
which seems to be fixed in LuaJIT 2.1

LuaJIT 2.1 is still in beta, but is still actively developed:
https://github.com/LuaJIT/LuaJIT

[Pirulax]

I think vanilla Lua 5.1 has the same limit. Quick Google confirmed my thoughts.
I don't think Lua needs more than a few hunderd MB of memory, even on large servers.
The 4GB limit is more than enough.
Does this limit exist on 64bit?

[Pirulax]

Okay, so I was thinking about this.
Sbx (as said above) suggested to write a transpiler of some sort.
Now, what if we provided 2 Lua dlls? One with JIT, and one without.
This way servers could continue using regular Lua until they recompile their stuff to JIT (if they want to).
I think this would be a good solution, as we wouldn't have to fiddle around with a transpiler.

[Pirulax]

Some DC discussion we had:
https://discord.com/channels/801330706252038164/801409941088239627/830885707240570941

[MegadreamsBE]

Before JIT:

With JIT (experimental build by @Pirulax without Lua BitOps)

With JIT (experimental build by @Pirulax with Lua BitOps)

1. JIT can definitely help projects like these.
2. MTA truly adds a lot of overhead.
3. No data yet on how bad the performance hit on C calls would be.

[Pirulax]

Nice, so its about a 3x increase at least.
Seems like C calls have a lot of overhead, because, the JIT bitop library isn't special at all.

[MegadreamsBE]

It'll come down then to whether or nor the tradeoff is generally worth it. In this particular project it is but that's because it only uses very few C calls. In most servers I believe they wouldn't notice the performance hit if any though (although dx calls perhaps... but this project has them as well and it doesn't affect it much) but once again... needs testing.

[MegadreamsBE]

Turns out I had the default fps limit on. It managed to run at a nice 60 fps:

Enabled unlimited FPS and removed the fps lock on the emulator which resulted in:

Do note that it does fluctuate but the emulator is actually playable with JIT.

[Pirulax]

Answering Einheit101:
The benefits of LuaJIT:

• Better performance (at least 2x)
• Smart compiler, which will allow you to use OOP without a performance hit

Distadvantages:

• Low limit for memory addresses on x86 (2GB, not sure why). Maybe we can work around this issue by pre-allocating low memory addresses, and giving them to JIT. (As this is just a memory address limit in JIT)?
  This means that, only stuff handled by JIT has to be in the lower 2GB, the upper 2GB is totally fine.
  One smart way to deal with this would be to allocate a few hundred MB in the lower 2GB.
  Obviously, in big chunks. Now, if memory is starting to run low, we can always free some of it.
  This doesn't affect the server that much (you really should be running x64 server)

[ArranTuna]

* Better performance (at least 2x)

Does this mean 2x faster pure lua code (as in code that doesn't call MTA funcs) execution or 2x better performance in general (like having twice the FPS even if most of the CPU usage isn't due to lua code)?

[MegadreamsBE]

Well, my tests on my GameBoy emulator which doesn't rely as much on MTA functions indicate that the performance gains can be 5 to 6 times the original performance. With MTA calls I can definitely imagine it being less but depending on how your code is written I can definitely see at least double the performance being possible.

[Einheit-101]

I use a lot of mta functions, so it probably wont do anything at all to server performance. Tests will reveal the truth

[Pirulax]

I did test it on a RP server. The performance was 20-30% better on average.
If you would like I can give you a client+server to test it privately. Let me know, if you are up to it.

[Einheit-101]

Yes, i want to try this :)

[ArranTuna]

Are any servers actually struggling CPU wise?
Eg: CIT, 181 players online, CPU usage on logic thread was reading as low as 0%, average 5%. It's the clients that need the optimizations.

[Pirulax]

Ideally it will be implemented client-side as well.
Sadly, there are a lot of blockers as of now.