Android Backend Support #717
Conversation
|
Looks good, thanks for putting this together. Could you please manually trigger a run of the "Update Binaries" action on your repo as a test run. To do that:
|
There was a problem hiding this comment.
Thank you for this contribution! The overall looks good except one concern. I prefer to putting the android native libraries in a new package LLamaSharp.Backend.Android because those who want to use android will never use libraries of win/linux/osx. For those who want to publish a cross-platform app, they could install both LLamaSharp.Backend.Android and LLamaSharp.Backend.Cpu. Please refer to #489 to see how to add a new backend package.
The following items will further improve it but are not required in this PR.
- Add a workflow to run test in android. I know little about running C# apps on android but it seems that github action has already provided such support (reference).
- Add logics to support dynamic native library selection on android.
I'd like to work with item 2 after having this PR merged so please don't worry about it. :) Would you like to work with item 1 (in this PR or after this PR are both okay).
|
Totally aligned. I was pondering that it deserved its own backend as well but wasn't sure. Will work on these over the next few days. Should share an update soon. PS: Prefer to finish it up in this PR |
|
Could you upload the MAUI app's source with the android runtime libraries to test it? |
|
Hi @cmpktheo Attaching a binary compliled for |
|
Thanks @AmSmart ! I followed the android build instructions on llama.cpp page, but when I try to use the .so file, I get a null pointer calling the llama_model_default_params method (which basically returns a dummy structure) . It makes me think that there's either a flaw in my build process or some issue related to MAUI. This is my first attempt to build anything in MAUI so I might miss something. I would appreciate if you could share the build process so I can apply it to x86_64 architecture or a ready build x86_64 runtime and ideally a MAUI-Android sample app. |
|
@martindevans the issue seems to be with my build process since the native method returns null pointer. |
AmSmart
force-pushed
the
feature/add-android-support
branch
from
82e5020
to
cb08e1b
Compare
AmSmart
force-pushed
the
feature/add-android-support
branch
from
cb08e1b
to
c7b6980
Compare
|
@cmpktheo You can check the |
AmSmart
force-pushed
the
feature/add-android-support
branch
from
7541e8b
to
0511b96
Compare
|
@AsakusaRinne I've been a bit busy since our last conversation but I've finally gotten some time to address some of the concerns you listed earlier and made some other changes as well. Things to note
Let me know if there are anymore requested changes for this PR to be good to go. |
There was a problem hiding this comment.
Thank you for your effort! It looks good to me except a few minor issues.
With regards to providing tests, I'm thinking of a world where we have a Demo/Example .NET Android Project instead. The UnitTests already provide coverage for the underlying code and a demo app should suffice. That said, we could still proceed to write tests against the demo app and have them run in the workflow. Let me know what you think, I'm happy to contribute this in another PR.
Sure, the workflow is not required in this PR. It's a good idea to add an example/demo first. The main motivation for the android test workflow is that our core team members do not have android environment to test it. Thus it will be hard for us to know whether a PR breaks this feature. But no hurry, adding an example is a good first step. :)
Since Android doesn't support additional enhancements (CUDA, CUBLAS, AVX etc) at the moment, there is really no need to manually resolve the native library. I have added an if statement to exclude Android from the Manual Resolution in NativeApi.Load.cs
The native library loading system is also responsible for deciding the file path to load it. However, please don't worry, I'll do it after this PR.
|
|
||
| <None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama.dll"> |
There was a problem hiding this comment.
These targets are used in the unittest and example projects, for example:
https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Examples/LLama.Examples.csproj#L2
https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Unittest/LLama.Unittest.csproj#L2
Therefore, though I'm not sure whether this targets file should also be used in other backend packages, I don't think we should remove it. May I ask which behavior will be different if a targets file is used in the backend package?
| @@ -57,6 +57,11 @@ private static void SetDllImportResolver() | |||
| // NativeLibrary is not available on older runtimes. We'll have to depend on | |||
| // the normal runtime dll resolution there. | |||
| #if NET5_0_OR_GREATER | |||
|
|
|||
| // We don't need special dll resolution on Android | |||
| if (OperatingSystem.IsAndroid()) | |||
There was a problem hiding this comment.
The master branch has refactored the native library loading system so I'm afraid this code would not work. Don't worry, I can do it after having this PR merged.
| @@ -51,12 +51,12 @@ jobs: | |||
| - uses: actions/upload-artifact@v4 | |||
| with: | |||
| path: ./build/libllama.so | |||
| name: llama-bin-linux-${{ matrix.build }}-x64.so | |||
| name: llama-bin-linux-${{ matrix.build }}-x64 | |||
There was a problem hiding this comment.
@martindevans Could you please take a look of this? I'm not sure what it's expected to do with the original code.
As discussed in #695, Here's a PR that generates the shared library for Android. I have also added the new binaries to be copied in the CPU Backend
.nuspecfile. Please let me know if I am missing anything else.I have tested the generated
.soby loading manually into a .NET MAUI app which ran smoothly on my Android device. See video belowScreen_Recording_20240427_141010.mp4