Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build universal native libraries on macOS [2 XMR] #843

Open
woodser opened this issue Mar 26, 2024 · 22 comments
Open

Build universal native libraries on macOS [2 XMR] #843

woodser opened this issue Mar 26, 2024 · 22 comments
Labels
💰bounty There is a bounty on this issue

Comments

@woodser
Copy link
Contributor

woodser commented Mar 26, 2024 •
edited

Haveno can use native libraries instead of monero-wallet-rpc for wallet functionality using the monero-java project.

Currently, the native libraries built for macOS are platform-dependent, so they must be built for x86_64 and ARM64.

This issue requests making the native libraries universal on macOS, so multiple builds are not necessary.

This issue shares a bounty with the related issue: woodser/monero-java#81
@woodser woodser added the 💰bounty There is a bounty on this issue label Mar 26, 2024
@github-actions GitHub Actions
Copy link

There is a bounty on this issue, the amount is in the title. The reward will be awarded to the first person or group of people who resolves this issue.

If you are starting to work on this bounty, please write a comment, so that we can assign the issue to you. We expect contributors to provide a PR in a reasonable time frame or, in case of an extensive work, updates on their progresses. We will unassign the issue if we feel the assignee is not responsive or has abandoned the task.

Read the full conditions and details of our bounty system.

@woodser woodser changed the title Build universal native libraries on macOS [1.5 XMR] Build universal native libraries on macOS [2 XMR] Mar 27, 2024
@preland
Copy link
Contributor

preland commented Mar 28, 2024

I’ll look into this

@woodser
Copy link
Contributor Author

woodser commented Mar 28, 2024

Excellent, happy to have your help. Thanks.

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024

I'm curious; is the goal to avoid multiple builds or to have a universal native library on Mac?

@woodser
Copy link
Contributor Author

woodser commented Apr 2, 2024

Universal native library for macOS x86_64 and ARM64.

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024

Universal native library for macOS x86_64 and ARM64.

What if the universal native library is generated from already built x86_64 and ARM64 libraries? Then that involves multiple builds. That is why I ask.

@woodser
Copy link
Contributor Author

woodser commented Apr 2, 2024 •
edited

That could be one way to do it, but I was expecting to use cmake configuration and toolchains like monero-project, to avoid multiple builds and be able to do it from one machine: https://github.com/monero-project/monero/tree/master/cmake

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024 via email

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024 via email

@woodser
Copy link
Contributor Author

woodser commented Apr 2, 2024

I'm reading that multiple flags can be passed to the build command, e.g. CXXFLAGS="-arch x86_64 -arch arm64 -mmacosx-version-min=11.0 -std=c++11"

But I haven't looked into it deeply.

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024 via email

@woodser
Copy link
Contributor Author

woodser commented Apr 2, 2024

Nope, but sounds worth trying.

@niyid
Copy link
Contributor

niyid commented Apr 2, 2024 via email

@niyid
Copy link
Contributor

niyid commented Apr 5, 2024 via email

@woodser
Copy link
Contributor Author

woodser commented Apr 7, 2024

set(CMAKE_OSX_ARCHITECTURES "x86_64;arm64") did not work by adding it to CMakeLists.txt in monero-cpp and monero-java; the resulting libs work on x86_64 but not arm64.

@niyid
Copy link
Contributor

niyid commented Apr 7, 2024

...and you added the lipo command?

I have not tested this yet (no Mac):

cmake_minimum_required(VERSION 3.4.1)

project(monero-java-jni)

# Set architectures for Apple platforms
if (APPLE)
    set(ARCHITECTURES x86_64 arm64)
endif()

# For other platforms, use the original script
if (WIN32)
    add_definitions( "-D_GLIBCXX_USE_NANOSLEEP=1" )
    add_definitions( "-DWIN32_LEAN_AND_MEAN" )
    SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wa,-mbig-obj -O2 -fPIC -std=c++14 -F/Library/Frameworks -pthread -lcrypto -lcrypt32") 
else()
    SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC -std=c++14 -F/Library/Frameworks -pthread")
endif()

#SET(CMAKE_C_COMPILER /path/to/c/compiler)
#SET(CMAKE_CXX_COMPILER /path/to/cpp/compiler)

#############
# System
#############

set(MONERO_CPP "${CMAKE_SOURCE_DIR}/external/monero-cpp")
message(STATUS MONERO_CPP : ${MONERO_CPP} : ${MONERO_CPP})

set(MONERO_CPP_SRC "${MONERO_CPP}/src")
set(MONERO_PROJECT ${MONERO_CPP}/external/monero-project)
set(MONERO_PROJECT_SRC "${MONERO_PROJECT}/src")

# java header includes
if(NOT DEFINED ENV{JAVA_HOME} OR "$ENV{JAVA_HOME}" STREQUAL "")
  message(FATAL_ERROR "JAVA_HOME variable not set, for example: export JAVA_HOME=/path/to/jdk")
else()
  include_directories("$ENV{JAVA_HOME}")
  include_directories("$ENV{JAVA_HOME}/include")
  if (APPLE)
    include_directories("$ENV{JAVA_HOME}/include/darwin")
  elseif (WIN32)
    include_directories("$ENV{JAVA_HOME}/include/win32")
  else()
    include_directories("$ENV{JAVA_HOME}/include/linux")
  endif()
endif()

# monero-project header includes
include_directories("${MONERO_CPP}/external/libsodium/include/sodium")
include_directories("${MONERO_CPP}/external/openssl-sdk/include")
include_directories("${MONERO_CPP_SRC}/")
include_directories("${MONERO_PROJECT}/contrib/epee/include")
include_directories("${MONERO_PROJECT}/external/")
include_directories("${MONERO_PROJECT}/external/easylogging++")
include_directories("${MONERO_PROJECT}/external/rapidjson/include")
include_directories("${MONERO_PROJECT_SRC}/")
include_directories("${MONERO_PROJECT_SRC}/crypto")
include_directories("${MONERO_PROJECT_SRC}/crypto/crypto_ops_builder/include/")
include_directories("${MONERO_PROJECT_SRC}/wallet")
include_directories("${MONERO_PROJECT_SRC}/wallet/api")

# TODO: remove TRUEs, how are APPLE, DEPENDS, etc initialized?
if (TRUE OR HIDAPI_FOUND OR LibUSB_COMPILE_TEST_PASSED)
  if (APPLE)
    if(TRUE OR DEPENDS)
      list(APPEND EXTRA_LIBRARIES "-framework Foundation -framework IOKit -framework AppKit")
    else()
      find_library(COREFOUNDATION CoreFoundation)
      find_library(IOKIT IOKit)
      find_library(APPKIT AppKit)
      list(APPEND EXTRA_LIBRARIES ${IOKIT})
      list(APPEND EXTRA_LIBRARIES ${COREFOUNDATION})
      list(APPEND EXTRA_LIBRARIES ${APPKIT})
    endif()
  endif()
  if (WIN32)
    list(APPEND EXTRA_LIBRARIES setupapi)
  endif()
endif()

message(STATUS EXTRA_LIBRARIES: ${EXTRA_LIBRARIES})

############
# Boost
############

set(Boost_NO_BOOST_CMAKE 1)
set(Boost_USE_MULTITHREADED ON)
find_package(Boost 1.58 QUIET REQUIRED COMPONENTS chrono date_time filesystem program_options regex serialization wserialization system thread)
message(STATUS "Using Boost include dir at ${Boost_INCLUDE_DIR}")
include_directories(${Boost_INCLUDE_DIR})

############
# OpenSSL
############

if (APPLE AND NOT IOS)
  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fvisibility=default -std=c++14")
  if (NOT OPENSSL_ROOT_DIR)
      EXECUTE_PROCESS(COMMAND brew --prefix openssl
        OUTPUT_VARIABLE OPENSSL_ROOT_DIR
        OUTPUT_STRIP_TRAILING_WHITESPACE)
    message(STATUS "Using OpenSSL found at ${OPENSSL_ROOT_DIR}")
  endif()
endif()

find_package(OpenSSL REQUIRED)
message(STATUS "Using OpenSSL include dir at ${OPENSSL_INCLUDE_DIR}")
include_directories(${OPENSSL_INCLUDE_DIR})

if(STATIC AND NOT IOS)
  if(UNIX)
    set(OPENSSL_LIBRARIES "${OPENSSL_LIBRARIES};${CMAKE_DL_LIBS};${CMAKE_THREAD_LIBS_INIT}")
  endif()
endif()

if (WIN32)
  list(APPEND OPENSSL_LIBRARIES ws2_32 crypt32)
endif()

######################
# monero-cpp
######################

add_library(monero-cpp SHARED IMPORTED)

# import shared c++ library
if (APPLE)
  set_target_properties(monero-cpp PROPERTIES IMPORTED_LOCATION ./libmonero-cpp.dylib)
elseif (WIN32)
  set_target_properties(monero-cpp PROPERTIES IMPORTED_LOCATION ./libmonero-cpp.dll)
  set_target_properties(monero-cpp PROPERTIES IMPORTED_IMPLIB ./libmonero-cpp.dll.a)
else()
  set_target_properties(monero-cpp PROPERTIES IMPORTED_LOCATION ./libmonero-cpp.so)
endif()

# Check if building for macOS to combine builds into a universal library using lipo
if(APPLE)
    # Add flags and configurations for each architecture
    foreach(arch IN ITEMS ${ARCHITECTURES})
        set(CMAKE_C_FLAGS_${arch} "${CMAKE_C_FLAGS} -arch ${arch}")
        set(CMAKE_CXX_FLAGS_${arch} "${CMAKE_CXX_FLAGS} -arch ${arch}")
        set(CMAKE_LIBRARY_OUTPUT_DIRECTORY_${arch} ${CMAKE_BINARY_DIR}/${arch})
        add_library(monero-java-${arch} SHARED ${MONERO_JNI_SRC_FILES})
        target_link_libraries(monero-java-${arch}
            monero-cpp
            ${Boost_LIBRARIES}
            ${OPENSSL_LIBRARIES}
            ${EXTRA_LIBRARIES}
        )
    endforeach()

    add_custom_command(TARGET monero-java-x86_64 monero-java-arm64
        POST_BUILD COMMAND
        lipo -create
        $<TARGET_FILE:monero-java-x86_64>
        $<TARGET_FILE:monero-java-arm64>
        -output ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/libmonero-java.dylib
    )

    # Install the combined library
    INSTALL(TARGETS monero-java-x86_64
        RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT Runtime
        LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT Runtime
        ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT Development
    )
endif()

@NamesCode
Copy link

I can test this for you, I have an M2 mac and I can emulate an x86-64 one using UTM

@preland
Copy link
Contributor

preland commented Jun 2, 2024

I can test this for you, I have an M2 mac and I can emulate an x86-64 one using UTM

The main issue (sorry, I kinda dropped the ball on this issue, was working on other things) was/is getting monero to properly static compile for both x86_64 and arm64 on the same system. I was able to get that far in GitHub workflows, but I am still unable to properly test if lipo will create a proper universal binary (especially the cctools Linux port, which seems to hate GitHub workflows with a burning passion)

the most recent workflow run I did is here; if you could run lipo on the artifacts and see if it works properly, that would be highly appreciated. It should, but honestly it wouldn’t shock me at this point if it found a way to mess things up

@NamesCode
Copy link

NamesCode commented Jun 2, 2024 •
edited

I can test this for you, I have an M2 mac and I can emulate an x86-64 one using UTM

The main issue (sorry, I kinda dropped the ball on this issue, was working on other things) was/is getting monero to properly static compile for both x86_64 and arm64 on the same system. I was able to get that far in GitHub workflows, but I am still unable to properly test if lipo will create a proper universal binary (especially the cctools Linux port, which seems to hate GitHub workflows with a burning passion)

the most recent workflow run I did is here; if you could run lipo on the artifacts and see if it works properly, that would be highly appreciated. It should, but honestly it wouldn’t shock me at this point if it found a way to mess things up

Heyo! Good news!! after running lipo create ~/Downloads/Cross-Mac-x86_64/{Binary name here} ~/Downloads/Cross-Mac-aarch64/{Binary name here} -output ./{Binary name here} and chmod +x'ing the resulting binaries both worked perfectly on aarch64. No warnings or issues. Had a bit of a surprise when monerod started syncing too

The binaries themselves work in practice so you may be able to just run the above command in the final linker step?
I havent taken a look at the build.yml myself to see where it is erroring.

Proof it works too ^^:

image image

@niyid
Copy link
Contributor

niyid commented Jun 2, 2024

@ all, thank you all for your contribution and proving the idea works.

@NamesCode
Copy link

If there's anyway I can help further I'm more than happy too ^^

I'll see if I can take a look at the GitHub workflows tomorrow if you'd like @preland unless it's not going to be used in the final build?

@preland
Copy link
Contributor

preland commented Jun 2, 2024

If there's anyway I can help further I'm more than happy too ^^

I'll see if I can take a look at the GitHub workflows tomorrow if you'd like @preland unless it's not going to be used in the final build?

That would be appreciated; once that is working properly, then the path to getting this issue resolved will be as “simple” as monero->monero-cpp->monero-java->haveno

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💰bounty There is a bounty on this issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@woodser @niyid @NamesCode @preland