HoloKit is an optical see-through mixed reality headset that transforms your iPhone into a powerful stereoscopic AR device. With the HoloKit Unity SDK, developers can create immersive AR experiences in Unity, which can be viewed with HoloKit headset.
The SDK currently provides three core features:
- Stereoscopic Rendering
- Hand Tracking
- Hand Gesture Recognition
Stereoscopic rendering is the central feature of the SDK, allowing for the display of stereo images on the iPhone screen. By attaching your iPhone to the HoloKit, you can enjoy a captivating AR experience. Utilizing Apple Vision framework, the SDK detects user hand poses. Together with LiDAR depth sensor, it allows your iPhone to track the 3D positions of the user's hands. Furthermore, the SDK can recognize hand gestures such as pinching, serving as a trigger for specific operations in your project.
The SDK offers a built-in video recording feature which allows the user to record videos under both screen AR mode and stereoscopic mode.
HoloKit Unity SDK, built on the foundation of ARFoundation, is compatible with most ARFoundation features such as image tracking and plane detection. Upgrading your ARFoundation project to a stereoscopic AR project is straightforward with the HoloKit Unity SDK.
You can install HoloKit Unity SDK from the following git URL in Package Manager:
https://github.com/holokit/holokit-unity-sdk.git
or by directly adding the following line to the Packages/manifest.json file.
"io.holokit.unity-sdk": "https://github.com/holokit/holokit-unity-sdk.git"
HoloKit Unity SDK has been tested and found compatible with the following software versions:
- Unity 2022.3 LTS
- ARFoundation 5.1
We aim to continually test and verify compatibility with newer versions of these softwares.
HoloKit was primarily built for iOS devices. Please see the Supported iOS Devices to check if your iPhone is supported.
HoloKit currently only supports a limited number of Android phones. Please see the Supported Android Devices for more details.
If your Android phone is not supported, we also provide a calibration method to let you figure out the specs of your Android phone yourself. You can then enter your Android phone specs to the SDK to have your device rendering stereo images. Please see Phone Calibration Guide, which is a step-by-step guide on how to figure out the specs of your phone model.
Before using the HoloKit Unity SDK, ensure to adjust the project settings to meet ARFoundation's requirements. If you are already comfortable with ARFoundation, feel free to skip this section.
Upon successfully installation of the SDK package, the ARFoundation package will be installed automatically. Please note that the HoloKit Unity SDK is only compatible with ARFoundation 5.+. If you are using ARFoundation 4.+, please update to a more recent version.
For iOS-targeted projects, install the Apple ARKit XR Plugin package. For Android-targeted projects, the Google ARCore XR Plugin is required.
- Set Camera and Microphone Usage Descriptions: Navigate to
Project Settings > Player > Other Settings > Configurationand provide a user-friendly text forCamera Usage DescriptionandMicrophone usage description. TheCamera Usage Descriptionwill pop up when the iPhone requests access to the camera. TheMicrophone usage descriptionis required because the video recording feature of the SDK will access the microphone to record audio.
- Enable Apple ARKit: Under
Project Settings > XR Plug-in Management > Plug-in Providers, enable theApple ARKitoption.
- Disable Auto Graphics API: Navigate to
Project Settings > Player > Other Settings > Renderingand disableAuto Graphics API. In theGraphics APIslist, removeVulkanand ensure onlyOpenGLES3is listed.
- Set Minimum API Level: Proceed to
Project Settings > Player > Other Settings > Identificationand setMinimum API LeveltoAndroid 7.0 'Nougat' (API level 24)or higher.
- Configure Scripting Backend: In
Project Settings > Player > Other Settings > Configuration, setScripting BackendtoIL2CPP. UnderTarget Architectures, uncheckARMv7and checkARM64.
- Enable Google ARCore: Under
Project Settings > XR Plug-in Management > Plug-in Providers, enable theGoogle ARCoreoption.
The SDK package offers a samples for each of the three key features: stereoscopic rendering, hand tracking, and hand gesture recognition. Import these samples into your project and deploy them to mobile devices for a quick overview of each feature. These samples can also be used as templates for your projects.
Stereoscopic rendering is the core feature of the SDK, your app needs to display stereo images on the phone screen so that it can be viewed with the HoloKit headset.
The SDK provides two rendering modes: mono and stereo. Mono mode renders the background camera image with virtual content layered on top, akin to a standard ARFoundation project. In constrast, stereo mode renders two stereo images on a black background one the phone's screen, so that you can insert your phone onto the HoloKit headset to have stereoscopic AR experience.
 |
 |
|---|---|
| Mono rendering mode | Stereo rendering mode |
To integrate stereoscopic rendering into your project, the HoloKitCameraManager component is required. In fact, HoloKitCameraManager is the only mandatory component in the SDK that you must add into the scene. Instead of adding a default ARFoundation XR Origin object, we add a HoloKit XR Origin object into the scene by right-clicking in Unity Editor's hierarchy window and selecting XR/HoloKit/HoloKit XR Origin. The HoloKit XR Origin contains a HoloKit Camera object with the HoloKitCameraManager script, which enables both mono and stereo rendering modes.
When you start the game, HoloKitCameraManager initially enters mono mode. A UI button is required to transition between rendering modes. The current rendering mode can be retrieved and set with HoloKitCameraManager. The following code snippet illustrates how to switch the rendering mode.
// A reference to the HoloKitCameraManager component is required
[SerializedField] HoloKitCameraManager m_HoloKitCameraManager;
public void SwitchRenderMode()
{
// Get the current ScreenRenderMode value and assign a new value
holokitCamera.ScreenRenderMode = holokitCamera.ScreenRenderMode == ScreenRenderMode.Mono ? ScreenRenderMode.Stereo : ScreenRenderMode.Mono;
}
When the stereo mode is activated, the SDK spawns an alignment marker UI on the screen's upper right corner, enabling you to align your phone accurately once attached to the HoloKit headset.
The SDK tracks the user's hand, providing the 3D positions of the 21 hand joints of each hand.
To incorporate the hand tracking into your project, create an empty GameObject and add HandTrackingManager component to it. The script will automatically setup all necessary objects as child GameObjects.
HandTrackingManager requires depth information to calculate 3D hand positions, so add AROcclusionManager component to the HoloKit Camera GameObject and adjust its settings accordingly.
You can now build the project onto an iPhone to view the results.
HandTrackingManager can be configured to track either one hand or both hands. As hand tracking is energy-intensive, if dual hand tracking isn't necessary, we advise tracking only one hand to conserve energy. Set HandTrackingManager.MaxHandCount in its inspector within the Unity editor.
To obtain specific hand joint positions in code, use HandTrackingManager.GetHandJointPosition(int handIndex, JointName jointName). As the hand tracking algorithm can't distinguish handedness, the parameter handIndex simply represents the hand detection sequence. When only one hand is detected, handIndex should always be 0. If two hands are detected, handIndex can be either 0 or 1. The number of detected hands can be accessed via HandTrackingManager.HandCount.
Please note, the hand tracking feature is only compatible with iOS 14.0 or higher devices equipped with a LiDAR depth sensor.
The SDK can also recognizes user's hand gesture, which can serve as input triggers in your project. Currently there are only two available hand gestures: HandGesture.None and HandGesture.Pinched.
To utilize hand gesture recognition, add HandGestureRecognitionManager to the scene. The callback function HandGestureRecognitionManager.OnHandGestureChanged is invoked when a hand gesture transition is detected. Register this callback to receive notifications. You can also access the user's current hand gesture via HandGestureRecognitionManager.HandGesture. The following code snippet illustrates how to capture user hand gesture transitions.
[SerializeField] HandGestureRecognitionManager m_HandGestureRecognitionManager;
private void Start()
{
// Register the callback
m_HandGestureRecognitionManager.OnHandGestureChanged += OnHandGestureChanged;
}
private void OnHandGestureChanged(HandGesture handGesture)
{
// Execute desired action
}
Please note, the hand gesture recognition feature is only available on iOS 14.0 or higher devices.
You can use hand tracking and hand gesture recognition at the same time, simply add both HandTrackingManager and HandGestureRecognitionManager components to your scene.
HoloKit Unity SDK is based on ARFoundation. For a minimal ARFoundation scene requires two primary objects: an AR Session object and an XR Origin object with the Main Camera Object as its child.
To enable stereoscopic rendering, we need to upgrade the default Main Camera to a HoloKit Camera. To create an XR Origin with a HoloKit Camera as its child, right-click in the Unity Editor and select XR/HoloKit/HoloKit XR Origin to create a HoloKit XR Origin.
The HoloKit XR Orign object includes a HoloKit Camera capable of both screen AR and stereoscopic rendering.
To switch between the rendering modes, a UI button is necessary. To create a default UI panel which includes this mode-switching button, right-click and navigate to UI/HoloKit/HoloKit Default UI Canvas.
The default canvas also has a button for controlling video recording.
After setting up the AR Session, HoloKit XROrigin, and UI components, you can add any 3D object to the scene, such as a cube. When you build your project onto an iPhone, you'll be able to view your object in both screen AR mode and stereoscopic rendering mode with a HoloKit headset.
The SDK enables resetting the ARKit ARSession's world origin to a specified position and rotation by using the WorldOriginResetter script. Please notice that there must be a HoloKitARKitManager component in the scene, which is used by the WorldOriginResetter script.
Import the Reset World Origin package sample for how to use this feature.
The 4K HDR (High Dynamic Range) AR background video is a feature of ARKit 6, available on iOS 16 and later. This enhancement improves the quality of ARBackground video, leading to better video recording quality. To enable the 4K HDR option, use the ARBackgroundVideoEnhancementManager component. Please notice that there must be a HoloKitARKitManager component in the scene, which is used by the ARBackgroundVideoEnhancementManager script.
Import the AR Background Video Enhancement package sample for how to use this feature. Please note that not all iOS devices support 4K or HDR. For more information, please visit the ARKit's website and check out this WWDC session on ARKit 6.
If you encounter any issues, have queries, suggestions, or discover any bugs, we welcome you to join our Discord community or submit an issue. Your feedback is invaluable as we continually strive to update and improve the HoloKit Unity SDK.