Liberty-Way AMLS Landing Controller

This project is a part of project activity in Moscow Polytech University by students of group 181-311

All the code excluding the dependencies block, was written by Fern Lane, Vladislav Yasnetsky and Andrey Kabalin

More projects

Liberty-X

Drone flight controller

https://github.com/XxOinvizioNxX/Liberty-X

Eitude

AMLS Platform controller

https://github.com/XxOinvizioNxX/Liberty-Way/tree/main/Eitude

Sonarus

I2C ultrasonic rangefinder

https://github.com/XxOinvizioNxX/Liberty-Way/tree/main/Sonarus

GPS Mixer

Way to improve GPS navigation using multiple receivers

https://github.com/XxOinvizioNxX/Liberty-Way/tree/main/GPS-mixer

GPS to Serial

Android app to send phone GPS coordinates via USB serial port

https://github.com/XxOinvizioNxX/GPS-to-Serial

Feedback

Fern Lane (Author and CEO of the project) E-Mail: xxoinvizionxx@gmail.com, Twitter: @fern_hertz
Vladislav Yasnetsky (Project developer. Video editing, PR, hardware development). E-Mail: vlad.yasn@gmail.com
Andrey Kabalin (Project developer. GPS stabilization, hardware development) E-Mail: astik452@gmail.com

For project development

BTC: bc1qd2j53p9nplxcx4uyrv322t3mg0t93pz6m5lnft
ZEC: t1Jb5tH61zcSTy2QyfsxftUEWHikdSYpPoz
ETH: 0x284E6121362ea1C69528eDEdc309fC8b90fA5578

Dependencies

Flak (Web framework): https://github.com/pcdv/flak
MiniPID (PID Controller): https://github.com/tekdemo/MiniPID-Java
Log4j (Logback): https://logging.apache.org/log4j/1.2/
OpenCV (Computer vision): https://github.com/opencv/opencv
OpenCV-contib (Extra OpenCV modules): https://github.com/opencv/opencv_contrib
Commons-CLI (Arguments parser): https://commons.apache.org/proper/commons-cli/
jSerialComm (Serial communication): https://github.com/Fazecast/jSerialComm
GSon (JSON implementation): https://github.com/google/gson
Augmented-UI (WEB framework for Cyberpunk-style UI): https://github.com/propjockey/augmented-ui
Mapbox (Maps from mapbox): https://www.mapbox.com/
Bing Maps (Maps from Microsoft): https://www.bing.com/maps/

Description

This application is a part of the AMLS project (Autonomous Multi-rotor Landing System). The task of the project is to automatically land a drone on a platform in motion.

This application processes the frame received from the camera located on the platform which is looking up. On the bottom of the drone, there is an ARUco tag. The application detects the marker, estimates its position, then passes through the PID controller and sends the correction values to the drone.

Licenses

Liberty-Way is licensed under AGPLv3, you can find its contents in the main branch.

The dependencies are certificated by following licenses:

Apache-2.0 for Flak, Log4j, Opencv, OpenCV-contib, Commons-CLI and GSon
BSD-2-Clause License for Augmented-UI
GPL-3.0 for MiniPID
GNU-3.0 for jSerialComm
Mapbox: https://www.mapbox.com/pricing/

Logotype

AMLS and Liberty-X (Liberty-Drones) logo was designed by Fern Lane

Building and running

Liberty-Way is a cross-platform application and has been tested on:

Microsoft Windows 10 Pro 10.0.19041 N/A Build 19041, Intel(R) Core(TM) i7-9750H (x64), java version "1.8.0_281"
Microsoft Windows 11 Pro 10.0.22000 N/A Build 22000, Intel(R) Core(TM) i5-9300H (x64)
Microsoft Windows 10 Pro 10.0.19042 N/A Build 19042, Intel(R) Core(TM) i5-8300H (x64)
5.6.0-kali2-amd64 SMP Debian 5.6.14-1kali1 (2020-05-25) x86_64 GNU/Linux, Intel(R) Core(TM) i7-9750H (x64), openjdk version "11.0.7-ea"
Ubuntu 20.04.1 LTS, Intel(R) Core(TM) i7-9750H (x64), openjdk version "1.8.0_292"
macOS 11.6 20G165, Intel(R) Core(TM) i7-9750H (x64), openjdk version "11.0.12"
macOS 11.0.1 20B50, Intel(R) Core(TM) i5-7360U (x64), openjdk version "11.0.12"

You can run Liberty-Way on other operating systems. Recommended operating system - Windows.

It is strongly recommended building the native OpenCV library from a source for your operating system. Native libraries compiled on the listed operating systems are contained in the release archives, as well as in the main branch. If you build your native library, put the library file next to the Liberty-Way.jar file. Don't rename the native library file.

You can build the Liberty-Way application yourself or run a ready-made .jar from releases (recommended). Also, nightly builds (.jar) can be downloaded from main branch.

All the controls is performed only from the browser. When starting the application, you can specify the arguments:

 -c,--color                write colored logs.
 -i,--ip <arg>             server ip
 -sp,--server_port <arg>   web server port (0 - 65535)
 -vp,--video_port <arg>    video stream port (0 - 65535)

Also, the server address and ports can be specified in the configuration (settings.json)

When launched, the console displays the IP address of the user interface. You need to go to it in your browser

Run on Windows

Download and install Java x64
Download the latest Liberty-Way release from Releases
Unpack Liberty-Way archive
Run START_WINDOWS.bat

Run on macOS

Download and install Homebrew (/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)")
Install java11 or later with brew (brew install java11)
Check java version with java -version
If java version is not the same as installed, set JAVA_HOME environment variable (for example export JAVA_HOME=/usr/local/Cellar/openjdk@11/11.0.12/)
Install opencv dependencies (brew install opencv) (OpenCV is built into Liberty-Way, but other system dependencies are also required, so the easiest way is to let homebrew install them automatically)
Download the latest Liberty-Way release from Releases
Unpack Liberty-Way archive
Run START_MACOS.sh or navigate into Liberty-Way folder (using cd) and type java -Djava.library.path=. -jar Liberty-Way.jar -c

Run on Linux

Install Java (for example sudo apt-get install openjdk-11-jdk)
Check java version with java -version
If java version is not the same as installed, set JAVA_HOME environment variable (for example export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/)
Install opencv dependencies
- sudo apt install libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
- sudo apt install libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev
- sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev**
Download the latest Liberty-Way release from Releases
Unpack Liberty-Way archive
Run START_LINUX.sh or navigate into Liberty-Way folder (using cd) and type java -Djava.library.path=. -jar Liberty-Way.jar -c

Build Liberty-Way

Building Liberty-Way is done simply by typing ant main

Install JDK (1.8+)
Install ant
Install git
Set JAVA_HOME and ANT_HOME environment variables
Check java version with java -version
Check ant version with ant -version
Type git clone https://github.com/XxOinvizioNxX/Liberty-Way
Navigate to Liberty-Way folder cd Liberty-Way
Run ant script ant main

Build OpenCV native library on Linux

Install dependencies:
- sudo apt install build-essential git cmake
- sudo apt install libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
- sudo apt install libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev
- sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev**
Install Ant:
- sudo apt-get install ant
Install JDK. JDK 17 Installation Example:
- sudo mkdir /usr/lib/jvm
- sudo cd /usr/lib/jvm
- sudo wget https://download.oracle.com/java/17/latest/jdk-17_linux-x64_bin.tar.gz
- sudo tar -xvzf jdk-17_linux-x64_bin.tar.gz
- export J2SDKDIR="/usr/lib/jvm/jdk-17"
- export J2REDIR="/usr/lib/jvm/jdk-17/jre"
- export JAVA_HOME="/usr/lib/jvm/jdk-17"
- export DERBY_HOME="/usr/lib/jvm/jdk-17/db"
- export PATH=$JAVA_HOME/bin:$PATH
Check JDK and ant:
- java -version
- ant -version
- Both of these commands should show the JDK and ant versions without error. If not, check if you have installed JDK and ant and set system variables.
Go to the folder where you want to build opencv. Example:
- cd ~
- mkdir opencv
- cd opencv
Download OpenCV:
- wget https://github.com/opencv/opencv/archive/4.5.3.tar.gz
- mv 4.5.3.tar.gz opencv-4.5.3.tar.gz
- wget https://github.com/opencv/opencv_contrib/archive/4.5.3.tar.gz
- mv 4.5.3.tar.gz opencv_contrib-4.5.3.tar.gz
- tar -xf opencv-4.5.3.tar.gz
- tar -xf opencv_contrib-4.5.3.tar.gz
- cd opencv-4.5.3
- mkdir build
- cd build
- mkdir install
Run CMAKE:

cmake .. \
  -D WITH_IPP=OFF \
  -D BUILD_opencv_java=ON \
  -D CMAKE_BUILD_TYPE=RELEASE \
  -D CMAKE_INSTALL_PREFIX=../../opencv-4.5.3/build/install \
  -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.5.3/modules \
  -D BUILD_opencv_python2=OFF \
  -D BUILD_opencv_python3=OFF \
  -D INSTALL_PYTHON_EXAMPLES=OFF \
  -D INSTALL_C_EXAMPLES=OFF \
  -D OPENCV_ENABLE_NONFREE=ON \
  -D BUILD_EXAMPLES=OFF \
  -D OPENCV_JAVA_SOURCE_VERSION=1.8 \
  -D OPENCV_JAVA_TARGET_VERSION=1.8 \
  -D BUILD_SHARED_LIBS=OFF \
  -D BUILD_FAT_JAVA_LIB=ON

Make sure cmake ran without errors and Java: ant: and Java: JNI: are found in the log and Java wrappers: is YES. If not, check your JDK and ant installation again
Finally, run make (To speed up, you can use make -j NUMBER OF CPU CORES)

Configuration

Settings

These are the parameters presented in settings.json file that are used by the program and can be changed depending on of its application:

"marker_size": 6.0 - length of one side of the tracking marker (cm)

"max_exposure": -8 - default and maximum exposure value of the camera

"motors_turn_off_height": 32.0 - altitude of sending a packet to turn off the motors

"landing_allowed": true - whether to lower the drone during optical stabilization. If set to false, the drone will never land

"only_optical_stabilization": false - if true, Liberty-Way will not send start commands and waypoints

"max_marker_height": 200 - marker will not be calculated if found above this height

"pid_file": "pid.json" - file of PID configuration

"camera_matrix_file": "camera_matrix.json" - camera calibration (matrix)

"camera_distortions_file": "camera_distortions.json" - camera calibration (distortions)

"watermark_file": "watermark.png" - image of a watermark (top-right corner)

"web_resources_folder": "web/static" - folder of web resources (static folder)

"web_templates_folder": "web/templates" - folder of templates (html) for a web-page

"blackbox_folder": "blackbox/" - folder which stores blackbox .csv files

"frame_width": 1280 - resolution that is set to web-camera (width)

"frame_height": 720 - resolution that is set to web-camera (height)

"disable_auto_exposure": true - disabling/enabling auto exposure feature

"disable_auto_wb": true - disabling/enabling auto white balance feature

"disable_auto_focus": true - disabling/enabling autofocus feature

"default_server_host": "localhost" - server host which can be overridden by cmd argument

"default_server_port": 80 - server port which can be overridden by cmd argument

"default_video_port": 8080 - video port which can be overridden by cmd argument

"video_stream_enabled_by_default": true - should the video be enabled on the page from the start

"blackbox_enabled": true - should the blackbox feature be enabled by default

"serial_reconnect_time": 500 - how many milliseconds to try to open the serial port if it is lost

"udp_timeout": 2000 - UDP response timeout (milliseconds)

"telemetry_lost_time": 3000 - how many milliseconds it takes to consider telemetry (drone) lost

"platform_lost_time": 1000 - how many milliseconds it takes to consider platform lost

"platform_light_enable_threshold": 1000 - lower threshold of the surrounding light to enable additional lighting

"platform_light_disable_threshold": 3000 - upper threshold of the surrounding light to disable additional lighting

"platform_loop_timer": 100 - time of one cycle of receiving data from the platform

"fps_measure_period": 2000 - how many milliseconds to measure FPS

"adaptive_thresh_constant": 15 - detector of parameters (ARUCO)

"aruco_dictionary": 0 - index of used ARUco dictionary (default = 0 which is 50 4x4 marks)

"allowed_ids": [9] - array of allowed tracking markers ARUCO ids

"input_filter": 0.30 - Kalman filter coefficient

"setpoint_alignment_factor": 0.75 - floating setpoint to the desired position coefficient

"allowed_lost_frames": 10 - frames where the marker is not in the input frame

"landing_decrement": 0.20 - constant latitude decrement (cm)

"allowed_landing_range_xy": 5.0 - range of the landing allowance (cm)

"allowed_landing_range_yaw": 5.0 - range of the landing allowance (degrees)

"min_satellites_num_start": 0 - the minimum number of satellites is set for the start

"min_satellites_num": 0 - the minimum number of satellites allowed

"setpoint_x": 0.0 - setpoint of PID controller (absolute estimated x)

"setpoint_y": 0.0 - setpoint of PID controller (absolute estimated y)

"setpoint_yaw": 0.0 - setpoint of PID controller (absolute estimated yaw)

"drone_data_suffix_1": 238 - first of the unique pair symbols that show the end of the packet

"drone_data_suffix_2": 239 - second of the unique pair symbols that show the end of the packet

"platform_data_suffix_1": 238 - first of the unique pair symbols that show the end of the packet

"platform_data_suffix_2": 239 - second of the unique pair symbols that show the end of the packet

"push_osd_after_frames": 2 - after how many frames the image is being sent to the web-page

"planet_radius": 6378.137 - the radius of the planet the project is running on

"pressure_term_above_platform": 30 - amount of pressure (Pa) above the platform which is equivalent to 3 meters above the ground

"send_idle_cycles_num": 0 - amount of cycles in which telemetry data is sent between GPS data (0 - only GPS data)

"is_telemetry_necessary": true - is it necessary to use telemetry

"max_platform_speed": 10 - the allowed maximum speed for the platform

"send_idle_in_wait_mode": true - is it necessary to send telemetry data while the drone isn't active

"is_gps_prediction_allowed": false - is it necessary to predict future positions

"stop_prediction_on_distance": 10 - amount of meters on which the prediction of future positions stops operating

"log_fps": false - is it necessary to log fps while operating

"log_api_requests": false - is it necessary to log processed POST requests to API

PID

These are the PID regulation parameters for each processed axle (x, y, z and yaw) which can be found in the pid.json file:

"P",              proportional term coefficient 
"I",              integral term coefficient 
"D",              derivative term coefficient
"F",              feed-forward term (which is a rough prediction of the output value) coefficient
"ramp",           maximum rate that the output can increase per cycle
"limit",          acceptable maximum of the output value
"reversed",       should the output be in a reversed state with opposite value

Data packet structure

Structure of Liberty-Link packets that Liberty-Way sends to the drone

Each packet consists of 12 bytes. The first 8 bytes are the payload. Next comes 1 byte Link Command, which tells the drone what to do with this packet. Then 1 byte XOR of the check-sum of the previous 9 bytes. At the end there are 2 bytes indicating the end of the packet (indicated in three. By default, it is 0xEE 0xEF).

For each packet sent (even IDLE), the drone returns 1 or several bytes (the number is specified in the Liberty-X settings) of telemetry.

All data that is sent to the drone or comes from the drone (telemetry) is arranged in big-endian order.

IDLE (Link command 0)

In IDLE mode, the drone does not perform any new calculations and simply continues its flight in the previous mode.

The payload bytes must be equal to 0, the Link Command byte is equal to 0, then the check-sum (0) and 2 bytes of the end of the packet.