Discussion on on-chip calibration

[mksbabu]

Hi Team,

I want to add on-chip calibration feature to my application by using intel realsense SDK, done few changes as per white paper ( https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-d-on-chip-calibration-c-api ) of self calibration but getting below error when executing below code. could you please help me on how to mention "const float progress" value and what went wrong in below code since this is my first project on intelrealsense SDK with c++.

code snippet:
rs2::pipeline pipe;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 256, 144, RS2_FORMAT_Z16, 90);
rs2::device dev = pipe.start(cfg).get_device();
rs2::auto_calibrated_device cal = dev.asrs2::auto_calibrated_device();
float health;
rs2::calibration_table res = cal.run_on_chip_calibration("", &health, [&](const float progress) { /* On Progress */ },100);

Error:

Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C (KernelBase.dll) in learning.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Exception thrown at 0x00007FF9ACCD4B2C in learning.exe: Microsoft C++ exception: librealsense::invalid_value_exception at memory location 0x000000C6A3FAE6F0.
Exception thrown at 0x00007FF9ACCD4B2C in learning.exe: Microsoft C++ exception: librealsense::invalid_value_exception at memory location 0x000000C6A3FADD60.
Exception thrown at 0x00007FF9ACCD4B2C in learning.exe: Microsoft C++ exception: [rethrow] at memory location 0x0000000000000000.
Exception thrown at 0x00007FF9ACCD4B2C in learning.exe: Microsoft C++ exception: rs2::invalid_value_error at memory location 0x000000C6A3FAF260.
Unhandled exception at 0x00007FF9ACCD4B2C in learning.exe: Microsoft C++ exception: rs2::invalid_value_error at memory location 0x000000C6A3FAF260.

[MartyG-RealSense]

Hi @mksbabu Can you provide information about the questions below, please?

1. Which RealSense camera model are you using?

2. Are your RealSense SDK and firmware driver versions newer than SDK 2.33.1 and firmware 5.12.2.100 please? These are the minimum versions required for using self-calibration.

3. Are you developing your C++ program in Visual Studio on Windows? If you are using VS on Windows then I would recommend using the tutorial at the link below to set up the project's linkages to librealsense using three .props property sheet files provided by the RealSense SDK if you have not done so already.

https://github.com/EduardoWang/visual-studio-real-sense-record-and-playback-project

[mksbabu]

Hi @MartyG-RealSense

I am using intel real sense D435 camera, SDK version: 2.57.1 and firmware version:05.15.00.02.

yes I followed example what you mentioned above but getting same error. could you please help me how to pass parameters with values from my side for method cal.run_on_chip_calibration("", &health, [&](const float progress) { /* On Progress */ },100); (https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-d-on-chip-calibration-c-api)

Thanks in advance.

[MartyG-RealSense]

In the rs2::calibration_table res line, this instruction typically does not have a number such as '100' at the end of it. Have you tried the line without this number to confirm whether this is the 'stream number' that the error is referring to, please?

rs2::calibration_table res = cal.run_on_chip_calibration(json, &health, [&](const float progress) { /* On Progress */ });

[mksbabu]

Hi @MartyG-RealSense

I tried it without 100 but the issue is same and also found that one of the discussion issue related to on-chip calibration(#8337), tried with successful code(#8337 (comment)) which is mentioned in discussion but getting below error could you please help us on it
Error in realsense.exe:
Counter: 150
Error in Main: hwmon command 0x80( 8 3 0 0 ) failed (response -7= HW not ready)

console log of visual studio:
calibration log.txt

Thanks

[MartyG-RealSense]

If your code is okay then there are some computers that on-chip calibration will only work with up until librealsense SDK version 2.49.0, but will not work with versions newer than 2.49.0. If your computer is one of those that experiences this problem then there is no solution other than to use 2.49.0 to perform on-chip calibration, unfortunately.

[mksbabu]

Hi @MartyG-RealSense

I tried to run my code with 2.49.0 version but there is no luck, still same error:

hwmon command 0x80( 8 0 0 8 ) failed (response -7= HW not ready)

could you please help us on why this error is coming?

[MartyG-RealSense]

If you are using Windows, please check in the RealSense Viewer if the Infrared and Infrared 2 options are listed under the Stereo Module section of the Viewer's options side-panel. If they are missing and only Depth is listed then this could indicate that the camera drivers are damaged and need to be uninstalled and reinstalled as their absence can make activationof on-chip calibration fail.

If the Infrared and Infrared 2 options are listed then it is likely that your particular computer does not work with on-chip calibration. It will work fine on some computers but not others, and if your computer is one that has a problem with on-chip calibration activation then there is no way to fix it, unfortunately.

Cameras can alternatively be calibrated with the Dynamic Calibration tool that can be installed as a separate software package, though I understand that you would prefer to use on-chip so that it can be integrated into your application.

https://www.intel.com/content/www/us/en/download/645988/intel-realsense-d400-series-dynamic-calibration-tool.html

[mksbabu]

Hi @MartyG-RealSense

I am able to get health value from on-chip calibration method with below code but have a doubt on it as the health value is changing slightly for each time with below code. What's the reason behind it and Am I doing anything wrong from my side?
rs2::pipeline pipe;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 256, 144, RS2_FORMAT_Z16, 90);
rs2::device dev = pipe.start(cfg).get_device();
rs2::auto_calibrated_device cal = dev.asrs2::auto_calibrated_device();
float health;
std::string json = "{"calib type": 0,"speed": 0,"scan parameter": 0,"white wall mode": 0}";
rs2::calibration_table res = cal.run_on_chip_calibration(json, &health, [&](const float progress) { /* On Progress */ });

could you please suggest how to get 3D screen for intel real sense viewer(I mean which method and class will use to get 3D screen from librealsense library)?

[MartyG-RealSense]

Hi @mksbabu That sounds normal. The health check score may vary with each new calibration performed. The user can repeat the calibration process multiple times if they wish to do so until they achieve a score that they are satisfied with and then they can save that calibration to the camera hardware.

The 3D mode of the RealSense Viewer is a 3D pointcloud image that represents measurement of depth in meters in the real-world scene. Intel provide a C++ pointcloud example script called rs-pointcloud at the link below.

https://github.com/IntelRealSense/librealsense/tree/master/examples/pointcloud

[mksbabu]

thank you @MartyG-RealSense, will look into it.

[mksbabu]

Hi @MartyG-RealSense

could you please help us on how to get device details(rs2::device) by using depth 2d or 3d feeds of depth image in c++ code.
Thanks in advance

[MartyG-RealSense]

Getting device information does not require depth feeds. There is a C++ tutorial at the link below about querying the camera for its device details and displaying them.

https://www.codeproject.com/Articles/1233892/FluentRealSense-The-First-Steps-to-a-Simpler-RealS

[mksbabu]

Hi @MartyG-RealSense

could you please suggest how to get 2D depth image of intel realsense camera with c++ code?

[MartyG-RealSense]

If you want a 2D depth image and not a 3D pointcloud then the RealSense SDK's C++ example rs-capture might fit your needs.

https://github.com/IntelRealSense/librealsense/tree/master/examples/capture

[mksbabu]

Thank you @MartyG-RealSense

[mksbabu]

Hi @MartyG-RealSense

I want to convert depth data(rs2::frame depth) or Mat image which includes depth data of intel realsense camera into char* such as bytes because I want to use this depth data in my java application.
could you please suggest how to do conversion of depth data into bytes?

This is my code block:
rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
rs2::frame depth = data.get_depth_frame().apply_filter(color_map);
// Query frame size (width and height)
const int w = depth.asrs2::video_frame().get_width();
const int h = depth.asrs2::video_frame().get_height();
// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);

[MartyG-RealSense]

There is a C++ script at the link below which looks similar to your rs2::frame depth code but they nest their code within this instruction where they place a char definition:

int main(int argc, char * argv[]) try
{

https://www.twblogs.net/a/5c36203ebd9eee35b3a57a8c/?lang=zh-cn

---

If you are creating a RealSense Java application then an alternative approach that you could consider is using an unofficial Java wrapper created by a RealSense community member.

https://github.com/cansik/librealsense-java

[mksbabu]

Hi @MartyG-RealSense

Thank you for above information!!

I have below query when I tried on-chip calibration with two cameras are connected to my laptop.
For rs2::auto_calibrated_device cal = devices[0].asrs2::auto_calibrated_device(); => I am able to get health value with devices[0] but If I changed this line with devices[1].asrs2::auto_calibrated_device(); getting below error. could you please suggest what's the reason for this failure if index changed to 1?
Microsoft C++ exception: rs2::invalid_value_error at memory location 0x000000E2812FF150.

code:
int main(int argc, char* argv[]) {
list cameralist;
rs2::context ctx;
rs2::device_list devices = ctx.query_devices();
rs2::device devobject;
float health;
cout << "spandana devices " << devices[1] << endl;
rs2::pipeline pipe;
pipe.start();
rs2::auto_calibrated_device cal = devices[1].asrs2::auto_calibrated_device();
//rs2::calibration_table ress = cal.get_calibration_table();
//float health;
std::string json = "{"calib type": 1,"speed": 0,"scan parameter": 0,"white wall mode": 0}";
rs2::calibration_table res = cal.run_on_chip_calibration(json, &health, [&](const float progress) { /* On Progress */ });
std::cout << "result: " << health;
}

[MartyG-RealSense]

Let's try setting the index to 0 so that the script works, and inserting the line below after your ctx.query_devices() line to see if all attached cameras are detected and listed.

rs2::device_list devices = ctx.query_devices();
std::cout << "Found the following devices:\n" << std::endl;

[mksbabu]

Yeah, All cameras are detected and able to see all info but only issue is with cal.run_on_chip_calibration method if index set to second device such as 1 and it's working only with one device when index set to 0.
If I test those cameras individually on chip calibration method is working with both devices but only issue is with when connected with both devices

Is there anything like run_on_chip_calibration method will work with only one device only?

[MartyG-RealSense]

Does it work if you change your auto_calibrated_device line to this (using devobject instead of dev for the device):

rs2::auto_calibrated_device cal = devobject[1].as<rs2::auto_calibrated_device>();

[mksbabu]

Hi @MartyG-RealSense

we are facing below error while calculating ground truth distance with intel realsense camera. could you please help us on it.

Code:
try {
rs2::pipeline pipe;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 256, 144, RS2_FORMAT_Z16, 90);
rs2::device dev = pipe.start(cfg).get_device();
int limit = 50;
rs2::frame_queue queue(limit * 2, true);
rs2::frame_queue queue2(limit * 2, true);
rs2::frame_queue queue3(limit * 2, true);
rs2::auto_calibrated_device cal = dev.asrs2::auto_calibrated_device();
const auto callback = [&](const float progress) -> void {std::cout << progress << std::endl; };
float res = cal.calculate_target_z(queue, queue2, queue3, 175.0, 100.0, callback);

}
catch (const rs2::error& e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}

Error:
RealSense error calling rs2_calculate_target_z_cpp(device:000001DE3B88E4A0, queue1:000001DE3B98F0C0, queue2:000001DE3B9B8360, queue3:000001DE3BA0D730, target_width:175, target_height:100):
value is below allowed min for argument "rs2_frame_queue_size(queue1, error)"

[MartyG-RealSense]

The example C++ code in the on-chip calibration guide does not use 175, 100 in the rs2_calculate_target_z instruction.

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-d-on-chip-calibration-c-api

What happens if you remove these values?

float res = cal.calculate_target_z(queue, queue2, queue3, callback);

[mksbabu]

@MartyG-RealSense I am good with on-chip calibration, trying to use cal.calculate_target_z method for tare calibration.

[MartyG-RealSense]

it looks as though cal.calculate_target_z is part of focal length calibration, which is a separate function to tare calibration. For tare calibration, the documentation suggests using the instruction below, with ground_thruth (their spelling of ground truth) presumably performing the same function as getting the GT in the RealSense Viewer with the 'Get' button during tare calibration.

rs2::calibration_table res = cal.run_tare_calibration(ground_thruth, json, [&](const float progress) { /* On Progress */ });

[mksbabu]

yes @MartyG-RealSense but we are facing below error by passing rs2::frame_queue's while calculating ground truth distance with c++ code for tare calibration. If we get ground truth value then we will try to add cal.run_tare_calibration() method for calibration. could you please help us on what's the reason for below error.

Code:
try {
rs2::pipeline pipe;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 256, 144, RS2_FORMAT_Z16, 90);
rs2::device dev = pipe.start(cfg).get_device();
int limit = 50;
rs2::frame_queue queue(limit * 2, true);
rs2::frame_queue queue2(limit * 2, true);
rs2::frame_queue queue3(limit * 2, true);
rs2::auto_calibrated_device cal = dev.asrs2::auto_calibrated_device();
const auto callback = [&](const float progress) -> void {std::cout << progress << std::endl; };
float res = cal.calculate_target_z(queue, queue2, queue3, 175.0, 100.0, callback);

}
catch (const rs2::error& e)
{
std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
return EXIT_FAILURE;
}
Error:
RealSense error calling rs2_calculate_target_z_cpp(device:000001DE3B88E4A0, queue1:000001DE3B98F0C0, queue2:000001DE3B9B8360, queue3:000001DE3BA0D730, target_width:175, target_height:100):
value is below allowed min for argument "rs2_frame_queue_size(queue1, error)"

[MartyG-RealSense]

I can point you towards how another RealSense user handled controlling tare calibration with C++ at #11629 (comment)

[mksbabu]

@MartyG-RealSense I have already checked on it, they hardcoded ground truth value instead of getting by method but we need it to calculate dynamically

[MartyG-RealSense]

If you will always be calibrating at the same distance from the target then it should be fine to hard-code the value. You could find the value for your particular setup using the Get button in the Viewer's interface and then hardcode that value into your script.

A calibration specialist on the Intel RealSense team provides expert advice about tare at #10213 (comment)

[mksbabu]

Thanks @MartyG-RealSense , I will have look on it

[mksbabu]

Hi @MartyG-RealSense
we want to calculate ground truth dynamically instead of specific value but we are getting below error with c++ code and same function is working with intel realsense viewer and python code (https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/depth_auto_calibration_example.py) with camera position which is using for c++ code. could you please help us on where it's wrong with c++ code.
Error:
RealSense error calling rs2_calculate_target_z_cpp(device:000001D88D83F150, queue1:000001D88E3D1980, queue2:000001D88D9E07A0, queue3:000001D88D9D87A0, target_width:175, target_height:100):
Failed to extract target dimension info!

Code:
#include <librealsense2/rs.hpp>
#include <opencv2/opencv.hpp>
#include
using namespace std;
int main(int argc, char* argv[])
{
try {
rs2::pipeline pipe;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_INFRARED, 1280, 720, RS2_FORMAT_Y8, 30);
rs2::device dev = pipe.start(cfg).get_device();
rs2::frame_queue queue(50, true);
rs2::frame_queue queue2(50, true);
rs2::frame_queue queue3(50, true);
rs2::frame f;
int counter = 0;
while (counter<50) {
f = pipe.wait_for_frames();
if (f)
{
queue.enqueue(f);
}
++counter;
}
rs2::auto_calibrated_device cal = dev.asrs2::auto_calibrated_device();
float res = cal.calculate_target_z(queue, queue2, queue3, 175.0, 100.0);
}
catch (const rs2::error& e)
{
std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
return EXIT_FAILURE;
}
}