Aravissrc times out when frame acquisition is not at a fixed frame rate #528

PauloFavero · 2021-07-15T18:45:20Z

Describe the bug

Not able to use aravissrc as a source when triggering the camera by hardware. The pipeline does not send the buffer downstream. The goal is to generate snapshots when triggering the camera from the hardware.

To Reproduce
1 - Set the camera to generate frames on demand when triggered by hardware events:

arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control AcquisitionMode='Continuous'
arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control ExposureTime='200000'
arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control TriggerSelector='FrameStart'
arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control TriggerMode='On'
arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control TriggerActivation='FallingEdge'
arv-tool-0.8 -n 'BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282' control TriggerSource='Line4'

2 - Launch gstd client:

gstd -p 5000 -n 2

3 - In another terminal, create and play the pipeline:

 gst-client pipeline_create test aravissrc name=balluff num-buffers=10 \
 ! video/x-bayer, format=rggb, width=1920, height=1080, framerate=0/1, pixel-aspect-ratio=1/1 ! bayer2rgb \
 ! video/x-raw, format=RGBA ! videoconvert ! jpegenc ! multifilesink async=false location=test.jpeg
 
 gst-client debug_enable true
 gst-client debug_color true
 gst-client debug_threshold **:6

 gst-client element_set test balluff camera-name "BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282"

 sleep 5
 gst-client pipeline_play test

 sleep 3
 gst-client pipeline_play test

4 - Trigger the camera.

In our setup, we have a camera with pin 4 connected to a laser sensor. When the laser is interrupted, we have a failing edge that triggers the camera to capture an image. This behavior works very well when running the wxPropView provided by the Balluff SDK.

Expected behavior
Store a jpeg image in the filesystem when a frame is present in the pipeline.

Camera description:

Manufacturer: Balluf
Model: GmbH-BVS CA-GX0-0032AC-GX301282
Interface: Gigabit M12 para Ethernet

Platform description:

Aravis 0.8.10
OS: Ubuntu 20.04
Hardware x86_64

Additional context

GSTD Logs:
gstd_logs.txt

Multifilesink without async prop:

When running the pipeline without setting the multifilesink async prop to false we get the following output:

streamer     | Setting pipeline to PAUSED ...
gstreamer     | Pipeline is live and does not need PREROLL ...
gstreamer     | Setting pipeline to PLAYING ...
gstreamer     | ERROR: from element /GstPipeline:pipeline0/GstAravis:aravis0: Internal data stream error.
gstreamer     | Additional debug info:
gstreamer     | gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstAravis:aravis0:
gstreamer     | streaming stopped, reason not-negotiated (-4)
gstreamer     | Execution ended after 0:00:00.000038603
gstreamer     | Setting pipeline to PAUSED ...
gstreamer     | Setting pipeline to READY ...
gstreamer     | Setting pipeline to NULL ...
gstreamer     | Freeing pipeline ...
gstreamer exited with code 1

Balluff settings (wxPropView):

The text was updated successfully, but these errors were encountered:

Opala76 · 2021-07-26T16:44:44Z

Looking forward to get a feedback about this issue :)

EmmanuelP · 2021-07-26T17:01:17Z

@PauloFavero Did you try arv-camera-test ?

PauloFavero · 2021-07-26T18:01:48Z

Hi @EmmanuelP

I just tried with arv-camera-test and it works. It is streaming at a maximum of 6 frames per second.

Does this test suggest that something else is wrong in the pipeline? Or is it something related to the arvsrc element?

Thanks

EmmanuelP · 2021-07-27T08:04:35Z

It is probably an issue with your pipeline. The simplest pipeline that is supposed to work is:

aravissrc ! videoconvert ! xvimagesink

You may want to use the helper script in the gst build directory:

./gst/gst-aravis-launch aravissrc ! videoconvert ! xvimagesink

PauloFavero · 2021-08-05T19:29:26Z

This is the setup that I am using to trigger the camera:

The Arduino is generating a square wave of 8Hz. This is the reason why the arv-camera-test-0.8 is showing 8 frames per second. The arv-camera-test keeps streaming in case If I disconnect the trigger or the trigger frequency drops down to zero during runtime.

This is the output from arv-camera-test:

$arv-camera-test-0.8
Looking for the first available camera
vendor name           = BALLUFF GmbH
model name            = BVS CA-GX0-0032AC
device serial number  = GX301282
image width           = 2064
image height          = 1544
horizontal binning    = 1
vertical binning      = 1
exposure              = 500 µs
gain                  = 0 dB
payload               = 3186816 bytes
gv n_stream channels  = 1
gv current channel    = 0
gv packet delay       = 0 ns
gv packet size        = 576 bytes
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  7 frames/s -    22.3 MiB/s
  9 frames/s -    28.7 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s
  8 frames/s -    25.5 MiB/s

The trigger in our application is not constant or continuous. It means that the camera is triggered when something appears in front of it. It could be in a certain frequency or sporadically.

We did more testing with the suggested pipeline and we narrowed down the problem a little bit. We created the following pipeline:

gst-launch-1.0 aravissrc camera-name="BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282" ! video/x-bayer, format=rggb ! bayer2rgb ! video/x-raw, format=RGBA ! videoconvert ! xvimagesink

When we have a constant frame rate being generated from the external trigger (Arduino square wave) it works perfectly. The problem appears when the pipeline starts without a buffer (no trigger) or if the trigger stops while the pipeline is running. The error output log below was generated when starting the pipeline without the trigger running. The error is the same when the trigger stops.

Test with GStreamer launcher with a frequency of 0Hz.

gst-launch-1.0 aravissrc camera-name="BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282" ! video/x-bayer, format=rggb ! bayer2rgb ! video/x-raw, format=RGBA ! videoconvert ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstAravis:aravis0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstAravis:aravis0:
streaming stopped, reason error (-5)
Execution ended after 0:00:02.004351405
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

It seems that there is a timeout somewhere that stops the pipeline when there is no buffer being streamed.

Does you know how to proceed in this case? Does it seem to be something related to the aravissrc element?

EmmanuelP · 2021-08-06T06:30:31Z

Hi,

Thanks for the analysis.

You are probably hitting the default 2s timeout aravissrc is using trying to pull a buffer when operating without a fixed frame rate.

aravis/gst/gstaravis.c

Line 45 in 1300e6b

#define GST_ARAVIS_BUFFER_TIMEOUT_DEFAULT 2000000

aravis/gst/gstaravis.c

Lines 261 to 276 in 1300e6b

    
           if (!error && frame_rate != NULL) { 
        
           	double dbl_frame_rate; 
        
           	dbl_frame_rate = (double) gst_value_get_fraction_numerator (frame_rate) / 
        
           		(double) gst_value_get_fraction_denominator (frame_rate); 
        
           	GST_DEBUG_OBJECT (gst_aravis, "Frame rate = %g Hz", dbl_frame_rate); 
        
           	arv_camera_set_frame_rate (gst_aravis->camera, dbl_frame_rate, &error); 
        
           	if (dbl_frame_rate > 0.0) 
        
           		gst_aravis->buffer_timeout_us = MAX (GST_ARAVIS_BUFFER_TIMEOUT_DEFAULT, 
        
           						     3e6 / dbl_frame_rate); 
        
           	else 
        
           		gst_aravis->buffer_timeout_us = GST_ARAVIS_BUFFER_TIMEOUT_DEFAULT; 
        
           } else 
        
           	gst_aravis->buffer_timeout_us = GST_ARAVIS_BUFFER_TIMEOUT_DEFAULT;

aravis/gst/gstaravis.c

Lines 519 to 525 in 1300e6b

    
           do { 
        
           	if (arv_buffer) arv_stream_push_buffer (gst_aravis->stream, arv_buffer); 
        
           	arv_buffer = arv_stream_timeout_pop_buffer (gst_aravis->stream, gst_aravis->buffer_timeout_us); 
        
           } while (arv_buffer != NULL && arv_buffer_get_status (arv_buffer) != ARV_BUFFER_STATUS_SUCCESS); 
        
           if (arv_buffer == NULL) 
        
           	goto error;

A quick fix would be to increase this default timeout, but that obviously does not sound like a proper solution.

The correct fix implies a better understanding of what is the expected behaviour of GstPushSrc::create override.

kbegiedza · 2022-04-16T17:56:50Z

Do we have any workaround ?

EmmanuelP · 2022-04-22T15:33:45Z

The easiest workaround is to increase the timeout value and recompile Aravis.

But a proper solution needs a better understanding of GStreamer. Any help welcome.

TIS-Edgar · 2022-04-25T05:27:59Z

Hi Emmanuel,

Maybe I can help. I suspect your problem is not the create function but the gstreamer latency handling.
At least that is where we had problems when dealing with TriggerMode.

Basically what happens is your source must be synchronized with the gstreamer clock.
Since it is a live source it can't give reliable increasing timestamps and instead works with durations.
Ideally that would mean first timestamp (0) + duration of frame. New timestamp + duration of frame, etc.
Since we cannot know what the duration of a frame is (some come faster, some slower, thus never being 100% accurate) gstreamer wants to know a latency range. Aravis uses the default implementation which is either min==max==0 or min==max==framerate that I could not find.

I suspect that that is the cause of your problems. Our cameras do not offer 0/1 so i can't verify this.
A better implementation would be min=framerate and max=GST_CLOCK_TIME_NONE.
That basically says expect the next buffer between average duration and the heat death of the universe,
giving you more than enough time to trigger or extended exposure times, etc.

A latency implementation that can almost be copied would be ours: https://github.com/TheImagingSource/tiscamera/blob/master/src/gstreamer-1.0/gsttcammainsrc.cpp#L1500

video/x-bayer, format=rggb, width=1920, height=1080, framerate=0/1

I am surprised that that is even a legal value. Setting framerate=8/1 seems like a good idea. It would certainly enable the default implementation to work better.

To get a better idea of what the source is doing some logging might help.

gst-launch-1.0 --gst-debug=aravissrc:5basesrc:5 aravissrc camera-name="BALLUFF GmbH-BVS CA-GX0-0032AC-GX301282" ! video/x-bayer, format=rggb ! bayer2rgb ! video/x-raw, format=RGBA ! videoconvert ! xvimagesink

Does that help or am I overlooking an obvious bit of information?

turowicz · 2022-05-23T12:38:54Z

@EmmanuelP tl;dr nobody from this project knows how to add a property to aravissrc and pull it up in code as a timeout value?

EmmanuelP · 2022-05-23T12:42:02Z

@turowicz Adding yet another property is not the right solution. @TIS-Edgar gave some thoughts on how to solve this issue properly, but nobody stepped up with a fix yet.

turowicz · 2022-05-23T12:45:03Z

@EmmanuelP looks like this issue is not exactly the same as one I've reported. I do not need to change the framerate.

TIS-Edgar · 2022-05-23T13:14:34Z

Sorry, I was unsure if anyone wanted to give additional input and had a bit of additional projects to deal with.

@PauloFavero I created a branch that implements the latency feature I described. You can find it here https://github.com/TIS-Edgar/aravis/tree/issue/528/add-gst-query-latency

Would it be possible for you to test and verify that this solves this problem? I am unable to test that with our cameras and would like to know that before making an official merge request.

TIS-Edgar · 2022-06-28T05:44:18Z

@EmmanuelP Since I received no comments on the feature branch from @PauloFavero, should I simply open a merge request so that this can keep moving?

EmmanuelP · 2022-06-28T05:55:15Z

@EmmanuelP Since I received no comments on the feature branch from @PauloFavero, should I simply open a merge request so that this can keep moving?

Yes please.

EmmanuelP · 2022-07-07T08:41:11Z

Should be fixed by #686. Please reopen if the issue persists.

EmmanuelP · 2022-08-12T09:25:24Z

It looks like the issue is not completely fixed, as reported here: #704

We still use a 2s timeout when waiting for a buffer.

aravis/gst/gstaravis.c

Line 311 in 43b3ea4

    
           GST_DEBUG_OBJECT (gst_aravis, "Buffer timeout = %" G_GUINT64_FORMAT " µs", gst_aravis->buffer_timeout_us);

We can not really increase this default timeout, and already busy waiting for 2 seconds is wrong. A proper solution requires a better knowledge of how we are supposed implement this sort of intermittent source.

Any help appreciated !

lc-spxl · 2022-08-12T11:58:08Z

My use case is with 4x cameras with hardware trigger on each except one.

I used a spinnaker gst pipeline before which seems to wait for the initial buffer.

I am looking at

aravis/gst/gstaravis.c

Lines 555 to 561 in 43b3ea4

    
           do { 
        
           	if (arv_buffer) arv_stream_push_buffer (gst_aravis->stream, arv_buffer); 
        
           	arv_buffer = arv_stream_timeout_pop_buffer (gst_aravis->stream, gst_aravis->buffer_timeout_us); 
        
           } while (arv_buffer != NULL && arv_buffer_get_status (arv_buffer) != ARV_BUFFER_STATUS_SUCCESS); 
        
           if (arv_buffer == NULL) 
        
           	goto error;

Which seems the right place to wait for a frame, eventually checking if a trigger is enabled.

lc-spxl · 2022-08-29T10:56:33Z

@EmmanuelP above PR has just been tested in lab with hardware trigger configured. It works properly by starting streaming once the camera without trigger stream the first frame. Thanks

GabrielMonS · 2023-07-10T11:34:59Z

Hello, we are very interested in this feature. Will this be merged in main anytime soon? Thank you for your time.

PauloFavero changed the title ~~Aravissrc don't get buffers from a camera with I/O trigger~~ Aravissrc unable to get buffers from a camera with I/O trigger Jul 15, 2021

PauloFavero changed the title ~~Aravissrc unable to get buffers from a camera with I/O trigger~~ Aravissrc unable to get snapshots from a camera with I/O trigger Jul 15, 2021

EmmanuelP added the 5. Gstreamer Issue in GStreamer plugin label Jul 27, 2021

EmmanuelP changed the title ~~Aravissrc unable to get snapshots from a camera with I/O trigger~~ Aravissrc times out when frame acquisition is not at a fixed frame rate Aug 6, 2021

EmmanuelP added the 1. bug Problems, incorrect behavior or appearance label Aug 6, 2021

EmmanuelP mentioned this issue May 23, 2022

Configure buffer timeout for GST #675

Closed

TIS-Edgar mentioned this issue Jun 28, 2022

gst: Add GstQueryLatency handling #686

Merged

EmmanuelP closed this as completed Jul 7, 2022

lc-spxl mentioned this issue Aug 11, 2022

gst pipeline closes (-5) without waiting for hardware trigger #704

Closed

EmmanuelP reopened this Aug 12, 2022

EmmanuelP added the 4. Help wanted Help is welcome to resolve the issue label Aug 12, 2022

lc-spxl linked a pull request Aug 12, 2022 that will close this issue

Keep reading if TriggerMode is set to On #705

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aravissrc times out when frame acquisition is not at a fixed frame rate #528

Aravissrc times out when frame acquisition is not at a fixed frame rate #528

PauloFavero commented Jul 15, 2021 •

edited

Opala76 commented Jul 26, 2021

EmmanuelP commented Jul 26, 2021

PauloFavero commented Jul 26, 2021

EmmanuelP commented Jul 27, 2021

PauloFavero commented Aug 5, 2021

EmmanuelP commented Aug 6, 2021

kbegiedza commented Apr 16, 2022

EmmanuelP commented Apr 22, 2022

TIS-Edgar commented Apr 25, 2022

turowicz commented May 23, 2022

EmmanuelP commented May 23, 2022 •

edited

turowicz commented May 23, 2022

TIS-Edgar commented May 23, 2022

TIS-Edgar commented Jun 28, 2022

EmmanuelP commented Jun 28, 2022

EmmanuelP commented Jul 7, 2022

EmmanuelP commented Aug 12, 2022 •

edited

lc-spxl commented Aug 12, 2022

lc-spxl commented Aug 29, 2022

GabrielMonS commented Jul 10, 2023

Aravissrc times out when frame acquisition is not at a fixed frame rate #528

Aravissrc times out when frame acquisition is not at a fixed frame rate #528

Comments

PauloFavero commented Jul 15, 2021 • edited

Opala76 commented Jul 26, 2021

EmmanuelP commented Jul 26, 2021

PauloFavero commented Jul 26, 2021

EmmanuelP commented Jul 27, 2021

PauloFavero commented Aug 5, 2021

EmmanuelP commented Aug 6, 2021

kbegiedza commented Apr 16, 2022

EmmanuelP commented Apr 22, 2022

TIS-Edgar commented Apr 25, 2022

turowicz commented May 23, 2022

EmmanuelP commented May 23, 2022 • edited

turowicz commented May 23, 2022

TIS-Edgar commented May 23, 2022

TIS-Edgar commented Jun 28, 2022

EmmanuelP commented Jun 28, 2022

EmmanuelP commented Jul 7, 2022

EmmanuelP commented Aug 12, 2022 • edited

lc-spxl commented Aug 12, 2022

lc-spxl commented Aug 29, 2022

GabrielMonS commented Jul 10, 2023

PauloFavero commented Jul 15, 2021 •

edited

EmmanuelP commented May 23, 2022 •

edited

EmmanuelP commented Aug 12, 2022 •

edited