Does anyone know what the elt buffer size limit is?

[bikerdan]

What is the largest the elt.buffer_size can be set to? Would it be possible to set it to say 2GB?

[tayloramurphy]

@aaronsteers @edgarrmondragon do you all know?

[edgarrmondragon]

Not off the top of my head, but the absolute upper limit and how to query it surely changes from system to system. Safe values are probably something well below the memory constraints of the computer or container.

[bikerdan]

So as far as we know, the limits are with the system itself? Or could there be limits on the code and libraries used for the buffer?

[edgarrmondragon]

Yeah, the limits are in the system itself. The value is passed to asyncio.create_subprocess_exec, and there are no hard limits there afaict.

[aaronsteers]

Agreed with @edgarrmondragon above. My understanding is that this is only limited by the available RAM on the machine.

Larger values will increase the amount of unprocessed data that can be held in the target's input buffer before the tap must be paused. If there's enough RAM to hold the entire dataset in memory, then in theory the tap could run entirely to completion without ever needing to be paused. (Although the target's own performance and runtime duration may still be unchanged.)

[bikerdan]

Thanks for the clarification.