MOD-6572 replace semaphore with wait in jobq

[meiravgri]

This PR significantly modifies the threadpool's pulling mechanism, setting the stage for more flexible runtime configurations.

Changing pulling synchronization machanism

Previously, the threadpool was synchronized using a semaphore-based waiting mechanism for job retrieval. This PR replaces that mechanism with a condition variable.

The revised job pulling mechanism operates as follows:
2. Threads enter a waiting state on a condition variable when the job queue is both empty and running.
3. Upon adding a new job to the queue:

1. Single job addition triggers a signal to one thread.
2. Adding multiple jobs results in a broadcast signal, waking up all threads for processing.

Configure threadpool at runtime

Threadpool state

can be either initialized - has n_threads valid and ready to pull threads,
or uninitialized - some or all of the threads may have exited.
Threadpool state is set to uninitialized when terminate_when_empty or destroy are called.

Removed threads_all_idle condition variable

drain and wait are internally implemented with a busy wait.
calling wait() is equivalent to calling drain(threshold = 0, yieldCB = nullptr)

Thread state

A thread can be in one of three states: running, terminate_when_empty, or dead.
The thread state can be configured by pushing a change_state_job to the admin priority queue.

Pause and resume

When pausing the threadpool, the jobq state is changed to paused. pause function will return when there are no more jobs in progress (i.e num_working_threads == 0)

Intialization

The new design assumes that the threadpool is initialized by the main thread.
All threadpool initialization is lazy and occurs upon the first push to the queue.
Since the GC pushes to the jobq from the bg, it breaks this assumption. Hence, the GC thpool initialized upon module startup.

Auestethic changes

Add name to the threadpool

The threadpool name is used to set the thepool's threads names.
a thread name is <thpool_name>-<rand_id>.
As the maximum number of threads per thpool is 10,000, rand_id is a random number between 0 to 9,999.

Remove thpool->threads array

This array was unnecessary and has been removed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.25%. Comparing base (1c97090) to head (2de2d3b).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4446   +/-   ##
=======================================
  Coverage   86.25%   86.25%           
=======================================
  Files         190      190           
  Lines       34822    34822           
=======================================
+ Hits        30035    30037    +2     
+ Misses       4787     4785    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GuyAv46]

Nice!

[GuyAv46]

Consider passing the pool name as a parameter. I would at least add a comment about why we chose this name

[meiravgri]

2 rows above the array explictly initialized to size 1 with a comment "Only used by the coordinator, so 1 is enough"

[GuyAv46]

Consider moving into priority_queue_push_chain to avoid duplications

[meiravgri]

I don't think it's the priority queue's responsibility to initialize the threads.

1. I added a wrapper function priority_queue_push_chain_init_threads to the threadpool section
2. priority_queue_push_chain_unsafe: replaced redisearch_thpool_t * argument that was used only to get the priority queue, with a priority_queue *

[GuyAv46]

Very well done