Question: What a correct way of pthreads start-up?

[pavelkumbrasev]

Hi everyone,
I'm working on improving oneTBB to support Emscripten and specifically on this issue our users faced.
The issue is that threads (tbb-workers) would not arrive into parallel regions and then everything is executed serially.
Our thread launch/wake-up model is "binary tree" where main thread will launch/wake-up only 2 threads and this threads will propagate signal to their subsequent threads.
Let's consider this example. It contains two implementations:

1. Serial creation of threads. (#define SERIAL 1 should be used)
2. Tree like structure where each thread creates subsequent threads.

#include <chrono>
#include <iostream>
#include <thread>
#include <atomic>
#include <vector>
#include <functional>


int main() {
    int num_threads = std::thread::hardware_concurrency();

    std::atomic<int> barrier{num_threads};
    std::vector<std::thread> threads;

#if SERIAL
    auto thread_func = [&] {
        auto ticket = --barrier;
        while (barrier > 0) {}
        std::cout << ticket << std::endl;
    };

    for (int i = 0; i < num_threads-1; ++i) {
        threads.emplace_back(thread_func);
    }

    thread_func();
#else
    threads.reserve(num_threads);

    new (&threads[0]) std::thread{}; // Empty root thread
    std::function<void(int)> thread_func = [&] (int node_index) {
        int left_child_index = 2 * node_index + 1;
        int right_child_index = 2 * node_index + 2;

        if (left_child_index < num_threads) {
            new (&threads[left_child_index]) std::thread(thread_func, left_child_index);
        }

        if (right_child_index < num_threads) {
            new (&threads[right_child_index]) std::thread(thread_func, right_child_index);
        }

        auto ticket = --barrier;
        while (barrier > 0) {
            // std::this_thread::yield();
        }
        std::cout << ticket << std::endl;
    };

    thread_func(0);
#endif
    for (int i = 0; i < num_threads-1; ++i) {
        if (threads[i].joinable()) {
            threads[i].join();
        }
    }

}

Using -sPROXY_TO_PTHREAD both code paths work with no problem. The problem is that not all the user can use it without refactoring.
So I tried to experiment without this flag. Seems SERIAL code path is always working and it doesn't matter will I call join immediately after thread creation or use busy wait with no yields (as presented in example).
However, tree like approach hangs unless I add std::this_thread::yield() into busy wait.
[1] As I understood it happens because the first two threads will synchronously wait for main thread to help them with thread creation which it will never do because it locked in busy loop.

Now I'm trying to figure our what will be the best approach for thread creation with Emscripten.
So I have several questions:

1. Is the above [1] correct?
2. What is the difference with -sPROXY_TO_PTHREAD? Why tree-like approach is working with the flag? How does it helps for subsequent threads creation?
3. What do you think will there be performance difference in this approaches if main thread participate in thread creation no matter what?

[jellychen]

1. It's worth noting that if the memory allocator in Emscripten is not mimalloc, its multi-threaded malloc implementation has a global lock.
2. Mimalloc appears to not have the global lock limitation, but I have tested and found that TBB also has issues under the mimalloc allocator.
3. The operation of PROXY_TO_PTHREAD seems very similar to the warm-up process I've performed, just based on feeling.

[sbc100]

The core issue here is that new workers (the things that host pthreads) can start running if the main browser thread yeilds to the event loop.

Because your main function is blocking, if your main function on the main browser thread then it cannot create new workers. PROXY_TO_PTHREAD simply runs your main function on a background thread, leaving the main browser thread free to handle new worker creation. An alternative to this approach would be to use -sPTHREAD_POOL_SIZE=N to create N workers ahead of time, alleviating the need to create new ones at runtime at all.

See https://emscripten.org/docs/porting/pthreads.html#blocking-on-the-main-browser-thread

[pavelkumbrasev]

Hi @sbc100, I've tried to run the tree start-up with -sPTHREAD_POOL_SIZE=N but it still hangs. Could you please comment on this?

[sbc100]

Have you tired -sPROXY_TO_PTHREAD?

[pavelkumbrasev]

As I stated in original question I have tried it and it works fine. The problem is that oneTBB is a threading library and we cannot force all our users to use -sPROXY_TO_PTHREAD (because it requires refactoring) so I'm trying to consider all possible solutions.

[sbc100]

One possible solution is to create works that is dedicated to starting other workers. Right now the main thread is starting new workers, which is leading to this issue. In that past I don't think all browsers supports nested workers.. but if they then we could get rid of the entire PTHREAD_POOL_SIZE concept that just have new thread created by the dedicated "workers-creating-worker".

i'm not sure how much work this would be.

[pavelkumbrasev]

This "workers-creating-worker" would be really nice to have