Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SO_INCOMING_CPU support to net.sock_option_flag_in #11366

Merged
merged 2 commits into from
Jun 9, 2024

Conversation

masaori335
Copy link
Contributor

@masaori335 masaori335 commented May 20, 2024

Address #5165.

Add a option (128) for SO_INCOMING_CPU to the proxy.config.net.sock_option_flag_in.

Background

There was a bug with SO_REUSEPORT since v4.6. But it's fixed by v6.2 and some distros backported it.
Now we can use SO_INCOMING_CPU, SO_REUSEPORT and CPU Affinity 4 for perfect locality.

torvalds/linux@b261eda

Ref.

    SO_INCOMING_CPU (gettable since Linux 3.19, settable since Linux 4.4)

          Sets or gets the CPU affinity of a socket.  Expects an
          integer flag.

              int cpu = 1;
              setsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu,
                         sizeof(cpu));

          Because all of the packets for a single stream (i.e., all
          packets for the same 4-tuple) arrive on the single RX
          queue that is associated with a particular CPU, the
          typical use case is to employ one listening process per RX
          queue, with the incoming flow being handled by a listener
          on the same CPU that is handling the RX queue.  This
          provides optimal NUMA behavior and keeps CPU caches hot.

https://www.man7.org/linux/man-pages/man7/socket.7.html

@masaori335 masaori335 self-assigned this May 20, 2024
@masaori335 masaori335 added this to the 10.1.0 milestone May 20, 2024
@masaori335 masaori335 changed the title Add SO_INCOMING_CPU support on net.sock_option_flag_in Add SO_INCOMING_CPU support to net.sock_option_flag_in May 20, 2024
EThread *ethread = this_ethread();

#ifdef TS_USE_HWLOC
cpu = ethread->hwloc_obj->os_index;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hwloc_obj::os_index looks "OS-provided physical index number" and it's working with exec_thread.affinity: 4 as expected. However, hwloc has any better API?
https://github.com/open-mpi/hwloc/blob/63a8288d31a1baf67a909466aba9a022c78ca7b1/include/hwloc.h#L491

@masaori335 masaori335 merged commit 266b666 into apache:master Jun 9, 2024
15 checks passed
@masaori335
Copy link
Contributor Author

@cmcfarlen I'd request to 10.0.x backport. But, this is a new feature, so we can wait for 10.1.x, if you want.

@cmcfarlen
Copy link
Contributor

Cherry-picked to v10.0.x

cmcfarlen pushed a commit that referenced this pull request Jun 10, 2024
* Add SO_INCOMING_CPU support to net.sock_option_flag_in

* Fix FreeBSD Build

(cherry picked from commit 266b666)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: picked-10.0.0
Development

Successfully merging this pull request may close these issues.

None yet

3 participants