TCP CCAs in Rust #1062

vobst · 2024-02-11T22:10:04Z

Hi all!

I've been experimenting with writing TCP congestion control algorithms (CCAs) in Rust. These patches contain abstractions for writing modules that define a single CCA, a minimal proof-of-concept CCA, and a reimplementation of BIC (the predecessor of the CUBIC algorithm that is the default choice in the kernel).

The end goal of this work is to eventually have a new CCA, i.e., one with no current upstream implementation, implemented in Rust at some point in the (probably distant) future. With that in mind, I'd appreciate if you could give my abstraction design a close look. I'm happy about all kinds of feedback (design, structure, implementation, comments, docs,...) but since this is a pretty early stage, high-level things may be more useful that tiny details :) (Note: The relevant patches are the last three.)

By now, I've been using these patches for a few days on my host system (these very characters are sent on a connection that uses bic_rust ;) ), but I lack the knowledge/setup to perform proper testing against the C implementation in a lab setup; I'd appreciate some help/suggestions on that. For what it helps, here is an Arch Linux package that the Arch-users among you can use to easily build a kernel package for your host system.

In general, these choices select Rust BIC as the default for all connections:

Symbol: RUST_TCP_ABSTRACTIONS [=y]                                                                                              │
  │ Type  : bool                                                                                                                    │
  │ Defined at net/ipv4/Kconfig:469                                                                                                 │
  │   Prompt: TCP: Rust abstractions                                                                                                │
  │   Depends on: NET [=y] && INET [=y] && RUST [=y]                                                                                │
  │   Location:                                                                                                                     │
  │     -> Networking support (NET [=y])                                                                                            │
  │       -> Networking options                                                                                                     │
  │         -> TCP/IP networking (INET [=y])                                                                                        │
  │ (1)       -> TCP: Rust abstractions (RUST_TCP_ABSTRACTIONS [=y])
Symbol: TCP_CONG_BIC_RUST [=y]                                                                                                  │
  │ Type  : tristate                                                                                                                │
  │ Defined at net/ipv4/Kconfig:505                                                                                                 │
  │   Prompt: Binary Increase Congestion (BIC) control (Rust rewrite)                                                               │
  │   Depends on: NET [=y] && INET [=y] && TCP_CONG_ADVANCED [=y] && RUST_TCP_ABSTRACTIONS [=y]                                     │
  │   Location:                                                                                                                     │
  │     -> Networking support (NET [=y])                                                                                            │
  │       -> Networking options                                                                                                     │
  │         -> TCP/IP networking (INET [=y])                                                                                        │
  │           -> TCP: advanced congestion control (TCP_CONG_ADVANCED [=y])                                                          │
  │ (4)         -> Binary Increase Congestion (BIC) control (Rust rewrite) (TCP_CONG_BIC_RUST [=y])
  │ Symbol: DEFAULT_BIC_RUST [=y]                                                                                                   │
  │ Type  : bool                                                                                                                    │
  │ Defined at net/ipv4/Kconfig:709                                                                                                 │
  │   Prompt: Bic (Rust)                                                                                                            │
  │   Depends on: <choice>                                                                                                          │
  │   Visible if: <choice> && TCP_CONG_BIC_RUST [=y]=y                                                                              │
  │   Location:                                                                                                                     │
  │     -> Networking support (NET [=y])                                                                                            │
  │       -> Networking options                                                                                                     │
  │         -> TCP/IP networking (INET [=y])                                                                                        │
  │           -> TCP: advanced congestion control (TCP_CONG_ADVANCED [=y])                                                          │
  │             -> Default TCP congestion control (<choice> [=y])                                                                   │
  │ (2)           -> Bic (Rust) (DEFAULT_BIC_RUST [=y])

And these sysctl knobs can be used to set it at runtime:

$ cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno bic bic_rust cubic my_cca
$ echo bic_rust | sudo tee /proc/sys/net/ipv4/tcp_congestion_control

Note: There are a bunch of print statements that are obviously not intended to remain there forever, but I helps to see the algorithm "at work".

tgross35 · 2024-02-12T21:42:43Z

net/ipv4/tcp_bic_rust.rs

+//!     Binary Increase Congestion Control (BIC) for Fast Long-Distance
+//!     Networks - Lisong Xu, Khaled Harfoush, and Injong Rhee
+//!     IEEE INFOCOM 2004, Hong Kong, China, 2004, pp. 2514-2524 vol.4
+//!     doi: 10.1109/INFCOM.2004.1354672


Could you add a link if available?

Yes. There is a (dead) link in the C file. The Wayback Machine has a backup.

The DOI link would be the cleanest option, however, there the pdf is behind a paywall. Other sources for the pdf exist, but nothing I'd want to link from a kernel source file ;)

tgross35 · 2024-02-12T21:45:41Z

net/ipv4/tcp_bic_rust.rs

+/// In binary search, go to point: `cwnd + (W_max - cwnd) / BICTCP_B`.
+// SAFETY: Obvious.
+// TODO: Convert to `new::(x).unwrap()` once 'const_option' is stabilised.
+const BICTCP_B: NonZeroU32 = unsafe { NonZeroU32::new_unchecked(4) };


Note that you will need a // SAFETY: ... comment for these, even though obvious.

However - since you use a lot of these constants as u32 rather than needing them to be NonZero*, it could be more clear to just make them u32 and do a static assert that they are nonzero.

Note that you will need a // SAFETY: ... comment for these, even though obvious.

In fact, it is not even going to compile when passing zero ... playground ... but I'll still add some more meaningful comment.

However - since you use a lot of these constants as u32 rather than needing them to be NonZero*, it could be
more clear to just make them u32 and do a static assert that they are nonzero.

Hmm, eventually I'd like to convert most of these constants to module parameters to match the C implementation. Ideally, there will be an option to have module parameters of type NonZeroXYZ because currently accidentally panicking your kernel is as easy as echo 0 > /sys/module/tcp_bic/parameters/max_increment. Therefore, even though it doesn't make much sense at present, I'd leave them as NonZeroU32 to make transitioning easier once there are the required module parameters.

net/ipv4/tcp_bic_rust.rs

tgross35 · 2024-02-12T22:16:01Z

rust/kernel/lib.rs

@@ -66,7 +66,7 @@ const __LOG_PREFIX: &[u8] = b"rust_kernel\0";
 /// The top level entrypoint to implementing a kernel module.
 ///
 /// For any teardown or cleanup operations, your type may implement [`Drop`].
-pub trait Module: Sized + Sync {
+pub trait Module: Sized + Sync + Send {


I'm not sure if we should add this, I think some modules have the requirement that they must be set up and torn down from the same context. (I know this comes from Wedson's work)

tgross35 · 2024-02-12T22:43:48Z

rust/kernel/net/tcp/cong.rs

+    fn cong_avoid(sk: &mut Sock<Self>, ack: u32, acked: u32);
+
+    /// Called before the sender's congestion state is changed.
+    fn set_state(sk: &mut Sock<Self>, new_state: State) {}


These default implementations should have a kernel::build_error(VTABLE_DEFAULT_ERROR). My explanation at https://lore.kernel.org/rust-for-linux/CALNs47svLHD3=LFrhi=zf4hdr--e6hGjTzT5X_U9yd8q5r7G7g@mail.gmail.com/ may help

rust/kernel/net/tcp/cong.rs

tgross35 · 2024-02-12T22:46:59Z

rust/kernel/net/tcp/cong.rs

+/// - `inet_csk_ca(sk)` points to a valid instance of `T::Data`, which belongs
+///   to the instance of the algorithm used by this socket. A callback has
+///   exclusive, mutable access to this data.
+pub struct Sock<T: Algorithm + ?Sized> {


This struct lives in net/sock.h right? It should be in the same module here.

I feel like the C side sock vs. socket might not be something we want to keep, but I don't have any better suggestion.

Making this type generic over a CCA algorithm seems somewhat specialized.

rust/kernel/net/tcp/cong.rs

rust/kernel/time.rs

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

When generating the `rust-project.json`, also search for modules in the `net/` folder. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

This allows modules to be initialised in-place in pinned memory, which enables the usage of pinned types (e.g., mutexes, spinlocks, driver registrations, etc.) in modules without any extra allocations. Drivers that don't need this may continue to implement `Module` without any changes. Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com> [kernel@valentinobst.de: remove feature return_position_impl_trait_in_trait as it is now stabilised] [kernel@valentinobst.de: remove `Send` trait bound on `Module` and `InPlaceModule`]

We'll need it, for example, when calling `register_filesystem` to initialise a file system registration. Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>

In net/tcp time values are usually 32bit wide unsigned integers, and either in units of jiffies, microseconds or milliseconds. Add types, constants, and functions to work with 32bit time values. This is, for example, used in the CUBIC and BIC CCAs. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

Add a macro to determine the size of a structure field at compile time. This is used by the CCA abstractions to ensure that the private data of every CCA will fit into the space that the kernel provides for it. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

# Describe the purpose of this series. The information you put here # will be used by the project maintainer to make a decision whether # your patches should be reviewed, and in what priority order. Please be # very detailed and link to any relevant discussions or sites that the # maintainer can review to better understand your proposed changes. If you # only have a single patch in your series, the contents of the cover # letter will be appended to the "under-the-cut" portion of the patch. # Lines starting with # will be removed from the cover letter. You can # use them to add notes or reminders to yourself. If you want to use # markdown headers in your cover letter, start the line with ">#". # You can add trailers to the cover letter. Any email addresses found in # these trailers will be added to the addresses specified/generated # during the b4 send stage. You can also run "b4 prep --auto-to-cc" to # auto-populate the To: and Cc: trailers based on the code being # modified. Signed-off-by: Valentin Obst <kernel@valentinobst.de> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 1, "change-id": "20240301-tcp-cca-rfc-1a0fbcaa533c", "prefixes": [] } }

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

Add an example that uses the `module_cca` macro and the `Algorithm` trait to implement a minimal CCA. IMPORTANT: This CCA is not compliant with the relevant RFCs and must not be used outside of test environments. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

Reimplement the Binary Increase Congestion (BIC) control algorithm in Rust. BIC is one of the smallest CCAs in the kernel and this mainly serves as a minimal example for a real-world algorithm. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

CUBIC is the default CCA since 2.6.18. Missing features compared to the C implementation: - configuration via module parameters, - exporting callbacks to BPF programs as kfuncs. Changes compared to the C implementation: - uses only SI units for time, i.e., no jiffies and `BICTCP_HZ`, Signed-off-by: Valentin Obst <kernel@valentinobst.de>

tgross35 reviewed Feb 12, 2024

View reviewed changes

ojeda force-pushed the rust-next branch from 2392e2a to e3c3d34 Compare February 18, 2024 20:22

vobst force-pushed the tcp-cca-rfc branch 2 times, most recently from 0ea9f23 to 8399bea Compare February 18, 2024 21:44

vobst changed the base branch from rust-next to rust-net February 18, 2024 21:50

vobst changed the base branch from rust-net to rust-next February 18, 2024 21:51

vobst changed the base branch from rust-next to rust-net February 19, 2024 09:35

vobst marked this pull request as draft February 19, 2024 09:36

Valentin Obst added 2 commits February 19, 2024 10:46

net/tcp: add logging to BIC

a275bc8

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

scripts/gen_rust_analyzer: search modules in net/

12e579b

When generating the `rust-project.json`, also search for modules in the `net/` folder. Signed-off-by: Valentin Obst <kernel@valentinobst.de>

vobst force-pushed the tcp-cca-rfc branch 3 times, most recently from 257334e to 1986ac1 Compare February 25, 2024 20:35

wedsonaf and others added 11 commits March 2, 2024 15:41

rust: init: introduce Opaque::try_ffi_init

a13e700

We'll need it, for example, when calling `register_filesystem` to initialise a file system registration. Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>

rust/net: add sock, tcp_sock and icsk wrappers

ded5104

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

rust/net: add CCA abstractions

f410fa0

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

rust/net: add implementation of HyStart

67c08f7

Signed-off-by: Valentin Obst <kernel@valentinobst.de>

vobst force-pushed the tcp-cca-rfc branch from 1986ac1 to 2253d8e Compare March 9, 2024 21:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TCP CCAs in Rust #1062

TCP CCAs in Rust #1062

vobst commented Feb 11, 2024 •

edited

tgross35 Feb 12, 2024

vobst Feb 14, 2024 •

edited

tgross35 Feb 12, 2024

vobst Feb 14, 2024 •

edited

tgross35 Feb 12, 2024

tgross35 Feb 12, 2024

tgross35 Feb 12, 2024

TCP CCAs in Rust #1062

Are you sure you want to change the base?

TCP CCAs in Rust #1062

Conversation

vobst commented Feb 11, 2024 • edited

tgross35 Feb 12, 2024

Choose a reason for hiding this comment

vobst Feb 14, 2024 • edited

Choose a reason for hiding this comment

tgross35 Feb 12, 2024

Choose a reason for hiding this comment

vobst Feb 14, 2024 • edited

Choose a reason for hiding this comment

tgross35 Feb 12, 2024

Choose a reason for hiding this comment

tgross35 Feb 12, 2024

Choose a reason for hiding this comment

tgross35 Feb 12, 2024

Choose a reason for hiding this comment

vobst commented Feb 11, 2024 •

edited

vobst Feb 14, 2024 •

edited

vobst Feb 14, 2024 •

edited