Slow work during testing on VM #352

grishy · 2023-07-08T15:44:17Z

Hi 👋

I wanted to add mDNS support for wireguard and decided to take userspace module for it. But during the test I found the strange thing that the bandwidth dropped a lot and the ping increased in boringtun. This does not happen when switching to kernel or Go implementation (just switch WG_QUICK_USERSPACE_IMPLEMENTATION).

The latest version is used as mentioned in the README. This appeared after some time of testing. In the beginning everything was fine. As you can see the CPU is not busy.

vagrant@wgserver:~$ boringtun-cli -V
boringtun 0.6.0

vagrant@wgserver:~$ rustup -V
rustup 1.26.0 (5af9b9484 2023-04-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.70.0 (90c541806 2023-05-31)`

vagrant@wgserver:~$ hostnamectl
 Static hostname: wgserver
 Pretty hostname: wg_server
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: 794e8af11ca343c3ac86176f4506a03e
         Boot ID: 5c51191b3166487ea3d9dad3aa1ec591
  Virtualization: oracle
Operating System: Ubuntu 23.04
          Kernel: Linux 6.2.0-24-generic
    Architecture: x86-64
 Hardware Vendor: innotek GmbH
  Hardware Model: VirtualBox
Firmware Version: VirtualBox

P.S. Maybe I should also mention in the README that WG_QUICK_USERSPACE_IMPLEMENTATION parameter doesn't work without fixes because it doesn't go into basic mode inside if

~ Sergei

The text was updated successfully, but these errors were encountered:

grishy · 2023-07-08T16:18:27Z

If you know how to debug this and its interesting , I will be glad to hear :)
(I am not a Rust professional)

grishy · 2023-07-08T17:23:35Z

I tested with a remote server (one of VM - cloud VM) and got the same results

Noah-Kennedy · 2023-07-08T17:33:37Z

Interesting!

Can you walk through the different versions and see if this is a regression?

Noah-Kennedy · 2023-07-08T17:40:23Z

The cli used to just be boringtun, before we split the crates.

grishy · 2023-07-08T20:33:27Z

I will try it again in the cloud and if it works, I will write 👍

grishy · 2023-07-08T20:39:24Z

On clean new env same speed ~~BUT now we have 100% CPU.~~ Just same, one core. Load not from iperf side, because in load less 1%. By kernel I guess.

P.S. Kernel and wg-go OK

Also OS version is different + kernel 5.15:

> hostnamectl
 Static hostname: server
       Icon name: computer-vm
         Chassis: vm
      Machine ID: bfa83f0910c74cd0b5936a1ec423c924
         Boot ID: 9701f19ae2814bb5ae25d01104db086a
  Virtualization: microsoft
Operating System: Ubuntu 22.04.2 LTS
          Kernel: Linux 5.15.0-76-generic
    Architecture: x86-64
 Hardware Vendor: Red Hat
  Hardware Model: KVM

grishy · 2023-07-08T20:46:14Z

v0.5.2 - have a problem during installation.

cargo install boringtun-cli@0.5.2

Output:

...many errors...
...
error[E0412]: cannot find type `StaticSecret` in crate `x25519_dalek`
   --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/boringtun-0.5.2/src/noise/mod.rs:295:39
    |
295 |         static_private: x25519_dalek::StaticSecret,
    |                                       ^^^^^^^^^^^^ help: a struct with a similar name exists: `SharedSecret`
    |
   ::: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/x25519-dalek-2.0.0-rc.3/src/x25519.rs:275:1
    |
275 | pub struct SharedSecret(pub(crate) MontgomeryPoint);
    | ----------------------- similarly named struct `SharedSecret` defined here

error[E0412]: cannot find type `StaticSecret` in crate `x25519_dalek`
   --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/boringtun-0.5.2/src/noise/mod.rs:331:39
    |
331 |         static_private: x25519_dalek::StaticSecret,
    |                                       ^^^^^^^^^^^^ help: a struct with a similar name exists: `SharedSecret`
    |
   ::: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/x25519-dalek-2.0.0-rc.3/src/x25519.rs:275:1
    |
275 | pub struct SharedSecret(pub(crate) MontgomeryPoint);
    | ----------------------- similarly named struct `SharedSecret` defined here

Some errors have detailed explanations: E0412, E0433.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `boringtun` (lib) due to 13 previous errors
error: failed to compile `boringtun-cli v0.5.2`, intermediate artifacts can be found at `/tmp/cargo-installqRD3Ni`

grishy · 2023-07-08T20:58:34Z

@Noah-Kennedy can you try to reproduce it?

Noah-Kennedy · 2023-07-08T21:24:01Z

Ignore v0.5.2, I need to yank it and put out a patch release due to an issue with how cargo resolves release candidate semver.

Noah-Kennedy · 2023-07-08T21:38:22Z

I've posted v0.5.3 now

grishy · 2023-07-09T07:21:11Z

Do we have CLI for v0.5.3?

> cargo install boringtun@0.5.3
    Updating crates.io index
error: there is nothing to install in `boringtun v0.5.3`, because it has no binaries
`cargo install` is only for installing programs, and can't be used with libraries.
To use a library crate, add it as a dependency to a Cargo project with `cargo add`.

grishy · 2023-07-09T07:29:10Z

I built from source - works great! Same servers as before.

sudo WG_QUICK_USERSPACE_IMPLEMENTATION=/root/boringtun/target/release/boringtun-cli WG_SUDO=1 wg-quick up wg0

> iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.8.0.2, port 39806
[  5] local 10.8.0.1 port 5201 connected to 10.8.0.2 port 39812
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  21.0 MBytes   176 Mbits/sec
[  5]   1.00-2.00   sec  23.1 MBytes   194 Mbits/sec
[  5]   2.00-3.00   sec  22.3 MBytes   187 Mbits/sec
[  5]   3.00-4.00   sec  21.7 MBytes   182 Mbits/sec
[  5]   4.00-5.00   sec  21.9 MBytes   184 Mbits/sec
[  5]   5.00-6.00   sec  21.3 MBytes   178 Mbits/sec
[  5]   6.00-7.00   sec  22.7 MBytes   191 Mbits/sec
[  5]   7.00-8.00   sec  23.1 MBytes   194 Mbits/sec
[  5]   8.00-9.00   sec  21.3 MBytes   179 Mbits/sec
[  5]   9.00-10.00  sec  21.8 MBytes   183 Mbits/sec
[  5]  10.00-10.04  sec   942 KBytes   177 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec   221 MBytes   185 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Without WG:

> iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.8.0.2, port 32888
[  5] local 10.8.0.1 port 5201 connected to 10.8.0.2 port 32894
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  21.1 MBytes   177 Mbits/sec
[  5]   1.00-2.00   sec  24.0 MBytes   201 Mbits/sec
[  5]   2.00-3.00   sec  22.7 MBytes   190 Mbits/sec
[  5]   3.00-4.00   sec  25.0 MBytes   210 Mbits/sec
[  5]   4.00-5.00   sec  24.3 MBytes   204 Mbits/sec
[  5]   5.00-6.00   sec  23.9 MBytes   201 Mbits/sec
[  5]   6.00-7.00   sec  24.1 MBytes   202 Mbits/sec
[  5]   7.00-8.00   sec  22.6 MBytes   189 Mbits/sec
[  5]   8.00-9.00   sec  21.2 MBytes   178 Mbits/sec
[  5]   9.00-10.00  sec  23.3 MBytes   196 Mbits/sec
[  5]  10.00-10.05  sec  1017 KBytes   181 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.05  sec   233 MBytes   195 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

grishy · 2023-07-09T15:36:37Z

So it's OK in v0.5.2

Noah-Kennedy · 2023-07-10T14:42:22Z

Do we have CLI for v0.5.3?

> cargo install boringtun@0.5.3
    Updating crates.io index
error: there is nothing to install in `boringtun v0.5.3`, because it has no binaries
`cargo install` is only for installing programs, and can't be used with libraries.
To use a library crate, add it as a dependency to a Cargo project with `cargo add`.

Oh, sorry, I should have mentioned, I pushed up a new library version, the cli will just use 0.5.3 since its the latest matching that semver.

Noah-Kennedy · 2023-07-10T14:42:54Z

So it's OK in v0.5.2

Interesting, I'll have a look at what changed

r7vme · 2023-09-15T08:25:02Z

I have similar issue with v0.6.0. With v0.6.0 I get 8.23 Mbits/sec vs v0.5.2 with ~ 1.10 Gbits/sec. Running on x84_64 KVM VMs with Ubuntu 22.04 (5.15.0-83-generic).

grishy · 2023-11-03T17:33:30Z

@Noah-Kennedy ping me if you want to recheck it with fixes 🙂

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slow work during testing on VM #352

Slow work during testing on VM #352

grishy commented Jul 8, 2023 •

edited

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023

Noah-Kennedy commented Jul 8, 2023 •

edited

Noah-Kennedy commented Jul 8, 2023

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023 •

edited

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023

Noah-Kennedy commented Jul 8, 2023 •

edited

Noah-Kennedy commented Jul 8, 2023

grishy commented Jul 9, 2023

grishy commented Jul 9, 2023 •

edited

grishy commented Jul 9, 2023

Noah-Kennedy commented Jul 10, 2023

Noah-Kennedy commented Jul 10, 2023

r7vme commented Sep 15, 2023

grishy commented Nov 3, 2023

Slow work during testing on VM #352

Slow work during testing on VM #352

Comments

grishy commented Jul 8, 2023 • edited

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023

Noah-Kennedy commented Jul 8, 2023 • edited

Noah-Kennedy commented Jul 8, 2023

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023 • edited

grishy commented Jul 8, 2023

grishy commented Jul 8, 2023

Noah-Kennedy commented Jul 8, 2023 • edited

Noah-Kennedy commented Jul 8, 2023

grishy commented Jul 9, 2023

grishy commented Jul 9, 2023 • edited

grishy commented Jul 9, 2023

Noah-Kennedy commented Jul 10, 2023

Noah-Kennedy commented Jul 10, 2023

r7vme commented Sep 15, 2023

grishy commented Nov 3, 2023

grishy commented Jul 8, 2023 •

edited

Noah-Kennedy commented Jul 8, 2023 •

edited

grishy commented Jul 8, 2023 •

edited

Noah-Kennedy commented Jul 8, 2023 •

edited

grishy commented Jul 9, 2023 •

edited