Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add TLS support to gnet #435

Open
wants to merge 49 commits into
base: dev
Choose a base branch
from
Open

feat: Add TLS support to gnet #435

wants to merge 49 commits into from

Commits on Jan 25, 2023

  1. Configuration menu
    Copy the full SHA
    697b56a View commit details
    Browse the repository at this point in the history
  2. 1. merge tls to go 1.20rc3 as close as possible

    2. change the gnet API name for the TLS server & client
    3. gnet TLS write returns the exact number of bytes
        written to the socket rather than the lenght of data.
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    2e073d2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fe87eeb View commit details
    Browse the repository at this point in the history
  4. Memory optimization: add the elastic wrapper EMsgBuffer

    to MsgBuffer so that the tls conn not longer holds the actual buffer
    when the connection is idle.
    
    Other updates:
    1. add defaultSize in MsgBuffer
    2. fix the condition to clean up the buffer
         (i > blockSize to i >= blockSize)
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    7c5336a View commit details
    Browse the repository at this point in the history
  5. Add kernel TLS support

    1. The kernel TLS implementation is based on
         https://github.com/jim3ma/go.git
         branch: dev.ktls.1.16.3
    2. Supports: TLS1.2 & TLS 1.3
    3. Supported cipher suites:
         AES_128_GCM_SHA256
         AES_256_GCM_SHA384
         CHACHA20_POLY1305_SHA256
    4. Server side has been tested and it works.
        Client side needs to be tested later
    5. TODO: add sendfile(), TLS_TX_ZEROCOPY_RO (device offload),
        and TLS_RX_EXPECT_NO_PAD. (See
        https://docs.kernel.org/networking/tls.html#optional-optimizations)
        for details.
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    3394893 View commit details
    Browse the repository at this point in the history
  6. Fix typos

    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    40e9536 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c7d0993 View commit details
    Browse the repository at this point in the history
  8. Add supports to TLS_TX_ZEROCOPY_RO and TLS_RX_EXPECT_NO_PAD,

    but not tested yet
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    582f146 View commit details
    Browse the repository at this point in the history
  9. bug: Fix KTLS readRecordOrCCS return EOF

    data should use the local declaration rather than re-declaring
    in the if statement, which results len(data) is 0 on line 794,
    resulting EOF.
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    29768bc View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    ee43463 View commit details
    Browse the repository at this point in the history
  11. Bug: Fix kTLS 1.3 RX not working on kernel 5.15

    =======================================
    1. disable kTLS 1.3 RX on kernel 5.15
    2. check zero copy on kernel 5.19
    3. check tls 1.3 no pad on kernel 6.0
    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    8e71e26 View commit details
    Browse the repository at this point in the history
  12. comment out dead code

    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    3e95281 View commit details
    Browse the repository at this point in the history
  13. update go version to 1.20

    0-haha committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    af39088 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. Configuration menu
    Copy the full SHA
    492f83e View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2023

  1. opt: TLS writes the data into the socket directly

    ======================================
    1. TLS writes the data into the socket directly rather than writing the
        data into the buffer. the data is buffered only if error unix.EAGAIN
        occurs.
    2. Add "tlsEnabled bool" to control when to use tlsconn.Write(). The
        reason is that tlsconn.Write() encrypt the data, then calls
        gnetConn.Write() which could potently call either gnetConn.write()
        or gnetConn.writeTLS(). Therefore, we make "tlsEnabled" to false
        before calling tlsconn.Write(), and then restore "tlsEnabled" to
        true after that.
    3. tlsconn.flush() calls gnetConn.Flush() to flush the buffer
        immediately. Therefore, we don't need to call gnetConn.Flush() in
        gnet TLS handshake phase as tlsconn.Handshake() calls
        gnetConn.Flush() implicitly.
    0-haha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    94ad7e8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    43bf39f View commit details
    Browse the repository at this point in the history
  3. opt: remove the dead code

    0-haha committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    76acc42 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. opt: zero-copy buffer in gnet TLS implementation

    ========================================
    Redesign the buffer in gnet TLS implementation
    to achieve zero-copy.
    
    Background:
    - tlsconn.rawInput: raw input from TCP to hold the TLS record
    - tlsconn.input: buffer to hold decrypted TLS record
    - tlsconn.hand: buffer to hold handshake data
    - tlsconn.sendBuf: buffer to hold sending data
    
    Problems:
    - Memory copy in TLS read:
      In the previous implementation, tlsconn.input refers
      to the gnetConn.inboundBuffer. To decrypted, we copy
      el.buffer to tlsconn.rawInput. The TLS connection,
      write the decrypted data to tlsconn.input, which is
      gnetConn.inboundBuffer. When el.eventHandler.OnTraffic()
      is triggered, gnetConn.Next() and gnet.Conn.Peek()
      can trigger more data copy as it can write to c.loop.cache()
    - Memory copy in TLS write:
      In the previous implementation, all encrypted data are
      first written to tlsconn.sendBuf, which refers to
      gnetConn.outboundBuffer. Then, tlsconn.Write() calls
      gnetConn.Write() which flushes the buffer to the socket
    
    New implementation:
    We designed LazyBuffer (lb) which has a buf []byte and its
    reference ref *[]byte. In the lazy mode, lb.ref is always nil,
    lb.buf is readonly. When calling lb.Write(), lb request a buffer
    from the sync.Pool, and copies lb.buf to the new buffer.
    Both lb.buf and lb.ref point to the new buffer.
    - New TLS read:
      With LazyBuffer, we let tlsconn.rawInput refer to el.buffer.
      Decrypted data stores in tlsconn.rawInput as well. tlsconn.Data()
      returns the reference of all decrypted data, and will be
      assigned to gnetConn.buffer.
    - New TLS write:
      tlsconn.Write() first encrypts the data, then calls
      gnetConn.WriteTCP() which directly writes the data
      to the socket.
    - New TLS handshake:
      we restore the tlsconn.Buffering flag which is only used in
      the handshake. Incoming handshake data is stored
      in tlsconn.hand and will be discarded immediately
      after being used. Outgoing handshake data is buffered
      in tlsconn.sendBuf, and will be flushed after calling
      tlsconn.flush() which calls gnetConn.WriteTCP() which
      directly writes the data to the socket.
    0-haha committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    c377ece View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d24fd00 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3f21522 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. Configuration menu
    Copy the full SHA
    d13ead1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b1b7bc5 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2023

  1. Configuration menu
    Copy the full SHA
    213300a View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2023

  1. crypto/tls: replace all usages of BytesOrPanic

    Message marshalling makes use of BytesOrPanic a lot, under the
    assumption that it will never panic. This assumption was incorrect, and
    specifically crafted handshakes could trigger panics. Rather than just
    surgically replacing the usages of BytesOrPanic in paths that could
    panic, replace all usages of it with proper error returns in case there
    are other ways of triggering panics which we didn't find.
    
    In one specific case, the tree routed by expandLabel, we replace the
    usage of BytesOrPanic, but retain a panic. This function already
    explicitly panicked elsewhere, and returning an error from it becomes
    rather painful because it requires changing a large number of APIs.
    The marshalling is unlikely to ever panic, as the inputs are all either
    fixed length, or already limited to the sizes required. If it were to
    panic, it'd likely only be during development. A close inspection shows
    no paths for a user to cause a panic currently.
    
    This patches ends up being rather large, since it requires routing
    errors back through functions which previously had no error returns.
    Where possible I've tried to use helpers that reduce the verbosity
    of frequently repeated stanzas, and to make the diffs as minimal as
    possible.
    
    Thanks to Marten Seemann for reporting this issue.
    
    Updates #58001
    Fixes #58359
    Fixes CVE-2022-41724
    
    Change-Id: Ieb55867ef0a3e1e867b33f09421932510cb58851
    Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1679436
    Reviewed-by: Julie Qiu <julieqiu@google.com>
    TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com>
    Run-TryBot: Roland Shoemaker <bracewell@google.com>
    Reviewed-by: Damien Neil <dneil@google.com>
    (cherry picked from commit 1d4e6ca9454f6cf81d30c5361146fb5988f1b5f6)
    Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1728205
    Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
    Reviewed-on: https://go-review.googlesource.com/c/go/+/468121
    Reviewed-by: Than McIntosh <thanm@google.com>
    Auto-Submit: Michael Pratt <mpratt@google.com>
    TryBot-Bypass: Michael Pratt <mpratt@google.com>
    Run-TryBot: Michael Pratt <mpratt@google.com>
    rolandshoemaker authored and 0-haha committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    e054d94 View commit details
    Browse the repository at this point in the history
  2. Fix: add missing ctx

    0-haha committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    2b05f32 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2023

  1. Configuration menu
    Copy the full SHA
    5217a6a View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2023

  1. Configuration menu
    Copy the full SHA
    f45a29f View commit details
    Browse the repository at this point in the history
  2. Fix golangci-lint

    0-haha committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    d4ab072 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a0bf9d9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    369338e View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. fix: make comments to english

    0-haha committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    0ccefca View commit details
    Browse the repository at this point in the history
  2. fix: typos in comments

    0-haha committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    37393e2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2705b62 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    bef64fa View commit details
    Browse the repository at this point in the history

Commits on May 21, 2023

  1. Configuration menu
    Copy the full SHA
    25c4638 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ccc7c28 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2023

  1. Configuration menu
    Copy the full SHA
    d35e196 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2023

  1. Configuration menu
    Copy the full SHA
    9a79add View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. Configuration menu
    Copy the full SHA
    f6206bb View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2023

  1. Configuration menu
    Copy the full SHA
    9b98998 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Configuration menu
    Copy the full SHA
    d25b6ab View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. Configuration menu
    Copy the full SHA
    9015fae View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. fix the typo

    0-haha committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    18c311d View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2023

  1. Configuration menu
    Copy the full SHA
    e174dc7 View commit details
    Browse the repository at this point in the history

Commits on Jan 1, 2024

  1. Configuration menu
    Copy the full SHA
    ecdf787 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2024

  1. Merge branch 'dev' into dev

    0-haha committed Mar 3, 2024
    Configuration menu
    Copy the full SHA
    6191b85 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. Configuration menu
    Copy the full SHA
    d78adc6 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Configuration menu
    Copy the full SHA
    7de6c58 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. Configuration menu
    Copy the full SHA
    01c9175 View commit details
    Browse the repository at this point in the history