Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the decoding issues series: BPE Tokenizer #1854

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Fix the decoding issues series: BPE Tokenizer #1854

wants to merge 15 commits into from
+1,658 −41

Commits on Feb 10, 2024

  1. bpe_tokenizer implementation

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    a29a3c8 View commit details
    Browse the repository at this point in the history

  2. Add files via upload

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    0806bc3 View commit details
    Browse the repository at this point in the history

  3. Update CMakeLists.txt

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    f8c8d49 View commit details
    Browse the repository at this point in the history

  4. Update Makefile

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    221d8d9 View commit details
    Browse the repository at this point in the history

  5. Add files via upload

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    14fef7c View commit details
    Browse the repository at this point in the history

  6. Reduce error rate

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    56a7a22 View commit details
    Browse the repository at this point in the history

  7. reduce error rate

    @bobqianic
    bobqianic committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    047ae5b View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
    @bobqianic @ggerganov
    bobqianic and ggerganov committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    99e5322 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. fix bugs

    @bobqianic
    bobqianic committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    0f6ad6c View commit details
    Browse the repository at this point in the history

  2. Add files via upload

    @bobqianic
    bobqianic committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    b668927 View commit details
    Browse the repository at this point in the history

  3. Add files via upload

    @bobqianic
    bobqianic committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    dac6533 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2024

  1. Add files via upload

    @bobqianic
    bobqianic committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    6eb97e9 View commit details
    Browse the repository at this point in the history

  2. rewrite bpe_gpt2_preprocess

    @bobqianic
    bobqianic committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    53d58fb View commit details
    Browse the repository at this point in the history

  3. Merge pull request #9 from bobqianic/fix 

    Fix
    @bobqianic
    bobqianic committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    b7bc969 View commit details
    Browse the repository at this point in the history

  4. Add files via upload

    @bobqianic
    bobqianic committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    9d3d1d2 View commit details
    Browse the repository at this point in the history