v0.2.2

Key Changes

• Enable observability in JetStream Server (prometheus metrics)
• Enable JAX profiler support on single-host JetStream Server
• Support both text and token ids I/O for JetStream Decode API
• Add health check API
• Support MLPerf evaluation
• Enable JetStream Server E2E tests
• Increase unit test coverage (>=96%)

What's Changed

• Accuracy eval mlperf by @jwyang-google in #76
• Add metadata metrics by @yeandy in #77
• Fix pad_tokens function description by @FanhaiLu1 in #80
• Prometheus Metrics by @Bslabe123 in #71
• Update JetStream grpc proto to support I/O with text and token ids by @JoeZijunZhou in #78
• Update benchmark script to easily test llama-3 by @bhavya01 in #83
• Unit test coverage cleanup by @JoeZijunZhou in #81
• Allow tokenizer to customize stop_tokens by @qihqi in #84
• Decode Batch Percentage Metrics/Improved Scraping by @Bslabe123 in #82
• Bump requests from 2.31.0 to 2.32.0 in the pip group across 1 directory by @dependabot in #86
• Add profiling support and update docs by @JoeZijunZhou in #85
• Add ray disaggregated serving support by @FanhaiLu1 in #87
• Ensure server warmup before benchmark by @JoeZijunZhou in #91
• Add healthcheck support for JetStream by @vivianrwu in #90
• Add JetStream E2E test CI by @JoeZijunZhou in #89
• Release v0.2.2 by @JoeZijunZhou in #95

New Contributors

• @jwyang-google made their first contribution in #76
• @Bslabe123 made their first contribution in #71
• @vivianrwu made their first contribution in #90

Full Changelog: v0.2.1...v0.2.2

v0.2.1

Key Changes

• Support Llama3 tokenizer
• JetStream Tokenizer refactor
• Disaggregation preparation work

What's Changed

• add sample_idx in InputRequest for debugging by @morgandu in #32
• Update README.md with user guides by @JoeZijunZhou in #34
• Update README.md with PT user guide by @JoeZijunZhou in #35
• Reorganize unit tests and update CICD by @JoeZijunZhou in #37
• Add badges for JetStream by @JoeZijunZhou in #38
• Bump idna from 3.6 to 3.7 by @dependabot in #39
• Reformat benchmark metrics by @yeandy in #42
• Update server host default value by @JoeZijunZhou in #43
• Refactor readme by @FanhaiLu1 in #41
• Add missing Documentation by @FanhaiLu1 in #47
• Update README.md to fix broken link by @charbull in #50
• Add np padded token support by @FanhaiLu1 in #49
• Format token utils and test by @FanhaiLu1 in #51
• Align Tokenizer in JetStream by @JoeZijunZhou in #40
• Do nothing for nd array in copy_to_host_async by @FanhaiLu1 in #52
• Add jax_padding support driver and server lib by @FanhaiLu1 in #54
• Update maxtext user guide by @JoeZijunZhou in #56
• Fix benchmark script type issue by @JoeZijunZhou in #59
• Fix requester flag default value by @JoeZijunZhou in #60
• Fix float division by zero in benchmark by @FanhaiLu1 in #62
• Register IFRT proxy backend when proxy is defined in the jax_platforms by @zhihaoshan-google in #63
• Add an abstract class for Tokenizer by @bhavya01 in #53
• refactor slice_to_num_chips to adapt to Cloud config by @zhihaoshan-google in #65
• Support llama3 tokenizer by @bhavya01 in #67
• Prerequisite work for supporting disaggregation: by @zhihaoshan-google in #68
• Create init.py in Jetstream/third_party by @bhavya01 in #69
• Add tokenize_and_pad function to backward compatible by @FanhaiLu1 in #70
• Release v0.2.1 by @JoeZijunZhou in #72
• Bump tqdm from 4.66.1 to 4.66.3 in the pip group across 1 directory by @dependabot in #73
• Release v0.2.1 with docs update by @JoeZijunZhou in #74

New Contributors

• @dependabot made their first contribution in #39
• @yeandy made their first contribution in #42
• @charbull made their first contribution in #50
• @zhihaoshan-google made their first contribution in #63
• @bhavya01 made their first contribution in #53

Full Changelog: v0.2.0...v0.2.1

v0.2.0

Major Changes

• Support JetStream MaxText inference on Cloud TPU VM
• Support JetStream Pytorch inference on Cloud TPU VM
• Support Continuous Batching with interleaved mode in JetStream
• Support online serving benchmarking

What's Changed

• Add unit tests CI github action by @JoeZijunZhou in #1
• Refine thread in orchestrator by @JoeZijunZhou in #2
• Optimize maximum threads to saturate decoding capacity by @JoeZijunZhou in #3
• Add benchmarks maximum threads config by @JoeZijunZhou in #4
• First support necessary for MaxText by @rwitten in #5
• Support gracefully stopping orchestrator and server by @JoeZijunZhou in #6
• Save request outputs and add eval accuracy support by @FanhaiLu1 in #8
• Use parameter based num as inference request max output length by @FanhaiLu1 in #10
• Fix output token drop issue by @JoeZijunZhou in #9
• Add option to warm up by @qihqi in #11
• Replace token_list with generated_text in saved outputs by @FanhaiLu1 in #12
• Refine requester util by @JoeZijunZhou in #15
• Adds filtering for sharegpt based on conversation starter. by @patemotter in #17
• Allows more requests than available data. by @patemotter in #19
• Fix starvation with async server and interleaving optimization by @JoeZijunZhou in #13
• Add Token util unit test by @FanhaiLu1 in #20
• Fix llama2 decode bug in tokenizer by @FanhaiLu1 in #22
• Fix whitespace replacement bug by @FanhaiLu1 in #24
• Update benchmark to run openorca dataset by @morgandu in #21
• Add model ckpt conversion and AQT scripts for JetStream MaxText Serving by @JoeZijunZhou in #23
• Refactor to sample before tokenize by @morgandu in #26
• Update ckpt conversion scripts by @JoeZijunZhou in #25
• move tokenizer model to third party llama2 by @FanhaiLu1 in #27
• Support JetStream MaxText user guide by @JoeZijunZhou in #28
• Enable pylint linter and pyink formatter by @JoeZijunZhou in #29
• Update README by @JoeZijunZhou in #30
• Release v0.2.0 by @JoeZijunZhou in #31

New Contributors

• @JoeZijunZhou made their first contribution in #1
• @rwitten made their first contribution in #5
• @FanhaiLu1 made their first contribution in #8
• @qihqi made their first contribution in #11
• @patemotter made their first contribution in #17
• @morgandu made their first contribution in #21

Full Changelog: https://github.com/google/JetStream/commits/v0.2.0