v0.9.0-beta

v0.9 🚀🚀🚀

We have got a big one for you! Today we are releasing version 0.9 of SpacetimeDB. It's been a while since our last official release and we have got a huge number changes and performance improvements.

Most notably performance has improved by 10x - 100x or more for certain parts of the database! Subscription evaluation in particular has gotten almost 100 times faster! This makes an enormous difference for people shipping production games. 🏃🏃🏃

Version 0.9 is the same version that we ran the Closed Alpha for the MMOPRG BitCraft, which had 700 players playing concurrently in a single database! We are battle testing SpacetimeDB with our production game!

We're so excited to see what people build with this new version!

As a reminder you can upgrade from the previous release with the spacetime upgrade command. If you are on macOS and you've installed through brew we recommend using brew install clockworklabs/tap/spacetime to upgrade.

Highlights

• Vastly improved metrics gathering performance overhead
• Stabilized new transaction log format for a 2-3x reduction in transaction log size and much faster reply times
• Reduced unnecessary copying and cloning in core database paths
• Improved threading model for optimal performance
• Fixed a critical issue for misbehaving clients or clients with very poor latency
• Multi-column index selection for the query! macro
• Huge number of critical performance and bug fixes!

What's Changed

• Bump Rust to 1.76.0 by @coolreader18 in #811
• test: Add performance benchmark for incremental join query by @joshua-spacetime in #795
• perf(subscriptions): Add metric counter for subscribe calls by @joshua-spacetime in #817
• ResultInspectExt is obsolete by @coolreader18 in #818
• chore(datastore): Remove debug instrumentation by @joshua-spacetime in #812
• Fix benchmarks by @coolreader18 in #819
• This fixes replaying of the transaction log to no longer check constraints by @cloutiertyler in #806
• ScanIterByColRange: Avoid unnecessary ProductValue allocations by @gefjon in #804
• Update upgrade-version tool by @coolreader18 in #826
• chore(836): Remove tracing instrumentation from database iterators by @joshua-spacetime in #837
• test(829): Benchmark incremental update for table subscription by @joshua-spacetime in #830
• Verify that the upgrade-version tool still works for every PR by @jdetter in #825
• To make eval subscription concurrent by @Shubham8287 in #807
• Remove unused pre-publish.duck file by @bfops in #841
• Fix: Publish fails if rustup not present by @bfops in #814
• test: Deduplicate utilities for creating tables by @joshua-spacetime in #842
• Add workflow to tag latest docker image on release by @drogus in #756
• test(835): Benchmark full subscription eval for table scans and join by @joshua-spacetime in #839
• Core: depend on "serde" from lib by @gefjon in #845
• Reduce allocations for prometheus with_label_values by @coolreader18 in #810
• fix: Wrong algebraic type in query benchmarks by @joshua-spacetime in #849
• fix: Record durations for aborted and read transactions by @joshua-spacetime in #850
• Fix publish issues 0.8.2 by @jdetter in #853
• Fix identity subcommand ordering by @jdetter in #857
• v0.8.2 Version Upgrade by @jdetter in #858
• core: Ignore indexed-only constraints when updating by @kim in #734
• Use a version label for tag builds by @drogus in #751
• Fix issues caused by disabling metrics by @jdetter in #856
• Add delete_by_{nonunique_column} method to #[spacetimedb(table)] by @gefjon in #859
• Fix deadlock by @coolreader18 in #854
• Fix bench errors and include in CI by @coolreader18 in #855
• Fix consistency issue by @coolreader18 in #861
• Refactor some ReadColumn stuff + relational_db tests by @Centril in #847
• Added extra check to create_table transactionality test by @cloutiertyler in #866
• system_tables, mut_tx, and friends: bye bye to_product_value by @Centril in #851
• Fix bootstrapping ids for relating objects like indexes by @mamcx in #777
• dedup some with_label_values + reduce cloning in bootstrapping by @Centril in #867
• Set name for Rayon threads by @gefjon in #870
• Adds an isolation_level argument to begin_mut_tx calls by @cloutiertyler in #864
• chore: Remove instrumentation from DataKey deserializers by @joshua-spacetime in #852
• Replace DbProgram with a leaner version that compile directly to queries without environment by @mamcx in #860
• Use a copy-on-write structure for RelValue by @Centril in #869
• Fix module import warnings by @coolreader18 in #820
• Add GitHub workflow to check for PRs with merge-blocking labels by @bfops in #862
• Script to run perf against SpacetimeDB by @gefjon in #882
• HostThreadPool: don't run Rayon threads in Tokio blocking threads by @gefjon in #881
• Rename types in benchmarks by @kazimuth in #803
• num_threads = available_concurrency by @coolreader18 in #887
• Metric for request RTT by @Shubham8287 in #872
• serialise disconnect logic by @Shubham8287 in #886
• Run initial subscriptions evals on rayon by @coolreader18 in #888
• Revert "Run initial subscriptions evals on rayon" by @cloutiertyler in #894
• C#: reuse Consume helper by @RReverser in #704
• refactor(890): Improve magic constant for index join selection by @joshua-spacetime in #895
• VM: Wrap Header in Arc to avoid cloning by @gefjon in #897
• Parallelize QuerySet::eval by @coolreader18 in #891
• Remove row_pk from client API; hash rows not row_ids on client by @gefjon in #840
• perf(832): Remove redundant row deduplication in subscriptions by @joshua-spacetime in #863
• acquire remove subscription lock in spawn_blocking by @Shubham8287 in #903
• Adjust benchmarks yaml to run callgrind + with better comments by @kazimuth in #907
• perf(831): Remove row_pk computation from query by @joshua-spacetime in #908
• core: Fix host controller to not replace module if lifecyle hooks failed by @kim in #904
• Correctly show the error for AmbiguousField and simplify the code by @mamcx in #910
• Adding an index selector that take in account multi-column indexes (and improve the query! macro) by @mamcx in #694
• Revert 694 by @jdetter in #914
• Fix the RowCount estimation for select operator by @mamcx in #900
• Remove Tables from query plans by @gefjon in #912
• iter_by_col_eq: take AV by ref + impl RangeBounds<..> for &AV by @Centril in #925
• perf(747): Single query execution for multiple subscriptions by @joshua-spacetime in #917
• Re-land mult-col index selection for queries by @Centril in #918
• Use recv_many in ws_client_actor by @coolreader18 in #913
• proto changes by @Shubham8287 in #924
• Single-table subscription queries: plan once, run repeatedly by @gefjon in #928
• Improve generate help text by @jdetter in #934
• Avoid Header::find_pos_by_name in eval_incr. by @gefjon in #935
• Kick clients that are backing up their message channel by @coolreader18 in #930
• chore: Remove debug instrumentation from hot path by @joshua-spacetime in #936
• Incremental joins: compile once, run repeatedly. by @gefjon in #938
• chore: Header does not need to be Ord by @joshua-spacetime in #942
• Remove perfcnt for now by @kazimuth in #941
• perf: Make op_type special case fast for selections by @joshua-spacetime in #943
• chore: Remove instrumentation from subscription eval hot path by @joshua-spacetime in #945
• perf: Update magic constant for join rewrite by @joshua-spacetime in #944
• Truly remove perfcnt by @mamcx in #946
• Also poll handle_queue while flushing by @coolreader18 in #947
• refactor: Incremental evaluation tests for index joins by @joshua-spacetime in #952
• Increase sdk tests' timeout to 60 seconds by @gefjon in #956
• fix: Incremental evaluation for index joins by @joshua-spacetime in #957
• test(954): Incremental evaluation for index joins by @joshua-spacetime in #958
• perf(816): Compile inner joins ahead of time for incremental evaluation by @joshua-spacetime in #964
• Fix bench for location, restore single-column indexes by @mamcx in #967
• Rewrite smoketests as python unittests by @coolreader18 in #692
• Code motion: QueryCode => QueryExpr; CrudCode => CrudExpr by @Centril in #975
• iter_filtered_chunks: avoid PVs by @Centril in #978
• Simplify RelOps<'a> for CatalogCursor<I> by @Centril in #979
• spacetime publish: Add --wasm-file flag by @bfops in #883
• Nix From<IndexScan> for ColumnOp & fix the multi-col bug by @Centril in #980
• Add a Testing section to the PR template by @bfops in #898
• Misc tweaks to tools/perf.sh by @bfops in #937
• fix BufReader for Cursor impl by @Centril in #982
• Remove #[tracing::instrument...] from InstanceEnv::*_by_col_eq by @Centril in #983
• core: Fix schema checks in database updates, again by @kim in #974
• Log a warning when doing iter_by_col_range without an index by @gefjon in #971
• Distinguish between inner and semijoins in QueryExpr AST. by @gefjon in #969
• perf(833): Do not clone query plan during execution by @Centril in #981
• perf(813): Avoid materialization of product values in subscriptions by @Centril in #959
• perf(933): Clone bsatn instead of product values in incremental update by @Centril in #951
• fix: Optimize query plan in iter_filtered_chunks by @Centril in #939
• Don't do alignment computations during BFLATN ser/de by @gefjon in #986
• Remove useless usages of RowCount by @mamcx in #987
• Nuke to_mem_table_with_op_type by @gefjon in #990
• incr-join, find_updates: avoid unncecessary clones & use partition by @Centril in #988
• test: Subscriptions should not drop read lock early by @joshua-spacetime in #995
• fix(996): Do not release database lock early for subscriptions by @joshua-spacetime in #997
• fix: Consistency test for subscription message ordering by @joshua-spacetime in #1007
• Show the error text of the server when a sql call fails on cli by @mamcx in #1004
• fix(1009): Multi-column index selection for query macro by @joshua-spacetime in #1012
• test: Multi-column index selection through query macro by @joshua-spacetime in #1001
• BFLATN -> BSATN fast-path for fixed-length rows by @gefjon in #1005
• Bump Rust to 1.77 + fix warnings + use Bound::map by @Centril in #1020
• perf(1024): Remove serialization from tx execution thread by @joshua-spacetime in #1027
• Add AlgebraicValue::take + move test-code in btree_index to tests by @Centril in #1028
• Split DatabaseTableUpdate in deletes/inserts vecs by @Centril in #1019
• Fixed an issue which caused metrics to only be recorded for on-disk databases by @cloutiertyler in #901
• Binary WebSocket API: Brotli-compress all outgoing messages by @gefjon in #1026
• (1/3) Commitlog: Base implementation "sans I/O" by @kim in #919
• (2/3) Commitlog: Add I/O based on regular files by @kim in #920
• (3/3) Commitlog: Add canonical txdata payload by @kim in #921
• Make 'op_type' an Enum by @mamcx in #1032
• Prune deps from bindings dependency tree by @coolreader18 in #1014
• Bump to reqwest 0.12 (uses hyper 1.0) by @coolreader18 in #1031
• Implement 'sql' parsing for simple 'Enums' by @mamcx in #1018
• C#: add experimental NativeAOT-LLVM support by @RReverser in #713
• Batch metrics with ctx by @Shubham8287 in #916
• Remove dead code in subscription by @Centril in #1039
• Consistently use --identity -i for identity args to CLI by @gefjon in #1041
• use nohasher_hash and ahash instead of siphash13 by @Centril in #1040
• Preserve debuginfo (e.g. symbols) in wasm-opt by @coolreader18 in #1013
• Reject large SQL queries by @gefjon in #1037
• IndexSemijoin::next: do a bit less cloning and work by @Centril in #1044
• Detect unsatisfiable range queries; warn and short-circuit. by @gefjon in #1036
• Metrics were not measuring the time it took to commit the tx by @cloutiertyler in #1045
• fix(1051): Reset queue length metrics on module restart by @joshua-spacetime in #1053
• fix: Disable unused metrics by @joshua-spacetime in #1056
• refactor(1058): Remove feature flag for disabling metrics by @joshua-spacetime in #1059
• Drop commitlog logging to trace to avoid spamming host logs by @gefjon in #1073
• disable iter metrics by @Shubham8287 in #1074
• Move lib::{name,recovery} to client-api-messages by @coolreader18 in #570
• core: Make init_module_host consistent with update_module_host by @kim in #977
• Durability: Traits and implementation in terms of commitlog by @kim in #922
• Implement 'sql' parsing for 'Identity, Address' in hex format by @mamcx in #786
• Swap the location of tags in the BFLATN encoding by @kazimuth in #1063
• core: Integrate new commitlog + durability by @kim in #926
• core: More compact TxData by @kim in #950
• core,commitlog: Re-instantiate commitlog disk usage reporting by @kim in #955

Full Changelog: v0.8.2-beta-hotfix7...v0.9.0-beta