GitHub - nalgeon/redka: Redis re-implemented with SQLite

Redka aims to reimplement the good parts of Redis with SQLite, while remaining compatible with Redis API.

Notable features:

Data does not have to fit in RAM.
ACID transactions.
SQL views for better introspection and reporting.
Both in-process (Go API) and standalone (RESP) servers.
Redis-compatible commands and wire protocol.

This is a work in progress. See below for the current status and roadmap.

Commands • Installation • Usage • Persistence • Performance • Roadmap • More

Commands

Redka aims to support five core Redis data types: strings, lists, sets, hashes, and sorted sets.

Strings

Strings are the most basic Redis type, representing a sequence of bytes. Redka supports the following string-related commands:

Command      Go API                 Description
-------      ------                 -----------
DECR         DB.Str().Incr          Decrements the integer value of a key by one.
DECRBY       DB.Str().Incr          Decrements a number from the integer value of a key.
GET          DB.Str().Get           Returns the value of a key.
GETSET       DB.Str().SetWith       Sets the key to a new value and returns the prev value.
INCR         DB.Str().Incr          Increments the integer value of a key by one.
INCRBY       DB.Str().Incr          Increments the integer value of a key by a number.
INCRBYFLOAT  DB.Str().IncrFloat     Increments the float value of a key by a number.
MGET         DB.Str().GetMany       Returns the values of one or more keys.
MSET         DB.Str().SetMany       Sets the values of one or more keys.
PSETEX       DB.Str().SetExpires    Sets the value and expiration time (in ms) of a key.
SET          DB.Str().Set           Sets the value of a key.
SETEX        DB.Str().SetExpires    Sets the value and expiration (in sec) time of a key.
SETNX        DB.Str().SetWith       Sets the value of a key when the key doesn't exist.

The following string-related commands are not planned for 1.0:

APPEND  GETDEL  GETEX  GETRANGE  LCS  MSETNX  SETRANGE  STRLEN  SUBSTR

Lists

Lists are lists of strings sorted by insertion order. Redka aims to support the following list-related commands in 1.0:

LINDEX  LINSERT  LLEN  LPOP  LPUSHX  LRANGE  LREM  LSET
LTRIM  RPOP  RPOPLPUSH  RPUSH  RPUSHX

Sets

Sets are unordered collections of unique strings. Redka aims to support the following set-related commands in 1.0:

SADD  SCARD  SDIFF  SDIFFSTORE  SINTER  SINTERSTORE
SISMEMBER  SMEMBERS  SMOVE  SPOP  SRANDMEMBER  SREM
SUNION  SUNIONSTORE

Hashes

Hashes are field-value (hash)maps. Redka supports the following hash-related commands:

Command       Go API                  Description
-------       ------------------      -----------
HDEL          DB.Hash().Delete        Deletes one or more fields and their values.
HEXISTS       DB.Hash().Exists        Determines whether a field exists.
HGET          DB.Hash().Get           Returns the value of a field.
HGETALL       DB.Hash().Items         Returns all fields and values.
HINCRBY       DB.Hash().Incr          Increments the integer value of a field.
HINCRBYFLOAT  DB.Hash().IncrFloat     Increments the float value of a field.
HKEYS         DB.Hash().Keys          Returns all fields.
HLEN          DB.Hash().Len           Returns the number of fields.
HMGET         DB.Hash().GetMany       Returns the values of multiple fields.
HMSET         DB.Hash().SetMany       Sets the values of multiple fields.
HSCAN         DB.Hash().Scanner       Iterates over fields and values.
HSET          DB.Hash().SetMany       Sets the values of one or more fields.
HSETNX        DB.Hash().SetNotExists  Sets the value of a field when it doesn't exist.
HVALS         DB.Hash().Exists        Returns all values.

The following hash-related commands are not planned for 1.0:

HRANDFIELD  HSTRLEN

Sorted sets

Sorted sets (zsets) are collections of unique strings ordered by each string's associated score. Redka supports the following sorted set related commands:

Command           Go API                  Description
-------           ------                  -----------
ZADD              DB.ZSet().AddMany       Adds or updates one or more members of a set.
ZCARD             DB.ZSet().Len           Returns the number of members in a set.
ZCOUNT            DB.ZSet().Count         Returns the number of members of a set within a range of scores.
ZINCRBY           DB.ZSet().Incr          Increments the score of a member in a set.
ZINTER            DB.ZSet().InterWith     Returns the intersection of multiple sets.
ZINTERSTORE       DB.ZSet().InterWith     Stores the intersection of multiple sets in a key.
ZRANGE            DB.ZSet().RangeWith     Returns members of a set within a range of indexes.
ZRANGEBYSCORE     DB.ZSet().RangeWith     Returns members of a set within a range of scores.
ZRANK             DB.ZSet().GetRank       Returns the index of a member in a set ordered by ascending scores.
ZREM              DB.ZSet().Delete        Removes one or more members from a set.
ZREMRANGEBYRANK   DB.ZSet().DeleteWith    Removes members of a set within a range of indexes.
ZREMRANGEBYSCORE  DB.ZSet().DeleteWith    Removes members of a set within a range of scores.
ZREVRANGE         DB.ZSet().RangeWith     Returns members of a set within a range of indexes in reverse order.
ZREVRANGEBYSCORE  DB.ZSet().RangeWith     Returns members of a set within a range of scores in reverse order.
ZREVRANK          DB.ZSet().GetRankRev    Returns the index of a member in a set ordered by descending scores.
ZSCAN             DB.ZSet().Scan          Iterates over members and scores of a set.
ZSCORE            DB.ZSet().GetScore      Returns the score of a member in a set.
ZUNION            DB.ZSet().UnionWith     Returns the union of multiple sets.
ZUNIONSTORE       DB.ZSet().UnionWith     Stores the union of multiple sets in a key.

The following sorted set related commands are not planned for 1.0:

BZMPOP  BZPOPMAX  BZPOPMIN  ZDIFF  ZDIFFSTORE  ZINTERCARD
ZLEXCOUNT  ZMPOP  ZMSCORE  ZPOPMAX  ZPOPMIN  ZRANDMEMBER
ZRANGEBYLEX  ZRANGESTORE  ZREMRANGEBYLEX  ZREVRANGEBYLEX

Key management

Redka supports the following key management (generic) commands:

Command    Go API                    Description
-------    ------                    -----------
DEL        DB.Key().Delete           Deletes one or more keys.
EXISTS     DB.Key().Count            Determines whether one or more keys exist.
EXPIRE     DB.Key().Expire           Sets the expiration time of a key (in seconds).
EXPIREAT   DB.Key().ExpireAt         Sets the expiration time of a key to a Unix timestamp.
KEYS       DB.Key().Keys             Returns all key names that match a pattern.
PERSIST    DB.Key().Persist          Removes the expiration time of a key.
PEXPIRE    DB.Key().Expire           Sets the expiration time of a key in ms.
PEXPIREAT  DB.Key().ExpireAt         Sets the expiration time of a key to a Unix ms timestamp.
RANDOMKEY  DB.Key().Random           Returns a random key name from the database.
RENAME     DB.Key().Rename           Renames a key and overwrites the destination.
RENAMENX   DB.Key().RenameNotExists  Renames a key only when the target key name doesn't exist.
SCAN       DB.Key().Scanner          Iterates over the key names in the database.

The following generic commands are not planned for 1.0:

COPY  DUMP  EXPIRETIME  MIGRATE  MOVE  OBJECT  PEXPIRETIME
PTTL  RESTORE  SORT  SORT_RO  TOUCH  TTL  TYPE  UNLINK
WAIT  WAITAOF

Transactions

Redka supports the following transaction commands:

Command    Go API                 Description
-------    ------                 -----------
DISCARD    DB.View / DB.Update    Discards a transaction.
EXEC       DB.View / DB.Update    Executes all commands in a transaction.
MULTI      DB.View / DB.Update    Starts a transaction.

Unlike Redis, Redka's transactions are fully ACID, providing automatic rollback in case of failure.

The following transaction commands are not planned for 1.0:

UNWATCH  WATCH

Server/connection management

Redka supports only a couple of server and connection management commands:

Command    Go API                Description
-------    ------                -----------
ECHO       -                     Returns the given string.
FLUSHDB    DB.Key().DeleteAll    Remove all keys from the database.

The rest of the server and connection management commands are not planned for 1.0.

Installation

Redka can be installed as a standalone Redis-compatible server, or as a Go module for in-process use.

Standalone server

Redka server is a single-file binary. Download it from the releases.

Linux (x86 CPU only):

curl -L -O "https://github.com/nalgeon/redka/releases/download/v0.3.0/redka_linux_amd64.zip"
unzip redka_linux_amd64.zip
chmod +x redka

macOS (both x86 and ARM/Apple Silicon CPU):

curl -L -O "https://github.com/nalgeon/redka/releases/download/v0.3.0/redka_darwin_amd64.zip"
unzip redka_darwin_amd64.zip
# remove the build from quarantine
# (macOS disables unsigned binaries)
xattr -d com.apple.quarantine redka
chmod +x redka

Or pull with Docker as follows (x86/ARM):

docker pull nalgeon/redka

Or build from source (requires Go 1.22 and GCC):

git clone https://github.com/nalgeon/redka.git
cd redka
make setup build
# the path to the binary after the build
# will be ./build/redka

Go module

Install the module as follows:

go get github.com/nalgeon/redka

You'll also need an SQLite driver. Use github.com/mattn/go-sqlite3 if you don't mind CGO. Otherwise use a pure Go driver modernc.org/sqlite. Install either with go get like this:

go get github.com/mattn/go-sqlite3

Usage

Redka can be used as a standalone Redis-compatible server, or as an embeddable in-process server with Go API.

Standalone server

Redka server is a single-file binary. After downloading and unpacking the release asset, run it as follows:

redka [-h host] [-p port] [db-path]

For example:

./redka
./redka data.db
./redka -h 0.0.0.0 -p 6379 data.db

Server defaults are host localhost, port 6379 and empty DB path.

Running without a DB path creates an in-memory database. The data is not persisted in this case, and will be gone when the server is stopped.

You can also run Redka with Docker as follows:

# database inside the container
# will be lost when the container stops
docker run --rm -p 6379:6379 nalgeon/redka

# persistent database
# using the /path/to/data host directory
docker run --rm -p 6379:6379 -v /path/to/data:/data nalgeon/redka

# in-memory database, custom post
docker run --rm -p 6380:6380 nalgeon/redka redka -h 0.0.0.0 -p 6380

Server defaults in Docker are host 0.0.0.0, port 6379 and DB path /data/redka.db.

Once the server is running, connect to it using redis-cli or an API client like redis-py or go-redis — just as you would with Redis.

redis-cli -h localhost -p 6379

127.0.0.1:6379> echo hello
"hello"
127.0.0.1:6379> set name alice
OK
127.0.0.1:6379> get name
"alice"

In-process server

The primary object in Redka is the DB. To open or create your database, use the redka.Open() function:

package main

import (
    "log"

    _ "github.com/mattn/go-sqlite3"
    "github.com/nalgeon/redka"
)

func main() {
    // Open or create the data.db file.
    db, err := redka.Open("data.db", nil)
    if err != nil {
        log.Fatal(err)
    }
    // Always close the database when you are finished.
    defer db.Close()
    // ...
}

Don't forget to import the driver (here I use github.com/mattn/go-sqlite3). Using modernc.org/sqlite is slightly different, see example/modernc/main.go for details.

To open an in-memory database that doesn't persist to disk, use the following path:

// All data is lost when the database is closed.
redka.Open("file:redka?mode=memory&cache=shared")

After opening the database, call redka.DB methods to run individual commands:

db.Str().Set("name", "alice")
db.Str().Set("age", 25)

count, err := db.Key().Count("name", "age", "city")
slog.Info("count", "count", count, "err", err)

name, err := db.Str().Get("name")
slog.Info("get", "name", name, "err", err)

count count=2 err=<nil>
get name="alice" err=<nil>

See the full example in example/simple/main.go.

Use transactions to batch commands. There are View (read-only transaction) and Update (writable transaction) methods for this:

updCount := 0
err := db.Update(func(tx *redka.Tx) error {
    err := tx.Str().Set("name", "bob")
    if err != nil {
        return err
    }
    updCount++

    err = tx.Str().Set("age", 50)
    if err != nil {
        return err
    }
    updCount++
    return nil
})
slog.Info("updated", "count", updCount, "err", err)

updated count=2 err=<nil>

See the full example in example/tx/main.go.

See the package documentation for API reference.

Persistence

Redka stores data in a SQLite database using the following tables:

rkey
---
id       integer primary key
key      text not null
type     integer not null    -- 1 string, 2 list, 3 set, 4 hash, 5 sorted set
version  integer not null    -- incremented when the key value is updated
etime    integer             -- expiration timestamp in unix milliseconds
mtime    integer not null    -- modification timestamp in unix milliseconds

rstring
---
key_id   integer not null    -- FK -> rkey.id
value    blob not null

rhash
---
key_id   integer not null    -- FK -> rkey.id
field    text not null
value    blob not null

rzset
---
key_id   integer not null    -- FK -> rkey.id
elem     blob not null
score    real not null

To access the data with SQL, use views instead of tables:

select * from vstring;

┌────────┬──────┬───────┬───────┬─────────────────────┐
│ key_id │ key  │ value │ etime │        mtime        │
├────────┼──────┼───────┼───────┼─────────────────────┤
│ 1      │ name │ alice │       │ 2024-04-03 16:58:14 │
│ 2      │ age  │ 50    │       │ 2024-04-03 16:34:52 │
└────────┴──────┴───────┴───────┴─────────────────────┘

etime and mtime are in UTC.

There is a separate view for every data type:

vstring  vhash  vzset

Performance

I've compared Redka with Redis using redis-benchmark with the following parameters:

10 parallel connections
1000000 requests
10000 randomized keys
GET/SET commands

SQLite settings:

pragma journal_mode = wal;
pragma synchronous = normal;
pragma temp_store = memory;
pragma mmap_size = 268435456;
pragma foreign_keys = on;

Hardware: Apple M1 8-core CPU, 16GB RAM

Redis:

redis-server --appendonly no
redis-benchmark -p 6379 -q -c 10 -n 1000000 -r 10000 -t get,set

SET: 133262.25 requests per second, p50=0.055 msec
GET: 139217.59 requests per second, p50=0.055 msec

Redka (in-memory):

./redka -p 6380
redis-benchmark -p 6380 -q -c 10 -n 1000000 -r 10000 -t get,set

SET: 30084.24 requests per second, p50=0.255 msec
GET: 63011.97 requests per second, p50=0.103 msec

Redka (persisted to disk):

./redka -p 6380 data.db
redis-benchmark -p 6380 -q -c 10 -n 1000000 -r 10000 -t get,set

SET: 21913.01 requests per second, p50=0.335 msec
GET: 56795.59 requests per second, p50=0.119 msec

So while Redka is 2-6 times slower than Redis (not surprising, since we are comparing a relational database to a key-value data store), it can still do 22K writes/sec and 57K reads/sec, which is pretty good if you ask me.

Note that running in a container may result in poorer performance.

Roadmap

The project is on its way to 1.0.

The 1.0 release will include the following features:

✅ Strings.
⏳ Lists.
⬜ Sets.
✅ Hashes.
✅ Sorted sets.
✅ Key management.
✅ Transactions.

✅ = done, ⏳ = in progress, ⬜ = next in line

Future versions may include additional data types (such as streams, HyperLogLog or geo), features like publish/subscribe, and more commands for existing types.

Features I'd rather not implement even in future versions:

Lua scripting.
Authentication and ACLs.
Multiple databases.
Watch/unwatch.

Features I definitely don't want to implement:

Cluster.
Sentinel.

More information

Contributing

Contributions are welcome. For anything other than bugfixes, please first open an issue to discuss what you want to change.

Be sure to add or update tests as appropriate.

Acknowledgements

Redka would not be possible without these great projects and their creators:

Redis (Salvatore Sanfilippo). It's such an amazing idea to go beyond the get-set paradigm and provide a convenient API for more complex data structures.
SQLite (D. Richard Hipp). The in-process database powering the world.
Redcon (Josh Baker). A very clean and convenient implementation of a RESP server.

Logo font by Ek Type.

License

The software is available under the BSD-3-Clause license.

Stay tuned

★ Subscribe to stay on top of new features.

Name		Name	Last commit message	Last commit date
Latest commit History 137 Commits
.github/workflows		.github/workflows
cmd		cmd
example		example
internal		internal
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
logo.svg		logo.svg
redka.go		redka.go
redka_test.go		redka_test.go

License

nalgeon/redka

Folders and files

Latest commit

History

Repository files navigation

Commands

Strings

Lists

Sets

Hashes

Sorted sets

Key management

Transactions

Server/connection management

Installation

Standalone server

Go module

Usage

Standalone server

In-process server

Persistence

Performance

Roadmap

More information

Contributing

Acknowledgements

License

Stay tuned

About

Topics

Resources

License

Stars

Watchers

Forks

Languages