From 20f54e34d8356afc50e3def8c0d83d873c9a9759 Mon Sep 17 00:00:00 2001 From: "hinto.janai" Date: Tue, 3 Sep 2024 19:36:10 -0400 Subject: [PATCH] db abstraction --- books/architecture/book.toml | 2 +- books/architecture/src/SUMMARY.md | 39 ++++++++++--------- .../{blockchain.md => blockchain/intro.md} | 0 .../src/storage/blockchain/schema/intro.md | 1 + .../src/storage/blockchain/schema/multimap.md | 1 + .../src/storage/blockchain/schema/tables.md | 1 + .../src/storage/blockchain/types.md | 1 + .../src/storage/common/initialization.md | 1 + .../architecture/src/storage/common/intro.md | 1 + books/architecture/src/storage/common/ops.md | 1 + .../src/storage/common/requests.md | 1 + .../src/storage/common/responses.md | 1 + .../src/storage/common/shutdown.md | 1 + .../src/storage/common/thread-model.md | 1 + .../architecture/src/storage/common/types.md | 1 + .../src/storage/database-abstraction.md | 1 - books/architecture/src/storage/db/backends.md | 1 + .../src/storage/db/common/initialization.md | 1 + .../src/storage/db/common/intro.md | 1 + .../architecture/src/storage/db/common/ops.md | 1 + .../src/storage/db/common/requests.md | 1 + .../src/storage/db/common/responses.md | 1 + .../src/storage/db/common/shutdown.md | 1 + .../src/storage/db/common/thread-model.md | 1 + .../src/storage/db/common/types.md | 1 + .../src/storage/db/database-abstraction.md | 21 ++++++++++ .../src/storage/db/issues/extra.md | 1 + .../src/storage/db/issues/unaligned.md | 1 + .../src/storage/db/layers/types.md | 1 + books/architecture/src/storage/intro.md | 6 +-- .../architecture/src/storage/txpool/intro.md | 1 + 31 files changed, 70 insertions(+), 24 deletions(-) rename books/architecture/src/storage/{blockchain.md => blockchain/intro.md} (100%) create mode 100644 books/architecture/src/storage/blockchain/schema/intro.md create mode 100644 books/architecture/src/storage/blockchain/schema/multimap.md create mode 100644 books/architecture/src/storage/blockchain/schema/tables.md create mode 100644 books/architecture/src/storage/blockchain/types.md create mode 100644 books/architecture/src/storage/common/initialization.md create mode 100644 books/architecture/src/storage/common/intro.md create mode 100644 books/architecture/src/storage/common/ops.md create mode 100644 books/architecture/src/storage/common/requests.md create mode 100644 books/architecture/src/storage/common/responses.md create mode 100644 books/architecture/src/storage/common/shutdown.md create mode 100644 books/architecture/src/storage/common/thread-model.md create mode 100644 books/architecture/src/storage/common/types.md delete mode 100644 books/architecture/src/storage/database-abstraction.md create mode 100644 books/architecture/src/storage/db/backends.md create mode 100644 books/architecture/src/storage/db/common/initialization.md create mode 100644 books/architecture/src/storage/db/common/intro.md create mode 100644 books/architecture/src/storage/db/common/ops.md create mode 100644 books/architecture/src/storage/db/common/requests.md create mode 100644 books/architecture/src/storage/db/common/responses.md create mode 100644 books/architecture/src/storage/db/common/shutdown.md create mode 100644 books/architecture/src/storage/db/common/thread-model.md create mode 100644 books/architecture/src/storage/db/common/types.md create mode 100644 books/architecture/src/storage/db/database-abstraction.md create mode 100644 books/architecture/src/storage/db/issues/extra.md create mode 100644 books/architecture/src/storage/db/issues/unaligned.md create mode 100644 books/architecture/src/storage/db/layers/types.md create mode 100644 books/architecture/src/storage/txpool/intro.md diff --git a/books/architecture/book.toml b/books/architecture/book.toml index 996f7fe..a1c8f7f 100644 --- a/books/architecture/book.toml +++ b/books/architecture/book.toml @@ -14,4 +14,4 @@ renderer = ["html"] default-theme = "ayu" preferred-dark-theme = "ayu" git-repository-url = "https://github.com/Cuprate/architecture-book" -additional-css = ["last-changed.css"] +additional-css = ["last-changed.css"] \ No newline at end of file diff --git a/books/architecture/src/SUMMARY.md b/books/architecture/src/SUMMARY.md index a29ce90..5dbda3b 100644 --- a/books/architecture/src/SUMMARY.md +++ b/books/architecture/src/SUMMARY.md @@ -28,8 +28,8 @@ --- - [🟒 Storage](storage/intro.md) - - [🟒 Database abstraction](storage/db/intro.md) - - [🟒 Backends](storage/db/backends/intro.md) + - [🟒 Database abstraction](storage/db/database-abstraction.md) + - [🟒 Backends](storage/db/backends.md) - [🟒 Layers](storage/db/layers/intro.md) - [🟒 Backend](storage/db/layers/backend.md) - [🟒 ConcreteEnv](storage/db/layers/concrete_env.md) @@ -37,25 +37,26 @@ - [🟒 Syncing](storage/db/syncing.md) - [🟒 Resizing](storage/db/resizing.md) - [🟒 (De)serialization](storage/db/serde.md) - - [🟒 Schema](storage/db/schema/intro.md) - - [🟒 Tables](storage/db/schema/tables.md) - - [🟒 Multimap tables](storage/db/schema/multimap.md) - [🟒 Known issues and tradeoffs](storage/db/issues/intro.md) - - [🟒 Traits abstracting backends](storage/db/issues/traits.md) - - [🟒 Hot-swappable backends](storage/db/issues/hot-swap.md) - - [🟒 Copying unaligned bytes](storage/db/issues/copy.md) + - [🟒 Abstracting backends](storage/db/issues/traits.md) + - [🟒 Hot-swap](storage/db/issues/hot-swap.md) + - [🟒 Unaligned bytes](storage/db/issues/unaligned.md) - [🟒 Endianness](storage/db/issues/endian.md) - - [🟒 Extra table data](storage/db/issues/extra-table-data.md) - - [🟒 The layers](storage/db/layers/intro.md) - - [🟒 `ops`](storage/db/layers/ops.md) - - [🟒 `tower::Service`](storage/db/service/intro.md) - - [🟒 Initialization](storage/db/service/initialization.md) - - [🟒 Requests](storage/db/service/requests.md) - - [🟒 Responses](storage/db/service/responses.md) - - [🟒 Thread model](storage/db/service/thread-model.md) - - [🟒 Shutdown](storage/db/service/shutdown.md) - - [βšͺ️ Blockchain](storage/blockchain.md) - - [βšͺ️ Transaction pool](storage/transaction-pool.md) + - [🟒 Extra table data](storage/db/issues/extra.md) + - [🟒 Common behavior](storage/common/intro.md) + - [🟒 Types](storage/common/types.md) + - [🟒 `ops`](storage/common/ops.md) + - [🟒 `tower::Service`](storage/common/intro.md) + - [🟒 Initialization](storage/common/initialization.md) + - [🟒 Requests](storage/common/requests.md) + - [🟒 Responses](storage/common/responses.md) + - [🟒 Thread model](storage/common/thread-model.md) + - [🟒 Shutdown](storage/common/shutdown.md) + - [βšͺ️ Blockchain](storage/blockchain/intro.md) + - [🟒 Schema](storage/blockchain/schema/intro.md) + - [🟒 Tables](storage/blockchain/schema/tables.md) + - [🟒 Multimap tables](storage/blockchain/schema/multimap.md) + - [βšͺ️ Transaction pool](storage/txpool/intro.md) - [βšͺ️ Pruning](storage/pruning.md) --- diff --git a/books/architecture/src/storage/blockchain.md b/books/architecture/src/storage/blockchain/intro.md similarity index 100% rename from books/architecture/src/storage/blockchain.md rename to books/architecture/src/storage/blockchain/intro.md diff --git a/books/architecture/src/storage/blockchain/schema/intro.md b/books/architecture/src/storage/blockchain/schema/intro.md new file mode 100644 index 0000000..02715fa --- /dev/null +++ b/books/architecture/src/storage/blockchain/schema/intro.md @@ -0,0 +1 @@ +# 🟒 Schema diff --git a/books/architecture/src/storage/blockchain/schema/multimap.md b/books/architecture/src/storage/blockchain/schema/multimap.md new file mode 100644 index 0000000..52657cd --- /dev/null +++ b/books/architecture/src/storage/blockchain/schema/multimap.md @@ -0,0 +1 @@ +# 🟒 Multimap tables diff --git a/books/architecture/src/storage/blockchain/schema/tables.md b/books/architecture/src/storage/blockchain/schema/tables.md new file mode 100644 index 0000000..dd1fe4d --- /dev/null +++ b/books/architecture/src/storage/blockchain/schema/tables.md @@ -0,0 +1 @@ +# 🟒 Tables diff --git a/books/architecture/src/storage/blockchain/types.md b/books/architecture/src/storage/blockchain/types.md new file mode 100644 index 0000000..ba238d6 --- /dev/null +++ b/books/architecture/src/storage/blockchain/types.md @@ -0,0 +1 @@ +# 🟒 Types diff --git a/books/architecture/src/storage/common/initialization.md b/books/architecture/src/storage/common/initialization.md new file mode 100644 index 0000000..59f994c --- /dev/null +++ b/books/architecture/src/storage/common/initialization.md @@ -0,0 +1 @@ +# 🟒 Initialization diff --git a/books/architecture/src/storage/common/intro.md b/books/architecture/src/storage/common/intro.md new file mode 100644 index 0000000..b29e971 --- /dev/null +++ b/books/architecture/src/storage/common/intro.md @@ -0,0 +1 @@ +# 🟒 Common behavior diff --git a/books/architecture/src/storage/common/ops.md b/books/architecture/src/storage/common/ops.md new file mode 100644 index 0000000..2bfd3f9 --- /dev/null +++ b/books/architecture/src/storage/common/ops.md @@ -0,0 +1 @@ +# 🟒 ops diff --git a/books/architecture/src/storage/common/requests.md b/books/architecture/src/storage/common/requests.md new file mode 100644 index 0000000..7391375 --- /dev/null +++ b/books/architecture/src/storage/common/requests.md @@ -0,0 +1 @@ +# 🟒 Requests diff --git a/books/architecture/src/storage/common/responses.md b/books/architecture/src/storage/common/responses.md new file mode 100644 index 0000000..3ba68e0 --- /dev/null +++ b/books/architecture/src/storage/common/responses.md @@ -0,0 +1 @@ +# 🟒 Responses diff --git a/books/architecture/src/storage/common/shutdown.md b/books/architecture/src/storage/common/shutdown.md new file mode 100644 index 0000000..cfe83db --- /dev/null +++ b/books/architecture/src/storage/common/shutdown.md @@ -0,0 +1 @@ +# 🟒 Shutdown diff --git a/books/architecture/src/storage/common/thread-model.md b/books/architecture/src/storage/common/thread-model.md new file mode 100644 index 0000000..b2addfc --- /dev/null +++ b/books/architecture/src/storage/common/thread-model.md @@ -0,0 +1 @@ +# 🟒 Thread model diff --git a/books/architecture/src/storage/common/types.md b/books/architecture/src/storage/common/types.md new file mode 100644 index 0000000..ba238d6 --- /dev/null +++ b/books/architecture/src/storage/common/types.md @@ -0,0 +1 @@ +# 🟒 Types diff --git a/books/architecture/src/storage/database-abstraction.md b/books/architecture/src/storage/database-abstraction.md deleted file mode 100644 index b21a192..0000000 --- a/books/architecture/src/storage/database-abstraction.md +++ /dev/null @@ -1 +0,0 @@ -# βšͺ️ Database abstraction diff --git a/books/architecture/src/storage/db/backends.md b/books/architecture/src/storage/db/backends.md new file mode 100644 index 0000000..b2ec6e4 --- /dev/null +++ b/books/architecture/src/storage/db/backends.md @@ -0,0 +1 @@ +# 🟒 Backends diff --git a/books/architecture/src/storage/db/common/initialization.md b/books/architecture/src/storage/db/common/initialization.md new file mode 100644 index 0000000..59f994c --- /dev/null +++ b/books/architecture/src/storage/db/common/initialization.md @@ -0,0 +1 @@ +# 🟒 Initialization diff --git a/books/architecture/src/storage/db/common/intro.md b/books/architecture/src/storage/db/common/intro.md new file mode 100644 index 0000000..b29e971 --- /dev/null +++ b/books/architecture/src/storage/db/common/intro.md @@ -0,0 +1 @@ +# 🟒 Common behavior diff --git a/books/architecture/src/storage/db/common/ops.md b/books/architecture/src/storage/db/common/ops.md new file mode 100644 index 0000000..2bfd3f9 --- /dev/null +++ b/books/architecture/src/storage/db/common/ops.md @@ -0,0 +1 @@ +# 🟒 ops diff --git a/books/architecture/src/storage/db/common/requests.md b/books/architecture/src/storage/db/common/requests.md new file mode 100644 index 0000000..7391375 --- /dev/null +++ b/books/architecture/src/storage/db/common/requests.md @@ -0,0 +1 @@ +# 🟒 Requests diff --git a/books/architecture/src/storage/db/common/responses.md b/books/architecture/src/storage/db/common/responses.md new file mode 100644 index 0000000..3ba68e0 --- /dev/null +++ b/books/architecture/src/storage/db/common/responses.md @@ -0,0 +1 @@ +# 🟒 Responses diff --git a/books/architecture/src/storage/db/common/shutdown.md b/books/architecture/src/storage/db/common/shutdown.md new file mode 100644 index 0000000..cfe83db --- /dev/null +++ b/books/architecture/src/storage/db/common/shutdown.md @@ -0,0 +1 @@ +# 🟒 Shutdown diff --git a/books/architecture/src/storage/db/common/thread-model.md b/books/architecture/src/storage/db/common/thread-model.md new file mode 100644 index 0000000..b2addfc --- /dev/null +++ b/books/architecture/src/storage/db/common/thread-model.md @@ -0,0 +1 @@ +# 🟒 Thread model diff --git a/books/architecture/src/storage/db/common/types.md b/books/architecture/src/storage/db/common/types.md new file mode 100644 index 0000000..ba238d6 --- /dev/null +++ b/books/architecture/src/storage/db/common/types.md @@ -0,0 +1 @@ +# 🟒 Types diff --git a/books/architecture/src/storage/db/database-abstraction.md b/books/architecture/src/storage/db/database-abstraction.md new file mode 100644 index 0000000..5baa69e --- /dev/null +++ b/books/architecture/src/storage/db/database-abstraction.md @@ -0,0 +1,21 @@ +# Database abstraction +[`cuprate_database`](https://doc.cuprate.org/cuprate_database) is Cuprate’s database abstraction. + +This crate abstracts various database backends with `trait`s. + +All backends have the following attributes: + +- [Embedded](https://en.wikipedia.org/wiki/Embedded_database) +- [Multiversion concurrency control](https://en.wikipedia.org/wiki/Multiversion_concurrency_control) +- [ACID](https://en.wikipedia.org/wiki/ACID) +- Are `(key, value)` oriented and have the expected API (`get()`, `insert()`, `delete()`) +- Are table oriented (`"table_name" -> (key, value)`) +- Allows concurrent readers + +The currently implemented backends are: +- [`heed`](https://github.com/meilisearch/heed) (LMDB) +- [`redb`](https://github.com/cberner/redb) + +Said precicely, `cuprate_database` is the embedded database other Cuprate +crates interact with instead of using any particular backend implementation. +This allows the backend to be swapped and/or future backends to be implemented. \ No newline at end of file diff --git a/books/architecture/src/storage/db/issues/extra.md b/books/architecture/src/storage/db/issues/extra.md new file mode 100644 index 0000000..c8fe44d --- /dev/null +++ b/books/architecture/src/storage/db/issues/extra.md @@ -0,0 +1 @@ +# 🟒 Extra table data diff --git a/books/architecture/src/storage/db/issues/unaligned.md b/books/architecture/src/storage/db/issues/unaligned.md new file mode 100644 index 0000000..af02c15 --- /dev/null +++ b/books/architecture/src/storage/db/issues/unaligned.md @@ -0,0 +1 @@ +# 🟒 Unaligned bytes diff --git a/books/architecture/src/storage/db/layers/types.md b/books/architecture/src/storage/db/layers/types.md new file mode 100644 index 0000000..ba238d6 --- /dev/null +++ b/books/architecture/src/storage/db/layers/types.md @@ -0,0 +1 @@ +# 🟒 Types diff --git a/books/architecture/src/storage/intro.md b/books/architecture/src/storage/intro.md index e78538e..5edd51d 100644 --- a/books/architecture/src/storage/intro.md +++ b/books/architecture/src/storage/intro.md @@ -1,5 +1,5 @@ # Storage -This section covers all things related to the long-term on-disk storage of data within Cuprate. +This section covers all things related to the on-disk storage of data within Cuprate. ## Overview The quick overview is that Cuprate has a [database abstraction crate](./database-abstraction.md) @@ -12,9 +12,9 @@ This database abstraction crate is then used by all crates that need on-disk sto ## Service The interface provided by all crates building on-top of the database abstraction is a [`tower::Service`](https://docs.rs/tower), i.e. -database requests are sent, and database responses are received asynchronously. +database requests/responses are sent/received asynchronously. -As the interface details are similar across databases (threadpool, read operations, write operations), +As the interface details are similar across crates (threadpool, read operations, write operations), the interface itself is abstracted in the [`cuprate_database_service`](./db/layers/intro.md) crate, which is then used by the crates. diff --git a/books/architecture/src/storage/txpool/intro.md b/books/architecture/src/storage/txpool/intro.md new file mode 100644 index 0000000..4eb139b --- /dev/null +++ b/books/architecture/src/storage/txpool/intro.md @@ -0,0 +1 @@ +# βšͺ️ Transaction pool