From 86017f329146b685af88d5295d0745fc8b76b5bb Mon Sep 17 00:00:00 2001 From: "hinto.janai" Date: Tue, 3 Sep 2024 17:22:15 -0400 Subject: [PATCH] add files, intro --- books/architecture/src/SUMMARY.md | 54 +++++++++---------- .../src/storage/db/backends/intro.md | 1 + books/architecture/src/storage/db/intro.md | 1 + .../src/storage/db/issues/copy.md | 1 + .../src/storage/db/issues/endian.md | 1 + .../src/storage/db/issues/extra-table-data.md | 1 + .../src/storage/db/issues/hot-swap.md | 1 + .../src/storage/db/issues/intro.md | 1 + .../src/storage/db/issues/traits.md | 1 + .../src/storage/db/layers/backend.md | 1 + .../src/storage/db/layers/concrete_env.md | 1 + .../src/storage/db/layers/intro.md | 1 + .../architecture/src/storage/db/layers/ops.md | 1 + .../src/storage/db/layers/service.md | 1 + .../src/storage/db/layers/trait.md | 1 + books/architecture/src/storage/db/resizing.md | 1 + .../src/storage/db/schema/intro.md | 1 + .../src/storage/db/schema/multimap.md | 1 + .../src/storage/db/schema/tables.md | 1 + books/architecture/src/storage/db/serde.md | 1 + .../src/storage/db/service/initialization.md | 1 + .../src/storage/db/service/intro.md | 1 + .../src/storage/db/service/requests.md | 1 + .../src/storage/db/service/responses.md | 1 + .../src/storage/db/service/shutdown.md | 1 + .../src/storage/db/service/thread-model.md | 1 + books/architecture/src/storage/db/syncing.md | 1 + books/architecture/src/storage/intro.md | 35 +++++++++++- 28 files changed, 87 insertions(+), 28 deletions(-) create mode 100644 books/architecture/src/storage/db/backends/intro.md create mode 100644 books/architecture/src/storage/db/intro.md create mode 100644 books/architecture/src/storage/db/issues/copy.md create mode 100644 books/architecture/src/storage/db/issues/endian.md create mode 100644 books/architecture/src/storage/db/issues/extra-table-data.md create mode 100644 books/architecture/src/storage/db/issues/hot-swap.md create mode 100644 books/architecture/src/storage/db/issues/intro.md create mode 100644 books/architecture/src/storage/db/issues/traits.md create mode 100644 books/architecture/src/storage/db/layers/backend.md create mode 100644 books/architecture/src/storage/db/layers/concrete_env.md create mode 100644 books/architecture/src/storage/db/layers/intro.md create mode 100644 books/architecture/src/storage/db/layers/ops.md create mode 100644 books/architecture/src/storage/db/layers/service.md create mode 100644 books/architecture/src/storage/db/layers/trait.md create mode 100644 books/architecture/src/storage/db/resizing.md create mode 100644 books/architecture/src/storage/db/schema/intro.md create mode 100644 books/architecture/src/storage/db/schema/multimap.md create mode 100644 books/architecture/src/storage/db/schema/tables.md create mode 100644 books/architecture/src/storage/db/serde.md create mode 100644 books/architecture/src/storage/db/service/initialization.md create mode 100644 books/architecture/src/storage/db/service/intro.md create mode 100644 books/architecture/src/storage/db/service/requests.md create mode 100644 books/architecture/src/storage/db/service/responses.md create mode 100644 books/architecture/src/storage/db/service/shutdown.md create mode 100644 books/architecture/src/storage/db/service/thread-model.md create mode 100644 books/architecture/src/storage/db/syncing.md diff --git a/books/architecture/src/SUMMARY.md b/books/architecture/src/SUMMARY.md index d0c641a..a29ce90 100644 --- a/books/architecture/src/SUMMARY.md +++ b/books/architecture/src/SUMMARY.md @@ -27,33 +27,33 @@ --- -- [⚪️ Storage](storage/intro.md) - - [⚪️ Database abstraction](storage/db/intro.md) - - [⚪️ Backends](storage/db/backends/intro.md) - - [⚪️ Layers](storage/db/layers/intro.md) - - [⚪️ Backend](storage/db/layers/backend.md) - - [⚪️ ConcreteEnv](storage/db/layers/concrete_env.md) - - [⚪️ Trait](storage/db/layers/trait.md) - - [⚪️ `ops`](storage/db/layers/ops.md) - - [⚪️ Service](storage/db/layers/service.md) - - [⚪️ The service](storage/db/service/intro.md) - - [⚪️ Initialization](storage/db/service/initialization.md) - - [⚪️ Requests](storage/db/service/requests.md) - - [⚪️ Responses](storage/db/service/responses.md) - - [⚪️ Thread model](storage/db/service/thread-model.md) - - [⚪️ Shutdown](storage/db/service/shutdown.md) - - [⚪️ Syncing](storage/db/syncing.md) - - [⚪️ Resizing](storage/db/resizing.md) - - [⚪️ (De)serialization](storage/db/serde.md) - - [⚪️ Schema](storage/db/schema/intro.md) - - [⚪️ Tables](storage/db/schema/tables.md) - - [⚪️ Multimap tables](storage/db/schema/multimap.md) - - [⚪️ Known issues and tradeoffs](storage/db/issues/intro.md) - - [⚪️ Traits abstracting backends](storage/db/issues/traits.md) - - [⚪️ Hot-swappable backends](storage/db/issues/hot-swap.md) - - [⚪️ Copying unaligned bytes](storage/db/issues/copy.md) - - [⚪️ Endianness](storage/db/issues/endian.md) - - [⚪️ Extra table data](storage/db/issues/extra-table-data.md) +- [🟢 Storage](storage/intro.md) + - [🟢 Database abstraction](storage/db/intro.md) + - [🟢 Backends](storage/db/backends/intro.md) + - [🟢 Layers](storage/db/layers/intro.md) + - [🟢 Backend](storage/db/layers/backend.md) + - [🟢 ConcreteEnv](storage/db/layers/concrete_env.md) + - [🟢 Trait](storage/db/layers/trait.md) + - [🟢 Syncing](storage/db/syncing.md) + - [🟢 Resizing](storage/db/resizing.md) + - [🟢 (De)serialization](storage/db/serde.md) + - [🟢 Schema](storage/db/schema/intro.md) + - [🟢 Tables](storage/db/schema/tables.md) + - [🟢 Multimap tables](storage/db/schema/multimap.md) + - [🟢 Known issues and tradeoffs](storage/db/issues/intro.md) + - [🟢 Traits abstracting backends](storage/db/issues/traits.md) + - [🟢 Hot-swappable backends](storage/db/issues/hot-swap.md) + - [🟢 Copying unaligned bytes](storage/db/issues/copy.md) + - [🟢 Endianness](storage/db/issues/endian.md) + - [🟢 Extra table data](storage/db/issues/extra-table-data.md) + - [🟢 The layers](storage/db/layers/intro.md) + - [🟢 `ops`](storage/db/layers/ops.md) + - [🟢 `tower::Service`](storage/db/service/intro.md) + - [🟢 Initialization](storage/db/service/initialization.md) + - [🟢 Requests](storage/db/service/requests.md) + - [🟢 Responses](storage/db/service/responses.md) + - [🟢 Thread model](storage/db/service/thread-model.md) + - [🟢 Shutdown](storage/db/service/shutdown.md) - [⚪️ Blockchain](storage/blockchain.md) - [⚪️ Transaction pool](storage/transaction-pool.md) - [⚪️ Pruning](storage/pruning.md) diff --git a/books/architecture/src/storage/db/backends/intro.md b/books/architecture/src/storage/db/backends/intro.md new file mode 100644 index 0000000..250cdf3 --- /dev/null +++ b/books/architecture/src/storage/db/backends/intro.md @@ -0,0 +1 @@ +# ⚪️ Backends diff --git a/books/architecture/src/storage/db/intro.md b/books/architecture/src/storage/db/intro.md new file mode 100644 index 0000000..b21a192 --- /dev/null +++ b/books/architecture/src/storage/db/intro.md @@ -0,0 +1 @@ +# ⚪️ Database abstraction diff --git a/books/architecture/src/storage/db/issues/copy.md b/books/architecture/src/storage/db/issues/copy.md new file mode 100644 index 0000000..f13746d --- /dev/null +++ b/books/architecture/src/storage/db/issues/copy.md @@ -0,0 +1 @@ +# ⚪️ Copying unaligned bytes diff --git a/books/architecture/src/storage/db/issues/endian.md b/books/architecture/src/storage/db/issues/endian.md new file mode 100644 index 0000000..d1271ec --- /dev/null +++ b/books/architecture/src/storage/db/issues/endian.md @@ -0,0 +1 @@ +# ⚪️ Endianness diff --git a/books/architecture/src/storage/db/issues/extra-table-data.md b/books/architecture/src/storage/db/issues/extra-table-data.md new file mode 100644 index 0000000..b0215c2 --- /dev/null +++ b/books/architecture/src/storage/db/issues/extra-table-data.md @@ -0,0 +1 @@ +# ⚪️ Extra table data diff --git a/books/architecture/src/storage/db/issues/hot-swap.md b/books/architecture/src/storage/db/issues/hot-swap.md new file mode 100644 index 0000000..2cbab13 --- /dev/null +++ b/books/architecture/src/storage/db/issues/hot-swap.md @@ -0,0 +1 @@ +# ⚪️ Hot-swappable backends diff --git a/books/architecture/src/storage/db/issues/intro.md b/books/architecture/src/storage/db/issues/intro.md new file mode 100644 index 0000000..20ab7b5 --- /dev/null +++ b/books/architecture/src/storage/db/issues/intro.md @@ -0,0 +1 @@ +# ⚪️ Known issues and tradeoffs diff --git a/books/architecture/src/storage/db/issues/traits.md b/books/architecture/src/storage/db/issues/traits.md new file mode 100644 index 0000000..9740130 --- /dev/null +++ b/books/architecture/src/storage/db/issues/traits.md @@ -0,0 +1 @@ +# ⚪️ Traits abstracting backends diff --git a/books/architecture/src/storage/db/layers/backend.md b/books/architecture/src/storage/db/layers/backend.md new file mode 100644 index 0000000..568ecaf --- /dev/null +++ b/books/architecture/src/storage/db/layers/backend.md @@ -0,0 +1 @@ +# ⚪️ Backend diff --git a/books/architecture/src/storage/db/layers/concrete_env.md b/books/architecture/src/storage/db/layers/concrete_env.md new file mode 100644 index 0000000..98018a9 --- /dev/null +++ b/books/architecture/src/storage/db/layers/concrete_env.md @@ -0,0 +1 @@ +# ⚪️ ConcreteEnv diff --git a/books/architecture/src/storage/db/layers/intro.md b/books/architecture/src/storage/db/layers/intro.md new file mode 100644 index 0000000..701ca85 --- /dev/null +++ b/books/architecture/src/storage/db/layers/intro.md @@ -0,0 +1 @@ +# ⚪️ Layers diff --git a/books/architecture/src/storage/db/layers/ops.md b/books/architecture/src/storage/db/layers/ops.md new file mode 100644 index 0000000..f37989c --- /dev/null +++ b/books/architecture/src/storage/db/layers/ops.md @@ -0,0 +1 @@ +# ⚪️ ops diff --git a/books/architecture/src/storage/db/layers/service.md b/books/architecture/src/storage/db/layers/service.md new file mode 100644 index 0000000..1ec8479 --- /dev/null +++ b/books/architecture/src/storage/db/layers/service.md @@ -0,0 +1 @@ +# ⚪️ Service diff --git a/books/architecture/src/storage/db/layers/trait.md b/books/architecture/src/storage/db/layers/trait.md new file mode 100644 index 0000000..6a441d8 --- /dev/null +++ b/books/architecture/src/storage/db/layers/trait.md @@ -0,0 +1 @@ +# ⚪️ Trait diff --git a/books/architecture/src/storage/db/resizing.md b/books/architecture/src/storage/db/resizing.md new file mode 100644 index 0000000..777e2d4 --- /dev/null +++ b/books/architecture/src/storage/db/resizing.md @@ -0,0 +1 @@ +# ⚪️ Resizing diff --git a/books/architecture/src/storage/db/schema/intro.md b/books/architecture/src/storage/db/schema/intro.md new file mode 100644 index 0000000..010e736 --- /dev/null +++ b/books/architecture/src/storage/db/schema/intro.md @@ -0,0 +1 @@ +# ⚪️ Schema diff --git a/books/architecture/src/storage/db/schema/multimap.md b/books/architecture/src/storage/db/schema/multimap.md new file mode 100644 index 0000000..13dd36b --- /dev/null +++ b/books/architecture/src/storage/db/schema/multimap.md @@ -0,0 +1 @@ +# ⚪️ Multimap tables diff --git a/books/architecture/src/storage/db/schema/tables.md b/books/architecture/src/storage/db/schema/tables.md new file mode 100644 index 0000000..60c0465 --- /dev/null +++ b/books/architecture/src/storage/db/schema/tables.md @@ -0,0 +1 @@ +# ⚪️ Tables diff --git a/books/architecture/src/storage/db/serde.md b/books/architecture/src/storage/db/serde.md new file mode 100644 index 0000000..5c559e5 --- /dev/null +++ b/books/architecture/src/storage/db/serde.md @@ -0,0 +1 @@ +# ⚪️ (De)serialization diff --git a/books/architecture/src/storage/db/service/initialization.md b/books/architecture/src/storage/db/service/initialization.md new file mode 100644 index 0000000..d96fb30 --- /dev/null +++ b/books/architecture/src/storage/db/service/initialization.md @@ -0,0 +1 @@ +# ⚪️ Initialization diff --git a/books/architecture/src/storage/db/service/intro.md b/books/architecture/src/storage/db/service/intro.md new file mode 100644 index 0000000..c6b2c74 --- /dev/null +++ b/books/architecture/src/storage/db/service/intro.md @@ -0,0 +1 @@ +# ⚪️ The service diff --git a/books/architecture/src/storage/db/service/requests.md b/books/architecture/src/storage/db/service/requests.md new file mode 100644 index 0000000..3f6a1e9 --- /dev/null +++ b/books/architecture/src/storage/db/service/requests.md @@ -0,0 +1 @@ +# ⚪️ Requests diff --git a/books/architecture/src/storage/db/service/responses.md b/books/architecture/src/storage/db/service/responses.md new file mode 100644 index 0000000..404ae0c --- /dev/null +++ b/books/architecture/src/storage/db/service/responses.md @@ -0,0 +1 @@ +# ⚪️ Responses diff --git a/books/architecture/src/storage/db/service/shutdown.md b/books/architecture/src/storage/db/service/shutdown.md new file mode 100644 index 0000000..1bdc92d --- /dev/null +++ b/books/architecture/src/storage/db/service/shutdown.md @@ -0,0 +1 @@ +# ⚪️ Shutdown diff --git a/books/architecture/src/storage/db/service/thread-model.md b/books/architecture/src/storage/db/service/thread-model.md new file mode 100644 index 0000000..4bcfa01 --- /dev/null +++ b/books/architecture/src/storage/db/service/thread-model.md @@ -0,0 +1 @@ +# ⚪️ Thread model diff --git a/books/architecture/src/storage/db/syncing.md b/books/architecture/src/storage/db/syncing.md new file mode 100644 index 0000000..7cc80f1 --- /dev/null +++ b/books/architecture/src/storage/db/syncing.md @@ -0,0 +1 @@ +# ⚪️ Syncing diff --git a/books/architecture/src/storage/intro.md b/books/architecture/src/storage/intro.md index 214cf15..e78538e 100644 --- a/books/architecture/src/storage/intro.md +++ b/books/architecture/src/storage/intro.md @@ -1 +1,34 @@ -# ⚪️ Storage +# Storage +This section covers all things related to the long-term on-disk storage of data within Cuprate. + +## Overview +The quick overview is that Cuprate has a [database abstraction crate](./database-abstraction.md) +that handles "low-level" database details such as key and value (de)serialization, tables, transactions, etc. + +This database abstraction crate is then used by all crates that need on-disk storage, i.e. the +- [Blockchain database](./blockchain.md) +- [Transaction pool database](./transaction-pool.md) + +## Service +The interface provided by all crates building on-top of the +database abstraction is a [`tower::Service`](https://docs.rs/tower), i.e. +database requests are sent, and database responses are received asynchronously. + +As the interface details are similar across databases (threadpool, read operations, write operations), +the interface itself is abstracted in the [`cuprate_database_service`](./db/layers/intro.md) crate, +which is then used by the crates. + +## Diagram +This is a roughly how database crates are set up. + +```text + ┌─────────────────┐ +┌──────────────────────────────────┐ │ │ +│ Some crate that needs a database │ ┌────────────────┐ │ │ +│ │ │ Public │ │ │ +│ ┌──────────────────────────────┐ │─►│ tower::Service │◄─►│ Rest of Cuprate │ +│ │ Database abstraction │ │ │ API │ │ │ +│ └──────────────────────────────┘ │ └────────────────┘ │ │ +└──────────────────────────────────┘ │ │ + └─────────────────┘ +```