abstraction

books/architecture/src/SUMMARY.md

@@ -29,11 +29,10 @@
 
 - [🟢 Storage](storage/intro.md)
     - [🟢 Database abstraction](storage/db/database-abstraction.md)
-        - [🟢 Backends](storage/db/backends.md)
-        - [🟢 Layers](storage/db/layers/intro.md)
-            - [🟢 Backend](storage/db/layers/backend.md)
-            - [🟢 ConcreteEnv](storage/db/layers/concrete_env.md)
-            - [🟢 Trait](storage/db/layers/trait.md)
+        - [🟢 Abstraction](storage/db/abstraction/intro.md)
+            - [🟢 Backend](storage/db/abstraction/backend.md)
+            - [🟢 ConcreteEnv](storage/db/abstraction/concrete_env.md)
+            - [🟢 Trait](storage/db/abstraction/trait.md)
         - [🟢 Syncing](storage/db/syncing.md)
         - [🟢 Resizing](storage/db/resizing.md)
         - [🟢 (De)serialization](storage/db/serde.md)

books/architecture/src/storage/db/abstraction/backend.md

@@ -0,0 +1,50 @@
+# Backend
+First, we need an actual database implementation.
+
+`cuprate-database`'s `trait`s allow abstracting over the actual database, such that any backend in particular could be used.
+
+This page is an enumeration of all the backends Cuprate has, has tried, and may try in the future.
+
+## `heed`
+The default database used is [`heed`](https://github.com/meilisearch/heed) (LMDB). The upstream versions from [`crates.io`](https://crates.io/crates/heed) are used. `LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
+
+`heed`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
+
+| Filename   | Purpose |
+|------------|---------|
+| `data.mdb` | Main data file
+| `lock.mdb` | Database lock file
+
+`heed`-specific notes:
+- [There is a maximum reader limit](https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1372). Other potential processes (e.g. `xmrblocks`) that are also reading the `data.mdb` file need to be accounted for
+- [LMDB does not work on remote filesystem](https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/lmdb.h#L129)
+
+## `redb`
+The 2nd database backend is the 100% Rust [`redb`](https://github.com/cberner/redb).
+
+The upstream versions from [`crates.io`](https://crates.io/crates/redb) are used.
+
+`redb`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
+
+| Filename    | Purpose |
+|-------------|---------|
+| `data.redb` | Main data file
+
+<!-- TODO: document DB on remote filesystem (does redb allow this?) -->
+
+## `redb-memory`
+This backend is 100% the same as `redb`, although, it uses [`redb::backend::InMemoryBackend`](https://docs.rs/redb/2.1.2/redb/backends/struct.InMemoryBackend.html) which is a database that completely resides in memory instead of a file.
+
+All other details about this should be the same as the normal `redb` backend.
+
+## `sanakirja`
+[`sanakirja`](https://docs.rs/sanakirja) was a candidate as a backend, however there were problems with maximum value sizes.
+
+The default maximum value size is [1012 bytes](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.Storable.html) which was too small for our requirements. Using [`sanakirja::Slice`](https://docs.rs/sanakirja/1.4.1/sanakirja/union.Slice.html) and [sanakirja::UnsizedStorage](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.UnsizedStorable.html) was attempted, but there were bugs found when inserting a value in-between `512..=4096` bytes.
+
+As such, it is not implemented.
+
+## `MDBX`
+[`MDBX`](https://erthink.github.io/libmdbx) was a candidate as a backend, however MDBX deprecated the custom key/value comparison functions, this makes it a bit trickier to implement multimap tables. It is also quite similar to the main backend LMDB (of which it was originally a fork of).
+
+As such, it is not implemented (yet).

books/architecture/src/storage/db/abstraction/concrete_env.md

@@ -0,0 +1,15 @@
+# `ConcreteEnv`
+After a backend is selected, the main database environment struct is "abstracted" by putting it in the non-generic, concrete [`struct ConcreteEnv`](https://doc.cuprate.org/cuprate_database/struct.ConcreteEnv.html).
+
+This is the main object used when handling the database directly.
+
+This struct contains all the data necessary to operate the database.
+The actual database backend `ConcreteEnv` will use internally depends on which backend feature is used.
+
+`ConcreteEnv` itself is not too important, what is important is that:
+1. It allows callers to not directly reference any particular backend environment
+1. It implements [`trait Env`](https://doc.cuprate.org/cuprate_database/trait.Env.html) which opens the door to all the other database traits
+
+The equivalent "database environment" objects in the backends themselves are:
+- [`heed::Env`](https://docs.rs/heed/0.20.0/heed/struct.Env.html)
+- [`redb::Database`](https://docs.rs/redb/2.1.0/redb/struct.Database.html)

books/architecture/src/storage/db/abstraction/intro.md

@@ -0,0 +1,33 @@
+# Abstraction
+This next section details how `cuprate_database` abstracts multiple database backends into 1 API.
+
+## Diagram
+A simple diagram describing the responsibilities/relationship of `cuprate_database`.
+
+```text
+┌───────────────────────────────────────────────────────────────────────┐
+│ cuprate_database                                                      │
+│                                                                       │
+│ ┌───────────────────────────┐     ┌─────────────────────────────────┐ │
+│ │ Database traits           │     │ Backends                        │ │
+│ │ ┌─────┐┌──────┐┌────────┐ │     │ ┌─────────────┐ ┌─────────────┐ │ │
+│ │ │ Env ││ TxRw ││ ...    │ ├─────┤ │ heed (LMDB) │ │ redb        │ │ │
+│ │ └─────┘└──────┘└────────┘ │     │ └─────────────┘ └─────────────┘ │ │
+│ └──────────┬─────────────┬──┘     └──┬──────────────────────────────┘ │
+│            │             └─────┬─────┘                                │
+│            │         ┌─────────┴──────────────┐                       │
+│            │         │ Database types         │                       │
+│            │         │ ┌─────────────┐┌─────┐ │                       │
+│            │         │ │ ConcreteEnv ││ ... │ │                       │
+│            │         │ └─────────────┘└─────┘ │                       │
+│            │         └─────────┬──────────────┘                       │
+│            │                   │                                      │
+└────────────┼───────────────────┼──────────────────────────────────────┘
+             │                   │
+             └───────────────────┤
+                                 │
+                                 ▼
+                     ┌───────────────────────┐
+                     │ cuprate_database user │
+                     └───────────────────────┘
+```

books/architecture/src/storage/db/abstraction/trait.md

@@ -0,0 +1,49 @@
+# Trait
+`cuprate_database` provides a set of `trait`s that abstract over the various database backends.
+
+This allows the function signatures and behavior to stay the same but allows for swapping out databases in an easier fashion.
+
+All common behavior of the backend's are encapsulated here and used instead of using the backend directly.
+
+Examples:
+- [`trait Env`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/env.rs)
+- [`trait {TxRo, TxRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/transaction.rs)
+- [`trait {DatabaseRo, DatabaseRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/database.rs)
+
+For example, instead of calling `heed` or `redb`'s `get()` function directly, `DatabaseRo::get()` is called.
+
+## Usage
+With a `ConcreteEnv` and a particular backend selected,
+we can now start using it alongside these traits to start
+doing database operations in a generic manner.
+
+An example:
+
+```rust
+use cuprate_database::{
+    ConcreteEnv,
+    config::ConfigBuilder,
+    Env, EnvInner,
+    DatabaseRo, DatabaseRw, TxRo, TxRw,
+};
+
+// Initialize the database environment.
+let env = ConcreteEnv::open(config)?;
+
+// Open up a transaction + tables for writing.
+let env_inner = env.env_inner();
+let tx_rw = env_inner.tx_rw()?;
+env_inner.create_db::<Table>(&tx_rw)?;
+
+// Write data to the table.
+{
+	let mut table = env_inner.open_db_rw::<Table>(&tx_rw)?;
+	table.put(&0, &1)?;
+}
+
+// Commit the transaction.
+TxRw::commit(tx_rw)?;
+```
+
+As seen above, there is no direct call to `heed` or `redb`.
+Their functionality is abstracted behind `ConcreteEnv` and the `trait`s.

books/architecture/src/storage/db/layers/backend.md

@@ -1 +0,0 @@
-# ⚪️ Backend

books/architecture/src/storage/db/layers/concrete_env.md

@@ -1 +0,0 @@
-# ⚪️ ConcreteEnv

books/architecture/src/storage/db/layers/intro.md

@@ -1 +0,0 @@
-# ⚪️ Layers

books/architecture/src/storage/db/layers/ops.md

@@ -1 +0,0 @@
-# ⚪️ ops

books/architecture/src/storage/db/layers/service.md

@@ -1 +0,0 @@
-# ⚪️ Service

books/architecture/src/storage/db/layers/trait.md

@@ -1 +0,0 @@
-# ⚪️ Trait

books/architecture/src/storage/db/layers/types.md

@@ -1 +0,0 @@
-# 🟢 Types