mirror of https://github.com/Cuprate/cuprate.git synced 2024-12-23 03:59:31 +00:00

database: Resizes, Shutdown, Flushing (#68 )

* error: add `NeedsResize`

accidently removed, was `MapFull` before.

We need this because we as the writer thread must
react to this error in order to resize.

The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist

* env/backend: add `MANUAL_RESIZE` and `resize()`

* heed: respect `ReadersFull`, comment errors

* free: add `resize_memory_map()`

* env: add `Env::current_map_size`

* service: add `resize_map()`

* database: make `Env` itself cheaply clonable + threadsafe

`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.

* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`

* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`

* database: add `src/service/state.rs`

* service: add to `state.rs`, add `DatabaseState` to readers/writer

* add `parking_lot`

Okay, turns out we need to take locks in
`database/src/service/state.rs`...

`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.

* state: use `crossbeam::atomic::AtomicCell`

We have crossbeam as a dep anyway.

* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`

* service: add reader shutdown handle, use `Select` for msgs

* service: remove `state.rs`

We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.

* service: fix misc reader/writer stuff

* database: add `config.rs`

* service: use `ReaderThreads` when spawning readers

* service: impl `shutdown()` for readers/writer

* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`

Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.

If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.

Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.

* remove `parking_lot`

Could be used as the database mutual exclusion lock.

Needs to be tested against `std`.

* config: add `path`

* env: `path()` -> `config()`, backend: impl `Drop`

* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect

* backend: add `Config` to `ConcreteEnv`

* service: use `std:🧵:Builder` for readers/writer

* config: `PathBuf` -> `Cow<'static, Path>`

* misc docs, remove `RuntimeError::ShuttingDown`

* service: init & shutdown docs

* heed: impl `Env::resize_map()`

* heed: impl `Env::current_map_size()`

* lib.rs: add example crate usage test

* heed: `RwLock` comment

* helper: add `cuprate_database_dir()`

* config: use `cuprate_database_dir()`

* lib.rs: TODO example test

* database: add `page_size`

The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772

* free: impl `resize_memory_map()`

* free: docs

* env: add `disk_size_bytes()`

* config: impl `From<$num>` for `ReaderThreads`

* move `fs`-related constants to `cuprate_helper::fs`

* docs

* add `resize.rs`, `ResizeAlgorithm`

* env: use `ResizeAlgorithm` in `resize_map()`

* TODO: account for LMDB reader limit

* resize: docs, add `page_size()`, impl `fixed_bytes()`

* resize: impl `percent()`

* resize: docs

* env: take `ResizeAlgorithm` by value (it impls `Copy`)

* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`

* config: `From<Into<usize>>` for `ReaderThreads`

Co-authored-by: Boog900 <boog900@tutanota.com>

* env: move mutual exclusion doc to backend

* free: update invariant doc

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/service/mod.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* fix `[allow(unused_imports)] // docs`

* move DB filename from `helper/` -> `database/`

* config: use `DATABASE_FILENAME`

* config: add `db_file_path()`

* lib: add non-`service` usage invariant docs

* table: seal `Table` trait, impl for all `crate::tables`

* fix docs

* fix docs pt.2

---------

Co-authored-by: Boog900 <boog900@tutanota.com>

2024-02-25 19:46:36 +00:00

6.3 KiB

Raw Blame History

Database

Cuprate's database implementation.

Documentation
File Structure
Backends
- heed
- sanakirja
Layers
- Database
- Trait
- [ConcreteEnv](#concreteenvConcreteEnv
- Thread-pool
- Service
Resizing
Flushing
(De)serialization

Documentation

In general, documentation for database/ is split into 3:

Documentation location	Purpose
`database/README.md`	High level design of `cuprate-database`
`cuprate-database`	Practical usage documentation/warnings/notes/etc
Source file `// comments`	Implementation-specific details (e.g, how many reader threads to spawn?)

This README serves as the overview/design document.

For actual practical usage, cuprate-database's types and general usage are documented via standard Rust tooling.

Run:

cargo doc --package cuprate-database --open

at the root of the repo to open/read the documentation.

If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many // Regular comments that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.

The code within src/ is also littered with some grep-able comments containing some keywords:

Word	Meaning
`INVARIANT`	This code makes an assumption that must be upheld for correctness
`SAFETY`	This `unsafe` code is okay, for `x,y,z` reasons
`FIXME`	This code works but isn't ideal
`HACK`	This code is a brittle workaround
`PERF`	This code is weird for performance reasons
`TODO`	This must be implemented; There should be 0 of these in production code
`SOMEDAY`	This should be implemented... someday

File Structure

A quick reference of the structure of the folders & files in cuprate-database.

Note that lib.rs/mod.rs files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding mod.rs.

`src/`

The top-level src/ files.

File	Purpose
`config.rs`	Database `Env` configuration
`constants.rs`	General constants used throughout `cuprate-database`
`database.rs`	Abstracted database; `trait Database`
`env.rs`	Abstracted database environment; `trait Env`
`error.rs`	Database error types
`free.rs`	General free functions (related to the database)
`key.rs`	Abstracted database keys; `trait Key`
`pod.rs`	Data (de)serialization; `trait Pod`
`table.rs`	Database table abstraction; `trait Table`
`tables.rs`	All the table definitions used by `cuprate-database`
`transaction.rs`	Database transaction abstraction; `trait RoTx`, `trait RwTx`

`src/ops/`

This folder contains the cupate_database::ops module.

TODO: more detailed descriptions.

File	Purpose
`alt_block.rs`	Alternative blocks
`block.rs`	Blocks
`blockchain.rs`	Blockchain-related
`output.rs`	Outputs
`property.rs`	Properties
`spent_key.rs`	Spent keys
`tx.rs`	Transactions

`src/service/`

This folder contains the cupate_database::service module.

File	Purpose
`free.rs`	General free functions used (related to `cuprate_database::service`)
`read.rs`	Read thread-pool definitions and logic
`request.rs`	Read/write `Request`s to the database
`response.rs`	Read/write `Response`'s from the database
`tests.rs`	Thread-pool tests and test helper functions
`write.rs`	Write thread-pool definitions and logic

`src/backend/`

This folder contains the actual database crates used as the backend for cuprate-database.

Each backend has its own folder.

Folder	Purpose
`heed/`	Backend using using forked `heed`
`sanakirja/`	Backend using `sanakirja`

All backends follow the same file structure:

File	Purpose
`database.rs`	Implementation of `trait Database`
`env.rs`	Implementation of `trait Env`
`error.rs`	Implementation of backend's errors to `cuprate_database`'s error types
`transaction.rs`	Implementation of `trait RoTx/RwTx`
`types.rs`	Type aliases for long backend-specific types

Backends

cuprate-database's traits abstract over various actual databases.

Each database's implementation is located in its respective file in src/backend/${DATABASE_NAME}.rs.

`heed`

The default database used is a modified fork of heed, located at Cuprate/heed.

To generate documentation of the fork for local use:

git clone --recursive https://github.com/Cuprate/heed
cargo doc

LMDB should not need to be installed as heed has a build script that pulls it in automatically.

TODO: document max readers limit: 059028a30a/src/blockchain_db/lmdb/db_lmdb.cpp (L1372). Other potential processes (e.g. xmrblocks) that are also reading the data.mdb file need to be accounted for.

`sanakirja`

TODO

Layers

TODO: update with accurate information when ready, update image.

Database

Trait

ConcreteEnv

Thread

Service

Resizing

TODO: document resize algorithm:

Exactly when it occurs
How much bytes are added

All backends follow the same algorithm.

Flushing

TODO: document disk flushing behavior.

Config options
Backend-specific behavior

(De)serialization

TODO: document Pod and how databases use (de)serialize objects when storing/fetching, essentially using <[u8], [u8]>.

6.3 KiB Raw Blame History

Database

Documentation

File Structure

src/

src/ops/

src/service/

src/backend/

Backends

heed

sanakirja

Layers

Database

Trait

ConcreteEnv

Thread

Service

Resizing

Flushing

(De)serialization

6.3 KiB

Raw Blame History

`src/`

`src/ops/`

`src/service/`

`src/backend/`

`heed`

`sanakirja`