51d9ccd02d
* error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std:🧵:Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com> |
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.toml | ||
| README.md | ||
Database
Cuprate's database implementation.
- Documentation
- File Structure
- Backends
- Layers
- Database
- Trait
- [ConcreteEnv](#concreteenvConcreteEnv
- Thread-pool
- Service
- Resizing
- Flushing
- (De)serialization
Documentation
In general, documentation for database/ is split into 3:
| Documentation location | Purpose |
|---|---|
database/README.md |
High level design of cuprate-database |
cuprate-database |
Practical usage documentation/warnings/notes/etc |
Source file // comments |
Implementation-specific details (e.g, how many reader threads to spawn?) |
This README serves as the overview/design document.
For actual practical usage, cuprate-database's types and general usage are documented via standard Rust tooling.
Run:
cargo doc --package cuprate-database --open
at the root of the repo to open/read the documentation.
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many // Regular comments that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
The code within src/ is also littered with some grep-able comments containing some keywords:
| Word | Meaning |
|---|---|
INVARIANT |
This code makes an assumption that must be upheld for correctness |
SAFETY |
This unsafe code is okay, for x,y,z reasons |
FIXME |
This code works but isn't ideal |
HACK |
This code is a brittle workaround |
PERF |
This code is weird for performance reasons |
TODO |
This must be implemented; There should be 0 of these in production code |
SOMEDAY |
This should be implemented... someday |
File Structure
A quick reference of the structure of the folders & files in cuprate-database.
Note that lib.rs/mod.rs files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding mod.rs.
src/
The top-level src/ files.
| File | Purpose |
|---|---|
config.rs |
Database Env configuration |
constants.rs |
General constants used throughout cuprate-database |
database.rs |
Abstracted database; trait Database |
env.rs |
Abstracted database environment; trait Env |
error.rs |
Database error types |
free.rs |
General free functions (related to the database) |
key.rs |
Abstracted database keys; trait Key |
pod.rs |
Data (de)serialization; trait Pod |
table.rs |
Database table abstraction; trait Table |
tables.rs |
All the table definitions used by cuprate-database |
transaction.rs |
Database transaction abstraction; trait RoTx, trait RwTx |
src/ops/
This folder contains the cupate_database::ops module.
TODO: more detailed descriptions.
| File | Purpose |
|---|---|
alt_block.rs |
Alternative blocks |
block.rs |
Blocks |
blockchain.rs |
Blockchain-related |
output.rs |
Outputs |
property.rs |
Properties |
spent_key.rs |
Spent keys |
tx.rs |
Transactions |
src/service/
This folder contains the cupate_database::service module.
| File | Purpose |
|---|---|
free.rs |
General free functions used (related to cuprate_database::service) |
read.rs |
Read thread-pool definitions and logic |
request.rs |
Read/write Requests to the database |
response.rs |
Read/write Response's from the database |
tests.rs |
Thread-pool tests and test helper functions |
write.rs |
Write thread-pool definitions and logic |
src/backend/
This folder contains the actual database crates used as the backend for cuprate-database.
Each backend has its own folder.
| Folder | Purpose |
|---|---|
heed/ |
Backend using using forked heed |
sanakirja/ |
Backend using sanakirja |
All backends follow the same file structure:
| File | Purpose |
|---|---|
database.rs |
Implementation of trait Database |
env.rs |
Implementation of trait Env |
error.rs |
Implementation of backend's errors to cuprate_database's error types |
transaction.rs |
Implementation of trait RoTx/RwTx |
types.rs |
Type aliases for long backend-specific types |
Backends
cuprate-database's traits abstract over various actual databases.
Each database's implementation is located in its respective file in src/backend/${DATABASE_NAME}.rs.
heed
The default database used is a modified fork of heed, located at Cuprate/heed.
To generate documentation of the fork for local use:
git clone --recursive https://github.com/Cuprate/heed
cargo doc
LMDB should not need to be installed as heed has a build script that pulls it in automatically.
TODO: document max readers limit: 059028a30a/src/blockchain_db/lmdb/db_lmdb.cpp (L1372). Other potential processes (e.g. xmrblocks) that are also reading the data.mdb file need to be accounted for.
sanakirja
TODO
Layers
TODO: update with accurate information when ready, update image.
Database
Trait
ConcreteEnv
Thread
Service
Resizing
TODO: document resize algorithm:
- Exactly when it occurs
- How much bytes are added
All backends follow the same algorithm.
Flushing
TODO: document disk flushing behavior.
- Config options
- Backend-specific behavior
(De)serialization
TODO: document Pod and how databases use (de)serialize objects when storing/fetching, essentially using <[u8], [u8]>.