c65eb0a3ca
* ops: add `trait MoneroR{o,w}`
* update `trait MoneroR{o,w}` bounds
* types: add `BlockInfoLatest` type alias
* block: impl most core functions
* types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916
* fix table type test
* cargo.toml: add `{cuprate-types, monero-serai}`
* add_block: add all other block data
* ops: remove unneeded `block` functions
* env: add `EnvInner::open_db_rw_all()`
* types: fix test
* block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()`
* add `trait Tables[Mut]` and use it in `EnvInner`
* block: use `TablesMut`
* tables: replace manual impl with `define_trait_tables!()`
* tables: docs for `trait Tables[Mut]`
* tables: doc functions
* create `call_fn_on_all_tables_or_early_return!()` macro
* block: cleanup signatures + bodies
* block: more fn's, docs
* block: add `doc_{error,single,bulk}!()`
* remove `ops/monero.rs`
* move `height()` to `ops/blockchain.rs`
* add `ops/macros.rs`
* tx: add fn signatures
* output: fix fn signatures
* ops: expose `_inner()` functions
* block: add `add_block_header{_bulk, _inner}()`
* ops: remove doc_{fn,inner}!()`
* ops: remove `_{inner,bulk}()`, lifetime + generics
* update lib/mod docs
* ops: add and use `doc_add_block_inner_invariant!()`
* ops: add docs/return to inner `add_block()` functions
* add_block(): extract and use fn for {key_image, output}
* ops: more fn body impl + `add_block()`
* cargo: add `monero-pruning`
* ops: extract out `tx` functions from `add_block()`
* property: add `db_version()`
* ops: `pop_block()` body, remove other `pop_block` fn's
* types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation`
* block: put `block_blob`, pass `Tables` to sub-functions
`impl TablesMut` can't mutably pass multiple tables since
it takes `&mut self`, so all functions unfortunately have to
take a full `&mut impl TablesMut` even though they only need
a few tables.
* database: add `DatabaseRw::take()`
useful for `pop_block()` where we need the value afterwards
* block: deserialize tx's from `block_blobs` in `pop_block()`
* blockchain: `height()` -> `chain_height()`
* output: fix `amount_index`
* ops: fix unlock_time, chain_height
* `BlockInfoV{1,2,3}` -> `BlockInfo`
* constants: add `DATABASE_VERSION`
* database: add `DatabaseRw::update()`
* output: use `DatabaseRw::update()` in `remove_output()`
* add `TxBlobs` table, ignore pruning tables
* block: mostly impl `add_block()` body
* ops: comments
* add_block: miner v2 tx commitment, height cast
* block: impl `pop_block()`
* block: mostly impl `get_block()`
* block: impl `get_block_{from_height,header,header_from_height}`
* add `OutputFlags` bitflags
* add_block: u32::try_into(height: u32), use `OutputFlags`
* tx: impl `get_{tx,tx_from_id}()`
* tx: move docs tests to `#[test]`
testing everything in 1 go is more natural since e.g:
`add_tx()` is followed by `get_tx()`
* tables: add `trait TablesIter`, `all_tables_empty()`
This allows `TablesMut` to be a superset of `Tables`
and use all its accessor functions.
* use cuprate-test-utils, fix tx tests
* block: `add_block()` take block by ref
* tx: use all txs in tests
* output: add `all_tx_functions()` test
* add_block: check current height against input
* block: map more fields in `get_block()`
* block: remove `get_block()`, doc tests, fix `get_block_header()`
* block: dummy values in test
* heed: use `last/first()` instead of `unsafe` cursors
We no longer have DUP semantics and also hard to debug errors
were popping up on `del_current()`...
* heed: fix `DatabaseRw::delete`
Ok(true) means the key did not exist, so we must return
Err(RuntimeError::KeyNotFound)
* block: `add_block()` (dummy value) test
* ops: `key_image` tests
* cleanup, docs, tests
* backend: test `take()` & `update()`
* docs
* remove `OutputFlags::NONE`
* add_block(): add asserts, panic docs, `should_panic` tests
* backend: remove `Ok(())` in `Database::delete` if already deleted
redb already does this, so heed so match
* block: move block operations after tx/outputs
* `amount == 0` -> `amount == 1`
* Nit: StorableVec::wrap_ref
* `saturating_sub(1)` -> `- 1`
* add `TxOutputs` table
* add_block(): add to `tx_outputs` table
* fix `DatabaseRo::update`
* add_tx(): take `block_height` as input
* tx: add/remove from `TxOutputs` table
* output: remove if `amount == 1` -> `amount_index == 0`
* output: fix `add_output()`'s `amount_index` calculation
* output: fix `add_output()`'s `amount_index` calculation again
* output: tests for `amount_index/num_outputs`
* block: `num_outputs - 1` and `take()` -> `get()`
We don't need to `take()` since the call afterwards to
`remove_output()` removes the entry
* block: swap `get_block_extended_header[_from_height]()`
* move `{key_image,output}` handling `add_block()` -> `add_tx()`
* blockchain: add doc to `top_block_height()`
* block: manual panic -> `assert_eq!()`
* test-utils: add `block_blob` to `VerifiedBlockInformation`
field introduced in this PR
* ops: use real block/tx data in tests
* block: `total_generated_coins` -> `cumulative_generated_coins`
* fix clippy, docs, TODOs
* `cumulative_generated_coins()`: `block/` -> `blockchain/`
* blockchain: add `cumulative_generated_coins()` tests
* Update database/src/ops/block.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* `cumulative_generated_coins()` docs for pre-block-0 special case
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.toml | ||
| README.md | ||
Database
Cuprate's database implementation.
Documentation
In general, documentation for database/ is split into 3:
| Documentation location | Purpose |
|---|---|
database/README.md |
High level design of cuprate-database |
cuprate-database |
Practical usage documentation/warnings/notes/etc |
Source file // comments |
Implementation-specific details (e.g, how many reader threads to spawn?) |
This README serves as the overview/design document.
For actual practical usage, cuprate-database's types and general usage are documented via standard Rust tooling.
Run:
cargo doc --package cuprate-database --open
at the root of the repo to open/read the documentation.
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many // Regular comments that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
The code within src/ is also littered with some grep-able comments containing some keywords:
| Word | Meaning |
|---|---|
INVARIANT |
This code makes an assumption that must be upheld for correctness |
SAFETY |
This unsafe code is okay, for x,y,z reasons |
FIXME |
This code works but isn't ideal |
HACK |
This code is a brittle workaround |
PERF |
This code is weird for performance reasons |
TODO |
This must be implemented; There should be 0 of these in production code |
SOMEDAY |
This should be implemented... someday |
File Structure
A quick reference of the structure of the folders & files in cuprate-database.
Note that lib.rs/mod.rs files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding mod.rs.
src/
The top-level src/ files.
| File | Purpose |
|---|---|
config.rs |
Database Env configuration |
constants.rs |
General constants used throughout cuprate-database |
database.rs |
Abstracted database; trait DatabaseR{o,w} |
env.rs |
Abstracted database environment; trait Env |
error.rs |
Database error types |
free.rs |
General free functions (related to the database) |
key.rs |
Abstracted database keys; trait Key |
resize.rs |
Database resizing algorithms |
storable.rs |
Data (de)serialization; trait Storable |
table.rs |
Database table abstraction; trait Table |
tables.rs |
All the table definitions used by cuprate-database |
transaction.rs |
Database transaction abstraction; trait TxR{o,w} |
types.rs |
Database table schema types |
src/ops/
This folder contains the cupate_database::ops module.
TODO: more detailed descriptions.
| File | Purpose |
|---|---|
alt_block.rs |
Alternative blocks |
block.rs |
Blocks |
blockchain.rs |
Blockchain-related |
output.rs |
Outputs |
property.rs |
Properties |
spent_key.rs |
Spent keys |
tx.rs |
Transactions |
src/service/
This folder contains the cupate_database::service module.
| File | Purpose |
|---|---|
free.rs |
General free functions used (related to cuprate_database::service) |
read.rs |
Read thread-pool definitions and logic |
tests.rs |
Thread-pool tests and test helper functions |
types.rs |
cuprate_database::service-related type aliases |
write.rs |
Write thread-pool definitions and logic |
src/backend/
This folder contains the actual database crates used as the backend for cuprate-database.
Each backend has its own folder.
| Folder | Purpose |
|---|---|
heed/ |
Backend using using forked heed |
sanakirja/ |
Backend using sanakirja |
All backends follow the same file structure:
| File | Purpose |
|---|---|
database.rs |
Implementation of trait DatabaseR{o,w} |
env.rs |
Implementation of trait Env |
error.rs |
Implementation of backend's errors to cuprate_database's error types |
storable.rs |
Compatibility layer between cuprate_database::Storable and backend-specific (de)serialization |
tests.rs |
Tests for the specific backend |
transaction.rs |
Implementation of trait TxR{o,w} |
types.rs |
Type aliases for long backend-specific types |
Backends
cuprate-database's traits abstract over various actual databases.
Each database's implementation is located in its respective file in src/backend/${DATABASE_NAME}.rs.
heed
The default database used is heed (LMDB).
LMDB should not need to be installed as heed has a build script that pulls it in automatically.
heed's filenames inside Cuprate's database folder (~/.local/share/cuprate/database/) are:
| Filename | Purpose |
|---|---|
data.mdb |
Main data file |
lock.mdb |
Database lock file |
TODO: document max readers limit: 059028a30a/src/blockchain_db/lmdb/db_lmdb.cpp (L1372). Other potential processes (e.g. xmrblocks) that are also reading the data.mdb file need to be accounted for.
TODO: document DB on remote filesystem: b8e54b4c31/libraries/liblmdb/lmdb.h (L129).
redb
The 2nd database backend is the 100% Rust redb.
The upstream versions from crates.io are used.
redb's filenames inside Cuprate's database folder (~/.local/share/cuprate/database/) are:
| Filename | Purpose |
|---|---|
data.redb |
Main data file |
TODO: document DB on remote filesystem (does redb allow this?)
redb-memory
This backend is 100% the same as redb, although, it uses redb::backend::InMemoryBackend which is a key-value store that completely resides in memory instead of a file.
All other details about this should be the same as the normal redb backend.
sanakirja
sanakirja was a candidate as a backend, however there were problems with maximum value sizes.
The default maximum value size is 1012 bytes which was too small for our requirements. Using sanakirja::Slice and sanakirja::UnsizedStorage was attempted, but there were bugs found when inserting a value in-between 512..=4096 bytes.
As such, it is not implemented.
MDBX
MDBX was a candidate as a backend, however MDBX deprecated the custom key/value comparison functions, this makes it a bit trickier to implement duplicate tables. It is also quite similar to the main backend LMDB (of which it was originally a fork of).
As such, it is not implemented (yet).
Layers
TODO: update with accurate information when ready, update image.
Database
Trait
ConcreteEnv
Thread
Service
Resizing
TODO: document resize algorithm:
- Exactly when it occurs
- How much bytes are added
All backends follow the same algorithm.
Flushing
TODO: document disk flushing behavior.
- Config options
- Backend-specific behavior
(De)serialization
TODO: document Storable and how databases (de)serialize types when storing/fetching.