SyntheticBird/cuprate-hinto-janai
1
0
Fork 0
mirror of https://github.com/hinto-janai/cuprate.git synced 2024-10-30 07:47:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
cuprate-hinto-janai/database/README.md
hinto-janai 51d9ccd02d
database: Resizes, Shutdown, Flushing (#68) 
* error: add `NeedsResize`

accidently removed, was `MapFull` before.

We need this because we as the writer thread must
react to this error in order to resize.

The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist

* env/backend: add `MANUAL_RESIZE` and `resize()`

* heed: respect `ReadersFull`, comment errors

* free: add `resize_memory_map()`

* env: add `Env::current_map_size`

* service: add `resize_map()`

* database: make `Env` itself cheaply clonable + threadsafe

`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.

* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`

* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`

* database: add `src/service/state.rs`

* service: add to `state.rs`, add `DatabaseState` to readers/writer

* add `parking_lot`

Okay, turns out we need to take locks in
`database/src/service/state.rs`...

`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.

* state: use `crossbeam::atomic::AtomicCell`

We have crossbeam as a dep anyway.

* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`

* service: add reader shutdown handle, use `Select` for msgs

* service: remove `state.rs`

We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.

* service: fix misc reader/writer stuff

* database: add `config.rs`

* service: use `ReaderThreads` when spawning readers

* service: impl `shutdown()` for readers/writer

* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`

Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.

If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.

Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.

* remove `parking_lot`

Could be used as the database mutual exclusion lock.

Needs to be tested against `std`.

* config: add `path`

* env: `path()` -> `config()`, backend: impl `Drop`

* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect

* backend: add `Config` to `ConcreteEnv`

* service: use `std:🧵:Builder` for readers/writer

* config: `PathBuf` -> `Cow<'static, Path>`

* misc docs, remove `RuntimeError::ShuttingDown`

* service: init & shutdown docs

* heed: impl `Env::resize_map()`

* heed: impl `Env::current_map_size()`

* lib.rs: add example crate usage test

* heed: `RwLock` comment

* helper: add `cuprate_database_dir()`

* config: use `cuprate_database_dir()`

* lib.rs: TODO example test

* database: add `page_size`

The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772

* free: impl `resize_memory_map()`

* free: docs

* env: add `disk_size_bytes()`

* config: impl `From<$num>` for `ReaderThreads`

* move `fs`-related constants to `cuprate_helper::fs`

* docs

* add `resize.rs`, `ResizeAlgorithm`

* env: use `ResizeAlgorithm` in `resize_map()`

* TODO: account for LMDB reader limit

* resize: docs, add `page_size()`, impl `fixed_bytes()`

* resize: impl `percent()`

* resize: docs

* env: take `ResizeAlgorithm` by value (it impls `Copy`)

* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`

* config: `From<Into<usize>>` for `ReaderThreads`

Co-authored-by: Boog900 <boog900@tutanota.com>

* env: move mutual exclusion doc to backend

* free: update invariant doc

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/service/mod.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* fix `[allow(unused_imports)] // docs`

* move DB filename from `helper/` -> `database/`

* config: use `DATABASE_FILENAME`

* config: add `db_file_path()`

* lib: add non-`service` usage invariant docs

* table: seal `Table` trait, impl for all `crate::tables`

* fix docs

* fix docs pt.2

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00

170 lines
No EOL
6.3 KiB
Markdown
Raw Blame History

# Database
Cuprate's database implementation.
<!-- Did you know markdown automatically increments number lists, even if they are all 1...? -->
1. [Documentation](#documentation)
1. [File Structure](#file-structure)
- [`src/`](#src)
- [`src/ops`](#src-ops)
- [`src/service/`](#src-service)
- [`src/backend/`](#src-backend)
1. [Backends](#backends)
- [`heed`](#heed)
- [`sanakirja`](#sanakirja)
1. [Layers](#layers)
- [Database](#database)
- [Trait](#trait)
- [ConcreteEnv](#concreteenvConcreteEnv
- [Thread-pool](#thread-pool)
- [Service](#service)
1. [Resizing](#resizing)
1. [Flushing](#flushing)
1. [(De)serialization](#deserialization)
---
# Documentation
In general, documentation for `database/` is split into 3:
| Documentation location | Purpose |
|---------------------------|---------|
| `database/README.md` | High level design of `cuprate-database`
| `cuprate-database` | Practical usage documentation/warnings/notes/etc
| Source file `// comments` | Implementation-specific details (e.g, how many reader threads to spawn?)
This README serves as the overview/design document.
For actual practical usage, `cuprate-database`'s types and general usage are documented via standard Rust tooling.
Run:
```bash
cargo doc --package cuprate-database --open
```
at the root of the repo to open/read the documentation.
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many `// Regular comments` that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
The code within `src/` is also littered with some `grep`-able comments containing some keywords:
| Word | Meaning |
|-------------|---------|
| `INVARIANT` | This code makes an _assumption_ that must be upheld for correctness
| `SAFETY` | This `unsafe` code is okay, for `x,y,z` reasons
| `FIXME` | This code works but isn't ideal
| `HACK` | This code is a brittle workaround
| `PERF` | This code is weird for performance reasons
| `TODO` | This must be implemented; There should be 0 of these in production code
| `SOMEDAY` | This should be implemented... someday
# File Structure
A quick reference of the structure of the folders & files in `cuprate-database`.
Note that `lib.rs/mod.rs` files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding `mod.rs`.
## `src/`
The top-level `src/` files.
| File | Purpose |
|------------------|---------|
| `config.rs` | Database `Env` configuration
| `constants.rs` | General constants used throughout `cuprate-database`
| `database.rs` | Abstracted database; `trait Database`
| `env.rs` | Abstracted database environment; `trait Env`
| `error.rs` | Database error types
| `free.rs` | General free functions (related to the database)
| `key.rs` | Abstracted database keys; `trait Key`
| `pod.rs` | Data (de)serialization; `trait Pod`
| `table.rs` | Database table abstraction; `trait Table`
| `tables.rs` | All the table definitions used by `cuprate-database`
| `transaction.rs` | Database transaction abstraction; `trait RoTx`, `trait RwTx`
## `src/ops/`
This folder contains the `cupate_database::ops` module.
TODO: more detailed descriptions.
| File | Purpose |
|-----------------|---------|
| `alt_block.rs` | Alternative blocks
| `block.rs` | Blocks
| `blockchain.rs` | Blockchain-related
| `output.rs` | Outputs
| `property.rs` | Properties
| `spent_key.rs` | Spent keys
| `tx.rs` | Transactions
## `src/service/`
This folder contains the `cupate_database::service` module.
| File | Purpose |
|----------------|---------|
| `free.rs` | General free functions used (related to `cuprate_database::service`)
| `read.rs` | Read thread-pool definitions and logic
| `request.rs` | Read/write `Request`s to the database
| `response.rs` | Read/write `Response`'s from the database
| `tests.rs` | Thread-pool tests and test helper functions
| `write.rs` | Write thread-pool definitions and logic
## `src/backend/`
This folder contains the actual database crates used as the backend for `cuprate-database`.
Each backend has its own folder.
| Folder | Purpose |
|--------------|---------|
| `heed/` | Backend using using forked [`heed`](https://github.com/Cuprate/heed)
| `sanakirja/` | Backend using [`sanakirja`](https://docs.rs/sanakirja)
All backends follow the same file structure:
| File | Purpose |
|------------------|---------|
| `database.rs` | Implementation of `trait Database`
| `env.rs` | Implementation of `trait Env`
| `error.rs` | Implementation of backend's errors to `cuprate_database`'s error types
| `transaction.rs` | Implementation of `trait RoTx/RwTx`
| `types.rs` | Type aliases for long backend-specific types
# Backends
`cuprate-database`'s `trait`s abstract over various actual databases.
Each database's implementation is located in its respective file in `src/backend/${DATABASE_NAME}.rs`.
## `heed`
The default database used is a modified fork of [`heed`](https://github.com/meilisearch/heed), located at [`Cuprate/heed`](https://github.com/Cuprate/heed).
To generate documentation of the fork for local use:
```bash
git clone --recursive https://github.com/Cuprate/heed
cargo doc
```
`LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
TODO: document max readers limit: https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1372. Other potential processes (e.g. `xmrblocks`) that are also reading the `data.mdb` file need to be accounted for.
## `sanakirja`
TODO
# Layers
TODO: update with accurate information when ready, update image.
## Database
## Trait
## ConcreteEnv
## Thread
## Service
# Resizing
TODO: document resize algorithm:
- Exactly when it occurs
- How much bytes are added
All backends follow the same algorithm.
# Flushing
TODO: document disk flushing behavior.
- Config options
- Backend-specific behavior
# (De)serialization
TODO: document `Pod` and how databases use (de)serialize objects when storing/fetching, essentially using `<[u8], [u8]>`.
Reference in a new issue View git blame Copy permalink