mirror of
https://github.com/hinto-janai/cuprate.git
synced 2025-01-10 12:54:44 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hinto-janai
|
4876e159e7
|
repo: add skeleton directories, fix labels (#139)
* `database/` -> `storage/`, create skeleton `storage/` crates * add skeleton root directories * books: add skeleton * ci: fix labeler.yml * fix CONTRIBUTING.md * fix workspace Cargo.toml * fix `storage/` * rename helper fs fn, `cuprate_database` -> `cuprate_blockchain` * fix Cargo.toml * cuprate-blockchain: revert find-and-replace Since it has to be ported to the book anyway, there's not much reason to update this and create review diffs. * labeler.yml: remove `A-docs` from `books/` changes Although books are documentation, the `A-books` label already exists for this |
||
|
hinto-janai
|
38ede35468
|
database: more README docs (#126)
* tables: change order * readme: add `Schema` * readme: use type aliases * readme: add `10. Known issues and tradeoffs` * !! * readme: add `The service` section * readme: fill `Known issues and tradeoffs` section * readme: fixes * readme: fixes * readme: review fixes |
||
|
hinto-janai
|
fb3d41ccbb
|
database: final docs + cleanup (#117)
* re-apply 'main' merge + doc patches * fix redb lints * update readme * add `lib.rs, ops, service` doc-test examples * docs for `config`, `ops`, add doc-tests * remove merge error incorrect leftover code from previous merge * doc top-level types * docs: error, tables, types * misc docs, TODO, FIXME, SOMEDAY fixes * change clippy lints * tests: add `tables_are_sorted()` * move `tables_are_sorted()` test to `backend/tests.rs` * readme formatting * small fixes * readme fixes * docs: `helper/` * docs: `types/` * database/README.md fixes * doc fixes * types: doc fixes * fixes * all review changes |
||
|
hinto-janai
|
9c27ba5791
|
database: impl service fn bodies (#113)
* write: impl write_block() * ops: add `get_block_info()` * read: impl block fn's * read: fix signatures * service: wrap `ConcreteEnv` in `RwLock` and doc why * heed: use `read-txn-no-tls` for `Send` read transactions * service: remove RwLock, impl some read functions * read: impl `outputs()` * read: flatten indentation, add `thread_local()` * read: impl `number_outputs_with_amount()` * tests: add `AssertTableLen` * ops: replace all table len asserts with `AssertTableLen` * service: initial tests * service: finish most tests * service: fix bad block data in test * tables: fix incorrect doc * service: add `ReadRequest::Outputs` test * read: use macros for set/getting `ThreadLocal`'s based on backend * small fixes * fix review * small fixes * read: fix ThreadLocal macros for `redb` * read: move `Output` mapping to `crate::free` it's needed in tests too * service: check output value correctness in tests * helper: add timelock <-> u64 mapping functions * free: use `u64_to_timelock()` * read: rct outputs * read: fix variable name * read: use ThreadLocal for both backends * heed: use Mutex for `HeedTableRo`'s read tx * block: add miner_tx * heed: remove Table bound oops * Revert "heed: use Mutex for `HeedTableRo`'s read tx" This reverts commit 7e8aae016c55802070ccf7d152aa8966984d7186. * add `UnsafeSendable` * read: use `UnsafeSendable` for `heed`, branch on backend * read: safety docs * cargo.toml: re-add `read-txn-no-tls` for heed * ops: fix tests, remove miner_tx * fix tx_idx calculation, account for RCT outputs in tests * read: docs, fix `get_tables!()` for both backends * fix clippy * database: `unsafe trait DatabaseRo` * tx: use correct tx_id * free: remove miner_tx comment * free: remove `amount` input for rct outputs * ops: split `add_tx` inputs * read: use `UnsafeSendable` for all backends * heed: update safety comment * move output functions `free` -> `ops` * read: fix `chain_height()` handling * remove serde on `UnsafeSendable` * de-dup docs on `trait DatabaseRo`, `get_tables!()` * Update database/src/unsafe_sendable.rs Co-authored-by: Boog900 <boog900@tutanota.com> --------- Co-authored-by: Boog900 <boog900@tutanota.com> |
||
|
hinto-janai
|
c65eb0a3ca
|
database: implement ops/ (#102)
* ops: add `trait MoneroR{o,w}`
* update `trait MoneroR{o,w}` bounds
* types: add `BlockInfoLatest` type alias
* block: impl most core functions
* types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916
* fix table type test
* cargo.toml: add `{cuprate-types, monero-serai}`
* add_block: add all other block data
* ops: remove unneeded `block` functions
* env: add `EnvInner::open_db_rw_all()`
* types: fix test
* block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()`
* add `trait Tables[Mut]` and use it in `EnvInner`
* block: use `TablesMut`
* tables: replace manual impl with `define_trait_tables!()`
* tables: docs for `trait Tables[Mut]`
* tables: doc functions
* create `call_fn_on_all_tables_or_early_return!()` macro
* block: cleanup signatures + bodies
* block: more fn's, docs
* block: add `doc_{error,single,bulk}!()`
* remove `ops/monero.rs`
* move `height()` to `ops/blockchain.rs`
* add `ops/macros.rs`
* tx: add fn signatures
* output: fix fn signatures
* ops: expose `_inner()` functions
* block: add `add_block_header{_bulk, _inner}()`
* ops: remove doc_{fn,inner}!()`
* ops: remove `_{inner,bulk}()`, lifetime + generics
* update lib/mod docs
* ops: add and use `doc_add_block_inner_invariant!()`
* ops: add docs/return to inner `add_block()` functions
* add_block(): extract and use fn for {key_image, output}
* ops: more fn body impl + `add_block()`
* cargo: add `monero-pruning`
* ops: extract out `tx` functions from `add_block()`
* property: add `db_version()`
* ops: `pop_block()` body, remove other `pop_block` fn's
* types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation`
* block: put `block_blob`, pass `Tables` to sub-functions
`impl TablesMut` can't mutably pass multiple tables since
it takes `&mut self`, so all functions unfortunately have to
take a full `&mut impl TablesMut` even though they only need
a few tables.
* database: add `DatabaseRw::take()`
useful for `pop_block()` where we need the value afterwards
* block: deserialize tx's from `block_blobs` in `pop_block()`
* blockchain: `height()` -> `chain_height()`
* output: fix `amount_index`
* ops: fix unlock_time, chain_height
* `BlockInfoV{1,2,3}` -> `BlockInfo`
* constants: add `DATABASE_VERSION`
* database: add `DatabaseRw::update()`
* output: use `DatabaseRw::update()` in `remove_output()`
* add `TxBlobs` table, ignore pruning tables
* block: mostly impl `add_block()` body
* ops: comments
* add_block: miner v2 tx commitment, height cast
* block: impl `pop_block()`
* block: mostly impl `get_block()`
* block: impl `get_block_{from_height,header,header_from_height}`
* add `OutputFlags` bitflags
* add_block: u32::try_into(height: u32), use `OutputFlags`
* tx: impl `get_{tx,tx_from_id}()`
* tx: move docs tests to `#[test]`
testing everything in 1 go is more natural since e.g:
`add_tx()` is followed by `get_tx()`
* tables: add `trait TablesIter`, `all_tables_empty()`
This allows `TablesMut` to be a superset of `Tables`
and use all its accessor functions.
* use cuprate-test-utils, fix tx tests
* block: `add_block()` take block by ref
* tx: use all txs in tests
* output: add `all_tx_functions()` test
* add_block: check current height against input
* block: map more fields in `get_block()`
* block: remove `get_block()`, doc tests, fix `get_block_header()`
* block: dummy values in test
* heed: use `last/first()` instead of `unsafe` cursors
We no longer have DUP semantics and also hard to debug errors
were popping up on `del_current()`...
* heed: fix `DatabaseRw::delete`
Ok(true) means the key did not exist, so we must return
Err(RuntimeError::KeyNotFound)
* block: `add_block()` (dummy value) test
* ops: `key_image` tests
* cleanup, docs, tests
* backend: test `take()` & `update()`
* docs
* remove `OutputFlags::NONE`
* add_block(): add asserts, panic docs, `should_panic` tests
* backend: remove `Ok(())` in `Database::delete` if already deleted
redb already does this, so heed so match
* block: move block operations after tx/outputs
* `amount == 0` -> `amount == 1`
* Nit: StorableVec::wrap_ref
* `saturating_sub(1)` -> `- 1`
* add `TxOutputs` table
* add_block(): add to `tx_outputs` table
* fix `DatabaseRo::update`
* add_tx(): take `block_height` as input
* tx: add/remove from `TxOutputs` table
* output: remove if `amount == 1` -> `amount_index == 0`
* output: fix `add_output()`'s `amount_index` calculation
* output: fix `add_output()`'s `amount_index` calculation again
* output: tests for `amount_index/num_outputs`
* block: `num_outputs - 1` and `take()` -> `get()`
We don't need to `take()` since the call afterwards to
`remove_output()` removes the entry
* block: swap `get_block_extended_header[_from_height]()`
* move `{key_image,output}` handling `add_block()` -> `add_tx()`
* blockchain: add doc to `top_block_height()`
* block: manual panic -> `assert_eq!()`
* test-utils: add `block_blob` to `VerifiedBlockInformation`
field introduced in this PR
* ops: use real block/tx data in tests
* block: `total_generated_coins` -> `cumulative_generated_coins`
* fix clippy, docs, TODOs
* `cumulative_generated_coins()`: `block/` -> `blockchain/`
* blockchain: add `cumulative_generated_coins()` tests
* Update database/src/ops/block.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* `cumulative_generated_coins()` docs for pre-block-0 special case
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
de931f8630
|
database: implement tables and types (#91)
* table: set `Key` and `Value` to `?Sized`
* types: add `SCHEMA.md` types
* tables: add `SCHEMA.md` tables
* redb: let value be `?Sized`
* types: add casting doc tests and `pub`
* backend: create all tables in `Env::open()`
* backend: open all tables in `tests.rs`
* remove `TestType{1,2}`, use `Output` in tests
* backend: add tests for all tables/keys/values
* add `NumOutputs` table
* small doc fixes
* tables: doc
* types: add `PreRctOutputId`
* types: `BlockInfoV2 { cumulative_rct_outs: u32 -> u64 }`
|
||
|
hinto-janai
|
272ef18eb6
|
database: replace Pod with bytemuck (#81)
* database: add `bytemuck`
* database: add `types` module, replace `Pod` with `bytemuck` traits
* types: docs
* types: more docs
* types: align safety msg
* types: docs
* misc docs
* add `storable.rs`
* add `slice.rs`
* storable: impl `impl_storable_checked_bit_pattern!()`
* database: TODO: fix `DatabaseRo::get_range` lifetimes
* key/table: trait bound fixes
* misc fixes
* remove `borsh`
- Doesn't work on must types
- Probably won't use it anyway
- Most things impl `serde`
* key: add `new_with_max_secondary()`
* key: add `new_with_max_secondary()`
* heed: add `StorableHeed` for `Storable` compat
* redb: add `StorableRedb{Key,Value}` for `Storable` compat
* storable: add `Debug` bound and `fixed_width()`
* redb: fix `'static` bound
* storable: docs
* `pod.rs` tests -> `storable.rs`
* redb: add `Storable` tests
* storable: add doc tests
* redb: simplify `Storable` tests
* heed: add `Storable` tests
* misc docs/fixes
* cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9`
* update readme
* docs
* fix README
* table: remove `CONSTANT_SIZE`
* database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
|
||
|
hinto-janai
|
51d9ccd02d
|
database: Resizes, Shutdown, Flushing (#68)
* error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std:🧵:Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com> |
||
|
hinto-janai
|
331d4d3e7f
|
Database (#35)
* rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos |