mirror of
https://github.com/hinto-janai/cuprate.git
synced 2025-01-26 20:35:56 +00:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hinto-janai
|
9c27ba5791
|
database: impl service fn bodies (#113)
* write: impl write_block() * ops: add `get_block_info()` * read: impl block fn's * read: fix signatures * service: wrap `ConcreteEnv` in `RwLock` and doc why * heed: use `read-txn-no-tls` for `Send` read transactions * service: remove RwLock, impl some read functions * read: impl `outputs()` * read: flatten indentation, add `thread_local()` * read: impl `number_outputs_with_amount()` * tests: add `AssertTableLen` * ops: replace all table len asserts with `AssertTableLen` * service: initial tests * service: finish most tests * service: fix bad block data in test * tables: fix incorrect doc * service: add `ReadRequest::Outputs` test * read: use macros for set/getting `ThreadLocal`'s based on backend * small fixes * fix review * small fixes * read: fix ThreadLocal macros for `redb` * read: move `Output` mapping to `crate::free` it's needed in tests too * service: check output value correctness in tests * helper: add timelock <-> u64 mapping functions * free: use `u64_to_timelock()` * read: rct outputs * read: fix variable name * read: use ThreadLocal for both backends * heed: use Mutex for `HeedTableRo`'s read tx * block: add miner_tx * heed: remove Table bound oops * Revert "heed: use Mutex for `HeedTableRo`'s read tx" This reverts commit 7e8aae016c55802070ccf7d152aa8966984d7186. * add `UnsafeSendable` * read: use `UnsafeSendable` for `heed`, branch on backend * read: safety docs * cargo.toml: re-add `read-txn-no-tls` for heed * ops: fix tests, remove miner_tx * fix tx_idx calculation, account for RCT outputs in tests * read: docs, fix `get_tables!()` for both backends * fix clippy * database: `unsafe trait DatabaseRo` * tx: use correct tx_id * free: remove miner_tx comment * free: remove `amount` input for rct outputs * ops: split `add_tx` inputs * read: use `UnsafeSendable` for all backends * heed: update safety comment * move output functions `free` -> `ops` * read: fix `chain_height()` handling * remove serde on `UnsafeSendable` * de-dup docs on `trait DatabaseRo`, `get_tables!()` * Update database/src/unsafe_sendable.rs Co-authored-by: Boog900 <boog900@tutanota.com> --------- Co-authored-by: Boog900 <boog900@tutanota.com> |
||
|
hinto-janai
|
c65eb0a3ca
|
database: implement ops/ (#102)
* ops: add `trait MoneroR{o,w}`
* update `trait MoneroR{o,w}` bounds
* types: add `BlockInfoLatest` type alias
* block: impl most core functions
* types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916
* fix table type test
* cargo.toml: add `{cuprate-types, monero-serai}`
* add_block: add all other block data
* ops: remove unneeded `block` functions
* env: add `EnvInner::open_db_rw_all()`
* types: fix test
* block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()`
* add `trait Tables[Mut]` and use it in `EnvInner`
* block: use `TablesMut`
* tables: replace manual impl with `define_trait_tables!()`
* tables: docs for `trait Tables[Mut]`
* tables: doc functions
* create `call_fn_on_all_tables_or_early_return!()` macro
* block: cleanup signatures + bodies
* block: more fn's, docs
* block: add `doc_{error,single,bulk}!()`
* remove `ops/monero.rs`
* move `height()` to `ops/blockchain.rs`
* add `ops/macros.rs`
* tx: add fn signatures
* output: fix fn signatures
* ops: expose `_inner()` functions
* block: add `add_block_header{_bulk, _inner}()`
* ops: remove doc_{fn,inner}!()`
* ops: remove `_{inner,bulk}()`, lifetime + generics
* update lib/mod docs
* ops: add and use `doc_add_block_inner_invariant!()`
* ops: add docs/return to inner `add_block()` functions
* add_block(): extract and use fn for {key_image, output}
* ops: more fn body impl + `add_block()`
* cargo: add `monero-pruning`
* ops: extract out `tx` functions from `add_block()`
* property: add `db_version()`
* ops: `pop_block()` body, remove other `pop_block` fn's
* types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation`
* block: put `block_blob`, pass `Tables` to sub-functions
`impl TablesMut` can't mutably pass multiple tables since
it takes `&mut self`, so all functions unfortunately have to
take a full `&mut impl TablesMut` even though they only need
a few tables.
* database: add `DatabaseRw::take()`
useful for `pop_block()` where we need the value afterwards
* block: deserialize tx's from `block_blobs` in `pop_block()`
* blockchain: `height()` -> `chain_height()`
* output: fix `amount_index`
* ops: fix unlock_time, chain_height
* `BlockInfoV{1,2,3}` -> `BlockInfo`
* constants: add `DATABASE_VERSION`
* database: add `DatabaseRw::update()`
* output: use `DatabaseRw::update()` in `remove_output()`
* add `TxBlobs` table, ignore pruning tables
* block: mostly impl `add_block()` body
* ops: comments
* add_block: miner v2 tx commitment, height cast
* block: impl `pop_block()`
* block: mostly impl `get_block()`
* block: impl `get_block_{from_height,header,header_from_height}`
* add `OutputFlags` bitflags
* add_block: u32::try_into(height: u32), use `OutputFlags`
* tx: impl `get_{tx,tx_from_id}()`
* tx: move docs tests to `#[test]`
testing everything in 1 go is more natural since e.g:
`add_tx()` is followed by `get_tx()`
* tables: add `trait TablesIter`, `all_tables_empty()`
This allows `TablesMut` to be a superset of `Tables`
and use all its accessor functions.
* use cuprate-test-utils, fix tx tests
* block: `add_block()` take block by ref
* tx: use all txs in tests
* output: add `all_tx_functions()` test
* add_block: check current height against input
* block: map more fields in `get_block()`
* block: remove `get_block()`, doc tests, fix `get_block_header()`
* block: dummy values in test
* heed: use `last/first()` instead of `unsafe` cursors
We no longer have DUP semantics and also hard to debug errors
were popping up on `del_current()`...
* heed: fix `DatabaseRw::delete`
Ok(true) means the key did not exist, so we must return
Err(RuntimeError::KeyNotFound)
* block: `add_block()` (dummy value) test
* ops: `key_image` tests
* cleanup, docs, tests
* backend: test `take()` & `update()`
* docs
* remove `OutputFlags::NONE`
* add_block(): add asserts, panic docs, `should_panic` tests
* backend: remove `Ok(())` in `Database::delete` if already deleted
redb already does this, so heed so match
* block: move block operations after tx/outputs
* `amount == 0` -> `amount == 1`
* Nit: StorableVec::wrap_ref
* `saturating_sub(1)` -> `- 1`
* add `TxOutputs` table
* add_block(): add to `tx_outputs` table
* fix `DatabaseRo::update`
* add_tx(): take `block_height` as input
* tx: add/remove from `TxOutputs` table
* output: remove if `amount == 1` -> `amount_index == 0`
* output: fix `add_output()`'s `amount_index` calculation
* output: fix `add_output()`'s `amount_index` calculation again
* output: tests for `amount_index/num_outputs`
* block: `num_outputs - 1` and `take()` -> `get()`
We don't need to `take()` since the call afterwards to
`remove_output()` removes the entry
* block: swap `get_block_extended_header[_from_height]()`
* move `{key_image,output}` handling `add_block()` -> `add_tx()`
* blockchain: add doc to `top_block_height()`
* block: manual panic -> `assert_eq!()`
* test-utils: add `block_blob` to `VerifiedBlockInformation`
field introduced in this PR
* ops: use real block/tx data in tests
* block: `total_generated_coins` -> `cumulative_generated_coins`
* fix clippy, docs, TODOs
* `cumulative_generated_coins()`: `block/` -> `blockchain/`
* blockchain: add `cumulative_generated_coins()` tests
* Update database/src/ops/block.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* `cumulative_generated_coins()` docs for pre-block-0 special case
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
2813c92505
|
database: make open_db_rw() take &TxRw (#104)
* env: take `&TxRw` in `open_db_rw()` * heed: use `UnsafeCell` for write transactions * backend: update tests * add `trait DatabaseIter<T: Table>` * heed: impl `trait DatabaseIter`, inline shared iter `fn`s * env: make `open_db_ro()` return `impl DatabaseIter` * heed: use full path for transaction fn calls * tests: fix tx/table tests * backend: fix redb * heed: `UnsafeCell` -> `RefCell` * docs * remove unneeded `// SAFETY` * Update database/src/backend/heed/env.rs Co-authored-by: Boog900 <boog900@tutanota.com> --------- Co-authored-by: Boog900 <boog900@tutanota.com> |
||
|
hinto-janai
|
21697b8af5
|
database: add more fn to trait DatabaseR{o,w} (#96)
* database: add more fn signatures * database: fix docs * heed: add new fn signatures with `todo!()` * heed: add shared fn sigantures * database: make `iter()` return `(key, value)` * database: make `get_range()` return `(key, value)` * database: make `first()/last()` return `(key, value)` * heed: impl new fn signatures * redb: impl new fn signatures * backend: update tests for new `fn`s * backend: update table tests * heed: fix `retain()` predicate behavior * database: add `keys()`, `values()`, `pop_first()`, `pop_last()` * heed: impl `keys()`, `values()`, `pop_first()`, `pop_last()` * redb: impl `keys()`, `values()`, `pop_first()`, `pop_last()` * database: add `contains()` * backend: add `contains()` to tests * database: remove `retain()` * database: only return value in `get_range()` * database: `DatabaseRw::clear()` -> `EnvInner::clear_db()` * redb: update `clear_db()` invariant doc * fix redb * cargo.toml: avoid merge conflict * update `get_range()` docs * redb: re-create table in `clear_db()` |
||
|
hinto-janai
|
5e7ee57482
|
database: redb 1.5.0 -> redb 2.0.0 (#95)
* `redb 1.5.0` -> `redb 2.0.0`
* `Redb{Key,Value}` -> `redb::{Key,Value}`
* redb: remove unneeded lifetimes
* database: remove read table lifetime
* redb: remove read table lifetime
* heed: remove `'tx` lifetime on read table
* remove `'env, 'tx` lifetime from read/write table
* redb: remove `'env, 'tx` lifetime from read/write table
* heed: remove `'env, 'tx` lifetime from read/write table
* redb: update `TxRo::commit()` docs
|
||
|
hinto-janai
|
3656a1ada7
|
database: return owned T in Storable (#92)
* add `storable_slice.rs` * storable: `from_bytes(&[u8]) -> Self` * heed: return `T` in `Storable`, use `StorableSlice` in tests * remove `value_guard.rs` * redb: update `Storable` to return owned `T` * database: return `T` directly, remove `'a` lifetime * replace `[]` with `StorableSlice` * tables: add lifetime to `tables!()` and generated structs * backend: fix tests * heed: update fn signatures * storable: add `StorableVec`, remove `Storable::ALIGN` * tables/types: remove slice lifetimes, use `StorableVec` * backend: use `StorableVec`, fix tests * types: fix tests to use owned returned value * heed: fix `Storable` impl and tests * storable: `StorableVec` docs + tests * backend: assert values are detached from `get_range()` iterator * database: add docs * udpate docs, remove unneeded clippy `allow` * replace `ToOwnedDebug` with `std::fmt::Debug` * impl `StorableBytes` * backend: add `StorableBytes` to tests * readme: remove `to_owned_debug.rs` reference |
||
|
hinto-janai
|
8f22d8ab79
|
database: impl trait function bodies for heed & redb (#85)
* env: remove `T: Table` for `Env::create_tables()`
It creates _all_ tables, not a specific `T: Table`
* heed: half impl `Env::open()`, some TODOs
* heed: add `HeedTxR{o,w}<'env>`
* workspace/cargo: add `parking_lot`
* remove `parking_lot`
`MappedRwLockGuard` doesn't solve the `returning reference to
object owned by function` error when returning heed's lock guard
+ the tx so we'll be going with `std`
* env: add `{EnvInner,TxRoInput,TxRwInput}` and getter `fn()`s
* env: fix tx <-> table lifetimes, fix `Env::create_tables()`
* heed: impl `DatabaseRo`
* heed: impl `DatabaseRw`
* database: add `src/backend/${BACKEND}/tests.rs`
* heed: impl more of `Env::open()`
* redb: fix trait signatures, add `trait ValueGuard`
* accommodate `DatabaseRo` bounds for both `{heed,redb}`
* fold `get_range()` generic + bounds
* add `TxCreator`, add `heed` tests
* env: `TxCreator` -> `EnvInner` which doubles as DB/Table opener
* database: `DatabaseRw<'tx>` -> `DatabaseRw<'db, 'tx>`
* heed: `db_read_write()` test
* database: fix `get()` lifetimes, heed: add `db_read_write()` test
* database: remove `'env` lifetime from `DatabaseRo<'env, 'tx>`
not needed for immutable references
* redb: fix new `{Env, EnvInner, DatabaseR{o,w}}` bounds
* redb: impl `Database` traits
* redb: impl `TxR{o,w}` traits
* redb: impl `Env`
* redb: open/create tables in `Env::open`
* redb: enable tests, add tmp `Storable` printlns
* redb: fix alignment issue with `Cow` and `from_bytes_unaligned()`
* redb: only allocate bytes when alignment != 1
* heed: remove custom iterator from `range()`
* storable: conditionally allocat on misaligned bytes in `from_bytes`
* add database guard
* database: AccessGuard -> ValueGuard
* storable: add `ALIGN` and `from_bytes_unaligned()`
* redb: 1 serde type `StorableRedb`, fix impl
* cow serde, trait bounds, fix backend's where bounds
- Uses Cow for `redb`'s deserialization
- Conforms `heed` to use Cow (but not as the actual key/value)
- Conforms the `cuprate_database` trait API to use Cow
- Adds `ToOwned + Debug` (and permutation) trait bounds
- Solves 23098573148968945687345349657398 compiler errors due
to aforementioned trait bounds, causing `where` to be everywhere
* fix docs, use fully qualified Tx functions for tests
* backend: check value guard contains value in test
* add `Storable::ALIGN` tests, doc TODOs
* add `trait ToOwnedDebug`
* traits: `ToOwned + Debug` -> `ToOwnedDebug`
* heed: remove `ToOwned` + `Debug` bounds
* redb: remove `ToOwned` + `Debug` bounds
* add `ValueGuard`, replace signatures, fix `redb`
* heed: fix for `ValueGuard`
* docs, tests
* redb: add `CowRange` for `T::Key` -> `Cow<'_, T::Key>` conversion
* separate `config.rs` -> `config/`
* ci: combine tests, run both `heed` and `redb` tests
* ci: fix workflow
* backend: add `resize()` test
* ci: remove windows-specific update
* ci: remove update + windows default set
* backend: add `unreachable` tests, fix docs
* trait docs
* ci: fix
* Update database/src/backend/heed/env.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/env.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/redb/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/redb/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* readme: fix `value_guard.rs`
* heed: remove unneeded clippy + fix formatting
* heed: create and use `create_table()` in `Env::open()`
* redb: create and use `create_table()` in `Env::open()`
* redb: remove unneeded clippy
* fix clippy, remove `<[u8], [u8]>` docs
---------
Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
272ef18eb6
|
database: replace Pod with bytemuck (#81)
* database: add `bytemuck`
* database: add `types` module, replace `Pod` with `bytemuck` traits
* types: docs
* types: more docs
* types: align safety msg
* types: docs
* misc docs
* add `storable.rs`
* add `slice.rs`
* storable: impl `impl_storable_checked_bit_pattern!()`
* database: TODO: fix `DatabaseRo::get_range` lifetimes
* key/table: trait bound fixes
* misc fixes
* remove `borsh`
- Doesn't work on must types
- Probably won't use it anyway
- Most things impl `serde`
* key: add `new_with_max_secondary()`
* key: add `new_with_max_secondary()`
* heed: add `StorableHeed` for `Storable` compat
* redb: add `StorableRedb{Key,Value}` for `Storable` compat
* storable: add `Debug` bound and `fixed_width()`
* redb: fix `'static` bound
* storable: docs
* `pod.rs` tests -> `storable.rs`
* redb: add `Storable` tests
* storable: add doc tests
* redb: simplify `Storable` tests
* heed: add `Storable` tests
* misc docs/fixes
* cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9`
* update readme
* docs
* fix README
* table: remove `CONSTANT_SIZE`
* database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
|
||
|
hinto-janai
|
240e579066
|
database: replace sanakirja with redb (#80)
* cargo: replace `sanakirja` with `redb`
* database: update docs `sanakirja` -> `redb`
* lib: add TODO for `ConcreteEnv` generic replacement
* database: split `trait Database` -> `trait Database{Read,Write}`
* heed: add `struct HeedTable{Ro,Rw}` to match `redb` behavior
* ops: remove imports for now
* env: fix `&mut` bound on RwTx
* database: impl `redb`, type-checks
* fix heed trait impls, `Database{Read,Write}` -> `Database{Ro,Rw}`
* redb: impl `From<_>` for `RuntimeError`
* update readme
* heed: document `HeedTableR{o,w}` types
* env: doc `sync()` invariant
* database: document data & lock filenames
* misc docs, `redb` durability impl, `'db` -> `'env`
* redb: fixes
* misc docs and fixes
* Update database/README.md
Co-authored-by: Boog900 <boog900@tutanota.com>
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
331d4d3e7f
|
Database (#35)
* rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos |