mirror of
https://github.com/Cuprate/cuprate.git
synced 2025-01-04 01:49:27 +00:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hinto-janai
|
ad7b750d76
|
database: impl service::{Request,Response} mappings (#101)
* add `cuprate-types`
* remove `cuprate_database::service::{request,response}`
* use `cuprate_types::service::{request,response}`
* service: fix `Request` `match`'s
* service: create `ReadRequest` function mappings
* service: create `WriteRequest` function mappings
* service: add rough `WriteRequest` retry loop
* service: handle `RuntimeError::ResizeNeeded` in writer
* add `{R,r}o` exception to typos
* docs
* env: make `resize_map()` return new memory map byte size
* write: proactively handle resizes
`add_block()` takes `VerifiedBlockInformation` such
that it can just take the inner blobs of data.
This is a problem when reactively resizing since we no longer
have the block struct we just gave away so we're forced to `.clone()`
each retry.
Instead of that - we will proactively resize so the resize error
will never occur in the first place.
* read: use type aliases
* docs
* fix import
* write: handle resizes reactively
* service: panic if response can't be sent back
* write: add loop unreachable asserts
* service: print and drop error instead of panic
* write: fix retry loop off-by-1
* write: fix docs
* review changes
* update readme
* remove `BlockBatchInRange` request/response
* Update database/README.md
Co-authored-by: Boog900 <boog900@tutanota.com>
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
8f22d8ab79
|
database: impl trait function bodies for heed & redb (#85)
* env: remove `T: Table` for `Env::create_tables()`
It creates _all_ tables, not a specific `T: Table`
* heed: half impl `Env::open()`, some TODOs
* heed: add `HeedTxR{o,w}<'env>`
* workspace/cargo: add `parking_lot`
* remove `parking_lot`
`MappedRwLockGuard` doesn't solve the `returning reference to
object owned by function` error when returning heed's lock guard
+ the tx so we'll be going with `std`
* env: add `{EnvInner,TxRoInput,TxRwInput}` and getter `fn()`s
* env: fix tx <-> table lifetimes, fix `Env::create_tables()`
* heed: impl `DatabaseRo`
* heed: impl `DatabaseRw`
* database: add `src/backend/${BACKEND}/tests.rs`
* heed: impl more of `Env::open()`
* redb: fix trait signatures, add `trait ValueGuard`
* accommodate `DatabaseRo` bounds for both `{heed,redb}`
* fold `get_range()` generic + bounds
* add `TxCreator`, add `heed` tests
* env: `TxCreator` -> `EnvInner` which doubles as DB/Table opener
* database: `DatabaseRw<'tx>` -> `DatabaseRw<'db, 'tx>`
* heed: `db_read_write()` test
* database: fix `get()` lifetimes, heed: add `db_read_write()` test
* database: remove `'env` lifetime from `DatabaseRo<'env, 'tx>`
not needed for immutable references
* redb: fix new `{Env, EnvInner, DatabaseR{o,w}}` bounds
* redb: impl `Database` traits
* redb: impl `TxR{o,w}` traits
* redb: impl `Env`
* redb: open/create tables in `Env::open`
* redb: enable tests, add tmp `Storable` printlns
* redb: fix alignment issue with `Cow` and `from_bytes_unaligned()`
* redb: only allocate bytes when alignment != 1
* heed: remove custom iterator from `range()`
* storable: conditionally allocat on misaligned bytes in `from_bytes`
* add database guard
* database: AccessGuard -> ValueGuard
* storable: add `ALIGN` and `from_bytes_unaligned()`
* redb: 1 serde type `StorableRedb`, fix impl
* cow serde, trait bounds, fix backend's where bounds
- Uses Cow for `redb`'s deserialization
- Conforms `heed` to use Cow (but not as the actual key/value)
- Conforms the `cuprate_database` trait API to use Cow
- Adds `ToOwned + Debug` (and permutation) trait bounds
- Solves 23098573148968945687345349657398 compiler errors due
to aforementioned trait bounds, causing `where` to be everywhere
* fix docs, use fully qualified Tx functions for tests
* backend: check value guard contains value in test
* add `Storable::ALIGN` tests, doc TODOs
* add `trait ToOwnedDebug`
* traits: `ToOwned + Debug` -> `ToOwnedDebug`
* heed: remove `ToOwned` + `Debug` bounds
* redb: remove `ToOwned` + `Debug` bounds
* add `ValueGuard`, replace signatures, fix `redb`
* heed: fix for `ValueGuard`
* docs, tests
* redb: add `CowRange` for `T::Key` -> `Cow<'_, T::Key>` conversion
* separate `config.rs` -> `config/`
* ci: combine tests, run both `heed` and `redb` tests
* ci: fix workflow
* backend: add `resize()` test
* ci: remove windows-specific update
* ci: remove update + windows default set
* backend: add `unreachable` tests, fix docs
* trait docs
* ci: fix
* Update database/src/backend/heed/env.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/env.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/transaction.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/redb/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/redb/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/backend/heed/database.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* readme: fix `value_guard.rs`
* heed: remove unneeded clippy + fix formatting
* heed: create and use `create_table()` in `Env::open()`
* redb: create and use `create_table()` in `Env::open()`
* redb: remove unneeded clippy
* fix clippy, remove `<[u8], [u8]>` docs
---------
Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
272ef18eb6
|
database: replace Pod with bytemuck (#81)
* database: add `bytemuck`
* database: add `types` module, replace `Pod` with `bytemuck` traits
* types: docs
* types: more docs
* types: align safety msg
* types: docs
* misc docs
* add `storable.rs`
* add `slice.rs`
* storable: impl `impl_storable_checked_bit_pattern!()`
* database: TODO: fix `DatabaseRo::get_range` lifetimes
* key/table: trait bound fixes
* misc fixes
* remove `borsh`
- Doesn't work on must types
- Probably won't use it anyway
- Most things impl `serde`
* key: add `new_with_max_secondary()`
* key: add `new_with_max_secondary()`
* heed: add `StorableHeed` for `Storable` compat
* redb: add `StorableRedb{Key,Value}` for `Storable` compat
* storable: add `Debug` bound and `fixed_width()`
* redb: fix `'static` bound
* storable: docs
* `pod.rs` tests -> `storable.rs`
* redb: add `Storable` tests
* storable: add doc tests
* redb: simplify `Storable` tests
* heed: add `Storable` tests
* misc docs/fixes
* cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9`
* update readme
* docs
* fix README
* table: remove `CONSTANT_SIZE`
* database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
|
||
|
hinto-janai
|
240e579066
|
database: replace sanakirja with redb (#80)
* cargo: replace `sanakirja` with `redb`
* database: update docs `sanakirja` -> `redb`
* lib: add TODO for `ConcreteEnv` generic replacement
* database: split `trait Database` -> `trait Database{Read,Write}`
* heed: add `struct HeedTable{Ro,Rw}` to match `redb` behavior
* ops: remove imports for now
* env: fix `&mut` bound on RwTx
* database: impl `redb`, type-checks
* fix heed trait impls, `Database{Read,Write}` -> `Database{Ro,Rw}`
* redb: impl `From<_>` for `RuntimeError`
* update readme
* heed: document `HeedTableR{o,w}` types
* env: doc `sync()` invariant
* database: document data & lock filenames
* misc docs, `redb` durability impl, `'db` -> `'env`
* redb: fixes
* misc docs and fixes
* Update database/README.md
Co-authored-by: Boog900 <boog900@tutanota.com>
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
|
||
|
hinto-janai
|
51d9ccd02d
|
database: Resizes, Shutdown, Flushing (#68)
* error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std:🧵:Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com> |