Commit graph

10 commits

Author SHA1 Message Date
hinto.janai
65d3a5b0c6
Merge branch 'main' into test 2024-04-16 16:53:40 -04:00
hinto.janai
ba5a495a1d
add README.md 2024-03-28 16:03:05 -04:00
hinto-janai
ae0d9ff36a
database: add redb-memory backend (#97)
* add `redb-memory` feature

* redb: use `redb::backend::InMemoryBackend` if enabled

* readme: add `redb-memory` section

* ci: add `redb-memory` testing

* ci: remove `redb-memory` testing

probably not worth adding time to CI for this
2024-03-27 00:50:10 +00:00
hinto-janai
3656a1ada7
database: return owned T in Storable (#92)
* add `storable_slice.rs`

* storable: `from_bytes(&[u8]) -> Self`

* heed: return `T` in `Storable`, use `StorableSlice` in tests

* remove `value_guard.rs`

* redb: update `Storable` to return owned `T`

* database: return `T` directly, remove `'a` lifetime

* replace `[]` with `StorableSlice`

* tables: add lifetime to `tables!()` and generated structs

* backend: fix tests

* heed: update fn signatures

* storable: add `StorableVec`, remove `Storable::ALIGN`

* tables/types: remove slice lifetimes, use `StorableVec`

* backend: use `StorableVec`, fix tests

* types: fix tests to use owned returned value

* heed: fix `Storable` impl and tests

* storable: `StorableVec` docs + tests

* backend: assert values are detached from `get_range()` iterator

* database: add docs

* udpate docs, remove unneeded clippy `allow`

* replace `ToOwnedDebug` with `std::fmt::Debug`

* impl `StorableBytes`

* backend: add `StorableBytes` to tests

* readme: remove `to_owned_debug.rs` reference
2024-03-22 21:11:48 +00:00
hinto-janai
8f22d8ab79
database: impl trait function bodies for heed & redb (#85)
* env: remove `T: Table` for `Env::create_tables()`

It creates _all_ tables, not a specific `T: Table`

* heed: half impl `Env::open()`, some TODOs

* heed: add `HeedTxR{o,w}<'env>`

* workspace/cargo: add `parking_lot`

* remove `parking_lot`

`MappedRwLockGuard` doesn't solve the `returning reference to
object owned by function` error when returning heed's lock guard
+ the tx so we'll be going with `std`

* env: add `{EnvInner,TxRoInput,TxRwInput}` and getter `fn()`s

* env: fix tx <-> table lifetimes, fix `Env::create_tables()`

* heed: impl `DatabaseRo`

* heed: impl `DatabaseRw`

* database: add `src/backend/${BACKEND}/tests.rs`

* heed: impl more of `Env::open()`

* redb: fix trait signatures, add `trait ValueGuard`

* accommodate `DatabaseRo` bounds for both `{heed,redb}`

* fold `get_range()` generic + bounds

* add `TxCreator`, add `heed` tests

* env: `TxCreator` -> `EnvInner` which doubles as DB/Table opener

* database: `DatabaseRw<'tx>` -> `DatabaseRw<'db, 'tx>`

* heed: `db_read_write()` test

* database: fix `get()` lifetimes, heed: add `db_read_write()` test

* database: remove `'env` lifetime from `DatabaseRo<'env, 'tx>`

not needed for immutable references

* redb: fix new `{Env, EnvInner, DatabaseR{o,w}}` bounds

* redb: impl `Database` traits

* redb: impl `TxR{o,w}` traits

* redb: impl `Env`

* redb: open/create tables in `Env::open`

* redb: enable tests, add tmp `Storable` printlns

* redb: fix alignment issue with `Cow` and `from_bytes_unaligned()`

* redb: only allocate bytes when alignment != 1

* heed: remove custom iterator from `range()`

* storable: conditionally allocat on misaligned bytes in `from_bytes`

* add database guard

* database: AccessGuard -> ValueGuard

* storable: add `ALIGN` and `from_bytes_unaligned()`

* redb: 1 serde type `StorableRedb`, fix impl

* cow serde, trait bounds, fix backend's where bounds

- Uses Cow for `redb`'s deserialization
- Conforms `heed` to use Cow (but not as the actual key/value)
- Conforms the `cuprate_database` trait API to use Cow
- Adds `ToOwned + Debug` (and permutation) trait bounds
- Solves 23098573148968945687345349657398 compiler errors due
  to aforementioned trait bounds, causing `where` to be everywhere

* fix docs, use fully qualified Tx functions for tests

* backend: check value guard contains value in test

* add `Storable::ALIGN` tests, doc TODOs

* add `trait ToOwnedDebug`

* traits: `ToOwned + Debug` -> `ToOwnedDebug`

* heed: remove `ToOwned` + `Debug` bounds

* redb: remove `ToOwned` + `Debug` bounds

* add `ValueGuard`, replace signatures, fix `redb`

* heed: fix for `ValueGuard`

* docs, tests

* redb: add `CowRange` for `T::Key` -> `Cow<'_, T::Key>` conversion

* separate `config.rs` -> `config/`

* ci: combine tests, run both `heed` and `redb` tests

* ci: fix workflow

* backend: add `resize()` test

* ci: remove windows-specific update

* ci: remove update + windows default set

* backend: add `unreachable` tests, fix docs

* trait docs

* ci: fix

* Update database/src/backend/heed/env.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/heed/env.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/heed/transaction.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/heed/transaction.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/heed/transaction.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/redb/database.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/redb/database.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/backend/heed/database.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* readme: fix `value_guard.rs`

* heed: remove unneeded clippy + fix formatting

* heed: create and use `create_table()` in `Env::open()`

* redb: create and use `create_table()` in `Env::open()`

* redb: remove unneeded clippy

* fix clippy, remove `<[u8], [u8]>` docs

---------

Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
Co-authored-by: Boog900 <boog900@tutanota.com>
2024-03-13 22:05:24 +00:00
hinto-janai
272ef18eb6
database: replace Pod with bytemuck (#81)
* database: add `bytemuck`

* database: add `types` module, replace `Pod` with `bytemuck` traits

* types: docs

* types: more docs

* types: align safety msg

* types: docs

* misc docs

* add `storable.rs`

* add `slice.rs`

* storable: impl `impl_storable_checked_bit_pattern!()`

* database: TODO: fix `DatabaseRo::get_range` lifetimes

* key/table: trait bound fixes

* misc fixes

* remove `borsh`

- Doesn't work on must types
- Probably won't use it anyway
- Most things impl `serde`

* key: add `new_with_max_secondary()`

* key: add `new_with_max_secondary()`

* heed: add `StorableHeed` for `Storable` compat

* redb: add `StorableRedb{Key,Value}` for `Storable` compat

* storable: add `Debug` bound and `fixed_width()`

* redb: fix `'static` bound

* storable: docs

* `pod.rs` tests -> `storable.rs`

* redb: add `Storable` tests

* storable: add doc tests

* redb: simplify `Storable` tests

* heed: add `Storable` tests

* misc docs/fixes

* cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9`

* update readme

* docs

* fix README

* table: remove `CONSTANT_SIZE`

* database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
2024-03-03 22:26:39 +00:00
hinto-janai
240e579066
database: replace sanakirja with redb (#80)
* cargo: replace `sanakirja` with `redb`

* database: update docs `sanakirja` -> `redb`

* lib: add TODO for `ConcreteEnv` generic replacement

* database: split `trait Database` -> `trait Database{Read,Write}`

* heed: add `struct HeedTable{Ro,Rw}` to match `redb` behavior

* ops: remove imports for now

* env: fix `&mut` bound on RwTx

* database: impl `redb`, type-checks

* fix heed trait impls, `Database{Read,Write}` -> `Database{Ro,Rw}`

* redb: impl `From<_>` for `RuntimeError`

* update readme

* heed: document `HeedTableR{o,w}` types

* env: doc `sync()` invariant

* database: document data & lock filenames

* misc docs, `redb` durability impl, `'db` -> `'env`

* redb: fixes

* misc docs and fixes

* Update database/README.md

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-29 17:40:15 +00:00
hinto-janai
51d9ccd02d
database: Resizes, Shutdown, Flushing (#68)
* error: add `NeedsResize`

accidently removed, was `MapFull` before.

We need this because we as the writer thread must
react to this error in order to resize.

The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist

* env/backend: add `MANUAL_RESIZE` and `resize()`

* heed: respect `ReadersFull`, comment errors

* free: add `resize_memory_map()`

* env: add `Env::current_map_size`

* service: add `resize_map()`

* database: make `Env` itself cheaply clonable + threadsafe

`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.

* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`

* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`

* database: add `src/service/state.rs`

* service: add to `state.rs`, add `DatabaseState` to readers/writer

* add `parking_lot`

Okay, turns out we need to take locks in
`database/src/service/state.rs`...

`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.

* state: use `crossbeam::atomic::AtomicCell`

We have crossbeam as a dep anyway.

* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`

* service: add reader shutdown handle, use `Select` for msgs

* service: remove `state.rs`

We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.

* service: fix misc reader/writer stuff

* database: add `config.rs`

* service: use `ReaderThreads` when spawning readers

* service: impl `shutdown()` for readers/writer

* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`

Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.

If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.

Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.

* remove `parking_lot`

Could be used as the database mutual exclusion lock.

Needs to be tested against `std`.

* config: add `path`

* env: `path()` -> `config()`, backend: impl `Drop`

* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect

* backend: add `Config` to `ConcreteEnv`

* service: use `std:🧵:Builder` for readers/writer

* config: `PathBuf` -> `Cow<'static, Path>`

* misc docs, remove `RuntimeError::ShuttingDown`

* service: init & shutdown docs

* heed: impl `Env::resize_map()`

* heed: impl `Env::current_map_size()`

* lib.rs: add example crate usage test

* heed: `RwLock` comment

* helper: add `cuprate_database_dir()`

* config: use `cuprate_database_dir()`

* lib.rs: TODO example test

* database: add `page_size`

The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772

* free: impl `resize_memory_map()`

* free: docs

* env: add `disk_size_bytes()`

* config: impl `From<$num>` for `ReaderThreads`

* move `fs`-related constants to `cuprate_helper::fs`

* docs

* add `resize.rs`, `ResizeAlgorithm`

* env: use `ResizeAlgorithm` in `resize_map()`

* TODO: account for LMDB reader limit

* resize: docs, add `page_size()`, impl `fixed_bytes()`

* resize: impl `percent()`

* resize: docs

* env: take `ResizeAlgorithm` by value (it impls `Copy`)

* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`

* config: `From<Into<usize>>` for `ReaderThreads`

Co-authored-by: Boog900 <boog900@tutanota.com>

* env: move mutual exclusion doc to backend

* free: update invariant doc

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/service/mod.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* fix `[allow(unused_imports)] // docs`

* move DB filename from `helper/` -> `database/`

* config: use `DATABASE_FILENAME`

* config: add `db_file_path()`

* lib: add non-`service` usage invariant docs

* table: seal `Table` trait, impl for all `crate::tables`

* fix docs

* fix docs pt.2

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00
hinto-janai
f7bd1304e2
database: Errors (#62)
* error: add variants to `RuntimeError`

* error: remove `<BackendError>` generic

we can just map each backend error variant <-> our error as needed

* backend: impl `From<heed::Error>` for `RuntimeError`

* add `Never` type to allow foreign trait implementations

This is a newtype to workaround `sanakirja::Storable` not being
able to be implemented on `std::convert::Infallible` due to
"foreign trait on foreign type" rules.

* revert 0342848, fix `sanakirja` trait bounds

K/V will always be `[u8]`, not the concrete type
so it does not need to be bounded.

* backend: fix `sanakijra` K/V generics

* sanakirja: add `error.rs`

* error: remove serde traits

* heed: add `todo!()` for error mappings

* error: add `Corrupt` variant

* sanakirja: finish error mappings

* heed: finish error mappings

* error: add to error types

* env: use `InitError` in `Env::open()`

* error: docs

* heed: remove `serde.rs`

Not needed if all K/V's stored are `[u8]`.

* heed: use `heed::types::Bytes` as K/V

* database: docs

* heed: impl `From<heed::Error>` for `InitError`

* sanakirja: impl `From<sanakirja::Error>` for `InitError`

* error: fix doc warnings

* heed: fix `clippy::match_same_arms` in error match

* readme: add TODO

* error: add `BoxError`, and fatal/unknown variants

* heed: use `Fatal/Unknown` variants in errors

* sanakirja: use `Fatal/Unknown` variants in errors

* clippy

* sanakijra: remove `fallible_impl_from`

* error: remove `RuntimeError::InvalidVersion`

* error: remove `RuntimeError` variants that should panic

* error: remove `InitError::Fatal`

We will exit on all errors regardless.

Any non-enumrated variants will use `InitError::Unknown`.

* heed: fix error mappings

* constants: add `CUPRATE_DATABASE_CORRUPT_MSG`

* sanakijra: fix error mappings

* heed: fix import

* comments/docs

* key: fix docs
2024-02-17 13:00:14 +00:00
hinto-janai
331d4d3e7f
Database (#35)
* rename `database` -> `old_database`

Keeping it around for reference until new implementation is complete.

* create new `database/` skeleton

* add `DATABASE.md` design doc skeleton

* move design doc to `database/README.md`

* add rough code

* `lib.rs` -> `gist.rs`

* database: use workspace deps

* workspace: include `database/` as member

CI will now include this crate.

* cargo fmt

* database: `AGPL` -> `MIT`

* readme: add `TODO`s

* add base files

* cargo.toml: add `heed` feature

* fix clippy

* lib.rs: add extremely pedantic lints

* readme: add `# Backends`

* cargo.toml: add `cfg-if`

* add `backend/` structure

* base `database.rs`

* cargo.toml: add `borsh`

* backend: add `DATABASE_BACKEND`

* base `error.rs`

* base `database.rs`

* base `transaction.rs`

* base `table.rs`

* lib.rs: add imports

* add `pod.rs`

* pod: use `Read/Write`, add tests for all primitive numbers

* pod: impl Pod for `Vec<u8>`, `[u8; N]`

* pod: add docs, add `private::Sealed`

* pod: `to_writer`, `from_reader`

The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.

Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`

* pod: impl Pod for `Box<[u8]>`

* pod: return `Err` on incorrect length in `from_bytes()`

* pod: docs

* pod: impl Pod for `Arc<[u8]>`

* readme: docs

* database: add `create_table()`, `get_table()`

* table: `Pod` bound

* backend: move into directories

* pod: add `into_bytes()`

* heed: impl `BytesEncode`, `BytesDecode`

* add `actor`, `service` features

The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.

* add `lib.rs` docs

* service: add `service.rs`

* service: add `reader.rs`

* service: add `writer.rs`

* service: add `request.rs` & `response.rs`

* cargo.toml: add `crossbeam`

* service: add/use `enum Request`, `enum Response`

* service: basic signatures for thread-pools, `Writer` -> `Writers`

* service: split `tower::Service<ReadRequest/WriteRequest>`

* service: impl `tower::Service` for `Database(Reader|Writer)`

* service: add `init()`, impl basic `Reader/Writer` pools

* service: add example `Request`'s

* service: add example `ReadRequest` handling

* temporarily allow clippy lints

* readme: update file structure

* transaction: add `RoTx::get_range()`

* service: module docs

* cargo.toml: add `cuprate-helper`

* service: scale readers/writers based on thread count

* database: change lifetimes

* heed: impl Database for `heed`

* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits

* docs

* service: `read.rs` docs

* service: `write.rs` docs

* service: request/response docs

* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`

* service: leak database into `&'static`

* database: add `#[inline]`, `#[cold]`

* service: `free.rs` docs, more `#[inline]` + `#[cold]`

* service: add `shutdown()`

* service: add `Request::Shutdown`

* service: `shutdown()` docs

* heed: hide concrete tx types

* lib.rs: add terms

* split `Env` <-> `Database`

* cargo.toml: add `paste`

* database: add `tables/`

* impl `serde/borsh` where possible

* tables: add `Tables`, add test docs

* make db backend mutually exclusive to fix `--all-features`

* tables: use `$()*` in `tables!()`

* cargo.toml: add `sanakirja 1.4.0`

* sanakirja: impl `Env`

* sanakirja: impl `Database`

* sanakirja: impl `Transaction`

* table: temporarily fix `sanakirja` K/V bounds

* table: fix `#[cfg]`

* cargo.toml: fix deps

* lib.rs: docs

* service: docs

* readme: add files, update `# Documentation`, add `# Layers`

* readme: `src/` file purpose

* readme: `src/service/` file purpose

* readme: `src/backend/` file purpose

* fix `Cargo.lock` merge conflict

* database: remove `gist.rs`

* add to `constants.rs`

* add top `//! comments` for files/modules

* constants: add sanity-check test

* service: add `only_one_database` test in `free.rs`

* service: add `tests.rs`

* remove unneeded markers + imports

* backend: fix `get_range()`'s trait `impl` return

* env: add `create_tables_if_needed()`, don't return `Option<Db>`

* sort imports by `CONTRIBUTING.md` rules

oops sorry boog

* add `monero.rs`

* monero: docs

* database: add missing `RoTx/RwTx` inputs

* backend: add missing `RoTx/RwTx` inputs

* `monero.rs` trait -> free functions in `ops/`

* pod: make methods infallible

* ci: add `rustup update` step

* service: use `Arc` instead of leaking, remove `db_read/db_write`

* service: use `InfallibleOneshotReceiver` for readers

* service: shutdown on error, add todos

* service: remove `Request`

* service: combine `ReadResponse` and `WriteResponse`

* service: use `InfallibleOneshotReceiver` for writer

* service: only spawn 1 writer, don't allow cloning writer handle

* table: add associated `const CONSTANT_SIZE`

* add `key.rs` + `trait Key`, add bound to `Table`

* fix typos
2024-02-13 17:43:25 +00:00