Cuprate/cuprate
Cuprate/cuprate
1
0
Fork 0
mirror of https://github.com/Cuprate/cuprate.git synced 2024-12-23 03:59:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
cuprate/database/README.md

138 lines
5.1 KiB
Markdown
Raw Normal View History

Database (#35) * rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos
2024-02-13 17:43:25 +00:00
# Database
Cuprate's database implementation.
<!-- Did you know markdown automatically increments number lists, even if they are all 1...? -->
1. [Documentation](#documentation)
1. [File Structure](#file-structure)
- [`src/`](#src)
- [`src/service/`](#src-service)
- [`src/backend/`](#src-backend)
1. [Backends](#backends)
- [`heed`](#heed)
- [`sanakirja`](#sanakirja)
1. [Layers](#layers)
- [Database](#database)
- [Trait](#trait)
- [ConcreteDatabase](#concretedatabase)
- [Thread-pool](#thread-pool)
- [Service](#service)
---
# Documentation
In general, documentation for `database/` is split into 3:
| Documentation location | Purpose |
|---------------------------|---------|
| `database/README.md` | High level design of `cuprate-database`
| `cuprate-database` | Practical usage documentation/warnings/notes/etc
| Source file `// comments` | Implementation-specific details (e.g, how many reader threads to spawn?)
This README serves as the overview/design document.
For actual practical usage, `cuprate-database`'s types and general usage are documented via standard Rust tooling.
Run:
```bash
cargo doc --package cuprate-database --open
```
at the root of the repo to open/read the documentation.
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many `// Regular comments` that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
The code within `src/` is also littered with some `grep`-able comments containing some keywords:
| Word | Meaning |
|-------------|---------|
| `INVARIANT` | This code makes an _assumption_ that must be upheld for correctness
| `SAFETY` | This `unsafe` code is okay, for `x,y,z` reasons
| `FIXME` | This code works but isn't ideal
| `HACK` | This code is a brittle workaround
| `PERF` | This code is weird for performance reasons
| `TODO` | This has to be implemented
| `SOMEDAY` | This should be implemented... someday
# File Structure
A quick reference of the structure of the folders & files in `cuprate-database`.
Note that `lib.rs/mod.rs` files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding `mod.rs`.
## `src/`
The top-level `src/` files.
| File/Folder | Purpose |
|------------------|---------|
| `constants.rs` | General constants used throughout `cuprate-database`
| `database.rs` | Abstracted database; `trait Database`
| `env.rs` | Abstracted database environment; `trait Env`
| `error.rs` | Database error types
| `free.rs` | General free functions (related to the database)
| `key.rs` | Abstracted database keys; `trait Key`
| `pod.rs` | Data (de)serialization; `trait Pod`
| `table.rs` | Database table abstraction; `trait Table`
| `tables.rs` | All the table definitions used by `cuprate-database`
| `transaction.rs` | Database transaction abstraction; `trait RoTx`, `trait RwTx`
## `src/service/`
This folder contains the `cupate_database::service` module.
| File/Folder | Purpose |
|----------------|---------|
| `free.rs` | General free functions used (related to `cuprate_database::service`)
| `read.rs` | Read thread-pool definitions and logic
| `request.rs` | Read/write `Request`s to the database
| `response.rs` | Read/write `Response`'s from the database
| `write.rs` | Write thread-pool definitions and logic
## `src/backend/`
This folder contains the actual database crates used as the backend for `cuprate-database`.
Each backend has its own folder.
| File/Folder | Purpose |
|--------------|---------|
| `heed/` | Backend using using forked [`heed`](https://github.com/Cuprate/heed)
| `sanakirja/` | Backend using [`sanakirja`](https://docs.rs/sanakirja)
### `src/backend/heed/`
| File/Folder | Purpose |
|------------------|---------|
| `database.rs` | Implementation of `trait Database`
| `env.rs` | Implementation of `trait Env`
| `serde.rs` | Data (de)serialization implementations
| `transaction.rs` | Implementation of `trait RoTx/RwTx`
### `src/backend/sanakirja/`
| File/Folder | Purpose |
|------------------|---------|
| `database.rs` | Implementation of `trait Database`
| `env.rs` | Implementation of `trait Env`
| `transaction.rs` | Implementation of `trait RoTx/RwTx`
# Backends
`cuprate-database`'s `trait`s abstract over various actual databases.
Each database's implementation is located in its respective file in `src/backend/${DATABASE_NAME}.rs`.
## `heed`
The default database used is a modified fork of [`heed`](https://github.com/meilisearch/heed), located at [`Cuprate/heed`](https://github.com/Cuprate/heed).
To generate documentation of the fork for local use:
```bash
git clone --recursive https://github.com/Cuprate/heed
cargo doc
```
`LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
## `sanakirja`
TODO
# Layers
TODO: update with accurate information when ready, update image.
## Database
## Trait
## ConcreteDatabase
## Thread
## Service
Reference in a new issue Copy permalink