mirror of
https://github.com/hinto-janai/cuprate.git
synced 2025-03-12 09:31:30 +00:00
database/ -> storage/`
This commit is contained in:
hinto.janai
parent
45656fe653
commit
c7c2631457
105 changed files with 10125 additions and 0 deletions
598
storage/cuprate-database/README.md
Normal file
598
storage/cuprate-database/README.md
Normal file
|
|
@ -0,0 +1,598 @@
|
|||
# Database
|
||||
Cuprate's database implementation.
|
||||
|
||||
- [1. Documentation](#1-documentation)
|
||||
- [2. File structure](#2-file-structure)
|
||||
- [2.1 `src/`](#21-src)
|
||||
- [2.2 `src/backend/`](#22-srcbackend)
|
||||
- [2.3 `src/config/`](#23-srcconfig)
|
||||
- [2.4 `src/ops/`](#24-srcops)
|
||||
- [2.5 `src/service/`](#25-srcservice)
|
||||
- [3. Backends](#3-backends)
|
||||
- [3.1 heed](#31-heed)
|
||||
- [3.2 redb](#32-redb)
|
||||
- [3.3 redb-memory](#33-redb-memory)
|
||||
- [3.4 sanakirja](#34-sanakirja)
|
||||
- [3.5 MDBX](#35-mdbx)
|
||||
- [4. Layers](#4-layers)
|
||||
- [4.1 Backend](#41-backend)
|
||||
- [4.2 Trait](#42-trait)
|
||||
- [4.3 ConcreteEnv](#43-concreteenv)
|
||||
- [4.4 ops](#44-ops)
|
||||
- [4.5 service](#45-service)
|
||||
- [5. The service](#5-the-service)
|
||||
- [5.1 Initialization](#51-initialization)
|
||||
- [5.2 Requests](#53-requests)
|
||||
- [5.3 Responses](#54-responses)
|
||||
- [5.4 Thread model](#52-thread-model)
|
||||
- [5.5 Shutdown](#55-shutdown)
|
||||
- [6. Syncing](#6-Syncing)
|
||||
- [7. Resizing](#7-resizing)
|
||||
- [8. (De)serialization](#8-deserialization)
|
||||
- [9. Schema](#9-schema)
|
||||
- [9.1 Tables](#91-tables)
|
||||
- [9.2 Multimap tables](#92-multimap-tables)
|
||||
- [10. Known issues and tradeoffs](#10-known-issues-and-tradeoffs)
|
||||
- [10.1 Traits abstracting backends](#101-traits-abstracting-backends)
|
||||
- [10.2 Hot-swappable backends](#102-hot-swappable-backends)
|
||||
- [10.3 Copying unaligned bytes](#103-copying-unaligned-bytes)
|
||||
- [10.4 Endianness](#104-endianness)
|
||||
- [10.5 Extra table data](#105-extra-table-data)
|
||||
|
||||
---
|
||||
|
||||
## 1. Documentation
|
||||
Documentation for `database/` is split into 3 locations:
|
||||
|
||||
| Documentation location | Purpose |
|
||||
|---------------------------|---------|
|
||||
| `database/README.md` | High level design of `cuprate-database`
|
||||
| `cuprate-database` | Practical usage documentation/warnings/notes/etc
|
||||
| Source file `// comments` | Implementation-specific details (e.g, how many reader threads to spawn?)
|
||||
|
||||
This README serves as the implementation design document.
|
||||
|
||||
For actual practical usage, `cuprate-database`'s types and general usage are documented via standard Rust tooling.
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cargo doc --package cuprate-database --open
|
||||
```
|
||||
at the root of the repo to open/read the documentation.
|
||||
|
||||
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many `// Regular comments` that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
|
||||
|
||||
The code within `src/` is also littered with some `grep`-able comments containing some keywords:
|
||||
|
||||
| Word | Meaning |
|
||||
|-------------|---------|
|
||||
| `INVARIANT` | This code makes an _assumption_ that must be upheld for correctness
|
||||
| `SAFETY` | This `unsafe` code is okay, for `x,y,z` reasons
|
||||
| `FIXME` | This code works but isn't ideal
|
||||
| `HACK` | This code is a brittle workaround
|
||||
| `PERF` | This code is weird for performance reasons
|
||||
| `TODO` | This must be implemented; There should be 0 of these in production code
|
||||
| `SOMEDAY` | This should be implemented... someday
|
||||
|
||||
## 2. File structure
|
||||
A quick reference of the structure of the folders & files in `cuprate-database`.
|
||||
|
||||
Note that `lib.rs/mod.rs` files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding `mod.rs`.
|
||||
|
||||
### 2.1 `src/`
|
||||
The top-level `src/` files.
|
||||
|
||||
| File | Purpose |
|
||||
|------------------------|---------|
|
||||
| `constants.rs` | General constants used throughout `cuprate-database`
|
||||
| `database.rs` | Abstracted database; `trait DatabaseR{o,w}`
|
||||
| `env.rs` | Abstracted database environment; `trait Env`
|
||||
| `error.rs` | Database error types
|
||||
| `free.rs` | General free functions (related to the database)
|
||||
| `key.rs` | Abstracted database keys; `trait Key`
|
||||
| `resize.rs` | Database resizing algorithms
|
||||
| `storable.rs` | Data (de)serialization; `trait Storable`
|
||||
| `table.rs` | Database table abstraction; `trait Table`
|
||||
| `tables.rs` | All the table definitions used by `cuprate-database`
|
||||
| `tests.rs` | Utilities for `cuprate_database` testing
|
||||
| `transaction.rs` | Database transaction abstraction; `trait TxR{o,w}`
|
||||
| `types.rs` | Database-specific types
|
||||
| `unsafe_unsendable.rs` | Marker type to impl `Send` for objects not `Send`
|
||||
|
||||
### 2.2 `src/backend/`
|
||||
This folder contains the implementation for actual databases used as the backend for `cuprate-database`.
|
||||
|
||||
Each backend has its own folder.
|
||||
|
||||
| Folder/File | Purpose |
|
||||
|-------------|---------|
|
||||
| `heed/` | Backend using using [`heed`](https://github.com/meilisearch/heed) (LMDB)
|
||||
| `redb/` | Backend using [`redb`](https://github.com/cberner/redb)
|
||||
| `tests.rs` | Backend-agnostic tests
|
||||
|
||||
All backends follow the same file structure:
|
||||
|
||||
| File | Purpose |
|
||||
|------------------|---------|
|
||||
| `database.rs` | Implementation of `trait DatabaseR{o,w}`
|
||||
| `env.rs` | Implementation of `trait Env`
|
||||
| `error.rs` | Implementation of backend's errors to `cuprate_database`'s error types
|
||||
| `storable.rs` | Compatibility layer between `cuprate_database::Storable` and backend-specific (de)serialization
|
||||
| `transaction.rs` | Implementation of `trait TxR{o,w}`
|
||||
| `types.rs` | Type aliases for long backend-specific types
|
||||
|
||||
### 2.3 `src/config/`
|
||||
This folder contains the `cupate_database::config` module; configuration options for the database.
|
||||
|
||||
| File | Purpose |
|
||||
|---------------------|---------|
|
||||
| `config.rs` | Main database `Config` struct
|
||||
| `reader_threads.rs` | Reader thread configuration for `service` thread-pool
|
||||
| `sync_mode.rs` | Disk sync configuration for backends
|
||||
|
||||
### 2.4 `src/ops/`
|
||||
This folder contains the `cupate_database::ops` module.
|
||||
|
||||
These are higher-level functions abstracted over the database, that are Monero-related.
|
||||
|
||||
| File | Purpose |
|
||||
|-----------------|---------|
|
||||
| `block.rs` | Block related (main functions)
|
||||
| `blockchain.rs` | Blockchain related (height, cumulative values, etc)
|
||||
| `key_image.rs` | Key image related
|
||||
| `macros.rs` | Macros specific to `ops/`
|
||||
| `output.rs` | Output related
|
||||
| `property.rs` | Database properties (pruned, version, etc)
|
||||
| `tx.rs` | Transaction related
|
||||
|
||||
### 2.5 `src/service/`
|
||||
This folder contains the `cupate_database::service` module.
|
||||
|
||||
The `async`hronous request/response API other Cuprate crates use instead of managing the database directly themselves.
|
||||
|
||||
| File | Purpose |
|
||||
|----------------|---------|
|
||||
| `free.rs` | General free functions used (related to `cuprate_database::service`)
|
||||
| `read.rs` | Read thread-pool definitions and logic
|
||||
| `tests.rs` | Thread-pool tests and test helper functions
|
||||
| `types.rs` | `cuprate_database::service`-related type aliases
|
||||
| `write.rs` | Writer thread definitions and logic
|
||||
|
||||
## 3. Backends
|
||||
`cuprate-database`'s `trait`s allow abstracting over the actual database, such that any backend in particular could be used.
|
||||
|
||||
Each database's implementation for those `trait`'s are located in its respective folder in `src/backend/${DATABASE_NAME}/`.
|
||||
|
||||
### 3.1 heed
|
||||
The default database used is [`heed`](https://github.com/meilisearch/heed) (LMDB). The upstream versions from [`crates.io`](https://crates.io/crates/heed) are used. `LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
|
||||
|
||||
`heed`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
|
||||
|
||||
| Filename | Purpose |
|
||||
|------------|---------|
|
||||
| `data.mdb` | Main data file
|
||||
| `lock.mdb` | Database lock file
|
||||
|
||||
`heed`-specific notes:
|
||||
- [There is a maximum reader limit](https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1372). Other potential processes (e.g. `xmrblocks`) that are also reading the `data.mdb` file need to be accounted for
|
||||
- [LMDB does not work on remote filesystem](https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/lmdb.h#L129)
|
||||
|
||||
### 3.2 redb
|
||||
The 2nd database backend is the 100% Rust [`redb`](https://github.com/cberner/redb).
|
||||
|
||||
The upstream versions from [`crates.io`](https://crates.io/crates/redb) are used.
|
||||
|
||||
`redb`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
|
||||
|
||||
| Filename | Purpose |
|
||||
|-------------|---------|
|
||||
| `data.redb` | Main data file
|
||||
|
||||
<!-- TODO: document DB on remote filesystem (does redb allow this?) -->
|
||||
|
||||
### 3.3 redb-memory
|
||||
This backend is 100% the same as `redb`, although, it uses `redb::backend::InMemoryBackend` which is a database that completely resides in memory instead of a file.
|
||||
|
||||
All other details about this should be the same as the normal `redb` backend.
|
||||
|
||||
### 3.4 sanakirja
|
||||
[`sanakirja`](https://docs.rs/sanakirja) was a candidate as a backend, however there were problems with maximum value sizes.
|
||||
|
||||
The default maximum value size is [1012 bytes](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.Storable.html) which was too small for our requirements. Using [`sanakirja::Slice`](https://docs.rs/sanakirja/1.4.1/sanakirja/union.Slice.html) and [sanakirja::UnsizedStorage](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.UnsizedStorable.html) was attempted, but there were bugs found when inserting a value in-between `512..=4096` bytes.
|
||||
|
||||
As such, it is not implemented.
|
||||
|
||||
### 3.5 MDBX
|
||||
[`MDBX`](https://erthink.github.io/libmdbx) was a candidate as a backend, however MDBX deprecated the custom key/value comparison functions, this makes it a bit trickier to implement [`9.2 Multimap tables`](#92-multimap-tables). It is also quite similar to the main backend LMDB (of which it was originally a fork of).
|
||||
|
||||
As such, it is not implemented (yet).
|
||||
|
||||
## 4. Layers
|
||||
`cuprate_database` is logically abstracted into 5 layers, with each layer being built upon the last.
|
||||
|
||||
Starting from the lowest:
|
||||
1. Backend
|
||||
2. Trait
|
||||
3. ConcreteEnv
|
||||
4. `ops`
|
||||
5. `service`
|
||||
|
||||
<!-- TODO: insert image here after database/ split -->
|
||||
|
||||
### 4.1 Backend
|
||||
This is the actual database backend implementation (or a Rust shim over one).
|
||||
|
||||
Examples:
|
||||
- `heed` (LMDB)
|
||||
- `redb`
|
||||
|
||||
`cuprate_database` itself just uses a backend, it does not implement one.
|
||||
|
||||
All backends have the following attributes:
|
||||
- [Embedded](https://en.wikipedia.org/wiki/Embedded_database)
|
||||
- [Multiversion concurrency control](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
||||
- [ACID](https://en.wikipedia.org/wiki/ACID)
|
||||
- Are `(key, value)` oriented and have the expected API (`get()`, `insert()`, `delete()`)
|
||||
- Are table oriented (`"table_name" -> (key, value)`)
|
||||
- Allows concurrent readers
|
||||
|
||||
### 4.2 Trait
|
||||
`cuprate_database` provides a set of `trait`s that abstract over the various database backends.
|
||||
|
||||
This allows the function signatures and behavior to stay the same but allows for swapping out databases in an easier fashion.
|
||||
|
||||
All common behavior of the backend's are encapsulated here and used instead of using the backend directly.
|
||||
|
||||
Examples:
|
||||
- [`trait Env`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/env.rs)
|
||||
- [`trait {TxRo, TxRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/transaction.rs)
|
||||
- [`trait {DatabaseRo, DatabaseRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/database.rs)
|
||||
|
||||
For example, instead of calling `LMDB` or `redb`'s `get()` function directly, `DatabaseRo::get()` is called.
|
||||
|
||||
### 4.3 ConcreteEnv
|
||||
This is the non-generic, concrete `struct` provided by `cuprate_database` that contains all the data necessary to operate the database. The actual database backend `ConcreteEnv` will use internally depends on which backend feature is used.
|
||||
|
||||
`ConcreteEnv` implements `trait Env`, which opens the door to all the other traits.
|
||||
|
||||
The equivalent objects in the backends themselves are:
|
||||
- [`heed::Env`](https://docs.rs/heed/0.20.0/heed/struct.Env.html)
|
||||
- [`redb::Database`](https://docs.rs/redb/2.1.0/redb/struct.Database.html)
|
||||
|
||||
This is the main object used when handling the database directly, although that is not strictly necessary as a user if the [`4.5 service`](#45-service) layer is used.
|
||||
|
||||
### 4.4 ops
|
||||
These are Monero-specific functions that use the abstracted `trait` forms of the database.
|
||||
|
||||
Instead of dealing with the database directly:
|
||||
- `get()`
|
||||
- `delete()`
|
||||
|
||||
the `ops` layer provides more abstract functions that deal with commonly used Monero operations:
|
||||
- `add_block()`
|
||||
- `pop_block()`
|
||||
|
||||
### 4.5 service
|
||||
The final layer abstracts the database completely into a [Monero-specific `async` request/response API](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/types/src/service.rs#L18-L78) using [`tower::Service`](https://docs.rs/tower/latest/tower/trait.Service.html).
|
||||
|
||||
For more information on this layer, see the next section: [`5. The service`](#5-the-service).
|
||||
|
||||
## 5. The service
|
||||
The main API `cuprate_database` exposes for other crates to use is the `cuprate_database::service` module.
|
||||
|
||||
This module exposes an `async` request/response API with `tower::Service`, backed by a threadpool, that allows reading/writing Monero-related data from/to the database.
|
||||
|
||||
`cuprate_database::service` itself manages the database using a separate writer thread & reader thread-pool, and uses the previously mentioned [`4.4 ops`](#44-ops) functions when responding to requests.
|
||||
|
||||
### 5.1 Initialization
|
||||
The service is started simply by calling: [`cuprate_database::service::init()`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/service/free.rs#L23).
|
||||
|
||||
This function initializes the database, spawns threads, and returns a:
|
||||
- Read handle to the database (cloneable)
|
||||
- Write handle to the database (not cloneable)
|
||||
|
||||
These "handles" implement the `tower::Service` trait, which allows sending requests and receiving responses `async`hronously.
|
||||
|
||||
### 5.2 Requests
|
||||
Along with the 2 handles, there are 2 types of requests:
|
||||
- [`ReadRequest`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/types/src/service.rs#L23-L90)
|
||||
- [`WriteRequest`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/types/src/service.rs#L93-L105)
|
||||
|
||||
`ReadRequest` is for retrieving various types of information from the database.
|
||||
|
||||
`WriteRequest` currently only has 1 variant: to write a block to the database.
|
||||
|
||||
### 5.3 Responses
|
||||
After sending one of the above requests using the read/write handle, the value returned is _not_ the response, yet an `async`hronous channel that will eventually return the response:
|
||||
```rust,ignore
|
||||
// Send a request.
|
||||
// tower::Service::call()
|
||||
// V
|
||||
let response_channel: Channel = read_handle.call(ReadResponse::ChainHeight)?;
|
||||
|
||||
// Await the response.
|
||||
let response: ReadResponse = response_channel.await?;
|
||||
|
||||
// Assert the response is what we expected.
|
||||
assert_eq!(matches!(response), Response::ChainHeight(_));
|
||||
```
|
||||
|
||||
After `await`ing the returned channel, a `Response` will eventually be returned when the `service` threadpool has fetched the value from the database and sent it off.
|
||||
|
||||
Both read/write requests variants match in name with `Response` variants, i.e.
|
||||
- `ReadRequest::ChainHeight` leads to `Response::ChainHeight`
|
||||
- `WriteRequest::WriteBlock` leads to `Response::WriteBlockOk`
|
||||
|
||||
### 5.4 Thread model
|
||||
As mentioned in the [`4. Layers`](#4-layers) section, the base database abstractions themselves are not concerned with parallelism, they are mostly functions to be called from a single-thread.
|
||||
|
||||
However, the `cuprate_database::service` API, _does_ have a thread model backing it.
|
||||
|
||||
When [`cuprate_database::service`'s initialization function](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/free.rs#L33-L44) is called, threads will be spawned and maintained until the user drops (disconnects) the returned handles.
|
||||
|
||||
The current behavior for thread count is:
|
||||
- [1 writer thread](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/write.rs#L52-L66)
|
||||
- [As many reader threads as there are system threads](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/read.rs#L104-L126)
|
||||
|
||||
For example, on a system with 32-threads, `cuprate_database` will spawn:
|
||||
- 1 writer thread
|
||||
- 32 reader threads
|
||||
|
||||
whose sole responsibility is to listen for database requests, access the database (potentially in parallel), and return a response.
|
||||
|
||||
Note that the `1 system thread = 1 reader thread` model is only the default setting, the reader thread count can be configured by the user to be any number between `1 .. amount_of_system_threads`.
|
||||
|
||||
The reader threads are managed by [`rayon`](https://docs.rs/rayon).
|
||||
|
||||
For an example of where multiple reader threads are used: given a request that asks if any key-image within a set already exists, `cuprate_database` will [split that work between the threads with `rayon`](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/read.rs#L490-L503).
|
||||
|
||||
### 5.5 Shutdown
|
||||
Once the read/write handles are `Drop`ed, the backing thread(pool) will gracefully exit, automatically.
|
||||
|
||||
Note the writer thread and reader threadpool aren't connected whatsoever; dropping the write handle will make the writer thread exit, however, the reader handle is free to be held onto and can be continued to be read from - and vice-versa for the write handle.
|
||||
|
||||
## 6. Syncing
|
||||
`cuprate_database`'s database has 5 disk syncing modes.
|
||||
|
||||
1. FastThenSafe
|
||||
1. Safe
|
||||
1. Async
|
||||
1. Threshold
|
||||
1. Fast
|
||||
|
||||
The default mode is `Safe`.
|
||||
|
||||
This means that upon each transaction commit, all the data that was written will be fully synced to disk. This is the slowest, but safest mode of operation.
|
||||
|
||||
Note that upon any database `Drop`, whether via `service` or dropping the database directly, the current implementation will sync to disk regardless of any configuration.
|
||||
|
||||
For more information on the other modes, read the documentation [here](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/config/sync_mode.rs#L63-L144).
|
||||
|
||||
## 7. Resizing
|
||||
Database backends that require manually resizing will, by default, use a similar algorithm as `monerod`'s.
|
||||
|
||||
Note that this only relates to the `service` module, where the database is handled by `cuprate_database` itself, not the user. In the case of a user directly using `cuprate_database`, it is up to them on how to resize.
|
||||
|
||||
Within `service`, the resizing logic defined [here](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/service/write.rs#L139-L201) does the following:
|
||||
|
||||
- If there's not enough space to fit a write request's data, start a resize
|
||||
- Each resize adds around [`1_073_745_920`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L104-L160) bytes to the current map size
|
||||
- A resize will be attempted `3` times before failing
|
||||
|
||||
There are other [resizing algorithms](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L38-L47) that define how the database's memory map grows, although currently the behavior of [`monerod`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L104-L160) is closely followed.
|
||||
|
||||
## 8. (De)serialization
|
||||
All types stored inside the database are either bytes already, or are perfectly bitcast-able.
|
||||
|
||||
As such, they do not incur heavy (de)serialization costs when storing/fetching them from the database. The main (de)serialization used is [`bytemuck`](https://docs.rs/bytemuck)'s traits and casting functions.
|
||||
|
||||
The size & layout of types is stable across compiler versions, as they are set and determined with [`#[repr(C)]`](https://doc.rust-lang.org/nomicon/other-reprs.html#reprc) and `bytemuck`'s derive macros such as [`bytemuck::Pod`](https://docs.rs/bytemuck/latest/bytemuck/derive.Pod.html).
|
||||
|
||||
Note that the data stored in the tables are still type-safe; we still refer to the key and values within our tables by the type.
|
||||
|
||||
The main deserialization `trait` for database storage is: [`cuprate_database::Storable`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L16-L115).
|
||||
|
||||
- Before storage, the type is [simply cast into bytes](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L125)
|
||||
- When fetching, the bytes are [simply cast into the type](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L130)
|
||||
|
||||
When a type is casted into bytes, [the reference is casted](https://docs.rs/bytemuck/latest/bytemuck/fn.bytes_of.html), i.e. this is zero-cost serialization.
|
||||
|
||||
However, it is worth noting that when bytes are casted into the type, [it is copied](https://docs.rs/bytemuck/latest/bytemuck/fn.pod_read_unaligned.html). This is due to byte alignment guarantee issues with both backends, see:
|
||||
- https://github.com/AltSysrq/lmdb-zero/issues/8
|
||||
- https://github.com/cberner/redb/issues/360
|
||||
|
||||
Without this, `bytemuck` will panic with [`TargetAlignmentGreaterAndInputNotAligned`](https://docs.rs/bytemuck/latest/bytemuck/enum.PodCastError.html#variant.TargetAlignmentGreaterAndInputNotAligned) when casting.
|
||||
|
||||
Copying the bytes fixes this problem, although it is more costly than necessary. However, in the main use-case for `cuprate_database` (the `service` module) the bytes would need to be owned regardless as the `Request/Response` API uses owned data types (`T`, `Vec<T>`, `HashMap<K, V>`, etc).
|
||||
|
||||
Practically speaking, this means lower-level database functions that normally look like such:
|
||||
```rust
|
||||
fn get(key: &Key) -> &Value;
|
||||
```
|
||||
end up looking like this in `cuprate_database`:
|
||||
```rust
|
||||
fn get(key: &Key) -> Value;
|
||||
```
|
||||
|
||||
Since each backend has its own (de)serialization methods, our types are wrapped in compatibility types that map our `Storable` functions into whatever is required for the backend, e.g:
|
||||
- [`StorableHeed<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/backend/heed/storable.rs#L11-L45)
|
||||
- [`StorableRedb<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/backend/redb/storable.rs#L11-L30)
|
||||
|
||||
Compatibility structs also exist for any `Storable` containers:
|
||||
- [`StorableVec<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L135-L191)
|
||||
- [`StorableBytes`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L208-L241)
|
||||
|
||||
Again, it's unfortunate that these must be owned, although in `service`'s use-case, they would have to be owned anyway.
|
||||
|
||||
## 9. Schema
|
||||
This following section contains Cuprate's database schema, it may change throughout the development of Cuprate, as such, nothing here is final.
|
||||
|
||||
### 9.1 Tables
|
||||
The `CamelCase` names of the table headers documented here (e.g. `TxIds`) are the actual type name of the table within `cuprate_database`.
|
||||
|
||||
Note that words written within `code blocks` mean that it is a real type defined and usable within `cuprate_database`. Other standard types like u64 and type aliases (TxId) are written normally.
|
||||
|
||||
Within `cuprate_database::tables`, the below table is essentially defined as-is with [a macro](https://github.com/Cuprate/cuprate/blob/31ce89412aa174fc33754f22c9a6d9ef5ddeda28/database/src/tables.rs#L369-L470).
|
||||
|
||||
Many of the data types stored are the same data types, although are different semantically, as such, a map of aliases used and their real data types is also provided below.
|
||||
|
||||
| Alias | Real Type |
|
||||
|----------------------------------------------------|-----------|
|
||||
| BlockHeight, Amount, AmountIndex, TxId, UnlockTime | u64
|
||||
| BlockHash, KeyImage, TxHash, PrunableHash | [u8; 32]
|
||||
|
||||
| Table | Key | Value | Description |
|
||||
|-------------------|----------------------|--------------------|-------------|
|
||||
| `BlockBlobs` | BlockHeight | `StorableVec<u8>` | Maps a block's height to a serialized byte form of a block
|
||||
| `BlockHeights` | BlockHash | BlockHeight | Maps a block's hash to its height
|
||||
| `BlockInfos` | BlockHeight | `BlockInfo` | Contains metadata of all blocks
|
||||
| `KeyImages` | KeyImage | () | This table is a set with no value, it stores transaction key images
|
||||
| `NumOutputs` | Amount | u64 | Maps an output's amount to the number of outputs with that amount
|
||||
| `Outputs` | `PreRctOutputId` | `Output` | This table contains legacy CryptoNote outputs which have clear amounts. This table will not contain an output with 0 amount.
|
||||
| `PrunedTxBlobs` | TxId | `StorableVec<u8>` | Contains pruned transaction blobs (even if the database is not pruned)
|
||||
| `PrunableTxBlobs` | TxId | `StorableVec<u8>` | Contains the prunable part of a transaction
|
||||
| `PrunableHashes` | TxId | PrunableHash | Contains the hash of the prunable part of a transaction
|
||||
| `RctOutputs` | AmountIndex | `RctOutput` | Contains RingCT outputs mapped from their global RCT index
|
||||
| `TxBlobs` | TxId | `StorableVec<u8>` | Serialized transaction blobs (bytes)
|
||||
| `TxIds` | TxHash | TxId | Maps a transaction's hash to its index/ID
|
||||
| `TxHeights` | TxId | BlockHeight | Maps a transaction's ID to the height of the block it comes from
|
||||
| `TxOutputs` | TxId | `StorableVec<u64>` | Gives the amount indices of a transaction's outputs
|
||||
| `TxUnlockTime` | TxId | UnlockTime | Stores the unlock time of a transaction (only if it has a non-zero lock time)
|
||||
|
||||
The definitions for aliases and types (e.g. `RctOutput`) are within the [`cuprate_database::types`](https://github.com/Cuprate/cuprate/blob/31ce89412aa174fc33754f22c9a6d9ef5ddeda28/database/src/types.rs#L51) module.
|
||||
|
||||
<!-- TODO(Boog900): We could split this table again into `RingCT (non-miner) Outputs` and `RingCT (miner) Outputs` as for miner outputs we can store the amount instead of commitment saving 24 bytes per miner output. -->
|
||||
|
||||
### 9.2 Multimap tables
|
||||
When referencing outputs, Monero will [use the amount and the amount index](https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/blockchain_db/lmdb/db_lmdb.cpp#L3447-L3449). This means 2 keys are needed to reach an output.
|
||||
|
||||
With LMDB you can set the `DUP_SORT` flag on a table and then set the key/value to:
|
||||
```rust
|
||||
Key = KEY_PART_1
|
||||
```
|
||||
```rust
|
||||
Value = {
|
||||
KEY_PART_2,
|
||||
VALUE // The actual value we are storing.
|
||||
}
|
||||
```
|
||||
|
||||
Then you can set a custom value sorting function that only takes `KEY_PART_2` into account; this is how `monerod` does it.
|
||||
|
||||
This requires that the underlying database supports:
|
||||
- multimap tables
|
||||
- custom sort functions on values
|
||||
- setting a cursor on a specific key/value
|
||||
|
||||
---
|
||||
|
||||
Another way to implement this is as follows:
|
||||
```rust
|
||||
Key = { KEY_PART_1, KEY_PART_2 }
|
||||
```
|
||||
```rust
|
||||
Value = VALUE
|
||||
```
|
||||
|
||||
Then the key type is simply used to look up the value; this is how `cuprate_database` does it.
|
||||
|
||||
For example, the key/value pair for outputs is:
|
||||
```rust
|
||||
PreRctOutputId => Output
|
||||
```
|
||||
where `PreRctOutputId` looks like this:
|
||||
```rust
|
||||
struct PreRctOutputId {
|
||||
amount: u64,
|
||||
amount_index: u64,
|
||||
}
|
||||
```
|
||||
|
||||
## 10. Known issues and tradeoffs
|
||||
`cuprate_database` takes many tradeoffs, whether due to:
|
||||
- Prioritizing certain values over others
|
||||
- Not having a better solution
|
||||
- Being "good enough"
|
||||
|
||||
This is a list of the larger ones, along with issues that don't have answers yet.
|
||||
|
||||
### 10.1 Traits abstracting backends
|
||||
Although all database backends used are very similar, they have some crucial differences in small implementation details that must be worked around when conforming them to `cuprate_database`'s traits.
|
||||
|
||||
Put simply: using `cuprate_database`'s traits is less efficient and more awkward than using the backend directly.
|
||||
|
||||
For example:
|
||||
- [Data types must be wrapped in compatibility layers when they otherwise wouldn't be](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/backend/heed/env.rs#L101-L116)
|
||||
- [There are types that only apply to a specific backend, but are visible to all](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/error.rs#L86-L89)
|
||||
- [There are extra layers of abstraction to smoothen the differences between all backends](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/env.rs#L62-L68)
|
||||
- [Existing functionality of backends must be taken away, as it isn't supported in the others](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/database.rs#L27-L34)
|
||||
|
||||
This is a _tradeoff_ that `cuprate_database` takes, as:
|
||||
- The backend itself is usually not the source of bottlenecks in the greater system, as such, small inefficiencies are OK
|
||||
- None of the lost functionality is crucial for operation
|
||||
- The ability to use, test, and swap between multiple database backends is [worth it](https://github.com/Cuprate/cuprate/pull/35#issuecomment-1952804393)
|
||||
|
||||
### 10.2 Hot-swappable backends
|
||||
Using a different backend is really as simple as re-building `cuprate_database` with a different feature flag:
|
||||
```bash
|
||||
# Use LMDB.
|
||||
cargo build --package cuprate-database --features heed
|
||||
|
||||
# Use redb.
|
||||
cargo build --package cuprate-database --features redb
|
||||
```
|
||||
|
||||
This is "good enough" for now, however ideally, this hot-swapping of backends would be able to be done at _runtime_.
|
||||
|
||||
As it is now, `cuprate_database` cannot compile both backends and swap based on user input at runtime; it must be compiled with a certain backend, which will produce a binary with only that backend.
|
||||
|
||||
This also means things like [CI testing multiple backends is awkward](https://github.com/Cuprate/cuprate/blob/main/.github/workflows/ci.yml#L132-L136), as we must re-compile with different feature flags instead.
|
||||
|
||||
### 10.3 Copying unaligned bytes
|
||||
As mentioned in [`8. (De)serialization`](#8-deserialization), bytes are _copied_ when they are turned into a type `T` due to unaligned bytes being returned from database backends.
|
||||
|
||||
Using a regular reference cast results in an improperly aligned type `T`; [such a type even existing causes undefined behavior](https://doc.rust-lang.org/reference/behavior-considered-undefined.html). In our case, `bytemuck` saves us by panicking before this occurs.
|
||||
|
||||
Thus, when using `cuprate_database`'s database traits, an _owned_ `T` is returned.
|
||||
|
||||
This is doubly unfortunately for `&[u8]` as this does not even need deserialization.
|
||||
|
||||
For example, `StorableVec` could have been this:
|
||||
```rust
|
||||
enum StorableBytes<'a, T: Storable> {
|
||||
Owned(T),
|
||||
Ref(&'a T),
|
||||
}
|
||||
```
|
||||
but this would require supporting types that must be copied regardless with the occasional `&[u8]` that can be returned without casting. This was hard to do so in a generic way, thus all `[u8]`'s are copied and returned as owned `StorableVec`s.
|
||||
|
||||
This is a _tradeoff_ `cuprate_database` takes as:
|
||||
- `bytemuck::pod_read_unaligned` is cheap enough
|
||||
- The main API, `service`, needs to return owned value anyway
|
||||
- Having no references removes a lot of lifetime complexity
|
||||
|
||||
The alternative is either:
|
||||
- Using proper (de)serialization instead of casting (which comes with its own costs)
|
||||
- Somehow fixing the alignment issues in the backends mentioned previously
|
||||
|
||||
### 10.4 Endianness
|
||||
`cuprate_database`'s (de)serialization and storage of bytes are native-endian, as in, byte storage order will depend on the machine it is running on.
|
||||
|
||||
As Cuprate's build-targets are all little-endian ([big-endian by default machines barely exist](https://en.wikipedia.org/wiki/Endianness#Hardware)), this doesn't matter much and the byte ordering can be seen as a constant.
|
||||
|
||||
Practically, this means `cuprated`'s database files can be transferred across computers, as can `monerod`'s.
|
||||
|
||||
### 10.5 Extra table data
|
||||
Some of `cuprate_database`'s tables differ from `monerod`'s tables, for example, the way [`9.2 Multimap tables`](#92-multimap-tables) tables are done requires that the primary key is stored _for all_ entries, compared to `monerod` only needing to store it once.
|
||||
|
||||
For example:
|
||||
```rust
|
||||
// `monerod` only stores `amount: 1` once,
|
||||
// `cuprated` stores it each time it appears.
|
||||
struct PreRctOutputId { amount: 1, amount_index: 0 }
|
||||
struct PreRctOutputId { amount: 1, amount_index: 1 }
|
||||
```
|
||||
|
||||
This means `cuprated`'s database will be slightly larger than `monerod`'s.
|
||||
|
||||
The current method `cuprate_database` uses will be "good enough" until usage shows that it must be optimized as multimap tables are tricky to implement across all backends.
|
||||
59
storage/database/Cargo.toml
Normal file
59
storage/database/Cargo.toml
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
[package]
|
||||
name = "cuprate-database"
|
||||
version = "0.0.0"
|
||||
edition = "2021"
|
||||
description = "Cuprate's database abstraction"
|
||||
license = "MIT"
|
||||
authors = ["hinto-janai"]
|
||||
repository = "https://github.com/Cuprate/cuprate/tree/main/database"
|
||||
keywords = ["cuprate", "database"]
|
||||
|
||||
[features]
|
||||
default = ["heed", "redb", "service"]
|
||||
# default = ["redb", "service"]
|
||||
# default = ["redb-memory", "service"]
|
||||
heed = ["dep:heed"]
|
||||
redb = ["dep:redb"]
|
||||
redb-memory = ["redb"]
|
||||
service = ["dep:crossbeam", "dep:futures", "dep:tokio", "dep:tokio-util", "dep:tower", "dep:rayon"]
|
||||
|
||||
[dependencies]
|
||||
bitflags = { workspace = true, features = ["serde", "bytemuck"] }
|
||||
bytemuck = { version = "1.14.3", features = ["must_cast", "derive", "min_const_generics", "extern_crate_alloc"] }
|
||||
bytes = { workspace = true }
|
||||
cfg-if = { workspace = true }
|
||||
# FIXME:
|
||||
# We only need the `thread` feature if `service` is enabled.
|
||||
# Figure out how to enable features of an already pulled in dependency conditionally.
|
||||
cuprate-helper = { path = "../helper", features = ["fs", "thread", "map"] }
|
||||
cuprate-types = { path = "../types", features = ["service"] }
|
||||
curve25519-dalek = { workspace = true }
|
||||
monero-pruning = { path = "../pruning" }
|
||||
monero-serai = { workspace = true, features = ["std"] }
|
||||
paste = { workspace = true }
|
||||
page_size = { version = "0.6.0" } # Needed for database resizes, they must be a multiple of the OS page size.
|
||||
thiserror = { workspace = true }
|
||||
|
||||
# `service` feature.
|
||||
crossbeam = { workspace = true, features = ["std"], optional = true }
|
||||
futures = { workspace = true, optional = true }
|
||||
tokio = { workspace = true, features = ["full"], optional = true }
|
||||
tokio-util = { workspace = true, features = ["full"], optional = true }
|
||||
tower = { workspace = true, features = ["full"], optional = true }
|
||||
thread_local = { workspace = true }
|
||||
rayon = { workspace = true, optional = true }
|
||||
|
||||
# Optional features.
|
||||
heed = { version = "0.20.0", features = ["read-txn-no-tls"], optional = true }
|
||||
redb = { version = "2.1.0", optional = true }
|
||||
serde = { workspace = true, optional = true }
|
||||
|
||||
[dev-dependencies]
|
||||
bytemuck = { version = "1.14.3", features = ["must_cast", "derive", "min_const_generics", "extern_crate_alloc"] }
|
||||
cuprate-helper = { path = "../helper", features = ["thread"] }
|
||||
cuprate-test-utils = { path = "../test-utils" }
|
||||
page_size = { version = "0.6.0" }
|
||||
tempfile = { version = "3.10.0" }
|
||||
pretty_assertions = { workspace = true }
|
||||
hex = { workspace = true }
|
||||
hex-literal = { workspace = true }
|
||||
598
storage/database/README.md
Normal file
598
storage/database/README.md
Normal file
|
|
@ -0,0 +1,598 @@
|
|||
# Database
|
||||
Cuprate's database implementation.
|
||||
|
||||
- [1. Documentation](#1-documentation)
|
||||
- [2. File structure](#2-file-structure)
|
||||
- [2.1 `src/`](#21-src)
|
||||
- [2.2 `src/backend/`](#22-srcbackend)
|
||||
- [2.3 `src/config/`](#23-srcconfig)
|
||||
- [2.4 `src/ops/`](#24-srcops)
|
||||
- [2.5 `src/service/`](#25-srcservice)
|
||||
- [3. Backends](#3-backends)
|
||||
- [3.1 heed](#31-heed)
|
||||
- [3.2 redb](#32-redb)
|
||||
- [3.3 redb-memory](#33-redb-memory)
|
||||
- [3.4 sanakirja](#34-sanakirja)
|
||||
- [3.5 MDBX](#35-mdbx)
|
||||
- [4. Layers](#4-layers)
|
||||
- [4.1 Backend](#41-backend)
|
||||
- [4.2 Trait](#42-trait)
|
||||
- [4.3 ConcreteEnv](#43-concreteenv)
|
||||
- [4.4 ops](#44-ops)
|
||||
- [4.5 service](#45-service)
|
||||
- [5. The service](#5-the-service)
|
||||
- [5.1 Initialization](#51-initialization)
|
||||
- [5.2 Requests](#53-requests)
|
||||
- [5.3 Responses](#54-responses)
|
||||
- [5.4 Thread model](#52-thread-model)
|
||||
- [5.5 Shutdown](#55-shutdown)
|
||||
- [6. Syncing](#6-Syncing)
|
||||
- [7. Resizing](#7-resizing)
|
||||
- [8. (De)serialization](#8-deserialization)
|
||||
- [9. Schema](#9-schema)
|
||||
- [9.1 Tables](#91-tables)
|
||||
- [9.2 Multimap tables](#92-multimap-tables)
|
||||
- [10. Known issues and tradeoffs](#10-known-issues-and-tradeoffs)
|
||||
- [10.1 Traits abstracting backends](#101-traits-abstracting-backends)
|
||||
- [10.2 Hot-swappable backends](#102-hot-swappable-backends)
|
||||
- [10.3 Copying unaligned bytes](#103-copying-unaligned-bytes)
|
||||
- [10.4 Endianness](#104-endianness)
|
||||
- [10.5 Extra table data](#105-extra-table-data)
|
||||
|
||||
---
|
||||
|
||||
## 1. Documentation
|
||||
Documentation for `database/` is split into 3 locations:
|
||||
|
||||
| Documentation location | Purpose |
|
||||
|---------------------------|---------|
|
||||
| `database/README.md` | High level design of `cuprate-database`
|
||||
| `cuprate-database` | Practical usage documentation/warnings/notes/etc
|
||||
| Source file `// comments` | Implementation-specific details (e.g, how many reader threads to spawn?)
|
||||
|
||||
This README serves as the implementation design document.
|
||||
|
||||
For actual practical usage, `cuprate-database`'s types and general usage are documented via standard Rust tooling.
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cargo doc --package cuprate-database --open
|
||||
```
|
||||
at the root of the repo to open/read the documentation.
|
||||
|
||||
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many `// Regular comments` that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
|
||||
|
||||
The code within `src/` is also littered with some `grep`-able comments containing some keywords:
|
||||
|
||||
| Word | Meaning |
|
||||
|-------------|---------|
|
||||
| `INVARIANT` | This code makes an _assumption_ that must be upheld for correctness
|
||||
| `SAFETY` | This `unsafe` code is okay, for `x,y,z` reasons
|
||||
| `FIXME` | This code works but isn't ideal
|
||||
| `HACK` | This code is a brittle workaround
|
||||
| `PERF` | This code is weird for performance reasons
|
||||
| `TODO` | This must be implemented; There should be 0 of these in production code
|
||||
| `SOMEDAY` | This should be implemented... someday
|
||||
|
||||
## 2. File structure
|
||||
A quick reference of the structure of the folders & files in `cuprate-database`.
|
||||
|
||||
Note that `lib.rs/mod.rs` files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding `mod.rs`.
|
||||
|
||||
### 2.1 `src/`
|
||||
The top-level `src/` files.
|
||||
|
||||
| File | Purpose |
|
||||
|------------------------|---------|
|
||||
| `constants.rs` | General constants used throughout `cuprate-database`
|
||||
| `database.rs` | Abstracted database; `trait DatabaseR{o,w}`
|
||||
| `env.rs` | Abstracted database environment; `trait Env`
|
||||
| `error.rs` | Database error types
|
||||
| `free.rs` | General free functions (related to the database)
|
||||
| `key.rs` | Abstracted database keys; `trait Key`
|
||||
| `resize.rs` | Database resizing algorithms
|
||||
| `storable.rs` | Data (de)serialization; `trait Storable`
|
||||
| `table.rs` | Database table abstraction; `trait Table`
|
||||
| `tables.rs` | All the table definitions used by `cuprate-database`
|
||||
| `tests.rs` | Utilities for `cuprate_database` testing
|
||||
| `transaction.rs` | Database transaction abstraction; `trait TxR{o,w}`
|
||||
| `types.rs` | Database-specific types
|
||||
| `unsafe_unsendable.rs` | Marker type to impl `Send` for objects not `Send`
|
||||
|
||||
### 2.2 `src/backend/`
|
||||
This folder contains the implementation for actual databases used as the backend for `cuprate-database`.
|
||||
|
||||
Each backend has its own folder.
|
||||
|
||||
| Folder/File | Purpose |
|
||||
|-------------|---------|
|
||||
| `heed/` | Backend using using [`heed`](https://github.com/meilisearch/heed) (LMDB)
|
||||
| `redb/` | Backend using [`redb`](https://github.com/cberner/redb)
|
||||
| `tests.rs` | Backend-agnostic tests
|
||||
|
||||
All backends follow the same file structure:
|
||||
|
||||
| File | Purpose |
|
||||
|------------------|---------|
|
||||
| `database.rs` | Implementation of `trait DatabaseR{o,w}`
|
||||
| `env.rs` | Implementation of `trait Env`
|
||||
| `error.rs` | Implementation of backend's errors to `cuprate_database`'s error types
|
||||
| `storable.rs` | Compatibility layer between `cuprate_database::Storable` and backend-specific (de)serialization
|
||||
| `transaction.rs` | Implementation of `trait TxR{o,w}`
|
||||
| `types.rs` | Type aliases for long backend-specific types
|
||||
|
||||
### 2.3 `src/config/`
|
||||
This folder contains the `cupate_database::config` module; configuration options for the database.
|
||||
|
||||
| File | Purpose |
|
||||
|---------------------|---------|
|
||||
| `config.rs` | Main database `Config` struct
|
||||
| `reader_threads.rs` | Reader thread configuration for `service` thread-pool
|
||||
| `sync_mode.rs` | Disk sync configuration for backends
|
||||
|
||||
### 2.4 `src/ops/`
|
||||
This folder contains the `cupate_database::ops` module.
|
||||
|
||||
These are higher-level functions abstracted over the database, that are Monero-related.
|
||||
|
||||
| File | Purpose |
|
||||
|-----------------|---------|
|
||||
| `block.rs` | Block related (main functions)
|
||||
| `blockchain.rs` | Blockchain related (height, cumulative values, etc)
|
||||
| `key_image.rs` | Key image related
|
||||
| `macros.rs` | Macros specific to `ops/`
|
||||
| `output.rs` | Output related
|
||||
| `property.rs` | Database properties (pruned, version, etc)
|
||||
| `tx.rs` | Transaction related
|
||||
|
||||
### 2.5 `src/service/`
|
||||
This folder contains the `cupate_database::service` module.
|
||||
|
||||
The `async`hronous request/response API other Cuprate crates use instead of managing the database directly themselves.
|
||||
|
||||
| File | Purpose |
|
||||
|----------------|---------|
|
||||
| `free.rs` | General free functions used (related to `cuprate_database::service`)
|
||||
| `read.rs` | Read thread-pool definitions and logic
|
||||
| `tests.rs` | Thread-pool tests and test helper functions
|
||||
| `types.rs` | `cuprate_database::service`-related type aliases
|
||||
| `write.rs` | Writer thread definitions and logic
|
||||
|
||||
## 3. Backends
|
||||
`cuprate-database`'s `trait`s allow abstracting over the actual database, such that any backend in particular could be used.
|
||||
|
||||
Each database's implementation for those `trait`'s are located in its respective folder in `src/backend/${DATABASE_NAME}/`.
|
||||
|
||||
### 3.1 heed
|
||||
The default database used is [`heed`](https://github.com/meilisearch/heed) (LMDB). The upstream versions from [`crates.io`](https://crates.io/crates/heed) are used. `LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
|
||||
|
||||
`heed`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
|
||||
|
||||
| Filename | Purpose |
|
||||
|------------|---------|
|
||||
| `data.mdb` | Main data file
|
||||
| `lock.mdb` | Database lock file
|
||||
|
||||
`heed`-specific notes:
|
||||
- [There is a maximum reader limit](https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1372). Other potential processes (e.g. `xmrblocks`) that are also reading the `data.mdb` file need to be accounted for
|
||||
- [LMDB does not work on remote filesystem](https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/lmdb.h#L129)
|
||||
|
||||
### 3.2 redb
|
||||
The 2nd database backend is the 100% Rust [`redb`](https://github.com/cberner/redb).
|
||||
|
||||
The upstream versions from [`crates.io`](https://crates.io/crates/redb) are used.
|
||||
|
||||
`redb`'s filenames inside Cuprate's database folder (`~/.local/share/cuprate/database/`) are:
|
||||
|
||||
| Filename | Purpose |
|
||||
|-------------|---------|
|
||||
| `data.redb` | Main data file
|
||||
|
||||
<!-- TODO: document DB on remote filesystem (does redb allow this?) -->
|
||||
|
||||
### 3.3 redb-memory
|
||||
This backend is 100% the same as `redb`, although, it uses `redb::backend::InMemoryBackend` which is a database that completely resides in memory instead of a file.
|
||||
|
||||
All other details about this should be the same as the normal `redb` backend.
|
||||
|
||||
### 3.4 sanakirja
|
||||
[`sanakirja`](https://docs.rs/sanakirja) was a candidate as a backend, however there were problems with maximum value sizes.
|
||||
|
||||
The default maximum value size is [1012 bytes](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.Storable.html) which was too small for our requirements. Using [`sanakirja::Slice`](https://docs.rs/sanakirja/1.4.1/sanakirja/union.Slice.html) and [sanakirja::UnsizedStorage](https://docs.rs/sanakirja/1.4.1/sanakirja/trait.UnsizedStorable.html) was attempted, but there were bugs found when inserting a value in-between `512..=4096` bytes.
|
||||
|
||||
As such, it is not implemented.
|
||||
|
||||
### 3.5 MDBX
|
||||
[`MDBX`](https://erthink.github.io/libmdbx) was a candidate as a backend, however MDBX deprecated the custom key/value comparison functions, this makes it a bit trickier to implement [`9.2 Multimap tables`](#92-multimap-tables). It is also quite similar to the main backend LMDB (of which it was originally a fork of).
|
||||
|
||||
As such, it is not implemented (yet).
|
||||
|
||||
## 4. Layers
|
||||
`cuprate_database` is logically abstracted into 5 layers, with each layer being built upon the last.
|
||||
|
||||
Starting from the lowest:
|
||||
1. Backend
|
||||
2. Trait
|
||||
3. ConcreteEnv
|
||||
4. `ops`
|
||||
5. `service`
|
||||
|
||||
<!-- TODO: insert image here after database/ split -->
|
||||
|
||||
### 4.1 Backend
|
||||
This is the actual database backend implementation (or a Rust shim over one).
|
||||
|
||||
Examples:
|
||||
- `heed` (LMDB)
|
||||
- `redb`
|
||||
|
||||
`cuprate_database` itself just uses a backend, it does not implement one.
|
||||
|
||||
All backends have the following attributes:
|
||||
- [Embedded](https://en.wikipedia.org/wiki/Embedded_database)
|
||||
- [Multiversion concurrency control](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
||||
- [ACID](https://en.wikipedia.org/wiki/ACID)
|
||||
- Are `(key, value)` oriented and have the expected API (`get()`, `insert()`, `delete()`)
|
||||
- Are table oriented (`"table_name" -> (key, value)`)
|
||||
- Allows concurrent readers
|
||||
|
||||
### 4.2 Trait
|
||||
`cuprate_database` provides a set of `trait`s that abstract over the various database backends.
|
||||
|
||||
This allows the function signatures and behavior to stay the same but allows for swapping out databases in an easier fashion.
|
||||
|
||||
All common behavior of the backend's are encapsulated here and used instead of using the backend directly.
|
||||
|
||||
Examples:
|
||||
- [`trait Env`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/env.rs)
|
||||
- [`trait {TxRo, TxRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/transaction.rs)
|
||||
- [`trait {DatabaseRo, DatabaseRw}`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/database.rs)
|
||||
|
||||
For example, instead of calling `LMDB` or `redb`'s `get()` function directly, `DatabaseRo::get()` is called.
|
||||
|
||||
### 4.3 ConcreteEnv
|
||||
This is the non-generic, concrete `struct` provided by `cuprate_database` that contains all the data necessary to operate the database. The actual database backend `ConcreteEnv` will use internally depends on which backend feature is used.
|
||||
|
||||
`ConcreteEnv` implements `trait Env`, which opens the door to all the other traits.
|
||||
|
||||
The equivalent objects in the backends themselves are:
|
||||
- [`heed::Env`](https://docs.rs/heed/0.20.0/heed/struct.Env.html)
|
||||
- [`redb::Database`](https://docs.rs/redb/2.1.0/redb/struct.Database.html)
|
||||
|
||||
This is the main object used when handling the database directly, although that is not strictly necessary as a user if the [`4.5 service`](#45-service) layer is used.
|
||||
|
||||
### 4.4 ops
|
||||
These are Monero-specific functions that use the abstracted `trait` forms of the database.
|
||||
|
||||
Instead of dealing with the database directly:
|
||||
- `get()`
|
||||
- `delete()`
|
||||
|
||||
the `ops` layer provides more abstract functions that deal with commonly used Monero operations:
|
||||
- `add_block()`
|
||||
- `pop_block()`
|
||||
|
||||
### 4.5 service
|
||||
The final layer abstracts the database completely into a [Monero-specific `async` request/response API](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/types/src/service.rs#L18-L78) using [`tower::Service`](https://docs.rs/tower/latest/tower/trait.Service.html).
|
||||
|
||||
For more information on this layer, see the next section: [`5. The service`](#5-the-service).
|
||||
|
||||
## 5. The service
|
||||
The main API `cuprate_database` exposes for other crates to use is the `cuprate_database::service` module.
|
||||
|
||||
This module exposes an `async` request/response API with `tower::Service`, backed by a threadpool, that allows reading/writing Monero-related data from/to the database.
|
||||
|
||||
`cuprate_database::service` itself manages the database using a separate writer thread & reader thread-pool, and uses the previously mentioned [`4.4 ops`](#44-ops) functions when responding to requests.
|
||||
|
||||
### 5.1 Initialization
|
||||
The service is started simply by calling: [`cuprate_database::service::init()`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/service/free.rs#L23).
|
||||
|
||||
This function initializes the database, spawns threads, and returns a:
|
||||
- Read handle to the database (cloneable)
|
||||
- Write handle to the database (not cloneable)
|
||||
|
||||
These "handles" implement the `tower::Service` trait, which allows sending requests and receiving responses `async`hronously.
|
||||
|
||||
### 5.2 Requests
|
||||
Along with the 2 handles, there are 2 types of requests:
|
||||
- [`ReadRequest`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/types/src/service.rs#L23-L90)
|
||||
- [`WriteRequest`](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/types/src/service.rs#L93-L105)
|
||||
|
||||
`ReadRequest` is for retrieving various types of information from the database.
|
||||
|
||||
`WriteRequest` currently only has 1 variant: to write a block to the database.
|
||||
|
||||
### 5.3 Responses
|
||||
After sending one of the above requests using the read/write handle, the value returned is _not_ the response, yet an `async`hronous channel that will eventually return the response:
|
||||
```rust,ignore
|
||||
// Send a request.
|
||||
// tower::Service::call()
|
||||
// V
|
||||
let response_channel: Channel = read_handle.call(ReadResponse::ChainHeight)?;
|
||||
|
||||
// Await the response.
|
||||
let response: ReadResponse = response_channel.await?;
|
||||
|
||||
// Assert the response is what we expected.
|
||||
assert_eq!(matches!(response), Response::ChainHeight(_));
|
||||
```
|
||||
|
||||
After `await`ing the returned channel, a `Response` will eventually be returned when the `service` threadpool has fetched the value from the database and sent it off.
|
||||
|
||||
Both read/write requests variants match in name with `Response` variants, i.e.
|
||||
- `ReadRequest::ChainHeight` leads to `Response::ChainHeight`
|
||||
- `WriteRequest::WriteBlock` leads to `Response::WriteBlockOk`
|
||||
|
||||
### 5.4 Thread model
|
||||
As mentioned in the [`4. Layers`](#4-layers) section, the base database abstractions themselves are not concerned with parallelism, they are mostly functions to be called from a single-thread.
|
||||
|
||||
However, the `cuprate_database::service` API, _does_ have a thread model backing it.
|
||||
|
||||
When [`cuprate_database::service`'s initialization function](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/free.rs#L33-L44) is called, threads will be spawned and maintained until the user drops (disconnects) the returned handles.
|
||||
|
||||
The current behavior for thread count is:
|
||||
- [1 writer thread](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/write.rs#L52-L66)
|
||||
- [As many reader threads as there are system threads](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/read.rs#L104-L126)
|
||||
|
||||
For example, on a system with 32-threads, `cuprate_database` will spawn:
|
||||
- 1 writer thread
|
||||
- 32 reader threads
|
||||
|
||||
whose sole responsibility is to listen for database requests, access the database (potentially in parallel), and return a response.
|
||||
|
||||
Note that the `1 system thread = 1 reader thread` model is only the default setting, the reader thread count can be configured by the user to be any number between `1 .. amount_of_system_threads`.
|
||||
|
||||
The reader threads are managed by [`rayon`](https://docs.rs/rayon).
|
||||
|
||||
For an example of where multiple reader threads are used: given a request that asks if any key-image within a set already exists, `cuprate_database` will [split that work between the threads with `rayon`](https://github.com/Cuprate/cuprate/blob/9c27ba5791377d639cb5d30d0f692c228568c122/database/src/service/read.rs#L490-L503).
|
||||
|
||||
### 5.5 Shutdown
|
||||
Once the read/write handles are `Drop`ed, the backing thread(pool) will gracefully exit, automatically.
|
||||
|
||||
Note the writer thread and reader threadpool aren't connected whatsoever; dropping the write handle will make the writer thread exit, however, the reader handle is free to be held onto and can be continued to be read from - and vice-versa for the write handle.
|
||||
|
||||
## 6. Syncing
|
||||
`cuprate_database`'s database has 5 disk syncing modes.
|
||||
|
||||
1. FastThenSafe
|
||||
1. Safe
|
||||
1. Async
|
||||
1. Threshold
|
||||
1. Fast
|
||||
|
||||
The default mode is `Safe`.
|
||||
|
||||
This means that upon each transaction commit, all the data that was written will be fully synced to disk. This is the slowest, but safest mode of operation.
|
||||
|
||||
Note that upon any database `Drop`, whether via `service` or dropping the database directly, the current implementation will sync to disk regardless of any configuration.
|
||||
|
||||
For more information on the other modes, read the documentation [here](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/config/sync_mode.rs#L63-L144).
|
||||
|
||||
## 7. Resizing
|
||||
Database backends that require manually resizing will, by default, use a similar algorithm as `monerod`'s.
|
||||
|
||||
Note that this only relates to the `service` module, where the database is handled by `cuprate_database` itself, not the user. In the case of a user directly using `cuprate_database`, it is up to them on how to resize.
|
||||
|
||||
Within `service`, the resizing logic defined [here](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/service/write.rs#L139-L201) does the following:
|
||||
|
||||
- If there's not enough space to fit a write request's data, start a resize
|
||||
- Each resize adds around [`1_073_745_920`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L104-L160) bytes to the current map size
|
||||
- A resize will be attempted `3` times before failing
|
||||
|
||||
There are other [resizing algorithms](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L38-L47) that define how the database's memory map grows, although currently the behavior of [`monerod`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L104-L160) is closely followed.
|
||||
|
||||
## 8. (De)serialization
|
||||
All types stored inside the database are either bytes already, or are perfectly bitcast-able.
|
||||
|
||||
As such, they do not incur heavy (de)serialization costs when storing/fetching them from the database. The main (de)serialization used is [`bytemuck`](https://docs.rs/bytemuck)'s traits and casting functions.
|
||||
|
||||
The size & layout of types is stable across compiler versions, as they are set and determined with [`#[repr(C)]`](https://doc.rust-lang.org/nomicon/other-reprs.html#reprc) and `bytemuck`'s derive macros such as [`bytemuck::Pod`](https://docs.rs/bytemuck/latest/bytemuck/derive.Pod.html).
|
||||
|
||||
Note that the data stored in the tables are still type-safe; we still refer to the key and values within our tables by the type.
|
||||
|
||||
The main deserialization `trait` for database storage is: [`cuprate_database::Storable`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L16-L115).
|
||||
|
||||
- Before storage, the type is [simply cast into bytes](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L125)
|
||||
- When fetching, the bytes are [simply cast into the type](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L130)
|
||||
|
||||
When a type is casted into bytes, [the reference is casted](https://docs.rs/bytemuck/latest/bytemuck/fn.bytes_of.html), i.e. this is zero-cost serialization.
|
||||
|
||||
However, it is worth noting that when bytes are casted into the type, [it is copied](https://docs.rs/bytemuck/latest/bytemuck/fn.pod_read_unaligned.html). This is due to byte alignment guarantee issues with both backends, see:
|
||||
- https://github.com/AltSysrq/lmdb-zero/issues/8
|
||||
- https://github.com/cberner/redb/issues/360
|
||||
|
||||
Without this, `bytemuck` will panic with [`TargetAlignmentGreaterAndInputNotAligned`](https://docs.rs/bytemuck/latest/bytemuck/enum.PodCastError.html#variant.TargetAlignmentGreaterAndInputNotAligned) when casting.
|
||||
|
||||
Copying the bytes fixes this problem, although it is more costly than necessary. However, in the main use-case for `cuprate_database` (the `service` module) the bytes would need to be owned regardless as the `Request/Response` API uses owned data types (`T`, `Vec<T>`, `HashMap<K, V>`, etc).
|
||||
|
||||
Practically speaking, this means lower-level database functions that normally look like such:
|
||||
```rust
|
||||
fn get(key: &Key) -> &Value;
|
||||
```
|
||||
end up looking like this in `cuprate_database`:
|
||||
```rust
|
||||
fn get(key: &Key) -> Value;
|
||||
```
|
||||
|
||||
Since each backend has its own (de)serialization methods, our types are wrapped in compatibility types that map our `Storable` functions into whatever is required for the backend, e.g:
|
||||
- [`StorableHeed<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/backend/heed/storable.rs#L11-L45)
|
||||
- [`StorableRedb<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/backend/redb/storable.rs#L11-L30)
|
||||
|
||||
Compatibility structs also exist for any `Storable` containers:
|
||||
- [`StorableVec<T>`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L135-L191)
|
||||
- [`StorableBytes`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/storable.rs#L208-L241)
|
||||
|
||||
Again, it's unfortunate that these must be owned, although in `service`'s use-case, they would have to be owned anyway.
|
||||
|
||||
## 9. Schema
|
||||
This following section contains Cuprate's database schema, it may change throughout the development of Cuprate, as such, nothing here is final.
|
||||
|
||||
### 9.1 Tables
|
||||
The `CamelCase` names of the table headers documented here (e.g. `TxIds`) are the actual type name of the table within `cuprate_database`.
|
||||
|
||||
Note that words written within `code blocks` mean that it is a real type defined and usable within `cuprate_database`. Other standard types like u64 and type aliases (TxId) are written normally.
|
||||
|
||||
Within `cuprate_database::tables`, the below table is essentially defined as-is with [a macro](https://github.com/Cuprate/cuprate/blob/31ce89412aa174fc33754f22c9a6d9ef5ddeda28/database/src/tables.rs#L369-L470).
|
||||
|
||||
Many of the data types stored are the same data types, although are different semantically, as such, a map of aliases used and their real data types is also provided below.
|
||||
|
||||
| Alias | Real Type |
|
||||
|----------------------------------------------------|-----------|
|
||||
| BlockHeight, Amount, AmountIndex, TxId, UnlockTime | u64
|
||||
| BlockHash, KeyImage, TxHash, PrunableHash | [u8; 32]
|
||||
|
||||
| Table | Key | Value | Description |
|
||||
|-------------------|----------------------|--------------------|-------------|
|
||||
| `BlockBlobs` | BlockHeight | `StorableVec<u8>` | Maps a block's height to a serialized byte form of a block
|
||||
| `BlockHeights` | BlockHash | BlockHeight | Maps a block's hash to its height
|
||||
| `BlockInfos` | BlockHeight | `BlockInfo` | Contains metadata of all blocks
|
||||
| `KeyImages` | KeyImage | () | This table is a set with no value, it stores transaction key images
|
||||
| `NumOutputs` | Amount | u64 | Maps an output's amount to the number of outputs with that amount
|
||||
| `Outputs` | `PreRctOutputId` | `Output` | This table contains legacy CryptoNote outputs which have clear amounts. This table will not contain an output with 0 amount.
|
||||
| `PrunedTxBlobs` | TxId | `StorableVec<u8>` | Contains pruned transaction blobs (even if the database is not pruned)
|
||||
| `PrunableTxBlobs` | TxId | `StorableVec<u8>` | Contains the prunable part of a transaction
|
||||
| `PrunableHashes` | TxId | PrunableHash | Contains the hash of the prunable part of a transaction
|
||||
| `RctOutputs` | AmountIndex | `RctOutput` | Contains RingCT outputs mapped from their global RCT index
|
||||
| `TxBlobs` | TxId | `StorableVec<u8>` | Serialized transaction blobs (bytes)
|
||||
| `TxIds` | TxHash | TxId | Maps a transaction's hash to its index/ID
|
||||
| `TxHeights` | TxId | BlockHeight | Maps a transaction's ID to the height of the block it comes from
|
||||
| `TxOutputs` | TxId | `StorableVec<u64>` | Gives the amount indices of a transaction's outputs
|
||||
| `TxUnlockTime` | TxId | UnlockTime | Stores the unlock time of a transaction (only if it has a non-zero lock time)
|
||||
|
||||
The definitions for aliases and types (e.g. `RctOutput`) are within the [`cuprate_database::types`](https://github.com/Cuprate/cuprate/blob/31ce89412aa174fc33754f22c9a6d9ef5ddeda28/database/src/types.rs#L51) module.
|
||||
|
||||
<!-- TODO(Boog900): We could split this table again into `RingCT (non-miner) Outputs` and `RingCT (miner) Outputs` as for miner outputs we can store the amount instead of commitment saving 24 bytes per miner output. -->
|
||||
|
||||
### 9.2 Multimap tables
|
||||
When referencing outputs, Monero will [use the amount and the amount index](https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/blockchain_db/lmdb/db_lmdb.cpp#L3447-L3449). This means 2 keys are needed to reach an output.
|
||||
|
||||
With LMDB you can set the `DUP_SORT` flag on a table and then set the key/value to:
|
||||
```rust
|
||||
Key = KEY_PART_1
|
||||
```
|
||||
```rust
|
||||
Value = {
|
||||
KEY_PART_2,
|
||||
VALUE // The actual value we are storing.
|
||||
}
|
||||
```
|
||||
|
||||
Then you can set a custom value sorting function that only takes `KEY_PART_2` into account; this is how `monerod` does it.
|
||||
|
||||
This requires that the underlying database supports:
|
||||
- multimap tables
|
||||
- custom sort functions on values
|
||||
- setting a cursor on a specific key/value
|
||||
|
||||
---
|
||||
|
||||
Another way to implement this is as follows:
|
||||
```rust
|
||||
Key = { KEY_PART_1, KEY_PART_2 }
|
||||
```
|
||||
```rust
|
||||
Value = VALUE
|
||||
```
|
||||
|
||||
Then the key type is simply used to look up the value; this is how `cuprate_database` does it.
|
||||
|
||||
For example, the key/value pair for outputs is:
|
||||
```rust
|
||||
PreRctOutputId => Output
|
||||
```
|
||||
where `PreRctOutputId` looks like this:
|
||||
```rust
|
||||
struct PreRctOutputId {
|
||||
amount: u64,
|
||||
amount_index: u64,
|
||||
}
|
||||
```
|
||||
|
||||
## 10. Known issues and tradeoffs
|
||||
`cuprate_database` takes many tradeoffs, whether due to:
|
||||
- Prioritizing certain values over others
|
||||
- Not having a better solution
|
||||
- Being "good enough"
|
||||
|
||||
This is a list of the larger ones, along with issues that don't have answers yet.
|
||||
|
||||
### 10.1 Traits abstracting backends
|
||||
Although all database backends used are very similar, they have some crucial differences in small implementation details that must be worked around when conforming them to `cuprate_database`'s traits.
|
||||
|
||||
Put simply: using `cuprate_database`'s traits is less efficient and more awkward than using the backend directly.
|
||||
|
||||
For example:
|
||||
- [Data types must be wrapped in compatibility layers when they otherwise wouldn't be](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/backend/heed/env.rs#L101-L116)
|
||||
- [There are types that only apply to a specific backend, but are visible to all](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/error.rs#L86-L89)
|
||||
- [There are extra layers of abstraction to smoothen the differences between all backends](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/env.rs#L62-L68)
|
||||
- [Existing functionality of backends must be taken away, as it isn't supported in the others](https://github.com/Cuprate/cuprate/blob/d0ac94a813e4cd8e0ed8da5e85a53b1d1ace2463/database/src/database.rs#L27-L34)
|
||||
|
||||
This is a _tradeoff_ that `cuprate_database` takes, as:
|
||||
- The backend itself is usually not the source of bottlenecks in the greater system, as such, small inefficiencies are OK
|
||||
- None of the lost functionality is crucial for operation
|
||||
- The ability to use, test, and swap between multiple database backends is [worth it](https://github.com/Cuprate/cuprate/pull/35#issuecomment-1952804393)
|
||||
|
||||
### 10.2 Hot-swappable backends
|
||||
Using a different backend is really as simple as re-building `cuprate_database` with a different feature flag:
|
||||
```bash
|
||||
# Use LMDB.
|
||||
cargo build --package cuprate-database --features heed
|
||||
|
||||
# Use redb.
|
||||
cargo build --package cuprate-database --features redb
|
||||
```
|
||||
|
||||
This is "good enough" for now, however ideally, this hot-swapping of backends would be able to be done at _runtime_.
|
||||
|
||||
As it is now, `cuprate_database` cannot compile both backends and swap based on user input at runtime; it must be compiled with a certain backend, which will produce a binary with only that backend.
|
||||
|
||||
This also means things like [CI testing multiple backends is awkward](https://github.com/Cuprate/cuprate/blob/main/.github/workflows/ci.yml#L132-L136), as we must re-compile with different feature flags instead.
|
||||
|
||||
### 10.3 Copying unaligned bytes
|
||||
As mentioned in [`8. (De)serialization`](#8-deserialization), bytes are _copied_ when they are turned into a type `T` due to unaligned bytes being returned from database backends.
|
||||
|
||||
Using a regular reference cast results in an improperly aligned type `T`; [such a type even existing causes undefined behavior](https://doc.rust-lang.org/reference/behavior-considered-undefined.html). In our case, `bytemuck` saves us by panicking before this occurs.
|
||||
|
||||
Thus, when using `cuprate_database`'s database traits, an _owned_ `T` is returned.
|
||||
|
||||
This is doubly unfortunately for `&[u8]` as this does not even need deserialization.
|
||||
|
||||
For example, `StorableVec` could have been this:
|
||||
```rust
|
||||
enum StorableBytes<'a, T: Storable> {
|
||||
Owned(T),
|
||||
Ref(&'a T),
|
||||
}
|
||||
```
|
||||
but this would require supporting types that must be copied regardless with the occasional `&[u8]` that can be returned without casting. This was hard to do so in a generic way, thus all `[u8]`'s are copied and returned as owned `StorableVec`s.
|
||||
|
||||
This is a _tradeoff_ `cuprate_database` takes as:
|
||||
- `bytemuck::pod_read_unaligned` is cheap enough
|
||||
- The main API, `service`, needs to return owned value anyway
|
||||
- Having no references removes a lot of lifetime complexity
|
||||
|
||||
The alternative is either:
|
||||
- Using proper (de)serialization instead of casting (which comes with its own costs)
|
||||
- Somehow fixing the alignment issues in the backends mentioned previously
|
||||
|
||||
### 10.4 Endianness
|
||||
`cuprate_database`'s (de)serialization and storage of bytes are native-endian, as in, byte storage order will depend on the machine it is running on.
|
||||
|
||||
As Cuprate's build-targets are all little-endian ([big-endian by default machines barely exist](https://en.wikipedia.org/wiki/Endianness#Hardware)), this doesn't matter much and the byte ordering can be seen as a constant.
|
||||
|
||||
Practically, this means `cuprated`'s database files can be transferred across computers, as can `monerod`'s.
|
||||
|
||||
### 10.5 Extra table data
|
||||
Some of `cuprate_database`'s tables differ from `monerod`'s tables, for example, the way [`9.2 Multimap tables`](#92-multimap-tables) tables are done requires that the primary key is stored _for all_ entries, compared to `monerod` only needing to store it once.
|
||||
|
||||
For example:
|
||||
```rust
|
||||
// `monerod` only stores `amount: 1` once,
|
||||
// `cuprated` stores it each time it appears.
|
||||
struct PreRctOutputId { amount: 1, amount_index: 0 }
|
||||
struct PreRctOutputId { amount: 1, amount_index: 1 }
|
||||
```
|
||||
|
||||
This means `cuprated`'s database will be slightly larger than `monerod`'s.
|
||||
|
||||
The current method `cuprate_database` uses will be "good enough" until usage shows that it must be optimized as multimap tables are tricky to implement across all backends.
|
||||
261
storage/database/src/backend/heed/database.rs
Normal file
261
storage/database/src/backend/heed/database.rs
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
//! Implementation of `trait Database` for `heed`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{cell::RefCell, ops::RangeBounds};
|
||||
|
||||
use crate::{
|
||||
backend::heed::types::HeedDb,
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
table::Table,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Heed Database Wrappers
|
||||
// Q. Why does `HeedTableR{o,w}` exist?
|
||||
// A. These wrapper types combine `heed`'s database/table
|
||||
// types with its transaction types. It exists to match
|
||||
// `redb`, which has this behavior built-in.
|
||||
//
|
||||
// `redb` forces us to abstract read/write semantics
|
||||
// at the _opened table_ level, so, we must match that in `heed`,
|
||||
// which abstracts it at the transaction level.
|
||||
//
|
||||
// We must also maintain the ability for
|
||||
// write operations to also read, aka, `Rw`.
|
||||
|
||||
/// An opened read-only database associated with a transaction.
|
||||
///
|
||||
/// Matches `redb::ReadOnlyTable`.
|
||||
pub(super) struct HeedTableRo<'tx, T: Table> {
|
||||
/// An already opened database table.
|
||||
pub(super) db: HeedDb<T::Key, T::Value>,
|
||||
/// The associated read-only transaction that opened this table.
|
||||
pub(super) tx_ro: &'tx heed::RoTxn<'tx>,
|
||||
}
|
||||
|
||||
/// An opened read/write database associated with a transaction.
|
||||
///
|
||||
/// Matches `redb::Table` (read & write).
|
||||
pub(super) struct HeedTableRw<'env, 'tx, T: Table> {
|
||||
/// An already opened database table.
|
||||
pub(super) db: HeedDb<T::Key, T::Value>,
|
||||
/// The associated read/write transaction that opened this table.
|
||||
pub(super) tx_rw: &'tx RefCell<heed::RwTxn<'env>>,
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Shared functions
|
||||
// FIXME: we cannot just deref `HeedTableRw -> HeedTableRo` and
|
||||
// call the functions since the database is held by value, so
|
||||
// just use these generic functions that both can call instead.
|
||||
|
||||
/// Shared [`DatabaseRo::get()`].
|
||||
#[inline]
|
||||
fn get<T: Table>(
|
||||
db: &HeedDb<T::Key, T::Value>,
|
||||
tx_ro: &heed::RoTxn<'_>,
|
||||
key: &T::Key,
|
||||
) -> Result<T::Value, RuntimeError> {
|
||||
db.get(tx_ro, key)?.ok_or(RuntimeError::KeyNotFound)
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::len()`].
|
||||
#[inline]
|
||||
fn len<T: Table>(
|
||||
db: &HeedDb<T::Key, T::Value>,
|
||||
tx_ro: &heed::RoTxn<'_>,
|
||||
) -> Result<u64, RuntimeError> {
|
||||
Ok(db.len(tx_ro)?)
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::first()`].
|
||||
#[inline]
|
||||
fn first<T: Table>(
|
||||
db: &HeedDb<T::Key, T::Value>,
|
||||
tx_ro: &heed::RoTxn<'_>,
|
||||
) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
db.first(tx_ro)?.ok_or(RuntimeError::KeyNotFound)
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::last()`].
|
||||
#[inline]
|
||||
fn last<T: Table>(
|
||||
db: &HeedDb<T::Key, T::Value>,
|
||||
tx_ro: &heed::RoTxn<'_>,
|
||||
) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
db.last(tx_ro)?.ok_or(RuntimeError::KeyNotFound)
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::is_empty()`].
|
||||
#[inline]
|
||||
fn is_empty<T: Table>(
|
||||
db: &HeedDb<T::Key, T::Value>,
|
||||
tx_ro: &heed::RoTxn<'_>,
|
||||
) -> Result<bool, RuntimeError> {
|
||||
Ok(db.is_empty(tx_ro)?)
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseIter Impl
|
||||
impl<T: Table> DatabaseIter<T> for HeedTableRo<'_, T> {
|
||||
#[inline]
|
||||
fn get_range<'a, Range>(
|
||||
&'a self,
|
||||
range: Range,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + 'a, RuntimeError>
|
||||
where
|
||||
Range: RangeBounds<T::Key> + 'a,
|
||||
{
|
||||
Ok(self.db.range(self.tx_ro, &range)?.map(|res| Ok(res?.1)))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn iter(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<(T::Key, T::Value), RuntimeError>> + '_, RuntimeError>
|
||||
{
|
||||
Ok(self.db.iter(self.tx_ro)?.map(|res| Ok(res?)))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn keys(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<T::Key, RuntimeError>> + '_, RuntimeError> {
|
||||
Ok(self.db.iter(self.tx_ro)?.map(|res| Ok(res?.0)))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn values(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + '_, RuntimeError> {
|
||||
Ok(self.db.iter(self.tx_ro)?.map(|res| Ok(res?.1)))
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRo Impl
|
||||
// SAFETY: `HeedTableRo: !Send` as it holds a reference to `heed::RoTxn: Send + !Sync`.
|
||||
unsafe impl<T: Table> DatabaseRo<T> for HeedTableRo<'_, T> {
|
||||
#[inline]
|
||||
fn get(&self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
get::<T>(&self.db, self.tx_ro, key)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn len(&self) -> Result<u64, RuntimeError> {
|
||||
len::<T>(&self.db, self.tx_ro)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn first(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
first::<T>(&self.db, self.tx_ro)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn last(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
last::<T>(&self.db, self.tx_ro)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn is_empty(&self) -> Result<bool, RuntimeError> {
|
||||
is_empty::<T>(&self.db, self.tx_ro)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRw Impl
|
||||
// SAFETY: The `Send` bound only applies to `HeedTableRo`.
|
||||
// `HeedTableRw`'s write transaction is `!Send`.
|
||||
unsafe impl<T: Table> DatabaseRo<T> for HeedTableRw<'_, '_, T> {
|
||||
#[inline]
|
||||
fn get(&self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
get::<T>(&self.db, &self.tx_rw.borrow(), key)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn len(&self) -> Result<u64, RuntimeError> {
|
||||
len::<T>(&self.db, &self.tx_rw.borrow())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn first(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
first::<T>(&self.db, &self.tx_rw.borrow())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn last(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
last::<T>(&self.db, &self.tx_rw.borrow())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn is_empty(&self) -> Result<bool, RuntimeError> {
|
||||
is_empty::<T>(&self.db, &self.tx_rw.borrow())
|
||||
}
|
||||
}
|
||||
|
||||
impl<T: Table> DatabaseRw<T> for HeedTableRw<'_, '_, T> {
|
||||
#[inline]
|
||||
fn put(&mut self, key: &T::Key, value: &T::Value) -> Result<(), RuntimeError> {
|
||||
Ok(self.db.put(&mut self.tx_rw.borrow_mut(), key, value)?)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn delete(&mut self, key: &T::Key) -> Result<(), RuntimeError> {
|
||||
self.db.delete(&mut self.tx_rw.borrow_mut(), key)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn take(&mut self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
// LMDB/heed does not return the value on deletion.
|
||||
// So, fetch it first - then delete.
|
||||
let value = get::<T>(&self.db, &self.tx_rw.borrow(), key)?;
|
||||
match self.db.delete(&mut self.tx_rw.borrow_mut(), key) {
|
||||
Ok(true) => Ok(value),
|
||||
Err(e) => Err(e.into()),
|
||||
// We just `get()`'ed the value - it is
|
||||
// incorrect for it to suddenly not exist.
|
||||
Ok(false) => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn pop_first(&mut self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let tx_rw = &mut self.tx_rw.borrow_mut();
|
||||
|
||||
// Get the value first...
|
||||
let Some((key, value)) = self.db.first(tx_rw)? else {
|
||||
return Err(RuntimeError::KeyNotFound);
|
||||
};
|
||||
|
||||
// ...then remove it.
|
||||
match self.db.delete(tx_rw, &key) {
|
||||
Ok(true) => Ok((key, value)),
|
||||
Err(e) => Err(e.into()),
|
||||
// We just `get()`'ed the value - it is
|
||||
// incorrect for it to suddenly not exist.
|
||||
Ok(false) => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn pop_last(&mut self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let tx_rw = &mut self.tx_rw.borrow_mut();
|
||||
|
||||
// Get the value first...
|
||||
let Some((key, value)) = self.db.last(tx_rw)? else {
|
||||
return Err(RuntimeError::KeyNotFound);
|
||||
};
|
||||
|
||||
// ...then remove it.
|
||||
match self.db.delete(tx_rw, &key) {
|
||||
Ok(true) => Ok((key, value)),
|
||||
Err(e) => Err(e.into()),
|
||||
// We just `get()`'ed the value - it is
|
||||
// incorrect for it to suddenly not exist.
|
||||
Ok(false) => unreachable!(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
347
storage/database/src/backend/heed/env.rs
Normal file
347
storage/database/src/backend/heed/env.rs
Normal file
|
|
@ -0,0 +1,347 @@
|
|||
//! Implementation of `trait Env` for `heed`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{
|
||||
cell::RefCell,
|
||||
num::NonZeroUsize,
|
||||
sync::{RwLock, RwLockReadGuard},
|
||||
};
|
||||
|
||||
use heed::{DatabaseOpenOptions, EnvFlags, EnvOpenOptions};
|
||||
|
||||
use crate::{
|
||||
backend::heed::{
|
||||
database::{HeedTableRo, HeedTableRw},
|
||||
storable::StorableHeed,
|
||||
types::HeedDb,
|
||||
},
|
||||
config::{Config, SyncMode},
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
env::{Env, EnvInner},
|
||||
error::{InitError, RuntimeError},
|
||||
resize::ResizeAlgorithm,
|
||||
table::Table,
|
||||
tables::call_fn_on_all_tables_or_early_return,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Consts
|
||||
/// Panic message when there's a table missing.
|
||||
const PANIC_MSG_MISSING_TABLE: &str =
|
||||
"cuprate_database::Env should uphold the invariant that all tables are already created";
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- ConcreteEnv
|
||||
/// A strongly typed, concrete database environment, backed by `heed`.
|
||||
pub struct ConcreteEnv {
|
||||
/// The actual database environment.
|
||||
///
|
||||
/// # Why `RwLock`?
|
||||
/// We need mutual exclusive access to the environment for resizing.
|
||||
///
|
||||
/// Using 2 atomics for mutual exclusion was considered:
|
||||
/// - `currently_resizing: AtomicBool`
|
||||
/// - `reader_count: AtomicUsize`
|
||||
///
|
||||
/// This is how `monerod` does it:
|
||||
/// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L354-L355>
|
||||
///
|
||||
/// `currently_resizing` would be set to `true` on resizes and
|
||||
/// `reader_count` would be spinned on until 0, at which point
|
||||
/// we are safe to resize.
|
||||
///
|
||||
/// Although, 3 atomic operations (check atomic bool, `reader_count++`, `reader_count--`)
|
||||
/// turns out to be roughly as expensive as acquiring a non-contended `RwLock`,
|
||||
/// the CPU sleeping instead of spinning is much better too.
|
||||
///
|
||||
/// # `unwrap()`
|
||||
/// This will be [`unwrap()`]ed everywhere.
|
||||
///
|
||||
/// If lock is poisoned, we want all of Cuprate to panic.
|
||||
env: RwLock<heed::Env>,
|
||||
|
||||
/// The configuration we were opened with
|
||||
/// (and in current use).
|
||||
pub(super) config: Config,
|
||||
}
|
||||
|
||||
impl Drop for ConcreteEnv {
|
||||
fn drop(&mut self) {
|
||||
// INVARIANT: drop(ConcreteEnv) must sync.
|
||||
//
|
||||
// SOMEDAY:
|
||||
// "if the environment has the MDB_NOSYNC flag set the flushes will be omitted,
|
||||
// and with MDB_MAPASYNC they will be asynchronous."
|
||||
// <http://www.lmdb.tech/doc/group__mdb.html#ga85e61f05aa68b520cc6c3b981dba5037>
|
||||
//
|
||||
// We need to do `mdb_env_set_flags(&env, MDB_NOSYNC|MDB_ASYNCMAP, 0)`
|
||||
// to clear the no sync and async flags such that the below `self.sync()`
|
||||
// _actually_ synchronously syncs.
|
||||
if let Err(_e) = crate::Env::sync(self) {
|
||||
// TODO: log error?
|
||||
}
|
||||
|
||||
// TODO: log that we are dropping the database.
|
||||
|
||||
// TODO: use tracing.
|
||||
// <https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/lmdb.h#L49-L61>
|
||||
let result = self.env.read().unwrap().clear_stale_readers();
|
||||
match result {
|
||||
Ok(n) => println!("LMDB stale readers cleared: {n}"),
|
||||
Err(e) => println!("LMDB stale reader clear error: {e:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Env Impl
|
||||
impl Env for ConcreteEnv {
|
||||
const MANUAL_RESIZE: bool = true;
|
||||
const SYNCS_PER_TX: bool = false;
|
||||
type EnvInner<'env> = RwLockReadGuard<'env, heed::Env>;
|
||||
type TxRo<'tx> = heed::RoTxn<'tx>;
|
||||
|
||||
/// HACK:
|
||||
/// `heed::RwTxn` is wrapped in `RefCell` to allow:
|
||||
/// - opening a database with only a `&` to it
|
||||
/// - allowing 1 write tx to open multiple tables
|
||||
///
|
||||
/// Our mutable accesses are safe and will not panic as:
|
||||
/// - Write transactions are `!Sync`
|
||||
/// - A table operation does not hold a reference to the inner cell
|
||||
/// once the call is over
|
||||
/// - The function to manipulate the table takes the same type
|
||||
/// of reference that the `RefCell` gets for that function
|
||||
///
|
||||
/// Also see:
|
||||
/// - <https://github.com/Cuprate/cuprate/pull/102#discussion_r1548695610>
|
||||
/// - <https://github.com/Cuprate/cuprate/pull/104>
|
||||
type TxRw<'tx> = RefCell<heed::RwTxn<'tx>>;
|
||||
|
||||
#[cold]
|
||||
#[inline(never)] // called once.
|
||||
fn open(config: Config) -> Result<Self, InitError> {
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1324>
|
||||
|
||||
let mut env_open_options = EnvOpenOptions::new();
|
||||
|
||||
// Map our `Config` sync mode to the LMDB environment flags.
|
||||
//
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1324>
|
||||
let flags = match config.sync_mode {
|
||||
SyncMode::Safe => EnvFlags::empty(),
|
||||
SyncMode::Async => EnvFlags::MAP_ASYNC,
|
||||
SyncMode::Fast => EnvFlags::NO_SYNC | EnvFlags::WRITE_MAP | EnvFlags::MAP_ASYNC,
|
||||
// SOMEDAY: dynamic syncs are not implemented.
|
||||
SyncMode::FastThenSafe | SyncMode::Threshold(_) => unimplemented!(),
|
||||
};
|
||||
|
||||
// SAFETY: the flags we're setting are 'unsafe'
|
||||
// from a data durability perspective, although,
|
||||
// the user config wanted this.
|
||||
//
|
||||
// MAYBE: We may need to open/create tables with certain flags
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1324>
|
||||
// MAYBE: Set comparison functions for certain tables
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1324>
|
||||
unsafe {
|
||||
env_open_options.flags(flags);
|
||||
}
|
||||
|
||||
// Set the memory map size to
|
||||
// (current disk size) + (a bit of leeway)
|
||||
// to account for empty databases where we
|
||||
// need to write same tables.
|
||||
#[allow(clippy::cast_possible_truncation)] // only 64-bit targets
|
||||
let disk_size_bytes = match std::fs::File::open(&config.db_file) {
|
||||
Ok(file) => file.metadata()?.len() as usize,
|
||||
// The database file doesn't exist, 0 bytes.
|
||||
Err(io_err) if io_err.kind() == std::io::ErrorKind::NotFound => 0,
|
||||
Err(io_err) => return Err(io_err.into()),
|
||||
};
|
||||
// Add leeway space.
|
||||
let memory_map_size = crate::resize::fixed_bytes(disk_size_bytes, 1_000_000 /* 1MB */);
|
||||
env_open_options.map_size(memory_map_size.get());
|
||||
|
||||
// Set the max amount of database tables.
|
||||
// We know at compile time how many tables there are.
|
||||
// SOMEDAY: ...how many?
|
||||
env_open_options.max_dbs(32);
|
||||
|
||||
// LMDB documentation:
|
||||
// ```
|
||||
// Number of slots in the reader table.
|
||||
// This value was chosen somewhat arbitrarily. 126 readers plus a
|
||||
// couple mutexes fit exactly into 8KB on my development machine.
|
||||
// ```
|
||||
// <https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/mdb.c#L794-L799>
|
||||
//
|
||||
// So, we're going to be following these rules:
|
||||
// - Use at least 126 reader threads
|
||||
// - Add 16 extra reader threads if <126
|
||||
//
|
||||
// FIXME: This behavior is from `monerod`:
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L1324>
|
||||
// I believe this could be adjusted percentage-wise so very high
|
||||
// thread PCs can benefit from something like (cuprated + anything that uses the DB in the future).
|
||||
// For now:
|
||||
// - No other program using our DB exists
|
||||
// - Almost no-one has a 126+ thread CPU
|
||||
let reader_threads =
|
||||
u32::try_from(config.reader_threads.as_threads().get()).unwrap_or(u32::MAX);
|
||||
env_open_options.max_readers(if reader_threads < 110 {
|
||||
126
|
||||
} else {
|
||||
reader_threads.saturating_add(16)
|
||||
});
|
||||
|
||||
// Create the database directory if it doesn't exist.
|
||||
std::fs::create_dir_all(config.db_directory())?;
|
||||
// Open the environment in the user's PATH.
|
||||
// SAFETY: LMDB uses a memory-map backed file.
|
||||
// <https://docs.rs/heed/0.20.0/heed/struct.EnvOpenOptions.html#method.open>
|
||||
let env = unsafe { env_open_options.open(config.db_directory())? };
|
||||
|
||||
/// Function that creates the tables based off the passed `T: Table`.
|
||||
fn create_table<T: Table>(
|
||||
env: &heed::Env,
|
||||
tx_rw: &mut heed::RwTxn<'_>,
|
||||
) -> Result<(), InitError> {
|
||||
DatabaseOpenOptions::new(env)
|
||||
.name(<T as Table>::NAME)
|
||||
.types::<StorableHeed<<T as Table>::Key>, StorableHeed<<T as Table>::Value>>()
|
||||
.create(tx_rw)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
let mut tx_rw = env.write_txn()?;
|
||||
// Create all tables.
|
||||
// FIXME: this macro is kinda awkward.
|
||||
{
|
||||
let env = &env;
|
||||
let tx_rw = &mut tx_rw;
|
||||
match call_fn_on_all_tables_or_early_return!(create_table(env, tx_rw)) {
|
||||
Ok(_) => (),
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
// INVARIANT: this should never return `ResizeNeeded` due to adding
|
||||
// some tables since we added some leeway to the memory map above.
|
||||
tx_rw.commit()?;
|
||||
|
||||
Ok(Self {
|
||||
env: RwLock::new(env),
|
||||
config,
|
||||
})
|
||||
}
|
||||
|
||||
fn config(&self) -> &Config {
|
||||
&self.config
|
||||
}
|
||||
|
||||
fn sync(&self) -> Result<(), RuntimeError> {
|
||||
Ok(self.env.read().unwrap().force_sync()?)
|
||||
}
|
||||
|
||||
fn resize_map(&self, resize_algorithm: Option<ResizeAlgorithm>) -> NonZeroUsize {
|
||||
let resize_algorithm = resize_algorithm.unwrap_or_else(|| self.config().resize_algorithm);
|
||||
|
||||
let current_size_bytes = self.current_map_size();
|
||||
let new_size_bytes = resize_algorithm.resize(current_size_bytes);
|
||||
|
||||
// SAFETY:
|
||||
// Resizing requires that we have
|
||||
// exclusive access to the database environment.
|
||||
// Our `heed::Env` is wrapped within a `RwLock`,
|
||||
// and we have a WriteGuard to it, so we're safe.
|
||||
//
|
||||
// <http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5>
|
||||
unsafe {
|
||||
// INVARIANT: `resize()` returns a valid `usize` to resize to.
|
||||
self.env
|
||||
.write()
|
||||
.unwrap()
|
||||
.resize(new_size_bytes.get())
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
new_size_bytes
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn current_map_size(&self) -> usize {
|
||||
self.env.read().unwrap().info().map_size
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn env_inner(&self) -> Self::EnvInner<'_> {
|
||||
self.env.read().unwrap()
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- EnvInner Impl
|
||||
impl<'env> EnvInner<'env, heed::RoTxn<'env>, RefCell<heed::RwTxn<'env>>>
|
||||
for RwLockReadGuard<'env, heed::Env>
|
||||
where
|
||||
Self: 'env,
|
||||
{
|
||||
#[inline]
|
||||
fn tx_ro(&'env self) -> Result<heed::RoTxn<'env>, RuntimeError> {
|
||||
Ok(self.read_txn()?)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn tx_rw(&'env self) -> Result<RefCell<heed::RwTxn<'env>>, RuntimeError> {
|
||||
Ok(RefCell::new(self.write_txn()?))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn open_db_ro<T: Table>(
|
||||
&self,
|
||||
tx_ro: &heed::RoTxn<'env>,
|
||||
) -> Result<impl DatabaseRo<T> + DatabaseIter<T>, RuntimeError> {
|
||||
// Open up a read-only database using our table's const metadata.
|
||||
Ok(HeedTableRo {
|
||||
db: self
|
||||
.open_database(tx_ro, Some(T::NAME))?
|
||||
.expect(PANIC_MSG_MISSING_TABLE),
|
||||
tx_ro,
|
||||
})
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn open_db_rw<T: Table>(
|
||||
&self,
|
||||
tx_rw: &RefCell<heed::RwTxn<'env>>,
|
||||
) -> Result<impl DatabaseRw<T>, RuntimeError> {
|
||||
let tx_ro = tx_rw.borrow();
|
||||
|
||||
// Open up a read/write database using our table's const metadata.
|
||||
Ok(HeedTableRw {
|
||||
db: self
|
||||
.open_database(&tx_ro, Some(T::NAME))?
|
||||
.expect(PANIC_MSG_MISSING_TABLE),
|
||||
tx_rw,
|
||||
})
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn clear_db<T: Table>(
|
||||
&self,
|
||||
tx_rw: &mut RefCell<heed::RwTxn<'env>>,
|
||||
) -> Result<(), RuntimeError> {
|
||||
let tx_rw = tx_rw.get_mut();
|
||||
|
||||
// Open the table first...
|
||||
let db: HeedDb<T::Key, T::Value> = self
|
||||
.open_database(tx_rw, Some(T::NAME))?
|
||||
.expect(PANIC_MSG_MISSING_TABLE);
|
||||
|
||||
// ...then clear it.
|
||||
Ok(db.clear(tx_rw)?)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
152
storage/database/src/backend/heed/error.rs
Normal file
152
storage/database/src/backend/heed/error.rs
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
//! Conversion from `heed::Error` -> `cuprate_database`'s errors.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use crate::constants::DATABASE_CORRUPT_MSG;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- InitError
|
||||
impl From<heed::Error> for crate::InitError {
|
||||
fn from(error: heed::Error) -> Self {
|
||||
use heed::Error as E1;
|
||||
use heed::MdbError as E2;
|
||||
|
||||
// Reference of all possible errors `heed` will return
|
||||
// upon using [`heed::EnvOpenOptions::open`]:
|
||||
// <https://docs.rs/heed/latest/src/heed/env.rs.html#149-219>
|
||||
match error {
|
||||
E1::Io(io_error) => Self::Io(io_error),
|
||||
E1::DatabaseClosing => Self::ShuttingDown,
|
||||
|
||||
// LMDB errors.
|
||||
E1::Mdb(mdb_error) => match mdb_error {
|
||||
E2::Invalid => Self::Invalid,
|
||||
E2::VersionMismatch => Self::InvalidVersion,
|
||||
|
||||
// "Located page was wrong type".
|
||||
// <https://docs.rs/heed/latest/heed/enum.MdbError.html#variant.Corrupted>
|
||||
//
|
||||
// "Requested page not found - this usually indicates corruption."
|
||||
// <https://docs.rs/heed/latest/heed/enum.MdbError.html#variant.PageNotFound>
|
||||
E2::Corrupted | E2::PageNotFound => Self::Corrupt,
|
||||
|
||||
// These errors shouldn't be returned on database init.
|
||||
E2::Incompatible
|
||||
| E2::Other(_)
|
||||
| E2::BadTxn
|
||||
| E2::Problem
|
||||
| E2::KeyExist
|
||||
| E2::NotFound
|
||||
| E2::MapFull
|
||||
| E2::ReadersFull
|
||||
| E2::PageFull
|
||||
| E2::DbsFull
|
||||
| E2::TlsFull
|
||||
| E2::TxnFull
|
||||
| E2::CursorFull
|
||||
| E2::MapResized
|
||||
| E2::BadRslot
|
||||
| E2::BadValSize
|
||||
| E2::BadDbi
|
||||
| E2::Panic => Self::Unknown(Box::new(mdb_error)),
|
||||
},
|
||||
|
||||
E1::BadOpenOptions { .. } | E1::Encoding(_) | E1::Decoding(_) => {
|
||||
Self::Unknown(Box::new(error))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- RuntimeError
|
||||
#[allow(clippy::fallible_impl_from)] // We need to panic sometimes.
|
||||
impl From<heed::Error> for crate::RuntimeError {
|
||||
/// # Panics
|
||||
/// This will panic on unrecoverable errors for safety.
|
||||
fn from(error: heed::Error) -> Self {
|
||||
use heed::Error as E1;
|
||||
use heed::MdbError as E2;
|
||||
|
||||
match error {
|
||||
// I/O errors.
|
||||
E1::Io(io_error) => Self::Io(io_error),
|
||||
|
||||
// LMDB errors.
|
||||
E1::Mdb(mdb_error) => match mdb_error {
|
||||
E2::KeyExist => Self::KeyExists,
|
||||
E2::NotFound => Self::KeyNotFound,
|
||||
E2::MapFull => Self::ResizeNeeded,
|
||||
|
||||
// Corruption errors, these have special panic messages.
|
||||
//
|
||||
// "Located page was wrong type".
|
||||
// <https://docs.rs/heed/latest/heed/enum.MdbError.html#variant.Corrupted>
|
||||
//
|
||||
// "Requested page not found - this usually indicates corruption."
|
||||
// <https://docs.rs/heed/latest/heed/enum.MdbError.html#variant.PageNotFound>
|
||||
E2::Corrupted | E2::PageNotFound => panic!("{mdb_error:#?}\n{DATABASE_CORRUPT_MSG}"),
|
||||
|
||||
// These errors should not occur, and if they do,
|
||||
// the best thing `cuprate_database` can do for
|
||||
// safety is to panic right here.
|
||||
E2::Panic
|
||||
| E2::PageFull
|
||||
| E2::Other(_)
|
||||
| E2::BadTxn
|
||||
| E2::Problem
|
||||
| E2::Invalid
|
||||
| E2::TlsFull
|
||||
| E2::TxnFull
|
||||
| E2::BadRslot
|
||||
| E2::VersionMismatch
|
||||
| E2::BadDbi => panic!("{mdb_error:#?}"),
|
||||
|
||||
// These errors are the same as above, but instead
|
||||
// of being errors we can't control, these are errors
|
||||
// that only happen if we write incorrect code.
|
||||
|
||||
// "Database contents grew beyond environment mapsize."
|
||||
// We should be resizing the map when needed, this error
|
||||
// occurring indicates we did _not_ do that, which is a bug
|
||||
// and we should panic.
|
||||
//
|
||||
// FIXME: This can also mean _another_ process wrote to our
|
||||
// LMDB file and increased the size. I don't think we need to accommodate for this.
|
||||
// <http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5>
|
||||
// Although `monerod` reacts to that instead of `MDB_MAP_FULL`
|
||||
// which is what `mdb_put()` returns so... idk?
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L526>
|
||||
| E2::MapResized
|
||||
// We should be setting `heed::EnvOpenOptions::max_readers()`
|
||||
// with our reader thread value in [`crate::config::Config`],
|
||||
// thus this error should never occur.
|
||||
// <http://www.lmdb.tech/doc/group__mdb.html#gae687966c24b790630be2a41573fe40e2>
|
||||
| E2::ReadersFull
|
||||
// Do not open more database tables than we initially started with.
|
||||
// We know this number at compile time (amount of `Table`'s) so this
|
||||
// should never happen.
|
||||
// <https://docs.rs/heed/0.20.0-alpha.9/heed/struct.EnvOpenOptions.html#method.max_dbs>
|
||||
// <https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#251>
|
||||
| E2::DbsFull
|
||||
// Don't do crazy multi-nested LMDB cursor stuff.
|
||||
| E2::CursorFull
|
||||
// <https://docs.rs/heed/0.20.0-alpha.9/heed/enum.MdbError.html#variant.Incompatible>
|
||||
| E2::Incompatible
|
||||
// Unsupported size of key/DB name/data, or wrong DUP_FIXED size.
|
||||
// Don't use a key that is `>511` bytes.
|
||||
// <http://www.lmdb.tech/doc/group__mdb.html#gaaf0be004f33828bf2fb09d77eb3cef94>
|
||||
| E2::BadValSize
|
||||
=> panic!("fix the database code! {mdb_error:#?}"),
|
||||
},
|
||||
|
||||
// Only if we write incorrect code.
|
||||
E1::DatabaseClosing | E1::BadOpenOptions { .. } | E1::Encoding(_) | E1::Decoding(_) => {
|
||||
panic!("fix the database code! {error:#?}")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
10
storage/database/src/backend/heed/mod.rs
Normal file
10
storage/database/src/backend/heed/mod.rs
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
//! Database backend implementation backed by `heed`.
|
||||
|
||||
mod env;
|
||||
pub use env::ConcreteEnv;
|
||||
|
||||
mod database;
|
||||
mod error;
|
||||
mod storable;
|
||||
mod transaction;
|
||||
mod types;
|
||||
122
storage/database/src/backend/heed/storable.rs
Normal file
122
storage/database/src/backend/heed/storable.rs
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
//! `cuprate_database::Storable` <-> `heed` serde trait compatibility layer.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use std::{borrow::Cow, marker::PhantomData};
|
||||
|
||||
use heed::{BoxedError, BytesDecode, BytesEncode};
|
||||
|
||||
use crate::storable::Storable;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- StorableHeed
|
||||
/// The glue struct that implements `heed`'s (de)serialization
|
||||
/// traits on any type that implements `cuprate_database::Storable`.
|
||||
///
|
||||
/// Never actually gets constructed, just used for trait bound translations.
|
||||
pub(super) struct StorableHeed<T>(PhantomData<T>)
|
||||
where
|
||||
T: Storable + ?Sized;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- BytesDecode
|
||||
impl<'a, T> BytesDecode<'a> for StorableHeed<T>
|
||||
where
|
||||
T: Storable + 'static,
|
||||
{
|
||||
type DItem = T;
|
||||
|
||||
#[inline]
|
||||
/// This function is infallible (will always return `Ok`).
|
||||
fn bytes_decode(bytes: &'a [u8]) -> Result<Self::DItem, BoxedError> {
|
||||
Ok(T::from_bytes(bytes))
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- BytesEncode
|
||||
impl<'a, T> BytesEncode<'a> for StorableHeed<T>
|
||||
where
|
||||
T: Storable + ?Sized + 'a,
|
||||
{
|
||||
type EItem = T;
|
||||
|
||||
#[inline]
|
||||
/// This function is infallible (will always return `Ok`).
|
||||
fn bytes_encode(item: &'a Self::EItem) -> Result<Cow<'a, [u8]>, BoxedError> {
|
||||
Ok(Cow::Borrowed(item.as_bytes()))
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use std::fmt::Debug;
|
||||
|
||||
use super::*;
|
||||
use crate::{StorableBytes, StorableVec};
|
||||
|
||||
// Each `#[test]` function has a `test()` to:
|
||||
// - log
|
||||
// - simplify trait bounds
|
||||
// - make sure the right function is being called
|
||||
|
||||
#[test]
|
||||
/// Assert `BytesEncode::bytes_encode` is accurate.
|
||||
fn bytes_encode() {
|
||||
fn test<T>(t: &T, expected: &[u8])
|
||||
where
|
||||
T: Storable + ?Sized,
|
||||
{
|
||||
println!("t: {t:?}, expected: {expected:?}");
|
||||
assert_eq!(
|
||||
<StorableHeed::<T> as BytesEncode>::bytes_encode(t).unwrap(),
|
||||
expected
|
||||
);
|
||||
}
|
||||
|
||||
test::<()>(&(), &[]);
|
||||
test::<u8>(&0, &[0]);
|
||||
test::<u16>(&1, &[1, 0]);
|
||||
test::<u32>(&2, &[2, 0, 0, 0]);
|
||||
test::<u64>(&3, &[3, 0, 0, 0, 0, 0, 0, 0]);
|
||||
test::<i8>(&-1, &[255]);
|
||||
test::<i16>(&-2, &[254, 255]);
|
||||
test::<i32>(&-3, &[253, 255, 255, 255]);
|
||||
test::<i64>(&-4, &[252, 255, 255, 255, 255, 255, 255, 255]);
|
||||
test::<StorableVec<u8>>(&StorableVec(vec![1, 2]), &[1, 2]);
|
||||
test::<StorableBytes>(&StorableBytes(bytes::Bytes::from_static(&[1, 2])), &[1, 2]);
|
||||
test::<[u8; 0]>(&[], &[]);
|
||||
test::<[u8; 1]>(&[255], &[255]);
|
||||
test::<[u8; 2]>(&[111, 0], &[111, 0]);
|
||||
test::<[u8; 3]>(&[1, 0, 1], &[1, 0, 1]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// Assert `BytesDecode::bytes_decode` is accurate.
|
||||
fn bytes_decode() {
|
||||
fn test<T>(bytes: &[u8], expected: &T)
|
||||
where
|
||||
T: Storable + PartialEq + ToOwned + Debug + 'static,
|
||||
T::Owned: Debug,
|
||||
{
|
||||
println!("bytes: {bytes:?}, expected: {expected:?}");
|
||||
assert_eq!(
|
||||
&<StorableHeed::<T> as BytesDecode>::bytes_decode(bytes).unwrap(),
|
||||
expected
|
||||
);
|
||||
}
|
||||
|
||||
test::<()>([].as_slice(), &());
|
||||
test::<u8>([0].as_slice(), &0);
|
||||
test::<u16>([1, 0].as_slice(), &1);
|
||||
test::<u32>([2, 0, 0, 0].as_slice(), &2);
|
||||
test::<u64>([3, 0, 0, 0, 0, 0, 0, 0].as_slice(), &3);
|
||||
test::<i8>([255].as_slice(), &-1);
|
||||
test::<i16>([254, 255].as_slice(), &-2);
|
||||
test::<i32>([253, 255, 255, 255].as_slice(), &-3);
|
||||
test::<i64>([252, 255, 255, 255, 255, 255, 255, 255].as_slice(), &-4);
|
||||
test::<StorableVec<u8>>(&[1, 2], &StorableVec(vec![1, 2]));
|
||||
test::<StorableBytes>(&[1, 2], &StorableBytes(bytes::Bytes::from_static(&[1, 2])));
|
||||
test::<[u8; 0]>([].as_slice(), &[]);
|
||||
test::<[u8; 1]>([255].as_slice(), &[255]);
|
||||
test::<[u8; 2]>([111, 0].as_slice(), &[111, 0]);
|
||||
test::<[u8; 3]>([1, 0, 1].as_slice(), &[1, 0, 1]);
|
||||
}
|
||||
}
|
||||
41
storage/database/src/backend/heed/transaction.rs
Normal file
41
storage/database/src/backend/heed/transaction.rs
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
//! Implementation of `trait TxRo/TxRw` for `heed`.
|
||||
|
||||
use std::cell::RefCell;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
error::RuntimeError,
|
||||
transaction::{TxRo, TxRw},
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- TxRo
|
||||
impl TxRo<'_> for heed::RoTxn<'_> {
|
||||
fn commit(self) -> Result<(), RuntimeError> {
|
||||
Ok(heed::RoTxn::commit(self)?)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- TxRw
|
||||
impl TxRo<'_> for RefCell<heed::RwTxn<'_>> {
|
||||
fn commit(self) -> Result<(), RuntimeError> {
|
||||
TxRw::commit(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl TxRw<'_> for RefCell<heed::RwTxn<'_>> {
|
||||
fn commit(self) -> Result<(), RuntimeError> {
|
||||
Ok(heed::RwTxn::commit(self.into_inner())?)
|
||||
}
|
||||
|
||||
/// This function is infallible.
|
||||
fn abort(self) -> Result<(), RuntimeError> {
|
||||
heed::RwTxn::abort(self.into_inner());
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
8
storage/database/src/backend/heed/types.rs
Normal file
8
storage/database/src/backend/heed/types.rs
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
//! `heed` type aliases.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use crate::backend::heed::storable::StorableHeed;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Types
|
||||
/// The concrete database type for `heed`, usable for reads and writes.
|
||||
pub(super) type HeedDb<K, V> = heed::Database<StorableHeed<K>, StorableHeed<V>>;
|
||||
16
storage/database/src/backend/mod.rs
Normal file
16
storage/database/src/backend/mod.rs
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
//! Database backends.
|
||||
|
||||
cfg_if::cfg_if! {
|
||||
// If both backends are enabled, fallback to `heed`.
|
||||
// This is useful when using `--all-features`.
|
||||
if #[cfg(all(feature = "redb", not(feature = "heed")))] {
|
||||
mod redb;
|
||||
pub use redb::ConcreteEnv;
|
||||
} else {
|
||||
mod heed;
|
||||
pub use heed::ConcreteEnv;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
213
storage/database/src/backend/redb/database.rs
Normal file
213
storage/database/src/backend/redb/database.rs
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
//! Implementation of `trait DatabaseR{o,w}` for `redb`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::ops::RangeBounds;
|
||||
|
||||
use redb::ReadableTable;
|
||||
|
||||
use crate::{
|
||||
backend::redb::{
|
||||
storable::StorableRedb,
|
||||
types::{RedbTableRo, RedbTableRw},
|
||||
},
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
table::Table,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Shared functions
|
||||
// FIXME: we cannot just deref `RedbTableRw -> RedbTableRo` and
|
||||
// call the functions since the database is held by value, so
|
||||
// just use these generic functions that both can call instead.
|
||||
|
||||
/// Shared [`DatabaseRo::get()`].
|
||||
#[inline]
|
||||
fn get<T: Table + 'static>(
|
||||
db: &impl redb::ReadableTable<StorableRedb<T::Key>, StorableRedb<T::Value>>,
|
||||
key: &T::Key,
|
||||
) -> Result<T::Value, RuntimeError> {
|
||||
Ok(db.get(key)?.ok_or(RuntimeError::KeyNotFound)?.value())
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::len()`].
|
||||
#[inline]
|
||||
fn len<T: Table>(
|
||||
db: &impl redb::ReadableTable<StorableRedb<T::Key>, StorableRedb<T::Value>>,
|
||||
) -> Result<u64, RuntimeError> {
|
||||
Ok(db.len()?)
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::first()`].
|
||||
#[inline]
|
||||
fn first<T: Table>(
|
||||
db: &impl redb::ReadableTable<StorableRedb<T::Key>, StorableRedb<T::Value>>,
|
||||
) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let (key, value) = db.first()?.ok_or(RuntimeError::KeyNotFound)?;
|
||||
Ok((key.value(), value.value()))
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::last()`].
|
||||
#[inline]
|
||||
fn last<T: Table>(
|
||||
db: &impl redb::ReadableTable<StorableRedb<T::Key>, StorableRedb<T::Value>>,
|
||||
) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let (key, value) = db.last()?.ok_or(RuntimeError::KeyNotFound)?;
|
||||
Ok((key.value(), value.value()))
|
||||
}
|
||||
|
||||
/// Shared [`DatabaseRo::is_empty()`].
|
||||
#[inline]
|
||||
fn is_empty<T: Table>(
|
||||
db: &impl redb::ReadableTable<StorableRedb<T::Key>, StorableRedb<T::Value>>,
|
||||
) -> Result<bool, RuntimeError> {
|
||||
Ok(db.is_empty()?)
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseIter
|
||||
impl<T: Table + 'static> DatabaseIter<T> for RedbTableRo<T::Key, T::Value> {
|
||||
#[inline]
|
||||
fn get_range<'a, Range>(
|
||||
&'a self,
|
||||
range: Range,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + 'a, RuntimeError>
|
||||
where
|
||||
Range: RangeBounds<T::Key> + 'a,
|
||||
{
|
||||
Ok(ReadableTable::range(self, range)?.map(|result| {
|
||||
let (_key, value) = result?;
|
||||
Ok(value.value())
|
||||
}))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn iter(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<(T::Key, T::Value), RuntimeError>> + '_, RuntimeError>
|
||||
{
|
||||
Ok(ReadableTable::iter(self)?.map(|result| {
|
||||
let (key, value) = result?;
|
||||
Ok((key.value(), value.value()))
|
||||
}))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn keys(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<T::Key, RuntimeError>> + '_, RuntimeError> {
|
||||
Ok(ReadableTable::iter(self)?.map(|result| {
|
||||
let (key, _value) = result?;
|
||||
Ok(key.value())
|
||||
}))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn values(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + '_, RuntimeError> {
|
||||
Ok(ReadableTable::iter(self)?.map(|result| {
|
||||
let (_key, value) = result?;
|
||||
Ok(value.value())
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRo
|
||||
// SAFETY: Both `redb`'s transaction and table types are `Send + Sync`.
|
||||
unsafe impl<T: Table + 'static> DatabaseRo<T> for RedbTableRo<T::Key, T::Value> {
|
||||
#[inline]
|
||||
fn get(&self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
get::<T>(self, key)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn len(&self) -> Result<u64, RuntimeError> {
|
||||
len::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn first(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
first::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn last(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
last::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn is_empty(&self) -> Result<bool, RuntimeError> {
|
||||
is_empty::<T>(self)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRw
|
||||
// SAFETY: Both `redb`'s transaction and table types are `Send + Sync`.
|
||||
unsafe impl<T: Table + 'static> DatabaseRo<T> for RedbTableRw<'_, T::Key, T::Value> {
|
||||
#[inline]
|
||||
fn get(&self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
get::<T>(self, key)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn len(&self) -> Result<u64, RuntimeError> {
|
||||
len::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn first(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
first::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn last(&self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
last::<T>(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn is_empty(&self) -> Result<bool, RuntimeError> {
|
||||
is_empty::<T>(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl<T: Table + 'static> DatabaseRw<T> for RedbTableRw<'_, T::Key, T::Value> {
|
||||
// `redb` returns the value after function calls so we end with Ok(()) instead.
|
||||
|
||||
#[inline]
|
||||
fn put(&mut self, key: &T::Key, value: &T::Value) -> Result<(), RuntimeError> {
|
||||
redb::Table::insert(self, key, value)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn delete(&mut self, key: &T::Key) -> Result<(), RuntimeError> {
|
||||
redb::Table::remove(self, key)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn take(&mut self, key: &T::Key) -> Result<T::Value, RuntimeError> {
|
||||
if let Some(value) = redb::Table::remove(self, key)? {
|
||||
Ok(value.value())
|
||||
} else {
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn pop_first(&mut self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let (key, value) = redb::Table::pop_first(self)?.ok_or(RuntimeError::KeyNotFound)?;
|
||||
Ok((key.value(), value.value()))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn pop_last(&mut self) -> Result<(T::Key, T::Value), RuntimeError> {
|
||||
let (key, value) = redb::Table::pop_last(self)?.ok_or(RuntimeError::KeyNotFound)?;
|
||||
Ok((key.value(), value.value()))
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
226
storage/database/src/backend/redb/env.rs
Normal file
226
storage/database/src/backend/redb/env.rs
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
//! Implementation of `trait Env` for `redb`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
backend::redb::storable::StorableRedb,
|
||||
config::{Config, SyncMode},
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
env::{Env, EnvInner},
|
||||
error::{InitError, RuntimeError},
|
||||
table::Table,
|
||||
tables::call_fn_on_all_tables_or_early_return,
|
||||
TxRw,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- ConcreteEnv
|
||||
/// A strongly typed, concrete database environment, backed by `redb`.
|
||||
pub struct ConcreteEnv {
|
||||
/// The actual database environment.
|
||||
env: redb::Database,
|
||||
|
||||
/// The configuration we were opened with
|
||||
/// (and in current use).
|
||||
config: Config,
|
||||
|
||||
/// A cached, redb version of `cuprate_database::config::SyncMode`.
|
||||
/// `redb` needs the sync mode to be set _per_ TX, so we
|
||||
/// will continue to use this value every `Env::tx_rw`.
|
||||
durability: redb::Durability,
|
||||
}
|
||||
|
||||
impl Drop for ConcreteEnv {
|
||||
fn drop(&mut self) {
|
||||
// INVARIANT: drop(ConcreteEnv) must sync.
|
||||
if let Err(e) = self.sync() {
|
||||
// TODO: use tracing
|
||||
println!("{e:#?}");
|
||||
}
|
||||
|
||||
// TODO: log that we are dropping the database.
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Env Impl
|
||||
impl Env for ConcreteEnv {
|
||||
const MANUAL_RESIZE: bool = false;
|
||||
const SYNCS_PER_TX: bool = false;
|
||||
type EnvInner<'env> = (&'env redb::Database, redb::Durability);
|
||||
type TxRo<'tx> = redb::ReadTransaction;
|
||||
type TxRw<'tx> = redb::WriteTransaction;
|
||||
|
||||
#[cold]
|
||||
#[inline(never)] // called once.
|
||||
fn open(config: Config) -> Result<Self, InitError> {
|
||||
// SOMEDAY: dynamic syncs are not implemented.
|
||||
let durability = match config.sync_mode {
|
||||
// FIXME: There's also `redb::Durability::Paranoid`:
|
||||
// <https://docs.rs/redb/1.5.0/redb/enum.Durability.html#variant.Paranoid>
|
||||
// should we use that instead of Immediate?
|
||||
SyncMode::Safe => redb::Durability::Immediate,
|
||||
SyncMode::Async => redb::Durability::Eventual,
|
||||
SyncMode::Fast => redb::Durability::None,
|
||||
// SOMEDAY: dynamic syncs are not implemented.
|
||||
SyncMode::FastThenSafe | SyncMode::Threshold(_) => unimplemented!(),
|
||||
};
|
||||
|
||||
let env_builder = redb::Builder::new();
|
||||
|
||||
// FIXME: we can set cache sizes with:
|
||||
// env_builder.set_cache(bytes);
|
||||
|
||||
// Use the in-memory backend if the feature is enabled.
|
||||
let mut env = if cfg!(feature = "redb-memory") {
|
||||
env_builder.create_with_backend(redb::backends::InMemoryBackend::new())?
|
||||
} else {
|
||||
// Create the database directory if it doesn't exist.
|
||||
std::fs::create_dir_all(config.db_directory())?;
|
||||
|
||||
// Open the database file, create if needed.
|
||||
let db_file = std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(false)
|
||||
.open(config.db_file())?;
|
||||
|
||||
env_builder.create_file(db_file)?
|
||||
};
|
||||
|
||||
// Create all database tables.
|
||||
// `redb` creates tables if they don't exist.
|
||||
// <https://docs.rs/redb/latest/redb/struct.WriteTransaction.html#method.open_table>
|
||||
|
||||
/// Function that creates the tables based off the passed `T: Table`.
|
||||
fn create_table<T: Table>(tx_rw: &redb::WriteTransaction) -> Result<(), InitError> {
|
||||
let table: redb::TableDefinition<
|
||||
'static,
|
||||
StorableRedb<<T as Table>::Key>,
|
||||
StorableRedb<<T as Table>::Value>,
|
||||
> = redb::TableDefinition::new(<T as Table>::NAME);
|
||||
|
||||
// `redb` creates tables on open if not already created.
|
||||
tx_rw.open_table(table)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Create all tables.
|
||||
// FIXME: this macro is kinda awkward.
|
||||
let mut tx_rw = env.begin_write()?;
|
||||
{
|
||||
let tx_rw = &mut tx_rw;
|
||||
match call_fn_on_all_tables_or_early_return!(create_table(tx_rw)) {
|
||||
Ok(_) => (),
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
tx_rw.commit()?;
|
||||
|
||||
// Check for file integrity.
|
||||
// FIXME: should we do this? is it slow?
|
||||
env.check_integrity()?;
|
||||
|
||||
Ok(Self {
|
||||
env,
|
||||
config,
|
||||
durability,
|
||||
})
|
||||
}
|
||||
|
||||
fn config(&self) -> &Config {
|
||||
&self.config
|
||||
}
|
||||
|
||||
fn sync(&self) -> Result<(), RuntimeError> {
|
||||
// `redb`'s syncs are tied with write transactions,
|
||||
// so just create one, don't do anything and commit.
|
||||
let mut tx_rw = self.env.begin_write()?;
|
||||
tx_rw.set_durability(redb::Durability::Paranoid);
|
||||
TxRw::commit(tx_rw)
|
||||
}
|
||||
|
||||
fn env_inner(&self) -> Self::EnvInner<'_> {
|
||||
(&self.env, self.durability)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- EnvInner Impl
|
||||
impl<'env> EnvInner<'env, redb::ReadTransaction, redb::WriteTransaction>
|
||||
for (&'env redb::Database, redb::Durability)
|
||||
where
|
||||
Self: 'env,
|
||||
{
|
||||
#[inline]
|
||||
fn tx_ro(&'env self) -> Result<redb::ReadTransaction, RuntimeError> {
|
||||
Ok(self.0.begin_read()?)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn tx_rw(&'env self) -> Result<redb::WriteTransaction, RuntimeError> {
|
||||
// `redb` has sync modes on the TX level, unlike heed,
|
||||
// which sets it at the Environment level.
|
||||
//
|
||||
// So, set the durability here before returning the TX.
|
||||
let mut tx_rw = self.0.begin_write()?;
|
||||
tx_rw.set_durability(self.1);
|
||||
Ok(tx_rw)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn open_db_ro<T: Table>(
|
||||
&self,
|
||||
tx_ro: &redb::ReadTransaction,
|
||||
) -> Result<impl DatabaseRo<T> + DatabaseIter<T>, RuntimeError> {
|
||||
// Open up a read-only database using our `T: Table`'s const metadata.
|
||||
let table: redb::TableDefinition<'static, StorableRedb<T::Key>, StorableRedb<T::Value>> =
|
||||
redb::TableDefinition::new(T::NAME);
|
||||
|
||||
// INVARIANT: Our `?` error conversion will panic if the table does not exist.
|
||||
Ok(tx_ro.open_table(table)?)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn open_db_rw<T: Table>(
|
||||
&self,
|
||||
tx_rw: &redb::WriteTransaction,
|
||||
) -> Result<impl DatabaseRw<T>, RuntimeError> {
|
||||
// Open up a read/write database using our `T: Table`'s const metadata.
|
||||
let table: redb::TableDefinition<'static, StorableRedb<T::Key>, StorableRedb<T::Value>> =
|
||||
redb::TableDefinition::new(T::NAME);
|
||||
|
||||
// `redb` creates tables if they don't exist, so this should never panic.
|
||||
// <https://docs.rs/redb/latest/redb/struct.WriteTransaction.html#method.open_table>
|
||||
Ok(tx_rw.open_table(table)?)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn clear_db<T: Table>(&self, tx_rw: &mut redb::WriteTransaction) -> Result<(), RuntimeError> {
|
||||
let table: redb::TableDefinition<
|
||||
'static,
|
||||
StorableRedb<<T as Table>::Key>,
|
||||
StorableRedb<<T as Table>::Value>,
|
||||
> = redb::TableDefinition::new(<T as Table>::NAME);
|
||||
|
||||
// INVARIANT:
|
||||
// This `delete_table()` will not run into this `TableAlreadyOpen` error:
|
||||
// <https://docs.rs/redb/2.0.0/src/redb/transactions.rs.html#382>
|
||||
// which will panic in the `From` impl, as:
|
||||
//
|
||||
// 1. Only 1 `redb::WriteTransaction` can exist at a time
|
||||
// 2. We have exclusive access to it
|
||||
// 3. So it's not being used to open a table since that needs `&tx_rw`
|
||||
//
|
||||
// Reader-open tables do not affect this, if they're open the below is still OK.
|
||||
redb::WriteTransaction::delete_table(tx_rw, table)?;
|
||||
// Re-create the table.
|
||||
// `redb` creates tables if they don't exist, so this should never panic.
|
||||
redb::WriteTransaction::open_table(tx_rw, table)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
172
storage/database/src/backend/redb/error.rs
Normal file
172
storage/database/src/backend/redb/error.rs
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
//! Conversion from `redb`'s errors -> `cuprate_database`'s errors.
|
||||
//!
|
||||
//! HACK: There's a lot of `_ =>` usage here because
|
||||
//! `redb`'s errors are `#[non_exhaustive]`...
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
constants::DATABASE_CORRUPT_MSG,
|
||||
error::{InitError, RuntimeError},
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- InitError
|
||||
impl From<redb::DatabaseError> for InitError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::Database::open`](https://docs.rs/redb/1.5.0/redb/struct.Database.html#method.open).
|
||||
fn from(error: redb::DatabaseError) -> Self {
|
||||
use redb::DatabaseError as E;
|
||||
use redb::StorageError as E2;
|
||||
|
||||
// Reference of all possible errors `redb` will return
|
||||
// upon using `redb::Database::open`:
|
||||
// <https://docs.rs/redb/1.5.0/src/redb/db.rs.html#908-923>
|
||||
match error {
|
||||
E::RepairAborted => Self::Corrupt,
|
||||
E::UpgradeRequired(_) => Self::InvalidVersion,
|
||||
E::Storage(s_error) => match s_error {
|
||||
E2::Io(e) => Self::Io(e),
|
||||
E2::Corrupted(_) => Self::Corrupt,
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(s_error)),
|
||||
},
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(error)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<redb::StorageError> for InitError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::Database::open`](https://docs.rs/redb/1.5.0/redb/struct.Database.html#method.check_integrity)
|
||||
fn from(error: redb::StorageError) -> Self {
|
||||
use redb::StorageError as E;
|
||||
|
||||
match error {
|
||||
E::Io(e) => Self::Io(e),
|
||||
E::Corrupted(_) => Self::Corrupt,
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(error)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<redb::TransactionError> for InitError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::Database::begin_write`](https://docs.rs/redb/1.5.0/redb/struct.Database.html#method.begin_write)
|
||||
fn from(error: redb::TransactionError) -> Self {
|
||||
match error {
|
||||
redb::TransactionError::Storage(error) => error.into(),
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(error)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<redb::TableError> for InitError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::WriteTransaction::open_table`](https://docs.rs/redb/1.5.0/redb/struct.WriteTransaction.html#method.open_table)
|
||||
fn from(error: redb::TableError) -> Self {
|
||||
use redb::TableError as E;
|
||||
|
||||
match error {
|
||||
E::Storage(error) => error.into(),
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(error)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<redb::CommitError> for InitError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::WriteTransaction::commit`](https://docs.rs/redb/1.5.0/redb/struct.WriteTransaction.html#method.commit)
|
||||
fn from(error: redb::CommitError) -> Self {
|
||||
match error {
|
||||
redb::CommitError::Storage(error) => error.into(),
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => Self::Unknown(Box::new(error)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- RuntimeError
|
||||
#[allow(clippy::fallible_impl_from)] // We need to panic sometimes.
|
||||
impl From<redb::TransactionError> for RuntimeError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::Database::begin_write`](https://docs.rs/redb/1.5.0/redb/struct.Database.html#method.begin_write)
|
||||
/// - [`redb::Database::begin_read`](https://docs.rs/redb/1.5.0/redb/struct.Database.html#method.begin_read)
|
||||
fn from(error: redb::TransactionError) -> Self {
|
||||
match error {
|
||||
redb::TransactionError::Storage(error) => error.into(),
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::fallible_impl_from)] // We need to panic sometimes.
|
||||
impl From<redb::CommitError> for RuntimeError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::WriteTransaction::commit`](https://docs.rs/redb/1.5.0/redb/struct.WriteTransaction.html#method.commit)
|
||||
fn from(error: redb::CommitError) -> Self {
|
||||
match error {
|
||||
redb::CommitError::Storage(error) => error.into(),
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::fallible_impl_from)] // We need to panic sometimes.
|
||||
impl From<redb::TableError> for RuntimeError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::WriteTransaction::open_table`](https://docs.rs/redb/1.5.0/redb/struct.WriteTransaction.html#method.open_table)
|
||||
/// - [`redb::ReadTransaction::open_table`](https://docs.rs/redb/1.5.0/redb/struct.ReadTransaction.html#method.open_table)
|
||||
fn from(error: redb::TableError) -> Self {
|
||||
use redb::TableError as E;
|
||||
|
||||
match error {
|
||||
E::Storage(error) => error.into(),
|
||||
|
||||
// Only if we write incorrect code.
|
||||
E::TableTypeMismatch { .. }
|
||||
| E::TableIsMultimap(_)
|
||||
| E::TableIsNotMultimap(_)
|
||||
| E::TypeDefinitionChanged { .. }
|
||||
| E::TableDoesNotExist(_)
|
||||
| E::TableAlreadyOpen(..) => panic!("fix the database code! {error:#?}"),
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::fallible_impl_from)] // We need to panic sometimes.
|
||||
impl From<redb::StorageError> for RuntimeError {
|
||||
/// Created by `redb` in:
|
||||
/// - [`redb::Table`](https://docs.rs/redb/1.5.0/redb/struct.Table.html) functions
|
||||
/// - [`redb::ReadOnlyTable`](https://docs.rs/redb/1.5.0/redb/struct.ReadOnlyTable.html) functions
|
||||
fn from(error: redb::StorageError) -> Self {
|
||||
use redb::StorageError as E;
|
||||
|
||||
match error {
|
||||
E::Io(e) => Self::Io(e),
|
||||
E::Corrupted(s) => panic!("{s:#?}\n{DATABASE_CORRUPT_MSG}"),
|
||||
E::ValueTooLarge(s) => panic!("fix the database code! {s:#?}"),
|
||||
E::LockPoisoned(s) => panic!("{s:#?}"),
|
||||
|
||||
// HACK: Handle new errors as `redb` adds them.
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
9
storage/database/src/backend/redb/mod.rs
Normal file
9
storage/database/src/backend/redb/mod.rs
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
//! Database backend implementation backed by `sanakirja`.
|
||||
|
||||
mod env;
|
||||
pub use env::ConcreteEnv;
|
||||
mod database;
|
||||
mod error;
|
||||
mod storable;
|
||||
mod transaction;
|
||||
mod types;
|
||||
221
storage/database/src/backend/redb/storable.rs
Normal file
221
storage/database/src/backend/redb/storable.rs
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
//! `cuprate_database::Storable` <-> `redb` serde trait compatibility layer.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use std::{cmp::Ordering, fmt::Debug, marker::PhantomData};
|
||||
|
||||
use redb::TypeName;
|
||||
|
||||
use crate::{key::Key, storable::Storable};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- StorableRedb
|
||||
/// The glue structs that implements `redb`'s (de)serialization
|
||||
/// traits on any type that implements `cuprate_database::Key`.
|
||||
///
|
||||
/// Never actually get constructed, just used for trait bound translations.
|
||||
#[derive(Debug)]
|
||||
pub(super) struct StorableRedb<T>(PhantomData<T>)
|
||||
where
|
||||
T: Storable;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- redb::Key
|
||||
// If `Key` is also implemented, this can act as a `redb::Key`.
|
||||
impl<T> redb::Key for StorableRedb<T>
|
||||
where
|
||||
T: Key + 'static,
|
||||
{
|
||||
#[inline]
|
||||
fn compare(left: &[u8], right: &[u8]) -> Ordering {
|
||||
<T as Key>::compare(left, right)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- redb::Value
|
||||
impl<T> redb::Value for StorableRedb<T>
|
||||
where
|
||||
T: Storable + 'static,
|
||||
{
|
||||
type SelfType<'a> = T where Self: 'a;
|
||||
type AsBytes<'a> = &'a [u8] where Self: 'a;
|
||||
|
||||
#[inline]
|
||||
fn fixed_width() -> Option<usize> {
|
||||
<T as Storable>::BYTE_LENGTH
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn from_bytes<'a>(data: &'a [u8]) -> Self::SelfType<'static>
|
||||
where
|
||||
Self: 'a,
|
||||
{
|
||||
<T as Storable>::from_bytes(data)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn as_bytes<'a, 'b: 'a>(value: &'a Self::SelfType<'b>) -> &'a [u8]
|
||||
where
|
||||
Self: 'a + 'b,
|
||||
{
|
||||
<T as Storable>::as_bytes(value)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn type_name() -> TypeName {
|
||||
TypeName::new(std::any::type_name::<T>())
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
#[allow(clippy::needless_pass_by_value)]
|
||||
mod test {
|
||||
use super::*;
|
||||
use crate::{StorableBytes, StorableVec};
|
||||
|
||||
// Each `#[test]` function has a `test()` to:
|
||||
// - log
|
||||
// - simplify trait bounds
|
||||
// - make sure the right function is being called
|
||||
|
||||
#[test]
|
||||
/// Assert `redb::Key::compare` works for `StorableRedb`.
|
||||
fn compare() {
|
||||
fn test<T>(left: T, right: T, expected: Ordering)
|
||||
where
|
||||
T: Key + 'static,
|
||||
{
|
||||
println!("left: {left:?}, right: {right:?}, expected: {expected:?}");
|
||||
assert_eq!(
|
||||
<StorableRedb::<T> as redb::Key>::compare(
|
||||
<StorableRedb::<T> as redb::Value>::as_bytes(&left),
|
||||
<StorableRedb::<T> as redb::Value>::as_bytes(&right)
|
||||
),
|
||||
expected
|
||||
);
|
||||
}
|
||||
|
||||
test::<i64>(-1, 2, Ordering::Greater); // bytes are greater, not the value
|
||||
test::<u64>(0, 1, Ordering::Less);
|
||||
test::<[u8; 2]>([1, 1], [1, 0], Ordering::Greater);
|
||||
test::<[u8; 3]>([1, 2, 3], [1, 2, 3], Ordering::Equal);
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// Assert `redb::Key::fixed_width` is accurate.
|
||||
fn fixed_width() {
|
||||
fn test<T>(expected: Option<usize>)
|
||||
where
|
||||
T: Storable + 'static,
|
||||
{
|
||||
assert_eq!(<StorableRedb::<T> as redb::Value>::fixed_width(), expected);
|
||||
}
|
||||
|
||||
test::<()>(Some(0));
|
||||
test::<u8>(Some(1));
|
||||
test::<u16>(Some(2));
|
||||
test::<u32>(Some(4));
|
||||
test::<u64>(Some(8));
|
||||
test::<i8>(Some(1));
|
||||
test::<i16>(Some(2));
|
||||
test::<i32>(Some(4));
|
||||
test::<i64>(Some(8));
|
||||
test::<StorableVec<u8>>(None);
|
||||
test::<StorableBytes>(None);
|
||||
test::<[u8; 0]>(Some(0));
|
||||
test::<[u8; 1]>(Some(1));
|
||||
test::<[u8; 2]>(Some(2));
|
||||
test::<[u8; 3]>(Some(3));
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// Assert `redb::Key::as_bytes` is accurate.
|
||||
fn as_bytes() {
|
||||
fn test<T>(t: &T, expected: &[u8])
|
||||
where
|
||||
T: Storable + 'static,
|
||||
{
|
||||
println!("t: {t:?}, expected: {expected:?}");
|
||||
assert_eq!(<StorableRedb::<T> as redb::Value>::as_bytes(t), expected);
|
||||
}
|
||||
|
||||
test::<()>(&(), &[]);
|
||||
test::<u8>(&0, &[0]);
|
||||
test::<u16>(&1, &[1, 0]);
|
||||
test::<u32>(&2, &[2, 0, 0, 0]);
|
||||
test::<u64>(&3, &[3, 0, 0, 0, 0, 0, 0, 0]);
|
||||
test::<i8>(&-1, &[255]);
|
||||
test::<i16>(&-2, &[254, 255]);
|
||||
test::<i32>(&-3, &[253, 255, 255, 255]);
|
||||
test::<i64>(&-4, &[252, 255, 255, 255, 255, 255, 255, 255]);
|
||||
test::<StorableVec<u8>>(&StorableVec([1, 2].to_vec()), &[1, 2]);
|
||||
test::<StorableBytes>(&StorableBytes(bytes::Bytes::from_static(&[1, 2])), &[1, 2]);
|
||||
test::<[u8; 0]>(&[], &[]);
|
||||
test::<[u8; 1]>(&[255], &[255]);
|
||||
test::<[u8; 2]>(&[111, 0], &[111, 0]);
|
||||
test::<[u8; 3]>(&[1, 0, 1], &[1, 0, 1]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// Assert `redb::Key::from_bytes` is accurate.
|
||||
fn from_bytes() {
|
||||
fn test<T>(bytes: &[u8], expected: &T)
|
||||
where
|
||||
T: Storable + PartialEq + 'static,
|
||||
{
|
||||
println!("bytes: {bytes:?}, expected: {expected:?}");
|
||||
assert_eq!(
|
||||
&<StorableRedb::<T> as redb::Value>::from_bytes(bytes),
|
||||
expected
|
||||
);
|
||||
}
|
||||
|
||||
test::<()>([].as_slice(), &());
|
||||
test::<u8>([0].as_slice(), &0);
|
||||
test::<u16>([1, 0].as_slice(), &1);
|
||||
test::<u32>([2, 0, 0, 0].as_slice(), &2);
|
||||
test::<u64>([3, 0, 0, 0, 0, 0, 0, 0].as_slice(), &3);
|
||||
test::<i8>([255].as_slice(), &-1);
|
||||
test::<i16>([254, 255].as_slice(), &-2);
|
||||
test::<i32>([253, 255, 255, 255].as_slice(), &-3);
|
||||
test::<i64>([252, 255, 255, 255, 255, 255, 255, 255].as_slice(), &-4);
|
||||
test::<StorableVec<u8>>(&[1, 2], &StorableVec(vec![1, 2]));
|
||||
test::<StorableBytes>(&[1, 2], &StorableBytes(bytes::Bytes::from_static(&[1, 2])));
|
||||
test::<[u8; 0]>([].as_slice(), &[]);
|
||||
test::<[u8; 1]>([255].as_slice(), &[255]);
|
||||
test::<[u8; 2]>([111, 0].as_slice(), &[111, 0]);
|
||||
test::<[u8; 3]>([1, 0, 1].as_slice(), &[1, 0, 1]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// Assert `redb::Key::type_name` returns unique names.
|
||||
/// The name itself isn't tested, the invariant is that
|
||||
/// they are all unique.
|
||||
fn type_name() {
|
||||
// Can't use a proper set because `redb::TypeName: !Ord`.
|
||||
let set = [
|
||||
<StorableRedb<()> as redb::Value>::type_name(),
|
||||
<StorableRedb<u8> as redb::Value>::type_name(),
|
||||
<StorableRedb<u16> as redb::Value>::type_name(),
|
||||
<StorableRedb<u32> as redb::Value>::type_name(),
|
||||
<StorableRedb<u64> as redb::Value>::type_name(),
|
||||
<StorableRedb<i8> as redb::Value>::type_name(),
|
||||
<StorableRedb<i16> as redb::Value>::type_name(),
|
||||
<StorableRedb<i32> as redb::Value>::type_name(),
|
||||
<StorableRedb<i64> as redb::Value>::type_name(),
|
||||
<StorableRedb<StorableVec<u8>> as redb::Value>::type_name(),
|
||||
<StorableRedb<StorableBytes> as redb::Value>::type_name(),
|
||||
<StorableRedb<[u8; 0]> as redb::Value>::type_name(),
|
||||
<StorableRedb<[u8; 1]> as redb::Value>::type_name(),
|
||||
<StorableRedb<[u8; 2]> as redb::Value>::type_name(),
|
||||
<StorableRedb<[u8; 3]> as redb::Value>::type_name(),
|
||||
];
|
||||
|
||||
// Check every permutation is unique.
|
||||
for (index, i) in set.iter().enumerate() {
|
||||
for (index2, j) in set.iter().enumerate() {
|
||||
if index != index2 {
|
||||
assert_ne!(i, j);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
38
storage/database/src/backend/redb/transaction.rs
Normal file
38
storage/database/src/backend/redb/transaction.rs
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
//! Implementation of `trait TxRo/TxRw` for `redb`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
error::RuntimeError,
|
||||
transaction::{TxRo, TxRw},
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- TxRo
|
||||
impl TxRo<'_> for redb::ReadTransaction {
|
||||
/// This function is infallible.
|
||||
fn commit(self) -> Result<(), RuntimeError> {
|
||||
// `redb`'s read transactions cleanup automatically when all references are dropped.
|
||||
//
|
||||
// There is `close()`:
|
||||
// <https://docs.rs/redb/2.0.0/redb/struct.ReadTransaction.html#method.close>
|
||||
// but this will error if there are outstanding references, i.e. an open table.
|
||||
// This is unwanted behavior in our case, so we don't call this.
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- TxRw
|
||||
impl TxRw<'_> for redb::WriteTransaction {
|
||||
fn commit(self) -> Result<(), RuntimeError> {
|
||||
Ok(self.commit()?)
|
||||
}
|
||||
|
||||
fn abort(self) -> Result<(), RuntimeError> {
|
||||
Ok(self.abort()?)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
11
storage/database/src/backend/redb/types.rs
Normal file
11
storage/database/src/backend/redb/types.rs
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
//! `redb` type aliases.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Types
|
||||
use crate::backend::redb::storable::StorableRedb;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Types
|
||||
/// The concrete type for readable `redb` tables.
|
||||
pub(super) type RedbTableRo<K, V> = redb::ReadOnlyTable<StorableRedb<K>, StorableRedb<V>>;
|
||||
|
||||
/// The concrete type for readable/writable `redb` tables.
|
||||
pub(super) type RedbTableRw<'tx, K, V> = redb::Table<'tx, StorableRedb<K>, StorableRedb<V>>;
|
||||
550
storage/database/src/backend/tests.rs
Normal file
550
storage/database/src/backend/tests.rs
Normal file
|
|
@ -0,0 +1,550 @@
|
|||
//! Tests for `cuprate_database`'s backends.
|
||||
//!
|
||||
//! These tests are fully trait-based, meaning there
|
||||
//! is no reference to `backend/`-specific types.
|
||||
//!
|
||||
//! As such, which backend is tested is
|
||||
//! dependant on the feature flags used.
|
||||
//!
|
||||
//! | Feature flag | Tested backend |
|
||||
//! |---------------|----------------|
|
||||
//! | Only `redb` | `redb`
|
||||
//! | Anything else | `heed`
|
||||
//!
|
||||
//! `redb`, and it only must be enabled for it to be tested.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
|
||||
use crate::{
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
env::{Env, EnvInner},
|
||||
error::RuntimeError,
|
||||
resize::ResizeAlgorithm,
|
||||
storable::StorableVec,
|
||||
tables::{
|
||||
BlockBlobs, BlockHeights, BlockInfos, KeyImages, NumOutputs, Outputs, PrunableHashes,
|
||||
PrunableTxBlobs, PrunedTxBlobs, RctOutputs, TxBlobs, TxHeights, TxIds, TxOutputs,
|
||||
TxUnlockTime,
|
||||
},
|
||||
tables::{TablesIter, TablesMut},
|
||||
tests::tmp_concrete_env,
|
||||
transaction::{TxRo, TxRw},
|
||||
types::{
|
||||
Amount, AmountIndex, AmountIndices, BlockBlob, BlockHash, BlockHeight, BlockInfo, KeyImage,
|
||||
Output, OutputFlags, PreRctOutputId, PrunableBlob, PrunableHash, PrunedBlob, RctOutput,
|
||||
TxBlob, TxHash, TxId, UnlockTime,
|
||||
},
|
||||
ConcreteEnv,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
/// Simply call [`Env::open`]. If this fails, something is really wrong.
|
||||
#[test]
|
||||
fn open() {
|
||||
tmp_concrete_env();
|
||||
}
|
||||
|
||||
/// Create database transactions, but don't write any data.
|
||||
#[test]
|
||||
fn tx() {
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
|
||||
TxRo::commit(env_inner.tx_ro().unwrap()).unwrap();
|
||||
TxRw::commit(env_inner.tx_rw().unwrap()).unwrap();
|
||||
TxRw::abort(env_inner.tx_rw().unwrap()).unwrap();
|
||||
}
|
||||
|
||||
/// Open (and verify) that all database tables
|
||||
/// exist already after calling [`Env::open`].
|
||||
#[test]
|
||||
fn open_db() {
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
|
||||
// Open all tables in read-only mode.
|
||||
// This should be updated when tables are modified.
|
||||
env_inner.open_db_ro::<BlockBlobs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<BlockHeights>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<BlockInfos>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<KeyImages>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<NumOutputs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<Outputs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<PrunableHashes>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<PrunableTxBlobs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<PrunedTxBlobs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<RctOutputs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<TxBlobs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<TxHeights>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<TxIds>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<TxOutputs>(&tx_ro).unwrap();
|
||||
env_inner.open_db_ro::<TxUnlockTime>(&tx_ro).unwrap();
|
||||
TxRo::commit(tx_ro).unwrap();
|
||||
|
||||
// Open all tables in read/write mode.
|
||||
env_inner.open_db_rw::<BlockBlobs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<BlockHeights>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<BlockInfos>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<KeyImages>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<NumOutputs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<Outputs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<PrunableHashes>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<PrunableTxBlobs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<PrunedTxBlobs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<RctOutputs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<TxBlobs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<TxHeights>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<TxIds>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<TxOutputs>(&tx_rw).unwrap();
|
||||
env_inner.open_db_rw::<TxUnlockTime>(&tx_rw).unwrap();
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
/// Test `Env` resizes.
|
||||
#[test]
|
||||
fn resize() {
|
||||
// This test is only valid for `Env`'s that need to resize manually.
|
||||
if !ConcreteEnv::MANUAL_RESIZE {
|
||||
return;
|
||||
}
|
||||
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
|
||||
// Resize by the OS page size.
|
||||
let page_size = crate::resize::page_size();
|
||||
let old_size = env.current_map_size();
|
||||
env.resize_map(Some(ResizeAlgorithm::FixedBytes(page_size)));
|
||||
|
||||
// Assert it resized exactly by the OS page size.
|
||||
let new_size = env.current_map_size();
|
||||
assert_eq!(new_size, old_size + page_size.get());
|
||||
}
|
||||
|
||||
/// Test that `Env`'s that don't manually resize.
|
||||
#[test]
|
||||
#[should_panic = "unreachable"]
|
||||
fn non_manual_resize_1() {
|
||||
if ConcreteEnv::MANUAL_RESIZE {
|
||||
unreachable!();
|
||||
} else {
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
env.resize_map(None);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[should_panic = "unreachable"]
|
||||
fn non_manual_resize_2() {
|
||||
if ConcreteEnv::MANUAL_RESIZE {
|
||||
unreachable!();
|
||||
} else {
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
env.current_map_size();
|
||||
}
|
||||
}
|
||||
|
||||
/// Test all `DatabaseR{o,w}` operations.
|
||||
#[test]
|
||||
fn db_read_write() {
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut table = env_inner.open_db_rw::<Outputs>(&tx_rw).unwrap();
|
||||
|
||||
/// The (1st) key.
|
||||
const KEY: PreRctOutputId = PreRctOutputId {
|
||||
amount: 1,
|
||||
amount_index: 123,
|
||||
};
|
||||
/// The expected value.
|
||||
const VALUE: Output = Output {
|
||||
key: [35; 32],
|
||||
height: 45_761_798,
|
||||
output_flags: OutputFlags::empty(),
|
||||
tx_idx: 2_353_487,
|
||||
};
|
||||
/// How many `(key, value)` pairs will be inserted.
|
||||
const N: u64 = 100;
|
||||
|
||||
/// Assert 2 `Output`'s are equal, and that accessing
|
||||
/// their fields don't result in an unaligned panic.
|
||||
fn assert_same(output: Output) {
|
||||
assert_eq!(output, VALUE);
|
||||
assert_eq!(output.key, VALUE.key);
|
||||
assert_eq!(output.height, VALUE.height);
|
||||
assert_eq!(output.output_flags, VALUE.output_flags);
|
||||
assert_eq!(output.tx_idx, VALUE.tx_idx);
|
||||
}
|
||||
|
||||
assert!(table.is_empty().unwrap());
|
||||
|
||||
// Insert keys.
|
||||
let mut key = KEY;
|
||||
for _ in 0..N {
|
||||
table.put(&key, &VALUE).unwrap();
|
||||
key.amount += 1;
|
||||
}
|
||||
|
||||
assert_eq!(table.len().unwrap(), N);
|
||||
|
||||
// Assert the first/last `(key, value)`s are there.
|
||||
{
|
||||
assert!(table.contains(&KEY).unwrap());
|
||||
let get: Output = table.get(&KEY).unwrap();
|
||||
assert_same(get);
|
||||
|
||||
let first: Output = table.first().unwrap().1;
|
||||
assert_same(first);
|
||||
|
||||
let last: Output = table.last().unwrap().1;
|
||||
assert_same(last);
|
||||
}
|
||||
|
||||
// Commit transactions, create new ones.
|
||||
drop(table);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let table_ro = env_inner.open_db_ro::<Outputs>(&tx_ro).unwrap();
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut table = env_inner.open_db_rw::<Outputs>(&tx_rw).unwrap();
|
||||
|
||||
// Assert the whole range is there.
|
||||
{
|
||||
let range = table_ro.get_range(..).unwrap();
|
||||
let mut i = 0;
|
||||
for result in range {
|
||||
let value: Output = result.unwrap();
|
||||
assert_same(value);
|
||||
|
||||
i += 1;
|
||||
}
|
||||
assert_eq!(i, N);
|
||||
}
|
||||
|
||||
// `get_range()` tests.
|
||||
let mut key = KEY;
|
||||
key.amount += N;
|
||||
let range = KEY..key;
|
||||
|
||||
// Assert count is correct.
|
||||
assert_eq!(
|
||||
N as usize,
|
||||
table_ro.get_range(range.clone()).unwrap().count()
|
||||
);
|
||||
|
||||
// Assert each returned value from the iterator is owned.
|
||||
{
|
||||
let mut iter = table_ro.get_range(range.clone()).unwrap();
|
||||
let value: Output = iter.next().unwrap().unwrap(); // 1. take value out
|
||||
drop(iter); // 2. drop the `impl Iterator + 'a`
|
||||
assert_same(value); // 3. assert even without the iterator, the value is alive
|
||||
}
|
||||
|
||||
// Assert each value is the same.
|
||||
{
|
||||
let mut iter = table_ro.get_range(range).unwrap();
|
||||
for _ in 0..N {
|
||||
let value: Output = iter.next().unwrap().unwrap();
|
||||
assert_same(value);
|
||||
}
|
||||
}
|
||||
|
||||
// Assert `update()` works.
|
||||
{
|
||||
const HEIGHT: u32 = 999;
|
||||
|
||||
assert_ne!(table.get(&KEY).unwrap().height, HEIGHT);
|
||||
|
||||
table
|
||||
.update(&KEY, |mut value| {
|
||||
value.height = HEIGHT;
|
||||
Some(value)
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(table.get(&KEY).unwrap().height, HEIGHT);
|
||||
}
|
||||
|
||||
// Assert deleting works.
|
||||
{
|
||||
table.delete(&KEY).unwrap();
|
||||
let value = table.get(&KEY);
|
||||
assert!(!table.contains(&KEY).unwrap());
|
||||
assert!(matches!(value, Err(RuntimeError::KeyNotFound)));
|
||||
// Assert the other `(key, value)` pairs are still there.
|
||||
let mut key = KEY;
|
||||
key.amount += N - 1; // we used inclusive `0..N`
|
||||
let value = table.get(&key).unwrap();
|
||||
assert_same(value);
|
||||
}
|
||||
|
||||
// Assert `take()` works.
|
||||
{
|
||||
let mut key = KEY;
|
||||
key.amount += 1;
|
||||
let value = table.take(&key).unwrap();
|
||||
assert_eq!(value, VALUE);
|
||||
|
||||
let get = table.get(&KEY);
|
||||
assert!(!table.contains(&key).unwrap());
|
||||
assert!(matches!(get, Err(RuntimeError::KeyNotFound)));
|
||||
|
||||
// Assert the other `(key, value)` pairs are still there.
|
||||
key.amount += 1;
|
||||
let value = table.get(&key).unwrap();
|
||||
assert_same(value);
|
||||
}
|
||||
|
||||
drop(table);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
|
||||
// Assert `clear_db()` works.
|
||||
{
|
||||
let mut tx_rw = env_inner.tx_rw().unwrap();
|
||||
env_inner.clear_db::<Outputs>(&mut tx_rw).unwrap();
|
||||
let table = env_inner.open_db_rw::<Outputs>(&tx_rw).unwrap();
|
||||
assert!(table.is_empty().unwrap());
|
||||
for n in 0..N {
|
||||
let mut key = KEY;
|
||||
key.amount += n;
|
||||
let value = table.get(&key);
|
||||
assert!(matches!(value, Err(RuntimeError::KeyNotFound)));
|
||||
assert!(!table.contains(&key).unwrap());
|
||||
}
|
||||
|
||||
// Reader still sees old value.
|
||||
assert!(!table_ro.is_empty().unwrap());
|
||||
|
||||
// Writer sees updated value (nothing).
|
||||
assert!(table.is_empty().unwrap());
|
||||
}
|
||||
}
|
||||
|
||||
/// Assert that `key`'s in database tables are sorted in
|
||||
/// an ordered B-Tree fashion, i.e. `min_value -> max_value`.
|
||||
#[test]
|
||||
fn tables_are_sorted() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables_mut = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
// Insert `{5, 4, 3, 2, 1, 0}`, assert each new
|
||||
// number inserted is the minimum `first()` value.
|
||||
for key in (0..6).rev() {
|
||||
tables_mut.num_outputs_mut().put(&key, &123).unwrap();
|
||||
let (first, _) = tables_mut.num_outputs_mut().first().unwrap();
|
||||
assert_eq!(first, key);
|
||||
}
|
||||
|
||||
drop(tables_mut);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
|
||||
// Assert iterators are ordered.
|
||||
{
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
let t = tables.num_outputs_iter();
|
||||
let iter = t.iter().unwrap();
|
||||
let keys = t.keys().unwrap();
|
||||
for ((i, iter), key) in (0..6).zip(iter).zip(keys) {
|
||||
let (iter, _) = iter.unwrap();
|
||||
let key = key.unwrap();
|
||||
assert_eq!(i, iter);
|
||||
assert_eq!(iter, key);
|
||||
}
|
||||
}
|
||||
|
||||
let mut tables_mut = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
let t = tables_mut.num_outputs_mut();
|
||||
|
||||
// Assert the `first()` values are the minimum, i.e. `{0, 1, 2}`
|
||||
for key in 0..3 {
|
||||
let (first, _) = t.first().unwrap();
|
||||
assert_eq!(first, key);
|
||||
t.delete(&key).unwrap();
|
||||
}
|
||||
|
||||
// Assert the `last()` values are the maximum, i.e. `{5, 4, 3}`
|
||||
for key in (3..6).rev() {
|
||||
let (last, _) = tables_mut.num_outputs_mut().last().unwrap();
|
||||
assert_eq!(last, key);
|
||||
tables_mut.num_outputs_mut().delete(&key).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Table Tests
|
||||
/// Test multiple tables and their key + values.
|
||||
///
|
||||
/// Each one of these tests:
|
||||
/// - Opens a specific table
|
||||
/// - Essentially does the `db_read_write` test
|
||||
macro_rules! test_tables {
|
||||
($(
|
||||
$table:ident, // Table type
|
||||
$key_type:ty => // Key (type)
|
||||
$value_type:ty, // Value (type)
|
||||
$key:expr => // Key (the value)
|
||||
$value:expr, // Value (the value)
|
||||
)* $(,)?) => { paste::paste! { $(
|
||||
// Test function's name is the table type in `snake_case`.
|
||||
#[test]
|
||||
fn [<$table:snake>]() {
|
||||
// Open the database env and table.
|
||||
let (env, _tempdir) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
let mut tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut table = env_inner.open_db_rw::<$table>(&mut tx_rw).unwrap();
|
||||
|
||||
/// The expected key.
|
||||
const KEY: $key_type = $key;
|
||||
// The expected value.
|
||||
let value: $value_type = $value;
|
||||
|
||||
// Assert a passed value is equal to the const value.
|
||||
let assert_eq = |v: &$value_type| {
|
||||
assert_eq!(v, &value);
|
||||
};
|
||||
|
||||
// Insert the key.
|
||||
table.put(&KEY, &value).unwrap();
|
||||
// Assert key is there.
|
||||
{
|
||||
let value: $value_type = table.get(&KEY).unwrap();
|
||||
assert_eq(&value);
|
||||
}
|
||||
|
||||
assert!(table.contains(&KEY).unwrap());
|
||||
assert_eq!(table.len().unwrap(), 1);
|
||||
|
||||
// Commit transactions, create new ones.
|
||||
drop(table);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
let mut tx_rw = env_inner.tx_rw().unwrap();
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let mut table = env_inner.open_db_rw::<$table>(&tx_rw).unwrap();
|
||||
let table_ro = env_inner.open_db_ro::<$table>(&tx_ro).unwrap();
|
||||
|
||||
// Assert `get_range()` works.
|
||||
{
|
||||
let range = KEY..;
|
||||
assert_eq!(1, table_ro.get_range(range.clone()).unwrap().count());
|
||||
let mut iter = table_ro.get_range(range).unwrap();
|
||||
let value = iter.next().unwrap().unwrap();
|
||||
assert_eq(&value);
|
||||
}
|
||||
|
||||
// Assert deleting works.
|
||||
{
|
||||
table.delete(&KEY).unwrap();
|
||||
let value = table.get(&KEY);
|
||||
assert!(matches!(value, Err(RuntimeError::KeyNotFound)));
|
||||
assert!(!table.contains(&KEY).unwrap());
|
||||
assert_eq!(table.len().unwrap(), 0);
|
||||
}
|
||||
|
||||
table.put(&KEY, &value).unwrap();
|
||||
|
||||
// Assert `clear_db()` works.
|
||||
{
|
||||
drop(table);
|
||||
env_inner.clear_db::<$table>(&mut tx_rw).unwrap();
|
||||
let table = env_inner.open_db_rw::<$table>(&mut tx_rw).unwrap();
|
||||
let value = table.get(&KEY);
|
||||
assert!(matches!(value, Err(RuntimeError::KeyNotFound)));
|
||||
assert!(!table.contains(&KEY).unwrap());
|
||||
assert_eq!(table.len().unwrap(), 0);
|
||||
}
|
||||
}
|
||||
)*}};
|
||||
}
|
||||
|
||||
// Notes:
|
||||
// - Keep this sorted A-Z (by table name)
|
||||
test_tables! {
|
||||
BlockBlobs, // Table type
|
||||
BlockHeight => BlockBlob, // Key type => Value type
|
||||
123 => StorableVec(vec![1,2,3,4,5,6,7,8]), // Actual key => Actual value
|
||||
|
||||
BlockHeights,
|
||||
BlockHash => BlockHeight,
|
||||
[32; 32] => 123,
|
||||
|
||||
BlockInfos,
|
||||
BlockHeight => BlockInfo,
|
||||
123 => BlockInfo {
|
||||
timestamp: 1,
|
||||
cumulative_generated_coins: 123,
|
||||
weight: 321,
|
||||
cumulative_difficulty_low: 111,
|
||||
cumulative_difficulty_high: 111,
|
||||
block_hash: [54; 32],
|
||||
cumulative_rct_outs: 2389,
|
||||
long_term_weight: 2389,
|
||||
},
|
||||
|
||||
KeyImages,
|
||||
KeyImage => (),
|
||||
[32; 32] => (),
|
||||
|
||||
NumOutputs,
|
||||
Amount => AmountIndex,
|
||||
123 => 123,
|
||||
|
||||
TxBlobs,
|
||||
TxId => TxBlob,
|
||||
123 => StorableVec(vec![1,2,3,4,5,6,7,8]),
|
||||
|
||||
TxIds,
|
||||
TxHash => TxId,
|
||||
[32; 32] => 123,
|
||||
|
||||
TxHeights,
|
||||
TxId => BlockHeight,
|
||||
123 => 123,
|
||||
|
||||
TxOutputs,
|
||||
TxId => AmountIndices,
|
||||
123 => StorableVec(vec![1,2,3,4,5,6,7,8]),
|
||||
|
||||
TxUnlockTime,
|
||||
TxId => UnlockTime,
|
||||
123 => 123,
|
||||
|
||||
Outputs,
|
||||
PreRctOutputId => Output,
|
||||
PreRctOutputId {
|
||||
amount: 1,
|
||||
amount_index: 2,
|
||||
} => Output {
|
||||
key: [1; 32],
|
||||
height: 1,
|
||||
output_flags: OutputFlags::empty(),
|
||||
tx_idx: 3,
|
||||
},
|
||||
|
||||
PrunedTxBlobs,
|
||||
TxId => PrunedBlob,
|
||||
123 => StorableVec(vec![1,2,3,4,5,6,7,8]),
|
||||
|
||||
PrunableTxBlobs,
|
||||
TxId => PrunableBlob,
|
||||
123 => StorableVec(vec![1,2,3,4,5,6,7,8]),
|
||||
|
||||
PrunableHashes,
|
||||
TxId => PrunableHash,
|
||||
123 => [32; 32],
|
||||
|
||||
RctOutputs,
|
||||
AmountIndex => RctOutput,
|
||||
123 => RctOutput {
|
||||
key: [1; 32],
|
||||
height: 1,
|
||||
output_flags: OutputFlags::empty(),
|
||||
tx_idx: 3,
|
||||
commitment: [3; 32],
|
||||
},
|
||||
}
|
||||
31
storage/database/src/config/backend.rs
Normal file
31
storage/database/src/config/backend.rs
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
//! SOMEDAY
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{
|
||||
borrow::Cow,
|
||||
num::NonZeroUsize,
|
||||
path::{Path, PathBuf},
|
||||
};
|
||||
|
||||
#[cfg(feature = "serde")]
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use cuprate_helper::fs::cuprate_database_dir;
|
||||
|
||||
use crate::{
|
||||
config::{ReaderThreads, SyncMode},
|
||||
constants::DATABASE_DATA_FILENAME,
|
||||
resize::ResizeAlgorithm,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Backend
|
||||
/// SOMEDAY: allow runtime hot-swappable backends.
|
||||
#[derive(Copy, Clone, Debug, Default, PartialEq, PartialOrd, Eq, Ord, Hash)]
|
||||
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
|
||||
pub enum Backend {
|
||||
#[default]
|
||||
/// SOMEDAY
|
||||
Heed,
|
||||
/// SOMEDAY
|
||||
Redb,
|
||||
}
|
||||
237
storage/database/src/config/config.rs
Normal file
237
storage/database/src/config/config.rs
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
//! The main [`Config`] struct, holding all configurable values.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{
|
||||
borrow::Cow,
|
||||
path::{Path, PathBuf},
|
||||
};
|
||||
|
||||
#[cfg(feature = "serde")]
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use cuprate_helper::fs::cuprate_database_dir;
|
||||
|
||||
use crate::{
|
||||
config::{ReaderThreads, SyncMode},
|
||||
constants::DATABASE_DATA_FILENAME,
|
||||
resize::ResizeAlgorithm,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- ConfigBuilder
|
||||
/// Builder for [`Config`].
|
||||
///
|
||||
// SOMEDAY: there's are many more options to add in the future.
|
||||
#[derive(Debug, Clone, PartialEq, PartialOrd)]
|
||||
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
|
||||
pub struct ConfigBuilder {
|
||||
/// [`Config::db_directory`].
|
||||
db_directory: Option<Cow<'static, Path>>,
|
||||
|
||||
/// [`Config::sync_mode`].
|
||||
sync_mode: Option<SyncMode>,
|
||||
|
||||
/// [`Config::reader_threads`].
|
||||
reader_threads: Option<ReaderThreads>,
|
||||
|
||||
/// [`Config::resize_algorithm`].
|
||||
resize_algorithm: Option<ResizeAlgorithm>,
|
||||
}
|
||||
|
||||
impl ConfigBuilder {
|
||||
/// Create a new [`ConfigBuilder`].
|
||||
///
|
||||
/// [`ConfigBuilder::build`] can be called immediately
|
||||
/// after this function to use default values.
|
||||
pub const fn new() -> Self {
|
||||
Self {
|
||||
db_directory: None,
|
||||
sync_mode: None,
|
||||
reader_threads: None,
|
||||
resize_algorithm: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build into a [`Config`].
|
||||
///
|
||||
/// # Default values
|
||||
/// If [`ConfigBuilder::db_directory`] was not called,
|
||||
/// the default [`cuprate_database_dir`] will be used.
|
||||
///
|
||||
/// For all other values, [`Default::default`] is used.
|
||||
pub fn build(self) -> Config {
|
||||
// INVARIANT: all PATH safety checks are done
|
||||
// in `helper::fs`. No need to do them here.
|
||||
let db_directory = self
|
||||
.db_directory
|
||||
.unwrap_or_else(|| Cow::Borrowed(cuprate_database_dir()));
|
||||
|
||||
// Add the database filename to the directory.
|
||||
let db_file = {
|
||||
let mut db_file = db_directory.to_path_buf();
|
||||
db_file.push(DATABASE_DATA_FILENAME);
|
||||
Cow::Owned(db_file)
|
||||
};
|
||||
|
||||
Config {
|
||||
db_directory,
|
||||
db_file,
|
||||
sync_mode: self.sync_mode.unwrap_or_default(),
|
||||
reader_threads: self.reader_threads.unwrap_or_default(),
|
||||
resize_algorithm: self.resize_algorithm.unwrap_or_default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Set a custom database directory (and file) [`Path`].
|
||||
#[must_use]
|
||||
pub fn db_directory(mut self, db_directory: PathBuf) -> Self {
|
||||
self.db_directory = Some(Cow::Owned(db_directory));
|
||||
self
|
||||
}
|
||||
|
||||
/// Tune the [`ConfigBuilder`] for the highest performing,
|
||||
/// but also most resource-intensive & maybe risky settings.
|
||||
///
|
||||
/// Good default for testing, and resource-available machines.
|
||||
#[must_use]
|
||||
pub fn fast(mut self) -> Self {
|
||||
self.sync_mode = Some(SyncMode::Fast);
|
||||
self.reader_threads = Some(ReaderThreads::OnePerThread);
|
||||
self.resize_algorithm = Some(ResizeAlgorithm::default());
|
||||
self
|
||||
}
|
||||
|
||||
/// Tune the [`ConfigBuilder`] for the lowest performing,
|
||||
/// but also least resource-intensive settings.
|
||||
///
|
||||
/// Good default for resource-limited machines, e.g. a cheap VPS.
|
||||
#[must_use]
|
||||
pub fn low_power(mut self) -> Self {
|
||||
self.sync_mode = Some(SyncMode::default());
|
||||
self.reader_threads = Some(ReaderThreads::One);
|
||||
self.resize_algorithm = Some(ResizeAlgorithm::default());
|
||||
self
|
||||
}
|
||||
|
||||
/// Set a custom [`SyncMode`].
|
||||
#[must_use]
|
||||
pub const fn sync_mode(mut self, sync_mode: SyncMode) -> Self {
|
||||
self.sync_mode = Some(sync_mode);
|
||||
self
|
||||
}
|
||||
|
||||
/// Set a custom [`ReaderThreads`].
|
||||
#[must_use]
|
||||
pub const fn reader_threads(mut self, reader_threads: ReaderThreads) -> Self {
|
||||
self.reader_threads = Some(reader_threads);
|
||||
self
|
||||
}
|
||||
|
||||
/// Set a custom [`ResizeAlgorithm`].
|
||||
#[must_use]
|
||||
pub const fn resize_algorithm(mut self, resize_algorithm: ResizeAlgorithm) -> Self {
|
||||
self.resize_algorithm = Some(resize_algorithm);
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for ConfigBuilder {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
db_directory: Some(Cow::Borrowed(cuprate_database_dir())),
|
||||
sync_mode: Some(SyncMode::default()),
|
||||
reader_threads: Some(ReaderThreads::default()),
|
||||
resize_algorithm: Some(ResizeAlgorithm::default()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Config
|
||||
/// Database [`Env`](crate::Env) configuration.
|
||||
///
|
||||
/// This is the struct passed to [`Env::open`](crate::Env::open) that
|
||||
/// allows the database to be configured in various ways.
|
||||
///
|
||||
/// For construction, either use [`ConfigBuilder`] or [`Config::default`].
|
||||
///
|
||||
// SOMEDAY: there's are many more options to add in the future.
|
||||
#[derive(Debug, Clone, PartialEq, PartialOrd)]
|
||||
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
|
||||
pub struct Config {
|
||||
//------------------------ Database PATHs
|
||||
// These are private since we don't want
|
||||
// users messing with them after construction.
|
||||
/// The directory used to store all database files.
|
||||
///
|
||||
/// By default, if no value is provided in the [`Config`]
|
||||
/// constructor functions, this will be [`cuprate_database_dir`].
|
||||
///
|
||||
// SOMEDAY: we should also support `/etc/cuprated.conf`.
|
||||
// This could be represented with an `enum DbPath { Default, Custom, Etc, }`
|
||||
pub(crate) db_directory: Cow<'static, Path>,
|
||||
/// The actual database data file.
|
||||
///
|
||||
/// This is private, and created from the above `db_directory`.
|
||||
pub(crate) db_file: Cow<'static, Path>,
|
||||
|
||||
/// Disk synchronization mode.
|
||||
pub sync_mode: SyncMode,
|
||||
|
||||
/// Database reader thread count.
|
||||
pub reader_threads: ReaderThreads,
|
||||
|
||||
/// Database memory map resizing algorithm.
|
||||
///
|
||||
/// This is used as the default fallback, but
|
||||
/// custom algorithms can be used as well with
|
||||
/// [`Env::resize_map`](crate::Env::resize_map).
|
||||
pub resize_algorithm: ResizeAlgorithm,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
/// Create a new [`Config`] with sane default settings.
|
||||
///
|
||||
/// The [`Config::db_directory`] will be [`cuprate_database_dir`].
|
||||
///
|
||||
/// All other values will be [`Default::default`].
|
||||
///
|
||||
/// Same as [`Config::default`].
|
||||
///
|
||||
/// ```rust
|
||||
/// use cuprate_database::{config::*, resize::*, DATABASE_DATA_FILENAME};
|
||||
/// use cuprate_helper::fs::*;
|
||||
///
|
||||
/// let config = Config::new();
|
||||
///
|
||||
/// assert_eq!(config.db_directory(), cuprate_database_dir());
|
||||
/// assert!(config.db_file().starts_with(cuprate_database_dir()));
|
||||
/// assert!(config.db_file().ends_with(DATABASE_DATA_FILENAME));
|
||||
/// assert_eq!(config.sync_mode, SyncMode::default());
|
||||
/// assert_eq!(config.reader_threads, ReaderThreads::default());
|
||||
/// assert_eq!(config.resize_algorithm, ResizeAlgorithm::default());
|
||||
/// ```
|
||||
pub fn new() -> Self {
|
||||
ConfigBuilder::default().build()
|
||||
}
|
||||
|
||||
/// Return the absolute [`Path`] to the database directory.
|
||||
pub const fn db_directory(&self) -> &Cow<'_, Path> {
|
||||
&self.db_directory
|
||||
}
|
||||
|
||||
/// Return the absolute [`Path`] to the database data file.
|
||||
pub const fn db_file(&self) -> &Cow<'_, Path> {
|
||||
&self.db_file
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for Config {
|
||||
/// Same as [`Config::new`].
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::config::*;
|
||||
/// assert_eq!(Config::default(), Config::new());
|
||||
/// ```
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
47
storage/database/src/config/mod.rs
Normal file
47
storage/database/src/config/mod.rs
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
//! Database [`Env`](crate::Env) configuration.
|
||||
//!
|
||||
//! This module contains the main [`Config`]uration struct
|
||||
//! for the database [`Env`](crate::Env)ironment, and types
|
||||
//! related to configuration settings.
|
||||
//!
|
||||
//! The main constructor is the [`ConfigBuilder`].
|
||||
//!
|
||||
//! These configurations are processed at runtime, meaning
|
||||
//! the `Env` can/will dynamically adjust its behavior
|
||||
//! based on these values.
|
||||
//!
|
||||
//! # Example
|
||||
//! ```rust
|
||||
//! use cuprate_database::{
|
||||
//! Env,
|
||||
//! config::{ConfigBuilder, ReaderThreads, SyncMode}
|
||||
//! };
|
||||
//!
|
||||
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
//! let db_dir = tempfile::tempdir()?;
|
||||
//!
|
||||
//! let config = ConfigBuilder::new()
|
||||
//! // Use a custom database directory.
|
||||
//! .db_directory(db_dir.path().to_path_buf())
|
||||
//! // Use as many reader threads as possible (when using `service`).
|
||||
//! .reader_threads(ReaderThreads::OnePerThread)
|
||||
//! // Use the fastest sync mode.
|
||||
//! .sync_mode(SyncMode::Fast)
|
||||
//! // Build into `Config`
|
||||
//! .build();
|
||||
//!
|
||||
//! // Start a database `service` using this configuration.
|
||||
//! let (reader_handle, _) = cuprate_database::service::init(config.clone())?;
|
||||
//! // It's using the config we provided.
|
||||
//! assert_eq!(reader_handle.env().config(), &config);
|
||||
//! # Ok(()) }
|
||||
//! ```
|
||||
|
||||
mod config;
|
||||
pub use config::{Config, ConfigBuilder};
|
||||
|
||||
mod reader_threads;
|
||||
pub use reader_threads::ReaderThreads;
|
||||
|
||||
mod sync_mode;
|
||||
pub use sync_mode::SyncMode;
|
||||
189
storage/database/src/config/reader_threads.rs
Normal file
189
storage/database/src/config/reader_threads.rs
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
//! Database [`Env`](crate::Env) configuration.
|
||||
//!
|
||||
//! This module contains the main [`Config`]uration struct
|
||||
//! for the database [`Env`](crate::Env)ironment, and data
|
||||
//! structures related to any configuration setting.
|
||||
//!
|
||||
//! These configurations are processed at runtime, meaning
|
||||
//! the `Env` can/will dynamically adjust its behavior
|
||||
//! based on these values.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::num::NonZeroUsize;
|
||||
|
||||
#[cfg(feature = "serde")]
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- ReaderThreads
|
||||
/// Amount of database reader threads to spawn when using [`service`](crate::service).
|
||||
///
|
||||
/// This controls how many reader thread `service`'s
|
||||
/// thread-pool will spawn to receive and send requests/responses.
|
||||
///
|
||||
/// It does nothing outside of `service`.
|
||||
///
|
||||
/// It will always be at least 1, up until the amount of threads on the machine.
|
||||
///
|
||||
/// The main function used to extract an actual
|
||||
/// usable thread count out of this is [`ReaderThreads::as_threads`].
|
||||
#[derive(Copy, Clone, Debug, Default, PartialEq, PartialOrd)]
|
||||
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
|
||||
pub enum ReaderThreads {
|
||||
#[default]
|
||||
/// Spawn 1 reader thread per available thread on the machine.
|
||||
///
|
||||
/// For example, a `32-thread` system will spawn
|
||||
/// `32` reader threads using this setting.
|
||||
OnePerThread,
|
||||
|
||||
/// Only spawn 1 reader thread.
|
||||
One,
|
||||
|
||||
/// Spawn a specified amount of reader threads.
|
||||
///
|
||||
/// Note that no matter how large this value, it will be
|
||||
/// ultimately capped at the amount of system threads.
|
||||
///
|
||||
/// # `0`
|
||||
/// `ReaderThreads::Number(0)` represents "use maximum value",
|
||||
/// as such, it is equal to [`ReaderThreads::OnePerThread`].
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::config::*;
|
||||
/// let reader_threads = ReaderThreads::from(0_usize);
|
||||
/// assert!(matches!(reader_threads, ReaderThreads::OnePerThread));
|
||||
/// ```
|
||||
Number(usize),
|
||||
|
||||
/// Spawn a specified % of reader threads.
|
||||
///
|
||||
/// This must be a value in-between `0.0..1.0`
|
||||
/// where `1.0` represents [`ReaderThreads::OnePerThread`].
|
||||
///
|
||||
/// # Example
|
||||
/// For example, using a `16-core, 32-thread` Ryzen 5950x CPU:
|
||||
///
|
||||
/// | Input | Total thread used |
|
||||
/// |------------------------------------|-------------------|
|
||||
/// | `ReaderThreads::Percent(0.0)` | 32 (maximum value)
|
||||
/// | `ReaderThreads::Percent(0.5)` | 16
|
||||
/// | `ReaderThreads::Percent(0.75)` | 24
|
||||
/// | `ReaderThreads::Percent(1.0)` | 32
|
||||
/// | `ReaderThreads::Percent(2.0)` | 32 (saturating)
|
||||
/// | `ReaderThreads::Percent(f32::NAN)` | 32 (non-normal default)
|
||||
///
|
||||
/// # `0.0`
|
||||
/// `ReaderThreads::Percent(0.0)` represents "use maximum value",
|
||||
/// as such, it is equal to [`ReaderThreads::OnePerThread`].
|
||||
///
|
||||
/// # Not quite `0.0`
|
||||
/// If the thread count multiplied by the percentage ends up being
|
||||
/// non-zero, but not 1 thread, the minimum value 1 will be returned.
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::config::*;
|
||||
/// assert_eq!(ReaderThreads::Percent(0.000000001).as_threads().get(), 1);
|
||||
/// ```
|
||||
Percent(f32),
|
||||
}
|
||||
|
||||
impl ReaderThreads {
|
||||
/// This converts [`ReaderThreads`] into a safe, usable
|
||||
/// number representing how many threads to spawn.
|
||||
///
|
||||
/// This function will always return a number in-between `1..=total_thread_count`.
|
||||
///
|
||||
/// It uses [`cuprate_helper::thread::threads()`] internally to determine the total thread count.
|
||||
///
|
||||
/// # Example
|
||||
/// ```rust
|
||||
/// use cuprate_database::config::ReaderThreads as Rt;
|
||||
///
|
||||
/// let total_threads: std::num::NonZeroUsize =
|
||||
/// cuprate_helper::thread::threads();
|
||||
///
|
||||
/// assert_eq!(Rt::OnePerThread.as_threads(), total_threads);
|
||||
///
|
||||
/// assert_eq!(Rt::One.as_threads().get(), 1);
|
||||
///
|
||||
/// assert_eq!(Rt::Number(0).as_threads(), total_threads);
|
||||
/// assert_eq!(Rt::Number(1).as_threads().get(), 1);
|
||||
/// assert_eq!(Rt::Number(usize::MAX).as_threads(), total_threads);
|
||||
///
|
||||
/// assert_eq!(Rt::Percent(0.01).as_threads().get(), 1);
|
||||
/// assert_eq!(Rt::Percent(0.0).as_threads(), total_threads);
|
||||
/// assert_eq!(Rt::Percent(1.0).as_threads(), total_threads);
|
||||
/// assert_eq!(Rt::Percent(f32::NAN).as_threads(), total_threads);
|
||||
/// assert_eq!(Rt::Percent(f32::INFINITY).as_threads(), total_threads);
|
||||
/// assert_eq!(Rt::Percent(f32::NEG_INFINITY).as_threads(), total_threads);
|
||||
///
|
||||
/// // Percentage only works on more than 1 thread.
|
||||
/// if total_threads.get() > 1 {
|
||||
/// assert_eq!(
|
||||
/// Rt::Percent(0.5).as_threads().get(),
|
||||
/// (total_threads.get() as f32 / 2.0) as usize,
|
||||
/// );
|
||||
/// }
|
||||
/// ```
|
||||
//
|
||||
// INVARIANT:
|
||||
// LMDB will error if we input zero, so don't allow that.
|
||||
// <https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/mdb.c#L4687>
|
||||
pub fn as_threads(&self) -> NonZeroUsize {
|
||||
let total_threads = cuprate_helper::thread::threads();
|
||||
|
||||
match self {
|
||||
Self::OnePerThread => total_threads, // use all threads
|
||||
Self::One => NonZeroUsize::MIN, // one
|
||||
Self::Number(n) => match NonZeroUsize::new(*n) {
|
||||
Some(n) => std::cmp::min(n, total_threads), // saturate at total threads
|
||||
None => total_threads, // 0 == maximum value
|
||||
},
|
||||
|
||||
// We handle the casting loss.
|
||||
#[allow(
|
||||
clippy::cast_precision_loss,
|
||||
clippy::cast_possible_truncation,
|
||||
clippy::cast_sign_loss
|
||||
)]
|
||||
Self::Percent(f) => {
|
||||
// If non-normal float, use the default (all threads).
|
||||
if !f.is_normal() || !(0.0..=1.0).contains(f) {
|
||||
return total_threads;
|
||||
}
|
||||
|
||||
// 0.0 == maximum value.
|
||||
if *f == 0.0 {
|
||||
return total_threads;
|
||||
}
|
||||
|
||||
// Calculate percentage of total threads.
|
||||
let thread_percent = (total_threads.get() as f32) * f;
|
||||
match NonZeroUsize::new(thread_percent as usize) {
|
||||
Some(n) => std::cmp::min(n, total_threads), // saturate at total threads.
|
||||
None => {
|
||||
// We checked for `0.0` above, so what this
|
||||
// being 0 means that the percentage was _so_
|
||||
// low it made our thread count something like
|
||||
// 0.99. In this case, just use 1 thread.
|
||||
NonZeroUsize::MIN
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<T: Into<usize>> From<T> for ReaderThreads {
|
||||
/// Create a [`ReaderThreads::Number`].
|
||||
///
|
||||
/// If `value` is `0`, this will return [`ReaderThreads::OnePerThread`].
|
||||
fn from(value: T) -> Self {
|
||||
let u: usize = value.into();
|
||||
if u == 0 {
|
||||
Self::OnePerThread
|
||||
} else {
|
||||
Self::Number(u)
|
||||
}
|
||||
}
|
||||
}
|
||||
135
storage/database/src/config/sync_mode.rs
Normal file
135
storage/database/src/config/sync_mode.rs
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
//! Database [`Env`](crate::Env) configuration.
|
||||
//!
|
||||
//! This module contains the main [`Config`]uration struct
|
||||
//! for the database [`Env`](crate::Env)ironment, and data
|
||||
//! structures related to any configuration setting.
|
||||
//!
|
||||
//! These configurations are processed at runtime, meaning
|
||||
//! the `Env` can/will dynamically adjust its behavior
|
||||
//! based on these values.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
|
||||
#[cfg(feature = "serde")]
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- SyncMode
|
||||
/// Disk synchronization mode.
|
||||
///
|
||||
/// This controls how/when the database syncs its data to disk.
|
||||
///
|
||||
/// Regardless of the variant chosen, dropping [`Env`](crate::Env)
|
||||
/// will always cause it to fully sync to disk.
|
||||
///
|
||||
/// # Sync vs Async
|
||||
/// All invariants except [`SyncMode::Async`] & [`SyncMode::Fast`]
|
||||
/// are `synchronous`, as in the database will wait until the OS has
|
||||
/// finished syncing all the data to disk before continuing.
|
||||
///
|
||||
/// `SyncMode::Async` & `SyncMode::Fast` are `asynchronous`, meaning
|
||||
/// the database will _NOT_ wait until the data is fully synced to disk
|
||||
/// before continuing. Note that this doesn't mean the database itself
|
||||
/// won't be synchronized between readers/writers, but rather that the
|
||||
/// data _on disk_ may not be immediately synchronized after a write.
|
||||
///
|
||||
/// Something like:
|
||||
/// ```rust,ignore
|
||||
/// db.put("key", value);
|
||||
/// db.get("key");
|
||||
/// ```
|
||||
/// will be fine, most likely pulling from memory instead of disk.
|
||||
///
|
||||
/// # SOMEDAY
|
||||
/// Dynamic sync's are not yet supported.
|
||||
///
|
||||
/// Only:
|
||||
///
|
||||
/// - [`SyncMode::Safe`]
|
||||
/// - [`SyncMode::Async`]
|
||||
/// - [`SyncMode::Fast`]
|
||||
///
|
||||
/// are supported, all other variants will panic on [`crate::Env::open`].
|
||||
#[derive(Copy, Clone, Debug, Default, PartialEq, PartialOrd, Eq, Ord, Hash)]
|
||||
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
|
||||
pub enum SyncMode {
|
||||
/// Use [`SyncMode::Fast`] until fully synced,
|
||||
/// then use [`SyncMode::Safe`].
|
||||
///
|
||||
// # SOMEDAY: how to implement this?
|
||||
// ref: <https://github.com/monero-project/monero/issues/1463>
|
||||
// monerod-solution: <https://github.com/monero-project/monero/pull/1506>
|
||||
// cuprate-issue: <https://github.com/Cuprate/cuprate/issues/78>
|
||||
//
|
||||
// We could:
|
||||
// ```rust,ignore
|
||||
// if current_db_block <= top_block.saturating_sub(N) {
|
||||
// // don't sync()
|
||||
// } else {
|
||||
// // sync()
|
||||
// }
|
||||
// ```
|
||||
// where N is some threshold we pick that is _close_ enough
|
||||
// to being synced where we want to start being safer.
|
||||
//
|
||||
// Essentially, when we are in a certain % range of being finished,
|
||||
// switch to safe mode, until then, go fast.
|
||||
FastThenSafe,
|
||||
|
||||
#[default]
|
||||
/// Fully sync to disk per transaction.
|
||||
///
|
||||
/// Every database transaction commit will
|
||||
/// fully sync all data to disk, _synchronously_,
|
||||
/// so the database (writer) halts until synced.
|
||||
///
|
||||
/// This is expected to be very slow.
|
||||
///
|
||||
/// This matches:
|
||||
/// - LMDB without any special sync flags
|
||||
/// - [`redb::Durability::Immediate`](https://docs.rs/redb/1.5.0/redb/enum.Durability.html#variant.Immediate)
|
||||
Safe,
|
||||
|
||||
/// Asynchrously sync to disk per transaction.
|
||||
///
|
||||
/// This is the same as [`SyncMode::Safe`],
|
||||
/// but the syncs will be asynchronous, i.e.
|
||||
/// each transaction commit will sync to disk,
|
||||
/// but only eventually, not necessarily immediately.
|
||||
///
|
||||
/// This matches:
|
||||
/// - [`MDB_MAPASYNC`](http://www.lmdb.tech/doc/group__mdb__env.html#gab034ed0d8e5938090aef5ee0997f7e94)
|
||||
/// - [`redb::Durability::Eventual`](https://docs.rs/redb/1.5.0/redb/enum.Durability.html#variant.Eventual)
|
||||
Async,
|
||||
|
||||
/// Fully sync to disk after we cross this transaction threshold.
|
||||
///
|
||||
/// After committing [`usize`] amount of database
|
||||
/// transactions, it will be sync to disk.
|
||||
///
|
||||
/// `0` behaves the same as [`SyncMode::Safe`], and a ridiculously large
|
||||
/// number like `usize::MAX` is practically the same as [`SyncMode::Fast`].
|
||||
Threshold(usize),
|
||||
|
||||
/// Only flush at database shutdown.
|
||||
///
|
||||
/// This is the fastest, yet unsafest option.
|
||||
///
|
||||
/// It will cause the database to never _actively_ sync,
|
||||
/// letting the OS decide when to flush data to disk.
|
||||
///
|
||||
/// This matches:
|
||||
/// - [`MDB_NOSYNC`](http://www.lmdb.tech/doc/group__mdb__env.html#ga5791dd1adb09123f82dd1f331209e12e) + [`MDB_MAPASYNC`](http://www.lmdb.tech/doc/group__mdb__env.html#gab034ed0d8e5938090aef5ee0997f7e94)
|
||||
/// - [`redb::Durability::None`](https://docs.rs/redb/1.5.0/redb/enum.Durability.html#variant.None)
|
||||
///
|
||||
/// `monerod` reference: <https://github.com/monero-project/monero/blob/7b7958bbd9d76375c47dc418b4adabba0f0b1785/src/blockchain_db/lmdb/db_lmdb.cpp#L1380-L1381>
|
||||
///
|
||||
/// # Corruption
|
||||
/// In the case of a system crash, the database
|
||||
/// may become corrupted when using this option.
|
||||
//
|
||||
// FIXME: we could call this `unsafe`
|
||||
// and use that terminology in the config file
|
||||
// so users know exactly what they are getting
|
||||
// themselves into.
|
||||
Fast,
|
||||
}
|
||||
86
storage/database/src/constants.rs
Normal file
86
storage/database/src/constants.rs
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
//! General constants used throughout `cuprate-database`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use cfg_if::cfg_if;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Version
|
||||
/// Current major version of the database.
|
||||
///
|
||||
/// Returned by [`crate::ops::property::db_version`].
|
||||
///
|
||||
/// This is incremented by 1 when `cuprate_database`'s
|
||||
/// structure/schema/tables change.
|
||||
///
|
||||
/// This is akin to `VERSION` in `monerod`:
|
||||
/// <https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/blockchain_db/lmdb/db_lmdb.cpp#L57>
|
||||
pub const DATABASE_VERSION: u64 = 0;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Error Messages
|
||||
/// Corrupt database error message.
|
||||
///
|
||||
/// The error message shown to end-users in panic
|
||||
/// messages if we think the database is corrupted.
|
||||
///
|
||||
/// This is meant to be user-friendly.
|
||||
pub const DATABASE_CORRUPT_MSG: &str = r"Cuprate has encountered a fatal error. The database may be corrupted.
|
||||
|
||||
TODO: instructions on:
|
||||
1. What to do
|
||||
2. How to fix (re-sync, recover, etc)
|
||||
3. General advice for preventing corruption
|
||||
4. etc";
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Misc
|
||||
/// Static string of the `crate` being used as the database backend.
|
||||
///
|
||||
/// | Backend | Value |
|
||||
/// |---------|-------|
|
||||
/// | `heed` | `"heed"`
|
||||
/// | `redb` | `"redb"`
|
||||
pub const DATABASE_BACKEND: &str = {
|
||||
cfg_if! {
|
||||
if #[cfg(all(feature = "redb", not(feature = "heed")))] {
|
||||
"redb"
|
||||
} else {
|
||||
"heed"
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
/// Cuprate's database filename.
|
||||
///
|
||||
/// Used in [`Config::db_file`](crate::config::Config::db_file).
|
||||
///
|
||||
/// | Backend | Value |
|
||||
/// |---------|-------|
|
||||
/// | `heed` | `"data.mdb"`
|
||||
/// | `redb` | `"data.redb"`
|
||||
pub const DATABASE_DATA_FILENAME: &str = {
|
||||
cfg_if! {
|
||||
if #[cfg(all(feature = "redb", not(feature = "heed")))] {
|
||||
"data.redb"
|
||||
} else {
|
||||
"data.mdb"
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
/// Cuprate's database lock filename.
|
||||
///
|
||||
/// | Backend | Value |
|
||||
/// |---------|-------|
|
||||
/// | `heed` | `Some("lock.mdb")`
|
||||
/// | `redb` | `None` (redb doesn't use a file lock)
|
||||
pub const DATABASE_LOCK_FILENAME: Option<&str> = {
|
||||
cfg_if! {
|
||||
if #[cfg(all(feature = "redb", not(feature = "heed")))] {
|
||||
None
|
||||
} else {
|
||||
Some("lock.mdb")
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {}
|
||||
216
storage/database/src/database.rs
Normal file
216
storage/database/src/database.rs
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
//! Abstracted database table operations; `trait DatabaseRo` & `trait DatabaseRw`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::ops::RangeBounds;
|
||||
|
||||
use crate::{error::RuntimeError, table::Table};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseIter
|
||||
/// Generic post-fix documentation for `DatabaseIter` methods.
|
||||
macro_rules! doc_iter {
|
||||
() => {
|
||||
r"Although the returned iterator itself is tied to the lifetime
|
||||
of `&self`, the returned values from the iterator are _owned_.
|
||||
|
||||
# Errors
|
||||
The construction of the iterator itself may error.
|
||||
|
||||
Each iteration of the iterator has the potential to error as well."
|
||||
};
|
||||
}
|
||||
|
||||
/// Database (key-value store) read-only iteration abstraction.
|
||||
///
|
||||
/// These are read-only iteration-related operations that
|
||||
/// can only be called from [`DatabaseRo`] objects.
|
||||
///
|
||||
/// # Hack
|
||||
/// This is a HACK to get around the fact [`DatabaseRw`] tables
|
||||
/// cannot safely return values returning lifetimes, as such,
|
||||
/// only read-only tables implement this trait.
|
||||
///
|
||||
/// - <https://github.com/Cuprate/cuprate/pull/102#discussion_r1548695610>
|
||||
/// - <https://github.com/Cuprate/cuprate/pull/104>
|
||||
pub trait DatabaseIter<T: Table> {
|
||||
/// Get an [`Iterator`] of value's corresponding to a range of keys.
|
||||
///
|
||||
/// For example:
|
||||
/// ```rust,ignore
|
||||
/// // This will return all 100 values corresponding
|
||||
/// // to the keys `{0, 1, 2, ..., 100}`.
|
||||
/// self.get_range(0..100);
|
||||
/// ```
|
||||
///
|
||||
/// Although the returned iterator itself is tied to the lifetime
|
||||
/// of `&'a self`, the returned values from the iterator are _owned_.
|
||||
///
|
||||
#[doc = doc_iter!()]
|
||||
fn get_range<'a, Range>(
|
||||
&'a self,
|
||||
range: Range,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + 'a, RuntimeError>
|
||||
where
|
||||
Range: RangeBounds<T::Key> + 'a;
|
||||
|
||||
/// Get an [`Iterator`] that returns the `(key, value)` types for this database.
|
||||
#[doc = doc_iter!()]
|
||||
#[allow(clippy::iter_not_returning_iterator)]
|
||||
fn iter(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<(T::Key, T::Value), RuntimeError>> + '_, RuntimeError>;
|
||||
|
||||
/// Get an [`Iterator`] that returns _only_ the `key` type for this database.
|
||||
#[doc = doc_iter!()]
|
||||
fn keys(&self)
|
||||
-> Result<impl Iterator<Item = Result<T::Key, RuntimeError>> + '_, RuntimeError>;
|
||||
|
||||
/// Get an [`Iterator`] that returns _only_ the `value` type for this database.
|
||||
#[doc = doc_iter!()]
|
||||
fn values(
|
||||
&self,
|
||||
) -> Result<impl Iterator<Item = Result<T::Value, RuntimeError>> + '_, RuntimeError>;
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRo
|
||||
/// Generic post-fix documentation for `DatabaseR{o,w}` methods.
|
||||
macro_rules! doc_database {
|
||||
() => {
|
||||
r"# Errors
|
||||
This will return [`RuntimeError::KeyNotFound`] if:
|
||||
- Input does not exist OR
|
||||
- Database is empty"
|
||||
};
|
||||
}
|
||||
|
||||
/// Database (key-value store) read abstraction.
|
||||
///
|
||||
/// This is a read-only database table,
|
||||
/// write operations are defined in [`DatabaseRw`].
|
||||
///
|
||||
/// # Safety
|
||||
/// The table type that implements this MUST be `Send`.
|
||||
///
|
||||
/// However if the table holds a reference to a transaction:
|
||||
/// - only the transaction only has to be `Send`
|
||||
/// - the table cannot implement `Send`
|
||||
///
|
||||
/// For example:
|
||||
///
|
||||
/// `heed`'s transactions are `Send` but `HeedTableRo` contains a `&`
|
||||
/// to the transaction, as such, if `Send` were implemented on `HeedTableRo`
|
||||
/// then 1 transaction could be used to open multiple tables, then sent to
|
||||
/// other threads - this would be a soundness hole against `HeedTableRo`.
|
||||
///
|
||||
/// `&T` is only `Send` if `T: Sync`.
|
||||
///
|
||||
/// `heed::RoTxn: !Sync`, therefore our table
|
||||
/// holding `&heed::RoTxn` must NOT be `Send`.
|
||||
///
|
||||
/// - <https://doc.rust-lang.org/std/marker/trait.Sync.html>
|
||||
/// - <https://doc.rust-lang.org/nomicon/send-and-sync.html>
|
||||
pub unsafe trait DatabaseRo<T: Table> {
|
||||
/// Get the value corresponding to a key.
|
||||
#[doc = doc_database!()]
|
||||
fn get(&self, key: &T::Key) -> Result<T::Value, RuntimeError>;
|
||||
|
||||
/// Returns `true` if the database contains a value for the specified key.
|
||||
///
|
||||
/// # Errors
|
||||
/// Note that this will _never_ return `Err(RuntimeError::KeyNotFound)`,
|
||||
/// as in that case, `Ok(false)` will be returned.
|
||||
///
|
||||
/// Other errors may still occur.
|
||||
fn contains(&self, key: &T::Key) -> Result<bool, RuntimeError> {
|
||||
match self.get(key) {
|
||||
Ok(_) => Ok(true),
|
||||
Err(RuntimeError::KeyNotFound) => Ok(false),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the number of `(key, value)` pairs in the database.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will never return [`RuntimeError::KeyNotFound`].
|
||||
fn len(&self) -> Result<u64, RuntimeError>;
|
||||
|
||||
/// Returns the first `(key, value)` pair in the database.
|
||||
#[doc = doc_database!()]
|
||||
fn first(&self) -> Result<(T::Key, T::Value), RuntimeError>;
|
||||
|
||||
/// Returns the last `(key, value)` pair in the database.
|
||||
#[doc = doc_database!()]
|
||||
fn last(&self) -> Result<(T::Key, T::Value), RuntimeError>;
|
||||
|
||||
/// Returns `true` if the database contains no `(key, value)` pairs.
|
||||
///
|
||||
/// # Errors
|
||||
/// This can only return [`RuntimeError::Io`] on errors.
|
||||
fn is_empty(&self) -> Result<bool, RuntimeError>;
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRw
|
||||
/// Database (key-value store) read/write abstraction.
|
||||
///
|
||||
/// All [`DatabaseRo`] functions are also callable by [`DatabaseRw`].
|
||||
pub trait DatabaseRw<T: Table>: DatabaseRo<T> {
|
||||
/// Insert a key-value pair into the database.
|
||||
///
|
||||
/// This will overwrite any existing key-value pairs.
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
///
|
||||
/// This will never [`RuntimeError::KeyExists`].
|
||||
fn put(&mut self, key: &T::Key, value: &T::Value) -> Result<(), RuntimeError>;
|
||||
|
||||
/// Delete a key-value pair in the database.
|
||||
///
|
||||
/// This will return `Ok(())` if the key does not exist.
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
///
|
||||
/// This will never [`RuntimeError::KeyExists`].
|
||||
fn delete(&mut self, key: &T::Key) -> Result<(), RuntimeError>;
|
||||
|
||||
/// Delete and return a key-value pair in the database.
|
||||
///
|
||||
/// This is the same as [`DatabaseRw::delete`], however,
|
||||
/// it will serialize the `T::Value` and return it.
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
fn take(&mut self, key: &T::Key) -> Result<T::Value, RuntimeError>;
|
||||
|
||||
/// Fetch the value, and apply a function to it - or delete the entry.
|
||||
///
|
||||
/// This will call [`DatabaseRo::get`] and call your provided function `f` on it.
|
||||
///
|
||||
/// The [`Option`] `f` returns will dictate whether `update()`:
|
||||
/// - Updates the current value OR
|
||||
/// - Deletes the `(key, value)` pair
|
||||
///
|
||||
/// - If `f` returns `Some(value)`, that will be [`DatabaseRw::put`] as the new value
|
||||
/// - If `f` returns `None`, the entry will be [`DatabaseRw::delete`]d
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
fn update<F>(&mut self, key: &T::Key, mut f: F) -> Result<(), RuntimeError>
|
||||
where
|
||||
F: FnMut(T::Value) -> Option<T::Value>,
|
||||
{
|
||||
let value = DatabaseRo::get(self, key)?;
|
||||
|
||||
match f(value) {
|
||||
Some(value) => DatabaseRw::put(self, key, &value),
|
||||
None => DatabaseRw::delete(self, key),
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes and returns the first `(key, value)` pair in the database.
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
fn pop_first(&mut self) -> Result<(T::Key, T::Value), RuntimeError>;
|
||||
|
||||
/// Removes and returns the last `(key, value)` pair in the database.
|
||||
///
|
||||
#[doc = doc_database!()]
|
||||
fn pop_last(&mut self) -> Result<(T::Key, T::Value), RuntimeError>;
|
||||
}
|
||||
286
storage/database/src/env.rs
Normal file
286
storage/database/src/env.rs
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
//! Abstracted database environment; `trait Env`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::num::NonZeroUsize;
|
||||
|
||||
use crate::{
|
||||
config::Config,
|
||||
database::{DatabaseIter, DatabaseRo, DatabaseRw},
|
||||
error::{InitError, RuntimeError},
|
||||
resize::ResizeAlgorithm,
|
||||
table::Table,
|
||||
tables::{call_fn_on_all_tables_or_early_return, TablesIter, TablesMut},
|
||||
transaction::{TxRo, TxRw},
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Env
|
||||
/// Database environment abstraction.
|
||||
///
|
||||
/// Essentially, the functions that can be called on [`ConcreteEnv`](crate::ConcreteEnv).
|
||||
///
|
||||
/// # `Drop`
|
||||
/// Objects that implement [`Env`] _should_ probably
|
||||
/// [`Env::sync`] in their drop implementations,
|
||||
/// although, no invariant relies on this (yet).
|
||||
///
|
||||
/// # Lifetimes
|
||||
/// The lifetimes associated with `Env` have a sequential flow:
|
||||
/// 1. `ConcreteEnv`
|
||||
/// 2. `'env`
|
||||
/// 3. `'tx`
|
||||
/// 4. `'db`
|
||||
///
|
||||
/// As in:
|
||||
/// - open database tables only live as long as...
|
||||
/// - transactions which only live as long as the...
|
||||
/// - environment ([`EnvInner`])
|
||||
pub trait Env: Sized {
|
||||
//------------------------------------------------ Constants
|
||||
/// Does the database backend need to be manually
|
||||
/// resized when the memory-map is full?
|
||||
///
|
||||
/// # Invariant
|
||||
/// If this is `false`, that means this [`Env`]
|
||||
/// must _never_ return a [`RuntimeError::ResizeNeeded`].
|
||||
///
|
||||
/// If this is `true`, [`Env::resize_map`] & [`Env::current_map_size`]
|
||||
/// _must_ be re-implemented, as it just panics by default.
|
||||
const MANUAL_RESIZE: bool;
|
||||
|
||||
/// Does the database backend forcefully sync/flush
|
||||
/// to disk on every transaction commit?
|
||||
///
|
||||
/// This is used as an optimization.
|
||||
const SYNCS_PER_TX: bool;
|
||||
|
||||
//------------------------------------------------ Types
|
||||
/// The struct representing the actual backend's database environment.
|
||||
///
|
||||
/// This is used as the `self` in [`EnvInner`] functions, so whatever
|
||||
/// this type is, is what will be accessible from those functions.
|
||||
///
|
||||
// # HACK
|
||||
// For `heed`, this is just `heed::Env`, for `redb` this is
|
||||
// `(redb::Database, redb::Durability)` as each transaction
|
||||
// needs the sync mode set during creation.
|
||||
type EnvInner<'env>: EnvInner<'env, Self::TxRo<'env>, Self::TxRw<'env>>
|
||||
where
|
||||
Self: 'env;
|
||||
|
||||
/// The read-only transaction type of the backend.
|
||||
type TxRo<'env>: TxRo<'env> + 'env
|
||||
where
|
||||
Self: 'env;
|
||||
|
||||
/// The read/write transaction type of the backend.
|
||||
type TxRw<'env>: TxRw<'env> + 'env
|
||||
where
|
||||
Self: 'env;
|
||||
|
||||
//------------------------------------------------ Required
|
||||
/// Open the database environment, using the passed [`Config`].
|
||||
///
|
||||
/// # Invariants
|
||||
/// This function **must** create all tables listed in [`crate::tables`].
|
||||
///
|
||||
/// The rest of the functions depend on the fact
|
||||
/// they already exist, or else they will panic.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will error if the database could not be opened.
|
||||
///
|
||||
/// This is the only [`Env`] function that will return
|
||||
/// an [`InitError`] instead of a [`RuntimeError`].
|
||||
fn open(config: Config) -> Result<Self, InitError>;
|
||||
|
||||
/// Return the [`Config`] that this database was [`Env::open`]ed with.
|
||||
fn config(&self) -> &Config;
|
||||
|
||||
/// Fully sync the database caches to disk.
|
||||
///
|
||||
/// # Invariant
|
||||
/// This must **fully** and **synchronously** flush the database data to disk.
|
||||
///
|
||||
/// I.e., after this function returns, there must be no doubts
|
||||
/// that the data isn't synced yet, it _must_ be synced.
|
||||
///
|
||||
// FIXME: either this invariant or `sync()` itself will most
|
||||
// likely be removed/changed after `SyncMode` is finalized.
|
||||
///
|
||||
/// # Errors
|
||||
/// If there is a synchronization error, this should return an error.
|
||||
fn sync(&self) -> Result<(), RuntimeError>;
|
||||
|
||||
/// Resize the database's memory map to a
|
||||
/// new (bigger) size using a [`ResizeAlgorithm`].
|
||||
///
|
||||
/// By default, this function will use the `ResizeAlgorithm` in [`Env::config`].
|
||||
///
|
||||
/// If `resize_algorithm` is `Some`, that will be used instead.
|
||||
///
|
||||
/// This function returns the _new_ memory map size in bytes.
|
||||
///
|
||||
/// # Invariant
|
||||
/// This function _must_ be re-implemented if [`Env::MANUAL_RESIZE`] is `true`.
|
||||
///
|
||||
/// Otherwise, this function will panic with `unreachable!()`.
|
||||
#[allow(unused_variables)]
|
||||
fn resize_map(&self, resize_algorithm: Option<ResizeAlgorithm>) -> NonZeroUsize {
|
||||
unreachable!()
|
||||
}
|
||||
|
||||
/// What is the _current_ size of the database's memory map in bytes?
|
||||
///
|
||||
/// # Invariant
|
||||
/// 1. This function _must_ be re-implemented if [`Env::MANUAL_RESIZE`] is `true`.
|
||||
/// 2. This function must be accurate, as [`Env::resize_map()`] may depend on it.
|
||||
fn current_map_size(&self) -> usize {
|
||||
unreachable!()
|
||||
}
|
||||
|
||||
/// Return the [`Env::EnvInner`].
|
||||
///
|
||||
/// # Locking behavior
|
||||
/// When using the `heed` backend, [`Env::EnvInner`] is a
|
||||
/// `RwLockReadGuard`, i.e., calling this function takes a
|
||||
/// read lock on the `heed::Env`.
|
||||
///
|
||||
/// Be aware of this, as other functions (currently only
|
||||
/// [`Env::resize_map`]) will take a _write_ lock.
|
||||
fn env_inner(&self) -> Self::EnvInner<'_>;
|
||||
|
||||
//------------------------------------------------ Provided
|
||||
/// Return the amount of actual of bytes the database is taking up on disk.
|
||||
///
|
||||
/// This is the current _disk_ value in bytes, not the memory map.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will error if either:
|
||||
///
|
||||
/// - [`std::fs::File::open`]
|
||||
/// - [`std::fs::File::metadata`]
|
||||
///
|
||||
/// failed on the database file on disk.
|
||||
fn disk_size_bytes(&self) -> std::io::Result<u64> {
|
||||
// We have the direct PATH to the file,
|
||||
// no need to use backend-specific functions.
|
||||
//
|
||||
// SAFETY: as we are only accessing the metadata of
|
||||
// the file and not reading the bytes, it should be
|
||||
// fine even with a memory mapped file being actively
|
||||
// written to.
|
||||
Ok(std::fs::File::open(&self.config().db_file)?
|
||||
.metadata()?
|
||||
.len())
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseRo
|
||||
/// Document errors when opening tables in [`EnvInner`].
|
||||
macro_rules! doc_table_error {
|
||||
() => {
|
||||
r"# Errors
|
||||
This will only return [`RuntimeError::Io`] if it errors.
|
||||
|
||||
As all tables are created upon [`Env::open`],
|
||||
this function will never error because a table doesn't exist."
|
||||
};
|
||||
}
|
||||
|
||||
/// The inner [`Env`] type.
|
||||
///
|
||||
/// This type is created with [`Env::env_inner`] and represents
|
||||
/// the type able to generate transactions and open tables.
|
||||
///
|
||||
/// # Locking behavior
|
||||
/// As noted in `Env::env_inner`, this is a `RwLockReadGuard`
|
||||
/// when using the `heed` backend, be aware of this and do
|
||||
/// not hold onto an `EnvInner` for a long time.
|
||||
pub trait EnvInner<'env, Ro, Rw>
|
||||
where
|
||||
Self: 'env,
|
||||
Ro: TxRo<'env>,
|
||||
Rw: TxRw<'env>,
|
||||
{
|
||||
/// Create a read-only transaction.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will only return [`RuntimeError::Io`] if it errors.
|
||||
fn tx_ro(&'env self) -> Result<Ro, RuntimeError>;
|
||||
|
||||
/// Create a read/write transaction.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will only return [`RuntimeError::Io`] if it errors.
|
||||
fn tx_rw(&'env self) -> Result<Rw, RuntimeError>;
|
||||
|
||||
/// Open a database in read-only mode.
|
||||
///
|
||||
/// The returned value can have [`DatabaseRo`]
|
||||
/// & [`DatabaseIter`] functions called on it.
|
||||
///
|
||||
/// This will open the database [`Table`]
|
||||
/// passed as a generic to this function.
|
||||
///
|
||||
/// ```rust,ignore
|
||||
/// let db = env.open_db_ro::<Table>(&tx_ro);
|
||||
/// // ^ ^
|
||||
/// // database table table metadata
|
||||
/// // (name, key/value type)
|
||||
/// ```
|
||||
///
|
||||
#[doc = doc_table_error!()]
|
||||
fn open_db_ro<T: Table>(
|
||||
&self,
|
||||
tx_ro: &Ro,
|
||||
) -> Result<impl DatabaseRo<T> + DatabaseIter<T>, RuntimeError>;
|
||||
|
||||
/// Open a database in read/write mode.
|
||||
///
|
||||
/// All [`DatabaseRo`] functions are also callable
|
||||
/// with the returned [`DatabaseRw`] structure.
|
||||
///
|
||||
/// Note that [`DatabaseIter`] functions are _not_
|
||||
/// available to [`DatabaseRw`] structures.
|
||||
///
|
||||
/// This will open the database [`Table`]
|
||||
/// passed as a generic to this function.
|
||||
///
|
||||
#[doc = doc_table_error!()]
|
||||
fn open_db_rw<T: Table>(&self, tx_rw: &Rw) -> Result<impl DatabaseRw<T>, RuntimeError>;
|
||||
|
||||
/// Open all tables in read/iter mode.
|
||||
///
|
||||
/// This calls [`EnvInner::open_db_ro`] on all database tables
|
||||
/// and returns a structure that allows access to all tables.
|
||||
///
|
||||
#[doc = doc_table_error!()]
|
||||
fn open_tables(&self, tx_ro: &Ro) -> Result<impl TablesIter, RuntimeError> {
|
||||
call_fn_on_all_tables_or_early_return! {
|
||||
Self::open_db_ro(self, tx_ro)
|
||||
}
|
||||
}
|
||||
|
||||
/// Open all tables in read-write mode.
|
||||
///
|
||||
/// This calls [`EnvInner::open_db_rw`] on all database tables
|
||||
/// and returns a structure that allows access to all tables.
|
||||
///
|
||||
#[doc = doc_table_error!()]
|
||||
fn open_tables_mut(&self, tx_rw: &Rw) -> Result<impl TablesMut, RuntimeError> {
|
||||
call_fn_on_all_tables_or_early_return! {
|
||||
Self::open_db_rw(self, tx_rw)
|
||||
}
|
||||
}
|
||||
|
||||
/// Clear all `(key, value)`'s from a database table.
|
||||
///
|
||||
/// This will delete all key and values in the passed
|
||||
/// `T: Table`, but the table itself will continue to exist.
|
||||
///
|
||||
/// Note that this operation is tied to `tx_rw`, as such this
|
||||
/// function's effects can be aborted using [`TxRw::abort`].
|
||||
///
|
||||
#[doc = doc_table_error!()]
|
||||
fn clear_db<T: Table>(&self, tx_rw: &mut Rw) -> Result<(), RuntimeError>;
|
||||
}
|
||||
94
storage/database/src/error.rs
Normal file
94
storage/database/src/error.rs
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
//! Database error types.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::fmt::Debug;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Types
|
||||
/// Alias for a thread-safe boxed error.
|
||||
type BoxError = Box<dyn std::error::Error + Send + Sync + 'static>;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- InitError
|
||||
/// Errors that occur during ([`Env::open`](crate::env::Env::open)).
|
||||
///
|
||||
/// # Handling
|
||||
/// As this is a database initialization error, the correct
|
||||
/// way to handle any of these occurring is probably just to
|
||||
/// exit the program.
|
||||
///
|
||||
/// There is not much we as Cuprate can do
|
||||
/// to recover from any of these errors.
|
||||
#[derive(thiserror::Error, Debug)]
|
||||
pub enum InitError {
|
||||
/// The given `Path/File` existed and was accessible,
|
||||
/// but was not a valid database file.
|
||||
#[error("database file exists but is not valid")]
|
||||
Invalid,
|
||||
|
||||
/// The given `Path/File` existed, was a valid
|
||||
/// database, but the version is incorrect.
|
||||
#[error("database file is valid, but version is incorrect")]
|
||||
InvalidVersion,
|
||||
|
||||
/// I/O error.
|
||||
#[error("database I/O error: {0}")]
|
||||
Io(#[from] std::io::Error),
|
||||
|
||||
/// The given `Path/File` existed,
|
||||
/// was a valid database, but it is corrupt.
|
||||
#[error("database file is corrupt")]
|
||||
Corrupt,
|
||||
|
||||
/// The database is currently in the process
|
||||
/// of shutting down and cannot respond.
|
||||
///
|
||||
/// # Notes
|
||||
/// This error can only occur with the `heed` backend when
|
||||
/// the database environment is opened _right_ at the same time
|
||||
/// another thread/process is closing it.
|
||||
///
|
||||
/// This will never occur with other backends.
|
||||
#[error("database is shutting down")]
|
||||
ShuttingDown,
|
||||
|
||||
/// An unknown error occurred.
|
||||
///
|
||||
/// This is for errors that cannot be recovered from,
|
||||
/// but we'd still like to panic gracefully.
|
||||
#[error("unknown error: {0}")]
|
||||
Unknown(BoxError),
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- RuntimeError
|
||||
/// Errors that occur _after_ successful ([`Env::open`](crate::env::Env::open)).
|
||||
///
|
||||
/// There are no errors for:
|
||||
/// 1. Missing tables
|
||||
/// 2. (De)serialization
|
||||
/// 3. Shutdown errors
|
||||
///
|
||||
/// as `cuprate_database` upholds the invariant that:
|
||||
///
|
||||
/// 1. All tables exist
|
||||
/// 2. (De)serialization never fails
|
||||
/// 3. The database (thread-pool) only shuts down when all channels are dropped
|
||||
#[derive(thiserror::Error, Debug)]
|
||||
pub enum RuntimeError {
|
||||
/// The given key already existed in the database.
|
||||
#[error("key already existed")]
|
||||
KeyExists,
|
||||
|
||||
/// The given key did not exist in the database.
|
||||
#[error("key/value pair was not found")]
|
||||
KeyNotFound,
|
||||
|
||||
/// The database memory map is full and needs a resize.
|
||||
///
|
||||
/// # Invariant
|
||||
/// This error can only occur if [`Env::MANUAL_RESIZE`](crate::Env::MANUAL_RESIZE) is `true`.
|
||||
#[error("database memory map must be resized")]
|
||||
ResizeNeeded,
|
||||
|
||||
/// A [`std::io::Error`].
|
||||
#[error("I/O error: {0}")]
|
||||
Io(#[from] std::io::Error),
|
||||
}
|
||||
11
storage/database/src/free.rs
Normal file
11
storage/database/src/free.rs
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
//! General free functions (related to the database).
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Free functions
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
58
storage/database/src/key.rs
Normal file
58
storage/database/src/key.rs
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
//! Database key abstraction; `trait Key`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::cmp::Ordering;
|
||||
|
||||
use crate::storable::Storable;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Table
|
||||
/// Database [`Table`](crate::table::Table) key metadata.
|
||||
///
|
||||
/// Purely compile time information for database table keys.
|
||||
//
|
||||
// FIXME: this doesn't need to exist right now but
|
||||
// may be used if we implement getting values using ranges.
|
||||
// <https://github.com/Cuprate/cuprate/pull/117#discussion_r1589378104>
|
||||
pub trait Key: Storable + Sized {
|
||||
/// The primary key type.
|
||||
type Primary: Storable;
|
||||
|
||||
/// Compare 2 [`Key`]'s against each other.
|
||||
///
|
||||
/// By default, this does a straight _byte_ comparison,
|
||||
/// not a comparison of the key's value.
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::*;
|
||||
/// assert_eq!(
|
||||
/// <u64 as Key>::compare([0].as_slice(), [1].as_slice()),
|
||||
/// std::cmp::Ordering::Less,
|
||||
/// );
|
||||
/// assert_eq!(
|
||||
/// <u64 as Key>::compare([1].as_slice(), [1].as_slice()),
|
||||
/// std::cmp::Ordering::Equal,
|
||||
/// );
|
||||
/// assert_eq!(
|
||||
/// <u64 as Key>::compare([2].as_slice(), [1].as_slice()),
|
||||
/// std::cmp::Ordering::Greater,
|
||||
/// );
|
||||
/// ```
|
||||
#[inline]
|
||||
fn compare(left: &[u8], right: &[u8]) -> Ordering {
|
||||
left.cmp(right)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Impl
|
||||
impl<T> Key for T
|
||||
where
|
||||
T: Storable + Sized,
|
||||
{
|
||||
type Primary = Self;
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
301
storage/database/src/lib.rs
Normal file
301
storage/database/src/lib.rs
Normal file
|
|
@ -0,0 +1,301 @@
|
|||
//! Cuprate's database abstraction.
|
||||
//!
|
||||
//! This documentation is mostly for practical usage of `cuprate_database`.
|
||||
//!
|
||||
//! For a high-level overview,
|
||||
//! see [`database/README.md`](https://github.com/Cuprate/cuprate/blob/main/database/README.md).
|
||||
//!
|
||||
//! # Purpose
|
||||
//! This crate does 3 things:
|
||||
//! 1. Abstracts various database backends with traits
|
||||
//! 2. Implements various `Monero` related [operations](ops), [tables], and [types]
|
||||
//! 3. Exposes a [`tower::Service`] backed by a thread-pool
|
||||
//!
|
||||
//! Each layer builds on-top of the previous.
|
||||
//!
|
||||
//! As a user of `cuprate_database`, consider using the higher-level [`service`] module,
|
||||
//! or at the very least the [`ops`] module instead of interacting with the database traits directly.
|
||||
//!
|
||||
//! With that said, many database traits and internals (like [`DatabaseRo::get`]) are exposed.
|
||||
//!
|
||||
//! # Terminology
|
||||
//! To be more clear on some terms used in this crate:
|
||||
//!
|
||||
//! | Term | Meaning |
|
||||
//! |------------------|--------------------------------------|
|
||||
//! | `Env` | The 1 database environment, the "whole" thing
|
||||
//! | `DatabaseR{o,w}` | A _actively open_ readable/writable `key/value` store
|
||||
//! | `Table` | Solely the metadata of a `Database` (the `key` and `value` types, and the name)
|
||||
//! | `TxR{o,w}` | A read/write transaction
|
||||
//! | `Storable` | A data that type can be stored in the database
|
||||
//!
|
||||
//! The dataflow is `Env` -> `Tx` -> `Database`
|
||||
//!
|
||||
//! Which reads as:
|
||||
//! 1. You have a database `Environment`
|
||||
//! 1. You open up a `Transaction`
|
||||
//! 1. You open a particular `Table` from that `Environment`, getting a `Database`
|
||||
//! 1. You can now read/write data from/to that `Database`
|
||||
//!
|
||||
//! # `ConcreteEnv`
|
||||
//! This crate exposes [`ConcreteEnv`], which is a non-generic/non-dynamic,
|
||||
//! concrete object representing a database [`Env`]ironment.
|
||||
//!
|
||||
//! The actual backend for this type is determined via feature flags.
|
||||
//!
|
||||
//! This object existing means `E: Env` doesn't need to be spread all through the codebase,
|
||||
//! however, it also means some small invariants should be kept in mind.
|
||||
//!
|
||||
//! As `ConcreteEnv` is just a re-exposed type which has varying inner types,
|
||||
//! it means some properties will change depending on the backend used.
|
||||
//!
|
||||
//! For example:
|
||||
//! - [`std::mem::size_of::<ConcreteEnv>`]
|
||||
//! - [`std::mem::align_of::<ConcreteEnv>`]
|
||||
//!
|
||||
//! Things like these functions are affected by the backend and inner data,
|
||||
//! and should not be relied upon. This extends to any `struct/enum` that contains `ConcreteEnv`.
|
||||
//!
|
||||
//! `ConcreteEnv` invariants you can rely on:
|
||||
//! - It implements [`Env`]
|
||||
//! - Upon [`Drop::drop`], all database data will sync to disk
|
||||
//!
|
||||
//! Note that `ConcreteEnv` itself is not a clonable type,
|
||||
//! it should be wrapped in [`std::sync::Arc`].
|
||||
//!
|
||||
//! <!-- SOMEDAY: replace `ConcreteEnv` with `fn Env::open() -> impl Env`/
|
||||
//! and use `<E: Env>` everywhere it is stored instead. This would allow
|
||||
//! generic-backed dynamic runtime selection of the database backend, i.e.
|
||||
//! the user can select which database backend they use. -->
|
||||
//!
|
||||
//! # Feature flags
|
||||
//! The `service` module requires the `service` feature to be enabled.
|
||||
//! See the module for more documentation.
|
||||
//!
|
||||
//! Different database backends are enabled by the feature flags:
|
||||
//! - `heed` (LMDB)
|
||||
//! - `redb`
|
||||
//!
|
||||
//! The default is `heed`.
|
||||
//!
|
||||
//! `tracing` is always enabled and cannot be disabled via feature-flag.
|
||||
//! <!-- FIXME: tracing should be behind a feature flag -->
|
||||
//!
|
||||
//! # Invariants when not using `service`
|
||||
//! `cuprate_database` can be used without the `service` feature enabled but
|
||||
//! there are some things that must be kept in mind when doing so.
|
||||
//!
|
||||
//! Failing to uphold these invariants may cause panics.
|
||||
//!
|
||||
//! 1. `LMDB` requires the user to resize the memory map resizing (see [`RuntimeError::ResizeNeeded`]
|
||||
//! 1. `LMDB` has a maximum reader transaction count, currently it is set to `128`
|
||||
//! 1. `LMDB` has [maximum key/value byte size](http://www.lmdb.tech/doc/group__internal.html#gac929399f5d93cef85f874b9e9b1d09e0) which must not be exceeded
|
||||
//!
|
||||
//! # Examples
|
||||
//! The below is an example of using `cuprate_database`'s
|
||||
//! lowest API, i.e. using the database directly.
|
||||
//!
|
||||
//! For examples of the higher-level APIs, see:
|
||||
//! - [`ops`]
|
||||
//! - [`service`]
|
||||
//!
|
||||
//! ```rust
|
||||
//! use cuprate_database::{
|
||||
//! ConcreteEnv,
|
||||
//! config::ConfigBuilder,
|
||||
//! Env, EnvInner,
|
||||
//! tables::{Tables, TablesMut},
|
||||
//! DatabaseRo, DatabaseRw, TxRo, TxRw,
|
||||
//! };
|
||||
//!
|
||||
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
//! // Create a configuration for the database environment.
|
||||
//! let db_dir = tempfile::tempdir()?;
|
||||
//! let config = ConfigBuilder::new()
|
||||
//! .db_directory(db_dir.path().to_path_buf())
|
||||
//! .build();
|
||||
//!
|
||||
//! // Initialize the database environment.
|
||||
//! let env = ConcreteEnv::open(config)?;
|
||||
//!
|
||||
//! // Open up a transaction + tables for writing.
|
||||
//! let env_inner = env.env_inner();
|
||||
//! let tx_rw = env_inner.tx_rw()?;
|
||||
//! let mut tables = env_inner.open_tables_mut(&tx_rw)?;
|
||||
//!
|
||||
//! // ⚠️ Write data to the tables directly.
|
||||
//! // (not recommended, use `ops` or `service`).
|
||||
//! const KEY_IMAGE: [u8; 32] = [88; 32];
|
||||
//! tables.key_images_mut().put(&KEY_IMAGE, &())?;
|
||||
//!
|
||||
//! // Commit the data written.
|
||||
//! drop(tables);
|
||||
//! TxRw::commit(tx_rw)?;
|
||||
//!
|
||||
//! // Read the data, assert it is correct.
|
||||
//! let tx_ro = env_inner.tx_ro()?;
|
||||
//! let tables = env_inner.open_tables(&tx_ro)?;
|
||||
//! let (key_image, _) = tables.key_images().first()?;
|
||||
//! assert_eq!(key_image, KEY_IMAGE);
|
||||
//! # Ok(()) }
|
||||
//! ```
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Lints
|
||||
// Forbid lints.
|
||||
// Our code, and code generated (e.g macros) cannot overrule these.
|
||||
#![forbid(
|
||||
// `unsafe` is allowed but it _must_ be
|
||||
// commented with `SAFETY: reason`.
|
||||
clippy::undocumented_unsafe_blocks,
|
||||
|
||||
// Never.
|
||||
unused_unsafe,
|
||||
redundant_semicolons,
|
||||
unused_allocation,
|
||||
coherence_leak_check,
|
||||
while_true,
|
||||
clippy::missing_docs_in_private_items,
|
||||
|
||||
// Maybe can be put into `#[deny]`.
|
||||
unconditional_recursion,
|
||||
for_loops_over_fallibles,
|
||||
unused_braces,
|
||||
unused_labels,
|
||||
keyword_idents,
|
||||
non_ascii_idents,
|
||||
variant_size_differences,
|
||||
single_use_lifetimes,
|
||||
|
||||
// Probably can be put into `#[deny]`.
|
||||
future_incompatible,
|
||||
let_underscore,
|
||||
break_with_label_and_loop,
|
||||
duplicate_macro_attributes,
|
||||
exported_private_dependencies,
|
||||
large_assignments,
|
||||
overlapping_range_endpoints,
|
||||
semicolon_in_expressions_from_macros,
|
||||
noop_method_call,
|
||||
unreachable_pub,
|
||||
)]
|
||||
// Deny lints.
|
||||
// Some of these are `#[allow]`'ed on a per-case basis.
|
||||
#![deny(
|
||||
clippy::all,
|
||||
clippy::correctness,
|
||||
clippy::suspicious,
|
||||
clippy::style,
|
||||
clippy::complexity,
|
||||
clippy::perf,
|
||||
clippy::pedantic,
|
||||
clippy::nursery,
|
||||
clippy::cargo,
|
||||
unused_doc_comments,
|
||||
unused_mut,
|
||||
missing_docs,
|
||||
deprecated,
|
||||
unused_comparisons,
|
||||
nonstandard_style
|
||||
)]
|
||||
#![allow(
|
||||
// FIXME: this lint affects crates outside of
|
||||
// `database/` for some reason, allow for now.
|
||||
clippy::cargo_common_metadata,
|
||||
|
||||
// FIXME: adding `#[must_use]` onto everything
|
||||
// might just be more annoying than useful...
|
||||
// although it is sometimes nice.
|
||||
clippy::must_use_candidate,
|
||||
|
||||
// FIXME: good lint but too many false positives
|
||||
// with our `Env` + `RwLock` setup.
|
||||
clippy::significant_drop_tightening,
|
||||
|
||||
// FIXME: good lint but is less clear in most cases.
|
||||
clippy::items_after_statements,
|
||||
|
||||
clippy::module_name_repetitions,
|
||||
clippy::module_inception,
|
||||
clippy::redundant_pub_crate,
|
||||
clippy::option_if_let_else,
|
||||
)]
|
||||
// Allow some lints when running in debug mode.
|
||||
#![cfg_attr(debug_assertions, allow(clippy::todo, clippy::multiple_crate_versions))]
|
||||
// Allow some lints in tests.
|
||||
#![cfg_attr(
|
||||
test,
|
||||
allow(
|
||||
clippy::cognitive_complexity,
|
||||
clippy::needless_pass_by_value,
|
||||
clippy::cast_possible_truncation,
|
||||
clippy::too_many_lines
|
||||
)
|
||||
)]
|
||||
// Only allow building 64-bit targets.
|
||||
//
|
||||
// This allows us to assume 64-bit
|
||||
// invariants in code, e.g. `usize as u64`.
|
||||
//
|
||||
// # Safety
|
||||
// As of 0d67bfb1bcc431e90c82d577bf36dd1182c807e2 (2024-04-12)
|
||||
// there are invariants relying on 64-bit pointer sizes.
|
||||
#[cfg(not(target_pointer_width = "64"))]
|
||||
compile_error!("Cuprate is only compatible with 64-bit CPUs");
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Public API
|
||||
// Import private modules, export public types.
|
||||
//
|
||||
// Documentation for each module is located in the respective file.
|
||||
|
||||
mod backend;
|
||||
pub use backend::ConcreteEnv;
|
||||
|
||||
pub mod config;
|
||||
|
||||
mod constants;
|
||||
pub use constants::{
|
||||
DATABASE_BACKEND, DATABASE_CORRUPT_MSG, DATABASE_DATA_FILENAME, DATABASE_LOCK_FILENAME,
|
||||
DATABASE_VERSION,
|
||||
};
|
||||
|
||||
mod database;
|
||||
pub use database::{DatabaseIter, DatabaseRo, DatabaseRw};
|
||||
|
||||
mod env;
|
||||
pub use env::{Env, EnvInner};
|
||||
|
||||
mod error;
|
||||
pub use error::{InitError, RuntimeError};
|
||||
|
||||
pub(crate) mod free;
|
||||
|
||||
pub mod resize;
|
||||
|
||||
mod key;
|
||||
pub use key::Key;
|
||||
|
||||
mod storable;
|
||||
pub use storable::{Storable, StorableBytes, StorableVec};
|
||||
|
||||
pub mod ops;
|
||||
|
||||
mod table;
|
||||
pub use table::Table;
|
||||
|
||||
pub mod tables;
|
||||
|
||||
pub mod types;
|
||||
|
||||
mod transaction;
|
||||
pub use transaction::{TxRo, TxRw};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Feature-gated
|
||||
#[cfg(feature = "service")]
|
||||
pub mod service;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Private
|
||||
#[cfg(test)]
|
||||
pub(crate) mod tests;
|
||||
|
||||
#[cfg(feature = "service")] // only needed in `service` for now
|
||||
pub(crate) mod unsafe_sendable;
|
||||
472
storage/database/src/ops/block.rs
Normal file
472
storage/database/src/ops/block.rs
Normal file
|
|
@ -0,0 +1,472 @@
|
|||
//! Blocks functions.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use bytemuck::TransparentWrapper;
|
||||
use monero_serai::block::Block;
|
||||
|
||||
use cuprate_helper::map::{combine_low_high_bits_to_u128, split_u128_into_low_high_bits};
|
||||
use cuprate_types::{ExtendedBlockHeader, VerifiedBlockInformation};
|
||||
|
||||
use crate::{
|
||||
database::{DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
ops::{
|
||||
blockchain::{chain_height, cumulative_generated_coins},
|
||||
macros::doc_error,
|
||||
output::get_rct_num_outputs,
|
||||
tx::{add_tx, remove_tx},
|
||||
},
|
||||
tables::{BlockHeights, BlockInfos, Tables, TablesMut},
|
||||
types::{BlockHash, BlockHeight, BlockInfo},
|
||||
StorableVec,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- `add_block_*`
|
||||
/// Add a [`VerifiedBlockInformation`] to the database.
|
||||
///
|
||||
/// This extracts all the data from the input block and
|
||||
/// maps/adds them to the appropriate database tables.
|
||||
///
|
||||
#[doc = doc_error!()]
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if:
|
||||
/// - `block.height > u32::MAX` (not normally possible)
|
||||
/// - `block.height` is not != [`chain_height`]
|
||||
///
|
||||
/// # Already exists
|
||||
/// This function will operate normally even if `block` already
|
||||
/// exists, i.e., this function will not return `Err` even if you
|
||||
/// call this function infinitely with the same block.
|
||||
// no inline, too big.
|
||||
pub fn add_block(
|
||||
block: &VerifiedBlockInformation,
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<(), RuntimeError> {
|
||||
//------------------------------------------------------ Check preconditions first
|
||||
|
||||
// Cast height to `u32` for storage (handled at top of function).
|
||||
// Panic (should never happen) instead of allowing DB corruption.
|
||||
// <https://github.com/Cuprate/cuprate/pull/102#discussion_r1560020991>
|
||||
assert!(
|
||||
u32::try_from(block.height).is_ok(),
|
||||
"block.height ({}) > u32::MAX",
|
||||
block.height,
|
||||
);
|
||||
|
||||
let chain_height = chain_height(tables.block_heights())?;
|
||||
assert_eq!(
|
||||
block.height, chain_height,
|
||||
"block.height ({}) != chain_height ({})",
|
||||
block.height, chain_height,
|
||||
);
|
||||
|
||||
// Expensive checks - debug only.
|
||||
#[cfg(debug_assertions)]
|
||||
{
|
||||
assert_eq!(block.block.serialize(), block.block_blob);
|
||||
assert_eq!(block.block.txs.len(), block.txs.len());
|
||||
for (i, tx) in block.txs.iter().enumerate() {
|
||||
assert_eq!(tx.tx_blob, tx.tx.serialize());
|
||||
assert_eq!(tx.tx_hash, block.block.txs[i]);
|
||||
}
|
||||
}
|
||||
|
||||
//------------------------------------------------------ Transaction / Outputs / Key Images
|
||||
// Add the miner transaction first.
|
||||
{
|
||||
let tx = &block.block.miner_tx;
|
||||
add_tx(tx, &tx.serialize(), &tx.hash(), &chain_height, tables)?;
|
||||
}
|
||||
|
||||
for tx in &block.txs {
|
||||
add_tx(&tx.tx, &tx.tx_blob, &tx.tx_hash, &chain_height, tables)?;
|
||||
}
|
||||
|
||||
//------------------------------------------------------ Block Info
|
||||
|
||||
// INVARIANT: must be below the above transaction loop since this
|
||||
// RCT output count needs account for _this_ block's outputs.
|
||||
let cumulative_rct_outs = get_rct_num_outputs(tables.rct_outputs())?;
|
||||
|
||||
let cumulative_generated_coins =
|
||||
cumulative_generated_coins(&block.height.saturating_sub(1), tables.block_infos())?
|
||||
+ block.generated_coins;
|
||||
|
||||
let (cumulative_difficulty_low, cumulative_difficulty_high) =
|
||||
split_u128_into_low_high_bits(block.cumulative_difficulty);
|
||||
|
||||
// Block Info.
|
||||
tables.block_infos_mut().put(
|
||||
&block.height,
|
||||
&BlockInfo {
|
||||
cumulative_difficulty_low,
|
||||
cumulative_difficulty_high,
|
||||
cumulative_generated_coins,
|
||||
cumulative_rct_outs,
|
||||
timestamp: block.block.header.timestamp,
|
||||
block_hash: block.block_hash,
|
||||
// INVARIANT: #[cfg] @ lib.rs asserts `usize == u64`
|
||||
weight: block.weight as u64,
|
||||
long_term_weight: block.long_term_weight as u64,
|
||||
},
|
||||
)?;
|
||||
|
||||
// Block blobs.
|
||||
tables
|
||||
.block_blobs_mut()
|
||||
.put(&block.height, StorableVec::wrap_ref(&block.block_blob))?;
|
||||
|
||||
// Block heights.
|
||||
tables
|
||||
.block_heights_mut()
|
||||
.put(&block.block_hash, &block.height)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- `pop_block`
|
||||
/// Remove the top/latest block from the database.
|
||||
///
|
||||
/// The removed block's data is returned.
|
||||
#[doc = doc_error!()]
|
||||
///
|
||||
/// In `pop_block()`'s case, [`RuntimeError::KeyNotFound`]
|
||||
/// will be returned if there are no blocks left.
|
||||
// no inline, too big
|
||||
pub fn pop_block(
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<(BlockHeight, BlockHash, Block), RuntimeError> {
|
||||
//------------------------------------------------------ Block Info
|
||||
// Remove block data from tables.
|
||||
let (block_height, block_hash) = {
|
||||
let (block_height, block_info) = tables.block_infos_mut().pop_last()?;
|
||||
(block_height, block_info.block_hash)
|
||||
};
|
||||
|
||||
// Block heights.
|
||||
tables.block_heights_mut().delete(&block_hash)?;
|
||||
|
||||
// Block blobs.
|
||||
// We deserialize the block blob into a `Block`, such
|
||||
// that we can remove the associated transactions later.
|
||||
let block_blob = tables.block_blobs_mut().take(&block_height)?.0;
|
||||
let block = Block::read(&mut block_blob.as_slice())?;
|
||||
|
||||
//------------------------------------------------------ Transaction / Outputs / Key Images
|
||||
remove_tx(&block.miner_tx.hash(), tables)?;
|
||||
for tx_hash in &block.txs {
|
||||
remove_tx(tx_hash, tables)?;
|
||||
}
|
||||
|
||||
Ok((block_height, block_hash, block))
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- `get_block_extended_header_*`
|
||||
/// Retrieve a [`ExtendedBlockHeader`] from the database.
|
||||
///
|
||||
/// This extracts all the data from the database tables
|
||||
/// needed to create a full `ExtendedBlockHeader`.
|
||||
///
|
||||
/// # Notes
|
||||
/// This is slightly more expensive than [`get_block_extended_header_from_height`]
|
||||
/// (1 more database lookup).
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_block_extended_header(
|
||||
block_hash: &BlockHash,
|
||||
tables: &impl Tables,
|
||||
) -> Result<ExtendedBlockHeader, RuntimeError> {
|
||||
get_block_extended_header_from_height(&tables.block_heights().get(block_hash)?, tables)
|
||||
}
|
||||
|
||||
/// Same as [`get_block_extended_header`] but with a [`BlockHeight`].
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_block_extended_header_from_height(
|
||||
block_height: &BlockHeight,
|
||||
tables: &impl Tables,
|
||||
) -> Result<ExtendedBlockHeader, RuntimeError> {
|
||||
let block_info = tables.block_infos().get(block_height)?;
|
||||
let block_blob = tables.block_blobs().get(block_height)?.0;
|
||||
let block = Block::read(&mut block_blob.as_slice())?;
|
||||
|
||||
let cumulative_difficulty = combine_low_high_bits_to_u128(
|
||||
block_info.cumulative_difficulty_low,
|
||||
block_info.cumulative_difficulty_high,
|
||||
);
|
||||
|
||||
// INVARIANT: #[cfg] @ lib.rs asserts `usize == u64`
|
||||
#[allow(clippy::cast_possible_truncation)]
|
||||
Ok(ExtendedBlockHeader {
|
||||
cumulative_difficulty,
|
||||
version: block.header.major_version,
|
||||
vote: block.header.minor_version,
|
||||
timestamp: block.header.timestamp,
|
||||
block_weight: block_info.weight as usize,
|
||||
long_term_weight: block_info.long_term_weight as usize,
|
||||
})
|
||||
}
|
||||
|
||||
/// Return the top/latest [`ExtendedBlockHeader`] from the database.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_block_extended_header_top(
|
||||
tables: &impl Tables,
|
||||
) -> Result<(ExtendedBlockHeader, BlockHeight), RuntimeError> {
|
||||
let height = chain_height(tables.block_heights())?.saturating_sub(1);
|
||||
let header = get_block_extended_header_from_height(&height, tables)?;
|
||||
Ok((header, height))
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Misc
|
||||
/// Retrieve a [`BlockInfo`] via its [`BlockHeight`].
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_block_info(
|
||||
block_height: &BlockHeight,
|
||||
table_block_infos: &impl DatabaseRo<BlockInfos>,
|
||||
) -> Result<BlockInfo, RuntimeError> {
|
||||
table_block_infos.get(block_height)
|
||||
}
|
||||
|
||||
/// Retrieve a [`BlockHeight`] via its [`BlockHash`].
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_block_height(
|
||||
block_hash: &BlockHash,
|
||||
table_block_heights: &impl DatabaseRo<BlockHeights>,
|
||||
) -> Result<BlockHeight, RuntimeError> {
|
||||
table_block_heights.get(block_hash)
|
||||
}
|
||||
|
||||
/// Check if a block exists in the database.
|
||||
///
|
||||
/// # Errors
|
||||
/// Note that this will never return `Err(RuntimeError::KeyNotFound)`,
|
||||
/// as in that case, `Ok(false)` will be returned.
|
||||
///
|
||||
/// Other errors may still occur.
|
||||
#[inline]
|
||||
pub fn block_exists(
|
||||
block_hash: &BlockHash,
|
||||
table_block_heights: &impl DatabaseRo<BlockHeights>,
|
||||
) -> Result<bool, RuntimeError> {
|
||||
table_block_heights.contains(block_hash)
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
#[allow(
|
||||
clippy::significant_drop_tightening,
|
||||
clippy::cognitive_complexity,
|
||||
clippy::too_many_lines
|
||||
)]
|
||||
mod test {
|
||||
use pretty_assertions::assert_eq;
|
||||
|
||||
use cuprate_test_utils::data::{block_v16_tx0, block_v1_tx2, block_v9_tx3};
|
||||
|
||||
use super::*;
|
||||
use crate::{
|
||||
ops::tx::{get_tx, tx_exists},
|
||||
tests::{assert_all_tables_are_empty, tmp_concrete_env, AssertTableLen},
|
||||
transaction::TxRw,
|
||||
Env, EnvInner,
|
||||
};
|
||||
|
||||
/// Tests all above block functions.
|
||||
///
|
||||
/// Note that this doesn't test the correctness of values added, as the
|
||||
/// functions have a pre-condition that the caller handles this.
|
||||
///
|
||||
/// It simply tests if the proper tables are mutated, and if the data
|
||||
/// stored and retrieved is the same.
|
||||
#[test]
|
||||
fn all_block_functions() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let mut blocks = [
|
||||
block_v1_tx2().clone(),
|
||||
block_v9_tx3().clone(),
|
||||
block_v16_tx0().clone(),
|
||||
];
|
||||
// HACK: `add_block()` asserts blocks with non-sequential heights
|
||||
// cannot be added, to get around this, manually edit the block height.
|
||||
for (height, block) in blocks.iter_mut().enumerate() {
|
||||
block.height = height as u64;
|
||||
assert_eq!(block.block.serialize(), block.block_blob);
|
||||
}
|
||||
let generated_coins_sum = blocks
|
||||
.iter()
|
||||
.map(|block| block.generated_coins)
|
||||
.sum::<u64>();
|
||||
|
||||
// Add blocks.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
for block in &blocks {
|
||||
// println!("add_block: {block:#?}");
|
||||
add_block(block, &mut tables).unwrap();
|
||||
}
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
// Assert all reads are OK.
|
||||
let block_hashes = {
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
|
||||
// Assert only the proper tables were added to.
|
||||
AssertTableLen {
|
||||
block_infos: 3,
|
||||
block_blobs: 3,
|
||||
block_heights: 3,
|
||||
key_images: 69,
|
||||
num_outputs: 41,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 111,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 8,
|
||||
tx_blobs: 8,
|
||||
tx_ids: 8,
|
||||
tx_heights: 8,
|
||||
tx_unlock_time: 3,
|
||||
}
|
||||
.assert(&tables);
|
||||
|
||||
// Check `cumulative` functions work.
|
||||
assert_eq!(
|
||||
cumulative_generated_coins(&2, tables.block_infos()).unwrap(),
|
||||
generated_coins_sum,
|
||||
);
|
||||
|
||||
// Both height and hash should result in getting the same data.
|
||||
let mut block_hashes = vec![];
|
||||
for block in &blocks {
|
||||
println!("blocks.iter(): hash: {}", hex::encode(block.block_hash));
|
||||
|
||||
let height = get_block_height(&block.block_hash, tables.block_heights()).unwrap();
|
||||
|
||||
println!("blocks.iter(): height: {height}");
|
||||
|
||||
assert!(block_exists(&block.block_hash, tables.block_heights()).unwrap());
|
||||
|
||||
let block_header_from_height =
|
||||
get_block_extended_header_from_height(&height, &tables).unwrap();
|
||||
let block_header_from_hash =
|
||||
get_block_extended_header(&block.block_hash, &tables).unwrap();
|
||||
|
||||
// Just an alias, these names are long.
|
||||
let b1 = block_header_from_hash;
|
||||
let b2 = block;
|
||||
assert_eq!(b1, block_header_from_height);
|
||||
assert_eq!(b1.version, b2.block.header.major_version);
|
||||
assert_eq!(b1.vote, b2.block.header.minor_version);
|
||||
assert_eq!(b1.timestamp, b2.block.header.timestamp);
|
||||
assert_eq!(b1.cumulative_difficulty, b2.cumulative_difficulty);
|
||||
assert_eq!(b1.block_weight, b2.weight);
|
||||
assert_eq!(b1.long_term_weight, b2.long_term_weight);
|
||||
|
||||
block_hashes.push(block.block_hash);
|
||||
|
||||
// Assert transaction reads are OK.
|
||||
for (i, tx) in block.txs.iter().enumerate() {
|
||||
println!("tx_hash: {:?}", hex::encode(tx.tx_hash));
|
||||
|
||||
assert!(tx_exists(&tx.tx_hash, tables.tx_ids()).unwrap());
|
||||
|
||||
let tx2 = get_tx(&tx.tx_hash, tables.tx_ids(), tables.tx_blobs()).unwrap();
|
||||
|
||||
assert_eq!(tx.tx_blob, tx2.serialize());
|
||||
assert_eq!(tx.tx_weight, tx2.weight());
|
||||
assert_eq!(tx.tx_hash, block.block.txs[i]);
|
||||
assert_eq!(tx.tx_hash, tx2.hash());
|
||||
}
|
||||
}
|
||||
|
||||
block_hashes
|
||||
};
|
||||
|
||||
{
|
||||
let len = block_hashes.len();
|
||||
let hashes: Vec<String> = block_hashes.iter().map(hex::encode).collect();
|
||||
println!("block_hashes: len: {len}, hashes: {hashes:?}");
|
||||
}
|
||||
|
||||
// Remove the blocks.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
for block_hash in block_hashes.into_iter().rev() {
|
||||
println!("pop_block(): block_hash: {}", hex::encode(block_hash));
|
||||
|
||||
let (_popped_height, popped_hash, _popped_block) = pop_block(&mut tables).unwrap();
|
||||
|
||||
assert_eq!(block_hash, popped_hash);
|
||||
|
||||
assert!(matches!(
|
||||
get_block_extended_header(&block_hash, &tables),
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
));
|
||||
}
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
assert_all_tables_are_empty(&env);
|
||||
}
|
||||
|
||||
/// We should panic if: `block.height` > `u32::MAX`
|
||||
#[test]
|
||||
#[should_panic(expected = "block.height (4294967296) > u32::MAX")]
|
||||
fn block_height_gt_u32_max() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
let mut block = block_v9_tx3().clone();
|
||||
|
||||
block.height = u64::from(u32::MAX) + 1;
|
||||
add_block(&block, &mut tables).unwrap();
|
||||
}
|
||||
|
||||
/// We should panic if: `block.height` != the chain height
|
||||
#[test]
|
||||
#[should_panic(
|
||||
expected = "assertion `left == right` failed: block.height (123) != chain_height (1)\n left: 123\n right: 1"
|
||||
)]
|
||||
fn block_height_not_chain_height() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
let mut block = block_v9_tx3().clone();
|
||||
// HACK: `add_block()` asserts blocks with non-sequential heights
|
||||
// cannot be added, to get around this, manually edit the block height.
|
||||
block.height = 0;
|
||||
|
||||
// OK, `0 == 0`
|
||||
assert_eq!(block.height, 0);
|
||||
add_block(&block, &mut tables).unwrap();
|
||||
|
||||
// FAIL, `123 != 1`
|
||||
block.height = 123;
|
||||
add_block(&block, &mut tables).unwrap();
|
||||
}
|
||||
}
|
||||
182
storage/database/src/ops/blockchain.rs
Normal file
182
storage/database/src/ops/blockchain.rs
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
//! Blockchain functions - chain height, generated coins, etc.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
database::DatabaseRo,
|
||||
error::RuntimeError,
|
||||
ops::macros::doc_error,
|
||||
tables::{BlockHeights, BlockInfos},
|
||||
types::BlockHeight,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Free Functions
|
||||
/// Retrieve the height of the chain.
|
||||
///
|
||||
/// This returns the chain-tip, not the [`top_block_height`].
|
||||
///
|
||||
/// For example:
|
||||
/// - The blockchain has 0 blocks => this returns `0`
|
||||
/// - The blockchain has 1 block (height 0) => this returns `1`
|
||||
/// - The blockchain has 2 blocks (height 1) => this returns `2`
|
||||
///
|
||||
/// So the height of a new block would be `chain_height()`.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn chain_height(
|
||||
table_block_heights: &impl DatabaseRo<BlockHeights>,
|
||||
) -> Result<BlockHeight, RuntimeError> {
|
||||
table_block_heights.len()
|
||||
}
|
||||
|
||||
/// Retrieve the height of the top block.
|
||||
///
|
||||
/// This returns the height of the top block, not the [`chain_height`].
|
||||
///
|
||||
/// For example:
|
||||
/// - The blockchain has 0 blocks => this returns `Err(RuntimeError::KeyNotFound)`
|
||||
/// - The blockchain has 1 block (height 0) => this returns `Ok(0)`
|
||||
/// - The blockchain has 2 blocks (height 1) => this returns `Ok(1)`
|
||||
///
|
||||
/// Note that in cases where no blocks have been written to the
|
||||
/// database yet, an error is returned: `Err(RuntimeError::KeyNotFound)`.
|
||||
///
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn top_block_height(
|
||||
table_block_heights: &impl DatabaseRo<BlockHeights>,
|
||||
) -> Result<BlockHeight, RuntimeError> {
|
||||
match table_block_heights.len()? {
|
||||
0 => Err(RuntimeError::KeyNotFound),
|
||||
height => Ok(height - 1),
|
||||
}
|
||||
}
|
||||
|
||||
/// Check how many cumulative generated coins there have been until a certain [`BlockHeight`].
|
||||
///
|
||||
/// This returns the total amount of Monero generated up to `block_height`
|
||||
/// (including the block itself) in atomic units.
|
||||
///
|
||||
/// For example:
|
||||
/// - on the genesis block `0`, this returns the amount block `0` generated
|
||||
/// - on the next block `1`, this returns the amount block `0` and `1` generated
|
||||
///
|
||||
/// If no blocks have been added and `block_height == 0`
|
||||
/// (i.e., the cumulative generated coins before genesis block is being calculated),
|
||||
/// this returns `Ok(0)`.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn cumulative_generated_coins(
|
||||
block_height: &BlockHeight,
|
||||
table_block_infos: &impl DatabaseRo<BlockInfos>,
|
||||
) -> Result<u64, RuntimeError> {
|
||||
match table_block_infos.get(block_height) {
|
||||
Ok(block_info) => Ok(block_info.cumulative_generated_coins),
|
||||
Err(RuntimeError::KeyNotFound) if block_height == &0 => Ok(0),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use pretty_assertions::assert_eq;
|
||||
|
||||
use cuprate_test_utils::data::{block_v16_tx0, block_v1_tx2, block_v9_tx3};
|
||||
|
||||
use super::*;
|
||||
use crate::{
|
||||
ops::block::add_block,
|
||||
tables::Tables,
|
||||
tests::{assert_all_tables_are_empty, tmp_concrete_env, AssertTableLen},
|
||||
transaction::TxRw,
|
||||
Env, EnvInner,
|
||||
};
|
||||
|
||||
/// Tests all above functions.
|
||||
///
|
||||
/// Note that this doesn't test the correctness of values added, as the
|
||||
/// functions have a pre-condition that the caller handles this.
|
||||
///
|
||||
/// It simply tests if the proper tables are mutated, and if the data
|
||||
/// stored and retrieved is the same.
|
||||
#[test]
|
||||
fn all_blockchain_functions() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let mut blocks = [
|
||||
block_v1_tx2().clone(),
|
||||
block_v9_tx3().clone(),
|
||||
block_v16_tx0().clone(),
|
||||
];
|
||||
let blocks_len = u64::try_from(blocks.len()).unwrap();
|
||||
|
||||
// Add blocks.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
assert!(matches!(
|
||||
top_block_height(tables.block_heights()),
|
||||
Err(RuntimeError::KeyNotFound),
|
||||
));
|
||||
assert_eq!(
|
||||
0,
|
||||
cumulative_generated_coins(&0, tables.block_infos()).unwrap()
|
||||
);
|
||||
|
||||
for (i, block) in blocks.iter_mut().enumerate() {
|
||||
let i = u64::try_from(i).unwrap();
|
||||
// HACK: `add_block()` asserts blocks with non-sequential heights
|
||||
// cannot be added, to get around this, manually edit the block height.
|
||||
block.height = i;
|
||||
add_block(block, &mut tables).unwrap();
|
||||
}
|
||||
|
||||
// Assert reads are correct.
|
||||
AssertTableLen {
|
||||
block_infos: 3,
|
||||
block_blobs: 3,
|
||||
block_heights: 3,
|
||||
key_images: 69,
|
||||
num_outputs: 41,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 111,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 8,
|
||||
tx_blobs: 8,
|
||||
tx_ids: 8,
|
||||
tx_heights: 8,
|
||||
tx_unlock_time: 3,
|
||||
}
|
||||
.assert(&tables);
|
||||
|
||||
assert_eq!(blocks_len, chain_height(tables.block_heights()).unwrap());
|
||||
assert_eq!(
|
||||
blocks_len - 1,
|
||||
top_block_height(tables.block_heights()).unwrap()
|
||||
);
|
||||
assert_eq!(
|
||||
cumulative_generated_coins(&0, tables.block_infos()).unwrap(),
|
||||
14_535_350_982_449,
|
||||
);
|
||||
assert_eq!(
|
||||
cumulative_generated_coins(&1, tables.block_infos()).unwrap(),
|
||||
17_939_125_004_612,
|
||||
);
|
||||
assert_eq!(
|
||||
cumulative_generated_coins(&2, tables.block_infos()).unwrap(),
|
||||
18_539_125_004_612,
|
||||
);
|
||||
assert!(matches!(
|
||||
cumulative_generated_coins(&3, tables.block_infos()),
|
||||
Err(RuntimeError::KeyNotFound),
|
||||
));
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
}
|
||||
}
|
||||
127
storage/database/src/ops/key_image.rs
Normal file
127
storage/database/src/ops/key_image.rs
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
//! Key image functions.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use crate::{
|
||||
database::{DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
ops::macros::{doc_add_block_inner_invariant, doc_error},
|
||||
tables::KeyImages,
|
||||
types::KeyImage,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Key image functions
|
||||
/// Add a [`KeyImage`] to the "spent" set in the database.
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn add_key_image(
|
||||
key_image: &KeyImage,
|
||||
table_key_images: &mut impl DatabaseRw<KeyImages>,
|
||||
) -> Result<(), RuntimeError> {
|
||||
table_key_images.put(key_image, &())
|
||||
}
|
||||
|
||||
/// Remove a [`KeyImage`] from the "spent" set in the database.
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn remove_key_image(
|
||||
key_image: &KeyImage,
|
||||
table_key_images: &mut impl DatabaseRw<KeyImages>,
|
||||
) -> Result<(), RuntimeError> {
|
||||
table_key_images.delete(key_image)
|
||||
}
|
||||
|
||||
/// Check if a [`KeyImage`] exists - i.e. if it is "spent".
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn key_image_exists(
|
||||
key_image: &KeyImage,
|
||||
table_key_images: &impl DatabaseRo<KeyImages>,
|
||||
) -> Result<bool, RuntimeError> {
|
||||
table_key_images.contains(key_image)
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use hex_literal::hex;
|
||||
|
||||
use super::*;
|
||||
use crate::{
|
||||
tables::{Tables, TablesMut},
|
||||
tests::{assert_all_tables_are_empty, tmp_concrete_env, AssertTableLen},
|
||||
transaction::TxRw,
|
||||
Env, EnvInner,
|
||||
};
|
||||
|
||||
/// Tests all above key-image functions.
|
||||
///
|
||||
/// Note that this doesn't test the correctness of values added, as the
|
||||
/// functions have a pre-condition that the caller handles this.
|
||||
///
|
||||
/// It simply tests if the proper tables are mutated, and if the data
|
||||
/// stored and retrieved is the same.
|
||||
#[test]
|
||||
fn all_key_image_functions() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let key_images = [
|
||||
hex!("be1c87fc8f958f68fbe346a18dfb314204dca7573f61aae14840b8037da5c286"),
|
||||
hex!("c5e4a592c11f34a12e13516ab2883b7c580d47b286b8fe8b15d57d2a18ade275"),
|
||||
hex!("93288b646f858edfb0997ae08d7c76f4599b04c127f108e8e69a0696ae7ba334"),
|
||||
hex!("726e9e3d8f826d24811183f94ff53aeba766c9efe6274eb80806f69b06bfa3fc"),
|
||||
];
|
||||
|
||||
// Add.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
for key_image in &key_images {
|
||||
println!("add_key_image(): {}", hex::encode(key_image));
|
||||
add_key_image(key_image, tables.key_images_mut()).unwrap();
|
||||
}
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
// Assert all reads are OK.
|
||||
{
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
|
||||
// Assert only the proper tables were added to.
|
||||
AssertTableLen {
|
||||
key_images: tables.key_images().len().unwrap(),
|
||||
..Default::default()
|
||||
}
|
||||
.assert(&tables);
|
||||
|
||||
for key_image in &key_images {
|
||||
println!("key_image_exists(): {}", hex::encode(key_image));
|
||||
key_image_exists(key_image, tables.key_images()).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
// Remove.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
for key_image in key_images {
|
||||
println!("remove_key_image(): {}", hex::encode(key_image));
|
||||
remove_key_image(&key_image, tables.key_images_mut()).unwrap();
|
||||
assert!(!key_image_exists(&key_image, tables.key_images()).unwrap());
|
||||
}
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
assert_all_tables_are_empty(&env);
|
||||
}
|
||||
}
|
||||
33
storage/database/src/ops/macros.rs
Normal file
33
storage/database/src/ops/macros.rs
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
//! Macros.
|
||||
//!
|
||||
//! These generate repetitive documentation
|
||||
//! for all the functions defined in `ops/`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Documentation macros
|
||||
/// Generate documentation for the required `# Error` section.
|
||||
macro_rules! doc_error {
|
||||
() => {
|
||||
r#"# Errors
|
||||
This function returns [`RuntimeError::KeyNotFound`] if the input (if applicable) doesn't exist or other `RuntimeError`'s on database errors."#
|
||||
};
|
||||
}
|
||||
pub(super) use doc_error;
|
||||
|
||||
/// Generate `# Invariant` documentation for internal `fn`'s
|
||||
/// that should be called directly with caution.
|
||||
macro_rules! doc_add_block_inner_invariant {
|
||||
() => {
|
||||
r#"# ⚠️ Invariant ⚠️
|
||||
This function mainly exists to be used internally by the parent function [`crate::ops::block::add_block`].
|
||||
|
||||
`add_block()` makes sure all data related to the input is mutated, while
|
||||
this function _does not_, it specifically mutates _particular_ tables.
|
||||
|
||||
This is usually undesired - although this function is still available to call directly.
|
||||
|
||||
When calling this function, ensure that either:
|
||||
1. This effect (incomplete database mutation) is what is desired, or that...
|
||||
2. ...the other tables will also be mutated to a correct state"#
|
||||
};
|
||||
}
|
||||
pub(super) use doc_add_block_inner_invariant;
|
||||
110
storage/database/src/ops/mod.rs
Normal file
110
storage/database/src/ops/mod.rs
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
//! Abstracted Monero database operations.
|
||||
//!
|
||||
//! This module contains many free functions that use the
|
||||
//! traits in this crate to generically call Monero-related
|
||||
//! database operations.
|
||||
//!
|
||||
//! # `impl Table`
|
||||
//! `ops/` functions take [`Tables`](crate::tables::Tables) and
|
||||
//! [`TablesMut`](crate::tables::TablesMut) directly - these are
|
||||
//! _already opened_ database tables.
|
||||
//!
|
||||
//! As such, the function puts the responsibility
|
||||
//! of transactions, tables, etc on the caller.
|
||||
//!
|
||||
//! This does mean these functions are mostly as lean
|
||||
//! as possible, so calling them in a loop should be okay.
|
||||
//!
|
||||
//! # Atomicity
|
||||
//! As transactions are handled by the _caller_ of these functions,
|
||||
//! it is up to the caller to decide what happens if one them return
|
||||
//! an error.
|
||||
//!
|
||||
//! To maintain atomicity, transactions should be [`abort`](crate::transaction::TxRw::abort)ed
|
||||
//! if one of the functions failed.
|
||||
//!
|
||||
//! For example, if [`add_block()`](block::add_block) is called and returns an [`Err`],
|
||||
//! `abort`ing the transaction that opened the input `TableMut` would reverse all tables
|
||||
//! mutated by `add_block()` up until the error, leaving it in the state it was in before
|
||||
//! `add_block()` was called.
|
||||
//!
|
||||
//! # Sub-functions
|
||||
//! The main functions within this module are mostly within the [`block`] module.
|
||||
//!
|
||||
//! Practically speaking, you should only be using 2 functions for mutation:
|
||||
//! - [`add_block`](block::add_block)
|
||||
//! - [`pop_block`](block::pop_block)
|
||||
//!
|
||||
//! The `block` functions are "parent" functions, calling other
|
||||
//! sub-functions such as [`add_output()`](output::add_output).
|
||||
//!
|
||||
//! `add_output()` itself only modifies output-related tables, while the `block` "parent"
|
||||
//! functions (like `add_block` and `pop_block`) modify all tables required.
|
||||
//!
|
||||
//! `add_block()` makes sure all data related to the input is mutated, while
|
||||
//! this sub-function _do not_, it specifically mutates _particular_ tables.
|
||||
//!
|
||||
//! When calling this sub-functions, ensure that either:
|
||||
//! 1. This effect (incomplete database mutation) is what is desired, or that...
|
||||
//! 2. ...the other tables will also be mutated to a correct state
|
||||
//!
|
||||
//! # Example
|
||||
//! Simple usage of `ops`.
|
||||
//!
|
||||
//! ```rust
|
||||
//! use hex_literal::hex;
|
||||
//!
|
||||
//! use cuprate_test_utils::data::block_v16_tx0;
|
||||
//!
|
||||
//! use cuprate_database::{
|
||||
//! ConcreteEnv,
|
||||
//! config::ConfigBuilder,
|
||||
//! Env, EnvInner,
|
||||
//! tables::{Tables, TablesMut},
|
||||
//! DatabaseRo, DatabaseRw, TxRo, TxRw,
|
||||
//! ops::block::{add_block, pop_block},
|
||||
//! };
|
||||
//!
|
||||
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
//! // Create a configuration for the database environment.
|
||||
//! let db_dir = tempfile::tempdir()?;
|
||||
//! let config = ConfigBuilder::new()
|
||||
//! .db_directory(db_dir.path().to_path_buf())
|
||||
//! .build();
|
||||
//!
|
||||
//! // Initialize the database environment.
|
||||
//! let env = ConcreteEnv::open(config)?;
|
||||
//!
|
||||
//! // Open up a transaction + tables for writing.
|
||||
//! let env_inner = env.env_inner();
|
||||
//! let tx_rw = env_inner.tx_rw()?;
|
||||
//! let mut tables = env_inner.open_tables_mut(&tx_rw)?;
|
||||
//!
|
||||
//! // Write a block to the database.
|
||||
//! let mut block = block_v16_tx0().clone();
|
||||
//! # block.height = 0;
|
||||
//! add_block(&block, &mut tables)?;
|
||||
//!
|
||||
//! // Commit the data written.
|
||||
//! drop(tables);
|
||||
//! TxRw::commit(tx_rw)?;
|
||||
//!
|
||||
//! // Read the data, assert it is correct.
|
||||
//! let tx_rw = env_inner.tx_rw()?;
|
||||
//! let mut tables = env_inner.open_tables_mut(&tx_rw)?;
|
||||
//! let (height, hash, serai_block) = pop_block(&mut tables)?;
|
||||
//!
|
||||
//! assert_eq!(height, 0);
|
||||
//! assert_eq!(serai_block, block.block);
|
||||
//! assert_eq!(hash, hex!("43bd1f2b6556dcafa413d8372974af59e4e8f37dbf74dc6b2a9b7212d0577428"));
|
||||
//! # Ok(()) }
|
||||
//! ```
|
||||
|
||||
pub mod block;
|
||||
pub mod blockchain;
|
||||
pub mod key_image;
|
||||
pub mod output;
|
||||
pub mod property;
|
||||
pub mod tx;
|
||||
|
||||
mod macros;
|
||||
371
storage/database/src/ops/output.rs
Normal file
371
storage/database/src/ops/output.rs
Normal file
|
|
@ -0,0 +1,371 @@
|
|||
//! Output functions.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use curve25519_dalek::{constants::ED25519_BASEPOINT_POINT, edwards::CompressedEdwardsY, Scalar};
|
||||
use monero_serai::{transaction::Timelock, H};
|
||||
|
||||
use cuprate_helper::map::u64_to_timelock;
|
||||
use cuprate_types::OutputOnChain;
|
||||
|
||||
use crate::{
|
||||
database::{DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
ops::macros::{doc_add_block_inner_invariant, doc_error},
|
||||
tables::{Outputs, RctOutputs, Tables, TablesMut, TxUnlockTime},
|
||||
types::{Amount, AmountIndex, Output, OutputFlags, PreRctOutputId, RctOutput},
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Pre-RCT Outputs
|
||||
/// Add a Pre-RCT [`Output`] to the database.
|
||||
///
|
||||
/// Upon [`Ok`], this function returns the [`PreRctOutputId`] that
|
||||
/// can be used to lookup the `Output` in [`get_output()`].
|
||||
///
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn add_output(
|
||||
amount: Amount,
|
||||
output: &Output,
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<PreRctOutputId, RuntimeError> {
|
||||
// FIXME: this would be much better expressed with a
|
||||
// `btree_map::Entry`-like API, fix `trait DatabaseRw`.
|
||||
let num_outputs = match tables.num_outputs().get(&amount) {
|
||||
// Entry with `amount` already exists.
|
||||
Ok(num_outputs) => num_outputs,
|
||||
// Entry with `amount` didn't exist, this is
|
||||
// the 1st output with this amount.
|
||||
Err(RuntimeError::KeyNotFound) => 0,
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
// Update the amount of outputs.
|
||||
tables.num_outputs_mut().put(&amount, &(num_outputs + 1))?;
|
||||
|
||||
let pre_rct_output_id = PreRctOutputId {
|
||||
amount,
|
||||
// The new `amount_index` is the length of amount of outputs with same amount.
|
||||
amount_index: num_outputs,
|
||||
};
|
||||
|
||||
tables.outputs_mut().put(&pre_rct_output_id, output)?;
|
||||
Ok(pre_rct_output_id)
|
||||
}
|
||||
|
||||
/// Remove a Pre-RCT [`Output`] from the database.
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn remove_output(
|
||||
pre_rct_output_id: &PreRctOutputId,
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<(), RuntimeError> {
|
||||
// Decrement the amount index by 1, or delete the entry out-right.
|
||||
// FIXME: this would be much better expressed with a
|
||||
// `btree_map::Entry`-like API, fix `trait DatabaseRw`.
|
||||
tables
|
||||
.num_outputs_mut()
|
||||
.update(&pre_rct_output_id.amount, |num_outputs| {
|
||||
// INVARIANT: Should never be 0.
|
||||
if num_outputs == 1 {
|
||||
None
|
||||
} else {
|
||||
Some(num_outputs - 1)
|
||||
}
|
||||
})?;
|
||||
|
||||
// Delete the output data itself.
|
||||
tables.outputs_mut().delete(pre_rct_output_id)
|
||||
}
|
||||
|
||||
/// Retrieve a Pre-RCT [`Output`] from the database.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_output(
|
||||
pre_rct_output_id: &PreRctOutputId,
|
||||
table_outputs: &impl DatabaseRo<Outputs>,
|
||||
) -> Result<Output, RuntimeError> {
|
||||
table_outputs.get(pre_rct_output_id)
|
||||
}
|
||||
|
||||
/// How many pre-RCT [`Output`]s are there?
|
||||
///
|
||||
/// This returns the amount of pre-RCT outputs currently stored.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_num_outputs(table_outputs: &impl DatabaseRo<Outputs>) -> Result<u64, RuntimeError> {
|
||||
table_outputs.len()
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- RCT Outputs
|
||||
/// Add an [`RctOutput`] to the database.
|
||||
///
|
||||
/// Upon [`Ok`], this function returns the [`AmountIndex`] that
|
||||
/// can be used to lookup the `RctOutput` in [`get_rct_output()`].
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn add_rct_output(
|
||||
rct_output: &RctOutput,
|
||||
table_rct_outputs: &mut impl DatabaseRw<RctOutputs>,
|
||||
) -> Result<AmountIndex, RuntimeError> {
|
||||
let amount_index = get_rct_num_outputs(table_rct_outputs)?;
|
||||
table_rct_outputs.put(&amount_index, rct_output)?;
|
||||
Ok(amount_index)
|
||||
}
|
||||
|
||||
/// Remove an [`RctOutput`] from the database.
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn remove_rct_output(
|
||||
amount_index: &AmountIndex,
|
||||
table_rct_outputs: &mut impl DatabaseRw<RctOutputs>,
|
||||
) -> Result<(), RuntimeError> {
|
||||
table_rct_outputs.delete(amount_index)
|
||||
}
|
||||
|
||||
/// Retrieve an [`RctOutput`] from the database.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_rct_output(
|
||||
amount_index: &AmountIndex,
|
||||
table_rct_outputs: &impl DatabaseRo<RctOutputs>,
|
||||
) -> Result<RctOutput, RuntimeError> {
|
||||
table_rct_outputs.get(amount_index)
|
||||
}
|
||||
|
||||
/// How many [`RctOutput`]s are there?
|
||||
///
|
||||
/// This returns the amount of RCT outputs currently stored.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_rct_num_outputs(
|
||||
table_rct_outputs: &impl DatabaseRo<RctOutputs>,
|
||||
) -> Result<u64, RuntimeError> {
|
||||
table_rct_outputs.len()
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Mapping functions
|
||||
/// Map an [`Output`] to a [`cuprate_types::OutputOnChain`].
|
||||
#[doc = doc_error!()]
|
||||
pub fn output_to_output_on_chain(
|
||||
output: &Output,
|
||||
amount: Amount,
|
||||
table_tx_unlock_time: &impl DatabaseRo<TxUnlockTime>,
|
||||
) -> Result<OutputOnChain, RuntimeError> {
|
||||
// FIXME: implement lookup table for common values:
|
||||
// <https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/ringct/rctOps.cpp#L322>
|
||||
let commitment = ED25519_BASEPOINT_POINT + H() * Scalar::from(amount);
|
||||
|
||||
let time_lock = if output
|
||||
.output_flags
|
||||
.contains(OutputFlags::NON_ZERO_UNLOCK_TIME)
|
||||
{
|
||||
u64_to_timelock(table_tx_unlock_time.get(&output.tx_idx)?)
|
||||
} else {
|
||||
Timelock::None
|
||||
};
|
||||
|
||||
let key = CompressedEdwardsY::from_slice(&output.key)
|
||||
.map(|y| y.decompress())
|
||||
.unwrap_or(None);
|
||||
|
||||
Ok(OutputOnChain {
|
||||
height: u64::from(output.height),
|
||||
time_lock,
|
||||
key,
|
||||
commitment,
|
||||
})
|
||||
}
|
||||
|
||||
/// Map an [`RctOutput`] to a [`cuprate_types::OutputOnChain`].
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if `rct_output`'s `commitment` fails to decompress
|
||||
/// into a valid [`EdwardsPoint`](curve25519_dalek::edwards::EdwardsPoint).
|
||||
///
|
||||
/// This should normally not happen as commitments that
|
||||
/// are stored in the database should always be valid.
|
||||
#[doc = doc_error!()]
|
||||
pub fn rct_output_to_output_on_chain(
|
||||
rct_output: &RctOutput,
|
||||
table_tx_unlock_time: &impl DatabaseRo<TxUnlockTime>,
|
||||
) -> Result<OutputOnChain, RuntimeError> {
|
||||
// INVARIANT: Commitments stored are valid when stored by the database.
|
||||
let commitment = CompressedEdwardsY::from_slice(&rct_output.commitment)
|
||||
.unwrap()
|
||||
.decompress()
|
||||
.unwrap();
|
||||
|
||||
let time_lock = if rct_output
|
||||
.output_flags
|
||||
.contains(OutputFlags::NON_ZERO_UNLOCK_TIME)
|
||||
{
|
||||
u64_to_timelock(table_tx_unlock_time.get(&rct_output.tx_idx)?)
|
||||
} else {
|
||||
Timelock::None
|
||||
};
|
||||
|
||||
let key = CompressedEdwardsY::from_slice(&rct_output.key)
|
||||
.map(|y| y.decompress())
|
||||
.unwrap_or(None);
|
||||
|
||||
Ok(OutputOnChain {
|
||||
height: u64::from(rct_output.height),
|
||||
time_lock,
|
||||
key,
|
||||
commitment,
|
||||
})
|
||||
}
|
||||
|
||||
/// Map an [`PreRctOutputId`] to an [`OutputOnChain`].
|
||||
///
|
||||
/// Note that this still support RCT outputs, in that case, [`PreRctOutputId::amount`] should be `0`.
|
||||
#[doc = doc_error!()]
|
||||
pub fn id_to_output_on_chain(
|
||||
id: &PreRctOutputId,
|
||||
tables: &impl Tables,
|
||||
) -> Result<OutputOnChain, RuntimeError> {
|
||||
// v2 transactions.
|
||||
if id.amount == 0 {
|
||||
let rct_output = get_rct_output(&id.amount_index, tables.rct_outputs())?;
|
||||
let output_on_chain = rct_output_to_output_on_chain(&rct_output, tables.tx_unlock_time())?;
|
||||
|
||||
Ok(output_on_chain)
|
||||
} else {
|
||||
// v1 transactions.
|
||||
let output = get_output(id, tables.outputs())?;
|
||||
let output_on_chain =
|
||||
output_to_output_on_chain(&output, id.amount, tables.tx_unlock_time())?;
|
||||
|
||||
Ok(output_on_chain)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use super::*;
|
||||
use crate::{
|
||||
tables::{Tables, TablesMut},
|
||||
tests::{assert_all_tables_are_empty, tmp_concrete_env, AssertTableLen},
|
||||
types::OutputFlags,
|
||||
Env, EnvInner,
|
||||
};
|
||||
|
||||
use pretty_assertions::assert_eq;
|
||||
|
||||
/// Dummy `Output`.
|
||||
const OUTPUT: Output = Output {
|
||||
key: [44; 32],
|
||||
height: 0,
|
||||
output_flags: OutputFlags::NON_ZERO_UNLOCK_TIME,
|
||||
tx_idx: 0,
|
||||
};
|
||||
|
||||
/// Dummy `RctOutput`.
|
||||
const RCT_OUTPUT: RctOutput = RctOutput {
|
||||
key: [88; 32],
|
||||
height: 1,
|
||||
output_flags: OutputFlags::empty(),
|
||||
tx_idx: 1,
|
||||
commitment: [100; 32],
|
||||
};
|
||||
|
||||
/// Dummy `Amount`
|
||||
const AMOUNT: Amount = 22;
|
||||
|
||||
/// Tests all above output functions when only inputting `Output` data (no Block).
|
||||
///
|
||||
/// Note that this doesn't test the correctness of values added, as the
|
||||
/// functions have a pre-condition that the caller handles this.
|
||||
///
|
||||
/// It simply tests if the proper tables are mutated, and if the data
|
||||
/// stored and retrieved is the same.
|
||||
#[test]
|
||||
fn all_output_functions() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
// Assert length is correct.
|
||||
assert_eq!(get_num_outputs(tables.outputs()).unwrap(), 0);
|
||||
assert_eq!(get_rct_num_outputs(tables.rct_outputs()).unwrap(), 0);
|
||||
|
||||
// Add outputs.
|
||||
let pre_rct_output_id = add_output(AMOUNT, &OUTPUT, &mut tables).unwrap();
|
||||
let amount_index = add_rct_output(&RCT_OUTPUT, tables.rct_outputs_mut()).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
pre_rct_output_id,
|
||||
PreRctOutputId {
|
||||
amount: AMOUNT,
|
||||
amount_index: 0,
|
||||
}
|
||||
);
|
||||
|
||||
// Assert all reads of the outputs are OK.
|
||||
{
|
||||
// Assert proper tables were added to.
|
||||
AssertTableLen {
|
||||
block_infos: 0,
|
||||
block_blobs: 0,
|
||||
block_heights: 0,
|
||||
key_images: 0,
|
||||
num_outputs: 1,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 1,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 1,
|
||||
tx_blobs: 0,
|
||||
tx_ids: 0,
|
||||
tx_heights: 0,
|
||||
tx_unlock_time: 0,
|
||||
}
|
||||
.assert(&tables);
|
||||
|
||||
// Assert length is correct.
|
||||
assert_eq!(get_num_outputs(tables.outputs()).unwrap(), 1);
|
||||
assert_eq!(get_rct_num_outputs(tables.rct_outputs()).unwrap(), 1);
|
||||
assert_eq!(1, tables.num_outputs().get(&AMOUNT).unwrap());
|
||||
|
||||
// Assert value is save after retrieval.
|
||||
assert_eq!(
|
||||
OUTPUT,
|
||||
get_output(&pre_rct_output_id, tables.outputs()).unwrap(),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
RCT_OUTPUT,
|
||||
get_rct_output(&amount_index, tables.rct_outputs()).unwrap(),
|
||||
);
|
||||
}
|
||||
|
||||
// Remove the outputs.
|
||||
{
|
||||
remove_output(&pre_rct_output_id, &mut tables).unwrap();
|
||||
remove_rct_output(&amount_index, tables.rct_outputs_mut()).unwrap();
|
||||
|
||||
// Assert value no longer exists.
|
||||
assert!(matches!(
|
||||
get_output(&pre_rct_output_id, tables.outputs()),
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
));
|
||||
assert!(matches!(
|
||||
get_rct_output(&amount_index, tables.rct_outputs()),
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
));
|
||||
|
||||
// Assert length is correct.
|
||||
assert_eq!(get_num_outputs(tables.outputs()).unwrap(), 0);
|
||||
assert_eq!(get_rct_num_outputs(tables.rct_outputs()).unwrap(), 0);
|
||||
}
|
||||
|
||||
assert_all_tables_are_empty(&env);
|
||||
}
|
||||
}
|
||||
39
storage/database/src/ops/property.rs
Normal file
39
storage/database/src/ops/property.rs
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
//! Database properties functions - version, pruning, etc.
|
||||
//!
|
||||
//! SOMEDAY: the database `properties` table is not yet implemented.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use monero_pruning::PruningSeed;
|
||||
|
||||
use crate::{error::RuntimeError, ops::macros::doc_error};
|
||||
//---------------------------------------------------------------------------------------------------- Free Functions
|
||||
/// SOMEDAY
|
||||
///
|
||||
#[doc = doc_error!()]
|
||||
///
|
||||
/// # Example
|
||||
/// ```rust
|
||||
/// # use cuprate_database::{*, tables::*, ops::block::*, ops::tx::*};
|
||||
/// // SOMEDAY
|
||||
/// ```
|
||||
#[inline]
|
||||
pub const fn get_blockchain_pruning_seed() -> Result<PruningSeed, RuntimeError> {
|
||||
// SOMEDAY: impl pruning.
|
||||
// We need a DB properties table.
|
||||
Ok(PruningSeed::NotPruned)
|
||||
}
|
||||
|
||||
/// SOMEDAY
|
||||
///
|
||||
#[doc = doc_error!()]
|
||||
///
|
||||
/// # Example
|
||||
/// ```rust
|
||||
/// # use cuprate_database::{*, tables::*, ops::block::*, ops::tx::*};
|
||||
/// // SOMEDAY
|
||||
/// ```
|
||||
#[inline]
|
||||
pub const fn db_version() -> Result<u64, RuntimeError> {
|
||||
// SOMEDAY: We need a DB properties table.
|
||||
Ok(crate::constants::DATABASE_VERSION)
|
||||
}
|
||||
434
storage/database/src/ops/tx.rs
Normal file
434
storage/database/src/ops/tx.rs
Normal file
|
|
@ -0,0 +1,434 @@
|
|||
//! Transaction functions.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use bytemuck::TransparentWrapper;
|
||||
use curve25519_dalek::{constants::ED25519_BASEPOINT_POINT, Scalar};
|
||||
use monero_serai::transaction::{Input, Timelock, Transaction};
|
||||
|
||||
use crate::{
|
||||
database::{DatabaseRo, DatabaseRw},
|
||||
error::RuntimeError,
|
||||
ops::{
|
||||
key_image::{add_key_image, remove_key_image},
|
||||
macros::{doc_add_block_inner_invariant, doc_error},
|
||||
output::{
|
||||
add_output, add_rct_output, get_rct_num_outputs, remove_output, remove_rct_output,
|
||||
},
|
||||
},
|
||||
tables::{TablesMut, TxBlobs, TxIds},
|
||||
types::{BlockHeight, Output, OutputFlags, PreRctOutputId, RctOutput, TxHash, TxId},
|
||||
StorableVec,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Private
|
||||
/// Add a [`Transaction`] (and related data) to the database.
|
||||
///
|
||||
/// The `block_height` is the block that this `tx` belongs to.
|
||||
///
|
||||
/// Note that the caller's input is trusted implicitly and no checks
|
||||
/// are done (in this function) whether the `block_height` is correct or not.
|
||||
///
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
///
|
||||
/// # Notes
|
||||
/// This function is different from other sub-functions and slightly more similar to
|
||||
/// [`add_block()`](crate::ops::block::add_block) in that it calls other sub-functions.
|
||||
///
|
||||
/// This function calls:
|
||||
/// - [`add_output()`]
|
||||
/// - [`add_rct_output()`]
|
||||
/// - [`add_key_image()`]
|
||||
///
|
||||
/// Thus, after [`add_tx`], those values (outputs and key images)
|
||||
/// will be added to database tables as well.
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if:
|
||||
/// - `block.height > u32::MAX` (not normally possible)
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn add_tx(
|
||||
tx: &Transaction,
|
||||
tx_blob: &Vec<u8>,
|
||||
tx_hash: &TxHash,
|
||||
block_height: &BlockHeight,
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<TxId, RuntimeError> {
|
||||
let tx_id = get_num_tx(tables.tx_ids_mut())?;
|
||||
|
||||
//------------------------------------------------------ Transaction data
|
||||
tables.tx_ids_mut().put(tx_hash, &tx_id)?;
|
||||
tables.tx_heights_mut().put(&tx_id, block_height)?;
|
||||
tables
|
||||
.tx_blobs_mut()
|
||||
.put(&tx_id, StorableVec::wrap_ref(tx_blob))?;
|
||||
|
||||
//------------------------------------------------------ Timelocks
|
||||
// Height/time is not differentiated via type, but rather:
|
||||
// "height is any value less than 500_000_000 and timestamp is any value above"
|
||||
// so the `u64/usize` is stored without any tag.
|
||||
//
|
||||
// <https://github.com/Cuprate/cuprate/pull/102#discussion_r1558504285>
|
||||
match tx.prefix.timelock {
|
||||
Timelock::None => (),
|
||||
Timelock::Block(height) => tables.tx_unlock_time_mut().put(&tx_id, &(height as u64))?,
|
||||
Timelock::Time(time) => tables.tx_unlock_time_mut().put(&tx_id, &time)?,
|
||||
}
|
||||
|
||||
//------------------------------------------------------ Pruning
|
||||
// SOMEDAY: implement pruning after `monero-serai` does.
|
||||
// if let PruningSeed::Pruned(decompressed_pruning_seed) = get_blockchain_pruning_seed()? {
|
||||
// SOMEDAY: what to store here? which table?
|
||||
// }
|
||||
|
||||
//------------------------------------------------------
|
||||
let Ok(height) = u32::try_from(*block_height) else {
|
||||
panic!("add_tx(): block_height ({block_height}) > u32::MAX");
|
||||
};
|
||||
|
||||
//------------------------------------------------------ Key Images
|
||||
// Is this a miner transaction?
|
||||
// Which table we add the output data to depends on this.
|
||||
// <https://github.com/monero-project/monero/blob/eac1b86bb2818ac552457380c9dd421fb8935e5b/src/blockchain_db/blockchain_db.cpp#L212-L216>
|
||||
let mut miner_tx = false;
|
||||
|
||||
// Key images.
|
||||
for inputs in &tx.prefix.inputs {
|
||||
match inputs {
|
||||
// Key images.
|
||||
Input::ToKey { key_image, .. } => {
|
||||
add_key_image(key_image.compress().as_bytes(), tables.key_images_mut())?;
|
||||
}
|
||||
// This is a miner transaction, set it for later use.
|
||||
Input::Gen(_) => miner_tx = true,
|
||||
}
|
||||
}
|
||||
|
||||
//------------------------------------------------------ Outputs
|
||||
// Output bit flags.
|
||||
// Set to a non-zero bit value if the unlock time is non-zero.
|
||||
let output_flags = match tx.prefix.timelock {
|
||||
Timelock::None => OutputFlags::empty(),
|
||||
Timelock::Block(_) | Timelock::Time(_) => OutputFlags::NON_ZERO_UNLOCK_TIME,
|
||||
};
|
||||
|
||||
let mut amount_indices = Vec::with_capacity(tx.prefix.outputs.len());
|
||||
|
||||
for (i, output) in tx.prefix.outputs.iter().enumerate() {
|
||||
let key = *output.key.as_bytes();
|
||||
|
||||
// Outputs with clear amounts.
|
||||
let amount_index = if let Some(amount) = output.amount {
|
||||
// RingCT (v2 transaction) miner outputs.
|
||||
if miner_tx && tx.prefix.version == 2 {
|
||||
// Create commitment.
|
||||
// <https://github.com/Cuprate/cuprate/pull/102#discussion_r1559489302>
|
||||
// FIXME: implement lookup table for common values:
|
||||
// <https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/ringct/rctOps.cpp#L322>
|
||||
let commitment = (ED25519_BASEPOINT_POINT
|
||||
+ monero_serai::H() * Scalar::from(amount))
|
||||
.compress()
|
||||
.to_bytes();
|
||||
|
||||
add_rct_output(
|
||||
&RctOutput {
|
||||
key,
|
||||
height,
|
||||
output_flags,
|
||||
tx_idx: tx_id,
|
||||
commitment,
|
||||
},
|
||||
tables.rct_outputs_mut(),
|
||||
)?
|
||||
// Pre-RingCT outputs.
|
||||
} else {
|
||||
add_output(
|
||||
amount,
|
||||
&Output {
|
||||
key,
|
||||
height,
|
||||
output_flags,
|
||||
tx_idx: tx_id,
|
||||
},
|
||||
tables,
|
||||
)?
|
||||
.amount_index
|
||||
}
|
||||
// RingCT outputs.
|
||||
} else {
|
||||
let commitment = tx.rct_signatures.base.commitments[i].compress().to_bytes();
|
||||
add_rct_output(
|
||||
&RctOutput {
|
||||
key,
|
||||
height,
|
||||
output_flags,
|
||||
tx_idx: tx_id,
|
||||
commitment,
|
||||
},
|
||||
tables.rct_outputs_mut(),
|
||||
)?
|
||||
};
|
||||
|
||||
amount_indices.push(amount_index);
|
||||
} // for each output
|
||||
|
||||
tables
|
||||
.tx_outputs_mut()
|
||||
.put(&tx_id, &StorableVec(amount_indices))?;
|
||||
|
||||
Ok(tx_id)
|
||||
}
|
||||
|
||||
/// Remove a transaction from the database with its [`TxHash`].
|
||||
///
|
||||
/// This returns the [`TxId`] and [`TxBlob`](crate::types::TxBlob) of the removed transaction.
|
||||
///
|
||||
#[doc = doc_add_block_inner_invariant!()]
|
||||
///
|
||||
/// # Notes
|
||||
/// As mentioned in [`add_tx`], this function will call other sub-functions:
|
||||
/// - [`remove_output()`]
|
||||
/// - [`remove_rct_output()`]
|
||||
/// - [`remove_key_image()`]
|
||||
///
|
||||
/// Thus, after [`remove_tx`], those values (outputs and key images)
|
||||
/// will be remove from database tables as well.
|
||||
///
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn remove_tx(
|
||||
tx_hash: &TxHash,
|
||||
tables: &mut impl TablesMut,
|
||||
) -> Result<(TxId, Transaction), RuntimeError> {
|
||||
//------------------------------------------------------ Transaction data
|
||||
let tx_id = tables.tx_ids_mut().take(tx_hash)?;
|
||||
let tx_blob = tables.tx_blobs_mut().take(&tx_id)?;
|
||||
tables.tx_heights_mut().delete(&tx_id)?;
|
||||
tables.tx_outputs_mut().delete(&tx_id)?;
|
||||
|
||||
//------------------------------------------------------ Pruning
|
||||
// SOMEDAY: implement pruning after `monero-serai` does.
|
||||
// table_prunable_hashes.delete(&tx_id)?;
|
||||
// table_prunable_tx_blobs.delete(&tx_id)?;
|
||||
// if let PruningSeed::Pruned(decompressed_pruning_seed) = get_blockchain_pruning_seed()? {
|
||||
// SOMEDAY: what to remove here? which table?
|
||||
// }
|
||||
|
||||
//------------------------------------------------------ Unlock Time
|
||||
match tables.tx_unlock_time_mut().delete(&tx_id) {
|
||||
Ok(()) | Err(RuntimeError::KeyNotFound) => (),
|
||||
// An actual error occurred, return.
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
|
||||
//------------------------------------------------------
|
||||
// Refer to the inner transaction type from now on.
|
||||
let tx = Transaction::read(&mut tx_blob.0.as_slice())?;
|
||||
|
||||
//------------------------------------------------------ Key Images
|
||||
// Is this a miner transaction?
|
||||
let mut miner_tx = false;
|
||||
for inputs in &tx.prefix.inputs {
|
||||
match inputs {
|
||||
// Key images.
|
||||
Input::ToKey { key_image, .. } => {
|
||||
remove_key_image(key_image.compress().as_bytes(), tables.key_images_mut())?;
|
||||
}
|
||||
// This is a miner transaction, set it for later use.
|
||||
Input::Gen(_) => miner_tx = true,
|
||||
}
|
||||
} // for each input
|
||||
|
||||
//------------------------------------------------------ Outputs
|
||||
// Remove each output in the transaction.
|
||||
for output in &tx.prefix.outputs {
|
||||
// Outputs with clear amounts.
|
||||
if let Some(amount) = output.amount {
|
||||
// RingCT miner outputs.
|
||||
if miner_tx && tx.prefix.version == 2 {
|
||||
let amount_index = get_rct_num_outputs(tables.rct_outputs())? - 1;
|
||||
remove_rct_output(&amount_index, tables.rct_outputs_mut())?;
|
||||
// Pre-RingCT outputs.
|
||||
} else {
|
||||
let amount_index = tables.num_outputs_mut().get(&amount)? - 1;
|
||||
remove_output(
|
||||
&PreRctOutputId {
|
||||
amount,
|
||||
amount_index,
|
||||
},
|
||||
tables,
|
||||
)?;
|
||||
}
|
||||
// RingCT outputs.
|
||||
} else {
|
||||
let amount_index = get_rct_num_outputs(tables.rct_outputs())? - 1;
|
||||
remove_rct_output(&amount_index, tables.rct_outputs_mut())?;
|
||||
}
|
||||
} // for each output
|
||||
|
||||
Ok((tx_id, tx))
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- `get_tx_*`
|
||||
/// Retrieve a [`Transaction`] from the database with its [`TxHash`].
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_tx(
|
||||
tx_hash: &TxHash,
|
||||
table_tx_ids: &impl DatabaseRo<TxIds>,
|
||||
table_tx_blobs: &impl DatabaseRo<TxBlobs>,
|
||||
) -> Result<Transaction, RuntimeError> {
|
||||
get_tx_from_id(&table_tx_ids.get(tx_hash)?, table_tx_blobs)
|
||||
}
|
||||
|
||||
/// Retrieve a [`Transaction`] from the database with its [`TxId`].
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_tx_from_id(
|
||||
tx_id: &TxId,
|
||||
table_tx_blobs: &impl DatabaseRo<TxBlobs>,
|
||||
) -> Result<Transaction, RuntimeError> {
|
||||
let tx_blob = table_tx_blobs.get(tx_id)?.0;
|
||||
Ok(Transaction::read(&mut tx_blob.as_slice())?)
|
||||
}
|
||||
|
||||
//----------------------------------------------------------------------------------------------------
|
||||
/// How many [`Transaction`]s are there?
|
||||
///
|
||||
/// This returns the amount of transactions currently stored.
|
||||
///
|
||||
/// For example:
|
||||
/// - 0 transactions exist => returns 0
|
||||
/// - 1 transactions exist => returns 1
|
||||
/// - 5 transactions exist => returns 5
|
||||
/// - etc
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn get_num_tx(table_tx_ids: &impl DatabaseRo<TxIds>) -> Result<u64, RuntimeError> {
|
||||
table_tx_ids.len()
|
||||
}
|
||||
|
||||
//----------------------------------------------------------------------------------------------------
|
||||
/// Check if a transaction exists in the database.
|
||||
///
|
||||
/// Returns `true` if it does, else `false`.
|
||||
#[doc = doc_error!()]
|
||||
#[inline]
|
||||
pub fn tx_exists(
|
||||
tx_hash: &TxHash,
|
||||
table_tx_ids: &impl DatabaseRo<TxIds>,
|
||||
) -> Result<bool, RuntimeError> {
|
||||
table_tx_ids.contains(tx_hash)
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use super::*;
|
||||
use crate::{
|
||||
tables::Tables,
|
||||
tests::{assert_all_tables_are_empty, tmp_concrete_env, AssertTableLen},
|
||||
transaction::TxRw,
|
||||
Env, EnvInner,
|
||||
};
|
||||
use cuprate_test_utils::data::{tx_v1_sig0, tx_v1_sig2, tx_v2_rct3};
|
||||
use pretty_assertions::assert_eq;
|
||||
|
||||
/// Tests all above tx functions when only inputting `Transaction` data (no Block).
|
||||
#[test]
|
||||
fn all_tx_functions() {
|
||||
let (env, _tmp) = tmp_concrete_env();
|
||||
let env_inner = env.env_inner();
|
||||
assert_all_tables_are_empty(&env);
|
||||
|
||||
// Monero `Transaction`, not database tx.
|
||||
let txs = [tx_v1_sig0(), tx_v1_sig2(), tx_v2_rct3()];
|
||||
|
||||
// Add transactions.
|
||||
let tx_ids = {
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
let tx_ids = txs
|
||||
.iter()
|
||||
.map(|tx| {
|
||||
println!("add_tx(): {tx:#?}");
|
||||
add_tx(&tx.tx, &tx.tx_blob, &tx.tx_hash, &0, &mut tables).unwrap()
|
||||
})
|
||||
.collect::<Vec<TxId>>();
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
|
||||
tx_ids
|
||||
};
|
||||
|
||||
// Assert all reads of the transactions are OK.
|
||||
let tx_hashes = {
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
|
||||
// Assert only the proper tables were added to.
|
||||
AssertTableLen {
|
||||
block_infos: 0,
|
||||
block_blobs: 0,
|
||||
block_heights: 0,
|
||||
key_images: 4, // added to key images
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
num_outputs: 9,
|
||||
outputs: 10, // added to outputs
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 2,
|
||||
tx_blobs: 3,
|
||||
tx_ids: 3,
|
||||
tx_heights: 3,
|
||||
tx_unlock_time: 1, // only 1 has a timelock
|
||||
}
|
||||
.assert(&tables);
|
||||
|
||||
// Both from ID and hash should result in getting the same transaction.
|
||||
let mut tx_hashes = vec![];
|
||||
for (i, tx_id) in tx_ids.iter().enumerate() {
|
||||
println!("tx_ids.iter(): i: {i}, tx_id: {tx_id}");
|
||||
|
||||
let tx_get_from_id = get_tx_from_id(tx_id, tables.tx_blobs()).unwrap();
|
||||
let tx_hash = tx_get_from_id.hash();
|
||||
let tx_get = get_tx(&tx_hash, tables.tx_ids(), tables.tx_blobs()).unwrap();
|
||||
|
||||
println!("tx_ids.iter(): tx_get_from_id: {tx_get_from_id:#?}, tx_get: {tx_get:#?}");
|
||||
|
||||
assert_eq!(tx_get_from_id.hash(), tx_get.hash());
|
||||
assert_eq!(tx_get_from_id.hash(), txs[i].tx_hash);
|
||||
assert_eq!(tx_get_from_id, tx_get);
|
||||
assert_eq!(tx_get, txs[i].tx);
|
||||
assert!(tx_exists(&tx_hash, tables.tx_ids()).unwrap());
|
||||
|
||||
tx_hashes.push(tx_hash);
|
||||
}
|
||||
|
||||
tx_hashes
|
||||
};
|
||||
|
||||
// Remove the transactions.
|
||||
{
|
||||
let tx_rw = env_inner.tx_rw().unwrap();
|
||||
let mut tables = env_inner.open_tables_mut(&tx_rw).unwrap();
|
||||
|
||||
for tx_hash in tx_hashes {
|
||||
println!("remove_tx(): tx_hash: {tx_hash:?}");
|
||||
|
||||
let (tx_id, _) = remove_tx(&tx_hash, &mut tables).unwrap();
|
||||
assert!(matches!(
|
||||
get_tx_from_id(&tx_id, tables.tx_blobs()),
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
));
|
||||
}
|
||||
|
||||
drop(tables);
|
||||
TxRw::commit(tx_rw).unwrap();
|
||||
}
|
||||
|
||||
assert_all_tables_are_empty(&env);
|
||||
}
|
||||
}
|
||||
307
storage/database/src/resize.rs
Normal file
307
storage/database/src/resize.rs
Normal file
|
|
@ -0,0 +1,307 @@
|
|||
//! Database memory map resizing algorithms.
|
||||
//!
|
||||
//! This modules contains [`ResizeAlgorithm`] which determines how the
|
||||
//! [`ConcreteEnv`](crate::ConcreteEnv) resizes its memory map when needing more space.
|
||||
//! This value is in [`Config`](crate::config::Config) and can be selected at runtime.
|
||||
//!
|
||||
//! Although, it is only used by `ConcreteEnv` if [`Env::MANUAL_RESIZE`](crate::env::Env::MANUAL_RESIZE) is `true`.
|
||||
//!
|
||||
//! The algorithms are available as free functions in this module as well.
|
||||
//!
|
||||
//! # Page size
|
||||
//! All free functions in this module will
|
||||
//! return a multiple of the OS page size ([`page_size()`]),
|
||||
//! [LMDB will error](http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5)
|
||||
//! if this is not the case.
|
||||
//!
|
||||
//! # Invariants
|
||||
//! All returned [`NonZeroUsize`] values of the free functions in this module
|
||||
//! (including [`ResizeAlgorithm::resize`]) uphold the following invariants:
|
||||
//! 1. It will always be `>=` the input `current_size_bytes`
|
||||
//! 2. It will always be a multiple of [`page_size()`]
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{num::NonZeroUsize, sync::OnceLock};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- ResizeAlgorithm
|
||||
/// The function/algorithm used by the
|
||||
/// database when resizing the memory map.
|
||||
///
|
||||
// # SOMEDAY
|
||||
// We could test around with different algorithms.
|
||||
// Calling `heed::Env::resize` is surprisingly fast,
|
||||
// around `0.0000082s` on my machine. We could probably
|
||||
// get away with smaller and more frequent resizes.
|
||||
// **With the caveat being we are taking a `WriteGuard` to a `RwLock`.**
|
||||
#[derive(Copy, Clone, Debug, PartialEq, PartialOrd)]
|
||||
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
|
||||
pub enum ResizeAlgorithm {
|
||||
/// Uses [`monero`].
|
||||
Monero,
|
||||
|
||||
/// Uses [`fixed_bytes`].
|
||||
FixedBytes(NonZeroUsize),
|
||||
|
||||
/// Uses [`percent`].
|
||||
Percent(f32),
|
||||
}
|
||||
|
||||
impl ResizeAlgorithm {
|
||||
/// Returns [`Self::Monero`].
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// assert!(matches!(ResizeAlgorithm::new(), ResizeAlgorithm::Monero));
|
||||
/// ```
|
||||
#[inline]
|
||||
pub const fn new() -> Self {
|
||||
Self::Monero
|
||||
}
|
||||
|
||||
/// Maps the `self` variant to the free functions in [`crate::resize`].
|
||||
///
|
||||
/// This function returns the _new_ memory map size in bytes.
|
||||
#[inline]
|
||||
pub fn resize(&self, current_size_bytes: usize) -> NonZeroUsize {
|
||||
match self {
|
||||
Self::Monero => monero(current_size_bytes),
|
||||
Self::FixedBytes(add_bytes) => fixed_bytes(current_size_bytes, add_bytes.get()),
|
||||
Self::Percent(f) => percent(current_size_bytes, *f),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for ResizeAlgorithm {
|
||||
/// Calls [`Self::new`].
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// assert_eq!(ResizeAlgorithm::new(), ResizeAlgorithm::default());
|
||||
/// ```
|
||||
#[inline]
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Free functions
|
||||
/// This function retrieves the system’s memory page size.
|
||||
///
|
||||
/// It is just [`page_size::get`](https://docs.rs/page_size) internally.
|
||||
///
|
||||
/// This caches the result, so this function is cheap after the 1st call.
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if the OS returns of page size of `0` (impossible?).
|
||||
#[inline]
|
||||
pub fn page_size() -> NonZeroUsize {
|
||||
/// Cached result of [`page_size()`].
|
||||
static PAGE_SIZE: OnceLock<NonZeroUsize> = OnceLock::new();
|
||||
*PAGE_SIZE
|
||||
.get_or_init(|| NonZeroUsize::new(page_size::get()).expect("page_size::get() returned 0"))
|
||||
}
|
||||
|
||||
/// Memory map resize closely matching `monerod`.
|
||||
///
|
||||
/// # Method
|
||||
/// This function mostly matches `monerod`'s current resize implementation[^1],
|
||||
/// and will increase `current_size_bytes` by `1 << 30`[^2] exactly then
|
||||
/// rounded to the nearest multiple of the OS page size.
|
||||
///
|
||||
/// [^1]: <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L549>
|
||||
///
|
||||
/// [^2]: `1_073_745_920`
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// // The value this function will increment by
|
||||
/// // (assuming page multiple of 4096).
|
||||
/// const N: usize = 1_073_741_824;
|
||||
///
|
||||
/// // 0 returns the minimum value.
|
||||
/// assert_eq!(monero(0).get(), N);
|
||||
///
|
||||
/// // Rounds up to nearest OS page size.
|
||||
/// assert_eq!(monero(1).get(), N + page_size().get());
|
||||
/// ```
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if adding onto `current_size_bytes` overflows [`usize::MAX`].
|
||||
///
|
||||
/// ```rust,should_panic
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// // Ridiculous large numbers panic.
|
||||
/// monero(usize::MAX);
|
||||
/// ```
|
||||
pub fn monero(current_size_bytes: usize) -> NonZeroUsize {
|
||||
/// The exact expression used by `monerod`
|
||||
/// when calculating how many bytes to add.
|
||||
///
|
||||
/// The nominal value is `1_073_741_824`.
|
||||
/// Not actually 1 GB but close enough I guess.
|
||||
///
|
||||
/// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L553>
|
||||
const ADD_SIZE: usize = 1_usize << 30;
|
||||
|
||||
let page_size = page_size().get();
|
||||
let new_size_bytes = current_size_bytes + ADD_SIZE;
|
||||
|
||||
// Round up the new size to the
|
||||
// nearest multiple of the OS page size.
|
||||
let remainder = new_size_bytes % page_size;
|
||||
|
||||
// INVARIANT: minimum is always at least `ADD_SIZE`.
|
||||
NonZeroUsize::new(if remainder == 0 {
|
||||
new_size_bytes
|
||||
} else {
|
||||
(new_size_bytes + page_size) - remainder
|
||||
})
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// Memory map resize by a fixed amount of bytes.
|
||||
///
|
||||
/// # Method
|
||||
/// This function will `current_size_bytes + add_bytes`
|
||||
/// and then round up to nearest OS page size.
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// let page_size: usize = page_size().get();
|
||||
///
|
||||
/// // Anything below the page size will round up to the page size.
|
||||
/// for i in 0..=page_size {
|
||||
/// assert_eq!(fixed_bytes(0, i).get(), page_size);
|
||||
/// }
|
||||
///
|
||||
/// // (page_size + 1) will round up to (page_size * 2).
|
||||
/// assert_eq!(fixed_bytes(page_size, 1).get(), page_size * 2);
|
||||
///
|
||||
/// // (page_size + page_size) doesn't require any rounding.
|
||||
/// assert_eq!(fixed_bytes(page_size, page_size).get(), page_size * 2);
|
||||
/// ```
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if adding onto `current_size_bytes` overflows [`usize::MAX`].
|
||||
///
|
||||
/// ```rust,should_panic
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// // Ridiculous large numbers panic.
|
||||
/// fixed_bytes(1, usize::MAX);
|
||||
/// ```
|
||||
pub fn fixed_bytes(current_size_bytes: usize, add_bytes: usize) -> NonZeroUsize {
|
||||
let page_size = page_size();
|
||||
let new_size_bytes = current_size_bytes + add_bytes;
|
||||
|
||||
// Guard against < page_size.
|
||||
if new_size_bytes <= page_size.get() {
|
||||
return page_size;
|
||||
}
|
||||
|
||||
// Round up the new size to the
|
||||
// nearest multiple of the OS page size.
|
||||
let remainder = new_size_bytes % page_size;
|
||||
|
||||
// INVARIANT: we guarded against < page_size above.
|
||||
NonZeroUsize::new(if remainder == 0 {
|
||||
new_size_bytes
|
||||
} else {
|
||||
(new_size_bytes + page_size.get()) - remainder
|
||||
})
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// Memory map resize by a percentage.
|
||||
///
|
||||
/// # Method
|
||||
/// This function will multiply `current_size_bytes` by `percent`.
|
||||
///
|
||||
/// Any input `<= 1.0` or non-normal float ([`f32::NAN`], [`f32::INFINITY`])
|
||||
/// will make the returning `NonZeroUsize` the same as `current_size_bytes`
|
||||
/// (rounded up to the OS page size).
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// let page_size: usize = page_size().get();
|
||||
///
|
||||
/// // Anything below the page size will round up to the page size.
|
||||
/// for i in 0..=page_size {
|
||||
/// assert_eq!(percent(i, 1.0).get(), page_size);
|
||||
/// }
|
||||
///
|
||||
/// // Same for 2 page sizes.
|
||||
/// for i in (page_size + 1)..=(page_size * 2) {
|
||||
/// assert_eq!(percent(i, 1.0).get(), page_size * 2);
|
||||
/// }
|
||||
///
|
||||
/// // Weird floats do nothing.
|
||||
/// assert_eq!(percent(page_size, f32::NAN).get(), page_size);
|
||||
/// assert_eq!(percent(page_size, f32::INFINITY).get(), page_size);
|
||||
/// assert_eq!(percent(page_size, f32::NEG_INFINITY).get(), page_size);
|
||||
/// assert_eq!(percent(page_size, -1.0).get(), page_size);
|
||||
/// assert_eq!(percent(page_size, 0.999).get(), page_size);
|
||||
/// ```
|
||||
///
|
||||
/// # Panics
|
||||
/// This function will panic if `current_size_bytes * percent`
|
||||
/// is closer to [`usize::MAX`] than the OS page size.
|
||||
///
|
||||
/// ```rust,should_panic
|
||||
/// # use cuprate_database::resize::*;
|
||||
/// // Ridiculous large numbers panic.
|
||||
/// percent(usize::MAX, 1.001);
|
||||
/// ```
|
||||
pub fn percent(current_size_bytes: usize, percent: f32) -> NonZeroUsize {
|
||||
// Guard against bad floats.
|
||||
use std::num::FpCategory;
|
||||
let percent = match percent.classify() {
|
||||
FpCategory::Normal => {
|
||||
if percent <= 1.0 {
|
||||
1.0
|
||||
} else {
|
||||
percent
|
||||
}
|
||||
}
|
||||
_ => 1.0,
|
||||
};
|
||||
|
||||
let page_size = page_size();
|
||||
|
||||
// INVARIANT: Allow `f32` <-> `usize` casting, we handle all cases.
|
||||
#[allow(
|
||||
clippy::cast_possible_truncation,
|
||||
clippy::cast_sign_loss,
|
||||
clippy::cast_precision_loss
|
||||
)]
|
||||
let new_size_bytes = ((current_size_bytes as f32) * percent) as usize;
|
||||
|
||||
// Panic if rounding up to the nearest page size would overflow.
|
||||
let new_size_bytes = if new_size_bytes > (usize::MAX - page_size.get()) {
|
||||
panic!("new_size_bytes is percent() near usize::MAX");
|
||||
} else {
|
||||
new_size_bytes
|
||||
};
|
||||
|
||||
// Guard against < page_size.
|
||||
if new_size_bytes <= page_size.get() {
|
||||
return page_size;
|
||||
}
|
||||
|
||||
// Round up the new size to the
|
||||
// nearest multiple of the OS page size.
|
||||
let remainder = new_size_bytes % page_size;
|
||||
|
||||
// INVARIANT: we guarded against < page_size above.
|
||||
NonZeroUsize::new(if remainder == 0 {
|
||||
new_size_bytes
|
||||
} else {
|
||||
(new_size_bytes + page_size.get()) - remainder
|
||||
})
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
40
storage/database/src/service/free.rs
Normal file
40
storage/database/src/service/free.rs
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
//! General free functions used (related to `cuprate_database::service`).
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::{
|
||||
config::Config,
|
||||
error::InitError,
|
||||
service::{DatabaseReadHandle, DatabaseWriteHandle},
|
||||
ConcreteEnv, Env,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Init
|
||||
#[cold]
|
||||
#[inline(never)] // Only called once (?)
|
||||
/// Initialize a database & thread-pool, and return a read/write handle to it.
|
||||
///
|
||||
/// Once the returned handles are [`Drop::drop`]ed, the reader
|
||||
/// thread-pool and writer thread will exit automatically.
|
||||
///
|
||||
/// # Errors
|
||||
/// This will forward the error if [`Env::open`] failed.
|
||||
pub fn init(config: Config) -> Result<(DatabaseReadHandle, DatabaseWriteHandle), InitError> {
|
||||
let reader_threads = config.reader_threads;
|
||||
|
||||
// Initialize the database itself.
|
||||
let db = Arc::new(ConcreteEnv::open(config)?);
|
||||
|
||||
// Spawn the Reader thread pool and Writer.
|
||||
let readers = DatabaseReadHandle::init(&db, reader_threads);
|
||||
let writer = DatabaseWriteHandle::init(db);
|
||||
|
||||
Ok((readers, writer))
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
130
storage/database/src/service/mod.rs
Normal file
130
storage/database/src/service/mod.rs
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
//! [`tower::Service`] integeration + thread-pool.
|
||||
//!
|
||||
//! ## `service`
|
||||
//! The `service` module implements the [`tower`] integration,
|
||||
//! along with the reader/writer thread-pool system.
|
||||
//!
|
||||
//! The thread-pool allows outside crates to communicate with it by
|
||||
//! sending database [`Request`][req_r]s and receiving [`Response`][resp]s `async`hronously -
|
||||
//! without having to actually worry and handle the database themselves.
|
||||
//!
|
||||
//! The system is managed by this crate, and only requires [`init`] by the user.
|
||||
//!
|
||||
//! This module must be enabled with the `service` feature.
|
||||
//!
|
||||
//! ## Handles
|
||||
//! The 2 handles to the database are:
|
||||
//! - [`DatabaseReadHandle`]
|
||||
//! - [`DatabaseWriteHandle`]
|
||||
//!
|
||||
//! The 1st allows any caller to send [`ReadRequest`][req_r]s.
|
||||
//!
|
||||
//! The 2nd allows any caller to send [`WriteRequest`][req_w]s.
|
||||
//!
|
||||
//! The `DatabaseReadHandle` can be shared as it is cheaply [`Clone`]able, however,
|
||||
//! the `DatabaseWriteHandle` cannot be cloned. There is only 1 place in Cuprate that
|
||||
//! writes, so it is passed there and used.
|
||||
//!
|
||||
//! ## Initialization
|
||||
//! The database & thread-pool system can be initialized with [`init()`].
|
||||
//!
|
||||
//! This causes the underlying database/threads to be setup
|
||||
//! and returns a read/write handle to that database.
|
||||
//!
|
||||
//! ## Shutdown
|
||||
//! Upon the above handles being dropped, the corresponding thread(s) will automatically exit, i.e:
|
||||
//! - The last [`DatabaseReadHandle`] is dropped => reader thread-pool exits
|
||||
//! - The last [`DatabaseWriteHandle`] is dropped => writer thread exits
|
||||
//!
|
||||
//! Upon dropping the [`crate::ConcreteEnv`]:
|
||||
//! - All un-processed database transactions are completed
|
||||
//! - All data gets flushed to disk (caused by [`Drop::drop`] impl on [`crate::ConcreteEnv`])
|
||||
//!
|
||||
//! ## Request and Response
|
||||
//! To interact with the database (whether reading or writing data),
|
||||
//! a `Request` can be sent using one of the above handles.
|
||||
//!
|
||||
//! Both the handles implement `tower::Service`, so they can be [`tower::Service::call`]ed.
|
||||
//!
|
||||
//! An `async`hronous channel will be returned from the call.
|
||||
//! This channel can be `.await`ed upon to (eventually) receive
|
||||
//! the corresponding `Response` to your `Request`.
|
||||
//!
|
||||
//! [req_r]: cuprate_types::service::ReadRequest
|
||||
//!
|
||||
//! [req_w]: cuprate_types::service::WriteRequest
|
||||
//!
|
||||
//! [resp]: cuprate_types::service::Response
|
||||
//!
|
||||
//! # Example
|
||||
//! Simple usage of `service`.
|
||||
//!
|
||||
//! ```rust
|
||||
//! use hex_literal::hex;
|
||||
//! use tower::{Service, ServiceExt};
|
||||
//!
|
||||
//! use cuprate_types::service::{ReadRequest, WriteRequest, Response};
|
||||
//! use cuprate_test_utils::data::block_v16_tx0;
|
||||
//!
|
||||
//! use cuprate_database::{ConcreteEnv, config::ConfigBuilder, Env};
|
||||
//!
|
||||
//! # #[tokio::main]
|
||||
//! # async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
//! // Create a configuration for the database environment.
|
||||
//! let db_dir = tempfile::tempdir()?;
|
||||
//! let config = ConfigBuilder::new()
|
||||
//! .db_directory(db_dir.path().to_path_buf())
|
||||
//! .build();
|
||||
//!
|
||||
//! // Initialize the database thread-pool.
|
||||
//! let (mut read_handle, mut write_handle) = cuprate_database::service::init(config)?;
|
||||
//!
|
||||
//! // Prepare a request to write block.
|
||||
//! let mut block = block_v16_tx0().clone();
|
||||
//! # block.height = 0 as u64; // must be 0th height or panic in `add_block()`
|
||||
//! let request = WriteRequest::WriteBlock(block);
|
||||
//!
|
||||
//! // Send the request.
|
||||
//! // We receive back an `async` channel that will
|
||||
//! // eventually yield the result when `service`
|
||||
//! // is done writing the block.
|
||||
//! let response_channel = write_handle.ready().await?.call(request);
|
||||
//!
|
||||
//! // Block write was OK.
|
||||
//! let response = response_channel.await?;
|
||||
//! assert_eq!(response, Response::WriteBlockOk);
|
||||
//!
|
||||
//! // Now, let's try getting the block hash
|
||||
//! // of the block we just wrote.
|
||||
//! let request = ReadRequest::BlockHash(0);
|
||||
//! let response_channel = read_handle.ready().await?.call(request);
|
||||
//! let response = response_channel.await?;
|
||||
//! assert_eq!(
|
||||
//! response,
|
||||
//! Response::BlockHash(
|
||||
//! hex!("43bd1f2b6556dcafa413d8372974af59e4e8f37dbf74dc6b2a9b7212d0577428")
|
||||
//! )
|
||||
//! );
|
||||
//!
|
||||
//! // This causes the writer thread on the
|
||||
//! // other side of this handle to exit...
|
||||
//! drop(write_handle);
|
||||
//! // ...and this causes the reader thread-pool to exit.
|
||||
//! drop(read_handle);
|
||||
//! # Ok(()) }
|
||||
//! ```
|
||||
|
||||
mod read;
|
||||
pub use read::DatabaseReadHandle;
|
||||
|
||||
mod write;
|
||||
pub use write::DatabaseWriteHandle;
|
||||
|
||||
mod free;
|
||||
pub use free::init;
|
||||
|
||||
// Internal type aliases for `service`.
|
||||
mod types;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
493
storage/database/src/service/read.rs
Normal file
493
storage/database/src/service/read.rs
Normal file
|
|
@ -0,0 +1,493 @@
|
|||
//! Database reader thread-pool definitions and logic.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{
|
||||
collections::{HashMap, HashSet},
|
||||
sync::Arc,
|
||||
task::{Context, Poll},
|
||||
};
|
||||
|
||||
use futures::{channel::oneshot, ready};
|
||||
use rayon::iter::{IntoParallelIterator, ParallelIterator};
|
||||
use thread_local::ThreadLocal;
|
||||
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
|
||||
use tokio_util::sync::PollSemaphore;
|
||||
|
||||
use cuprate_helper::asynch::InfallibleOneshotReceiver;
|
||||
use cuprate_types::{
|
||||
service::{ReadRequest, Response},
|
||||
ExtendedBlockHeader, OutputOnChain,
|
||||
};
|
||||
|
||||
use crate::{
|
||||
config::ReaderThreads,
|
||||
error::RuntimeError,
|
||||
ops::{
|
||||
block::{get_block_extended_header_from_height, get_block_info},
|
||||
blockchain::{cumulative_generated_coins, top_block_height},
|
||||
key_image::key_image_exists,
|
||||
output::id_to_output_on_chain,
|
||||
},
|
||||
service::types::{ResponseReceiver, ResponseResult, ResponseSender},
|
||||
tables::{BlockHeights, BlockInfos, Tables},
|
||||
types::{Amount, AmountIndex, BlockHeight, KeyImage, PreRctOutputId},
|
||||
ConcreteEnv, DatabaseRo, Env, EnvInner,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseReadHandle
|
||||
/// Read handle to the database.
|
||||
///
|
||||
/// This is cheaply [`Clone`]able handle that
|
||||
/// allows `async`hronously reading from the database.
|
||||
///
|
||||
/// Calling [`tower::Service::call`] with a [`DatabaseReadHandle`] & [`ReadRequest`]
|
||||
/// will return an `async`hronous channel that can be `.await`ed upon
|
||||
/// to receive the corresponding [`Response`].
|
||||
pub struct DatabaseReadHandle {
|
||||
/// Handle to the custom `rayon` DB reader thread-pool.
|
||||
///
|
||||
/// Requests are [`rayon::ThreadPool::spawn`]ed in this thread-pool,
|
||||
/// and responses are returned via a channel we (the caller) provide.
|
||||
pool: Arc<rayon::ThreadPool>,
|
||||
|
||||
/// Counting semaphore asynchronous permit for database access.
|
||||
/// Each [`tower::Service::poll_ready`] will acquire a permit
|
||||
/// before actually sending a request to the `rayon` DB threadpool.
|
||||
semaphore: PollSemaphore,
|
||||
|
||||
/// An owned permit.
|
||||
/// This will be set to [`Some`] in `poll_ready()` when we successfully acquire
|
||||
/// the permit, and will be [`Option::take()`]n after `tower::Service::call()` is called.
|
||||
///
|
||||
/// The actual permit will be dropped _after_ the rayon DB thread has finished
|
||||
/// the request, i.e., after [`map_request()`] finishes.
|
||||
permit: Option<OwnedSemaphorePermit>,
|
||||
|
||||
/// Access to the database.
|
||||
env: Arc<ConcreteEnv>,
|
||||
}
|
||||
|
||||
// `OwnedSemaphorePermit` does not implement `Clone`,
|
||||
// so manually clone all elements, while keeping `permit`
|
||||
// `None` across clones.
|
||||
impl Clone for DatabaseReadHandle {
|
||||
fn clone(&self) -> Self {
|
||||
Self {
|
||||
pool: Arc::clone(&self.pool),
|
||||
semaphore: self.semaphore.clone(),
|
||||
permit: None,
|
||||
env: Arc::clone(&self.env),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl DatabaseReadHandle {
|
||||
/// Initialize the `DatabaseReader` thread-pool backed by `rayon`.
|
||||
///
|
||||
/// This spawns `N` amount of `DatabaseReader`'s
|
||||
/// attached to `env` and returns a handle to the pool.
|
||||
///
|
||||
/// Should be called _once_ per actual database.
|
||||
#[cold]
|
||||
#[inline(never)] // Only called once.
|
||||
pub(super) fn init(env: &Arc<ConcreteEnv>, reader_threads: ReaderThreads) -> Self {
|
||||
// How many reader threads to spawn?
|
||||
let reader_count = reader_threads.as_threads().get();
|
||||
|
||||
// Spawn `rayon` reader threadpool.
|
||||
let pool = rayon::ThreadPoolBuilder::new()
|
||||
.num_threads(reader_count)
|
||||
.thread_name(|i| format!("cuprate_helper::service::read::DatabaseReader{i}"))
|
||||
.build()
|
||||
.unwrap();
|
||||
|
||||
// Create a semaphore with the same amount of
|
||||
// permits as the amount of reader threads.
|
||||
let semaphore = PollSemaphore::new(Arc::new(Semaphore::new(reader_count)));
|
||||
|
||||
// Return a handle to the pool.
|
||||
Self {
|
||||
pool: Arc::new(pool),
|
||||
semaphore,
|
||||
permit: None,
|
||||
env: Arc::clone(env),
|
||||
}
|
||||
}
|
||||
|
||||
/// Access to the actual database environment.
|
||||
///
|
||||
/// # ⚠️ Warning
|
||||
/// This function gives you access to the actual
|
||||
/// underlying database connected to by `self`.
|
||||
///
|
||||
/// I.e. it allows you to read/write data _directly_
|
||||
/// instead of going through a request.
|
||||
///
|
||||
/// Be warned that using the database directly
|
||||
/// in this manner has not been tested.
|
||||
#[inline]
|
||||
pub const fn env(&self) -> &Arc<ConcreteEnv> {
|
||||
&self.env
|
||||
}
|
||||
}
|
||||
|
||||
impl tower::Service<ReadRequest> for DatabaseReadHandle {
|
||||
type Response = Response;
|
||||
type Error = RuntimeError;
|
||||
type Future = ResponseReceiver;
|
||||
|
||||
#[inline]
|
||||
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
|
||||
// Check if we already have a permit.
|
||||
if self.permit.is_some() {
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
|
||||
// Acquire a permit before returning `Ready`.
|
||||
let permit =
|
||||
ready!(self.semaphore.poll_acquire(cx)).expect("this semaphore is never closed");
|
||||
|
||||
self.permit = Some(permit);
|
||||
Poll::Ready(Ok(()))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn call(&mut self, request: ReadRequest) -> Self::Future {
|
||||
let permit = self
|
||||
.permit
|
||||
.take()
|
||||
.expect("poll_ready() should have acquire a permit before calling call()");
|
||||
|
||||
// Response channel we `.await` on.
|
||||
let (response_sender, receiver) = oneshot::channel();
|
||||
|
||||
// Spawn the request in the rayon DB thread-pool.
|
||||
//
|
||||
// Note that this uses `self.pool` instead of `rayon::spawn`
|
||||
// such that any `rayon` parallel code that runs within
|
||||
// the passed closure uses the same `rayon` threadpool.
|
||||
//
|
||||
// INVARIANT:
|
||||
// The below `DatabaseReader` function impl block relies on this behavior.
|
||||
let env = Arc::clone(&self.env);
|
||||
self.pool.spawn(move || {
|
||||
let _permit: OwnedSemaphorePermit = permit;
|
||||
map_request(&env, request, response_sender);
|
||||
}); // drop(permit/env);
|
||||
|
||||
InfallibleOneshotReceiver::from(receiver)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Request Mapping
|
||||
// This function maps [`Request`]s to function calls
|
||||
// executed by the rayon DB reader threadpool.
|
||||
|
||||
/// Map [`Request`]'s to specific database handler functions.
|
||||
///
|
||||
/// This is the main entrance into all `Request` handler functions.
|
||||
/// The basic structure is:
|
||||
/// 1. `Request` is mapped to a handler function
|
||||
/// 2. Handler function is called
|
||||
/// 3. [`Response`] is sent
|
||||
fn map_request(
|
||||
env: &ConcreteEnv, // Access to the database
|
||||
request: ReadRequest, // The request we must fulfill
|
||||
response_sender: ResponseSender, // The channel we must send the response back to
|
||||
) {
|
||||
use ReadRequest as R;
|
||||
|
||||
/* SOMEDAY: pre-request handling, run some code for each request? */
|
||||
|
||||
let response = match request {
|
||||
R::BlockExtendedHeader(block) => block_extended_header(env, block),
|
||||
R::BlockHash(block) => block_hash(env, block),
|
||||
R::BlockExtendedHeaderInRange(range) => block_extended_header_in_range(env, range),
|
||||
R::ChainHeight => chain_height(env),
|
||||
R::GeneratedCoins => generated_coins(env),
|
||||
R::Outputs(map) => outputs(env, map),
|
||||
R::NumberOutputsWithAmount(vec) => number_outputs_with_amount(env, vec),
|
||||
R::CheckKIsNotSpent(set) => check_k_is_not_spent(env, set),
|
||||
};
|
||||
|
||||
if let Err(e) = response_sender.send(response) {
|
||||
// TODO: use tracing.
|
||||
println!("database reader failed to send response: {e:?}");
|
||||
}
|
||||
|
||||
/* SOMEDAY: post-request handling, run some code for each request? */
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Thread Local
|
||||
/// Q: Why does this exist?
|
||||
///
|
||||
/// A1: `heed`'s transactions and tables are not `Sync`, so we cannot use
|
||||
/// them with rayon, however, we set a feature such that they are `Send`.
|
||||
///
|
||||
/// A2: When sending to rayon, we want to ensure each read transaction
|
||||
/// is only being used by 1 thread only to scale reads
|
||||
///
|
||||
/// <https://github.com/Cuprate/cuprate/pull/113#discussion_r1576762346>
|
||||
#[inline]
|
||||
fn thread_local<T: Send>(env: &impl Env) -> ThreadLocal<T> {
|
||||
ThreadLocal::with_capacity(env.config().reader_threads.as_threads().get())
|
||||
}
|
||||
|
||||
/// Take in a `ThreadLocal<impl Tables>` and return an `&impl Tables + Send`.
|
||||
///
|
||||
/// # Safety
|
||||
/// See [`DatabaseRo`] docs.
|
||||
///
|
||||
/// We are safely using `UnsafeSendable` in `service`'s reader thread-pool
|
||||
/// as we are pairing our usage with `ThreadLocal` - only 1 thread
|
||||
/// will ever access a transaction at a time. This is an INVARIANT.
|
||||
///
|
||||
/// A `Mutex` was considered but:
|
||||
/// - It is less performant
|
||||
/// - It isn't technically needed for safety in our use-case
|
||||
/// - It causes `DatabaseIter` function return issues as there is a `MutexGuard` object
|
||||
///
|
||||
/// <https://github.com/Cuprate/cuprate/pull/113#discussion_r1581684698>
|
||||
///
|
||||
/// # Notes
|
||||
/// This is used for other backends as well instead of branching with `cfg_if`.
|
||||
/// The other backends (as of current) are `Send + Sync` so this is fine.
|
||||
/// <https://github.com/Cuprate/cuprate/pull/113#discussion_r1585618374>
|
||||
macro_rules! get_tables {
|
||||
($env_inner:ident, $tx_ro:ident, $tables:ident) => {{
|
||||
$tables.get_or_try(|| {
|
||||
#[allow(clippy::significant_drop_in_scrutinee)]
|
||||
match $env_inner.open_tables($tx_ro) {
|
||||
// SAFETY: see above macro doc comment.
|
||||
Ok(tables) => Ok(unsafe { crate::unsafe_sendable::UnsafeSendable::new(tables) }),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
})
|
||||
}};
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Handler functions
|
||||
// These are the actual functions that do stuff according to the incoming [`Request`].
|
||||
//
|
||||
// Each function name is a 1-1 mapping (from CamelCase -> snake_case) to
|
||||
// the enum variant name, e.g: `BlockExtendedHeader` -> `block_extended_header`.
|
||||
//
|
||||
// Each function will return the [`Response`] that we
|
||||
// should send back to the caller in [`map_request()`].
|
||||
//
|
||||
// INVARIANT:
|
||||
// These functions are called above in `tower::Service::call()`
|
||||
// using a custom threadpool which means any call to `par_*()` functions
|
||||
// will be using the custom rayon DB reader thread-pool, not the global one.
|
||||
//
|
||||
// All functions below assume that this is the case, such that
|
||||
// `par_*()` functions will not block the _global_ rayon thread-pool.
|
||||
|
||||
// FIXME: implement multi-transaction read atomicity.
|
||||
// <https://github.com/Cuprate/cuprate/pull/113#discussion_r1576874589>.
|
||||
|
||||
/// [`ReadRequest::BlockExtendedHeader`].
|
||||
#[inline]
|
||||
fn block_extended_header(env: &ConcreteEnv, block_height: BlockHeight) -> ResponseResult {
|
||||
// Single-threaded, no `ThreadLocal` required.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro()?;
|
||||
let tables = env_inner.open_tables(&tx_ro)?;
|
||||
|
||||
Ok(Response::BlockExtendedHeader(
|
||||
get_block_extended_header_from_height(&block_height, &tables)?,
|
||||
))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::BlockHash`].
|
||||
#[inline]
|
||||
fn block_hash(env: &ConcreteEnv, block_height: BlockHeight) -> ResponseResult {
|
||||
// Single-threaded, no `ThreadLocal` required.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro()?;
|
||||
let table_block_infos = env_inner.open_db_ro::<BlockInfos>(&tx_ro)?;
|
||||
|
||||
Ok(Response::BlockHash(
|
||||
get_block_info(&block_height, &table_block_infos)?.block_hash,
|
||||
))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::BlockExtendedHeaderInRange`].
|
||||
#[inline]
|
||||
fn block_extended_header_in_range(
|
||||
env: &ConcreteEnv,
|
||||
range: std::ops::Range<BlockHeight>,
|
||||
) -> ResponseResult {
|
||||
// Prepare tx/tables in `ThreadLocal`.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = thread_local(env);
|
||||
let tables = thread_local(env);
|
||||
|
||||
// Collect results using `rayon`.
|
||||
let vec = range
|
||||
.into_par_iter()
|
||||
.map(|block_height| {
|
||||
let tx_ro = tx_ro.get_or_try(|| env_inner.tx_ro())?;
|
||||
let tables = get_tables!(env_inner, tx_ro, tables)?.as_ref();
|
||||
get_block_extended_header_from_height(&block_height, tables)
|
||||
})
|
||||
.collect::<Result<Vec<ExtendedBlockHeader>, RuntimeError>>()?;
|
||||
|
||||
Ok(Response::BlockExtendedHeaderInRange(vec))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::ChainHeight`].
|
||||
#[inline]
|
||||
fn chain_height(env: &ConcreteEnv) -> ResponseResult {
|
||||
// Single-threaded, no `ThreadLocal` required.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro()?;
|
||||
let table_block_heights = env_inner.open_db_ro::<BlockHeights>(&tx_ro)?;
|
||||
let table_block_infos = env_inner.open_db_ro::<BlockInfos>(&tx_ro)?;
|
||||
|
||||
let chain_height = crate::ops::blockchain::chain_height(&table_block_heights)?;
|
||||
let block_hash =
|
||||
get_block_info(&chain_height.saturating_sub(1), &table_block_infos)?.block_hash;
|
||||
|
||||
Ok(Response::ChainHeight(chain_height, block_hash))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::GeneratedCoins`].
|
||||
#[inline]
|
||||
fn generated_coins(env: &ConcreteEnv) -> ResponseResult {
|
||||
// Single-threaded, no `ThreadLocal` required.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro()?;
|
||||
let table_block_heights = env_inner.open_db_ro::<BlockHeights>(&tx_ro)?;
|
||||
let table_block_infos = env_inner.open_db_ro::<BlockInfos>(&tx_ro)?;
|
||||
|
||||
let top_height = top_block_height(&table_block_heights)?;
|
||||
|
||||
Ok(Response::GeneratedCoins(cumulative_generated_coins(
|
||||
&top_height,
|
||||
&table_block_infos,
|
||||
)?))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::Outputs`].
|
||||
#[inline]
|
||||
fn outputs(env: &ConcreteEnv, outputs: HashMap<Amount, HashSet<AmountIndex>>) -> ResponseResult {
|
||||
// Prepare tx/tables in `ThreadLocal`.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = thread_local(env);
|
||||
let tables = thread_local(env);
|
||||
|
||||
// The 2nd mapping function.
|
||||
// This is pulled out from the below `map()` for readability.
|
||||
let inner_map = |amount, amount_index| -> Result<(AmountIndex, OutputOnChain), RuntimeError> {
|
||||
let tx_ro = tx_ro.get_or_try(|| env_inner.tx_ro())?;
|
||||
let tables = get_tables!(env_inner, tx_ro, tables)?.as_ref();
|
||||
|
||||
let id = PreRctOutputId {
|
||||
amount,
|
||||
amount_index,
|
||||
};
|
||||
|
||||
let output_on_chain = id_to_output_on_chain(&id, tables)?;
|
||||
|
||||
Ok((amount_index, output_on_chain))
|
||||
};
|
||||
|
||||
// Collect results using `rayon`.
|
||||
let map = outputs
|
||||
.into_par_iter()
|
||||
.map(|(amount, amount_index_set)| {
|
||||
Ok((
|
||||
amount,
|
||||
amount_index_set
|
||||
.into_par_iter()
|
||||
.map(|amount_index| inner_map(amount, amount_index))
|
||||
.collect::<Result<HashMap<AmountIndex, OutputOnChain>, RuntimeError>>()?,
|
||||
))
|
||||
})
|
||||
.collect::<Result<HashMap<Amount, HashMap<AmountIndex, OutputOnChain>>, RuntimeError>>()?;
|
||||
|
||||
Ok(Response::Outputs(map))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::NumberOutputsWithAmount`].
|
||||
#[inline]
|
||||
fn number_outputs_with_amount(env: &ConcreteEnv, amounts: Vec<Amount>) -> ResponseResult {
|
||||
// Prepare tx/tables in `ThreadLocal`.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = thread_local(env);
|
||||
let tables = thread_local(env);
|
||||
|
||||
// Cache the amount of RCT outputs once.
|
||||
// INVARIANT: #[cfg] @ lib.rs asserts `usize == u64`
|
||||
#[allow(clippy::cast_possible_truncation)]
|
||||
let num_rct_outputs = {
|
||||
let tx_ro = env_inner.tx_ro()?;
|
||||
let tables = env_inner.open_tables(&tx_ro)?;
|
||||
tables.rct_outputs().len()? as usize
|
||||
};
|
||||
|
||||
// Collect results using `rayon`.
|
||||
let map = amounts
|
||||
.into_par_iter()
|
||||
.map(|amount| {
|
||||
let tx_ro = tx_ro.get_or_try(|| env_inner.tx_ro())?;
|
||||
let tables = get_tables!(env_inner, tx_ro, tables)?.as_ref();
|
||||
|
||||
if amount == 0 {
|
||||
// v2 transactions.
|
||||
Ok((amount, num_rct_outputs))
|
||||
} else {
|
||||
// v1 transactions.
|
||||
match tables.num_outputs().get(&amount) {
|
||||
// INVARIANT: #[cfg] @ lib.rs asserts `usize == u64`
|
||||
#[allow(clippy::cast_possible_truncation)]
|
||||
Ok(count) => Ok((amount, count as usize)),
|
||||
// If we get a request for an `amount` that doesn't exist,
|
||||
// we return `0` instead of an error.
|
||||
Err(RuntimeError::KeyNotFound) => Ok((amount, 0)),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
})
|
||||
.collect::<Result<HashMap<Amount, usize>, RuntimeError>>()?;
|
||||
|
||||
Ok(Response::NumberOutputsWithAmount(map))
|
||||
}
|
||||
|
||||
/// [`ReadRequest::CheckKIsNotSpent`].
|
||||
#[inline]
|
||||
fn check_k_is_not_spent(env: &ConcreteEnv, key_images: HashSet<KeyImage>) -> ResponseResult {
|
||||
// Prepare tx/tables in `ThreadLocal`.
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = thread_local(env);
|
||||
let tables = thread_local(env);
|
||||
|
||||
// Key image check function.
|
||||
let key_image_exists = |key_image| {
|
||||
let tx_ro = tx_ro.get_or_try(|| env_inner.tx_ro())?;
|
||||
let tables = get_tables!(env_inner, tx_ro, tables)?.as_ref();
|
||||
key_image_exists(&key_image, tables.key_images())
|
||||
};
|
||||
|
||||
// FIXME:
|
||||
// Create/use `enum cuprate_types::Exist { Does, DoesNot }`
|
||||
// or similar instead of `bool` for clarity.
|
||||
// <https://github.com/Cuprate/cuprate/pull/113#discussion_r1581536526>
|
||||
//
|
||||
// Collect results using `rayon`.
|
||||
match key_images
|
||||
.into_par_iter()
|
||||
.map(key_image_exists)
|
||||
// If the result is either:
|
||||
// `Ok(true)` => a key image was found, return early
|
||||
// `Err` => an error was found, return early
|
||||
//
|
||||
// Else, `Ok(false)` will continue the iterator.
|
||||
.find_any(|result| !matches!(result, Ok(false)))
|
||||
{
|
||||
None | Some(Ok(false)) => Ok(Response::CheckKIsNotSpent(true)), // Key image was NOT found.
|
||||
Some(Ok(true)) => Ok(Response::CheckKIsNotSpent(false)), // Key image was found.
|
||||
Some(Err(e)) => Err(e), // A database error occurred.
|
||||
}
|
||||
}
|
||||
377
storage/database/src/service/tests.rs
Normal file
377
storage/database/src/service/tests.rs
Normal file
|
|
@ -0,0 +1,377 @@
|
|||
//! `crate::service` tests.
|
||||
//!
|
||||
//! This module contains general tests for the `service` implementation.
|
||||
|
||||
// This is only imported on `#[cfg(test)]` in `mod.rs`.
|
||||
#![allow(clippy::await_holding_lock, clippy::too_many_lines)]
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use std::{
|
||||
collections::{HashMap, HashSet},
|
||||
sync::Arc,
|
||||
};
|
||||
|
||||
use pretty_assertions::assert_eq;
|
||||
use tower::{Service, ServiceExt};
|
||||
|
||||
use cuprate_test_utils::data::{block_v16_tx0, block_v1_tx2, block_v9_tx3};
|
||||
use cuprate_types::{
|
||||
service::{ReadRequest, Response, WriteRequest},
|
||||
OutputOnChain, VerifiedBlockInformation,
|
||||
};
|
||||
|
||||
use crate::{
|
||||
config::ConfigBuilder,
|
||||
ops::{
|
||||
block::{get_block_extended_header_from_height, get_block_info},
|
||||
blockchain::chain_height,
|
||||
output::id_to_output_on_chain,
|
||||
},
|
||||
service::{init, DatabaseReadHandle, DatabaseWriteHandle},
|
||||
tables::{Tables, TablesIter},
|
||||
tests::AssertTableLen,
|
||||
types::{Amount, AmountIndex, PreRctOutputId},
|
||||
ConcreteEnv, DatabaseIter, DatabaseRo, Env, EnvInner, RuntimeError,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Helper functions
|
||||
/// Initialize the `service`.
|
||||
fn init_service() -> (
|
||||
DatabaseReadHandle,
|
||||
DatabaseWriteHandle,
|
||||
Arc<ConcreteEnv>,
|
||||
tempfile::TempDir,
|
||||
) {
|
||||
let tempdir = tempfile::tempdir().unwrap();
|
||||
let config = ConfigBuilder::new()
|
||||
.db_directory(tempdir.path().into())
|
||||
.low_power()
|
||||
.build();
|
||||
let (reader, writer) = init(config).unwrap();
|
||||
let env = reader.env().clone();
|
||||
(reader, writer, env, tempdir)
|
||||
}
|
||||
|
||||
/// This is the template used in the actual test functions below.
|
||||
///
|
||||
/// - Send write request(s)
|
||||
/// - Receive response(s)
|
||||
/// - Assert proper tables were mutated
|
||||
/// - Assert read requests lead to expected responses
|
||||
#[allow(clippy::future_not_send)] // INVARIANT: tests are using a single threaded runtime
|
||||
async fn test_template(
|
||||
// Which block(s) to add?
|
||||
block_fns: &[fn() -> &'static VerifiedBlockInformation],
|
||||
// Total amount of generated coins after the block(s) have been added.
|
||||
cumulative_generated_coins: u64,
|
||||
// What are the table lengths be after the block(s) have been added?
|
||||
assert_table_len: AssertTableLen,
|
||||
) {
|
||||
//----------------------------------------------------------------------- Write requests
|
||||
let (reader, mut writer, env, _tempdir) = init_service();
|
||||
|
||||
let env_inner = env.env_inner();
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
|
||||
// HACK: `add_block()` asserts blocks with non-sequential heights
|
||||
// cannot be added, to get around this, manually edit the block height.
|
||||
for (i, block_fn) in block_fns.iter().enumerate() {
|
||||
let mut block = block_fn().clone();
|
||||
block.height = i as u64;
|
||||
|
||||
// Request a block to be written, assert it was written.
|
||||
let request = WriteRequest::WriteBlock(block);
|
||||
let response_channel = writer.call(request);
|
||||
let response = response_channel.await.unwrap();
|
||||
assert_eq!(response, Response::WriteBlockOk);
|
||||
}
|
||||
|
||||
//----------------------------------------------------------------------- Reset the transaction
|
||||
drop(tables);
|
||||
drop(tx_ro);
|
||||
let tx_ro = env_inner.tx_ro().unwrap();
|
||||
let tables = env_inner.open_tables(&tx_ro).unwrap();
|
||||
|
||||
//----------------------------------------------------------------------- Assert all table lengths are correct
|
||||
assert_table_len.assert(&tables);
|
||||
|
||||
//----------------------------------------------------------------------- Read request prep
|
||||
// Next few lines are just for preparing the expected responses,
|
||||
// see further below for usage.
|
||||
|
||||
let extended_block_header_0 = Ok(Response::BlockExtendedHeader(
|
||||
get_block_extended_header_from_height(&0, &tables).unwrap(),
|
||||
));
|
||||
|
||||
let extended_block_header_1 = if block_fns.len() > 1 {
|
||||
Ok(Response::BlockExtendedHeader(
|
||||
get_block_extended_header_from_height(&1, &tables).unwrap(),
|
||||
))
|
||||
} else {
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
};
|
||||
|
||||
let block_hash_0 = Ok(Response::BlockHash(
|
||||
get_block_info(&0, tables.block_infos()).unwrap().block_hash,
|
||||
));
|
||||
|
||||
let block_hash_1 = if block_fns.len() > 1 {
|
||||
Ok(Response::BlockHash(
|
||||
get_block_info(&1, tables.block_infos()).unwrap().block_hash,
|
||||
))
|
||||
} else {
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
};
|
||||
|
||||
let range_0_1 = Ok(Response::BlockExtendedHeaderInRange(vec![
|
||||
get_block_extended_header_from_height(&0, &tables).unwrap(),
|
||||
]));
|
||||
|
||||
let range_0_2 = if block_fns.len() >= 2 {
|
||||
Ok(Response::BlockExtendedHeaderInRange(vec![
|
||||
get_block_extended_header_from_height(&0, &tables).unwrap(),
|
||||
get_block_extended_header_from_height(&1, &tables).unwrap(),
|
||||
]))
|
||||
} else {
|
||||
Err(RuntimeError::KeyNotFound)
|
||||
};
|
||||
|
||||
let chain_height = {
|
||||
let height = chain_height(tables.block_heights()).unwrap();
|
||||
let block_info = get_block_info(&height.saturating_sub(1), tables.block_infos()).unwrap();
|
||||
Ok(Response::ChainHeight(height, block_info.block_hash))
|
||||
};
|
||||
|
||||
let cumulative_generated_coins = Ok(Response::GeneratedCoins(cumulative_generated_coins));
|
||||
|
||||
let num_req = tables
|
||||
.outputs_iter()
|
||||
.keys()
|
||||
.unwrap()
|
||||
.map(Result::unwrap)
|
||||
.map(|key| key.amount)
|
||||
.collect::<Vec<Amount>>();
|
||||
|
||||
let num_resp = Ok(Response::NumberOutputsWithAmount(
|
||||
num_req
|
||||
.iter()
|
||||
.map(|amount| match tables.num_outputs().get(amount) {
|
||||
// INVARIANT: #[cfg] @ lib.rs asserts `usize == u64`
|
||||
#[allow(clippy::cast_possible_truncation)]
|
||||
Ok(count) => (*amount, count as usize),
|
||||
Err(RuntimeError::KeyNotFound) => (*amount, 0),
|
||||
Err(e) => panic!("{e:?}"),
|
||||
})
|
||||
.collect::<HashMap<Amount, usize>>(),
|
||||
));
|
||||
|
||||
// Contains a fake non-spent key-image.
|
||||
let ki_req = HashSet::from([[0; 32]]);
|
||||
let ki_resp = Ok(Response::CheckKIsNotSpent(true));
|
||||
|
||||
//----------------------------------------------------------------------- Assert expected response
|
||||
// Assert read requests lead to the expected responses.
|
||||
for (request, expected_response) in [
|
||||
(ReadRequest::BlockExtendedHeader(0), extended_block_header_0),
|
||||
(ReadRequest::BlockExtendedHeader(1), extended_block_header_1),
|
||||
(ReadRequest::BlockHash(0), block_hash_0),
|
||||
(ReadRequest::BlockHash(1), block_hash_1),
|
||||
(ReadRequest::BlockExtendedHeaderInRange(0..1), range_0_1),
|
||||
(ReadRequest::BlockExtendedHeaderInRange(0..2), range_0_2),
|
||||
(ReadRequest::ChainHeight, chain_height),
|
||||
(ReadRequest::GeneratedCoins, cumulative_generated_coins),
|
||||
(ReadRequest::NumberOutputsWithAmount(num_req), num_resp),
|
||||
(ReadRequest::CheckKIsNotSpent(ki_req), ki_resp),
|
||||
] {
|
||||
let response = reader.clone().oneshot(request).await;
|
||||
println!("response: {response:#?}, expected_response: {expected_response:#?}");
|
||||
match response {
|
||||
Ok(resp) => assert_eq!(resp, expected_response.unwrap()),
|
||||
Err(_) => assert!(matches!(response, _expected_response)),
|
||||
}
|
||||
}
|
||||
|
||||
//----------------------------------------------------------------------- Key image checks
|
||||
// Assert each key image we inserted comes back as "spent".
|
||||
for key_image in tables.key_images_iter().keys().unwrap() {
|
||||
let key_image = key_image.unwrap();
|
||||
let request = ReadRequest::CheckKIsNotSpent(HashSet::from([key_image]));
|
||||
let response = reader.clone().oneshot(request).await;
|
||||
println!("response: {response:#?}, key_image: {key_image:#?}");
|
||||
assert_eq!(response.unwrap(), Response::CheckKIsNotSpent(false));
|
||||
}
|
||||
|
||||
//----------------------------------------------------------------------- Output checks
|
||||
// Create the map of amounts and amount indices.
|
||||
//
|
||||
// FIXME: There's definitely a better way to map
|
||||
// `Vec<PreRctOutputId>` -> `HashMap<u64, HashSet<u64>>`
|
||||
let (map, output_count) = {
|
||||
let mut ids = tables
|
||||
.outputs_iter()
|
||||
.keys()
|
||||
.unwrap()
|
||||
.map(Result::unwrap)
|
||||
.collect::<Vec<PreRctOutputId>>();
|
||||
|
||||
ids.extend(
|
||||
tables
|
||||
.rct_outputs_iter()
|
||||
.keys()
|
||||
.unwrap()
|
||||
.map(Result::unwrap)
|
||||
.map(|amount_index| PreRctOutputId {
|
||||
amount: 0,
|
||||
amount_index,
|
||||
}),
|
||||
);
|
||||
|
||||
// Used later to compare the amount of Outputs
|
||||
// returned in the Response is equal to the amount
|
||||
// we asked for.
|
||||
let output_count = ids.len();
|
||||
|
||||
let mut map = HashMap::<Amount, HashSet<AmountIndex>>::new();
|
||||
for id in ids {
|
||||
map.entry(id.amount)
|
||||
.and_modify(|set| {
|
||||
set.insert(id.amount_index);
|
||||
})
|
||||
.or_insert_with(|| HashSet::from([id.amount_index]));
|
||||
}
|
||||
|
||||
(map, output_count)
|
||||
};
|
||||
|
||||
// Map `Output` -> `OutputOnChain`
|
||||
// This is the expected output from the `Response`.
|
||||
let outputs_on_chain = map
|
||||
.iter()
|
||||
.flat_map(|(amount, amount_index_set)| {
|
||||
amount_index_set.iter().map(|amount_index| {
|
||||
let id = PreRctOutputId {
|
||||
amount: *amount,
|
||||
amount_index: *amount_index,
|
||||
};
|
||||
id_to_output_on_chain(&id, &tables).unwrap()
|
||||
})
|
||||
})
|
||||
.collect::<Vec<OutputOnChain>>();
|
||||
|
||||
// Send a request for every output we inserted before.
|
||||
let request = ReadRequest::Outputs(map.clone());
|
||||
let response = reader.clone().oneshot(request).await;
|
||||
println!("Response::Outputs response: {response:#?}");
|
||||
let Ok(Response::Outputs(response)) = response else {
|
||||
panic!("{response:#?}")
|
||||
};
|
||||
|
||||
// Assert amount of `Amount`'s are the same.
|
||||
assert_eq!(map.len(), response.len());
|
||||
|
||||
// Assert we get back the same map of
|
||||
// `Amount`'s and `AmountIndex`'s.
|
||||
let mut response_output_count = 0;
|
||||
for (amount, output_map) in response {
|
||||
let amount_index_set = map.get(&amount).unwrap();
|
||||
|
||||
for (amount_index, output) in output_map {
|
||||
response_output_count += 1;
|
||||
assert!(amount_index_set.contains(&amount_index));
|
||||
assert!(outputs_on_chain.contains(&output));
|
||||
}
|
||||
}
|
||||
|
||||
// Assert the amount of `Output`'s returned is as expected.
|
||||
let table_output_len = tables.outputs().len().unwrap() + tables.rct_outputs().len().unwrap();
|
||||
assert_eq!(output_count as u64, table_output_len);
|
||||
assert_eq!(output_count, response_output_count);
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
/// Simply `init()` the service and then drop it.
|
||||
///
|
||||
/// If this test fails, something is very wrong.
|
||||
#[test]
|
||||
fn init_drop() {
|
||||
let (_reader, _writer, _env, _tempdir) = init_service();
|
||||
}
|
||||
|
||||
/// Assert write/read correctness of [`block_v1_tx2`].
|
||||
#[tokio::test]
|
||||
async fn v1_tx2() {
|
||||
test_template(
|
||||
&[block_v1_tx2],
|
||||
14_535_350_982_449,
|
||||
AssertTableLen {
|
||||
block_infos: 1,
|
||||
block_blobs: 1,
|
||||
block_heights: 1,
|
||||
key_images: 65,
|
||||
num_outputs: 41,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 111,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 0,
|
||||
tx_blobs: 3,
|
||||
tx_ids: 3,
|
||||
tx_heights: 3,
|
||||
tx_unlock_time: 1,
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Assert write/read correctness of [`block_v9_tx3`].
|
||||
#[tokio::test]
|
||||
async fn v9_tx3() {
|
||||
test_template(
|
||||
&[block_v9_tx3],
|
||||
3_403_774_022_163,
|
||||
AssertTableLen {
|
||||
block_infos: 1,
|
||||
block_blobs: 1,
|
||||
block_heights: 1,
|
||||
key_images: 4,
|
||||
num_outputs: 0,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 0,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 7,
|
||||
tx_blobs: 4,
|
||||
tx_ids: 4,
|
||||
tx_heights: 4,
|
||||
tx_unlock_time: 1,
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Assert write/read correctness of [`block_v16_tx0`].
|
||||
#[tokio::test]
|
||||
async fn v16_tx0() {
|
||||
test_template(
|
||||
&[block_v16_tx0],
|
||||
600_000_000_000,
|
||||
AssertTableLen {
|
||||
block_infos: 1,
|
||||
block_blobs: 1,
|
||||
block_heights: 1,
|
||||
key_images: 0,
|
||||
num_outputs: 0,
|
||||
pruned_tx_blobs: 0,
|
||||
prunable_hashes: 0,
|
||||
outputs: 0,
|
||||
prunable_tx_blobs: 0,
|
||||
rct_outputs: 1,
|
||||
tx_blobs: 1,
|
||||
tx_ids: 1,
|
||||
tx_heights: 1,
|
||||
tx_unlock_time: 1,
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
31
storage/database/src/service/types.rs
Normal file
31
storage/database/src/service/types.rs
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
//! Database service type aliases.
|
||||
//!
|
||||
//! Only used internally for our `tower::Service` impls.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Use
|
||||
use futures::channel::oneshot::Sender;
|
||||
|
||||
use cuprate_helper::asynch::InfallibleOneshotReceiver;
|
||||
use cuprate_types::service::Response;
|
||||
|
||||
use crate::error::RuntimeError;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Types
|
||||
/// The actual type of the response.
|
||||
///
|
||||
/// Either our [`Response`], or a database error occurred.
|
||||
pub(super) type ResponseResult = Result<Response, RuntimeError>;
|
||||
|
||||
/// The `Receiver` channel that receives the read response.
|
||||
///
|
||||
/// This is owned by the caller (the reader/writer thread)
|
||||
/// who `.await`'s for the response.
|
||||
///
|
||||
/// The channel itself should never fail,
|
||||
/// but the actual database operation might.
|
||||
pub(super) type ResponseReceiver = InfallibleOneshotReceiver<ResponseResult>;
|
||||
|
||||
/// The `Sender` channel for the response.
|
||||
///
|
||||
/// The database reader/writer thread uses this to send the database result to the caller.
|
||||
pub(super) type ResponseSender = Sender<ResponseResult>;
|
||||
245
storage/database/src/service/write.rs
Normal file
245
storage/database/src/service/write.rs
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
//! Database writer thread definitions and logic.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{
|
||||
sync::Arc,
|
||||
task::{Context, Poll},
|
||||
};
|
||||
|
||||
use futures::channel::oneshot;
|
||||
|
||||
use cuprate_helper::asynch::InfallibleOneshotReceiver;
|
||||
use cuprate_types::{
|
||||
service::{Response, WriteRequest},
|
||||
VerifiedBlockInformation,
|
||||
};
|
||||
|
||||
use crate::{
|
||||
env::{Env, EnvInner},
|
||||
error::RuntimeError,
|
||||
service::types::{ResponseReceiver, ResponseResult, ResponseSender},
|
||||
transaction::TxRw,
|
||||
ConcreteEnv,
|
||||
};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Constants
|
||||
/// Name of the writer thread.
|
||||
const WRITER_THREAD_NAME: &str = concat!(module_path!(), "::DatabaseWriter");
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseWriteHandle
|
||||
/// Write handle to the database.
|
||||
///
|
||||
/// This is handle that allows `async`hronously writing to the database,
|
||||
/// it is not [`Clone`]able as there is only ever 1 place within Cuprate
|
||||
/// that writes.
|
||||
///
|
||||
/// Calling [`tower::Service::call`] with a [`DatabaseWriteHandle`] & [`WriteRequest`]
|
||||
/// will return an `async`hronous channel that can be `.await`ed upon
|
||||
/// to receive the corresponding [`Response`].
|
||||
#[derive(Debug)]
|
||||
pub struct DatabaseWriteHandle {
|
||||
/// Sender channel to the database write thread-pool.
|
||||
///
|
||||
/// We provide the response channel for the thread-pool.
|
||||
pub(super) sender: crossbeam::channel::Sender<(WriteRequest, ResponseSender)>,
|
||||
}
|
||||
|
||||
impl DatabaseWriteHandle {
|
||||
/// Initialize the single `DatabaseWriter` thread.
|
||||
#[cold]
|
||||
#[inline(never)] // Only called once.
|
||||
pub(super) fn init(env: Arc<ConcreteEnv>) -> Self {
|
||||
// Initialize `Request/Response` channels.
|
||||
let (sender, receiver) = crossbeam::channel::unbounded();
|
||||
|
||||
// Spawn the writer.
|
||||
std::thread::Builder::new()
|
||||
.name(WRITER_THREAD_NAME.into())
|
||||
.spawn(move || {
|
||||
let this = DatabaseWriter { receiver, env };
|
||||
DatabaseWriter::main(this);
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
Self { sender }
|
||||
}
|
||||
}
|
||||
|
||||
impl tower::Service<WriteRequest> for DatabaseWriteHandle {
|
||||
type Response = Response;
|
||||
type Error = RuntimeError;
|
||||
type Future = ResponseReceiver;
|
||||
|
||||
#[inline]
|
||||
fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
|
||||
Poll::Ready(Ok(()))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn call(&mut self, request: WriteRequest) -> Self::Future {
|
||||
// Response channel we `.await` on.
|
||||
let (response_sender, receiver) = oneshot::channel();
|
||||
|
||||
// Send the write request.
|
||||
self.sender.send((request, response_sender)).unwrap();
|
||||
|
||||
InfallibleOneshotReceiver::from(receiver)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- DatabaseWriter
|
||||
/// The single database writer thread.
|
||||
pub(super) struct DatabaseWriter {
|
||||
/// Receiver side of the database request channel.
|
||||
///
|
||||
/// Any caller can send some requests to this channel.
|
||||
/// They send them alongside another `Response` channel,
|
||||
/// which we will eventually send to.
|
||||
receiver: crossbeam::channel::Receiver<(WriteRequest, ResponseSender)>,
|
||||
|
||||
/// Access to the database.
|
||||
env: Arc<ConcreteEnv>,
|
||||
}
|
||||
|
||||
impl Drop for DatabaseWriter {
|
||||
fn drop(&mut self) {
|
||||
// TODO: log the writer thread has exited?
|
||||
}
|
||||
}
|
||||
|
||||
impl DatabaseWriter {
|
||||
/// The `DatabaseWriter`'s main function.
|
||||
///
|
||||
/// The writer just loops in this function, handling requests forever
|
||||
/// until the request channel is dropped or a panic occurs.
|
||||
#[cold]
|
||||
#[inline(never)] // Only called once.
|
||||
fn main(self) {
|
||||
// 1. Hang on request channel
|
||||
// 2. Map request to some database function
|
||||
// 3. Execute that function, get the result
|
||||
// 4. Return the result via channel
|
||||
'main: loop {
|
||||
let Ok((request, response_sender)) = self.receiver.recv() else {
|
||||
// If this receive errors, it means that the channel is empty
|
||||
// and disconnected, meaning the other side (all senders) have
|
||||
// been dropped. This means "shutdown", and we return here to
|
||||
// exit the thread.
|
||||
//
|
||||
// Since the channel is empty, it means we've also processed
|
||||
// all requests. Since it is disconnected, it means future
|
||||
// ones cannot come in.
|
||||
return;
|
||||
};
|
||||
|
||||
/// How many times should we retry handling the request on resize errors?
|
||||
///
|
||||
/// This is 1 on automatically resizing databases, meaning there is only 1 iteration.
|
||||
const REQUEST_RETRY_LIMIT: usize = if ConcreteEnv::MANUAL_RESIZE { 3 } else { 1 };
|
||||
|
||||
// Map [`Request`]'s to specific database functions.
|
||||
//
|
||||
// Both will:
|
||||
// 1. Map the request to a function
|
||||
// 2. Call the function
|
||||
// 3. (manual resize only) If resize is needed, resize and retry
|
||||
// 4. (manual resize only) Redo step {1, 2}
|
||||
// 5. Send the function's `Result` back to the requester
|
||||
//
|
||||
// FIXME: there's probably a more elegant way
|
||||
// to represent this retry logic with recursive
|
||||
// functions instead of a loop.
|
||||
'retry: for retry in 0..REQUEST_RETRY_LIMIT {
|
||||
// FIXME: will there be more than 1 write request?
|
||||
// this won't have to be an enum.
|
||||
let response = match &request {
|
||||
WriteRequest::WriteBlock(block) => write_block(&self.env, block),
|
||||
};
|
||||
|
||||
// If the database needs to resize, do so.
|
||||
if ConcreteEnv::MANUAL_RESIZE && matches!(response, Err(RuntimeError::ResizeNeeded))
|
||||
{
|
||||
// If this is the last iteration of the outer `for` loop and we
|
||||
// encounter a resize error _again_, it means something is wrong.
|
||||
assert_ne!(
|
||||
retry, REQUEST_RETRY_LIMIT,
|
||||
"database resize failed maximum of {REQUEST_RETRY_LIMIT} times"
|
||||
);
|
||||
|
||||
// Resize the map, and retry the request handling loop.
|
||||
//
|
||||
// FIXME:
|
||||
// We could pass in custom resizes to account for
|
||||
// batches, i.e., we're about to add ~5GB of data,
|
||||
// add that much instead of the default 1GB.
|
||||
// <https://github.com/monero-project/monero/blob/059028a30a8ae9752338a7897329fe8012a310d5/src/blockchain_db/lmdb/db_lmdb.cpp#L665-L695>
|
||||
let old = self.env.current_map_size();
|
||||
let new = self.env.resize_map(None);
|
||||
|
||||
// TODO: use tracing.
|
||||
println!("resizing database memory map, old: {old}B, new: {new}B");
|
||||
|
||||
// Try handling the request again.
|
||||
continue 'retry;
|
||||
}
|
||||
|
||||
// Automatically resizing databases should not be returning a resize error.
|
||||
#[cfg(debug_assertions)]
|
||||
if !ConcreteEnv::MANUAL_RESIZE {
|
||||
assert!(
|
||||
!matches!(response, Err(RuntimeError::ResizeNeeded)),
|
||||
"auto-resizing database returned a ResizeNeeded error"
|
||||
);
|
||||
}
|
||||
|
||||
// Send the response back, whether if it's an `Ok` or `Err`.
|
||||
if let Err(e) = response_sender.send(response) {
|
||||
// TODO: use tracing.
|
||||
println!("database writer failed to send response: {e:?}");
|
||||
}
|
||||
|
||||
continue 'main;
|
||||
}
|
||||
|
||||
// Above retry loop should either:
|
||||
// - continue to the next ['main] loop or...
|
||||
// - ...retry until panic
|
||||
unreachable!();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Handler functions
|
||||
// These are the actual functions that do stuff according to the incoming [`Request`].
|
||||
//
|
||||
// Each function name is a 1-1 mapping (from CamelCase -> snake_case) to
|
||||
// the enum variant name, e.g: `BlockExtendedHeader` -> `block_extended_header`.
|
||||
//
|
||||
// Each function will return the [`Response`] that we
|
||||
// should send back to the caller in [`map_request()`].
|
||||
|
||||
/// [`WriteRequest::WriteBlock`].
|
||||
#[inline]
|
||||
fn write_block(env: &ConcreteEnv, block: &VerifiedBlockInformation) -> ResponseResult {
|
||||
let env_inner = env.env_inner();
|
||||
let tx_rw = env_inner.tx_rw()?;
|
||||
|
||||
let result = {
|
||||
let mut tables_mut = env_inner.open_tables_mut(&tx_rw)?;
|
||||
crate::ops::block::add_block(block, &mut tables_mut)
|
||||
};
|
||||
|
||||
match result {
|
||||
Ok(()) => {
|
||||
TxRw::commit(tx_rw)?;
|
||||
Ok(Response::WriteBlockOk)
|
||||
}
|
||||
Err(e) => {
|
||||
// INVARIANT: ensure database atomicity by aborting
|
||||
// the transaction on `add_block()` failures.
|
||||
TxRw::abort(tx_rw)
|
||||
.expect("could not maintain database atomicity by aborting write transaction");
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
347
storage/database/src/storable.rs
Normal file
347
storage/database/src/storable.rs
Normal file
|
|
@ -0,0 +1,347 @@
|
|||
//! (De)serialization for table keys & values.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
use std::{borrow::Borrow, fmt::Debug};
|
||||
|
||||
use bytemuck::Pod;
|
||||
use bytes::Bytes;
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Storable
|
||||
/// A type that can be stored in the database.
|
||||
///
|
||||
/// All keys and values in the database must be able
|
||||
/// to be (de)serialized into/from raw bytes (`[u8]`).
|
||||
///
|
||||
/// This trait represents types that can be **perfectly**
|
||||
/// casted/represented as raw bytes.
|
||||
///
|
||||
/// ## `bytemuck`
|
||||
/// Any type that implements:
|
||||
/// - [`bytemuck::Pod`]
|
||||
/// - [`Debug`]
|
||||
///
|
||||
/// will automatically implement [`Storable`].
|
||||
///
|
||||
/// This includes:
|
||||
/// - Most primitive types
|
||||
/// - All types in [`tables`](crate::tables)
|
||||
///
|
||||
/// See [`StorableVec`] & [`StorableBytes`] for storing slices of `T: Storable`.
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::*;
|
||||
/// # use std::borrow::*;
|
||||
/// let number: u64 = 0;
|
||||
///
|
||||
/// // Into bytes.
|
||||
/// let into = Storable::as_bytes(&number);
|
||||
/// assert_eq!(into, &[0; 8]);
|
||||
///
|
||||
/// // From bytes.
|
||||
/// let from: u64 = Storable::from_bytes(&into);
|
||||
/// assert_eq!(from, number);
|
||||
/// ```
|
||||
///
|
||||
/// ## Invariants
|
||||
/// No function in this trait is expected to panic.
|
||||
///
|
||||
/// The byte conversions must execute flawlessly.
|
||||
///
|
||||
/// ## Endianness
|
||||
/// This trait doesn't currently care about endianness.
|
||||
///
|
||||
/// Bytes are (de)serialized as-is, and `bytemuck`
|
||||
/// types are architecture-dependant.
|
||||
///
|
||||
/// Most likely, the bytes are little-endian, however
|
||||
/// that cannot be relied upon when using this trait.
|
||||
pub trait Storable: Debug {
|
||||
/// Is this type fixed width in byte length?
|
||||
///
|
||||
/// I.e., when converting `Self` to bytes, is it
|
||||
/// represented with a fixed length array of bytes?
|
||||
///
|
||||
/// # `Some`
|
||||
/// This should be `Some(usize)` on types like:
|
||||
/// - `u8`
|
||||
/// - `u64`
|
||||
/// - `i32`
|
||||
///
|
||||
/// where the byte length is known.
|
||||
///
|
||||
/// # `None`
|
||||
/// This should be `None` on any variable-length type like:
|
||||
/// - `str`
|
||||
/// - `[u8]`
|
||||
/// - `Vec<u8>`
|
||||
///
|
||||
/// # Examples
|
||||
/// ```rust
|
||||
/// # use cuprate_database::*;
|
||||
/// assert_eq!(<()>::BYTE_LENGTH, Some(0));
|
||||
/// assert_eq!(u8::BYTE_LENGTH, Some(1));
|
||||
/// assert_eq!(u16::BYTE_LENGTH, Some(2));
|
||||
/// assert_eq!(u32::BYTE_LENGTH, Some(4));
|
||||
/// assert_eq!(u64::BYTE_LENGTH, Some(8));
|
||||
/// assert_eq!(i8::BYTE_LENGTH, Some(1));
|
||||
/// assert_eq!(i16::BYTE_LENGTH, Some(2));
|
||||
/// assert_eq!(i32::BYTE_LENGTH, Some(4));
|
||||
/// assert_eq!(i64::BYTE_LENGTH, Some(8));
|
||||
/// assert_eq!(StorableVec::<u8>::BYTE_LENGTH, None);
|
||||
/// assert_eq!(StorableVec::<u64>::BYTE_LENGTH, None);
|
||||
/// ```
|
||||
const BYTE_LENGTH: Option<usize>;
|
||||
|
||||
/// Return `self` in byte form.
|
||||
fn as_bytes(&self) -> &[u8];
|
||||
|
||||
/// Create an owned [`Self`] from bytes.
|
||||
///
|
||||
/// # Blanket implementation
|
||||
/// The blanket implementation that covers all types used
|
||||
/// by `cuprate_database` will simply bitwise copy `bytes`
|
||||
/// into `Self`.
|
||||
///
|
||||
/// The bytes do not have be correctly aligned.
|
||||
fn from_bytes(bytes: &[u8]) -> Self;
|
||||
}
|
||||
|
||||
impl<T> Storable for T
|
||||
where
|
||||
Self: Pod + Debug,
|
||||
{
|
||||
const BYTE_LENGTH: Option<usize> = Some(std::mem::size_of::<T>());
|
||||
|
||||
#[inline]
|
||||
fn as_bytes(&self) -> &[u8] {
|
||||
bytemuck::bytes_of(self)
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn from_bytes(bytes: &[u8]) -> T {
|
||||
bytemuck::pod_read_unaligned(bytes)
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- StorableVec
|
||||
/// A [`Storable`] vector of `T: Storable`.
|
||||
///
|
||||
/// This is a wrapper around `Vec<T> where T: Storable`.
|
||||
///
|
||||
/// Slice types are owned both:
|
||||
/// - when returned from the database
|
||||
/// - in `put()`
|
||||
///
|
||||
/// This is needed as `impl Storable for Vec<T>` runs into impl conflicts.
|
||||
///
|
||||
/// # Example
|
||||
/// ```rust
|
||||
/// # use cuprate_database::*;
|
||||
/// //---------------------------------------------------- u8
|
||||
/// let vec: StorableVec<u8> = StorableVec(vec![0,1]);
|
||||
///
|
||||
/// // Into bytes.
|
||||
/// let into = Storable::as_bytes(&vec);
|
||||
/// assert_eq!(into, &[0,1]);
|
||||
///
|
||||
/// // From bytes.
|
||||
/// let from: StorableVec<u8> = Storable::from_bytes(&into);
|
||||
/// assert_eq!(from, vec);
|
||||
///
|
||||
/// //---------------------------------------------------- u64
|
||||
/// let vec: StorableVec<u64> = StorableVec(vec![0,1]);
|
||||
///
|
||||
/// // Into bytes.
|
||||
/// let into = Storable::as_bytes(&vec);
|
||||
/// assert_eq!(into, &[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0]);
|
||||
///
|
||||
/// // From bytes.
|
||||
/// let from: StorableVec<u64> = Storable::from_bytes(&into);
|
||||
/// assert_eq!(from, vec);
|
||||
/// ```
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, bytemuck::TransparentWrapper)]
|
||||
#[repr(transparent)]
|
||||
pub struct StorableVec<T>(pub Vec<T>);
|
||||
|
||||
impl<T> Storable for StorableVec<T>
|
||||
where
|
||||
T: Pod + Debug,
|
||||
{
|
||||
const BYTE_LENGTH: Option<usize> = None;
|
||||
|
||||
/// Casts the inner `Vec<T>` directly as bytes.
|
||||
#[inline]
|
||||
fn as_bytes(&self) -> &[u8] {
|
||||
bytemuck::must_cast_slice(&self.0)
|
||||
}
|
||||
|
||||
/// This always allocates a new `Vec<T>`,
|
||||
/// casting `bytes` into a vector of type `T`.
|
||||
#[inline]
|
||||
fn from_bytes(bytes: &[u8]) -> Self {
|
||||
Self(bytemuck::pod_collect_to_vec(bytes))
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> std::ops::Deref for StorableVec<T> {
|
||||
type Target = [T];
|
||||
#[inline]
|
||||
fn deref(&self) -> &[T] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Borrow<[T]> for StorableVec<T> {
|
||||
#[inline]
|
||||
fn borrow(&self) -> &[T] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- StorableBytes
|
||||
/// A [`Storable`] version of [`Bytes`].
|
||||
///
|
||||
/// ```rust
|
||||
/// # use cuprate_database::*;
|
||||
/// # use bytes::Bytes;
|
||||
/// let bytes: StorableBytes = StorableBytes(Bytes::from_static(&[0,1]));
|
||||
///
|
||||
/// // Into bytes.
|
||||
/// let into = Storable::as_bytes(&bytes);
|
||||
/// assert_eq!(into, &[0,1]);
|
||||
///
|
||||
/// // From bytes.
|
||||
/// let from: StorableBytes = Storable::from_bytes(&into);
|
||||
/// assert_eq!(from, bytes);
|
||||
/// ```
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
#[repr(transparent)]
|
||||
pub struct StorableBytes(pub Bytes);
|
||||
|
||||
impl Storable for StorableBytes {
|
||||
const BYTE_LENGTH: Option<usize> = None;
|
||||
|
||||
#[inline]
|
||||
fn as_bytes(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
|
||||
/// This always allocates a new `Bytes`.
|
||||
#[inline]
|
||||
fn from_bytes(bytes: &[u8]) -> Self {
|
||||
Self(Bytes::copy_from_slice(bytes))
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Deref for StorableBytes {
|
||||
type Target = [u8];
|
||||
#[inline]
|
||||
fn deref(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl Borrow<[u8]> for StorableBytes {
|
||||
#[inline]
|
||||
fn borrow(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
use super::*;
|
||||
|
||||
/// Serialize, deserialize, and compare that
|
||||
/// the intermediate/end results are correct.
|
||||
fn test_storable<const LEN: usize, T>(
|
||||
// The primitive number function that
|
||||
// converts the number into little endian bytes,
|
||||
// e.g `u8::to_le_bytes`.
|
||||
to_le_bytes: fn(T) -> [u8; LEN],
|
||||
// A `Vec` of the numbers to test.
|
||||
t: Vec<T>,
|
||||
) where
|
||||
T: Storable + Debug + Copy + PartialEq,
|
||||
{
|
||||
for t in t {
|
||||
let expected_bytes = to_le_bytes(t);
|
||||
|
||||
println!("testing: {t:?}, expected_bytes: {expected_bytes:?}");
|
||||
|
||||
// (De)serialize.
|
||||
let se: &[u8] = Storable::as_bytes(&t);
|
||||
let de = <T as Storable>::from_bytes(se);
|
||||
|
||||
println!("serialized: {se:?}, deserialized: {de:?}\n");
|
||||
|
||||
// Assert we wrote correct amount of bytes.
|
||||
if T::BYTE_LENGTH.is_some() {
|
||||
assert_eq!(se.len(), expected_bytes.len());
|
||||
}
|
||||
// Assert the data is the same.
|
||||
assert_eq!(de, t);
|
||||
}
|
||||
}
|
||||
|
||||
/// Create all the float tests.
|
||||
macro_rules! test_float {
|
||||
($(
|
||||
$float:ident // The float type.
|
||||
),* $(,)?) => {
|
||||
$(
|
||||
#[test]
|
||||
fn $float() {
|
||||
test_storable(
|
||||
$float::to_le_bytes,
|
||||
vec![
|
||||
-1.0,
|
||||
0.0,
|
||||
1.0,
|
||||
$float::MIN,
|
||||
$float::MAX,
|
||||
$float::INFINITY,
|
||||
$float::NEG_INFINITY,
|
||||
],
|
||||
);
|
||||
}
|
||||
)*
|
||||
};
|
||||
}
|
||||
|
||||
test_float! {
|
||||
f32,
|
||||
f64,
|
||||
}
|
||||
|
||||
/// Create all the (un)signed number tests.
|
||||
/// u8 -> u128, i8 -> i128.
|
||||
macro_rules! test_unsigned {
|
||||
($(
|
||||
$number:ident // The integer type.
|
||||
),* $(,)?) => {
|
||||
$(
|
||||
#[test]
|
||||
fn $number() {
|
||||
test_storable($number::to_le_bytes, vec![$number::MIN, 0, 1, $number::MAX]);
|
||||
}
|
||||
)*
|
||||
};
|
||||
}
|
||||
|
||||
test_unsigned! {
|
||||
u8,
|
||||
u16,
|
||||
u32,
|
||||
u64,
|
||||
u128,
|
||||
usize,
|
||||
i8,
|
||||
i16,
|
||||
i32,
|
||||
i64,
|
||||
i128,
|
||||
isize,
|
||||
}
|
||||
}
|
||||
31
storage/database/src/table.rs
Normal file
31
storage/database/src/table.rs
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
//! Database table abstraction; `trait Table`.
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Import
|
||||
|
||||
use crate::{key::Key, storable::Storable};
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Table
|
||||
/// Database table metadata.
|
||||
///
|
||||
/// Purely compile time information for database tables.
|
||||
///
|
||||
/// ## Sealed
|
||||
/// This trait is [`Sealed`](https://rust-lang.github.io/api-guidelines/future-proofing.html#sealed-traits-protect-against-downstream-implementations-c-sealed).
|
||||
///
|
||||
/// It is only implemented on the types inside [`tables`][crate::tables].
|
||||
pub trait Table: crate::tables::private::Sealed + 'static {
|
||||
/// Name of the database table.
|
||||
const NAME: &'static str;
|
||||
|
||||
/// Primary key type.
|
||||
type Key: Key + 'static;
|
||||
|
||||
/// Value type.
|
||||
type Value: Storable + 'static;
|
||||
}
|
||||
|
||||
//---------------------------------------------------------------------------------------------------- Tests
|
||||
#[cfg(test)]
|
||||
mod test {
|
||||
// use super::*;
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue