Commit graph

229 commits

Author SHA1 Message Date
hinto.janai
7e8aae016c
heed: use Mutex for HeedTableRo's read tx 2024-04-28 21:00:14 -04:00
hinto.janai
8b8bb6342f
read: use ThreadLocal for both backends 2024-04-28 20:45:05 -04:00
hinto.janai
bd246eb9a3
read: fix variable name 2024-04-28 16:10:39 -04:00
hinto.janai
5b227dbe4f
read: rct outputs 2024-04-28 15:55:18 -04:00
hinto.janai
98157c9347
free: use u64_to_timelock() 2024-04-28 13:54:08 -04:00
hinto.janai
8015267db3
helper: add timelock <-> u64 mapping functions 2024-04-28 13:53:20 -04:00
hinto.janai
118fd9a9e1
service: check output value correctness in tests 2024-04-26 21:01:57 -04:00
hinto.janai
a347f7fd9d
read: move Output mapping to crate::free
it's needed in tests too
2024-04-26 20:56:47 -04:00
hinto.janai
0e4ec3958d
read: fix ThreadLocal macros for redb 2024-04-26 20:33:12 -04:00
hinto.janai
64938bfec0
small fixes 2024-04-26 20:11:24 -04:00
hinto.janai
5f82c6aa6b
fix review 2024-04-26 18:02:10 -04:00
hinto.janai
3a172e05ac
small fixes 2024-04-25 20:29:05 -04:00
hinto.janai
9a507e7053
read: use macros for set/getting ThreadLocal's based on backend 2024-04-25 17:43:08 -04:00
hinto.janai
bb99739c59
service: add ReadRequest::Outputs test 2024-04-25 16:27:27 -04:00
hinto.janai
71e47eb64f
tables: fix incorrect doc 2024-04-25 15:54:49 -04:00
hinto.janai
51e9a14fb8
service: fix bad block data in test 2024-04-25 15:41:52 -04:00
hinto.janai
f3c46a109f
Merge branch 'main' into service 2024-04-25 15:39:37 -04:00
hinto-janai
32a44a2503
test-utils: compare data against RPC (#115)
* cargo: add `pretty_assertions`

* test-utils: add tests to compare data against RPC data, and fix

* small fixes

* database: fix `cumulative_generated_coins` from test data

block had wrong data which offset all of the asserts
2024-04-25 19:58:45 +01:00
hinto.janai
889532d1be
service: finish most tests 2024-04-24 19:21:13 -04:00
hinto.janai
034b64463b
Merge branch 'main' into service 2024-04-24 16:53:09 -04:00
hinto-janai
bf6c21c71e
database: split cumulative_difficulty into low/high bits (#114)
* types: split `cumulative_difficulty` into low/high bits

* helper: add `map` module

* database: use `helper`'s cumulative_diff functions

* helper: rename functions

splitting bits isn't necessarily `cumulative_difficulty` specific

* database: fix tests

* helper: docs

* helper: test output is low-endian bits

* helper: docs

* Update helper/src/map.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update helper/src/map.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-24 21:47:48 +01:00
hinto.janai
ae2a0be37e
service: initial tests 2024-04-24 16:19:00 -04:00
hinto.janai
c485b951f3
ops: replace all table len asserts with AssertTableLen 2024-04-24 16:17:28 -04:00
hinto.janai
1e260a4e9c
tests: add AssertTableLen 2024-04-24 16:06:06 -04:00
hinto.janai
ff5cca181b
read: impl number_outputs_with_amount() 2024-04-24 15:26:59 -04:00
hinto.janai
bf7c11ce84
read: flatten indentation, add thread_local() 2024-04-23 21:25:54 -04:00
hinto.janai
26dc7cffaf
read: impl outputs() 2024-04-23 21:14:14 -04:00
hinto.janai
0a01ddda2f
service: remove RwLock, impl some read functions 2024-04-23 18:57:21 -04:00
hinto.janai
6a0a17aead
heed: use read-txn-no-tls for Send read transactions 2024-04-23 18:56:52 -04:00
hinto.janai
ba1d96522f
service: wrap ConcreteEnv in RwLock and doc why 2024-04-23 16:47:12 -04:00
hinto.janai
4dbf7dafa9
read: fix signatures 2024-04-23 14:49:27 -04:00
hinto.janai
64304dc84c
read: impl block fn's 2024-04-23 14:30:00 -04:00
hinto.janai
5d8322dcba
ops: add get_block_info() 2024-04-23 13:32:33 -04:00
hinto.janai
a598fc38d2
write: impl write_block() 2024-04-23 13:20:57 -04:00
hinto-janai
c65eb0a3ca
database: implement ops/ (#102)
* ops: add `trait MoneroR{o,w}`

* update `trait MoneroR{o,w}` bounds

* types: add `BlockInfoLatest` type alias

* block: impl most core functions

* types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916

* fix table type test

* cargo.toml: add `{cuprate-types, monero-serai}`

* add_block: add all other block data

* ops: remove unneeded `block` functions

* env: add `EnvInner::open_db_rw_all()`

* types: fix test

* block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()`

* add `trait Tables[Mut]` and use it in `EnvInner`

* block: use `TablesMut`

* tables: replace manual impl with `define_trait_tables!()`

* tables: docs for `trait Tables[Mut]`

* tables: doc functions

* create `call_fn_on_all_tables_or_early_return!()` macro

* block: cleanup signatures + bodies

* block: more fn's, docs

* block: add `doc_{error,single,bulk}!()`

* remove `ops/monero.rs`

* move `height()` to `ops/blockchain.rs`

* add `ops/macros.rs`

* tx: add fn signatures

* output: fix fn signatures

* ops: expose `_inner()` functions

* block: add `add_block_header{_bulk, _inner}()`

* ops: remove doc_{fn,inner}!()`

* ops: remove `_{inner,bulk}()`, lifetime + generics

* update lib/mod docs

* ops: add and use `doc_add_block_inner_invariant!()`

* ops: add docs/return to inner `add_block()` functions

* add_block(): extract and use fn for {key_image, output}

* ops: more fn body impl + `add_block()`

* cargo: add `monero-pruning`

* ops: extract out `tx` functions from `add_block()`

* property: add `db_version()`

* ops: `pop_block()` body, remove other `pop_block` fn's

* types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation`

* block: put `block_blob`, pass `Tables` to sub-functions

`impl TablesMut` can't mutably pass multiple tables since
it takes `&mut self`, so all functions unfortunately have to
take a full `&mut impl TablesMut` even though they only need
a few tables.

* database: add `DatabaseRw::take()`

useful for `pop_block()` where we need the value afterwards

* block: deserialize tx's from `block_blobs` in `pop_block()`

* blockchain: `height()` -> `chain_height()`

* output: fix `amount_index`

* ops: fix unlock_time, chain_height

* `BlockInfoV{1,2,3}` -> `BlockInfo`

* constants: add `DATABASE_VERSION`

* database: add `DatabaseRw::update()`

* output: use `DatabaseRw::update()` in `remove_output()`

* add `TxBlobs` table, ignore pruning tables

* block: mostly impl `add_block()` body

* ops: comments

* add_block: miner v2 tx commitment, height cast

* block: impl `pop_block()`

* block: mostly impl `get_block()`

* block: impl `get_block_{from_height,header,header_from_height}`

* add `OutputFlags` bitflags

* add_block: u32::try_into(height: u32), use `OutputFlags`

* tx: impl `get_{tx,tx_from_id}()`

* tx: move docs tests to `#[test]`

testing everything in 1 go is more natural since e.g:
`add_tx()` is followed by `get_tx()`

* tables: add `trait TablesIter`, `all_tables_empty()`

This allows `TablesMut` to be a superset of `Tables`
and use all its accessor functions.

* use cuprate-test-utils, fix tx tests

* block: `add_block()` take block by ref

* tx: use all txs in tests

* output: add `all_tx_functions()` test

* add_block: check current height against input

* block: map more fields in `get_block()`

* block: remove `get_block()`, doc tests, fix `get_block_header()`

* block: dummy values in test

* heed: use `last/first()` instead of `unsafe` cursors

We no longer have DUP semantics and also hard to debug errors
were popping up on `del_current()`...

* heed: fix `DatabaseRw::delete`

Ok(true) means the key did not exist, so we must return
Err(RuntimeError::KeyNotFound)

* block: `add_block()` (dummy value) test

* ops: `key_image` tests

* cleanup, docs, tests

* backend: test `take()` & `update()`

* docs

* remove `OutputFlags::NONE`

* add_block(): add asserts, panic docs, `should_panic` tests

* backend: remove `Ok(())` in `Database::delete` if already deleted

redb already does this, so heed so match

* block: move block operations after tx/outputs

* `amount == 0` -> `amount == 1`

* Nit: StorableVec::wrap_ref

* `saturating_sub(1)` -> `- 1`

* add `TxOutputs` table

* add_block(): add to `tx_outputs` table

* fix `DatabaseRo::update`

* add_tx(): take `block_height` as input

* tx: add/remove from `TxOutputs` table

* output: remove if `amount == 1` -> `amount_index == 0`

* output: fix `add_output()`'s `amount_index` calculation

* output: fix `add_output()`'s `amount_index` calculation again

* output: tests for `amount_index/num_outputs`

* block: `num_outputs - 1` and `take()` -> `get()`

We don't need to `take()` since the call afterwards to
`remove_output()` removes the entry

* block: swap `get_block_extended_header[_from_height]()`

* move `{key_image,output}` handling `add_block()` -> `add_tx()`

* blockchain: add doc to `top_block_height()`

* block: manual panic -> `assert_eq!()`

* test-utils: add `block_blob` to `VerifiedBlockInformation`

field introduced in this PR

* ops: use real block/tx data in tests

* block: `total_generated_coins` -> `cumulative_generated_coins`

* fix clippy, docs, TODOs

* `cumulative_generated_coins()`: `block/` -> `blockchain/`

* blockchain: add `cumulative_generated_coins()` tests

* Update database/src/ops/block.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* `cumulative_generated_coins()` docs for pre-block-0 special case

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 02:53:29 +01:00
hinto-janai
ee22e81c7e
test-utils: add rpc module (#110)
* test-utils: impl `rpc` module

* client: use `spawn_blocking`

* client: add tests

* ignore test for now

* add example and `get_transaction_verification_data()`

* client: calculate proper `generated_coins`

* data: fix `generated_coins/reward` references

* data: fix height
2024-04-21 17:11:23 +01:00
dependabot[bot]
9ad7ea3fa0
build(deps): bump rustls from 0.22.2 to 0.22.4 (#112)
Bumps [rustls](https://github.com/rustls/rustls) from 0.22.2 to 0.22.4.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.22.2...v/0.22.4)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-21 01:34:17 +01:00
hinto-janai
fe2366c5df
test-utils: return &'static verified superset types (#108)
* data: return `&'static` verified superset types

* docs

* fix VerifiedBlockMap, use manual input, add block 5da0a3d

* free: use macros to generate accessor functions

* constants: use macro

* docs

* `tx_data` -> `tx_blob`
2024-04-21 01:34:04 +01:00
hinto-janai
ad7b750d76
database: impl service::{Request,Response} mappings (#101)
* add `cuprate-types`

* remove `cuprate_database::service::{request,response}`

* use `cuprate_types::service::{request,response}`

* service: fix `Request` `match`'s

* service: create `ReadRequest` function mappings

* service: create `WriteRequest` function mappings

* service: add rough `WriteRequest` retry loop

* service: handle `RuntimeError::ResizeNeeded` in writer

* add `{R,r}o` exception to typos

* docs

* env: make `resize_map()` return new memory map byte size

* write: proactively handle resizes

`add_block()` takes `VerifiedBlockInformation` such
that it can just take the inner blobs of data.

This is a problem when reactively resizing since we no longer
have the block struct we just gave away so we're forced to `.clone()`
each retry.

Instead of that - we will proactively resize so the resize error
will never occur in the first place.

* read: use type aliases

* docs

* fix import

* write: handle resizes reactively

* service: panic if response can't be sent back

* write: add loop unreachable asserts

* service: print and drop error instead of panic

* write: fix retry loop off-by-1

* write: fix docs

* review changes

* update readme

* remove `BlockBatchInRange` request/response

* Update database/README.md

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-16 23:05:38 +01:00
hinto-janai
e6465ec613
test-utils: add data/ (#107)
* test-utils: add `data/` + add lints + `clippy --fix`

* data: assert tx_hash is correct

* data: add `BLOCK_202612` doctest

* lib.rs: remove lints

* data: add more tx data, fix names

* revert `Cargo.toml` formatting

* data: add more block data

* free: add serialization tests

* remove clippy allows
2024-04-13 02:13:46 +01:00
dependabot[bot]
eb3227ef2d
build(deps): bump h2 from 0.3.24 to 0.3.26 (#105)
Bumps [h2](https://github.com/hyperium/h2) from 0.3.24 to 0.3.26.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/v0.3.26/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.3.24...v0.3.26)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-05 23:45:06 +01:00
hinto-janai
2813c92505
database: make open_db_rw() take &TxRw (#104)
* env: take `&TxRw` in `open_db_rw()`

* heed: use `UnsafeCell` for write transactions

* backend: update tests

* add `trait DatabaseIter<T: Table>`

* heed: impl `trait DatabaseIter`, inline shared iter `fn`s

* env: make `open_db_ro()` return `impl DatabaseIter`

* heed: use full path for transaction fn calls

* tests: fix tx/table tests

* backend: fix redb

* heed: `UnsafeCell` -> `RefCell`

* docs

* remove unneeded `// SAFETY`

* Update database/src/backend/heed/env.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-03 22:44:25 +01:00
hinto-janai
21697b8af5
database: add more fn to trait DatabaseR{o,w} (#96)
* database: add more fn signatures

* database: fix docs

* heed: add new fn signatures with `todo!()`

* heed: add shared fn sigantures

* database: make `iter()` return `(key, value)`

* database: make `get_range()` return `(key, value)`

* database: make `first()/last()` return `(key, value)`

* heed: impl new fn signatures

* redb: impl new fn signatures

* backend: update tests for new `fn`s

* backend: update table tests

* heed: fix `retain()` predicate behavior

* database: add `keys()`, `values()`, `pop_first()`, `pop_last()`

* heed: impl `keys()`, `values()`, `pop_first()`, `pop_last()`

* redb: impl `keys()`, `values()`, `pop_first()`, `pop_last()`

* database: add `contains()`

* backend: add `contains()` to tests

* database: remove `retain()`

* database: only return value in `get_range()`

* database: `DatabaseRw::clear()` -> `EnvInner::clear_db()`

* redb: update `clear_db()` invariant doc

* fix redb

* cargo.toml: avoid merge conflict

* update `get_range()` docs

* redb: re-create table in `clear_db()`
2024-03-30 01:33:38 +00:00
hinto-janai
ae0d9ff36a
database: add redb-memory backend (#97)
* add `redb-memory` feature

* redb: use `redb::backend::InMemoryBackend` if enabled

* readme: add `redb-memory` section

* ci: add `redb-memory` testing

* ci: remove `redb-memory` testing

probably not worth adding time to CI for this
2024-03-27 00:50:10 +00:00
hinto-janai
d503548716
Implement types/ (#94)
* workspace: add `bytemuck` to workspace

* add `types/`

* workspace: add `cuprate-types` to members

* copy `consensus/` types to `types/`

* remove `hard_fork/`

* extended_block_header: impl `Pod`, fix layout

* update `Request/Response`

* impl types

* fix `Response/Request`

* impl `borsh`

* workspace: add `strum`

* service: add `strum` traits

* remove `paste`, `serde_json`, `thiserror`

* remove `strum`

* VerifiedBlockInformation: remove `hf_vote`

* Update Cargo.toml

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-03-27 00:46:32 +00:00
hinto-janai
5e7ee57482
database: redb 1.5.0 -> redb 2.0.0 (#95)
* `redb 1.5.0` -> `redb 2.0.0`

* `Redb{Key,Value}` -> `redb::{Key,Value}`

* redb: remove unneeded lifetimes

* database: remove read table lifetime

* redb: remove read table lifetime

* heed: remove `'tx` lifetime on read table

* remove `'env, 'tx` lifetime from read/write table

* redb: remove `'env, 'tx` lifetime from read/write table

* heed: remove `'env, 'tx` lifetime from read/write table

* redb: update `TxRo::commit()` docs
2024-03-24 22:15:26 +00:00
hinto-janai
3656a1ada7
database: return owned T in Storable (#92)
* add `storable_slice.rs`

* storable: `from_bytes(&[u8]) -> Self`

* heed: return `T` in `Storable`, use `StorableSlice` in tests

* remove `value_guard.rs`

* redb: update `Storable` to return owned `T`

* database: return `T` directly, remove `'a` lifetime

* replace `[]` with `StorableSlice`

* tables: add lifetime to `tables!()` and generated structs

* backend: fix tests

* heed: update fn signatures

* storable: add `StorableVec`, remove `Storable::ALIGN`

* tables/types: remove slice lifetimes, use `StorableVec`

* backend: use `StorableVec`, fix tests

* types: fix tests to use owned returned value

* heed: fix `Storable` impl and tests

* storable: `StorableVec` docs + tests

* backend: assert values are detached from `get_range()` iterator

* database: add docs

* udpate docs, remove unneeded clippy `allow`

* replace `ToOwnedDebug` with `std::fmt::Debug`

* impl `StorableBytes`

* backend: add `StorableBytes` to tests

* readme: remove `to_owned_debug.rs` reference
2024-03-22 21:11:48 +00:00
hinto-janai
004bb153b4
database: use rayon for service's reader thread-pool (#93)
* add `rayon 1.9.0`

* service: re-impl reader threadpool with `rayon`

* service: impl `tower::Service` for writer

* backend: create db dir in `Env::open`

* service: read + write request/response tests

* docs, name changes

* service: always return `Poll::Ready` in writer

* service: use `spawn()` instead of `install()`

* service: replace `DatabaseReader` with free functions

* cargo: add `tokio-utils`

* service: acquire permit before `call()` for read requests

* service: acquire permit in tests

* docs

* service: use loop for write request tests

* service: use `ready!()`
2024-03-21 20:16:12 +00:00
93372fa4b5
P2P Address book & Handshake changes (#89)
* use tokio's delay queue for bans

* document handles

* remove peers from address book when retrieving

* ping inbound peers during handshakes

* support receiving pings during handshakes

* add peer to anchor before reducing whit list

* clippy

* comment handshakes

* typos

* sort `use`

* use `rand::prelude::*`

* review comments

* update macro
2024-03-20 20:58:12 +00:00
hinto-janai
de931f8630
database: implement tables and types (#91)
* table: set `Key` and `Value` to `?Sized`

* types: add `SCHEMA.md` types

* tables: add `SCHEMA.md` tables

* redb: let value be `?Sized`

* types: add casting doc tests and `pub`

* backend: create all tables in `Env::open()`

* backend: open all tables in `tests.rs`

* remove `TestType{1,2}`, use `Output` in tests

* backend: add tests for all tables/keys/values

* add `NumOutputs` table

* small doc fixes

* tables: doc

* types: add `PreRctOutputId`

* types: `BlockInfoV2 { cumulative_rct_outs: u32 -> u64 }`
2024-03-18 01:15:56 +00:00