Commit graph

42 commits

Author SHA1 Message Date
Boog900
88f7d1f212
download monerod in CI (#123)
* download monerod in CI

* move action file

* add macOS arm support

* remove reqwest from workspace

* undo whitespace changes

* fix indentation

* update comments

* add monerod to .gitignore

* Apply suggestions from code review

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>

* add back spacing

---------

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
2024-05-01 01:21:08 +01:00
hinto-janai
32a44a2503
test-utils: compare data against RPC (#115)
* cargo: add `pretty_assertions`

* test-utils: add tests to compare data against RPC data, and fix

* small fixes

* database: fix `cumulative_generated_coins` from test data

block had wrong data which offset all of the asserts
2024-04-25 19:58:45 +01:00
hinto-janai
d503548716
Implement types/ (#94)
* workspace: add `bytemuck` to workspace

* add `types/`

* workspace: add `cuprate-types` to members

* copy `consensus/` types to `types/`

* remove `hard_fork/`

* extended_block_header: impl `Pod`, fix layout

* update `Request/Response`

* impl types

* fix `Response/Request`

* impl `borsh`

* workspace: add `strum`

* service: add `strum` traits

* remove `paste`, `serde_json`, `thiserror`

* remove `strum`

* VerifiedBlockInformation: remove `hf_vote`

* Update Cargo.toml

Co-authored-by: Boog900 <boog900@tutanota.com>

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-03-27 00:46:32 +00:00
hinto-janai
004bb153b4
database: use rayon for service's reader thread-pool (#93)
* add `rayon 1.9.0`

* service: re-impl reader threadpool with `rayon`

* service: impl `tower::Service` for writer

* backend: create db dir in `Env::open`

* service: read + write request/response tests

* docs, name changes

* service: always return `Poll::Ready` in writer

* service: use `spawn()` instead of `install()`

* service: replace `DatabaseReader` with free functions

* cargo: add `tokio-utils`

* service: acquire permit before `call()` for read requests

* service: acquire permit in tests

* docs

* service: use loop for write request tests

* service: use `ready!()`
2024-03-21 20:16:12 +00:00
Boog900
93372fa4b5
P2P Address book & Handshake changes (#89)
* use tokio's delay queue for bans

* document handles

* remove peers from address book when retrieving

* ping inbound peers during handshakes

* support receiving pings during handshakes

* add peer to anchor before reducing whit list

* clippy

* comment handshakes

* typos

* sort `use`

* use `rand::prelude::*`

* review comments

* update macro
2024-03-20 20:58:12 +00:00
Boog900
159c8a3b48
levin: fragmented messages (#84)
* levin: fix fragmented messages & use bitflags

* levin: add a method to fragment a message

* levin: add tests for fragmented messages and fix issues

* fix docs

* tests: don't include bytes length

* levin: add support for sending fragmented
/ dummy messages

* fmt

* add fragmented handshake tests.

* fix handshake detection when fragmented
and alt (non-monero) protocol info

* add tracing logs

* remove `already_built`, this was an old way I was thinking of sending raw buckets

* clippy

* clippy 2

* Update net/levin/src/message.rs

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>

* review comments

* add timeout to tests

* Update net/levin/src/header.rs

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>

---------

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
2024-03-05 01:29:57 +00:00
Boog900
e560ecc2ee
Add a test for an inbound handshake from monerod (#75)
* monerod: set a random DB and remove dir when done

* add a test for an inbound monerod handshake

* don't fail if can't remove dir on windows

* Update test-utils/src/monerod.rs

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>

* use `temp_dir`

* use `tempfile`

---------

Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
2024-02-25 21:21:25 +00:00
hinto-janai
331d4d3e7f
Database (#35)
* rename `database` -> `old_database`

Keeping it around for reference until new implementation is complete.

* create new `database/` skeleton

* add `DATABASE.md` design doc skeleton

* move design doc to `database/README.md`

* add rough code

* `lib.rs` -> `gist.rs`

* database: use workspace deps

* workspace: include `database/` as member

CI will now include this crate.

* cargo fmt

* database: `AGPL` -> `MIT`

* readme: add `TODO`s

* add base files

* cargo.toml: add `heed` feature

* fix clippy

* lib.rs: add extremely pedantic lints

* readme: add `# Backends`

* cargo.toml: add `cfg-if`

* add `backend/` structure

* base `database.rs`

* cargo.toml: add `borsh`

* backend: add `DATABASE_BACKEND`

* base `error.rs`

* base `database.rs`

* base `transaction.rs`

* base `table.rs`

* lib.rs: add imports

* add `pod.rs`

* pod: use `Read/Write`, add tests for all primitive numbers

* pod: impl Pod for `Vec<u8>`, `[u8; N]`

* pod: add docs, add `private::Sealed`

* pod: `to_writer`, `from_reader`

The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.

Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`

* pod: impl Pod for `Box<[u8]>`

* pod: return `Err` on incorrect length in `from_bytes()`

* pod: docs

* pod: impl Pod for `Arc<[u8]>`

* readme: docs

* database: add `create_table()`, `get_table()`

* table: `Pod` bound

* backend: move into directories

* pod: add `into_bytes()`

* heed: impl `BytesEncode`, `BytesDecode`

* add `actor`, `service` features

The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.

* add `lib.rs` docs

* service: add `service.rs`

* service: add `reader.rs`

* service: add `writer.rs`

* service: add `request.rs` & `response.rs`

* cargo.toml: add `crossbeam`

* service: add/use `enum Request`, `enum Response`

* service: basic signatures for thread-pools, `Writer` -> `Writers`

* service: split `tower::Service<ReadRequest/WriteRequest>`

* service: impl `tower::Service` for `Database(Reader|Writer)`

* service: add `init()`, impl basic `Reader/Writer` pools

* service: add example `Request`'s

* service: add example `ReadRequest` handling

* temporarily allow clippy lints

* readme: update file structure

* transaction: add `RoTx::get_range()`

* service: module docs

* cargo.toml: add `cuprate-helper`

* service: scale readers/writers based on thread count

* database: change lifetimes

* heed: impl Database for `heed`

* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits

* docs

* service: `read.rs` docs

* service: `write.rs` docs

* service: request/response docs

* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`

* service: leak database into `&'static`

* database: add `#[inline]`, `#[cold]`

* service: `free.rs` docs, more `#[inline]` + `#[cold]`

* service: add `shutdown()`

* service: add `Request::Shutdown`

* service: `shutdown()` docs

* heed: hide concrete tx types

* lib.rs: add terms

* split `Env` <-> `Database`

* cargo.toml: add `paste`

* database: add `tables/`

* impl `serde/borsh` where possible

* tables: add `Tables`, add test docs

* make db backend mutually exclusive to fix `--all-features`

* tables: use `$()*` in `tables!()`

* cargo.toml: add `sanakirja 1.4.0`

* sanakirja: impl `Env`

* sanakirja: impl `Database`

* sanakirja: impl `Transaction`

* table: temporarily fix `sanakirja` K/V bounds

* table: fix `#[cfg]`

* cargo.toml: fix deps

* lib.rs: docs

* service: docs

* readme: add files, update `# Documentation`, add `# Layers`

* readme: `src/` file purpose

* readme: `src/service/` file purpose

* readme: `src/backend/` file purpose

* fix `Cargo.lock` merge conflict

* database: remove `gist.rs`

* add to `constants.rs`

* add top `//! comments` for files/modules

* constants: add sanity-check test

* service: add `only_one_database` test in `free.rs`

* service: add `tests.rs`

* remove unneeded markers + imports

* backend: fix `get_range()`'s trait `impl` return

* env: add `create_tables_if_needed()`, don't return `Option<Db>`

* sort imports by `CONTRIBUTING.md` rules

oops sorry boog

* add `monero.rs`

* monero: docs

* database: add missing `RoTx/RwTx` inputs

* backend: add missing `RoTx/RwTx` inputs

* `monero.rs` trait -> free functions in `ops/`

* pod: make methods infallible

* ci: add `rustup update` step

* service: use `Arc` instead of leaking, remove `db_read/db_write`

* service: use `InfallibleOneshotReceiver` for readers

* service: shutdown on error, add todos

* service: remove `Request`

* service: combine `ReadResponse` and `WriteResponse`

* service: use `InfallibleOneshotReceiver` for writer

* service: only spawn 1 writer, don't allow cloning writer handle

* table: add associated `const CONSTANT_SIZE`

* add `key.rs` + `trait Key`, add bound to `Table`

* fix typos
2024-02-13 17:43:25 +00:00
Boog900
b7df133175
consensus: fix batch handling when we don't have a full list of outputs. (#54)
* consensus: fix batch handling when we don't have a full list of outputs.

* change `scan_chain` to new API

* clippy

* add a test for calculating multiple difficulties

* fmt

* rx_seed -> rx_vms

* consensus-rules: ring members younger than current block.

* only create rx vms when required.

* fix rx initiation when syncing

* add single block verification (no batch)

* update serai
2024-02-13 00:51:11 +00:00
hinto-janai
630faed263
ci: include macos + windows (#52)
* ci: install boost, include macos + windows

* cryptonight: fix `MSVC`

* cryptonight: use `flag_if_supported()`

* fix cryptonight builds

* update randomX

* fix rx builds

* add memwipe

* include memwipe.c in build

* spawn monerod in msys2 for windows

* fix last commit

* install dependencies before spawning monerod

* remove --detach

* try another way of spawning monerod

* add /I

* download and spawn monerod as a part of tests

* add download.rs

* extend time for monerod spawn

* move sleep and show monerod output

* fix clippy

* change stdin to pipped

* #[cfg(unix)] on bytes::Buf

* fix macos capitalisation

* remove tar.bz2 on macos expected dir

* remove zip on windows expected dir

* fix todo

* add docs

* fix a couple typos

---------

Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-12 13:39:15 +00:00
hinto-janai
ba0f82c356
helper: use crossbeam::atomic::AtomicCell for atomic floats (#56)
* cargo.toml: add `crossbeam`

* helper: use `crossbeam::atomic::AtomicCell` for `AtomicF(32|64)`

* helper: atomic docs
2024-02-10 23:19:12 +00:00
Boog900
2b65be4b18
net: add epee-encoding and fixed-bytes 2024-01-29 22:44:59 +00:00
Boog900
ed598e374e
remove empty cuprate bin and common 2024-01-22 01:56:34 +00:00
Boog900
b20b6fdee1
remove primitive-types for crypto-bigint
we already have crypto-bigint in our tree
2024-01-21 15:18:25 +00:00
Boog900
b9334b6a90
clean up dependencies 2024-01-21 14:46:03 +00:00
hinto-janai
50894bef89
helper/ (#45)
* add `/helper/`

* add `num.rs`

* add `sys.rs`

* add `crypto.rs`

* add lints and mod to `lib.rs`

* `sys` -> `time`, add more free functions

straight from https://docs.rs/readable/latest/readable/time/index.html

* num: add `Number/Float` types, `cmp_float()`, `cmp_float_nan()`

* `common/src/tower_utils.rs` -> `helper/src/asynch.rs`

* gate modules with `#[cfg(feature = "...")]`

* add `thread.rs`

* cargo fmt

* thread: test out of 100

* add `atomic.rs`

* atomic: fix `fetch_update()`

* atomic: impl `fetch_*()` for atomic floats

* `#[no_std]` where possible

* asynch: remove `InstaFuture`

https://docs.rs/futures/latest/futures/future/fn.ready.html

* crypto: remove `check_point()`

* thread: return percent computation without static

* thread: add `low_priority_thread()`

https://docs.rs/lpt

* add rayon_spawn_async, remove crypto

* remove current_time_try

---------

Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
2024-01-21 00:04:09 +00:00
Boog900
40e64cc9c3
integrate RandomX, plus some other misc changes. 2024-01-05 22:36:57 +00:00
Boog900
f037532610
add ringCT rules 2023-12-24 21:07:28 +00:00
Boog900
84343a8297
move RPC scanning cache to borsh 2023-12-18 14:36:22 +00:00
Boog900
3eea0b73bd
Split consensus rules into separate crate. 2023-12-17 14:26:54 +00:00
Boog900
92652b26a2
add cli args to RPC scanning binary 2023-12-08 15:36:45 +00:00
Boog900
2c4cc1fb93
move address book to separate crate.
Also changes the address book to use the network zone trait
2023-12-08 15:03:01 +00:00
Boog900
8557073c15
p2p changes (#38)
* start re-working p2p to work with change monero-wire

* start re-working p2p to work with change monero-wire

adds back some changes from #22

* change the peer module to use the new API + fix a couple bugs

* remove peer set for now

* add try_from/from conversion between `Message` and
`Request`/`Response`

* Allow specifying other parameters in levin-cuprate

* add new `LevinCommand` enum and clean up monero-wire message de/encoding

* fix issues with merge

* start splitting up p2p crate into smaller crates.

* add monerod action from serai to test network code

* remove tracing in tests
2023-11-30 18:09:05 +00:00
hinto-janai
10b7400b17
Update workspace Cargo.toml (#36)
* add dependencies to workspace

* add workspace build profiles

* add potential dependencies
2023-11-16 01:10:46 +00:00
Boog900
2440ccbd8d
remove randomX lib 2023-10-29 00:08:37 +01:00
Boog900
d7798328b6
consensus: add block weight calculations + rcp speedup
Instead of asking for one block at a time, we ask for batches of blocks, which significantly speeds up getting blocks.
2023-09-28 12:21:06 +01:00
Boog900
57630ed9fb
init random-x module
super scaler program generation/ execution is complete.
2023-09-14 00:23:10 +01:00
Boog900
ab3c496bbd
add difficulty calculations 2023-09-06 15:54:49 +01:00
Boog900
42548f733d
add cryptonight bindings 2023-09-05 11:56:07 +01:00
Boog900
2f08978e67
init consensus rules crate 2023-09-03 23:50:38 +01:00
Boog900
477d9e42f3
change monero-wire to use epee-encoding (#25)
* change monero-wire to use epee-encoding

* Add back unified `Message` enum

* Run clippy & fmt

* add back default val for top_version
2023-07-17 18:43:34 +01:00
Boog900
5f20342736
initial Abscissa bin (#18)
* abscissa init

* remove package in cargo.toml

* cargo fmt + remove more stuff fro toml

* bump rust edition
2023-04-26 19:22:06 +01:00
Boog900
a187d9a357
initial p2p code (#8)
* init

* save

* use macro to create the levin body enum

* add protocol docs and cargo fmt

* add response validation

* add client functionality to connection + fmt

* Add new cuprate-common crate
this crate will hold stuff needed across cuprate crates

+ init handshaker

* add stagenet & testnet hardforks + tests

+ cargo fmt

* split peer and protocol into separate crates
+ add sync state watcher

* finish initial sync states and add some tests

* save

* add initial address book

* cargo fmt

* save

* add pruning module to cuprate-common

* more address book updates
- added an address book client
- add some more address book requests
- add "NetZone"

* lots of changes

* cargo fmt

* combine p2p into one crate
they were all linked anyway

* cargo fmt

* turn the handshaker into a statemachine

* cargo fmt

* reduce the amt of copies when decoding

+ remove reliance on monero-rs

* update time_from_timestamp func

* cargo fmt
+ change qr code link
+ remove clippy.toml
2023-04-24 22:37:40 +01:00
Someone Else
be43216b3f
initial database code (#6)
* commit to start the draft pull request.

added a space

* Please don't look to close.It might hurt your eyes

* impl associated types

* errors, docs & divided ro/rw tx

Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.

* more changes see description

changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.

* small changes

* Organized modules & implemented get_block_hash

* write prototype & error

Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.

* Multiple changes

- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction

* put mdbx as features with its dependency

* save

* refactored some method with macros

* more mdbx errors, docs correction, moved to error.rs

* finish nodup mdbx impl, errors.rs, macros, tables

Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.

* bye bye match statement in mdbx.rs

* defined all blockchain tables (not txpool)

* dupsort/fixed support, dupcursor, basic block interface

* tables, types, encoding and documentations

Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.

* replaced macros and added hfversion table

* save

* multiple changes

* modified database schema. deprecated output global index and splited up pre-rct from rct output.

* Fixed DupCursor function to return subkey (thx to rust turbofish inference).

* Added some output functions

* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.

* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.

* i hate lifetimes

* read-only method now use read-only tx

* initial output fn

* some tx functions. Yes I'll refactor them

* moved interface in a module

* redefined errors, more tx fn, None->error

* corrected a table + started blk fns

* save

* fixed TxOutputIdx + pop_block

* IIRC I finished initial interface fns

* fixed table name const + db build/check/open fn

* switched important tables to dummy keys + rm blockhfversion

* minor docs correction

* fixed mentioned issues

* make a test bin, just for fun

* fixed issues + cargo fmt

* removed monerod part

* fixed a comment
2023-04-20 18:20:32 +01:00
Someone Else
1d4abc841c
License correction for github (#10)
* Rename AGPL-3.0 to LICENSE-AGPL

* Create LICENSE-MIT (#9)

* Rename LICENSE to LICENSE.md

* remove rust version

* bump rust-version
2023-03-20 21:35:20 +01:00
Boog900
8844d23d2e
Change license to AGPL-3 or MIT depending on crate 2023-03-07 23:36:48 +00:00
Boog900
265fb3e895
initial net code (#2)
* initial commit
not ready

* add new levin lib

* fix decoding multiple messages

* make the levin lib async

* saving progress

* saving progress

* init documention of levin and reorganise net

* update monero-wire

* remove p2p folder - this PR is just for net

* update Cargo.toml

* net: update links in cargo.toml
2023-03-07 22:37:55 +00:00
fb05074a05 +docs, -fn, KeyValueDatabase trait + serde derive
Documented more functions to BlockchainDB, but I removed the ones that strictly concern
the database. These methods will be implemented under KeyValueDatabase trait.
Still needs to understand some things before starting to refactor blockchaindb functions.

Also enabled derive features for serde.
2023-02-11 18:59:55 +01:00
8ed2183a83 removed unused fn + clippy deny + Cargo
Removed some functions that'll be useless for blockchainDB. Docs the hardfork info ones.
Also set clippy to deny `expect()` and `panic!()` as /u/danda suggested on reddit.
Set blockchain_db in workspace dependencies + corrected the build rustflags.
2023-02-10 21:40:01 +01:00
4e9e688a7f put blockchain_db into library 2023-02-08 23:03:15 +01:00
1701d724dc forgot Cargo.toml 2023-02-08 13:54:55 +01:00
Someone Else
2c7cb27548
Initial commit 2023-02-07 14:44:32 +00:00