Any peer that's behind us while syncing is useless to us (though
not to them). This ensures that we don't get our peer slots filled
with peers that we can't use. Once we've synced, we can connect
to them and they can then sync off us if they want.
Essentially, one can send such a large amount of IDs that core exhausts
all free memory. This issue can theoretically be exploited using very
large CN blockchains, such as Monero.
This is a partial fix. Thanks and credit given to CryptoNote author
'cryptozoidberg' for collaboration and the fix. Also thanks to
'moneromooo'. Referencing HackerOne report #506595.
When all our outgoing peer slots are filled, we cycle one peer at
a time looking for syncing peers until we have at least two such
peers. This brings two advantages:
- Peers without incoming connections will find more syncing peers
that before, thereby strengthening network decentralization
- Peers will have more resistance to isolation attacks, as they
are more likely to find a "good" peer than they were before
RPC connections now have optional tranparent SSL.
An optional private key and certificate file can be passed,
using the --{rpc,daemon}-ssl-private-key and
--{rpc,daemon}-ssl-certificate options. Those have as
argument a path to a PEM format private private key and
certificate, respectively.
If not given, a temporary self signed certificate will be used.
SSL can be enabled or disabled using --{rpc}-ssl, which
accepts autodetect (default), disabled or enabled.
Access can be restricted to particular certificates using the
--rpc-ssl-allowed-certificates, which takes a list of
paths to PEM encoded certificates. This can allow a wallet to
connect to only the daemon they think they're connected to,
by forcing SSL and listing the paths to the known good
certificates.
To generate long term certificates:
openssl genrsa -out /tmp/KEY 4096
openssl req -new -key /tmp/KEY -out /tmp/REQ
openssl x509 -req -days 999999 -sha256 -in /tmp/REQ -signkey /tmp/KEY -out /tmp/CERT
/tmp/KEY is the private key, and /tmp/CERT is the certificate,
both in PEM format. /tmp/REQ can be removed. Adjust the last
command to set expiration date, etc, as needed. It doesn't
make a whole lot of sense for monero anyway, since most servers
will run with one time temporary self signed certificates anyway.
SSL support is transparent, so all communication is done on the
existing ports, with SSL autodetection. This means you can start
using an SSL daemon now, but you should not enforce SSL yet or
nothing will talk to you.
- Support for ".onion" in --add-exclusive-node and --add-peer
- Add --anonymizing-proxy for outbound Tor connections
- Add --anonymous-inbounds for inbound Tor connections
- Support for sharing ".onion" addresses over Tor connections
- Support for broadcasting transactions received over RPC exclusively
over Tor (else broadcast over public IP when Tor not enabled).
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
To help protect one's privacy from traffic volume analysis
for people using Tor or I2P. This will really fly once we
relay txes on a timer rather than on demand, though.
Off by default for now since it's wasteful and doesn't bring
anything until I2P's in.
3880bf39 cryptonote_protocol_handler.inl: remove span read just now that failed to pass some basic tests (stoffu)
da249fd5 cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool) (stoffu)
0e7ad2e2 Wallet API: generalize 'bool testnet' to 'NetworkType nettype' (stoffu)
af773211 Stagenet (stoffu)
cc9a0bee command_line: allow args to depend on more than one args (stoffu)
55f8d917 command_line::get_arg: remove 'required' for dependent args as they're always optional (stoffu)
450306a0 command line: allow has_arg to handle arg_descriptor<bool,false,true> #3318 (stoffu)
9f9e095a Use `genesis_tx` parameter in `generate_genesis_block`. #3261 (Jean Pierre Dudey)
43f5269f Wallets now do not depend on the daemon rpc lib (moneromooo-monero)
bb89ae8b move connection_basic and network_throttle from src/p2p to epee (moneromooo-monero)
4abf25f3 cryptonote_core does not depend on p2p anymore (moneromooo-monero)
Deleted 3 out of 4 calls to method connection_basic::sleep_before_packet
that were erroneous / superfluous, which enabled the elimination of a
"fudge" factor of 2.1 in connection_basic::set_rate_up_limit;
also ended the multiplying of limit values and numbers of bytes
transferred by 1024 before handing them over to the global throttle
objects
This is safer, as we don't risk break expectations (eg, requesting
block hashes and then receiving a late set of blocks). Dropping a
connection means another will be attempted in a fresh state.
Also bump the kick timeout to 5 minutes, to ensure we only kick
really idle peers.
The last known hash was calculated incorrectly, causing
further chain hash downloads to restart from the current
chain. When the block queue has close to 10k blocks waiting,
this causes frequent downloads of 10k more hashes, but
with only the last few hashes actually being useful.
It is unused, as it was apparently a future optimization,
and it leaks some information (though since pools publish
thei blocks they find, that amount seems small).
c867357a cryptonote_protocol: error handling on cleanup_handle_incoming_blocks (moneromooo-monero)
ce901fcb Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks (moneromooo-monero)
84fa015e core: guard against exceptions in handle_incoming_{block,tx} (moneromooo-monero)
Fix sync wedge corner case:
It could happen if a connection went into standby mode, while
it was the one which had requested the next span, and that span
was still waiting for the data, and that peer is not on the
main chain. Other peers can then start asking for that data
again and again, but never get it as only that forked peer does.
And various other fixes
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).
If monerod is started with an explicit sync mode, none of this
automation takes effect.
This was broken by the reorg fix, since we now have to add blocks
regardless of their starting height. We now check whether we know
the parent for the first block in the next span, or whether it was
requested. If neither, it's an orphan. If it is not known, but was
requested, we wait to get that block.
Add get_fork_version and add_ideal_fork_version to core so
cryptonote_protocol does not have to need the Blockchain
class directly, as it's not in its dependencies, and add
those to the fake core classes in tests too.
When a node is dropped, we stop considering its claimed blockchain
height as a factor in the target height calculation. This prevents
a runaway chain from being still thought to be the target even if
the nodes carrying it are dropped.
We won't even talk to a peer which claims a wrong version
for its top block. This will avoid syncing to known bad
peers in the first place.
Also add IP fails when failing to verify a block.
Connections can be dropped by the net_node layer,
unbeknownst to cryptonote_protocol, which would then
not flush any spans scheduled to that connection,
which would cause it to be only downloaded again
once it becomes the next span (possibly after a small
delay if it had been requested less than 5 seconds
ago).
A block queue is now placed between block download and
block processing. Blocks are now requested only from one
peer (unless starved).
Includes a new sync_info coommand.
All code which was using ip and port now uses a new IPv4 object,
subclass of a new network_address class. This will allow easy
addition of I2P addresses later (and also IPv6, etc).
Both old style and new style peer lists are now sent in the P2P
protocol, which is inefficient but allows peers using both
codebases to talk to each other. This will be removed in the
future. No other subclasses than IPv4 exist yet.
- only pause mining once we've got the lock (in practice, it'll
already be paused by another thread if we can't get the lock
at once though)
- do not call prepare_handle_incoming_blocks when we dismissed
all the blocks, it only causes cleanup_handle_incoming_blocks
to complain afterwards
8bdc86be protocol: speed up sync by minimizing duplicate work (moneromooo-monero)
61dfa310 epee: fix some log macros not printing context nicely (moneromooo-monero)
In particular, the prepare_handle_incoming_blocks call
is pretty lengthy, and entirely pointless in the common
case where several different connections will prepare
the exact same blocks.
- fix wrong block being used when a new block is received between
a node elaying a fluffy block and sending a new fluffy block
with txes a peer did not have
- misc a neverending ping pong requesting the same missing txids
when a new block is received in the meantime, causing the top
block to not be the one we need
- send the original fluffy block message block height when sending
a new fluffy block, not the current top height, which might
have been updated since
- avoid sending back the whole block blob when asking for txes,
send only the hash instead
- plus misc cleanup and additional debugging logs
0644eed7 Remove boost/foreach.cpp includes (Miguel Herranz)
36dd3e23 Replace BOOST_REVERSE_FOREACH with ranged for (Miguel Herranz)
629e3101 Replace BOOST_FOREACH with C++11 ranged for (Miguel Herranz)
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.
To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:
This one is (mostly) silent, only outputting fatal errors:
MONERO_LOGS=*:FATAL
This one is very verbose:
MONERO_LOGS=*:TRACE
This one is totally silent (logwise):
MONERO_LOGS=""
This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):
MONERO_LOGS=*:WARNING,verify:FATAL
Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE
Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:
MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE
Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.
Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.
The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
Added a new command to the P2P protocol definitions to allow querying for support flags.
Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags.
Added p2p constant for signaling fluffy block support.
Added get_pool_transaction function to cryptnote_core.
Added new commands to cryptonote protocol for relaying fluffy blocks.
Implemented handling of fluffy block command in cryptonote protocol.
Enabled fluffy block support in node initial configuration.
Implemented get_testnet function in cryptonote_core.
Made it so that fluffy blocks only run on testnet.
This will be when we can't find common ground between the peer's
short chain history and our blockchain.
This fixes bad peers claiming a higher blockchain height from never
dropped, and keeping the node in synchronizing state forever, since
we will never get blocks from that peer.
