monero-project/monero
monero-project/monero
1
0
Fork 0
mirror of https://github.com/monero-project/monero.git synced 2024-12-28 14:39:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl

2806 lines
128 KiB
Text
Raw Normal View History

2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
/// @file
/// @author rfree (current maintainer/user in monero.cc project - most of code is from CryptoNote)
Correct spelling mistakes.
2018-01-19 07:54:14 +00:00
/// @brief This is the original cryptonote protocol network-events handler, modified by us
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
Update copyright year to 2020 Update copyright year to 2020
2020-05-07 02:36:54 +00:00
// Copyright (c) 2014-2020, The Monero Project
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 22:07:58 +00:00
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
// (may contain code and/or modifications by other developers)
// developer rfree: this code is caller of our new network code, and is modded; e.g. for rate limiting
moved all stuff to github
2014-03-03 22:07:58 +00:00
#include <boost/interprocess/detail/atomic.hpp>
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
#include <list>
cryptonote_core does not depend on p2p anymore As a followon side effect, this makes a lot of inline code included only in particular cpp files (and instanciated when necessary.
2017-11-29 22:53:58 +00:00
#include <ctime>
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
extract some basic code from libcryptonote_core into libcryptonote_basic
2017-01-26 15:07:23 +00:00
#include "cryptonote_basic/cryptonote_format_utils.h"
moved all stuff to github
2014-03-03 22:07:58 +00:00
#include "profile_tools.h"
move connection_basic and network_throttle from src/p2p to epee These even had the epee namespace. This fixes some ugly circular dependencies.
2017-11-29 19:30:06 +00:00
#include "net/network_throttle-detail.hpp"
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
#include "common/pruning.h"
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
#include "common/util.h"
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
#undef MONERO_DEFAULT_LOG_CATEGORY
#define MONERO_DEFAULT_LOG_CATEGORY "net.cn"
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
protocol: log received messages in a separate category Makes it easier to log just what's going on on P2P
2017-01-22 10:14:45 +00:00
#define MLOG_P2P_MESSAGE(x) MCINFO("net.p2p.msg", context << x)
cryptonote_protocol_handler: add block/tx hashes in notify logs
2019-03-27 20:38:42 +00:00
#define MLOGIF_P2P_MESSAGE(init, test, x) \
do { \
const auto level = el::Level::Info; \
const char *cat = "net.p2p.msg"; \
if (ELPP->vRegistry()->allowed(level, cat)) { \
init; \
if (test) \
Properly format multiline logs As a side effect, colouring on Windows should now work regardless of version
2019-06-22 14:42:48 +00:00
el::base::Writer(level, el::Color::Default, __FILE__, __LINE__, ELPP_FUNC, el::base::DispatchAction::NormalLog).construct(cat) << x; \
cryptonote_protocol_handler: add block/tx hashes in notify logs
2019-03-27 20:38:42 +00:00
} \
} while(0)
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
#define MLOG_PEER_STATE(x) \
MCINFO(MONERO_DEFAULT_LOG_CATEGORY, context << "[" << epee::string_tools::to_string_hex(context.m_pruning_seed) << "] state: " << x << " in state " << cryptonote::get_protocol_state_string(context.m_state))
protocol: log received messages in a separate category Makes it easier to log just what's going on on P2P
2017-01-22 10:14:45 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
#define BLOCK_QUEUE_NSPANS_THRESHOLD 10 // chunks of N blocks
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
#define BLOCK_QUEUE_SIZE_THRESHOLD (100*1024*1024) // MB
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
#define BLOCK_QUEUE_FORCE_DOWNLOAD_NEAR_BLOCKS 1000
#define REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY (5 * 1000000) // microseconds
#define REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD (30 * 1000000) // microseconds
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
#define IDLE_PEER_KICK_TIME (240 * 1000000) // microseconds
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
#define NON_RESPONSIVE_PEER_KICK_TIME (20 * 1000000) // microseconds
protocol: drop connections which don't handshake after some time
2017-10-19 14:48:46 +00:00
#define PASSIVE_PEER_KICK_TIME (60 * 1000000) // microseconds
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
#define DROP_ON_SYNC_WEDGE_THRESHOLD (30 * 1000000000ull) // nanoseconds
#define LAST_ACTIVITY_STALL_THRESHOLD (2.0f) // seconds
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
#define DROP_PEERS_ON_SCORE -2
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
moved all stuff to github
2014-03-03 22:07:58 +00:00
namespace cryptonote
{
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//-----------------------------------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 22:07:58 +00:00
template<class t_core>
daemon: don't drop RPC with busy error when running offline
2018-01-25 13:35:17 +00:00
t_cryptonote_protocol_handler<t_core>::t_cryptonote_protocol_handler(t_core& rcore, nodetool::i_p2p_endpoint<connection_context>* p_net_layout, bool offline):m_core(rcore),
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_p2p(p_net_layout),
m_syncronized_connections_count(0),
daemon: don't drop RPC with busy error when running offline
2018-01-25 13:35:17 +00:00
m_synchronized(offline),
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
m_ask_for_txpool_complement(true),
daemon: add '--no-sync' arg to optionally disable blockchain sync
2019-02-25 01:31:45 +00:00
m_stopping(false),
m_no_sync(false)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
if(!m_p2p)
m_p2p = &m_p2p_stub;
}
//-----------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::init(const boost::program_options::variables_map& vm)
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_sync_timer.pause();
m_sync_timer.reset();
m_add_timer.pause();
m_add_timer.reset();
m_last_add_end_time = 0;
m_sync_spans_downloaded = 0;
m_sync_old_spans_downloaded = 0;
m_sync_bad_spans_downloaded = 0;
m_sync_download_chain_size = 0;
m_sync_download_objects_size = 0;
m_block_download_max_size = command_line::get_arg(vm, cryptonote::arg_block_download_max_size);
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
m_sync_pruned_blocks = command_line::get_arg(vm, cryptonote::arg_sync_pruned_blocks);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::deinit()
{
return true;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
void t_cryptonote_protocol_handler<t_core>::set_p2p_endpoint(nodetool::i_p2p_endpoint<connection_context>* p2p)
{
if(p2p)
m_p2p = p2p;
else
m_p2p = &m_p2p_stub;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::on_callback(cryptonote_connection_context& context)
{
LOG_PRINT_CCONTEXT_L2("callback fired");
CHECK_AND_ASSERT_MES_CC( context.m_callback_request_count > 0, false, "false callback fired, but context.m_callback_request_count=" << context.m_callback_request_count);
--context.m_callback_request_count;
if(context.m_state == cryptonote_connection_context::state_synchronizing)
{
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
NOTIFY_REQUEST_CHAIN::request r = {};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_needed_objects.clear();
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_core.get_short_chain_history(r.block_ids);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
handler_request_blocks_history( r.block_ids ); // change the limit(?), sleep(?)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
r.prune = m_sync_pruned_blocks;
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = NOTIFY_RESPONSE_CHAIN_ENTRY::ID;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_CHAIN: m_block_ids.size()=" << r.block_ids.size() );
moved all stuff to github
2014-03-03 22:07:58 +00:00
post_notify<NOTIFY_REQUEST_CHAIN>(r, context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting chain");
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
else if(context.m_state == cryptonote_connection_context::state_standby)
{
context.m_state = cryptonote_connection_context::state_synchronizing;
try_add_next_blocks(context);
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
some fixes
2014-03-20 11:46:11 +00:00
void t_cryptonote_protocol_handler<t_core>::log_connections()
{
std::stringstream ss;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
ss.precision(1);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
double down_sum = 0.0;
double down_curr_sum = 0.0;
double up_sum = 0.0;
double up_curr_sum = 0.0;
some fixes
2014-03-20 11:46:11 +00:00
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
ss << std::setw(30) << std::left << "Remote Host"
some fixes
2014-03-20 11:46:11 +00:00
<< std::setw(20) << "Peer id"
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
<< std::setw(20) << "Support Flags"
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< std::setw(30) << "Recv/Sent (inactive,sec)"
some fixes
2014-03-20 11:46:11 +00:00
<< std::setw(25) << "State"
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< std::setw(20) << "Livetime(sec)"
<< std::setw(12) << "Down (kB/s)"
<< std::setw(14) << "Down(now)"
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
<< std::setw(10) << "Up (kB/s)"
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< std::setw(13) << "Up(now)"
<< ENDL;
some fixes
2014-03-20 11:46:11 +00:00
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_p2p->for_each_connection([&](const connection_context& cntxt, nodetool::peerid_type peer_id, uint32_t support_flags)
some fixes
2014-03-20 11:46:11 +00:00
{
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
bool local_ip = cntxt.m_remote_address.is_local();
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
auto connection_time = time(NULL) - cntxt.m_started;
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
ss << std::setw(30) << std::left << std::string(cntxt.m_is_income ? " [INC]":"[OUT]") +
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
cntxt.m_remote_address.str()
always print peer IDs in the same format
2019-06-18 21:16:25 +00:00
<< std::setw(20) << nodetool::peerid_to_string(peer_id)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
<< std::setw(20) << std::hex << support_flags
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< std::setw(30) << std::to_string(cntxt.m_recv_cnt)+ "(" + std::to_string(time(NULL) - cntxt.m_last_recv) + ")" + "/" + std::to_string(cntxt.m_send_cnt) + "(" + std::to_string(time(NULL) - cntxt.m_last_send) + ")"
some fixes
2014-03-20 11:46:11 +00:00
<< std::setw(25) << get_protocol_state_string(cntxt.m_state)
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
<< std::setw(20) << std::to_string(time(NULL) - cntxt.m_started)
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< std::setw(12) << std::fixed << (connection_time == 0 ? 0.0 : cntxt.m_recv_cnt / connection_time / 1024)
<< std::setw(14) << std::fixed << cntxt.m_current_speed_down / 1024
<< std::setw(10) << std::fixed << (connection_time == 0 ? 0.0 : cntxt.m_send_cnt / connection_time / 1024)
<< std::setw(13) << std::fixed << cntxt.m_current_speed_up / 1024
<< (local_ip ? "[LAN]" : "")
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
<< std::left << (cntxt.m_remote_address.is_loopback() ? "[LOCALHOST]" : "") // 127.0.0.1
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
<< ENDL;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
if (connection_time > 1)
{
down_sum += (cntxt.m_recv_cnt / connection_time / 1024);
up_sum += (cntxt.m_send_cnt / connection_time / 1024);
}
down_curr_sum += (cntxt.m_current_speed_down / 1024);
up_curr_sum += (cntxt.m_current_speed_up / 1024);
some fixes
2014-03-20 11:46:11 +00:00
return true;
});
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
ss << ENDL
<< std::setw(125) << " "
<< std::setw(12) << down_sum
<< std::setw(14) << down_curr_sum
<< std::setw(10) << up_sum
<< std::setw(13) << up_curr_sum
<< ENDL;
LOG_PRINT_L0("Connections: " << ENDL << ss.str());
some fixes
2014-03-20 11:46:11 +00:00
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
// Returns a list of connection_info objects describing each open p2p connection
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
template<class t_core>
std::list<connection_info> t_cryptonote_protocol_handler<t_core>::get_connections()
{
std::list<connection_info> connections;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_p2p->for_each_connection([&](const connection_context& cntxt, nodetool::peerid_type peer_id, uint32_t support_flags)
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
{
connection_info cnx;
auto timestamp = time(NULL);
cnx.incoming = cntxt.m_is_income ? true : false;
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
cnx.address = cntxt.m_remote_address.str();
cnx.host = cntxt.m_remote_address.host_str();
cnx.ip = "";
cnx.port = "";
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
if (cntxt.m_remote_address.get_type_id() == epee::net_utils::ipv4_network_address::get_type_id())
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
{
cnx.ip = cnx.host;
cnx.port = std::to_string(cntxt.m_remote_address.as<epee::net_utils::ipv4_network_address>().port());
}
daemon: add --public-node mode, RPC port propagation over P2P
2019-02-24 08:47:49 +00:00
cnx.rpc_port = cntxt.m_rpc_port;
daemon, wallet: new pay for RPC use system Daemons intended for public use can be set up to require payment in the form of hashes in exchange for RPC service. This enables public daemons to receive payment for their work over a large number of calls. This system behaves similarly to a pool, so payment takes the form of valid blocks every so often, yielding a large one off payment, rather than constant micropayments. This system can also be used by third parties as a "paywall" layer, where users of a service can pay for use by mining Monero to the service provider's address. An example of this for web site access is Primo, a Monero mining based website "paywall": https://github.com/selene-kovri/primo This has some advantages: - incentive to run a node providing RPC services, thereby promoting the availability of third party nodes for those who can't run their own - incentive to run your own node instead of using a third party's, thereby promoting decentralization - decentralized: payment is done between a client and server, with no third party needed - private: since the system is "pay as you go", you don't need to identify yourself to claim a long lived balance - no payment occurs on the blockchain, so there is no extra transactional load - one may mine with a beefy server, and use those credits from a phone, by reusing the client ID (at the cost of some privacy) - no barrier to entry: anyone may run a RPC node, and your expected revenue depends on how much work you do - Sybil resistant: if you run 1000 idle RPC nodes, you don't magically get more revenue - no large credit balance maintained on servers, so they have no incentive to exit scam - you can use any/many node(s), since there's little cost in switching servers - market based prices: competition between servers to lower costs - incentive for a distributed third party node system: if some public nodes are overused/slow, traffic can move to others - increases network security - helps counteract mining pools' share of the network hash rate - zero incentive for a payer to "double spend" since a reorg does not give any money back to the miner And some disadvantages: - low power clients will have difficulty mining (but one can optionally mine in advance and/or with a faster machine) - payment is "random", so a server might go a long time without a block before getting one - a public node's overall expected payment may be small Public nodes are expected to compete to find a suitable level for cost of service. The daemon can be set up this way to require payment for RPC services: monerod --rpc-payment-address 4xxxxxx \ --rpc-payment-credits 250 --rpc-payment-difficulty 1000 These values are an example only. The --rpc-payment-difficulty switch selects how hard each "share" should be, similar to a mining pool. The higher the difficulty, the fewer shares a client will find. The --rpc-payment-credits switch selects how many credits are awarded for each share a client finds. Considering both options, clients will be awarded credits/difficulty credits for every hash they calculate. For example, in the command line above, 0.25 credits per hash. A client mining at 100 H/s will therefore get an average of 25 credits per second. For reference, in the current implementation, a credit is enough to sync 20 blocks, so a 100 H/s client that's just starting to use Monero and uses this daemon will be able to sync 500 blocks per second. The wallet can be set to automatically mine if connected to a daemon which requires payment for RPC usage. It will try to keep a balance of 50000 credits, stopping mining when it's at this level, and starting again as credits are spent. With the example above, a new client will mine this much credits in about half an hour, and this target is enough to sync 500000 blocks (currently about a third of the monero blockchain). There are three new settings in the wallet: - credits-target: this is the amount of credits a wallet will try to reach before stopping mining. The default of 0 means 50000 credits. - auto-mine-for-rpc-payment-threshold: this controls the minimum credit rate which the wallet considers worth mining for. If the daemon credits less than this ratio, the wallet will consider mining to be not worth it. In the example above, the rate is 0.25 - persistent-rpc-client-id: if set, this allows the wallet to reuse a client id across runs. This means a public node can tell a wallet that's connecting is the same as one that connected previously, but allows a wallet to keep their credit balance from one run to the other. Since the wallet only mines to keep a small credit balance, this is not normally worth doing. However, someone may want to mine on a fast server, and use that credit balance on a low power device such as a phone. If left unset, a new client ID is generated at each wallet start, for privacy reasons. To mine and use a credit balance on two different devices, you can use the --rpc-client-secret-key switch. A wallet's client secret key can be found using the new rpc_payments command in the wallet. Note: anyone knowing your RPC client secret key is able to use your credit balance. The wallet has a few new commands too: - start_mining_for_rpc: start mining to acquire more credits, regardless of the auto mining settings - stop_mining_for_rpc: stop mining to acquire more credits - rpc_payments: display information about current credits with the currently selected daemon The node has an extra command: - rpc_payments: display information about clients and their balances The node will forget about any balance for clients which have been inactive for 6 months. Balances carry over on node restart.
2018-02-11 15:15:56 +00:00
cnx.rpc_credits_per_hash = cntxt.m_rpc_credits_per_hash;
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
always print peer IDs in the same format
2019-06-18 21:16:25 +00:00
cnx.peer_id = nodetool::peerid_to_string(peer_id);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
cnx.support_flags = support_flags;
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
cnx.recv_count = cntxt.m_recv_cnt;
cryptonote_protocol: fix recv/send idle time before handshake
2017-08-09 20:22:05 +00:00
cnx.recv_idle_time = timestamp - std::max(cntxt.m_started, cntxt.m_last_recv);
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
cnx.send_count = cntxt.m_send_cnt;
cryptonote_protocol: fix recv/send idle time before handshake
2017-08-09 20:22:05 +00:00
cnx.send_idle_time = timestamp - std::max(cntxt.m_started, cntxt.m_last_send);
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
cnx.state = get_protocol_state_string(cntxt.m_state);
cnx.live_time = timestamp - cntxt.m_started;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
abstracted nework addresses All code which was using ip and port now uses a new IPv4 object, subclass of a new network_address class. This will allow easy addition of I2P addresses later (and also IPv6, etc). Both old style and new style peer lists are now sent in the P2P protocol, which is inefficient but allows peers using both codebases to talk to each other. This will be removed in the future. No other subclasses than IPv4 exist yet.
2017-05-27 10:35:54 +00:00
cnx.localhost = cntxt.m_remote_address.is_loopback();
cnx.local_ip = cntxt.m_remote_address.is_local();
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
auto connection_time = time(NULL) - cntxt.m_started;
if (connection_time == 0)
{
cnx.avg_download = 0;
cnx.avg_upload = 0;
}
else
{
cnx.avg_download = cntxt.m_recv_cnt / connection_time / 1024;
cnx.avg_upload = cntxt.m_send_cnt / connection_time / 1024;
}
cnx.current_download = cntxt.m_current_speed_down / 1024;
cnx.current_upload = cntxt.m_current_speed_up / 1024;
make connection_id a string in RPC It's sent as JSON, so raw binary is not appropriate
2017-11-17 23:52:50 +00:00
cnx.connection_id = epee::string_tools::pod_to_hex(cntxt.m_connection_id);
epee: add SSL support RPC connections now have optional tranparent SSL. An optional private key and certificate file can be passed, using the --{rpc,daemon}-ssl-private-key and --{rpc,daemon}-ssl-certificate options. Those have as argument a path to a PEM format private private key and certificate, respectively. If not given, a temporary self signed certificate will be used. SSL can be enabled or disabled using --{rpc}-ssl, which accepts autodetect (default), disabled or enabled. Access can be restricted to particular certificates using the --rpc-ssl-allowed-certificates, which takes a list of paths to PEM encoded certificates. This can allow a wallet to connect to only the daemon they think they're connected to, by forcing SSL and listing the paths to the known good certificates. To generate long term certificates: openssl genrsa -out /tmp/KEY 4096 openssl req -new -key /tmp/KEY -out /tmp/REQ openssl x509 -req -days 999999 -sha256 -in /tmp/REQ -signkey /tmp/KEY -out /tmp/CERT /tmp/KEY is the private key, and /tmp/CERT is the certificate, both in PEM format. /tmp/REQ can be removed. Adjust the last command to set expiration date, etc, as needed. It doesn't make a whole lot of sense for monero anyway, since most servers will run with one time temporary self signed certificates anyway. SSL support is transparent, so all communication is done on the existing ports, with SSL autodetection. This means you can start using an SSL daemon now, but you should not enforce SSL yet or nothing will talk to you.
2018-06-14 22:44:48 +00:00
cnx.ssl = cntxt.m_ssl;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
daemon: print peers' top height in sync_info
2017-08-12 09:59:54 +00:00
cnx.height = cntxt.m_remote_blockchain_height;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
cnx.pruning_seed = cntxt.m_pruning_seed;
daemon: display peer address type in print_cn
2019-05-31 09:09:24 +00:00
cnx.address_type = (uint8_t)cntxt.m_remote_address.get_type_id();
daemon: print peers' top height in sync_info
2017-08-12 09:59:54 +00:00
Added get_connections RPC call to daemon
2014-07-18 23:33:03 +00:00
connections.push_back(cnx);
return true;
});
return connections;
}
some fixes
2014-03-20 11:46:11 +00:00
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::process_payload_sync_data(const CORE_SYNC_DATA& hshd, cryptonote_connection_context& context, bool is_inital)
{
befor -> before Really unique yet consistent spelling mistake
2017-08-13 02:04:55 +00:00
if(context.m_state == cryptonote_connection_context::state_before_handshake && !is_inital)
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
if(context.m_state == cryptonote_connection_context::state_synchronizing)
return true;
protocol: add checks for top block hard fork version We won't even talk to a peer which claims a wrong version for its top block. This will avoid syncing to known bad peers in the first place. Also add IP fails when failing to verify a block.
2017-08-09 08:31:00 +00:00
// from v6, if the peer advertises a top block version, reject if it's not what it should be (will only work if no voting)
cryptonote_protocol: print peer versions when unexpected also avoid integer underflow on zero height
2017-12-07 21:39:48 +00:00
if (hshd.current_height > 0)
protocol: add checks for top block hard fork version We won't even talk to a peer which claims a wrong version for its top block. This will avoid syncing to known bad peers in the first place. Also add IP fails when failing to verify a block.
2017-08-09 08:31:00 +00:00
{
cryptonote_protocol: print peer versions when unexpected also avoid integer underflow on zero height
2017-12-07 21:39:48 +00:00
const uint8_t version = m_core.get_ideal_hard_fork_version(hshd.current_height - 1);
if (version >= 6 && version != hshd.top_version)
{
Only log an error if fork version is higher AND is not known.
2018-04-16 02:02:26 +00:00
if (version < hshd.top_version && version == m_core.get_ideal_hard_fork_version())
protocol: move the "peer claims higher version" warning to debug Because there's a neverending supply of cunts claiming a wrong version just to say "look at me" I guess
2020-05-19 16:13:49 +00:00
MDEBUG(context << " peer claims higher version than we think (" <<
cryptonote_protocol: fix missing space in version mismatch message
2017-12-31 17:32:57 +00:00
(unsigned)hshd.top_version << " for " << (hshd.current_height - 1) << " instead of " << (unsigned)version <<
protocol: move the "peer claims higher version" warning to debug Because there's a neverending supply of cunts claiming a wrong version just to say "look at me" I guess
2020-05-19 16:13:49 +00:00
") - we may be forked from the network and a software upgrade may be needed, or that peer is broken or malicious");
cryptonote_protocol: print peer versions when unexpected also avoid integer underflow on zero height
2017-12-07 21:39:48 +00:00
return false;
}
protocol: add checks for top block hard fork version We won't even talk to a peer which claims a wrong version for its top block. This will avoid syncing to known bad peers in the first place. Also add IP fails when failing to verify a block.
2017-08-09 08:31:00 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
// reject weird pruning schemes
if (hshd.pruning_seed)
{
const uint32_t log_stripes = tools::get_pruning_log_stripes(hshd.pruning_seed);
if (log_stripes != CRYPTONOTE_PRUNING_LOG_STRIPES || tools::get_pruning_stripe(hshd.pruning_seed) > (1u << log_stripes))
{
MWARNING(context << " peer claim unexpected pruning seed " << epee::string_tools::to_string_hex(hshd.pruning_seed) << ", disconnecting");
return false;
}
}
protocol: drop peers that decrease claimed height
2020-12-01 22:18:53 +00:00
if (hshd.current_height < context.m_remote_blockchain_height)
{
MINFO(context << "Claims " << hshd.current_height << ", claimed " << context.m_remote_blockchain_height << " before");
hit_score(context, 1);
}
daemon: print peers' top height in sync_info
2017-08-12 09:59:54 +00:00
context.m_remote_blockchain_height = hshd.current_height;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_pruning_seed = hshd.pruning_seed;
#ifdef CRYPTONOTE_PRUNING_DEBUG_SPOOF_SEED
context.m_pruning_seed = tools::make_pruning_seed(1 + (context.m_remote_address.as<epee::net_utils::ipv4_network_address>().ip()) % (1 << CRYPTONOTE_PRUNING_LOG_STRIPES), CRYPTONOTE_PRUNING_LOG_STRIPES);
LOG_INFO_CC(context, "New connection posing as pruning seed " << epee::string_tools::to_string_hex(context.m_pruning_seed) << ", seed address " << &context.m_pruning_seed);
#endif
daemon: print peers' top height in sync_info
2017-08-12 09:59:54 +00:00
Fix spurious synchronization status msgs Only update target height if it's actually greater than the current target. Only display "synchronized" when current height equals target.
2017-02-15 14:38:38 +00:00
uint64_t target = m_core.get_target_blockchain_height();
protocol: fix "sync done" message not appearing
2017-02-24 00:29:47 +00:00
if (target == 0)
target = m_core.get_current_blockchain_height();
Revert "Fix fake 'network synchronized, begin using' messages" This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2. The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
2016-09-25 18:08:31 +00:00
if(m_core.have_block(hshd.top_id))
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
context.m_state = cryptonote_connection_context::state_normal;
Daemon: Guard against reporting "synchronized" too early The added condition "hshd.current_height >= target" guards against reporting "synchronized" too early in the special situation that the very first peer sending us data is synced to a lower height than ourselves.
2020-01-05 15:38:39 +00:00
if(is_inital && hshd.current_height >= target && target == m_core.get_current_blockchain_height())
moved all stuff to github
2014-03-03 22:07:58 +00:00
on_connection_synchronized();
return true;
}
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
// No chain synchronization over hidden networks (tor, i2p, etc.)
if(context.m_remote_address.get_zone() != epee::net_utils::zone::public_)
{
context.m_state = cryptonote_connection_context::state_normal;
return true;
}
Fix spurious synchronization status msgs Only update target height if it's actually greater than the current target. Only display "synchronized" when current height equals target.
2017-02-15 14:38:38 +00:00
if (hshd.current_height > target)
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
/* As I don't know if accessing hshd from core could be a good practice,
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 20:50:13 +00:00
I prefer pushing target height to the core at the same time it is pushed to the user.
Nz. */
0.8.8update
2014-05-25 17:06:40 +00:00
int64_t diff = static_cast<int64_t>(hshd.current_height) - static_cast<int64_t>(m_core.get_current_blockchain_height());
cryptonote_protocol: simplify and remove unnecessary casts
2017-08-22 16:16:57 +00:00
uint64_t abs_diff = std::abs(diff);
remove "using namespace std" from headers It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-10-10 14:47:08 +00:00
uint64_t max_block_height = std::max(hshd.current_height,m_core.get_current_blockchain_height());
Stagenet
2018-02-16 11:04:04 +00:00
uint64_t last_block_v1 = m_core.get_nettype() == TESTNET ? 624633 : m_core.get_nettype() == MAINNET ? 1009826 : (uint64_t)-1;
remove "using namespace std" from headers It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-10-10 14:47:08 +00:00
uint64_t diff_v2 = max_block_height > last_block_v1 ? std::min(abs_diff, max_block_height - last_block_v1) : 0;
Properly format multiline logs As a side effect, colouring on Windows should now work regardless of version
2019-06-22 14:42:48 +00:00
MCLOG(is_inital ? el::Level::Info : el::Level::Debug, "global", el::Color::Yellow, context << "Sync data returned a new top block candidate: " << m_core.get_current_blockchain_height() << " -> " << hshd.current_height
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
<< " [Your node is " << abs_diff << " blocks (" << tools::get_human_readable_timespan((abs_diff - diff_v2) * DIFFICULTY_TARGET_V1 + diff_v2 * DIFFICULTY_TARGET_V2) << ") "
0.8.8update
2014-05-25 17:06:40 +00:00
<< (0 <= diff ? std::string("behind") : std::string("ahead"))
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
<< "] " << ENDL << "SYNCHRONIZATION started");
protocol: do not switch to unsafe sync mode for just a few blocks
2018-05-28 11:14:11 +00:00
if (hshd.current_height >= m_core.get_current_blockchain_height() + 5) // don't switch to unsafe mode just for a few blocks
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
protocol: do not switch to unsafe sync mode for just a few blocks
2018-05-28 11:14:11 +00:00
m_core.safesyncmode(false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
}
if (m_core.get_target_blockchain_height() == 0) // only when sync starts
{
m_sync_timer.resume();
m_sync_timer.reset();
m_add_timer.pause();
m_add_timer.reset();
m_last_add_end_time = 0;
m_sync_spans_downloaded = 0;
m_sync_old_spans_downloaded = 0;
m_sync_bad_spans_downloaded = 0;
m_sync_download_chain_size = 0;
m_sync_download_objects_size = 0;
}
m_core.set_target_blockchain_height((hshd.current_height));
Fix spurious synchronization status msgs Only update target height if it's actually greater than the current target. Only display "synchronized" when current height equals target.
2017-02-15 14:38:38 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MINFO(context << "Remote blockchain height: " << hshd.current_height << ", id: " << hshd.top_id);
daemon: add '--no-sync' arg to optionally disable blockchain sync
2019-02-25 01:31:45 +00:00
if (m_no_sync)
{
context.m_state = cryptonote_connection_context::state_normal;
return true;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
context.m_state = cryptonote_connection_context::state_synchronizing;
//let the socket to send response to handshake, but request callback, to let send request data after response
LOG_PRINT_CCONTEXT_L2("requesting callback");
++context.m_callback_request_count;
m_p2p->request_callback(context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting callback");
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::get_payload_sync_data(CORE_SYNC_DATA& hshd)
{
m_core.get_blockchain_top(hshd.current_height, hshd.top_id);
cryptonote_protocol: less strict check on top version on connect This allows peers who synced past a fork on the wrong height to reorg to the right chain after they updated their software to include the new version.
2017-08-23 00:22:12 +00:00
hshd.top_version = m_core.get_ideal_hard_fork_version(hshd.current_height);
Make difficulty 128 bit instead of 64 bit Based on Boolberry work by: jahrsg <jahr@jahr.me> cr.zoidberg <crypto.zoidberg@gmail.com>
2019-01-31 10:44:08 +00:00
difficulty_type wide_cumulative_difficulty = m_core.get_block_cumulative_difficulty(hshd.current_height);
fix wide difficulty conversion with some versions of boost
2019-05-01 19:57:34 +00:00
hshd.cumulative_difficulty = (wide_cumulative_difficulty & 0xffffffffffffffff).convert_to<uint64_t>();
hshd.cumulative_difficulty_top64 = ((wide_cumulative_difficulty >> 64) & 0xffffffffffffffff).convert_to<uint64_t>();
moved all stuff to github
2014-03-03 22:07:58 +00:00
hshd.current_height +=1;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
hshd.pruning_seed = m_core.get_blockchain_pruning_seed();
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
Change epee binary output from std::stringstream to byte_stream
2020-10-13 15:15:07 +00:00
bool t_cryptonote_protocol_handler<t_core>::get_payload_sync_data(epee::byte_slice& data)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
CORE_SYNC_DATA hsd = {};
moved all stuff to github
2014-03-03 22:07:58 +00:00
get_payload_sync_data(hsd);
epee::serialization::store_t_to_binary(hsd, data);
return true;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_notify_new_block(int command, NOTIFY_NEW_BLOCK::request& arg, cryptonote_connection_context& context)
{
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
MLOGIF_P2P_MESSAGE(crypto::hash hash; cryptonote::block b; bool ret = cryptonote::parse_and_validate_block_from_blob(arg.b.block, b, &hash);, ret, context << "Received NOTIFY_NEW_BLOCK " << hash << " (height " << arg.current_blockchain_height << ", " << arg.b.txs.size() << " txes)");
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(context.m_state != cryptonote_connection_context::state_normal)
return 1;
cryptonote_core: dandelion - use local height or median height if syncing
2020-11-20 11:37:19 +00:00
if(!is_synchronized()) // can happen if a peer connection goes to normal but another thread still hasn't finished adding queued blocks
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
LOG_DEBUG_CC(context, "Received new block while syncing, ignored");
return 1;
}
Pause miner before preparing for incoming blocks
2015-07-10 21:39:12 +00:00
m_core.pause_mine();
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<block_complete_entry> blocks;
Pause miner before preparing for incoming blocks
2015-07-10 21:39:12 +00:00
blocks.push_back(arg.b);
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
std::vector<block> pblocks;
cryptonote: avoid double parsing blocks when syncing
2018-11-23 13:05:48 +00:00
if (!m_core.prepare_handle_incoming_blocks(blocks, pblocks))
{
LOG_PRINT_CCONTEXT_L1("Block verification failed: prepare_handle_incoming_blocks failed, dropping connection");
drop_connection(context, false, false);
m_core.resume_mine();
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
for(auto tx_blob_it = arg.b.txs.begin(); tx_blob_it!=arg.b.txs.end();tx_blob_it++)
{
cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
m_core.handle_incoming_tx(*tx_blob_it, tvc, relay_method::block, true);
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(tvc.m_verifivation_failed)
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_CCONTEXT_L1("Block verification failed: transaction verification failed, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
m_core.cleanup_handle_incoming_blocks();
Pause miner before preparing for incoming blocks
2015-07-10 21:39:12 +00:00
m_core.resume_mine();
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
}
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
block_verification_context bvc = {};
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
m_core.handle_incoming_block(arg.b.block, pblocks.empty() ? NULL : &pblocks[0], bvc); // got block from handle_notify_new_block
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks(true))
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
m_core.resume_mine();
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_core.resume_mine();
if(bvc.m_verifivation_failed)
{
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
LOG_PRINT_CCONTEXT_L0("Block verification failed, dropping connection");
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
drop_connection_with_score(context, bvc.m_bad_pow ? P2P_IP_FAILS_BEFORE_BLOCK : 1, false);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
if(bvc.m_added_to_main_chain)
{
//TODO: Add here announce protocol usage
relay_block(arg, context);
}else if(bvc.m_marked_as_orphaned)
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_needed_objects.clear();
moved all stuff to github
2014-03-03 22:07:58 +00:00
context.m_state = cryptonote_connection_context::state_synchronizing;
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
NOTIFY_REQUEST_CHAIN::request r = {};
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_core.get_short_chain_history(r.block_ids);
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
r.prune = m_sync_pruned_blocks;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
handler_request_blocks_history( r.block_ids ); // change the limit(?), sleep(?)
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = NOTIFY_RESPONSE_CHAIN_ENTRY::ID;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_CHAIN: m_block_ids.size()=" << r.block_ids.size() );
moved all stuff to github
2014-03-03 22:07:58 +00:00
post_notify<NOTIFY_REQUEST_CHAIN>(r, context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting chain");
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_notify_new_fluffy_block(int command, NOTIFY_NEW_FLUFFY_BLOCK::request& arg, cryptonote_connection_context& context)
{
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
MLOGIF_P2P_MESSAGE(crypto::hash hash; cryptonote::block b; bool ret = cryptonote::parse_and_validate_block_from_blob(arg.b.block, b, &hash);, ret, context << "Received NOTIFY_NEW_FLUFFY_BLOCK " << hash << " (height " << arg.current_blockchain_height << ", " << arg.b.txs.size() << " txes)");
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
if(context.m_state != cryptonote_connection_context::state_normal)
return 1;
cryptonote_core: dandelion - use local height or median height if syncing
2020-11-20 11:37:19 +00:00
if(!is_synchronized()) // can happen if a peer connection goes to normal but another thread still hasn't finished adding queued blocks
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
LOG_DEBUG_CC(context, "Received new block while syncing, ignored");
return 1;
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.pause_mine();
block new_block;
transaction miner_tx;
if(parse_and_validate_block_from_blob(arg.b.block, new_block))
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
// This is a second notification, we must have asked for some missing tx
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
if(!context.m_requested_objects.empty())
{
// What we asked for != to what we received ..
if(context.m_requested_objects.size() != arg.b.txs.size())
{
LOG_ERROR_CCONTEXT
(
"NOTIFY_NEW_FLUFFY_BLOCK -> request/response mismatch, "
<< "block = " << epee::string_tools::pod_to_hex(get_blob_hash(arg.b.block))
<< ", requested = " << context.m_requested_objects.size()
<< ", received = " << new_block.tx_hashes.size()
<< ", dropping connection"
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
std::vector<tx_blob_entry> have_tx;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
// Instead of requesting missing transactions by hash like BTC,
// we do it by index (thanks to a suggestion from moneromooo) because
// we're way cooler .. and also because they're smaller than hashes.
//
// Also, remember to pepper some whitespace changes around to bother
// moneromooo ... only because I <3 him.
cryptonote_protocol: fix std::move usage on different types
2018-01-26 10:13:06 +00:00
std::vector<uint64_t> need_tx_indices;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
transaction tx;
crypto::hash tx_hash;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(auto& tx_blob: arg.b.txs)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
if(parse_and_validate_tx_from_blob(tx_blob.blob, tx))
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
try
{
if(!get_transaction_hash(tx, tx_hash))
{
LOG_PRINT_CCONTEXT_L1
(
"NOTIFY_NEW_FLUFFY_BLOCK: get_transaction_hash failed"
<< ", dropping connection"
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
}
catch(...)
{
LOG_PRINT_CCONTEXT_L1
(
"NOTIFY_NEW_FLUFFY_BLOCK: get_transaction_hash failed"
<< ", exception thrown"
<< ", dropping connection"
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
// hijacking m_requested objects in connection context to patch up
// a possible DOS vector pointed out by @monero-moo where peers keep
// sending (0...n-1) transactions.
// If requested objects is not empty, then we must have asked for
// some missing transacionts, make sure that they're all there.
//
// Can I safely re-use this field? I think so, but someone check me!
if(!context.m_requested_objects.empty())
{
auto req_tx_it = context.m_requested_objects.find(tx_hash);
if(req_tx_it == context.m_requested_objects.end())
{
LOG_ERROR_CCONTEXT
(
"Peer sent wrong transaction (NOTIFY_NEW_FLUFFY_BLOCK): "
<< "transaction with id = " << tx_hash << " wasn't requested, "
<< "dropping connection"
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
context.m_requested_objects.erase(req_tx_it);
}
// we might already have the tx that the peer
// sent in our pool, so don't verify again..
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
if(!m_core.pool_has_tx(tx_hash))
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
MDEBUG("Incoming tx " << tx_hash << " not in pool, adding");
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
if(!m_core.handle_incoming_tx(tx_blob, tvc, relay_method::block, true) || tvc.m_verifivation_failed)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
LOG_PRINT_CCONTEXT_L1("Block verification failed: transaction verification failed, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
//
// future todo:
// tx should only not be added to pool if verification failed, but
// maybe in the future could not be added for other reasons
// according to monero-moo so keep track of these separately ..
//
}
}
else
{
LOG_ERROR_CCONTEXT
(
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
"sent wrong tx: failed to parse and validate transaction: "
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
<< epee::string_tools::buff_to_hex_nodelimer(tx_blob.blob)
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
<< ", dropping connection"
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
}
}
// The initial size equality check could have been fooled if the sender
// gave us the number of transactions we asked for, but not the right
// ones. This check make sure the transactions we asked for were the
// ones we received.
if(context.m_requested_objects.size())
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
(
"NOTIFY_NEW_FLUFFY_BLOCK: peer sent the number of transaction requested"
<< ", but not the actual transactions requested"
<< ", context.m_requested_objects.size() = " << context.m_requested_objects.size()
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
<< ", dropping connection"
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
return 1;
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
size_t tx_idx = 0;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(auto& tx_hash: new_block.tx_hashes)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
cryptonote::blobdata txblob;
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
if(m_core.get_pool_transaction(tx_hash, txblob, relay_category::broadcasted))
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
have_tx.push_back({txblob, crypto::null_hash});
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
else
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
std::vector<crypto::hash> tx_ids;
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<transaction> txes;
std::vector<crypto::hash> missing;
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
tx_ids.push_back(tx_hash);
if (m_core.get_transactions(tx_ids, txes, missing) && missing.empty())
{
protocol: fix wrong tx being looked up from fluffy block Found by smooth
2017-03-13 19:16:25 +00:00
if (txes.size() == 1)
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
have_tx.push_back({tx_to_blob(txes.front()), crypto::null_hash});
protocol: fix wrong tx being looked up from fluffy block Found by smooth
2017-03-13 19:16:25 +00:00
}
else
{
MERROR("1 tx requested, none not found, but " << txes.size() << " returned");
m_core.resume_mine();
return 1;
}
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
}
else
{
MDEBUG("Tx " << tx_hash << " not found in pool");
need_tx_indices.push_back(tx_idx);
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
++tx_idx;
}
if(!need_tx_indices.empty()) // drats, we don't have everything..
{
// request non-mempool txs
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
MDEBUG("We are missing " << need_tx_indices.size() << " txes for this fluffy block");
for (auto txidx: need_tx_indices)
MDEBUG(" tx " << new_block.tx_hashes[txidx]);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
NOTIFY_REQUEST_FLUFFY_MISSING_TX::request missing_tx_req;
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
missing_tx_req.block_hash = get_block_hash(new_block);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
missing_tx_req.current_blockchain_height = arg.current_blockchain_height;
missing_tx_req.missing_tx_indices = std::move(need_tx_indices);
m_core.resume_mine();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_FLUFFY_MISSING_TX: missing_tx_indices.size()=" << missing_tx_req.missing_tx_indices.size() );
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
post_notify<NOTIFY_REQUEST_FLUFFY_MISSING_TX>(missing_tx_req, context);
}
else // whoo-hoo we've got em all ..
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
MDEBUG("We have all needed txes for this fluffy block");
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
block_complete_entry b;
b.block = arg.b.block;
b.txs = have_tx;
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<block_complete_entry> blocks;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
blocks.push_back(b);
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
std::vector<block> pblocks;
cryptonote: avoid double parsing blocks when syncing
2018-11-23 13:05:48 +00:00
if (!m_core.prepare_handle_incoming_blocks(blocks, pblocks))
{
LOG_PRINT_CCONTEXT_L0("Failure in prepare_handle_incoming_blocks");
m_core.resume_mine();
return 1;
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
block_verification_context bvc = {};
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
m_core.handle_incoming_block(arg.b.block, pblocks.empty() ? NULL : &pblocks[0], bvc); // got block from handle_notify_new_block
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks(true))
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
m_core.resume_mine();
return 1;
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.resume_mine();
if( bvc.m_verifivation_failed )
{
LOG_PRINT_CCONTEXT_L0("Block verification failed, dropping connection");
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
drop_connection_with_score(context, bvc.m_bad_pow ? P2P_IP_FAILS_BEFORE_BLOCK : 1, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
return 1;
}
if( bvc.m_added_to_main_chain )
{
//TODO: Add here announce protocol usage
NOTIFY_NEW_BLOCK::request reg_arg = AUTO_VAL_INIT(reg_arg);
reg_arg.current_blockchain_height = arg.current_blockchain_height;
protocol: fix fluffy to normal block dropping txes
2017-03-12 14:14:51 +00:00
reg_arg.b = b;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
relay_block(reg_arg, context);
}
else if( bvc.m_marked_as_orphaned )
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_needed_objects.clear();
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
context.m_state = cryptonote_connection_context::state_synchronizing;
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
NOTIFY_REQUEST_CHAIN::request r = {};
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
m_core.get_short_chain_history(r.block_ids);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
handler_request_blocks_history( r.block_ids ); // change the limit(?), sleep(?)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
r.prune = m_sync_pruned_blocks;
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = NOTIFY_RESPONSE_CHAIN_ENTRY::ID;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_CHAIN: m_block_ids.size()=" << r.block_ids.size() );
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
post_notify<NOTIFY_REQUEST_CHAIN>(r, context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting chain");
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
}
}
else
{
LOG_ERROR_CCONTEXT
(
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
"sent wrong block: failed to parse and validate block: "
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
<< epee::string_tools::buff_to_hex_nodelimer(arg.b.block)
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
<< ", dropping connection"
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
);
m_core.resume_mine();
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
return 1;
}
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
int t_cryptonote_protocol_handler<t_core>::handle_request_fluffy_missing_tx(int command, NOTIFY_REQUEST_FLUFFY_MISSING_TX::request& arg, cryptonote_connection_context& context)
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_REQUEST_FLUFFY_MISSING_TX (" << arg.missing_tx_indices.size() << " txes), block hash " << arg.block_hash);
cryptonote_protocol: reject requests/notifications before handshake Reported by xnbya
2020-05-17 13:48:40 +00:00
if (context.m_state == cryptonote_connection_context::state_before_handshake)
{
LOG_ERROR_CCONTEXT("Requested fluffy tx before handshake, dropping connection");
drop_connection(context, false, false);
return 1;
}
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<std::pair<cryptonote::blobdata, block>> local_blocks;
std::vector<cryptonote::blobdata> local_txs;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
block b;
if (!m_core.get_block_by_hash(arg.block_hash, b))
{
LOG_ERROR_CCONTEXT("failed to find block: " << arg.block_hash << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
return 1;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
std::vector<crypto::hash> txids;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
NOTIFY_NEW_FLUFFY_BLOCK::request fluffy_response;
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
fluffy_response.b.block = t_serializable_object_to_blob(b);
fluffy_response.current_blockchain_height = arg.current_blockchain_height;
cryptonote_protocol: fix another potential P2P DoS When asking for txes in a fluffy transaction, one might ask for the same (large) tx many times
2019-03-09 09:08:53 +00:00
std::vector<bool> seen(b.tx_hashes.size(), false);
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(auto& tx_idx: arg.missing_tx_indices)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
if(tx_idx < b.tx_hashes.size())
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
cryptonote_protocol_handler: fix crash in debug log
2017-06-01 18:42:33 +00:00
MDEBUG(" tx " << b.tx_hashes[tx_idx]);
cryptonote_protocol: fix another potential P2P DoS When asking for txes in a fluffy transaction, one might ask for the same (large) tx many times
2019-03-09 09:08:53 +00:00
if (seen[tx_idx])
{
LOG_ERROR_CCONTEXT
(
"Failed to handle request NOTIFY_REQUEST_FLUFFY_MISSING_TX"
<< ", request is asking for duplicate tx "
<< ", tx index = " << tx_idx << ", block tx count " << b.tx_hashes.size()
<< ", block_height = " << arg.current_blockchain_height
<< ", dropping connection"
);
drop_connection(context, true, false);
return 1;
}
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
txids.push_back(b.tx_hashes[tx_idx]);
cryptonote_protocol: fix another potential P2P DoS When asking for txes in a fluffy transaction, one might ask for the same (large) tx many times
2019-03-09 09:08:53 +00:00
seen[tx_idx] = true;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
else
{
LOG_ERROR_CCONTEXT
(
"Failed to handle request NOTIFY_REQUEST_FLUFFY_MISSING_TX"
<< ", request is asking for a tx whose index is out of bounds "
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
<< ", tx index = " << tx_idx << ", block tx count " << b.tx_hashes.size()
<< ", block_height = " << arg.current_blockchain_height
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
<< ", dropping connection"
);
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
return 1;
}
}
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<cryptonote::transaction> txs;
std::vector<crypto::hash> missed;
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
if (!m_core.get_transactions(txids, txs, missed))
{
LOG_ERROR_CCONTEXT("Failed to handle request NOTIFY_REQUEST_FLUFFY_MISSING_TX, "
<< "failed to get requested transactions");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
return 1;
}
if (!missed.empty() || txs.size() != txids.size())
{
LOG_ERROR_CCONTEXT("Failed to handle request NOTIFY_REQUEST_FLUFFY_MISSING_TX, "
<< missed.size() << " requested transactions not found" << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
return 1;
}
for(auto& tx: txs)
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
fluffy_response.b.txs.push_back({t_serializable_object_to_blob(tx), crypto::null_hash});
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
(
"-->>NOTIFY_RESPONSE_FLUFFY_MISSING_TX: "
<< ", txs.size()=" << fluffy_response.b.txs.size()
<< ", rsp.current_blockchain_height=" << fluffy_response.current_blockchain_height
);
post_notify<NOTIFY_NEW_FLUFFY_BLOCK>(fluffy_response, context);
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_notify_get_txpool_complement(int command, NOTIFY_GET_TXPOOL_COMPLEMENT::request& arg, cryptonote_connection_context& context)
{
MLOG_P2P_MESSAGE("Received NOTIFY_GET_TXPOOL_COMPLEMENT (" << arg.hashes.size() << " txes)");
cryptonote_protocol: reject requests/notifications before handshake Reported by xnbya
2020-05-17 13:48:40 +00:00
if(context.m_state != cryptonote_connection_context::state_normal)
return 1;
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
std::vector<std::pair<cryptonote::blobdata, block>> local_blocks;
std::vector<cryptonote::blobdata> local_txs;
std::vector<cryptonote::blobdata> txes;
if (!m_core.get_txpool_complement(arg.hashes, txes))
{
LOG_ERROR_CCONTEXT("failed to get txpool complement");
return 1;
}
NOTIFY_NEW_TRANSACTIONS::request new_txes;
new_txes.txs = std::move(txes);
MLOG_P2P_MESSAGE
(
"-->>NOTIFY_NEW_TRANSACTIONS: "
<< ", txs.size()=" << new_txes.txs.size()
);
post_notify<NOTIFY_NEW_TRANSACTIONS>(new_txes, context);
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_notify_new_transactions(int command, NOTIFY_NEW_TRANSACTIONS::request& arg, cryptonote_connection_context& context)
{
protocol: log received messages in a separate category Makes it easier to log just what's going on on P2P
2017-01-22 10:14:45 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_NEW_TRANSACTIONS (" << arg.txs.size() << " txes)");
cryptonote_protocol_handler: add block/tx hashes in notify logs
2019-03-27 20:38:42 +00:00
for (const auto &blob: arg.txs)
MLOGIF_P2P_MESSAGE(cryptonote::transaction tx; crypto::hash hash; bool ret = cryptonote::parse_and_validate_tx_from_blob(blob, tx, hash);, ret, "Including transaction " << hash);
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(context.m_state != cryptonote_connection_context::state_normal)
return 1;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
// while syncing, core will lock for a long time, so we ignore
// those txes as they aren't really needed anyway, and avoid a
// long block before replying
cryptonote_core: dandelion - use local height or median height if syncing
2020-11-20 11:37:19 +00:00
if(!is_synchronized())
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
{
LOG_DEBUG_CC(context, "Received new tx while syncing, ignored");
return 1;
}
Add randomized delay when forwarding txes from i2p/tor -> ipv4/6
2019-11-17 17:54:37 +00:00
/* If the txes were received over i2p/tor, the default is to "forward"
with a randomized delay to further enhance the "white noise" behavior,
potentially making it harder for ISP-level spies to determine which
inbound link sent the tx. If the sender disabled "white noise" over
i2p/tor, then the sender is "fluffing" (to only outbound) i2p/tor
connections with the `dandelionpp_fluff` flag set. The receiver (hidden
service) will immediately fluff in that scenario (i.e. this assumes that a
sybil spy will be unable to link an IP to an i2p/tor connection). */
const epee::net_utils::zone zone = context.m_remote_address.get_zone();
relay_method tx_relay = zone == epee::net_utils::zone::public_ ?
relay_method::stem : relay_method::forward;
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
std::vector<blobdata> stem_txs{};
std::vector<blobdata> fluff_txs{};
if (arg.dandelionpp_fluff)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
tx_relay = relay_method::fluff;
fluff_txs.reserve(arg.txs.size());
}
else
stem_txs.reserve(arg.txs.size());
for (auto& tx : arg.txs)
{
tx_verification_context tvc{};
if (!m_core.handle_incoming_tx({tx, crypto::null_hash}, tvc, tx_relay, true))
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_CCONTEXT_L1("Tx verification failed, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
switch (tvc.m_relay)
{
case relay_method::local:
case relay_method::stem:
stem_txs.push_back(std::move(tx));
break;
case relay_method::block:
case relay_method::fluff:
fluff_txs.push_back(std::move(tx));
break;
default:
Add randomized delay when forwarding txes from i2p/tor -> ipv4/6
2019-11-17 17:54:37 +00:00
case relay_method::forward: // not supposed to happen here
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
case relay_method::none:
break;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
if (!stem_txs.empty())
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
//TODO: add announce usage here
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
arg.dandelionpp_fluff = false;
arg.txs = std::move(stem_txs);
relay_transactions(arg, context.m_connection_id, context.m_remote_address.get_zone(), relay_method::stem);
}
if (!fluff_txs.empty())
{
//TODO: add announce usage here
arg.dandelionpp_fluff = true;
arg.txs = std::move(fluff_txs);
relay_transactions(arg, context.m_connection_id, context.m_remote_address.get_zone(), relay_method::fluff);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return 1;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_request_get_objects(int command, NOTIFY_REQUEST_GET_OBJECTS::request& arg, cryptonote_connection_context& context)
{
cryptonote_protocol: reject requests/notifications before handshake Reported by xnbya
2020-05-17 13:48:40 +00:00
if (context.m_state == cryptonote_connection_context::state_before_handshake)
{
LOG_ERROR_CCONTEXT("Requested objects before handshake, dropping connection");
drop_connection(context, false, false);
return 1;
}
Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT
2019-05-31 00:59:19 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_REQUEST_GET_OBJECTS (" << arg.blocks.size() << " blocks)");
if (arg.blocks.size() > CURRENCY_PROTOCOL_MAX_OBJECT_REQUEST_COUNT)
cryptonote_protocol_handler: prevent potential DoS Essentially, one can send such a large amount of IDs that core exhausts all free memory. This issue can theoretically be exploited using very large CN blockchains, such as Monero. This is a partial fix. Thanks and credit given to CryptoNote author 'cryptozoidberg' for collaboration and the fix. Also thanks to 'moneromooo'. Referencing HackerOne report #506595.
2019-03-09 09:11:44 +00:00
{
LOG_ERROR_CCONTEXT(
"Requested objects count is too big ("
Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT
2019-05-31 00:59:19 +00:00
<< arg.blocks.size() << ") expected not more then "
cryptonote_protocol: expand basic DoS protection Count transactions as well
2019-03-08 19:17:20 +00:00
<< CURRENCY_PROTOCOL_MAX_OBJECT_REQUEST_COUNT);
cryptonote_protocol_handler: prevent potential DoS Essentially, one can send such a large amount of IDs that core exhausts all free memory. This issue can theoretically be exploited using very large CN blockchains, such as Monero. This is a partial fix. Thanks and credit given to CryptoNote author 'cryptozoidberg' for collaboration and the fix. Also thanks to 'moneromooo'. Referencing HackerOne report #506595.
2019-03-09 09:11:44 +00:00
drop_connection(context, false, false);
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
NOTIFY_RESPONSE_GET_OBJECTS::request rsp;
if(!m_core.handle_get_objects(arg, rsp, context))
{
LOG_ERROR_CCONTEXT("failed to handle request NOTIFY_REQUEST_GET_OBJECTS, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return 1;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT
2019-05-31 00:59:19 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_RESPONSE_GET_OBJECTS: blocks.size()="
<< rsp.blocks.size() << ", rsp.m_current_blockchain_height=" << rsp.current_blockchain_height
<< ", missed_ids.size()=" << rsp.missed_ids.size());
moved all stuff to github
2014-03-03 22:07:58 +00:00
post_notify<NOTIFY_RESPONSE_GET_OBJECTS>(rsp, context);
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
//handler_response_blocks_now(sizeof(rsp)); // XXX
//handler_response_blocks_now(200);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
template<class t_core>
cryptonote_protocol: light cleanup
2017-07-08 16:59:39 +00:00
double t_cryptonote_protocol_handler<t_core>::get_avg_block_size()
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
CRITICAL_REGION_LOCAL(m_buffer_mutex);
cryptonote_protocol: light cleanup
2017-07-08 16:59:39 +00:00
if (m_avg_buffer.empty()) {
MWARNING("m_avg_buffer.size() == 0");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return 500;
}
cryptonote_protocol: light cleanup
2017-07-08 16:59:39 +00:00
double avg = 0;
for (const auto &element : m_avg_buffer) avg += element;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return avg / m_avg_buffer.size();
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
template<class t_core>
int t_cryptonote_protocol_handler<t_core>::handle_response_get_objects(int command, NOTIFY_RESPONSE_GET_OBJECTS::request& arg, cryptonote_connection_context& context)
{
Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT
2019-05-31 00:59:19 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_RESPONSE_GET_OBJECTS (" << arg.blocks.size() << " blocks)");
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("received objects");
boost::posix_time::ptime request_time = context.m_last_request_time;
context.m_last_request_time = boost::date_time::not_a_date_time;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
if (context.m_expect_response != NOTIFY_RESPONSE_GET_OBJECTS::ID)
{
LOG_ERROR_CCONTEXT("Got NOTIFY_RESPONSE_GET_OBJECTS out of the blue, dropping connection");
drop_connection(context, true, false);
return 1;
}
context.m_expect_response = 0;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// calculate size of request
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
size_t size = 0;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
size_t blocks_size = 0;
cryptonote_protocol: light cleanup
2017-07-08 16:59:39 +00:00
for (const auto &element : arg.blocks) {
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
blocks_size += element.block.size();
for (const auto &tx : element.txs)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
blocks_size += tx.blob.size();
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
size += blocks_size;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol: light cleanup
2017-07-08 16:59:39 +00:00
for (const auto &element : arg.missed_ids)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
size += sizeof(element.data);
size += sizeof(arg.current_blockchain_height);
{
CRITICAL_REGION_LOCAL(m_buffer_mutex);
m_avg_buffer.push_back(size);
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_spans_downloaded;
m_sync_download_objects_size += size;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
MDEBUG(context << " downloaded " << size << " bytes worth of blocks");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
/*using namespace boost::chrono;
auto point = steady_clock::now();
auto time_from_epoh = point.time_since_epoch();
auto sec = duration_cast< seconds >( time_from_epoh ).count();*/
//epee::net_utils::network_throttle_manager::get_global_throttle_inreq().logger_handle_net("log/dr-monero/net/req-all.data", sec, get_avg_block_size());
protocol: reject empty incoming block messages
2020-11-03 12:59:36 +00:00
if(arg.blocks.empty())
{
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_HAVE_OBJECTS: no blocks");
drop_connection(context, true, false);
++m_sync_bad_spans_downloaded;
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(context.m_last_response_height > arg.current_blockchain_height)
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_HAVE_OBJECTS: arg.m_current_blockchain_height=" << arg.current_blockchain_height
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< " < m_last_response_height=" << context.m_last_response_height << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
protocol: drop peers that decrease claimed height
2020-12-01 22:18:53 +00:00
if (arg.current_blockchain_height < context.m_remote_blockchain_height)
{
MINFO(context << "Claims " << arg.current_blockchain_height << ", claimed " << context.m_remote_blockchain_height << " before");
hit_score(context, 1);
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
context.m_remote_blockchain_height = arg.current_blockchain_height;
cryptonote_protocol: update target height when receiving blocks
2017-12-27 13:54:19 +00:00
if (context.m_remote_blockchain_height > m_core.get_target_blockchain_height())
m_core.set_target_blockchain_height(context.m_remote_blockchain_height);
moved all stuff to github
2014-03-03 22:07:58 +00:00
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
std::vector<crypto::hash> block_hashes;
block_hashes.reserve(arg.blocks.size());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
uint64_t start_height = std::numeric_limits<uint64_t>::max();
cryptonote::block b;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(const block_complete_entry& block_entry: arg.blocks)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Fix delayed exit when syncing
2016-12-04 12:27:45 +00:00
if (m_stopping)
{
return 1;
}
cryptonote: block parsing + hash calculation speedup This saves a duplicate serialization step
2018-12-13 19:47:47 +00:00
crypto::hash block_hash;
if(!parse_and_validate_block_from_blob(block_entry.block, b, block_hash))
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
LOG_ERROR_CCONTEXT("sent wrong block: failed to parse and validate block: "
<< epee::string_tools::buff_to_hex_nodelimer(block_entry.block) << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (b.miner_tx.vin.size() != 1 || b.miner_tx.vin.front().type() != typeid(txin_gen))
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
{
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
LOG_ERROR_CCONTEXT("sent wrong block: block: miner tx does not have exactly one txin_gen input"
<< epee::string_tools::buff_to_hex_nodelimer(block_entry.block) << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return 1;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (start_height == std::numeric_limits<uint64_t>::max())
start_height = boost::get<txin_gen>(b.miner_tx.vin[0]).height;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
auto req_it = context.m_requested_objects.find(block_hash);
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(req_it == context.m_requested_objects.end())
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_RESPONSE_GET_OBJECTS: block with id=" << epee::string_tools::pod_to_hex(get_blob_hash(block_entry.block))
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< " wasn't requested, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
if(b.tx_hashes.size() != block_entry.txs.size())
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_RESPONSE_GET_OBJECTS: block with id=" << epee::string_tools::pod_to_hex(get_blob_hash(block_entry.block))
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ", tx_hashes.size()=" << b.tx_hashes.size() << " mismatch with block_complete_entry.m_txs.size()=" << block_entry.txs.size() << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
context.m_requested_objects.erase(req_it);
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
block_hashes.push_back(block_hash);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if(!context.m_requested_objects.empty())
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MERROR(context << "returned not all requested objects (context.m_requested_objects.size()="
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
<< context.m_requested_objects.size() << "), dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++m_sync_bad_spans_downloaded;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
const bool pruned_ok = should_ask_for_pruned_data(context, start_height, arg.blocks.size(), true);
if (!pruned_ok)
{
// if we don't want pruned data, check we did not get any
for (block_complete_entry& block_entry: arg.blocks)
{
if (block_entry.pruned)
{
MERROR(context << "returned a pruned block, dropping connection");
drop_connection(context, false, false);
++m_sync_bad_spans_downloaded;
return 1;
}
if (block_entry.block_weight)
{
MERROR(context << "returned a block weight for a non pruned block, dropping connection");
drop_connection(context, false, false);
++m_sync_bad_spans_downloaded;
return 1;
}
for (const tx_blob_entry &tx_entry: block_entry.txs)
{
if (tx_entry.prunable_hash != crypto::null_hash)
{
MERROR(context << "returned at least one pruned object which we did not expect, dropping connection");
drop_connection(context, false, false);
++m_sync_bad_spans_downloaded;
return 1;
}
}
}
}
else
{
// we accept pruned data, check that if we got some, then no weights are zero
for (block_complete_entry& block_entry: arg.blocks)
{
if (block_entry.block_weight == 0 && block_entry.pruned)
{
MERROR(context << "returned at least one pruned block with 0 weight, dropping connection");
drop_connection(context, false, false);
++m_sync_bad_spans_downloaded;
return 1;
}
}
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
protocol: fix reorgs while syncing
2017-08-12 09:59:27 +00:00
MLOG_YELLOW(el::Level::Debug, context << " Got NEW BLOCKS inside of " << __FUNCTION__ << ": size: " << arg.blocks.size()
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
<< ", blocks: " << start_height << " - " << (start_height + arg.blocks.size() - 1) <<
" (pruning seed " << epee::string_tools::to_string_hex(context.m_pruning_seed) << ")");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// add that new span to the block queue
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const boost::posix_time::time_duration dt = now - request_time;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
const float rate = size * 1e6 / (dt.total_microseconds() + 1);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << " adding span: " << arg.blocks.size() << " at height " << start_height << ", " << dt.total_microseconds()/1e6 << " seconds, " << (rate/1024) << " kB/s, size now " << (m_block_queue.get_data_size() + blocks_size) / 1048576.f << " MB");
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
m_block_queue.add_blocks(start_height, arg.blocks, context.m_connection_id, context.m_remote_address, rate, blocks_size);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const crypto::hash last_block_hash = cryptonote::get_block_hash(b);
cryptonote_protocol: fix needless chain hashes downloads The last known hash was calculated incorrectly, causing further chain hash downloads to restart from the current chain. When the block queue has close to 10k blocks waiting, this causes frequent downloads of 10k more hashes, but with only the last few hashes actually being useful.
2017-09-17 13:13:59 +00:00
context.m_last_known_hash = last_block_hash;
cryptonote_protocol: fix "holes" in block download schedule
2017-08-15 16:49:06 +00:00
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
if (!m_core.get_test_drop_download() || !m_core.get_test_drop_download_height()) { // DISCARD BLOCKS for testing
return 1;
}
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
try_add_next_blocks(context);
return 1;
}
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
// Get an estimate for the remaining sync time from given current to target blockchain height, in seconds
template<class t_core>
uint64_t t_cryptonote_protocol_handler<t_core>::get_estimated_remaining_sync_seconds(uint64_t current_blockchain_height, uint64_t target_blockchain_height)
{
// The average sync speed varies so much, even averaged over quite long time periods like 10 minutes,
// that using some sliding window would be difficult to implement without often leading to bad estimates.
// The simplest strategy - always average sync speed over the maximum available interval i.e. since sync
// started at all (from "m_sync_start_time" and "m_sync_start_height") - gives already useful results
// and seems to be quite robust. Some quite special cases like "Internet connection suddenly becoming
// much faster after syncing already a long time, and staying fast" are not well supported however.
if (target_blockchain_height <= current_blockchain_height)
{
// Syncing stuck, or other special circumstance: Avoid errors, simply give back 0
return 0;
}
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
const boost::posix_time::time_duration sync_time = now - m_sync_start_time;
cryptonote::network_type nettype = m_core.get_nettype();
// Don't simply use remaining number of blocks for the estimate but "sync weight" as provided by
// "cumulative_block_sync_weight" which knows about strongly varying Monero mainnet block sizes
uint64_t synced_weight = tools::cumulative_block_sync_weight(nettype, m_sync_start_height, current_blockchain_height - m_sync_start_height);
float us_per_weight = (float)sync_time.total_microseconds() / (float)synced_weight;
uint64_t remaining_weight = tools::cumulative_block_sync_weight(nettype, current_blockchain_height, target_blockchain_height - current_blockchain_height);
float remaining_us = us_per_weight * (float)remaining_weight;
return (uint64_t)(remaining_us / 1e6);
}
// Return a textual remaining sync time estimate, or the empty string if waiting period not yet over
template<class t_core>
std::string t_cryptonote_protocol_handler<t_core>::get_periodic_sync_estimate(uint64_t current_blockchain_height, uint64_t target_blockchain_height)
{
std::string text = "";
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
boost::posix_time::time_duration period_sync_time = now - m_period_start_time;
if (period_sync_time > boost::posix_time::minutes(2))
{
// Period is over, time to report another estimate
uint64_t remaining_seconds = get_estimated_remaining_sync_seconds(current_blockchain_height, target_blockchain_height);
text = tools::get_human_readable_timespan(remaining_seconds);
// Start the new period
m_period_start_time = now;
}
return text;
}
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
template<class t_core>
int t_cryptonote_protocol_handler<t_core>::try_add_next_blocks(cryptonote_connection_context& context)
{
bool force_next_span = false;
{
// We try to lock the sync lock. If we can, it means no other thread is
// currently adding blocks, so we do that for as long as we can from the
// block queue. Then, we go back to download.
const boost::unique_lock<boost::mutex> sync{m_sync_lock, boost::try_to_lock};
if (!sync.owns_lock())
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MINFO(context << "Failed to lock m_sync_lock, going back to download");
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
goto skip;
}
MDEBUG(context << " lock m_sync_lock, adding blocks to chain...");
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("adding blocks");
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
{
cryptonote_protocol: tweaks to the syncing speedup - only pause mining once we've got the lock (in practice, it'll already be paused by another thread if we can't get the lock at once though) - do not call prepare_handle_incoming_blocks when we dismissed all the blocks, it only causes cleanup_handle_incoming_blocks to complain afterwards
2017-02-13 19:36:31 +00:00
m_core.pause_mine();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_add_timer.resume();
bool starting = true;
epee::misc_utils::auto_scope_leave_caller scope_exit_handler = epee::misc_utils::create_scope_leave_handler([this, &starting]() {
m_add_timer.pause();
m_core.resume_mine();
if (!starting)
m_last_add_end_time = tools::get_tick_count();
});
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
m_sync_start_time = boost::posix_time::microsec_clock::universal_time();
m_sync_start_height = m_core.get_current_blockchain_height();
m_period_start_time = m_sync_start_time;
cryptonote_protocol: tweaks to the syncing speedup - only pause mining once we've got the lock (in practice, it'll already be paused by another thread if we can't get the lock at once though) - do not call prepare_handle_incoming_blocks when we dismissed all the blocks, it only causes cleanup_handle_incoming_blocks to complain afterwards
2017-02-13 19:36:31 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
while (1)
{
const uint64_t previous_height = m_core.get_current_blockchain_height();
uint64_t start_height;
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<cryptonote::block_complete_entry> blocks;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
boost::uuids::uuid span_connection_id;
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
epee::net_utils::network_address span_origin;
if (!m_block_queue.get_next_span(start_height, blocks, span_connection_id, span_origin))
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
MDEBUG(context << " no next span found, going back to download");
break;
}
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
add empty container sanity checks when using front() and back()
2017-12-11 22:36:58 +00:00
if (blocks.empty())
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MERROR(context << "Next span has no blocks");
cryptonote_protocol_handler.inl: remove span read just now that failed to pass some basic tests
2018-04-28 02:10:59 +00:00
m_block_queue.remove_spans(span_connection_id, start_height);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
continue;
add empty container sanity checks when using front() and back()
2017-12-11 22:36:58 +00:00
}
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << " next span in the queue has blocks " << start_height << "-" << (start_height + blocks.size() - 1)
<< ", we need " << previous_height);
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
block new_block;
cryptonote: block parsing + hash calculation speedup This saves a duplicate serialization step
2018-12-13 19:47:47 +00:00
crypto::hash last_block_hash;
if (!parse_and_validate_block_from_blob(blocks.back().block, new_block, last_block_hash))
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
MERROR(context << "Failed to parse block, but it should already have been parsed");
m_block_queue.remove_spans(span_connection_id, start_height);
continue;
}
if (m_core.have_block(last_block_hash))
{
const uint64_t subchain_height = start_height + blocks.size();
LOG_DEBUG_CC(context, "These are old blocks, ignoring: blocks " << start_height << " - " << (subchain_height-1) << ", blockchain height " << m_core.get_current_blockchain_height());
m_block_queue.remove_spans(span_connection_id, start_height);
++m_sync_old_spans_downloaded;
continue;
}
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
if (!parse_and_validate_block_from_blob(blocks.front().block, new_block))
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MERROR(context << "Failed to parse block, but it should already have been parsed");
cryptonote_protocol_handler.inl: remove span read just now that failed to pass some basic tests
2018-04-28 02:10:59 +00:00
m_block_queue.remove_spans(span_connection_id, start_height);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
continue;
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
}
bool parent_known = m_core.have_block(new_block.prev_id);
if (!parent_known)
{
// it could be:
// - later in the current chain
// - later in an alt chain
// - orphan
// if it was requested, then it'll be resolved later, otherwise it's an orphan
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
bool parent_requested = m_block_queue.requested(new_block.prev_id);
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
if (!parent_requested)
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
// we might be able to ask for that block directly, as we now can request out of order,
// otherwise we continue out of order, unless this block is the one we need, in which
// case we request block hashes, though it might be safer to disconnect ?
if (start_height > previous_height)
{
if (should_drop_connection(context, get_next_needed_pruning_stripe().first))
{
MDEBUG(context << "Got block with unknown parent which was not requested, but peer does not have that block - dropping connection");
if (!context.m_is_income)
m_p2p->add_used_stripe_peer(context);
drop_connection(context, false, true);
return 1;
}
MDEBUG(context << "Got block with unknown parent which was not requested, but peer does not have that block - back to download");
goto skip;
}
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
// this can happen if a connection was sicced onto a late span, if it did not have those blocks,
// since we don't know that at the sic time
LOG_ERROR_CCONTEXT("Got block with unknown parent which was not requested - querying block hashes");
cryptonote_protocol_handler.inl: remove span read just now that failed to pass some basic tests
2018-04-28 02:10:59 +00:00
m_block_queue.remove_spans(span_connection_id, start_height);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
context.m_needed_objects.clear();
context.m_last_response_height = 0;
goto skip;
cryptonote_protocol: fix out of order addition This was broken by the reorg fix, since we now have to add blocks regardless of their starting height. We now check whether we know the parent for the first block in the next span, or whether it was requested. If neither, it's an orphan. If it is not known, but was requested, we wait to get that block.
2017-08-15 15:54:08 +00:00
}
// parent was requested, so we wait for it to be retrieved
MINFO(context << " parent was requested, we'll get back to it");
break;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
const boost::posix_time::ptime start = boost::posix_time::microsec_clock::universal_time();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (starting)
{
starting = false;
if (m_last_add_end_time)
{
const uint64_t tnow = tools::get_tick_count();
const uint64_t ns = tools::ticks_to_ns(tnow - m_last_add_end_time);
MINFO("Restarting adding block after idle for " << ns/1e9 << " seconds");
}
}
cryptonote_protocol: tweaks to the syncing speedup - only pause mining once we've got the lock (in practice, it'll already be paused by another thread if we can't get the lock at once though) - do not call prepare_handle_incoming_blocks when we dismissed all the blocks, it only causes cleanup_handle_incoming_blocks to complain afterwards
2017-02-13 19:36:31 +00:00
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
std::vector<block> pblocks;
cryptonote: avoid double parsing blocks when syncing
2018-11-23 13:05:48 +00:00
if (!m_core.prepare_handle_incoming_blocks(blocks, pblocks))
{
LOG_ERROR_CCONTEXT("Failure in prepare_handle_incoming_blocks");
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
drop_connections(span_origin);
cryptonote: avoid double parsing blocks when syncing
2018-11-23 13:05:48 +00:00
return 1;
}
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
if (!pblocks.empty() && pblocks.size() != blocks.size())
{
m_core.cleanup_handle_incoming_blocks();
LOG_ERROR_CCONTEXT("Internal error: blocks.size() != block_entry.txs.size()");
return 1;
}
protocol: speed up sync by minimizing duplicate work In particular, the prepare_handle_incoming_blocks call is pretty lengthy, and entirely pointless in the common case where several different connections will prepare the exact same blocks.
2017-02-12 17:17:30 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
uint64_t block_process_time_full = 0, transactions_process_time_full = 0;
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
size_t num_txs = 0, blockidx = 0;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
for(const block_complete_entry& block_entry: blocks)
Fix delayed exit when syncing
2016-12-04 12:27:45 +00:00
{
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (m_stopping)
{
m_core.cleanup_handle_incoming_blocks();
return 1;
}
Fix delayed exit when syncing
2016-12-04 12:27:45 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// process transactions
TIME_MEASURE_START(transactions_process_time);
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
num_txs += block_entry.txs.size();
std::vector<tx_verification_context> tvc;
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
m_core.handle_incoming_txs(block_entry.txs, tvc, relay_method::block, true);
cryptonote_protocol: internal error consistency size check
2017-12-24 13:04:40 +00:00
if (tvc.size() != block_entry.txs.size())
{
LOG_ERROR_CCONTEXT("Internal error: tvc.size() != block_entry.txs.size()");
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
if (!m_core.cleanup_handle_incoming_blocks())
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
return 1;
}
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return 1;
cryptonote_protocol: internal error consistency size check
2017-12-24 13:04:40 +00:00
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
std::vector<tx_blob_entry>::const_iterator it = block_entry.txs.begin();
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
for (size_t i = 0; i < tvc.size(); ++i, ++it)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
{
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
if(tvc[i].m_verifivation_failed)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
drop_connections(span_origin);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!m_p2p->for_connection(span_connection_id, [&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t f)->bool{
protocol: fix incorrect tx hash in log
2018-11-17 12:47:59 +00:00
cryptonote::transaction tx;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
crypto::hash txid;
if (it->prunable_hash == crypto::null_hash)
{
parse_and_validate_tx_from_blob(it->blob, tx, txid); // must succeed if we got here
}
else
{
parse_and_validate_tx_base_from_blob(it->blob, tx); // must succeed if we got here
txid = get_pruned_transaction_hash(tx, it->prunable_hash);
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
LOG_ERROR_CCONTEXT("transaction verification failed on NOTIFY_RESPONSE_GET_OBJECTS, tx_id = "
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
<< epee::string_tools::pod_to_hex(txid) << ", dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, true);
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return 1;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}))
LOG_ERROR_CCONTEXT("span connection id not found");
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks())
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
return 1;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// in case the peer had dropped beforehand, remove the span anyway so other threads can wake up and get it
m_block_queue.remove_spans(span_connection_id, start_height);
return 1;
}
}
TIME_MEASURE_FINISH(transactions_process_time);
transactions_process_time_full += transactions_process_time;
// process block
TIME_MEASURE_START(block_process_time);
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
block_verification_context bvc = {};
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
m_core.handle_incoming_block(block_entry.block, pblocks.empty() ? NULL : &pblocks[blockidx], bvc, false); // <--- process block
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if(bvc.m_verifivation_failed)
{
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
drop_connections(span_origin);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!m_p2p->for_connection(span_connection_id, [&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t f)->bool{
LOG_PRINT_CCONTEXT_L1("Block verification failed, dropping connection");
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
drop_connection_with_score(context, bvc.m_bad_pow ? P2P_IP_FAILS_BEFORE_BLOCK : 1, true);
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return 1;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}))
LOG_ERROR_CCONTEXT("span connection id not found");
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks())
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
return 1;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// in case the peer had dropped beforehand, remove the span anyway so other threads can wake up and get it
m_block_queue.remove_spans(span_connection_id, start_height);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return 1;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if(bvc.m_marked_as_orphaned)
{
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
drop_connections(span_origin);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!m_p2p->for_connection(span_connection_id, [&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t f)->bool{
LOG_PRINT_CCONTEXT_L1("Block received at sync phase was marked as orphaned, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, true, true);
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return 1;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}))
LOG_ERROR_CCONTEXT("span connection id not found");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks())
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
return 1;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// in case the peer had dropped beforehand, remove the span anyway so other threads can wake up and get it
m_block_queue.remove_spans(span_connection_id, start_height);
return 1;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
TIME_MEASURE_FINISH(block_process_time);
block_process_time_full += block_process_time;
core: avoid double parsing blocks after hoh
2018-11-23 11:15:04 +00:00
++blockidx;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
} // each download block
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << "Block process time (" << blocks.size() << " blocks, " << num_txs << " txs): " << block_process_time_full + transactions_process_time_full << " (" << transactions_process_time_full << "/" << block_process_time_full << ") ms");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol: error handling on cleanup_handle_incoming_blocks
2017-08-29 15:10:53 +00:00
if (!m_core.cleanup_handle_incoming_blocks())
{
LOG_PRINT_CCONTEXT_L0("Failure in cleanup_handle_incoming_blocks");
return 1;
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
protocol: fix reorgs while syncing
2017-08-12 09:59:27 +00:00
m_block_queue.remove_spans(span_connection_id, start_height);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint64_t current_blockchain_height = m_core.get_current_blockchain_height();
if (current_blockchain_height > previous_height)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint64_t target_blockchain_height = m_core.get_target_blockchain_height();
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
const boost::posix_time::time_duration dt = boost::posix_time::microsec_clock::universal_time() - start;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
std::string progress_message = "";
if (current_blockchain_height < target_blockchain_height)
{
uint64_t completion_percent = (current_blockchain_height * 100 / target_blockchain_height);
if (completion_percent == 100) // never show 100% if not actually up to date
completion_percent = 99;
progress_message = " (" + std::to_string(completion_percent) + "%, "
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
+ std::to_string(target_blockchain_height - current_blockchain_height) + " left";
std::string time_message = get_periodic_sync_estimate(current_blockchain_height, target_blockchain_height);
if (!time_message.empty())
{
uint64_t total_blocks_to_sync = target_blockchain_height - m_sync_start_height;
uint64_t total_blocks_synced = current_blockchain_height - m_sync_start_height;
progress_message += ", " + std::to_string(total_blocks_synced * 100 / total_blocks_to_sync) + "% of total synced";
progress_message += ", estimated " + time_message + " left";
}
progress_message += ")";
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
}
const uint32_t previous_stripe = tools::get_pruning_stripe(previous_height, target_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
const uint32_t current_stripe = tools::get_pruning_stripe(current_blockchain_height, target_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
std::string timing_message = "";
if (ELPP->vRegistry()->allowed(el::Level::Info, "sync-info"))
timing_message = std::string(" (") + std::to_string(dt.total_microseconds()/1e6) + " sec, "
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
+ std::to_string((current_blockchain_height - previous_height) * 1e6 / dt.total_microseconds())
+ " blocks/sec), " + std::to_string(m_block_queue.get_data_size() / 1048576.f) + " MB queued in "
+ std::to_string(m_block_queue.get_num_filled_spans()) + " spans, stripe "
+ std::to_string(previous_stripe) + " -> " + std::to_string(current_stripe);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (ELPP->vRegistry()->allowed(el::Level::Debug, "sync-info"))
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
timing_message += std::string(": ") + m_block_queue.get_overview(current_blockchain_height);
MGINFO_YELLOW("Synced " << current_blockchain_height << "/" << target_blockchain_height
<< progress_message << timing_message);
if (previous_stripe != current_stripe)
notify_new_stripe(context, current_stripe);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
}
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("stopping adding blocks");
if (should_download_next_span(context, false))
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
force_next_span = true;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
else if (should_drop_connection(context, get_next_needed_pruning_stripe().first))
{
if (!context.m_is_income)
{
m_p2p->add_used_stripe_peer(context);
drop_connection(context, false, false);
}
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
cryptonote_protocol: tweaks to the syncing speedup - only pause mining once we've got the lock (in practice, it'll already be paused by another thread if we can't get the lock at once though) - do not call prepare_handle_incoming_blocks when we dismissed all the blocks, it only causes cleanup_handle_incoming_blocks to complain afterwards
2017-02-13 19:36:31 +00:00
skip:
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!request_missing_objects(context, true, force_next_span))
{
LOG_ERROR_CCONTEXT("Failed to request missing objects, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
void t_cryptonote_protocol_handler<t_core>::notify_new_stripe(cryptonote_connection_context& cntxt, uint32_t stripe)
{
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool
{
if (cntxt.m_connection_id == context.m_connection_id)
return true;
if (context.m_state == cryptonote_connection_context::state_normal)
{
const uint32_t peer_stripe = tools::get_pruning_stripe(context.m_pruning_seed);
if (stripe && peer_stripe && peer_stripe != stripe)
return true;
context.m_state = cryptonote_connection_context::state_synchronizing;
LOG_PRINT_CCONTEXT_L2("requesting callback");
++context.m_callback_request_count;
m_p2p->request_callback(context);
MLOG_PEER_STATE("requesting callback");
}
return true;
});
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::on_idle()
{
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
m_idle_peer_kicker.do_call(boost::bind(&t_cryptonote_protocol_handler<t_core>::kick_idle_peers, this));
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
m_standby_checker.do_call(boost::bind(&t_cryptonote_protocol_handler<t_core>::check_standby_peers, this));
cryptonote_protocol_handler: search for syncing peers in "cruise mode" When all our outgoing peer slots are filled, we cycle one peer at a time looking for syncing peers until we have at least two such peers. This brings two advantages: - Peers without incoming connections will find more syncing peers that before, thereby strengthening network decentralization - Peers will have more resistance to isolation attacks, as they are more likely to find a "good" peer than they were before
2019-02-26 12:40:40 +00:00
m_sync_search_checker.do_call(boost::bind(&t_cryptonote_protocol_handler<t_core>::update_sync_search, this));
moved all stuff to github
2014-03-03 22:07:58 +00:00
return m_core.on_idle();
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
bool t_cryptonote_protocol_handler<t_core>::kick_idle_peers()
{
MTRACE("Checking for idle peers...");
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
std::vector<std::pair<boost::uuids::uuid, unsigned>> idle_peers;
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (context.m_state == cryptonote_connection_context::state_synchronizing && context.m_last_request_time != boost::date_time::not_a_date_time)
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
{
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
const boost::posix_time::time_duration dt = now - context.m_last_request_time;
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
const auto ms = dt.total_microseconds();
if (ms > IDLE_PEER_KICK_TIME || (context.m_expect_response && ms > NON_RESPONSIVE_PEER_KICK_TIME))
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
{
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
if (context.m_score-- >= 0)
{
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
MINFO(context << " kicking idle peer, last update " << (dt.total_microseconds() / 1.e6) << " seconds ago, expecting " << (int)context.m_expect_response);
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
LOG_PRINT_CCONTEXT_L2("requesting callback");
context.m_last_request_time = boost::date_time::not_a_date_time;
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = 0;
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
context.m_state = cryptonote_connection_context::state_standby; // we'll go back to adding, then (if we can't), download
++context.m_callback_request_count;
m_p2p->request_callback(context);
}
else
{
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
idle_peers.push_back(std::make_pair(context.m_connection_id, context.m_expect_response == 0 ? 1 : 5));
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
}
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
}
}
return true;
});
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
for (const auto &e: idle_peers)
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
{
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
const auto &uuid = e.first;
m_p2p->for_connection(uuid, [&](cryptonote_connection_context& ctx, nodetool::peerid_type peer_id, uint32_t f)->bool{
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
MINFO(ctx << "dropping idle peer with negative score");
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
drop_connection_with_score(ctx, e.second, false);
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
return true;
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
});
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
}
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
cryptonote_protocol_handler: search for syncing peers in "cruise mode" When all our outgoing peer slots are filled, we cycle one peer at a time looking for syncing peers until we have at least two such peers. This brings two advantages: - Peers without incoming connections will find more syncing peers that before, thereby strengthening network decentralization - Peers will have more resistance to isolation attacks, as they are more likely to find a "good" peer than they were before
2019-02-26 12:40:40 +00:00
bool t_cryptonote_protocol_handler<t_core>::update_sync_search()
{
const uint64_t target = m_core.get_target_blockchain_height();
const uint64_t height = m_core.get_current_blockchain_height();
if (target > height) // if we're not synced yet, don't do it
return true;
MTRACE("Checking for outgoing syncing peers...");
unsigned n_syncing = 0, n_synced = 0;
boost::uuids::uuid last_synced_peer_id(boost::uuids::nil_uuid());
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool
{
if (!peer_id || context.m_is_income) // only consider connected outgoing peers
return true;
if (context.m_state == cryptonote_connection_context::state_synchronizing)
++n_syncing;
if (context.m_state == cryptonote_connection_context::state_normal)
{
++n_synced;
if (!context.m_anchor)
last_synced_peer_id = context.m_connection_id;
}
return true;
});
MTRACE(n_syncing << " syncing, " << n_synced << " synced");
// if we're at max out peers, and not enough are syncing
if (n_synced + n_syncing >= m_max_out_peers && n_syncing < P2P_DEFAULT_SYNC_SEARCH_CONNECTIONS_COUNT && last_synced_peer_id != boost::uuids::nil_uuid())
{
if (!m_p2p->for_connection(last_synced_peer_id, [&](cryptonote_connection_context& ctx, nodetool::peerid_type peer_id, uint32_t f)->bool{
MINFO(ctx << "dropping synced peer, " << n_syncing << " syncing, " << n_synced << " synced");
drop_connection(ctx, false, false);
return true;
}))
MDEBUG("Failed to find peer we wanted to drop");
}
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
bool t_cryptonote_protocol_handler<t_core>::check_standby_peers()
{
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool
{
if (context.m_state == cryptonote_connection_context::state_standby)
{
LOG_PRINT_CCONTEXT_L2("requesting callback");
++context.m_callback_request_count;
m_p2p->request_callback(context);
}
return true;
});
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_request_chain(int command, NOTIFY_REQUEST_CHAIN::request& arg, cryptonote_connection_context& context)
{
protocol: log received messages in a separate category Makes it easier to log just what's going on on P2P
2017-01-22 10:14:45 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_REQUEST_CHAIN (" << arg.block_ids.size() << " blocks");
cryptonote_protocol: reject requests/notifications before handshake Reported by xnbya
2020-05-17 13:48:40 +00:00
if (context.m_state == cryptonote_connection_context::state_before_handshake)
{
LOG_ERROR_CCONTEXT("Requested chain before handshake, dropping connection");
drop_connection(context, false, false);
return 1;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
NOTIFY_RESPONSE_CHAIN_ENTRY::request r;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
if(!m_core.find_blockchain_supplement(arg.block_ids, !arg.prune, r))
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
LOG_ERROR_CCONTEXT("Failed to handle NOTIFY_REQUEST_CHAIN.");
return 1;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_RESPONSE_CHAIN_ENTRY: m_start_height=" << r.start_height << ", m_total_height=" << r.total_height << ", m_block_ids.size()=" << r.m_block_ids.size());
moved all stuff to github
2014-03-03 22:07:58 +00:00
post_notify<NOTIFY_RESPONSE_CHAIN_ENTRY>(r, context);
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool t_cryptonote_protocol_handler<t_core>::should_download_next_span(cryptonote_connection_context& context, bool standby)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<crypto::hash> hashes;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
boost::uuids::uuid span_connection_id;
boost::posix_time::ptime request_time;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
boost::uuids::uuid connection_id;
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
std::pair<uint64_t, uint64_t> span;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool filled;
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint64_t blockchain_height = m_core.get_current_blockchain_height();
if (context.m_remote_blockchain_height <= blockchain_height)
return false;
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
const bool has_next_block = tools::has_unpruned_block(blockchain_height, context.m_remote_blockchain_height, context.m_pruning_seed);
if (has_next_block)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (!m_block_queue.has_next_span(blockchain_height, filled, request_time, connection_id))
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << " we should download it as no peer reserved it");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return true;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (!filled)
{
const long dt = (now - request_time).total_microseconds();
if (dt >= REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD)
{
MDEBUG(context << " we should download it as it's not been received yet after " << dt/1e6);
return true;
}
// in standby, be ready to double download early since we're idling anyway
// let the fastest peer trigger first
long threshold;
const double dl_speed = context.m_max_speed_down;
if (standby && dt >= REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY && dl_speed > 0)
{
bool download = false;
if (m_p2p->for_connection(connection_id, [&](cryptonote_connection_context& ctx, nodetool::peerid_type peer_id, uint32_t f)->bool{
const time_t nowt = time(NULL);
const time_t time_since_last_recv = nowt - ctx.m_last_recv;
const float last_activity = std::min((float)time_since_last_recv, dt/1e6f);
const bool stalled = last_activity > LAST_ACTIVITY_STALL_THRESHOLD;
if (stalled)
{
MDEBUG(context << " we should download it as the downloading peer is stalling for " << nowt - ctx.m_last_recv << " seconds");
download = true;
return true;
}
// estimate the standby peer can give us 80% of its max speed
// and let it download if that speed is > N times as fast as the current one
// N starts at 10 after REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY,
// decreases to 1.25 at REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD,
// so that at times goes without the download being done, a retry becomes easier
const float max_multiplier = 10.f;
const float min_multiplier = 1.25f;
float multiplier = max_multiplier;
Fix time comparison math Dividing `dt` here by 1e6 converts it to seconds, but that is clearly wrong since `REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY` is measured in microseconds. As a result, this if statement was effectively never used.
2019-12-06 01:02:18 +00:00
if (dt >= REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY)
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
Fix time comparison math Dividing `dt` here by 1e6 converts it to seconds, but that is clearly wrong since `REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY` is measured in microseconds. As a result, this if statement was effectively never used.
2019-12-06 01:02:18 +00:00
multiplier = max_multiplier - (dt-REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY) * (max_multiplier - min_multiplier) / (REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD - REQUEST_NEXT_SCHEDULED_SPAN_THRESHOLD_STANDBY);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
multiplier = std::min(max_multiplier, std::max(min_multiplier, multiplier));
}
if (dl_speed * .8f > ctx.m_current_speed_down * multiplier)
{
MDEBUG(context << " we should download it as we are substantially faster (" << dl_speed << " vs "
<< ctx.m_current_speed_down << ", multiplier " << multiplier << " after " << dt/1e6 << " seconds)");
download = true;
return true;
}
return true;
}))
{
if (download)
return true;
}
else
{
MWARNING(context << " we should download it as the downloading peer is unexpectedly not known to us");
return true;
}
}
}
}
return false;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
bool t_cryptonote_protocol_handler<t_core>::should_drop_connection(cryptonote_connection_context& context, uint32_t next_stripe)
{
if (context.m_anchor)
{
MDEBUG(context << "This is an anchor peer, not dropping");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return false;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (context.m_pruning_seed == 0)
{
MDEBUG(context << "This peer is not striped, not dropping");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return false;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint32_t peer_stripe = tools::get_pruning_stripe(context.m_pruning_seed);
if (next_stripe == peer_stripe)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << "This peer has needed stripe " << peer_stripe << ", not dropping");
return false;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
const uint32_t local_stripe = tools::get_pruning_stripe(m_core.get_blockchain_pruning_seed());
protocol: fix syncing from peers being too choosy about peers
2019-10-17 17:35:22 +00:00
if (m_sync_pruned_blocks && local_stripe && next_stripe != local_stripe)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
{
MDEBUG(context << "We can sync pruned blocks off this peer, not dropping");
return false;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (!context.m_needed_objects.empty())
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint64_t next_available_block_height = context.m_last_response_height - context.m_needed_objects.size() + 1;
if (tools::has_unpruned_block(next_available_block_height, context.m_remote_blockchain_height, context.m_pruning_seed))
{
MDEBUG(context << "This peer has unpruned next block at height " << next_available_block_height << ", not dropping");
return false;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (next_stripe > 0)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
unsigned int n_out_peers = 0, n_peers_on_next_stripe = 0;
m_p2p->for_each_connection([&](cryptonote_connection_context& ctx, nodetool::peerid_type peer_id, uint32_t support_flags)->bool{
if (!ctx.m_is_income)
++n_out_peers;
if (ctx.m_state >= cryptonote_connection_context::state_synchronizing && tools::get_pruning_stripe(ctx.m_pruning_seed) == next_stripe)
++n_peers_on_next_stripe;
return true;
});
const uint32_t distance = (peer_stripe + (1<<CRYPTONOTE_PRUNING_LOG_STRIPES) - next_stripe) % (1<<CRYPTONOTE_PRUNING_LOG_STRIPES);
if ((n_out_peers >= m_max_out_peers && n_peers_on_next_stripe == 0) || (distance > 1 && n_peers_on_next_stripe <= 2) || distance > 2)
{
MDEBUG(context << "we want seed " << next_stripe << ", and either " << n_out_peers << " is at max out peers ("
<< m_max_out_peers << ") or distance " << distance << " from " << next_stripe << " to " << peer_stripe <<
" is too large and we have only " << n_peers_on_next_stripe << " peers on next seed, dropping connection to make space");
return true;
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << "End of checks, not dropping");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return false;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
void t_cryptonote_protocol_handler<t_core>::skip_unneeded_hashes(cryptonote_connection_context& context, bool check_block_queue) const
{
// take out blocks we already have
size_t skip = 0;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
while (skip < context.m_needed_objects.size() && (m_core.have_block(context.m_needed_objects[skip].first) || (check_block_queue && m_block_queue.have(context.m_needed_objects[skip].first))))
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
// if we're popping the last hash, record it so we can ask again from that hash,
// this prevents never being able to progress on peers we get old hash lists from
if (skip + 1 == context.m_needed_objects.size())
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
context.m_last_known_hash = context.m_needed_objects[skip].first;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
++skip;
}
if (skip > 0)
{
MDEBUG(context << "skipping " << skip << "/" << context.m_needed_objects.size() << " blocks");
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
context.m_needed_objects = std::vector<std::pair<crypto::hash, uint64_t>>(context.m_needed_objects.begin() + skip, context.m_needed_objects.end());
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
}
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
bool t_cryptonote_protocol_handler<t_core>::should_ask_for_pruned_data(cryptonote_connection_context& context, uint64_t first_block_height, uint64_t nblocks, bool check_block_weights) const
{
if (!m_sync_pruned_blocks)
return false;
if (!m_core.is_within_compiled_block_hash_area(first_block_height + nblocks - 1))
return false;
const uint32_t local_stripe = tools::get_pruning_stripe(m_core.get_blockchain_pruning_seed());
if (local_stripe == 0)
return false;
cryptonote_protocol: do not request pruned borromean sig txes We don't have a function to calculate their weight from a pruned version (yet).
2020-05-04 15:08:35 +00:00
// don't request pre-bulletprooof pruned blocks, we can't reconstruct their weight (yet)
protocol: fix asking for pruned blocks for v10 Old format is still alloewd there, this is a two fork update
2020-12-10 18:10:45 +00:00
static const uint64_t bp_fork_height = m_core.get_earliest_ideal_height_for_version(HF_VERSION_SMALLER_BP + 1);
cryptonote_protocol: don't synced pruned blocks before v11 There are a few Borromean proofs txes in the v8 era, and these aren't supported by get_pruned_transaction_weight. Moreover, only only the most recent variant of bulletproofs is currently supported.
2020-08-08 17:51:43 +00:00
if (first_block_height < bp_fork_height)
cryptonote_protocol: do not request pruned borromean sig txes We don't have a function to calculate their weight from a pruned version (yet).
2020-05-04 15:08:35 +00:00
return false;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
// assumes the span size is less or equal to the stripe size
bool full_data_needed = tools::get_pruning_stripe(first_block_height, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES) == local_stripe
|| tools::get_pruning_stripe(first_block_height + nblocks - 1, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES) == local_stripe;
if (full_data_needed)
return false;
if (check_block_weights && !m_core.has_block_weights(first_block_height, nblocks))
return false;
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
bool t_cryptonote_protocol_handler<t_core>::request_missing_objects(cryptonote_connection_context& context, bool check_having_blocks, bool force_next_span)
{
cryptonote_protocol: retry stale spans early Connections can be dropped by the net_node layer, unbeknownst to cryptonote_protocol, which would then not flush any spans scheduled to that connection, which would cause it to be only downloaded again once it becomes the next span (possibly after a small delay if it had been requested less than 5 seconds ago).
2017-07-11 20:48:54 +00:00
// flush stale spans
std::set<boost::uuids::uuid> live_connections;
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool{
live_connections.insert(context.m_connection_id);
return true;
});
m_block_queue.flush_stale_spans(live_connections);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
// if we don't need to get next span, and the block queue is full enough, wait a bit
bool start_from_current_chain = false;
if (!force_next_span)
{
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
do
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
size_t nspans = m_block_queue.get_num_filled_spans();
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
size_t size = m_block_queue.get_data_size();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const uint64_t bc_height = m_core.get_current_blockchain_height();
const auto next_needed_pruning_stripe = get_next_needed_pruning_stripe();
const uint32_t add_stripe = tools::get_pruning_stripe(bc_height, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
const uint32_t peer_stripe = tools::get_pruning_stripe(context.m_pruning_seed);
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
const uint32_t local_stripe = tools::get_pruning_stripe(m_core.get_blockchain_pruning_seed());
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const size_t block_queue_size_threshold = m_block_download_max_size ? m_block_download_max_size : BLOCK_QUEUE_SIZE_THRESHOLD;
bool queue_proceed = nspans < BLOCK_QUEUE_NSPANS_THRESHOLD || size < block_queue_size_threshold;
// get rid of blocks we already requested, or already have
skip_unneeded_hashes(context, true);
uint64_t next_needed_height = m_block_queue.get_next_needed_height(bc_height);
uint64_t next_block_height;
if (context.m_needed_objects.empty())
next_block_height = next_needed_height;
else
next_block_height = context.m_last_response_height - context.m_needed_objects.size() + 1;
protocol: fix syncing from peers being too choosy about peers
2019-10-17 17:35:22 +00:00
bool stripe_proceed_main = ((m_sync_pruned_blocks && local_stripe && add_stripe != local_stripe) || add_stripe == 0 || peer_stripe == 0 || add_stripe == peer_stripe) && (next_block_height < bc_height + BLOCK_QUEUE_FORCE_DOWNLOAD_NEAR_BLOCKS || next_needed_height < bc_height + BLOCK_QUEUE_FORCE_DOWNLOAD_NEAR_BLOCKS);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool stripe_proceed_secondary = tools::has_unpruned_block(next_block_height, context.m_remote_blockchain_height, context.m_pruning_seed);
bool proceed = stripe_proceed_main || (queue_proceed && stripe_proceed_secondary);
if (!stripe_proceed_main && !stripe_proceed_secondary && should_drop_connection(context, tools::get_pruning_stripe(next_block_height, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES)))
{
if (!context.m_is_income)
m_p2p->add_used_stripe_peer(context);
return false; // drop outgoing connections
}
MDEBUG(context << "proceed " << proceed << " (queue " << queue_proceed << ", stripe " << stripe_proceed_main << "/" <<
stripe_proceed_secondary << "), " << next_needed_pruning_stripe.first << "-" << next_needed_pruning_stripe.second <<
" needed, bc add stripe " << add_stripe << ", we have " << peer_stripe << "), bc_height " << bc_height);
MDEBUG(context << " - next_block_height " << next_block_height << ", seed " << epee::string_tools::to_string_hex(context.m_pruning_seed) <<
", next_needed_height "<< next_needed_height);
MDEBUG(context << " - last_response_height " << context.m_last_response_height << ", m_needed_objects size " << context.m_needed_objects.size());
// if we're waiting for next span, try to get it before unblocking threads below,
// or a runaway downloading of future spans might happen
if (stripe_proceed_main && should_download_next_span(context, true))
{
MDEBUG(context << " we should try for that next span too, we think we could get it faster, resuming");
force_next_span = true;
MLOG_PEER_STATE("resuming");
break;
}
if (proceed)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
if (context.m_state != cryptonote_connection_context::state_standby)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
LOG_DEBUG_CC(context, "Block queue is " << nspans << " and " << size << ", resuming");
MLOG_PEER_STATE("resuming");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
break;
}
cryptonote_protocol: guard against all threads in standby
2018-02-02 22:33:53 +00:00
// this one triggers if all threads are in standby, which should not happen,
// but happened at least once, so we unblock at least one thread if so
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
boost::unique_lock<boost::mutex> sync{m_sync_lock, boost::try_to_lock};
cryptonote_protocol: guard against all threads in standby
2018-02-02 22:33:53 +00:00
if (sync.owns_lock())
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool filled = false;
boost::posix_time::ptime time;
boost::uuids::uuid connection_id;
if (m_block_queue.has_next_span(m_core.get_current_blockchain_height(), filled, time, connection_id) && filled)
{
LOG_DEBUG_CC(context, "No other thread is adding blocks, and next span needed is ready, resuming");
MLOG_PEER_STATE("resuming");
context.m_state = cryptonote_connection_context::state_standby;
++context.m_callback_request_count;
m_p2p->request_callback(context);
return true;
}
else
{
sync.unlock();
cryptonote_protocol: guard against all threads in standby
2018-02-02 22:33:53 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
// if this has gone on for too long, drop incoming connection to guard against some wedge state
if (!context.m_is_income)
{
const uint64_t now = tools::get_tick_count();
const uint64_t dt = now - m_last_add_end_time;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
if (m_last_add_end_time && tools::ticks_to_ns(dt) >= DROP_ON_SYNC_WEDGE_THRESHOLD)
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
MDEBUG(context << "ns " << tools::ticks_to_ns(dt) << " from " << m_last_add_end_time << " and " << now);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << "Block addition seems to have wedged, dropping connection");
return false;
}
}
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
if (context.m_state != cryptonote_connection_context::state_standby)
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (!queue_proceed)
LOG_DEBUG_CC(context, "Block queue is " << nspans << " and " << size << ", pausing");
else if (!stripe_proceed_main && !stripe_proceed_secondary)
LOG_DEBUG_CC(context, "We do not have the stripe required to download another block, pausing");
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
context.m_state = cryptonote_connection_context::state_standby;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("pausing");
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
}
protocol: change standby mode to not wait sleeping
2018-12-07 12:32:50 +00:00
return true;
} while(0);
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
context.m_state = cryptonote_connection_context::state_synchronizing;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << " request_missing_objects: check " << check_having_blocks << ", force_next_span " << force_next_span
<< ", m_needed_objects " << context.m_needed_objects.size() << " lrh " << context.m_last_response_height << ", chain "
<< m_core.get_current_blockchain_height() << ", pruning seed " << epee::string_tools::to_string_hex(context.m_pruning_seed));
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if(context.m_needed_objects.size() || force_next_span)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
//we know objects that we need, request this objects
NOTIFY_REQUEST_GET_OBJECTS::request req;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
bool is_next = false;
moved all stuff to github
2014-03-03 22:07:58 +00:00
size_t count = 0;
cryptonote_protocol: large block sync size before v4
2017-08-16 20:19:53 +00:00
const size_t count_limit = m_core.get_block_sync_size(m_core.get_current_blockchain_height());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
std::pair<uint64_t, uint64_t> span = std::make_pair(0, 0);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
if (force_next_span)
{
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (span.second == 0)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<crypto::hash> hashes;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
boost::uuids::uuid span_connection_id;
boost::posix_time::ptime time;
span = m_block_queue.get_next_span_if_scheduled(hashes, span_connection_id, time);
if (span.second > 0)
{
is_next = true;
for (const auto &hash: hashes)
{
req.blocks.push_back(hash);
context.m_requested_objects.insert(hash);
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_block_queue.reset_next_span_time();
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (span.second == 0)
{
MDEBUG(context << " span size is 0");
if (context.m_last_response_height + 1 < context.m_needed_objects.size())
{
MERROR(context << " ERROR: inconsistent context: lrh " << context.m_last_response_height << ", nos " << context.m_needed_objects.size());
context.m_needed_objects.clear();
context.m_last_response_height = 0;
goto skip;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
skip_unneeded_hashes(context, false);
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
const uint64_t first_block_height = context.m_last_response_height - context.m_needed_objects.size() + 1;
cryptonote_protocol: do not request pruned borromean sig txes We don't have a function to calculate their weight from a pruned version (yet).
2020-05-04 15:08:35 +00:00
static const uint64_t bp_fork_height = m_core.get_earliest_ideal_height_for_version(8);
bool sync_pruned_blocks = m_sync_pruned_blocks && first_block_height >= bp_fork_height && m_core.get_blockchain_pruning_seed();
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
span = m_block_queue.reserve_span(first_block_height, context.m_last_response_height, count_limit, context.m_connection_id, context.m_remote_address, sync_pruned_blocks, m_core.get_blockchain_pruning_seed(), context.m_pruning_seed, context.m_remote_blockchain_height, context.m_needed_objects);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
MDEBUG(context << " span from " << first_block_height << ": " << span.first << "/" << span.second);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (span.second > 0)
{
const uint32_t stripe = tools::get_pruning_stripe(span.first, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
if (context.m_pruning_seed && stripe != tools::get_pruning_stripe(context.m_pruning_seed))
{
MDEBUG(context << " starting early on next seed (" << span.first << " with stripe " << stripe <<
", context seed " << epee::string_tools::to_string_hex(context.m_pruning_seed) << ")");
}
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
if (span.second == 0 && !force_next_span)
{
MDEBUG(context << " still no span reserved, we may be in the corner case of next span scheduled and everything else scheduled/filled");
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<crypto::hash> hashes;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
boost::uuids::uuid span_connection_id;
boost::posix_time::ptime time;
span = m_block_queue.get_next_span_if_scheduled(hashes, span_connection_id, time);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (span.second > 0 && !tools::has_unpruned_block(span.first, context.m_remote_blockchain_height, context.m_pruning_seed))
span = std::make_pair(0, 0);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (span.second > 0)
{
is_next = true;
for (const auto &hash: hashes)
{
req.blocks.push_back(hash);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
++count;
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
context.m_requested_objects.insert(hash);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
// that's atrocious O(n) wise, but this is rare
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
auto i = std::find_if(context.m_needed_objects.begin(), context.m_needed_objects.end(),
[&hash](const std::pair<crypto::hash, uint64_t> &o) { return o.first == hash; });
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
if (i != context.m_needed_objects.end())
context.m_needed_objects.erase(i);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
}
}
MDEBUG(context << " span: " << span.first << "/" << span.second << " (" << span.first << " - " << (span.first + span.second - 1) << ")");
if (span.second > 0)
{
if (!is_next)
{
const uint64_t first_context_block_height = context.m_last_response_height - context.m_needed_objects.size() + 1;
uint64_t skip = span.first - first_context_block_height;
if (skip > context.m_needed_objects.size())
{
MERROR("ERROR: skip " << skip << ", m_needed_objects " << context.m_needed_objects.size() << ", first_context_block_height" << first_context_block_height);
return false;
}
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
if (skip > 0)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
context.m_needed_objects = std::vector<std::pair<crypto::hash, uint64_t>>(context.m_needed_objects.begin() + skip, context.m_needed_objects.end());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (context.m_needed_objects.size() < span.second)
{
MERROR("ERROR: span " << span.first << "/" << span.second << ", m_needed_objects " << context.m_needed_objects.size());
return false;
}
for (size_t n = 0; n < span.second; ++n)
{
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
req.blocks.push_back(context.m_needed_objects[n].first);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
++count;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
context.m_requested_objects.insert(context.m_needed_objects[n].first);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
context.m_needed_objects = std::vector<std::pair<crypto::hash, uint64_t>>(context.m_needed_objects.begin() + span.second, context.m_needed_objects.end());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
req.prune = should_ask_for_pruned_data(context, span.first, span.second, true);
// if we need to ask for full data and that peer does not have the right stripe, we can't ask it
if (!req.prune && context.m_pruning_seed)
{
const uint32_t peer_stripe = tools::get_pruning_stripe(context.m_pruning_seed);
const uint32_t first_stripe = tools::get_pruning_stripe(span.first, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
const uint32_t last_stripe = tools::get_pruning_stripe(span.first + span.second - 1, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
protocol: fix syncing from peers being too choosy about peers
2019-10-17 17:35:22 +00:00
if ((((first_stripe && peer_stripe != first_stripe) || (last_stripe && peer_stripe != last_stripe)) && !m_sync_pruned_blocks) || (m_sync_pruned_blocks && req.prune))
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
{
MDEBUG(context << "We need full data, but the peer does not have it, dropping peer");
return false;
}
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = NOTIFY_RESPONSE_GET_OBJECTS::ID;
Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT
2019-05-31 00:59:19 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_GET_OBJECTS: blocks.size()=" << req.blocks.size()
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
<< "requested blocks count=" << count << " / " << count_limit << " from " << span.first << ", first hash " << req.blocks.front());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
//epee::net_utils::network_throttle_manager::get_global_throttle_inreq().logger_handle_net("log/dr-monero/net/req-all.data", sec, get_avg_block_size());
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
MDEBUG("Asking for " << (req.prune ? "pruned" : "full") << " data, start/end "
<< tools::get_pruning_stripe(span.first, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES)
<< "/" << tools::get_pruning_stripe(span.first + span.second - 1, context.m_remote_blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES)
<< ", ours " << tools::get_pruning_stripe(m_core.get_blockchain_pruning_seed()) << ", peer stripe " << tools::get_pruning_stripe(context.m_pruning_seed));
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
post_notify<NOTIFY_REQUEST_GET_OBJECTS>(req, context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting objects");
return true;
}
// if we're still around, we might be at a point where the peer is pruned, so we could either
// drop it to make space for other peers, or ask for a span further down the line
const uint32_t next_stripe = get_next_needed_pruning_stripe().first;
const uint32_t peer_stripe = tools::get_pruning_stripe(context.m_pruning_seed);
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
const uint32_t local_stripe = tools::get_pruning_stripe(m_core.get_blockchain_pruning_seed());
if (!(m_sync_pruned_blocks && peer_stripe == local_stripe) && next_stripe && peer_stripe && next_stripe != peer_stripe)
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
// at this point, we have to either close the connection, or start getting blocks past the
// current point, or become dormant
MDEBUG(context << "this peer is pruned at seed " << epee::string_tools::to_string_hex(context.m_pruning_seed) <<
", next stripe needed is " << next_stripe);
if (!context.m_is_income)
{
if (should_drop_connection(context, next_stripe))
{
m_p2p->add_used_stripe_peer(context);
return false; // drop outgoing connections
}
}
// we'll get back stuck waiting for the go ahead
context.m_state = cryptonote_connection_context::state_normal;
MLOG_PEER_STATE("Nothing to do for now, switching to normal state");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return true;
}
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
skip:
context.m_needed_objects.clear();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
// we might have been called from the "received chain entry" handler, and end up
// here because we can't use any of those blocks (maybe because all of them are
// actually already requested). In this case, if we can add blocks instead, do so
if (m_core.get_current_blockchain_height() < m_core.get_target_blockchain_height())
{
const boost::unique_lock<boost::mutex> sync{m_sync_lock, boost::try_to_lock};
if (sync.owns_lock())
{
uint64_t start_height;
std::vector<cryptonote::block_complete_entry> blocks;
boost::uuids::uuid span_connection_id;
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
epee::net_utils::network_address span_origin;
if (m_block_queue.get_next_span(start_height, blocks, span_connection_id, span_origin, true))
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
LOG_DEBUG_CC(context, "No other thread is adding blocks, resuming");
MLOG_PEER_STATE("will try to add blocks next");
context.m_state = cryptonote_connection_context::state_standby;
++context.m_callback_request_count;
m_p2p->request_callback(context);
return true;
}
}
}
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if(context.m_last_response_height < context.m_remote_blockchain_height-1)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{//we have to fetch more objects ids, request blockchain entry
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
Changed the use of boost:value_initialized for C++ list initializer
2019-06-26 14:39:41 +00:00
NOTIFY_REQUEST_CHAIN::request r = {};
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_core.get_short_chain_history(r.block_ids);
add empty container sanity checks when using front() and back()
2017-12-11 22:36:58 +00:00
CHECK_AND_ASSERT_MES(!r.block_ids.empty(), false, "Short chain history is empty");
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!start_from_current_chain)
{
protocol: fix reorgs while syncing
2017-08-12 09:59:27 +00:00
// we'll want to start off from where we are on that peer, which may not be added yet
move checkpoints in a separate library
2017-09-10 16:35:59 +00:00
if (context.m_last_known_hash != crypto::null_hash && r.block_ids.front() != context.m_last_known_hash)
protocol: fix reorgs while syncing
2017-08-12 09:59:27 +00:00
r.block_ids.push_front(context.m_last_known_hash);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
}
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
handler_request_blocks_history( r.block_ids ); // change the limit(?), sleep(?)
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
r.prune = m_sync_pruned_blocks;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//std::string blob; // for calculate size of request
//epee::serialization::store_t_to_binary(r, blob);
//epee::net_utils::network_throttle_manager::get_global_throttle_inreq().logger_handle_net("log/dr-monero/net/req-all.data", sec, get_avg_block_size());
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
//LOG_PRINT_CCONTEXT_L1("r = " << 200);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
cryptonote_protocol: kick idle synchronizing peers In case they dropped off downloading for any reason, they'll get sent to download again.
2017-08-16 18:27:16 +00:00
context.m_last_request_time = boost::posix_time::microsec_clock::universal_time();
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
context.m_expect_response = NOTIFY_RESPONSE_CHAIN_ENTRY::ID;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_P2P_MESSAGE("-->>NOTIFY_REQUEST_CHAIN: m_block_ids.size()=" << r.block_ids.size() << ", start_from_current_chain " << start_from_current_chain);
moved all stuff to github
2014-03-03 22:07:58 +00:00
post_notify<NOTIFY_REQUEST_CHAIN>(r, context);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("requesting chain");
moved all stuff to github
2014-03-03 22:07:58 +00:00
}else
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
{
CHECK_AND_ASSERT_MES(context.m_last_response_height == context.m_remote_blockchain_height-1
&& !context.m_needed_objects.size()
&& !context.m_requested_objects.size(), false, "request_missing_blocks final condition failed!"
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< "\r\nm_last_response_height=" << context.m_last_response_height
<< "\r\nm_remote_blockchain_height=" << context.m_remote_blockchain_height
<< "\r\nm_needed_objects.size()=" << context.m_needed_objects.size()
<< "\r\nm_requested_objects.size()=" << context.m_requested_objects.size()
0.8.8update
2014-05-25 17:06:40 +00:00
<< "\r\non connection [" << epee::net_utils::print_connection_context_short(context)<< "]");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
moved all stuff to github
2014-03-03 22:07:58 +00:00
context.m_state = cryptonote_connection_context::state_normal;
cryptonote_protocol: avoid spurious SYNCHRONIZED OK messages
2017-07-07 19:52:24 +00:00
if (context.m_remote_blockchain_height >= m_core.get_target_blockchain_height())
{
if (m_core.get_current_blockchain_height() >= m_core.get_target_blockchain_height())
{
MGINFO_GREEN("SYNCHRONIZED OK");
on_connection_synchronized();
}
}
else
{
protocol: drop peers that decrease claimed height
2020-12-01 22:18:53 +00:00
MINFO(context << " we've reached this peer's blockchain height (theirs " << context.m_remote_blockchain_height << ", our target " << m_core.get_target_blockchain_height());
cryptonote_protocol: avoid spurious SYNCHRONIZED OK messages
2017-07-07 19:52:24 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
return true;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::on_connection_synchronized()
{
bool val_expected = false;
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
uint64_t current_blockchain_height = m_core.get_current_blockchain_height();
if(!m_core.is_within_compiled_block_hash_area(current_blockchain_height) && m_synchronized.compare_exchange_strong(val_expected, true))
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Daemon: Print estimates for time until fully synced
2019-12-27 13:31:13 +00:00
if ((current_blockchain_height > m_sync_start_height) && (m_sync_spans_downloaded > 0))
{
uint64_t synced_blocks = current_blockchain_height - m_sync_start_height;
// Report only after syncing an "interesting" number of blocks:
if (synced_blocks > 20)
{
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
uint64_t synced_seconds = (now - m_sync_start_time).total_seconds();
if (synced_seconds == 0)
{
synced_seconds = 1;
}
float blocks_per_second = (1000 * synced_blocks / synced_seconds) / 1000.0f;
MGINFO_YELLOW("Synced " << synced_blocks << " blocks in "
<< tools::get_human_readable_timespan(synced_seconds) << " (" << blocks_per_second << " blocks per second)");
}
}
Update log messages to reflect fact we no longer need to type `exit` to save database before quitting
2017-01-23 01:03:04 +00:00
MGINFO_YELLOW(ENDL << "**********************************************************************" << ENDL
fix remaining bitmonero and simplewallet bits
2016-09-03 20:03:44 +00:00
<< "You are now synchronized with the network. You may now start monero-wallet-cli." << ENDL
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ENDL
Wallet, daemon: From 'help_advanced' back to 'help', and new 'apropos' command
2020-06-01 08:20:59 +00:00
<< "Use the \"help\" command to see the list of available commands." << ENDL
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< "**********************************************************************");
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_sync_timer.pause();
if (ELPP->vRegistry()->allowed(el::Level::Info, "sync-info"))
{
const uint64_t sync_time = m_sync_timer.value();
const uint64_t add_time = m_add_timer.value();
if (sync_time && add_time)
{
MCLOG_YELLOW(el::Level::Info, "sync-info", "Sync time: " << sync_time/1e9/60 << " min, idle time " <<
(100.f * (1.0f - add_time / (float)sync_time)) << "%" << ", " <<
(10 * m_sync_download_objects_size / 1024 / 1024) / 10.f << " + " <<
(10 * m_sync_download_chain_size / 1024 / 1024) / 10.f << " MB downloaded, " <<
100.0f * m_sync_old_spans_downloaded / m_sync_spans_downloaded << "% old spans, " <<
100.0f * m_sync_bad_spans_downloaded / m_sync_spans_downloaded << "% bad spans");
}
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_core.on_synchronized();
}
Cleanup test impact of adding safesyncmode() method
2017-08-22 13:54:18 +00:00
m_core.safesyncmode(true);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_p2p->clear_used_stripe_peers();
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
// ask for txpool complement from any suitable node if we did not yet
val_expected = true;
if (m_ask_for_txpool_complement.compare_exchange_strong(val_expected, false))
{
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool
{
if(context.m_state < cryptonote_connection_context::state_synchronizing)
{
MDEBUG(context << "not ready, ignoring");
return true;
}
if (!request_txpool_complement(context))
{
MERROR(context << "Failed to request txpool complement");
return true;
}
return false;
});
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
size_t t_cryptonote_protocol_handler<t_core>::get_synchronizing_connections_count()
{
size_t count = 0;
cryptonote_core does not depend on p2p anymore As a followon side effect, this makes a lot of inline code included only in particular cpp files (and instanciated when necessary.
2017-11-29 22:53:58 +00:00
m_p2p->for_each_connection([&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)->bool{
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(context.m_state == cryptonote_connection_context::state_synchronizing)
++count;
return true;
});
return count;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
int t_cryptonote_protocol_handler<t_core>::handle_response_chain_entry(int command, NOTIFY_RESPONSE_CHAIN_ENTRY::request& arg, cryptonote_connection_context& context)
{
protocol: log received messages in a separate category Makes it easier to log just what's going on on P2P
2017-01-22 10:14:45 +00:00
MLOG_P2P_MESSAGE("Received NOTIFY_RESPONSE_CHAIN_ENTRY: m_block_ids.size()=" << arg.m_block_ids.size()
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ", m_start_height=" << arg.start_height << ", m_total_height=" << arg.total_height);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("received chain");
protocol: drop peers that don't reply to queries
2020-12-03 13:22:52 +00:00
if (context.m_expect_response != NOTIFY_RESPONSE_CHAIN_ENTRY::ID)
{
LOG_ERROR_CCONTEXT("Got NOTIFY_RESPONSE_CHAIN_ENTRY out of the blue, dropping connection");
drop_connection(context, true, false);
return 1;
}
context.m_expect_response = 0;
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_last_request_time = boost::date_time::not_a_date_time;
m_sync_download_chain_size += arg.m_block_ids.size() * sizeof(crypto::hash);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(!arg.m_block_ids.size())
{
LOG_ERROR_CCONTEXT("sent empty m_block_ids, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, true, false);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
protocol: more restrictive checks on chain entry response
2020-12-10 14:00:40 +00:00
if (arg.total_height < arg.m_block_ids.size() || arg.start_height > arg.total_height - arg.m_block_ids.size() || arg.start_height >= m_core.get_current_blockchain_height())
cryptonote_protocol: sanity check chain hashes from peer
2017-12-07 22:22:14 +00:00
{
LOG_ERROR_CCONTEXT("sent invalid start/nblocks/height, dropping connection");
drop_connection(context, true, false);
return 1;
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
if (!arg.m_block_weights.empty() && arg.m_block_weights.size() != arg.m_block_ids.size())
{
LOG_ERROR_CCONTEXT("sent invalid block weight array, dropping connection");
drop_connection(context, true, false);
return 1;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MDEBUG(context << "first block hash " << arg.m_block_ids.front() << ", last " << arg.m_block_ids.back());
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
protocol: add a sanity check to the number of block hashes sent
2020-12-10 19:26:12 +00:00
if (arg.total_height >= CRYPTONOTE_MAX_BLOCK_NUMBER || arg.m_block_ids.size() > BLOCKS_IDS_SYNCHRONIZING_MAX_COUNT)
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
{
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_RESPONSE_CHAIN_ENTRY, with total_height=" << arg.total_height << " and block_ids=" << arg.m_block_ids.size());
drop_connection(context, false, false);
return 1;
}
protocol: drop peers that decrease claimed height
2020-12-01 22:18:53 +00:00
if (arg.total_height < context.m_remote_blockchain_height)
{
MINFO(context << "Claims " << arg.total_height << ", claimed " << context.m_remote_blockchain_height << " before");
hit_score(context, 1);
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
context.m_remote_blockchain_height = arg.total_height;
context.m_last_response_height = arg.start_height + arg.m_block_ids.size()-1;
if(context.m_last_response_height > context.m_remote_blockchain_height)
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
LOG_ERROR_CCONTEXT("sent wrong NOTIFY_RESPONSE_CHAIN_ENTRY, with m_total_height=" << arg.total_height
<< ", m_start_height=" << arg.start_height
<< ", m_block_ids.size()=" << arg.m_block_ids.size());
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return 1;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
uint64_t n_use_blocks = m_core.prevalidate_block_hashes(arg.start_height, arg.m_block_ids, arg.m_block_weights);
protocol: one more sanity check on usable data from a peer
2020-12-10 22:59:15 +00:00
if (n_use_blocks == 0 || n_use_blocks + HASH_OF_HASHES_STEP <= arg.m_block_ids.size())
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
{
protocol: drop (and increase fails) if most blocks hashes are invalid
2018-06-02 20:32:42 +00:00
LOG_ERROR_CCONTEXT("Most blocks are invalid, dropping connection");
drop_connection(context, true, false);
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
return 1;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
context.m_needed_objects.clear();
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
uint64_t added = 0;
protocol: more restrictive checks on chain entry response
2020-12-10 14:00:40 +00:00
std::unordered_set<crypto::hash> blocks_found;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
for (size_t i = 0; i < arg.m_block_ids.size(); ++i)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
protocol: more restrictive checks on chain entry response
2020-12-10 14:00:40 +00:00
if (!blocks_found.insert(arg.m_block_ids[i]).second)
{
LOG_ERROR_CCONTEXT("Duplicate blocks in chain entry response, dropping connection");
drop_connection(context, true, false);
return 1;
}
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
const uint64_t block_weight = arg.m_block_weights.empty() ? 0 : arg.m_block_weights[i];
context.m_needed_objects.push_back(std::make_pair(arg.m_block_ids[i], block_weight));
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
if (++added == n_use_blocks)
break;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
context.m_last_response_height -= arg.m_block_ids.size() - n_use_blocks;
moved all stuff to github
2014-03-03 22:07:58 +00:00
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
if (!request_missing_objects(context, false))
{
LOG_ERROR_CCONTEXT("Failed to request missing objects, dropping connection");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
drop_connection(context, false, false);
cryptonote_protocol_handler: sync speedup A block queue is now placed between block download and block processing. Blocks are now requested only from one peer (unless starved). Includes a new sync_info coommand.
2017-07-02 21:41:15 +00:00
return 1;
}
cryptonote_protocol: update target height when syncing too
2017-08-22 16:17:19 +00:00
if (arg.total_height > m_core.get_target_blockchain_height())
m_core.set_target_blockchain_height(arg.total_height);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return 1;
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool t_cryptonote_protocol_handler<t_core>::relay_block(NOTIFY_NEW_BLOCK::request& arg, cryptonote_connection_context& exclude_context)
{
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
NOTIFY_NEW_FLUFFY_BLOCK::request fluffy_arg = AUTO_VAL_INIT(fluffy_arg);
fluffy_arg.current_blockchain_height = arg.current_blockchain_height;
monerod can now sync from pruned blocks If the peer (whether pruned or not itself) supports sending pruned blocks to syncing nodes, the pruned version will be sent along with the hash of the pruned data and the block weight. The original tx hashes can be reconstructed from the pruned txes and theur prunable data hash. Those hashes and the block weights are hashes and checked against the set of precompiled hashes, ensuring the data we received is the original data. It is currently not possible to use this system when not using the set of precompiled hashes, since block weights can not otherwise be checked for validity. This is off by default for now, and is enabled by --sync-pruned-blocks
2019-09-16 12:18:34 +00:00
std::vector<tx_blob_entry> fluffy_txs;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
fluffy_arg.b = arg.b;
fluffy_arg.b.txs = fluffy_txs;
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
// sort peers between fluffy ones and others
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
std::vector<std::pair<epee::net_utils::zone, boost::uuids::uuid>> fullConnections, fluffyConnections;
Fix compiler warnings with Clang 6.0.0. monero/src/cryptonote_protocol/block_queue.cpp:208:44: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] static const boost::uuids::uuid uuid0 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ { } monero/src/wallet/wallet_rpc_server.cpp:1895:43: error: lambda capture 'wal' is not used [-Werror,-Wunused-lambda-capture] tools::signal_handler::install([&wrpc, &wal](int) { ^ monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:40: error: lambda capture 'arg' is not used [-Werror,-Wunused-lambda-capture] m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections... ^ monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:46: error: lambda capture 'fluffy_arg' is not used [-Werror,-Wunused-lambda-capture] m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections... ^ monero/src/blockchain_utilities/blockchain_export.cpp:181:3: error: bool literal returned from 'main' [-Werror,-Wmain] CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage"); ^ ~~~~~ monero/contrib/epee/include/misc_log_ex.h:180:97: note: expanded from macro 'CHECK_AND_ASSERT_MES' ...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0) ^ ~~~~~~~~~~~~ monero/src/blockchain_utilities/blockchain_export.cpp:195:3: error: bool literal returned from 'main' [-Werror,-Wmain] CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data"); ^ ~~~~~ monero/contrib/epee/include/misc_log_ex.h:180:97: note: expanded from macro 'CHECK_AND_ASSERT_MES' ...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0) ^ ~~~~~~~~~~~~
2017-10-07 16:21:04 +00:00
m_p2p->for_each_connection([this, &exclude_context, &fullConnections, &fluffyConnections](connection_context& context, nodetool::peerid_type peer_id, uint32_t support_flags)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
if (peer_id && exclude_context.m_connection_id != context.m_connection_id && context.m_remote_address.get_zone() == epee::net_utils::zone::public_)
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
Add a --fluffy-blocks option to relay blocks as fluffy blocks Defaults to off, but fluffy blocks are forced enabled on testnet
2017-09-02 16:11:20 +00:00
if(m_core.fluffy_blocks_enabled() && (support_flags & P2P_SUPPORT_FLAG_FLUFFY_BLOCKS))
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
LOG_DEBUG_CC(context, "PEER SUPPORTS FLUFFY BLOCKS - RELAYING THIN/COMPACT WHATEVER BLOCK");
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
fluffyConnections.push_back({context.m_remote_address.get_zone(), context.m_connection_id});
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
}
else
{
cryptonote_protocol: misc fluffy block fixes - fix wrong block being used when a new block is received between a node elaying a fluffy block and sending a new fluffy block with txes a peer did not have - misc a neverending ping pong requesting the same missing txids when a new block is received in the meantime, causing the top block to not be the one we need - send the original fluffy block message block height when sending a new fluffy block, not the current top height, which might have been updated since - avoid sending back the whole block blob when asking for txes, send only the hash instead - plus misc cleanup and additional debugging logs
2017-02-11 19:38:18 +00:00
LOG_DEBUG_CC(context, "PEER DOESN'T SUPPORT FLUFFY BLOCKS - RELAYING FULL BLOCK");
Adding initial support for broadcasting transactions over Tor - Support for ".onion" in --add-exclusive-node and --add-peer - Add --anonymizing-proxy for outbound Tor connections - Add --anonymous-inbounds for inbound Tor connections - Support for sharing ".onion" addresses over Tor connections - Support for broadcasting transactions received over RPC exclusively over Tor (else broadcast over public IP when Tor not enabled).
2018-12-16 17:57:44 +00:00
fullConnections.push_back({context.m_remote_address.get_zone(), context.m_connection_id});
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
}
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
}
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
return true;
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
});
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
// send fluffy ones first, we want to encourage people to run that
cryptonote_protocol: don't serialize an unneeded new block message Non fluffy block nodes should now be very rare
2018-08-01 13:56:37 +00:00
if (!fluffyConnections.empty())
{
Change epee binary output from std::stringstream to byte_stream
2020-10-13 15:15:07 +00:00
epee::byte_slice fluffyBlob;
epee::serialization::store_t_to_binary(fluffy_arg, fluffyBlob, 32 * 1024);
m_p2p->relay_notify_to_list(NOTIFY_NEW_FLUFFY_BLOCK::ID, epee::to_span(fluffyBlob), std::move(fluffyConnections));
cryptonote_protocol: don't serialize an unneeded new block message Non fluffy block nodes should now be very rare
2018-08-01 13:56:37 +00:00
}
if (!fullConnections.empty())
{
Change epee binary output from std::stringstream to byte_stream
2020-10-13 15:15:07 +00:00
epee::byte_slice fullBlob;
epee::serialization::store_t_to_binary(arg, fullBlob, 128 * 1024);
m_p2p->relay_notify_to_list(NOTIFY_NEW_BLOCK::ID, epee::to_span(fullBlob), std::move(fullConnections));
cryptonote_protocol: don't serialize an unneeded new block message Non fluffy block nodes should now be very rare
2018-08-01 13:56:37 +00:00
}
p2p: possibly fix crash in relay_blocks
2016-11-29 16:21:33 +00:00
cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool)
2018-04-28 02:07:51 +00:00
return true;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//------------------------------------------------------------------------------------------------------------------------
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
template<class t_core>
Adding Dandelion++ support to public networks: - New flag in NOTIFY_NEW_TRANSACTION to indicate stem mode - Stem loops detected in tx_pool.cpp - Embargo timeout for a blackhole attack during stem phase
2019-11-13 14:12:32 +00:00
bool t_cryptonote_protocol_handler<t_core>::relay_transactions(NOTIFY_NEW_TRANSACTIONS::request& arg, const boost::uuids::uuid& source, epee::net_utils::zone zone, relay_method tx_relay)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Adding support for hidden (anonymity) txpool
2019-11-02 20:36:03 +00:00
/* Push all outgoing transactions to this function. The behavior needs to
identify how the transaction is going to be relayed, and then update the
local mempool before doing the relay. The code was already updating the
DB twice on received transactions - it is difficult to workaround this
due to the internal design. */
Dandelion++: skip desynced peers in stem phase
2020-10-12 12:32:17 +00:00
return m_p2p->send_txs(std::move(arg.txs), zone, source, tx_relay) != epee::net_utils::zone::invalid;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
bool t_cryptonote_protocol_handler<t_core>::request_txpool_complement(cryptonote_connection_context &context)
{
NOTIFY_GET_TXPOOL_COMPLEMENT::request r = {};
if (!m_core.get_pool_transaction_hashes(r.hashes, false))
{
MERROR("Failed to get txpool hashes");
return false;
}
MLOG_P2P_MESSAGE("-->>NOTIFY_GET_TXPOOL_COMPLEMENT: hashes.size()=" << r.hashes.size() );
post_notify<NOTIFY_GET_TXPOOL_COMPLEMENT>(r, context);
MLOG_PEER_STATE("requesting txpool complement");
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
protocol: add scoring system to drop peers that don't behave
2020-12-01 22:25:07 +00:00
void t_cryptonote_protocol_handler<t_core>::hit_score(cryptonote_connection_context &context, int32_t score)
{
if (score <= 0)
{
MERROR("Negative score hit");
return;
}
context.m_score -= score;
if (context.m_score <= DROP_PEERS_ON_SCORE)
drop_connection_with_score(context, 5, false);
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
std::string t_cryptonote_protocol_handler<t_core>::get_peers_overview() const
{
std::stringstream ss;
const boost::posix_time::ptime now = boost::posix_time::microsec_clock::universal_time();
m_p2p->for_each_connection([&](const connection_context &ctx, nodetool::peerid_type peer_id, uint32_t support_flags) {
const uint32_t stripe = tools::get_pruning_stripe(ctx.m_pruning_seed);
char state_char = cryptonote::get_protocol_state_char(ctx.m_state);
ss << stripe + state_char;
if (ctx.m_last_request_time != boost::date_time::not_a_date_time)
ss << (((now - ctx.m_last_request_time).total_microseconds() > IDLE_PEER_KICK_TIME) ? "!" : "?");
ss << + " ";
return true;
});
return ss.str();
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
std::pair<uint32_t, uint32_t> t_cryptonote_protocol_handler<t_core>::get_next_needed_pruning_stripe() const
{
const uint64_t want_height_from_blockchain = m_core.get_current_blockchain_height();
const uint64_t want_height_from_block_queue = m_block_queue.get_next_needed_height(want_height_from_blockchain);
const uint64_t want_height = std::max(want_height_from_blockchain, want_height_from_block_queue);
uint64_t blockchain_height = m_core.get_target_blockchain_height();
// if we don't know the remote chain size yet, assume infinitely large so we get the right stripe if we're not near the tip
if (blockchain_height == 0)
blockchain_height = CRYPTONOTE_MAX_BLOCK_NUMBER;
const uint32_t next_pruning_stripe = tools::get_pruning_stripe(want_height, blockchain_height, CRYPTONOTE_PRUNING_LOG_STRIPES);
if (next_pruning_stripe == 0)
return std::make_pair(0, 0);
// if we already have a few peers on this stripe, but none on next one, try next one
unsigned int n_next = 0, n_subsequent = 0, n_others = 0;
const uint32_t subsequent_pruning_stripe = 1 + next_pruning_stripe % (1<<CRYPTONOTE_PRUNING_LOG_STRIPES);
m_p2p->for_each_connection([&](const connection_context &context, nodetool::peerid_type peer_id, uint32_t support_flags) {
if (context.m_state >= cryptonote_connection_context::state_synchronizing)
{
if (context.m_pruning_seed == 0 || tools::get_pruning_stripe(context.m_pruning_seed) == next_pruning_stripe)
++n_next;
else if (tools::get_pruning_stripe(context.m_pruning_seed) == subsequent_pruning_stripe)
++n_subsequent;
else
++n_others;
}
return true;
});
const bool use_next = (n_next > m_max_out_peers / 2 && n_subsequent <= 1) || (n_next > 2 && n_subsequent == 0);
const uint32_t ret_stripe = use_next ? subsequent_pruning_stripe: next_pruning_stripe;
MIDEBUG(const std::string po = get_peers_overview(), "get_next_needed_pruning_stripe: want height " << want_height << " (" <<
want_height_from_blockchain << " from blockchain, " << want_height_from_block_queue << " from block queue), stripe " <<
next_pruning_stripe << " (" << n_next << "/" << m_max_out_peers << " on it and " << n_subsequent << " on " <<
subsequent_pruning_stripe << ", " << n_others << " others) -> " << ret_stripe << " (+" <<
(ret_stripe - next_pruning_stripe + (1 << CRYPTONOTE_PRUNING_LOG_STRIPES)) % (1 << CRYPTONOTE_PRUNING_LOG_STRIPES) <<
"), current peers " << po);
return std::make_pair(next_pruning_stripe, ret_stripe);
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
bool t_cryptonote_protocol_handler<t_core>::needs_new_sync_connections() const
{
const uint64_t target = m_core.get_target_blockchain_height();
const uint64_t height = m_core.get_current_blockchain_height();
if (target && target <= height)
return false;
size_t n_out_peers = 0;
m_p2p->for_each_connection([&](cryptonote_connection_context& ctx, nodetool::peerid_type peer_id, uint32_t support_flags)->bool{
if (!ctx.m_is_income)
++n_out_peers;
return true;
});
if (n_out_peers >= m_max_out_peers)
return false;
return true;
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
void t_cryptonote_protocol_handler<t_core>::drop_connection_with_score(cryptonote_connection_context &context, unsigned score, bool flush_all_spans)
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
LOG_DEBUG_CC(context, "dropping connection id " << context.m_connection_id << " (pruning seed " <<
epee::string_tools::to_string_hex(context.m_pruning_seed) <<
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
"), score " << score << ", flush_all_spans " << flush_all_spans);
cryptonote_protocol_handler: log when dropping a peer
2018-05-01 15:21:43 +00:00
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
if (score > 0)
m_p2p->add_host_fail(context.m_remote_address, score);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
m_block_queue.flush_spans(context.m_connection_id, flush_all_spans);
protocol: fix use after free when dropping a connection
2018-11-26 17:19:25 +00:00
m_p2p->drop_connection(context);
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
ban peers sending bad pow outright PoW is expensive to verify, so be strict
2019-09-24 13:08:25 +00:00
void t_cryptonote_protocol_handler<t_core>::drop_connection(cryptonote_connection_context &context, bool add_fail, bool flush_all_spans)
{
return drop_connection_with_score(context, add_fail ? 1 : 0, flush_all_spans);
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
protocol: drop origin IP if a block fails to verify in sync mode It would otherwise be possible for a peer to send bad blocks, then disconnect and reconnect again, escaping bans
2020-12-08 17:55:24 +00:00
void t_cryptonote_protocol_handler<t_core>::drop_connections(const epee::net_utils::network_address address)
{
MWARNING("dropping connections to " << address.str());
m_p2p->add_host_fail(address, 5);
std::vector<boost::uuids::uuid> drop;
m_p2p->for_each_connection([&](const connection_context& cntxt, nodetool::peerid_type peer_id, uint32_t support_flags) {
if (address.is_same_host(cntxt.m_remote_address))
drop.push_back(cntxt.m_connection_id);
return true;
});
for (const boost::uuids::uuid &id: drop)
{
m_block_queue.flush_spans(id, true);
m_p2p->for_connection(id, [&](cryptonote_connection_context& context, nodetool::peerid_type peer_id, uint32_t f)->bool{
drop_connection(context, true, false);
return true;
});
}
}
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
void t_cryptonote_protocol_handler<t_core>::on_connection_close(cryptonote_connection_context &context)
{
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
uint64_t target = 0;
m_p2p->for_each_connection([&](const connection_context& cntxt, nodetool::peerid_type peer_id, uint32_t support_flags) {
if (cntxt.m_state >= cryptonote_connection_context::state_synchronizing && cntxt.m_connection_id != context.m_connection_id)
target = std::max(target, cntxt.m_remote_blockchain_height);
return true;
});
const uint64_t previous_target = m_core.get_target_blockchain_height();
if (target < previous_target)
{
MINFO("Target height decreasing from " << previous_target << " to " << target);
m_core.set_target_blockchain_height(target);
cryptonote_protocol: warn when the last connection goes
2019-04-14 09:03:33 +00:00
if (target == 0 && context.m_state > cryptonote_connection_context::state_before_handshake && !m_stopping)
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
{
cryptonote_protocol: warn when the last connection goes
2019-04-14 09:03:33 +00:00
MCWARNING("global", "monerod is now disconnected from the network");
protocol: request txpool contents when synced A newly synced Alice sends a (typically quite small) list of txids in the local tpxool to a random peer Bob, who then uses the existing tx relay system to send Alice any tx in his txpool which is not in the list Alice sent
2019-11-27 00:09:25 +00:00
m_ask_for_txpool_complement = true;
}
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
}
cryptonote_protocol: misc fixes to the new sync algorithm Fix sync wedge corner case: It could happen if a connection went into standby mode, while it was the one which had requested the next span, and that span was still waiting for the data, and that peer is not on the main chain. Other peers can then start asking for that data again and again, but never get it as only that forked peer does. And various other fixes
2017-08-18 19:14:23 +00:00
m_block_queue.flush_spans(context.m_connection_id, false);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
MLOG_PEER_STATE("closed");
cryptonote_protocol: keep target in sync with dropped connections When a node is dropped, we stop considering its claimed blockchain height as a factor in the target height calculation. This prevents a runaway chain from being still thought to be the target even if the nodes carrying it are dropped.
2017-08-09 19:49:24 +00:00
}
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
Fix delayed exit when syncing
2016-12-04 12:27:45 +00:00
//------------------------------------------------------------------------------------------------------------------------
template<class t_core>
void t_cryptonote_protocol_handler<t_core>::stop()
{
m_stopping = true;
m_core.stop();
}
2014 network limit 1.0a +utils +toc -doc -drmonero commands and options for network limiting works very well e.g. for 50 KiB/sec up and down ToS (QoS) flag peer number limit TODO some spikes in ingress/download TODO problems when other up and down limit added "otshell utils" - simple logging (with colors, text files channels)
2015-01-05 19:30:17 +00:00
} // namespace
[monerod] Added blocks remaining count during syncronisation. And percent if usefull (% < 99)
2018-07-25 08:16:01 +00:00
Reference in a new issue Copy permalink