added MRL6, version 2

publications/MRL-0006 - Difficulty Adjustment Algorithms in Cryptocurrency Protocols/main.tex

@@ -515,11 +515,11 @@
 \author[
    addressref={mrl},
    email={lab@monero.cc}
-]{\fnm{Sarang} \snm{Noether}}
+]{\fnm{Surae} \snm{Noether}}
 \author[
    addressref={mrl}
    email={lab@monero.cc}
-]{\fnm{Surae} \snm{Noether}}
+]{\fnm{Sarang} \snm{Noether}}
 
 
 \address[id=mrl]{
@@ -529,345 +529,331 @@
 
 \begin{abstractbox}
 \begin{abstract}
-As of this writing, the algorithm employed for difficulty adjustment in the CryptoNote reference code is known by the Monero Research Lab to be flawed. By dishonestly reporting timestamps, attackers can gain disproportionate control over network difficulty.  We verify this route of attack by testing the CryptoNote reference code against historical blockchain timestamp data; we use data that has been both deterministically manipulated and randomly manipulated to represent an attack. To do this, we reimplemented the difficulty adjustment algorithm in the Python programming language. We describe and illustrate the nature of the flaw and recommend a solution.
+As of this writing, the algorithm employed for difficulty adjustment in the CryptoNote reference code is known by the Monero Research Lab to be flawed. We describe and illustrate the nature of the flaw and recommend a solution. By dishonestly reporting timestamps, attackers can gain disproportionate control over network difficulty.  We verify this route of attack by auditing the CryptoNote reference difficulty adjustment code, which, we reimplement in the Python programming language.  We use a stochastic model of blockchain growth to test the CryptoNote reference difficulty formula against the more traditional Bitcoin difficulty formula. This allows us to test our difficulty formula against various hash rate scenarios.
 
 This research bulletin has not undergone peer review, and reflects only the results of internal investigation.
 \end{abstract}
 \end{abstractbox}
 \end{frontmatter}
 
-In this research bulletin, we audit difficulty assessment and adjustment in the CryptoNote reference code (from which the Monero cryptocurrency has been forked). This audit reveals that the difficulty assessment and adjustment algorithm is flawed in several ways. We discuss the desirable properties of difficulty and establish a more suitable difficulty adjustment algorithm. 
+\section{Introduction}\label{intro}
 
-To begin, though, we have to ask obvious questions. What is the hashing difficulty of a proof-of-work cryptocurrency? Each block that arrives on the network is assigned a difficulty score, so one interpretation of difficulty is the following: difficulty scores form a weighting scheme to a blockchain. However, many choices of weighting schemes can be described, none of which accomplish the intuitive tasks one desires from a difficulty scheme. If difficulty scores are independent of time and depend solely on block height, then network hash rate will have no practical consequence for difficulty. For example, by setting the difficulty of each block to be precisely half the previous difficulty with some initial difficulty on the genesis block, we see immediately that hash rate has no effect on difficulty. Furthermore, there is a monotonically increasing incentive to mine. For another example, by setting difficulty of each block to the current time, difficulty will linearly increase in time and we see, again, hash rate has no effect on difficulty. Hence, the answer ``difficulty is a weighting scheme'' seems dissatisfying. So what properties should difficulty exhibit, in addition to being a weighting scheme?
+In this research bulletin, we audit the difficulty assessment and adjustment components of the CryptoNote reference code. The difficulty score of the next block to be added to a blockchain depends on the sequence of timestamps of the blocks preceding it, together with those difficulty scores. However, we have no way of validating a timestamp, and so timestamps are vulnerable to manipulation. Thus, difficulty is vulnerable to manipulation.  
 
-The most important property of blockchain growth is that the number of block arrivals on the network should be held roughly constant in time. Hence, we intuitively conclude that if network hash rate has increased, difficulty should increase, and \textit{vice versa}. Difficulty should remain unchanged if and only if network hash rate has remained unchanged. It is not immediately obvious, however, what sort of relationship hash rate and difficulty ought to have. Should difficulty be directly proportional to hash rate, or should there be some more complicated relationship between the two? Should we set difficulty to the square of hash rate, or should difficulty be a linear function of hash rate? If difficulty is a linear function of hash rate, will any linear transformation suffice? If only some linear transformations work, what properties must we require from such a linear transformation?
+We first derive a model of blockchain growth incorporating nonhomogeneous Poisson processes and various difficulty assessment methods. We next use this model to identify the desirable properties of difficulty assessment methods. We implement our model in the Python programming language. Using this implementation, we compare the difficulty assessment method from the current CryptoNote reference code, the difficulty assessment method from the Bitcoin code, and an alternative difficulty assessment method which we propose as an alternative for Monero. This comparison is based on the desirable properties of difficulty assessment methods, and the net hashrate chosen by the experimenter.
 
-In Section \ref{model}, we construct a statistical model of blockchain construction using independent Bernoulli trials (each corresponding to hashing a particular nonce). Using this model, we conclude that difficulty assessments should be directly proportional to network hash rate. We also observe the equivalence between this process and a Poisson process under suitable assumptions, which appear to be reasonable in the cryptocurrency setting.
+The model we present herein is simplified in at least two critical ways compared to the true behavior of the network. First, we assume that there are no propagation delays of block discoveries, and second, we assume the usual Nakamoto parent coin selection rule. That is to say, although we will make some comments about parent coin selection rule in Section \ref{modelDef}, we will not investigate variants of parent coin selection rules such as Sompolinsky's GHOST rule in this document.
 
-In Section \ref{currentApproach}, we discuss the current approach to difficulty assessment in the CryptoNote reference code. In that section, we elaborate upon the current codebase, we attempt to identify the ostensible goals of the reference difficulty algorithm, and we discuss some successes and failures of that code with respect to both those goals as well as the desirable properties of difficulty.
+The modeler is stuck with a dilemma. On one hand, as we have done herein, way may assume no propagation delays occur in the cryptocurrency network, which leads to unrealistic behavior. With such an assumption, everyone has perfectly accurate data and thus every user will select the same parent coin, regardless of parent coin selection rule. Furthermore, in Poisson processes, arrivals may not occur simultaneously, and so we never have more than one chain in the blocktree. On the other hand, we may incorporate assumptions about propagation delays in a cryptocurrency network, but these assumptions are equivalent to assumptions about the cryptocurrency network structure and speed. In the former case, we are sacrificing realistic competitive behavior so that we are not forced to make unrealistic assumptions about network structure. In the latter case, we are gaining some realistic competitive behaviors but we are assuming much about network structure. 
 
-In Section \ref{proposedChanges}, we make some proposals for changes to the current difficulty assessment method. Our recommendations vary from simple to sophisticated, with advantages and drawbacks to each.
+However, if a researcher were to go down such a route, she would also need to study how rules regarding parent coin selection can impact blockchain dynamics in the presence of competing chains.  One could mitigate the problem of making assumptions about network structure by making empirical measurements such as average and standard deviation of block transmission times,  inferring data about the network structure. Studying these things in the context of forking blockchains and selfish mining would certainly be a project worthy of effort. However, this is beyond the scope of this document, which is primarily concerned with the behavior of difficulty scores in response to varying network hash rates. Hence, we go with the former assumption, with no propagation delays and with the Nakamoto parent coin selection rule.
 
-All discussions of drawbacks, flaws, and attack vulnerabilities herein are not meant to be comprehensive. Notice that, although this paper discusses the CryptoNote reference code, we discuss this from the perspective of the Monero codebase, which has been forked from the CryptoNote reference code.
+We define for the reader some notation we shall use in the sequel.  When block arrival rate is viewed as a nonhomogeneous Poisson process, we denote the instantaneous rate of block arrivals on the network as $\lambda(t)$. We wish to keep instantaneous block arrival rate close to our target block arrival rate, $\lambda^*$, which is a known constant. We denote instantaneous network hash rate as $H(t)$, which is unknown and not directly observable. We may occasionally refer to a cryptographic hash function $\mathcal{H}$, and a nonce, $x$. We consider the blockchain to be a sequence of blocks, $\mathcal{B}_0$, $\mathcal{B}_1$, $\ldots$. Each block, $\mathcal{B}_i$, consists of a difficutly score and a (possibly false) timestamp:
 
-\section{Modeling Blockchain Construction with Jenga, Bernoulli, and Poisson}\label{model}
+\[\mathcal{B}_0 = (t_0, d_0), \mathcal{B}_1 = (t_1, d_1), \mathcal{B}_2 = (t_2, d_2), \ldots\]
 
-Assume we have an unknown number of computers, $N(t)$, on a network with an unknown weighted connected graph structure, describing the computer network and transmission speeds. For simplicity, presume that each computer has a hash rate of $1 H/s$, so that $N(t)$ is both network hash rate and user participation. While this may seem unrealistic at first, keep in mind that a powerful mining computer can be viewed as a cluster of $1 H/s$ computers on the network with zero transmission delay between them. Mining pools can similarly be viewed as superclusters of such clusters with nearly zero transmission delays between them. We allow participants to join or leave the game at any real-valued time. The results of the coin flipping game are transmitted across the network so all participants have up-to-date data. Non-uniformity in the network leads each node to have its own data. Say the data accessible by the $i^{th}$ node up till time $t$ is $D_i(t)$. All nodes eventually receive the same data, although not necessarily at the same rate or in the same order. The coin flipping game proceeds as follows:
+We begin counting the genesis block as height $0$, so the block of height $n-1$ is the $n^{th}$ block to arrive; the number of inter-arrival times corresponds with block height this way. We denote a cryptographic hash function as $\mathcal{H}$, the block of height $n$ as $\mathcal{B}_{n}$, the difficulty of the block of height $n$ as $d_{n}$. We may occasionally refer to a nonce, which we denote $x$.
 
-The game is to build a stack of heads-up coins in the center of the room following certain rules. Before each flip, the $i^{th}$ participant uses the data made available to her thus far to assess the current state of the game. She uses this data to (a) choose a \textit{parent coin} from the current stack and (b) make a determination about the weight of the coin she is about to flip. Denote the probability of obtaining a heads as $p_i(t; D_i(t))$ and the probability of obtaining a tails as $1-p_i(t; D_i(t))$. If the flip is heads-up, she will stack her heads-up coin on the pre-determined parent coin and receive a monetary reward. In order to claim their reward, players announce the discovery time $\widehat{t}$ and the parent coin to the network. Note that $\widehat{t} \neq t$ in general. When the $j^{th}$ participant hears such an announcement, she adjusts her own information, $D_j(t)$, so as to keep an accurate model of the stack of coins.
+For some $m \geq 2$, we will generate a sequence of sample block arrival rates, $\hat{\lambda}_{i}$, for $i=m+1, m+2, m+3, \ldots$, each with sample size $m$. If $m=2$, we are only considering the inter-arrival time between the latest two blocks. From these, we compute network hash rate estimates, $\hat{H}_i$. For some $\ell \leq m$, we also compute $\overline{H}_i$, for each $i = \ell, \ell+1, \ell+2, \ldots$, the moving average of the network hash rate estimates with sample size $\ell$, $\overline{H}_i = \frac{1}{\ell}\sum_{j=1}^{\ell} \hat{H}_{n-j}$. If $\ell = 1$, then the moving average is simplified to our latest estimate of instantaneous hash rate. Notice that $m/\lambda^*$ is the expected length of time to observe $m$ arrivals from a homogeneous Poisson process with rate $\lambda^*$; so, for example, if our block arrival target is $\lambda^{*} = (60.0 s)^{-1}$ and we wish to use the last five minutes of data to compute our sample block arrival rate, we set $m=5$. On the other hand, if we wish to use the last two hours of data, we set $m=120$.
 
-When deciding how node $i$ should adjust $p_i$ based on $D_i$, we allow node $i$ to make the assumption that all other nodes on the network have the same data as itself. That is, node $i$ assumes that if $j \neq i$, then $D_j = D_i$. If so, the node will also conclude that other nodes have the same weight, i.e.\ $p_i = p_j$. Indeed, all nodes will \textit{eventually} have the same data, and probably only within a few seconds. While this assumption is not outrageously false, it is still false. Even if all actors are honest, computers within a pool will receive information about that pool before computers outside of that pool, due to transmission speeds if not selfish mining. Despite this unreasonable assumption, an honest node $i$ has no \textit{a priori} information available about the data made available to other players, and so this ostensibly false assumption is necessary.
 
-We largely ignore parent-coin selection rules in this document, although many interesting questions arise from these variations. For completeness, we shall discuss parent-coin selection for a moment. Many decision rules for choosing parent coins could be constructed, but one desirable restriction for a fair game is that, whatever the decision rule is employed for choosing parent coins, this rule is uniformly enforced across nodes. This way, two nodes at the same time with the same data will choose the same parent coin.  In \cite{nakamoto2008bitcoin}, Satoshi Nakamoto proposed that clients always stack new heads on top of the longest branch they can see. In \cite{sompolinsky2013accelerating}, Sompolinsky and Zohar recommend a greedy algorithm seeking the heaviest subtree approach: participants determine their parent coin by climbing the blocktree from the genesis block upward, and each time they are faced with a branch, they take the branch leading to the heaviest subtree. When they have finished climbing the blocktree, they have found their parent coin. 
 
-Both approaches can be viewed under a different light: assign a score to each block and choosing the block with the highest \textit{cumulative} score of all preceding blocks. A generalized Nakamoto rule can be constructed by stacking new heads on top of the longest branch in the coin tree in terms of some score function. This observation matches nicely with our description of difficulty as a weighting scheme applied to the blockchain, as well. Indeed, for any given difficulty scheme, a simple parent selection scheme could be ``choose as the parent coin the choice that has maximal \textit{cumulative difficulty}.'' From this perspective, we see that the generalized Nakamoto rule is sensitively dependent upon choice of difficulty algorithm.
+\section{Blockchain growth model}\label{modelDef}
 
-This is all we shall say on parent-coin selection rules in this document. On the other hand, we primarily focus on coin-weight selection, or rather, difficulty algorithm. One desirable restriction for a fair game is that the weight of the coin is determined across nodes uniformly, when possible. That is to say, two different computers on the network with the same data at the same time will flip coins with the same weights. That is to say, if $D_i(t_1) = D_j(t_2)$ and $t_1 = t_2$, then $p_i(t_1; D_i(t_1)) = p_j(t_2 \; D_j(t_2))$ for any $i, j$.
+In this section, we define a stochastic process that models the growth of the blockchain over time, and we justify why this model is a good representation of blockchain growth. We also make a few comments about parent coin selection rules. In Section \ref{hashrateattack}, we make a general criticism of traditional blockchain methods and describe a possibly novel route of attack based solely on blockchain dynamics.
 
-Although the game the network is playing may seem peculiar so far, the game becomes more peculiar when the fundamental unreliability of timestamps is taken into account. The discovering node announces the time $\widehat{t}$, not necessarily the time $t$. Indeed, the announced time of discovery of each flip, $\widehat{t}$, is subject to dishonest actors, calibration error on the part of the local machine, and error due to transmission speeds on the network. Indeed, machines with up to $2$ hours disparity in their local clocks can still be considered ostensibly honest actors due to the barbaric practice of Daylight Savings Time. Worse yet, there is no way, to our knowledge, of validating that timestamps are reported honestly. In fact, the data available to any given node, $D_i(t)$ is precisely a tree of (possibly false) timestamps representing the stack of coins.
+First, we describe our model, which may be represented formally in the following way; let us worry about justification in a moment. We use as input an unknown, \emph{a priori}, positive hash rate function $H(t) > 0$ with support containing the interval $[0,T)$ for some $T > 0$, where $T$ is a constant denoting the time we stop modeling the network. We could apply more assumptions about $H(t)$ if we feel this is not sufficient. For example, we could also presume that $H(t)$ is bounded below away from zero, say $H(t) > 1$, but this won't change our model very much. We could also presume $H(t)$ is piecewise constant (modeling user hash rates as either on the network at full speed or not), or we could assume $H(t)$ is generated by some other stochastic process, which may or may not be dependent upon the current state of the blockchain. For a general derivation, we simply assume that $H(t)$ is positive with support containing $[0,T)$.
 
-In the remainder of this document, unless there is danger of confusion, we will denote $N(t)$, $D_i(t)$, and $p_i(t; D_i(t))$ as $N$, $D_i$, and $p_i$, respectively, with the understanding that these quantities vary in time.
+We set initial difficulty $d_0 = 1$. A nonhomogeneous Poisson process governing block arrivals with counting process $N(t)$ is observed. Recall that $N(t)$ corresponds to the total number of block arrivals on the time interval $[0,t)$. Denote the instantaneous rate of block arrivals at time $t < T$ as $\lambda(t) = H(t)/d_{N(t)}$. The nonhomogeneous Poisson process gives rise to block arrival times $t_0, t_1, t_2, \ldots$, which may then be manipulated by a malicious user, giving rise to \emph{manipulated} block arrival times $\hat{t}_0, \hat{t}_1, \ldots$. The value of the denominator in the block arrival rate, $d_{N(t)}$, is block difficulty. This value is generated from a function that is determined by the manipulated block arrival times and the difficulties of the top blocks.  That is to say, we have some function $\phi$ and we set
+\[d_{N(t)} = \phi((\hat{t}_i, d_i)_{i=0}^{N(t)-1})\]
 
-\section{Difficulty is Hash Power}\label{diffIsHash}
+We define a blockchain in this context, then, as the stochastic process consisting of the sequence of ordered pairs $(\hat{t}_i, d_i)_{i=0}^{N(t)}$, where the difficulty $d_n = \phi((\hat{t}_i, d_i)_{i=0}^{n-1})$. Observe two facts: first, the \emph{latest} block (the block with $t_i = \max_j (t_j)$) is not necessarily the top block (which has height $n-1$) because the manipulated timestamps need not occur in the same order as their indices. Second, we have defined this notion so generally that it encompasses a variety of difficulty adjustment methods. For example, Bitcoin has a difficulty adjustment period of $2016$ blocks. That is to say, for Bitcoin, the function $\phi$ is dependent upon $n$, so that if $n \cong 0 (mod p)$ for some integer $p$, then for $k=0, 1, 2, \ldots, 2015$, $d_n = d_{n+k}$.
 
-In this subsection, we derive the constraint required on the coin weight, $p_i$, that ensures that, regardless of network hash rate, $N$, and regardless of network structure, the total number of heads-up coins being discovered across the whole network per unit time is approximately constant; namely, by setting difficulty proportional to network hash power, we will attain constant heads-up coins over time, on average. 
+To justify this model, recall how the proof-of-work competition for block validation works. Users collect transactions into blocks for validation and try to hash nonces together with block data in order to find a hash smaller than a certain target. That is to say, if a user on the network finds some $x$ such that $d_{n}\cdot \mathcal{H}(\mathcal{B} + x) < \texttt{fixed\_target}$, they have earned the right to declare a block as valid and they usually receive a block reward in the form of a coinbase transaction. Everyone who is working off of the same copy of the blockchain as each other will compute difficulty, $d_{n}$ in the same way. When a user finds such an $x$, they publish $\mathcal{B}$, $x$, and a possibly untrustworthy timestamp. They then recompute their difficulty. When a user hears about a new block, they add it to their copy of the blockchain and recompute their difficulty before trying more hashes.
 
-In most cryptocurrency protocols, as well as the CryptoNote protocol, a hash digest, $h(m)$, multiplied by network difficulty, $\mathcal{D}$, must be less than some constant, $c$, in order for a user on the network to have earned the right to declare a block valid. That is, if $h(m)\cdot\mathcal{D} \leq c$, then the user has earned the right to post a block to the blockchain and has won a block reward. A good hash algorithm, $h$, has an output space that is practically, if not mathematically, indistinguishable from the uniform distribution, and so this has probability of success that is roughly proportional to $c/\mathcal{D}$. This allows us to model the race for block validation by equating the act of testing a nonce against a difficulty target with flipping a coin with probability of heads $p=c/\mathcal{D}$. 
+Since the goal of the proof-of-work game is to find $x$ such that $d_{n} \cdot \mathcal{H}(\mathcal{B} + x) < \texttt{fixed\_target}$ and since the output of a good hash function is, in practice, indistinguishable from a uniform distribution, this implies the probability that any given nonce is a success is precisely$\texttt{fixed\_target}/d_{n}$. That is to say, each trial testing a nonce is a Bernoulli trial (weighted coin flip) with probability of success inversely proportional to the difficulty of the next block to be added. Without loss of generality, we may choose  $\texttt{fixed\_target}=1$; doing so calibrates the difficulty score such that $d_n=1$ corresponds with a success on each and every nonce. Hence, each trial testing a nonce is a Bernoulli trial (weighted coin flip) with probabiltiy of success $1/d_n$.
 
-Recall that the most important property of blockchain growth is to keep the rate of arrival of heads-up coins roughly constant, regardless of the number of coin-flippers and their speed of coin flips at any point in time. However, neither the number of users on the network nor their personal coin-flipping rates are directly observable. However, if a large number of such trials, $N$, is observed with a small probability of success, $p$, we can cite the approximation
-\[\binom{N}{k} p^k(1-p)^{N-k} \approx \frac{\lambda^k e^{-\lambda}}{k!}\]
-where $\lambda = Np$. The left hand side is the probability of finding $k$ successes in a sequence of $N$ independent Bernoulli trials, each with probability of success $p$. The right-hand side is the probability of finding $k$ arrivals in a time interval while observing a Poisson process with rate parameter $\lambda = Np$. That is to say, while hash rate is held constant at $N$ and difficulty is held constant at $\mathcal{D}$, blocks will arrive on the network in a process that is closely approximated by a Poisson process with rate $\lambda = cN/\mathcal{D}$. We wish to keep this rate of arrivals constant in time, regardless of how $N$ changes in time, so the natural choice is to pick $\mathcal{D} = cN/\lambda_{\text{target}}$, where $\lambda_{\text{target}}$ is our target block arrival rate. That is to say, the only difficulty choice that will maintain a constant (average) block arrival rate will be to choose difficulty to be proportional to network hash rate.
+Arrivals of heads-up coins in a sequence of coin flips, under suitable conditions, can be well approximated with a Poisson process. If we are given a constant $\lambda > 0$, a sequence of probabilities $\left\{p_n\right\}$ satisfying $n p_n \longrightarrow \lambda$ and a few other suitable conditions, then a binomial random variable $\text{Bin}(n,p_n)$ can be roughly approximated with the Poisson distribution with rate $\lambda$. The suitable conditions are technical but a good rule of thumb is that when $n \geq 20$ and $p \leq 1/20$, or if $n \geq 100$ and $np \leq 10$, we may use the Poisson approximation. In our case, we are talking about cryptocurrency networks with $n$ typically much greater than thousands of hashes per second, and with probability of success far lower than $1$ in $20$ nonces. Hence, we may approximate the proof-of-work block competition for validation by a Poisson process with rate $\lambda = np$. More technical arguments can be made toward this equivalence, but we shall be satisfied and proceed with a Poisson process model.
 
-We may re-attain this result without citing the Poisson approximation, if we are willing to deal in expectation and rely upon the Law of Large Numbers to ensure that variance away from the expectation is small. Indeed, each user flips coins at some fixed rate. Some users are faster than others, and some factions of users form collective pools with their coin-flipping machines in order to smooth out the unreliability in their rate of heads-up coin arrivals. However, regardless of user behavior, we can enumerate the list of the coin-flipping rates, $\left\{r_1, r_2, \ldots, r_n\right\}$, where $r_i$ measures the numbers of coin flips per unit time\footnote{Remember, coin flips are directly analagous to hashes, so a gigahash-per-second device corresponds to $r_i = 10^9$ flips per second}. Hence, network hash rate is precisely $N=\sum_i r_i$. Each user has access to data  describing the history of the game, $D_i$, which consists of both her own data as well as partial data provided by other users. She then draws her own conclusions about the bias of the coin she is flipping, $p_i$. Recall that if $p_i$ is the probability of a heads-up coin, and we make $r_i$ flips, the resulting number of heads follows a $\text{Binomial}(r_i,p_i)$ distribution, which has expectation $r_i p_i$. Thus, we can \textit{expect} the total number of arrivals across the network per second to be
-$$\text{Arrivals per second} = \sum_j (r_j\text{ coin flips per second})\cdot (p_j\text{ prob. of heads})$$
-and furthermore, if the number of users is large and the period of time over which we sample this game is large, this expectation will be fairly accurate and variance away from this expectation will be small
+Note that, in the Poisson process described above, $n$ is proportional to our global network hashrate, $H(t)$, and $p=1/d_n$. The block arrival rate will be $\lambda(t) = np = \beta H(t)/d_n$. Consider $\beta$. If difficulty is $1$, every nonce is a success, and if $H(t) = 1 H/s$, then a block will arrive with probability $1$ by time $t=1.0 s$, yielding a block arrival rate of $1.0 s^{-1}$. Hence, we have $1.0 s^{-1} = \beta \frac{1.0 H/s}{1}$. We conclude $\beta = 1$ and we are free to use the relationship
+\[\lambda = \frac{H}{d}\]
+Furthermore, since users assign difficulty $d_n$ according to some function of the preceding difficulties and timestamps, this is a nearly complete justification of our model in question. As mentioned in Section \ref{intro}, we largely ignore parent-coin selection rules in this document, although many interesting questions may arise (see Section \ref{furtherQuestions}). For completeness, however, we shall discuss parent-coin selection for a moment. 
 
-Citing our assumption that $D_i = D_j$ for each $i,j$, node $i$ may model the total number of arrivals across the network per second to be
-\begin{align*}\text{Arrivals per second} &= \sum_j r_j p_j\\
-&\approx \sum_j r_j p_i\text{ (by assumption)}\\
-&= p_i \sum_j r_j \\
-&= w_i N
-\end{align*}
-Of course, the left hand side of this equation must be held constant in order to maintain a constant rate of block arrivals. Recalling that the probability of a successful flip is $c/\mathcal{D}$ for some constant $c$, and for a target of $\lambda_{\text{target}}$ seconds per block, then we have
-\begin{align*}
-\lambda_{\text{target}} =& p_i N\\
-=& cN/\mathcal{D}\\
-\mathcal{D} =& cN/\lambda_{\text{target}}
-\end{align*}
-And we re-attain our result that difficulty ought to be proportional to network hash rate, $N$.
+Many decision rules for choosing parent coins could be constructed.  One common misconception of the Bitcoin whitepaper, \cite{nakamoto2008bitcoin}, is that Satoshi Nakamoto did not propose mining on the longest chain, but, in fact, the chain with the largest cumulative difficulty. In the case of Bitcoin, for which thousands of blocks in a row have the same difficulty score, this is often (although not always) equivalent to selecting a chain with the highest block height.  Alternative proposals have been made, e.g\ in \cite{sompolinsky2013accelerating} Sompolinsky and Zohar recommend a greedy algorithm seeking the heaviest subtree approach. Users determine their parent coin by climbing the blocktree from the genesis block upward, and each time they are faced with a branch, they take the branch leading to the heaviest subtree. When they have finished climbing the blocktree, they have found their parent coin. 
 
-\section{Orphaned Blocks}
-We pause to make an observation about orphaned coins, which actually has nothing to do with difficulty. Regardless of parent-coin selection, we will still occasionally, due to transmission speed errors, see computers producing orphan blocks. If all users mine honestly, this is the primary source of orphan blocks, but there is nothing to prevent users from colluding in a selfish mining attack, as described in \cite{eyal2014majority}. Due to some conflict in the community over the term ``orphaned block,'' a more descriptive term perhaps could be \textit{dead branches of the  blocktree}.
+These approaches are, in fact, identical if we view as a generalized Nakamoto rule: assign a generalized ``score'' to each block and choose the parent block by selecting the block with the largest \textit{cumulative} score of all preceding blocks. The only difference between these two methods is how the scores are computed. The original Nakamoto recommendation was to use difficulty of block $\mathcal{B}$ as the score function, whereas the recommendation by Sompolinsky and Zohar is to use the cumulative heaviness (in terms of difficulty) of the subtree that has a block $\mathcal{B}$ as its root.  From this perspective, we see that the generalized Nakamoto rule may be sensitive to both choices of score function and difficulty equation.
 
-Consider the following example modeled after Bitcoin with a block target of $\lambda_{\text{target}}=0.1$ blocks per minute; for the sake of argument, presume the network is a complete graph with a constant $N$ nodes and with the same transmission speed between any two nodes. We could model transmission time in the network between node $i$ and node $j$ as $\mu + e_{ij}$. Here, each $e_{ij}$ is a random variable with $E(e_{ij}) = 0$, the mean propagation time is $\mu \approx 6.5$ seconds following the times reported in \cite{decker2013information}; note that the times reported in that publication are somewhat out of date, but they are a sufficient starting point. When a node finds a heads-up coin at time $t$, then by time $t+6.5$, only half the network (on average) has heard of my new heads-up coin. What about a computer in the second half of the network, the blind half? A node in that half of the network flipping coins between time $t$ and $t+6.5$ may have chosen a different parent coin because it does not have as much information as I (or I chose a different parent coin because I have less information than it). Furthermore, one or the other branch will become part of the main chain eventually; these two events are mutually exclusive. Hence the total number of coin flips wasted is at least
-$$\text{Orphaned flips} \geq E\left[\text{Number of heads found by half the network in }[t,t+\mu)\right]$$
+This is all we shall say on parent-coin selection rules in this document. With that, we have justified the model of choice  All that remains is the focus of this document, which is to decide how difficulty is to be computed under the model described above.
+
+\subsection{Consequences of the model: Stalling}\label{hashrateattack}
+
+In this section, we discuss how the model presented in Section \ref{modelDef} is sensitive to sudden changes in hash rate.
+
+Consider a sudden change in hash rate; any traditional blockchain scheme is vulnerable to stalling if a large portion of the network suddenly withdraws its participation. We are creating a discrete object, the blockchain, from a continuous-time stochastic process; if the underlying forces giving rise to the creation of the blockchain are in a very rapid state of change between blocks, the results may be disastrous. Consider the scenario in which true network hash rate drops by several orders of magnitude very suddenly. For example, if Lex Luthor has control of a very large proportion of some cryptocurrency network, say $99.9\%$, Lex has some options. He can certainly re-write the history of the blockchain and give himself all of the money (the usual $50\%$ attack route). But he could also simply decide to switch off all of his machines.
+
+Difficulty $d_{n+1}$ remains unchanged, as it is based on an estimate of previous block arrival rates. After all, difficulty only updates and adjusts upon receipt of a new block timestamp. Now, however, block arrival rate is very close to zero blocks per second. Now the problem is that the network comes to a standstill; blocks are not arriving because difficulty was very high and a very large actor in the mining space took their equipment offline. Difficulty will not change before the next block arrives, which could take an arbitrarily long period of time. No one is mining because blocks are very rare, and no one is on the network, so blocks remain rare forever, killing transaction processing capabilities. Of course, a slow decay or smaller jumps in hash rate will not have this effect. 
+
+One standard assumption in cryptocurrencies is that no single attacker controls more than $50\%$ of the network (otherwise, the ledger may be re-written, and presumably the currency will lose all value). Hence, we wish to construct a difficulty algorithm robust against a sudden halving or doubling of net hash rate.  If hash rate is cut in half but difficulty remains the same, block inter-arrival times will be doubled, and if hash rate is doubled, the inter-arrival times will be halved. Hence, under the standard $50\%$ attacker assumption, no network will be stalled forever. However, with sufficiently long block adjustment periods, this can still be disastrous for a currency. Consider a situation in which the Bitcoin network hash rate is cut in half immediately after a difficutly adjustment. Transactions will now be processed in $20$ minute blocks, rather than $10$ minute blocks, and thus transaction processing rate on the network will be halved. Furthermore, due to the two week adjustment period in the Bitcoin difficulty adjustment code, we can expect the network to remain in this state \emph{for two weeks}. This would be an agonizingly long period of time for transaction processing speeds to be cut in half. Of course, the Bitcoin network need not be worried about such a scenario, for the size of the Bitcoin network is a good insulator against such attacks.
+
+Due to this, we are interested in measuring the robustness of the two difficulty equations of interest (the CryptoNote reference code and our new difficulty equation) when exposed to sudden large changes in hash rate. In Section \ref{compareToDropsAndJumps}, we will investigate hash rate functions that are piecewise constant, and we will investigate how rapidly the two difficulty adjustment algorithms respond to hash rate changes of varying magnitude.
+
+\subsection{Consequences of the model: Orphaned blocks}\label{orphanedBlocks}
+In this seciton,  we discuss how block arrival rate relates to the production of orphan blocks. Indeed, we lament the speed of light's inevitable restriction on propagation of data and the resulting orphaned coins.  Regardless of parent coin selection rule, regardless of network structure, no matter the speed of our network, we will still occasionally see computers producing orphan blocks. If all users mine honestly, this is the primary source of orphan blocks, but there is nothing to prevent users from colluding in a selfish mining attack, as described in \cite{eyal2014majority}. Due to some conflict in the community over the term ``orphaned block,'' a more descriptive term perhaps could be \textit{dead branches of the  blocktree}.
+
+Consider the following example modeled after Bitcoin with a block target of $\lambda_{\text{target}}=1/600$ blocks per second. For the sake of argument, presume the network is a complete graph with a constant $N$ nodes and with the same transmission speed between any two nodes. We could model transmission time in the network between node $i$ and node $j$ as $\mu + e_{ij}$ where each $e_{ij}$ is a random variable with $E(e_{ij}) = 0$ and the mean propagation time is $\mu$. In in \cite{decker2013information}, Decker and Wattenhofer measured the average Bitcoin propagation time between nodes to be $\mu \approx 6.5$ seconds. Although the mean propagation times reported in that publication are somewhat out of date, they are a sufficient starting point for this discussion. When a node finds a successful nonce at time $t$, they announce this fact on the network. By time $t+6.5$, only half the network (on average) has heard of this new block. What about a computer in the second half of the network, the blind half? A node in that half of the network hashing between time $t$ and $t+6.5$ will have chosen a different parent coin because it does not have as much information as the nodes in the first half of the network.  Furthermore, one or the other branch will become part of the main chain eventually; these two events are mutually exclusive. Hence the total number of coin flips wasted is, with high probability, bounded by
+\[\text{Orphaned flips} \geq E\left[\text{Number of heads found by half the network in }[t,t+\mu)\right]\]
 and if hash power is distributed approximately uniformly, we can take a simple average. This approximation works out to be $\mu\cdot \lambda_{\text{target}}$ flips, where $\lambda_{\text{target}}$ denotes our target block arrival rate. In this example, this would be about $0.01083$ orphaned blocks on average. In expectation, a Bitcoin miner can expect about $1.083$ of their blocks orphaned for every $100$ blocks mined (once every $16$-ish hours). In probability, a Bitcoin miner can be $95\%$ confident that at least one block has been orphaned for every $275$ blocks mined. An individual with $1\%$ of the Bitcoin network's hashing power can expect, on average, to receive $1$ out of every $100$ blocks rewards, which would occur with rate $\lambda_{\text{target}}/100 = 1/1000$ blocks per minute. Hence, to obtain $275$ blocks would require about $190$ days, and such a miner can be $95\%$ confident she will see one orphaned block every $190$ days.
 
 On the other hand, if we have a coin modeled after Monero, with a block target time of $\lambda_{\text{target}}=1.0$ blocks per minute (compared with Bitcoin's $\lambda_{\text{target}} = 0.1$ blocks per minute), but we choose an otherwise similar setup as above, we will see about $0.1083$ orphaned blocks per mined block on average, corresponding to a $6.5$-second propagation time. In expectation, a Monero miner can expect $1.083$ blocks orphaned/wasted for every $10$ mined blocks. In probability, a Monero miner can be $95\%$ confident that at least one block has been orphaned every $26$ blocks mined. If a miner has $1\%$ of the Monero network's hashing power, such a miner can expect to receive $1$ out of every $100$ block rewards, which would occur with rate $\lambda_{\text{target}}/100 = 1/100$ blocks per minute. With $95\%$ probability, such a miner can be $95\%$ confident she will see one orphaned block every $1.8$ days.
 
-Before drawing any conclusions, note that these estimates rely upon \textit{expectations}, which we justified by assuming uniformity in hashing power. This is, of course, very false in the case of Bitcoin, which can see many orders of magnitude difference in performance between various mining rigs. In the case of Monero, which has a somewhat egalitarian proof of work algorithm, this assumption is less problematic. Either way, these values should not be taken as particularly precise. Rather, these values are intended give a broad idea of a ``first-glimpse'' into orphan coin analysis. Having said that, notice that the rate of orphan block arrivals is proportional to the target block arrival rate: the arrival rate of Monero blocks is ten times the arrival rate of Bitcoin blocks, Monero miners can expect around $10\%$ of blocks to be orphan blocks, and Bitcoin miners can expect around $1\%$ of blocks to be orphan blocks. Hence, setting target block arrival rates lower (say, one block every two minutes or three minutes) will dramatically reduce the rate of orphans on the Monero network.
+Before drawing any conclusions, note that these estimates rely upon \textit{expectations}, which we justified by assuming uniformity in hashing power. This is, of course, very false in the case of Bitcoin, which can see many orders of magnitude difference in performance between various mining rigs. In the case of Monero, which has a somewhat egalitarian proof-of-work algorithm, this assumption is less problematic. Either way, these values should not be taken as particularly precise. Rather, these values are intended give a broad idea of a ``first-glimpse'' into orphan coin analysis. Having said that, notice that the rate of orphan block arrivals is proportional to the target block arrival rate: the arrival rate of Monero blocks is ten times the arrival rate of Bitcoin blocks, Monero miners can expect around $10\%$ of blocks to be orphan blocks, and Bitcoin miners can expect around $1\%$ of blocks to be orphan blocks. Hence, setting target block arrival rates lower (say, one block every two minutes or three minutes) will dramatically reduce the rate of orphans on the Monero network.
 
 
 
-\section{Current Monero Approach to Difficulty }\label{currentApproach}
-In this section, we examine the current approach in the Monero codebase, which was inherited from the CryptoNote reference code. We elaborate on the mechanism employed by this algorithm to assess and adjust difficulty. We discuss drawbacks and problems with this approach.
 
-\subsection{The Reference CryptoNote Difficulty Adjustment}
-We examined in detail the CryptoNote reference code (written in C). We re-implemented the code relevant to the problem at hand in Python. This code is available at INSERT URL HERE, and is also available in Appendix \ref{referenceCode}. This Python script was written by Sarang Noether so as to algorithmically match the C algorithm originally implemented in the reference code. We chose Python to re-implement the algorithm for clarity reasons.
+\section{Difficulty Adjustment Formula}\label{myCode}
 
-The cryptocurrency Monero (XMR) has a block target time of one block per minute. Since timestamps are reported in seconds, and due to a few other algorithmic reasons, this target time is expressed as its reciprocal, setting \texttt{target~}$= 60.0$ seconds per block.  The current algorithm considers the most recently received $720$ blocks as a sample. For notational convenience, in this section, we list these $720$ block timestamps as $\left\{t_1, t_2, \ldots, t_{720}\right\}$ (where later indices are received later in time) and their difficulties as $\left\{d_1, d_2, \ldots, d_{720}\right\}$. Given a new timestamp, $t_{721}$, the algorithm tries to answer the question ``to what difficulty should this new block be set?''  
+In this section we use the model defined in Section \ref{modelDef} to determine our difficulty adjustment function and we present it as a sequence of steps. Recall that the overall goal is to keep the true rate $\lambda(t)$ to be approximately $\lambda^{*}$, the target block arrival time. Say that a user is given a blockchain of height $n-1$, which has a sequence of (possibly out-of-order) timestamps and difficulties
+\[\mathcal{B}_0 = (t_0, d_0), \mathcal{B}_1 = (t_1, d_1), \mathcal{B}_2 = (t_2, d_2), \ldots, \mathcal{B}_{n-1} = (t_{n-1},d_{n-1})\]
+Before mining, the user shall place the timestamps in order, i.e.\ the \emph{order statistics}, and consider the top $m$ of these:
+\[t_{(n-1)} > t_{(n-2)} > \ldots > t_{(n-m)}\]
+Define $\Delta_{n-1} T = t_{(n-1)} - t_{(n-m)}$ for each $n \geq m$. Notice that each $\Delta_{n} T$ is computed from the order statistics of a different sequence, and timestamps may occur out of order. We now compute the sample block arrival rate at height $n-1$
+\[\hat{\lambda}_{n-1} := m/\Delta_{n-1}T\]
+This is a running estimate of the instantaneous block arrival rate based on the maximum likelihood of a Poisson process with a constant rate/intensity. Each time a new block arrives, the \emph{order statistics} are recomputed, and the sample block arrival rate is recomputed from the new \emph{order statistics}. Our goal is to keep these samples very close to our target, $\lambda^*$. Further, since $\lambda = H/d$, each of these block arrival rates come equipped with an estimate of network hash rate:
+\[\hat{H}_{n-1} = d_{n-1} \hat{\lambda}_{n-1} = m d_{n-1}/\Delta_{n-1} T\]
 
-These timestamps need not occur in order (that is, it is possible that $t_{i} > t_{i+1}$ in this list) due to reasons mentioned previously (calibration error in local clocks, transmission speeds across the network, and dishonest users). The \textit{order statistics} of a list of random variables is precisely that same list but sorted; denote the order statistics of these timestamps as $\left\{t_{(1)}, t_{(2)}, \ldots, t_{(720)}\right\}$ so that $t_{(1)} \leq t_{(2)} \leq \ldots \leq t_{(720)}$.
-
-Since each block is associated with both a timestamp and a difficulty, we can mathematically view the list of the past $720$ blocks as a list of \textit{ordered pairs}:
-$$(t_1,d_1),(t_2,d_2),\ldots,(t_{720},d_{720})$$
-The reference code currently sorts the \textit{first coordinate} of this list into its order statistics, yielding the list
-$$(t_{(1)},d_1),(t_{(2)},d_2),\ldots,(t_{(720)},d_{720})$$
-whereas a more natural choice may be to sort the entire list \textit{together} by its first coordinate. Observe that the difficulties listed are not necessarily related to the newly ordered timestamps. See Section \ref{fail} for a discussion on this and the possible ramifications.
-
-Next, the CryptoNote reference code discards the upper and lower $60$ blocks of the last $720$. This is simply discarding $1/6$ of the total list, the top and bottom outliers, keeping the middle $83.33\%$ of the list. All that is necessary for this step is to compute the indices at which we slice out the interesting information, and this appears to occur completely correctly\footnote{Unlike previous sorting and slicing problems in the CryptoNote codebase. See, for example, \cite{macheta2014counterfeiting}}.  This leaves the list of $600$ timestamps, $\left\{t_{(61)}, t_{(62)}, \ldots, t_{(660)}\right\}$, which is also ordered, and the difficulties, $\left\{d_{61}, \ldots, d_{660}\right\}$. Recall that these difficulties are associated with the original blocks, not necessarily the difficulties associated with the now-sorted blocks. Note that, by discarding the upper and lower $60$ blocks, a user may start mining until $30$ blocks have been added to the main chain and then log off without their hashing power ever impacting their own mining rate.
-
-The algorithm then uses the following formula for computing the difficulty score of the next block:
+While time series analysis tools may be used to make predictions of $\hat{H}_n$, we suspect that such predictions will not outperform simpler approaches\footnote{This may be the subject of a future research bulletin.}. In our case, for difficulty adjustment, it is unnecessary to predict future values of hash rate (which is a complicated problem) if we are capable of producing an accurate estimate of instantaneous hash rate (which is a less complicated problem). Thus we choose our sequence of difficulties $d_n$ so as to allow difficulty to track the simple moving average of these network hashrate estimates; this is, in fact, equivalent to presuming network hash rate will remain static and match our current estimate until the next block arrives. We compute the sample mean of the last $\ell$ estimates of network hash rate
+\begin{align*}
+\overline{H}_{n-1} =& \frac{1}{\ell}\sum_{i=1}^{\ell} \hat{H}_{n-i}\\
+ =& \frac{1}{\ell} \sum_{i=1}^{\ell} d_{n-i} \hat{\lambda}_{n-i}\\
+ =& \frac{1}{\ell} \sum_{i=1}^{\ell} m d_{n-i} /\Delta_{n-i} T
+\end{align*}
+Since $\lambda = H/d$, we may keep $\lambda(t)$ close to $\lambda^*$ by choosing
 \begin{align}
-d_{721} :=&\frac{\tau \cdot  \sum_{i=61}^{660} d_i + \left(t_{(660)} - t_{(61)}\right) - 1}{\left(t_{(660)} - t_{(61)}\right)} \label{diffFormInitial}
+d_{n+1} =& \overline{H}_n/\lambda^* \notag \\
+=& \frac{\overline{H}_n}{\overline{H}_{n-1}} \frac{\overline{H}_{n-1}}{\lambda^*}\notag \\
+d_{n+1}=& \frac{\overline{H}_n}{\overline{H}_{n-1}} d_{n}\label{diffeq}
 \end{align}
-where $\tau$ is the block target time $\tau = 60$ seconds per block. After close inspection, the offset $-1$ in the numerator appears to take into account that blocks must be separated by at least one second. Indeed, this latter term\footnote{While this bound may be tightened to $t_{(660)} - t_{(61)} \geq 599$, the clarity of this difficulty adjustment does not benefit by such an adjustment.}, $\frac{t_{(660)} - t_{(61)} - 1}{\tau} > 0$, and so $d_{721} > \frac{\tau}{t_{(660)} - t_{(61)}} \sum_i d_i$. 
+That is to say, hash rate and difficulty should move in lockstep. At this point, we should be somewhat comforted. If average network hash rate has doubled, difficulty should probably double, and if it has halved, difficulty should halve. Notice that if we were to set $\ell = 1$, we would abandon the simple moving average and simply have the instantaneous estimate
+\[d_{n+1} = \frac{\hat{H}_{n}}{\hat{H}_{n-1}} d_n\]
+Returning to the more general case, we have
+\begin{align*}
+d_{n+1}=& \frac{\overline{H}_n}{\overline{H}_{n-1}} d_{n}\\
+=& \frac{\frac{1}{\ell} \sum_{i=1}^{\ell} d_{n-i+1} \hat{\lambda}_{n-i+1}}{\frac{1}{\ell} \sum_{i=1}^{\ell} d_{n-i} \hat{\lambda}_{n-i}} d_n\\
+=& \frac{\sum_{i=1}^{\ell} m d_{n-i+1}/\Delta_{n-i+1} T}{\sum_{i=1}^{\ell} md_{n-i}/\Delta_{n-i} T}d_n\\
+=& \frac{\sum_{i=1}^{\ell}  d_{n-i+1}/\Delta_{n-i+1} T}{\sum_{i=1}^{\ell} d_{n-i}/\Delta_{n-i} T}d_n
+\end{align*}
+We may expand this a bit to write it as a difference equation:
+\begin{align*}
+d_{n+1}=& \frac{\frac{d_{n}}{\Delta_n T} + \frac{d_{n-1}}{\Delta_{n-1} T} + \cdots +  \frac{d_{n-\ell + 1}}{\Delta_{n-\ell + 1} T}}{\frac{d_{n-1}}{\Delta_{n-1} T} + \cdots +  \frac{d_{n-\ell }}{\Delta_{n-\ell } T}}d_n\\
+=& \left(1 + \frac{\frac{d_n}{\Delta_n T} - \frac{d_{n-\ell}}{\Delta_{n-\ell} T}}{\frac{d_{n-1}}{\Delta_{n-1} T} + \cdots +  \frac{d_{n-\ell }}{\Delta_{n-\ell } T}} \right) d_n\\
+d_{n+1} - d_n =& \frac{\frac{d_n}{\Delta_n T} - \frac{d_{n-\ell}}{\Delta_{n-\ell} T}}{\frac{d_{n-1}}{\Delta_{n-1} T} + \cdots +  \frac{d_{n-\ell }}{\Delta_{n-\ell } T}} d_n
+\end{align*}
+We will use these formulae interchangeably as our difficulty adjustment function, but for shorthand, we may remember the general inductive rule
+\[d_{n+1}= \frac{\overline{H}_n}{\overline{H}_{n-1}} d_{n}\]
 
-To clean up this formula, let us notationally emphasize that $t_{(660)} - t_{(61)}$ captures the dispersion of timestamps by writing $\Delta T := t_{(660)} - t_{(61)}$. Other approaches may choose different estimates of the dispersion of timestamps, $\Delta T$; dispersion is a fun thing to play with in statistics. We also write $d_{721} = d_{\text{Next}}$ to emphasize the notion that this approach is intended to compute future difficulties, and that we are not necessarily handcuffing ourselves to a $600$ minute window. We replace $\sum_{i=61}^{660} d_i$ with $n \overline{d}$ where $\overline{d}$ is the sample mean of difficulty of the blocks in our sample size for the same reason. Different summary statistics may be used for possibly great effect in various blockchain approaches.
+We now describe how to implement this difficulty adjustment algorithm. Presume we are given a blockchain of the form
+\[(t_0, d_0), (t_1, d_1), \ldots, (t_{n-1}, d_{n-1})\]
+and a difficulty score of the next block to be added, $d_n$. Furthermore, presume some timestamp, $t_n$, has just been announced on the network and we wish to compute $d_{n+1}$. Denote the sequence of hash rate estimates
+\[\texttt{HASH} = \hat{H}_{n-1}$, $\hat{H}_{n-2}, \ldots \hat{H}_{n-k}\]
+where $k = \min(n,m)$. Denote the average of this sequence
+\[\texttt{AVG\_HASH} = \frac{1}{600} \sum_{i=1}^{600} \hat{H}_{n-i}\]
 
-We choose all of this notation to emphasize that we can use many choices of sample of previous blocks and many choices of relevant time periods; the particular formula
+Recall that we reject a block (and its timestamp, $t_n$) as illegitimate if its timestamp is too far away from the latest timestamp. In particular, the CryptoNote reference code rejects a block if $\text{median}(t_{n-i})_{i=1}^{m} > t_n$ or if $t_n > 7200 + \max(t_{n-i})_{i=1}^{m}$. We see no reason to change this window of acceptance.  We execute the following procedure for any $n \geq 1$:
+\begin{enumerate}[(1)]
+\item Append $(t_n, d_n)$ to the blockchain if the timestamp $t_n$ is legitimate.
+\item Store the top $m$ blocks' timestamps temporarily as, say, $\hat{t}_1$, $\hat{t}_2$, $\ldots$, $\hat{t}_{m}$. If $n < m$, store as many as we have.
+\item Sort these stored timestamps into their order statistics, $\hat{t}_{(1)} < \hat{t}_{(2)} < \ldots < \hat{t}_{(m)}$.
+\item Compute the span of time represented by these timestamps, $\Delta_{n} T = \hat{t}_{(m)} - \hat{t}_{(1)}$. 
+\item If $n < m$, then set $\hat{\lambda}_{n} := n/\Delta_n T$. Otherwise, set $\hat{\lambda}_{n} := m/\Delta_n T$.
+\item Compute the estimated hash rate during this span of time, $\hat{H}_{n} = \hat{\lambda}_{n} d_{n}$. 
+\item Prepend $\hat{H}_n$ to $\texttt{HASH}$.
+\item If $n \geq m$, remove $\hat{H}_{n-m}$ from $\texttt{HASH}$.
+\item Compute the new average of the hash rate estimates. If $n < m$, set $\texttt{NEW\_AVG\_HASH} = \frac{1}{n} \sum_{i=0}^{n-1} \hat{H}_{n-i}$. Otherwise, set $\texttt{NEW\_AVG\_HASH} = \frac{1}{m} \sum_{i=0}^{m-1} \hat{H}_{n-i}$.
+\item If $n < m$, set new difficulty $d_{n+1} = 1$. Otherwise, set new difficulty to $d_{n+1} = d_{n} \cdot \texttt{NEW\_AVG\_HASH}/\texttt{AVG\_HASH}$.
+\item Re-set $\texttt{AVG\_HASH} \leftarrow \texttt{NEW\_AVG\_HASH}$. 
+\item Wait until a new block timestamp, $t_{n+1}$, to arrive and go back to step (1) when it does.
+\end{enumerate}
+
+
+\section{The Current CryptoNote Code}\label{refCode}
+
+In Appendix \ref{referenceCode}, we present a re-implementation by Sarang Noether of the reference CryptoNote difficulty assessment method in python. We describe that difficulty assessment method here and make comparisons between their difficulty adjustment algorithm and ours.
+
+The CryptoNote reference code computes the next difficulty score in the following manner:
+\begin{enumerate}[(1)]
+\item Store the top $720$ blocks' timestamps temporarily as, say, $\hat{t}_1$, $\hat{t}_2$, $\ldots$, $\hat{t}_{720}$.
+\item Store the top $720$ blocks' difficulty scores temporarily as, say,
+$\hat{d}_1$, $\hat{d}_2$, $\ldots$, $\hat{d}_{720}$.
+\item Sort the top timestamps in increasing order as $\hat{t}_{(1)}$, $\hat{t}_{(2)}$, $\ldots$, $\hat{t}_{(720)}$
+\item Eliminate outlying bottom $1/12$ and outlying top $1/12$ of each of these lists, leaving $(\hat{t}_{(61)}, \hat{t}_{(62)}, \ldots, \hat{t}_{(660)})$ and $(\hat{d}_{61}, \hat{d}_{62}, \ldots, \hat{d}_{660})$.
+\item Compute the span of time represented by these timestamps, $\Delta t = \hat{t}_{(660)} - \hat{t}_{(60)}$.
+\item Compute the sum of the difficulties, $D = \sum_{i=61}^{660} \hat{d}_i$.
+\item The difficulty of the next block, $\mathcal{B}_{H+1}$, given some target block arrival rate $\lambda$ blocks per unit time, is then computed from the formula
 \begin{align}
-d_{\text{next}} =& \frac{n\tau}{\Delta T}\overline{d} + \frac{\Delta T - 1}{\Delta T} \label{diffForm}
+\hat{d}_{721} :=&\frac{D/\lambda + \Delta T - 1}{\Delta T} \label{diffFormInitial}
 \end{align}
-will be referred to as the reference CryptoNote difficulty adjustment. In order to begin analyzing the reference CryptoNote difficulty adjustment, one must consider equilibria and stability of those equilibria. 
+\end{enumerate}
 
+We make some observations about this approach. First, as usual, timestamps added to the blockchain need not occur in order. In fact, we have no reason to expect that they will. Sorting the timestamps seems reasonable (in fact, we do this in our proposed algorithm in Section \ref{myCode}). Further, removing the outlying elements from the list also seems reasonable. However, notice that the timestamps were sorted separately from the difficulties. Hence, while an outlying timestamp may be removed, it's associated difficulty score is still included in the computation and the associated difficulty score of some other block is subsequently removed as an outlier (although the timestamp associated with the removed difficulty score may not have been an outlier originally!).  We discuss this problem in detail in Section \ref{reorderingproblems}.
 
-
-\subsection{Drawbacks of the CryptoNote Approach}\label{fail}
-This section discusses a few known drawbacks and failures of the current Monero codebase.
-
-\subsubsection{Difficulty Should Not Act Like a Clock}
-
-Notice that if all blocks have recently arrived on target, then $n\tau = \Delta T$, but not \emph{vice versa}, since $\Delta T$ is determined solely by the first and last timestamps. That is to say, we have the general fact that on-target block arrival times provide that $n\tau \approx \Delta T$, but $n\tau \approx \Delta T$ under very general circumstances with quite a bit of wiggle-room for the blocks that arrive between these two endpoints. Furthermore, $\frac{\Delta T - 1}{\Delta T} \approx 1$ and $\frac{n\tau}{\Delta T} \approx 1$ for most reasonable choices of $n$ and $\tau$. Consider, for example, Monero, which has $\tau = 60.0$ seconds per block and $n = 600$. For such a currency, we can expect, in most circumstances, $\Delta T \approx 36000$ and so $\frac{\Delta T - 1}{\Delta T} \approx 0.99997$.  Hence, the reference CryptoNote difficulty adjustment algorithm satisfies
-\[d_{\text{next}} \approx \overline{d} + 1\]
-under very general circumswtances, which demonstrates on an intuitive level that the reference CryptoNote difficulty algorithm acts like a clock rather than an estimate of hashing power.
-
-We can demonstrate this rigorously, rather than intuitively, however. We first show that no equilibrium possible in this system when all blocks have arrived on target. Intuitively, when designing a difficulty algorith, we ought to attain $d_{\text{next}} = \overline{d}$ whenever the network is static and unchanging, but this is not possible in the CryptoNote reference code. Indeed, assume that $d_{\text{next}} = \overline{d}$ so that equilibrium occurs and that all blocks have arrived on target so that $n \tau = \Delta T$.  Then we have
-
+Second, notice that the formula for $\hat{d}_{721}$ may seem a bit strange. Recall that, given $n$ sample inter-arrival times $S_1, S_2, \ldots, S_n$, then the maximum likelihood estimate of the Poisson rate is $\hat{\lambda} = n/\sum_i S_i$ or rather, if we have $n$ arrivals in a time interval of width $\Delta T$, then $\hat{\lambda}^{-1} = n/\Delta T$. These are the formulas used in Section \ref{myCode}. Also notice that we may replace a sum, $D = \sum_i d_i$, with a product of the mean, $D = n\overline{d}$. Since the CryptoNote reference code considers the middle $600$ blocks after slicing outliers, we may write
 \begin{align*}
-d_{\text{next}} =& \frac{n\tau}{\Delta T}\overline{d} + \frac{\Delta T - 1}{\Delta T}\\
-\overline{d} =& \overline{d} + \frac{\Delta T - 1}{\Delta T}
+\hat{d}_{721} =&\frac{D/\lambda + \Delta T - 1}{\Delta T}\\
+=& \frac{\hat{\lambda}}{\lambda} \overline{d} + \frac{\Delta T - 1}{\Delta T}
 \end{align*}
-Which can only occur if $\Delta T = 1$. However, $\Delta T \geq n$ since each block must occur at least one second later than the previous block\footnote{The CryptoNote reference code attaches a unique integer timestamp to each block, so blocks cannot have the same timestamp.}. 
-
-On the other hand, let us split our two assumptions up into two distinct assumptions. Rather than assuming that we are at an equilibrium \emph{and} all recent blocks have arrived on target, now let us assume only that we are at equilibrium. Then we have
-\begin{align*}
-d_{\text{next}} =& \frac{n\tau}{\Delta T} \overline{d} - \frac{\Delta T - 1}{\Delta T}\\
-\overline{d} =& \frac{n \tau}{\Delta T} \overline{d} - \frac{\Delta T - 1}{\Delta T}\\
-\overline{d} \Delta T =& n\tau \overline{d} - \Delta T + 1\\
-(1+\overline{d})\Delta T =& n\tau \overline{d} + 1\\
-\Delta T =& \frac{n \tau \overline{d} + 1}{\overline{d} + 1}\\
-=& n\tau + \frac{1 - n\tau}{\overline{d} + 1}
-\end{align*}
-Hence, under the assumption of equilibrium, we conclude that blocks can only arrive on target if $n\tau = 1$. Again, our target block arrival time is $\tau = 60.0$ for a Monero-like cryptocurrency, and $\tau = 1209600.0$ for a Bitcoin-like currency; in both cases, $n > 600$. Indeed, $n\tau = 1$ is only true for coins with a block target time less than one second. 
-
-However, we do have the result that if we are at equilibrium with a very large difficulty, then $\Delta T \approx n\tau$, i.e.\ blocks are arriving \textit{almost} on target.
-
-Now let us only assume that all recent blocks have arrived on target. If this is the case, then $\Delta T = n\tau$ and we have
-\begin{align*}
-d_{\text{next}} =& \frac{n\tau}{\Delta T}\overline{d} + \frac{\Delta T - 1}{\Delta T}\\
-=& \frac{n\tau}{n\tau}\overline{d} + \frac{n\tau - 1}{n\tau}\\
-=& \overline{d} + 1 - \frac{1}{n\tau}
-\end{align*}
-In the case of Monero, we have $n = 600$ and $\tau = 60.0$ so $1/n\tau$ is nearly negligible and we re-obtain our previous result that difficulty increases like clockwork. We conclude that if all blocks have recently arrived on target, then the reference CryptoNote difficulty adjustment increases \emph{about} linearly.
-
-Now consider the question: under what circumstances may difficulty drop? Recall that, in general, difficulty is a large, positive number so $1-\overline{d} < 0$:
-\begin{align*}
-d_{\text{next}} =& \frac{n\tau}{\Delta T}\overline{d} + \frac{\Delta T - 1}{\Delta T} < \overline{d}\\
-n\tau \overline{d} + (\Delta T - 1) <& \overline{d} \Delta T\\
-n\tau \overline{d} - 1 <& \Delta T (\overline{d} - 1)\\
-\frac{n\tau \overline{d} - 1}{\overline{d} - 1} <& \Delta T\\
-\overline{d}\frac{n\tau - \overline{d}^{-1}}{1 - \overline{d}^{-1}} < \Delta T
-\end{align*}
-Now, presuming difficulty is large (as it usually is), we have $\overline{d}^{-1} \approx 0$ and so we have the idea that difficulty will drop whenever $\overline{d} n \tau < \Delta T$, or rather, whenever $\overline{d} < \frac{\Delta T}{n\tau}$.
-
-Since $n^{-1}\Delta T$ is the sample mean of inter-arrival times of blocks, we see that this aspect of the difficulty algorithm acts as it should: if sample mean of inter-arrival times is very large enough compared to the target, then $\frac{\Delta T}{n\tau}$ is large enough compared to $\overline{d}$, and so difficulty drops. 
 
 
-\subsubsection{Ordering by Time but not Difficulty}
-The original CryptoNote developers sorted the list of ordered pairs
-$$(t_1, d_1),(t_2,d_2),\ldots,(t_{720},d_{720})$$
-by the first coordinate, the timestamps, yielding the list
-$$(t_{(1)},d_1), (t_{(2)},d_2),\ldots,(t_{(720)},d_{720}).$$
-However, we now clearly see that the block difficulty associated with the $n^{\text{th}}$ block in this list, $d_n$, is no longer associated with timestamp $t_n$, but instead the $n^{\text{th}}$ ordered timestamp, $t_{(n)}$. The severity of this oversight could be slight. As we saw above, under assumptions of a static network performing ``on target,'' difficulty will increase approximately linearly in time. Sorting timestamps without sorting the difficulty associated with those timestamps could lead to strange results, as the current algorithm re-assembles the timestamps of blocks without regard to their difficulty.
-
-To illustrate the potential for problems, consider that the next step in the Monero algorithm is to discard $1/6$ of the list according to outliers in the list $\left\{t_{(n)}\right\}$. But now the choice of sorting becomes strikingly important; there are going to be many instances in which we entirely discard block $X$ due to its timestamp, but preserve the difficulty assigned to block $X$ and simply \textit{re-assign that difficulty to a different block}.
-
-Statistically, this is attempting to assess some information from a population of people about their height and age, and then using that information to infer information about their gender. During the process, you take a sample of people and randomly permute their ages. You then discard the outlying people in terms of \textit{height} and conveniently forget that you permuted their ages in the first place! In the meanwhile, the fact that nice old people tend to be short gets entirely washed out of your statistical analysis. 
-
-\subsubsection{Incorrect Computation}
-The median in the algorithm is computed as a sum:
-
-\begin{lstlisting}[language=Python,basicstyle=\small,breaklines=true]
-# Apply the median rule
-for i in range( check_window, len( timestamps ) ):
-    try:
-        median = sum( timestamps[i-check_window:i] )
-        if timestamp[i] < median:
-            timestamp.pop( i )
-    except:
-        pass
-\end{lstlisting}
-Perhaps the original coders simply got their terminology confused. The mean is proportional to the sum, but the median is not.
-
-\section{Proposed Difficulty}\label{proposedChanges}
-
-In this section, we propose partial solutions to the problem of determining the difficulty of the next block to be added on the blockchain. We propose some methods of computing difficulty. The first method we propose is overly simplistic, but is based on good intuition. We discuss the drawbacks of this method and use those drawbacks to draw conclusions about how a ``good'' difficulty algorithm should behave. We then propose another method that is less simplistic and exploits the uniformly minimum variance unbiased estimator (UMVUE) of the block arrival rate assuming blocks arrive in a Poisson process. Finally, we propose another method that builds on the UMVUE method to include a so-called ``momentum'' term to prevent manipulation by malicious parties. Finally, we use Monte Carlo simulations, described in detail, to illustrate how our method is more reliable than the current Monero approach. 
-
-Recall that difficulty is intended to be proportional to the net hash power of the network; any difficulty computation should be based on a statistical estimate of the rate of block arrivals. The best possible estimate for this would come from an impartial and trusted observer with an accurate clock observing all hashes, both failures and successes, and their timestamps. This is absurd for a variety of reasons; not only is it impossible to establish the trustworthiness of any one given observer, it is exceedingly difficult to establish the accuracy of their clock, even under the generous assumption of trustworthiness; Einstein would be pleased by the difficulty of our situation, all puns intended. All we can do is allow each user to trust their own clock and their observed data. Our suggestions are based on this notion. 
-
-Our estimates take the form of the uniformly minimum variance unbiased estimator (UMVUE) of the rate of arrival of a Poisson process, which also happens to match the maximum likelihood estimate due to the fact that the inter-arrival times of a Poisson process are drawn from an exponential distribution, which is a member of the exponential family.
-
-We list the blocks on the blockchain in terms of height and in descending order, $\left\{\mathcal{B}_H,\mathcal{B}_{H-1}, \ldots,\mathcal{B}_1, \mathcal{B}_0\right\}$, where $\mathcal{B}_0$ denotes the genesis block, and the current height of the blockchain is $H+1$ (since we start counting at zero). The next block to be added will have height $H+2$ and will be labeled $\mathcal{B}_{H+1}$ with timestamp $t_{H+1}$. Our task is to compute difficulty $\mathcal{D}_{H+1}$. For each block $\mathcal{B}_i$, for $0 \leq i \leq H$, denote the timestamp and difficulty of that block to be $(t_i,\mathcal{D}_i)$. Presuming that $H > N+1$, the top $N+1$ blocks on the blockchain have timestamps and difficulties, in descending order, $\left\{(t_{H},\mathcal{D}_{H}),(t_{H-1},\mathcal{D}_{H-1}),\ldots,(t_{H-N},\mathcal{D}_{H-N})\right\}$.
-
-
-\subsection{Simplest Method}
-In this subsection, we discuss a na\"{i}ve method of difficulty adjustment; to be clear, we are not recommending this method, but in fact we are pointing out how a very simple method can be used to produce a better method than the current implementation. Using the Difficulty Equation \ref{difficultyEquation}, we see that difficulty is proportional to network hash rate:
-\[\mathcal{D} = k\tau \widehat{N\overline{r}}\]
-where $k$ is a constant, $\tau$ is our target number of seconds per block arrival, and $\widehat{N\overline{r}}$ is an estimate of network hash rate. Using the rate notation for a Poisson process, $\lambda = 1/\tau$, so we could easily write $\mathcal{D} = \frac{k}{\lambda} \widehat{N\overline{r}}$.
-
-The simplest method of assigning difficulty to $\mathcal{B}_{H+1}$ is to consider whether the rate of arrival has changed from block $\mathcal{B}_{H}$ to $\mathcal{B}_{H+1}$. Indeed, we could simply compare $\left|t_{H+1} - t_{H}\right|$ compared to $\left|t_{H} - t_{H-1}\right|$. A simple decision rule can be manufactured by setting $r:= \displaystyle \frac{\left|t_{H+1} - t_{H}\right|}{\left|t_{H} - t_{H-1}\right|}$ and then using the rule
-\begin{equation*}
-\begin{cases}
-r \gg 1 & \Rightarrow \text{~Decrease difficulty}\\
-r \approx 1 & \Rightarrow \text{~Do not change difficulty}\\
-r \ll 1 & \Rightarrow \text{~Increase difficulty}.
-\end{cases}
-\end{equation*}
-
-A specific implementation could be as simple as setting $\mathcal{D}_{H+1} = \mathcal{D}_{H}/cr$ when $r > 1+ r^{*}$, and setting $\mathcal{D}_{H+1} = \mathcal{D}_{H}/r$ when $r < 1 - r^{*}$ for some chosen threshold $0 < r^{*} < 1$ and constant $c$. Then, by setting an initial condition, the idea is that difficulty will more or less change continuously, or at least whenever net hash power has significantly changed. This algorithm is appealing to intuition. However, there are some issues.
-
-First, we are only taking one sample of block arrival times. Hence, large variability from stochasticity could easily cripple a cryptocurrency employing this algorithm without any degree of manipulation. For example, consider the case of a Monero-like cryptocurrency in which two blocks arrive on target, with $t_{H}-t_{H-1} = \tau = 60.0$, but then a third block arrives almost simultaneously with the second block, only one second later. Then $r = 1/60$, so difficulty will increase by a factor of $60.0$. Certainly, taking a sample of only one is inappropriate here.
-
-Second, this algorithm does not at all take into account the block target time $\tau$. This algorithm causes an adjustment in difficulty only under the circumstances that network behavior has changed, but not its current state. Recall that a user can manipulate the timestamps placed on her own blocks. There is nothing to prevent a user from mining several blocks in a row and setting their timestamps to manipulate the above algorithm.  A dishonest miner under these circumstances could first force difficulty to spike upward and then carefully release her blocks such that difficulty never returns to its previously lower levels; alternatively, a dishonest miner under these circumstances could carefully choose their timestamps so that $r \approx 1$ for each of their blocks, causing their contribution toward difficulty to be discounted. This illustrates how this method is insufficient for our purposes and leads us  to determine some further requirements for difficulty beyond a simple estimate of network hash rate.  However, this method is still superior to the current CryptoNote reference code, which acts like a clock!
-
-\subsection{Further Requirements For Difficulty}
-Any measure of difficulty must include a notion of the target block arrival time. Any measure of difficulty must be resistant to stochasticity; minimizing the variance of an estimate is a good way to ensure this. This can be obtained by taking a sample of a size larger than $N=1$, as we did in the previous example, and by using the uniformly minimum variance unbiased estimator (UMVUE) of inter-arrival times under the assumption that the underlying process is Poisson. These requirements must be balanced against the desire for difficulty to be fairly responsive; large sample sizes are unresponsive. Rapid changes are caused by large numbers of users dropping off the network, for example, due to mining multipools switching their target coins. Thus, our sample sizes can neither be too small (as we've already seen) or too large (to avoid a sluggish response).  Finally, we hope to find a method that is resistant to manipulation in a certain sense; we will discuss this below.
-
-\subsection{UMVUE Method}
-We are given a block target $\tau=1/\lambda$. We recall the asymptotic similarity between iterative continuous-time Bernoulli trials and the Poisson process. Given the Difficulty Equation \ref{difficultyEquation}, we have
-\[\mathcal{D} = \frac{k}{\lambda} N\overline{r} = k\tau N\overline{r}\]
-we will see a rate of arrival $\lambda=1/\tau$, where each hash has probability of success $k/\mathcal{D}$. Here's the trick: we know our previous value of $\mathcal{D}$, which is the difficulty of the previous block or, alternatively, an average of previous blocks. We also know our target value of $\frac{k}{\lambda}=k\tau$ and we wish to estimate $N\overline{r}$. But we can't observe $N\overline{r}$, we can only observe the actual rates of arrival. We may use these actual rates of arrival to compute the UMVUE, $\hat{\lambda}$, which may differ from our target rate of arrival, $\lambda$. Given the last $N$ blocks
-\[\left\{(t_{H}, \mathcal{D}_{H}), (t_{H-1}, \mathcal{D}_{H-1}), \ldots (t_{H-N+1}, \mathcal{D}_{H-N+1})\right\}\]
-we compute the order statistics in time
-\[t_{(H)}, t_{(H-1)}, \ldots, t_{(H-N+1)}\]
-where this list is simply the \textit{ordered} list of timestamps. In particular, $t_{(H)} = \text{max}\left\{t_i;H-N+1 \leq i \leq H\right\}$ and $t_{(H-N+1)} = \text{min}\left\{t_i; H-N+1 \leq i \leq H\right\}$. We obtain the uniformly minimum variance unbiased estimator of the rate of arrival for a Poisson process given this data 
-\[\widehat{\lambda} = N/(t_{(H)} - t_{(H-N+1)})\]
-The Difficulty Equation \ref{difficultyEquation} provides something along the lines of 
-\[\widehat{N\overline{r}} = \frac{\mathcal{D}_{\text{old}}}{k} \hat{\lambda}\]
-where $\mathcal{D}_{\text{old}}$ may be taken to be our previous value of difficulty, $\mathcal{D}_H$, or the average value of difficulty over the previous $N$ blocks, $\overline{\mathcal{D}}$, or some other ``current estimate of difficutly.'' No matter what, however, we may not exclude outliers from this estimate, for this would allow us to avoid counting the outlier blocks, and hence, any malicious actor could tailor their timestamps to look like outliers to be ignored. Here, $k$ is the constant as before, and $\hat{\lambda}$ is the UMVUE of our rate of arrival. This will provide us
-\begin{align*}
-\mathcal{D}_{H+1} =& k\tau \widehat{N\overline{r}}\\
-=& k\tau \frac{\mathcal{D}_{\text{old}}}{k}\hat{\lambda}\\
-=& \hat{\lambda}\tau \mathcal{D}_{\text{old}}
-\end{align*}
-A smooth difficulty adjustment algorithm would use $\mathcal{D}_\text{old} = \overline{\mathcal{D}}$, whereas if we have no concern for smoothness, we may use $\mathcal{D}_{\text{odl}} = \mathcal{D}_H$.  Since $\tau = 1/\lambda$, our multiplicative adjustment is the ratio $\widehat{\lambda}/\lambda$, or rather, we adjust our difficulty by the ratio of our observed rate of arrival to our target rate of arrival. 
-\begin{align}\label{proposedChangedAlgo}
-\mathcal{D}_{H+1} =& \frac{\widehat{\lambda}}{\lambda} \mathcal{D}_{\text{old}} 
+In the CryptoNote reference code, we have a block arrival target of $60.0 s$ per block, so to observe $600$ blocks should take, on average, $36000 s$, so $\frac{\Delta T - 1}{\Delta T} \approx 0.99997$ and this formula may be written as, approximately,
+\begin{align}
+\hat{d}_{721} =&  \frac{\hat{\lambda}}{\lambda} D + 1 \label{diffFormApprox}
 \end{align}
-We refer to Equation \ref{proposedChangedAlgo} as our Proposed Difficulty Algorithm. If $\hat{\lambda}$ is smaller than $\lambda$, then (a) fewer arrivals per second have been observed than our target and (b) this ratio is less than one, so difficulty will drop. Similarly, if $\hat{\lambda}$ is greater than $\lambda$, then (a) more arrivals per second have been observed than our target and (b) this ratio is greater than one so difficulty will increase. Note how similar this expression is to our so-called \textit{Simplest Method} from the previous section, which was based solely on the intuition of rates of arrivals.  Now our task has reduced to finding good estimates of $\hat{\lambda}$, which is best obtained with the UMVUE.
+We provide further discussion of this in Section \ref{clockproblems}.
+
+\subsection{Criticisms of the CryptoNote Reference Code: Sorting Timestamps Alone}\label{reorderingproblems}
+
+To see the gravity of the mistake made in re-ordering timestamps in isolation from the difficulties, consider throwing out the outlying measurements incorrectly from a more usual example. Let's say we are measuring the height and weight of everyone in Huntington-Ashland, West Virginia, the fattest city in America. We line everyone up and give them a number according to their position in line. This is their \emph{index}. We then measure their heights and their weights, proceeding from index $1$, the first person in line, down to \emph{index} $P$, the last person in line, where $P$ is the population of the town. These measurements yield the list of data $(H_1, W_1)$, $(H_2, W_2)$, and so on up to $(H_P, W_P)$. Of course, $H_i$ and $W_i$ correspond to the height and weight, respectively, of the $i^{th}$ person in line. Furthermore, let us presume that we wish to discard the top $1/12$ of the list and the bottom $1/12$ of the list, leaving $5/6$ of the population in the list. 
+
+However, before analyzing weight by removing outliers, presume we follow a similar procedure as the CryptoNote reference code. We order heights separately from weights:
+\begin{align*}
+\hat{H}_1 =& \min\left\{H_1, H_2, \ldots, H_P\right\}\\
+\hat{H}_2 =& \min\left\{H_1, H_2, \ldots, H_P \mid H_i \neq \hat{H}_1\right\}\\
+\hat{H}_3 =& \min\left\{H_1, H_2, \ldots, H_P \mid H_i \neq \hat{H}_1, \hat{H}_2\right\}\\
+\vdots
+\end{align*}
+Now, $\hat{H}_1$ corresponds to the height of the shortest person in town, but $W_1$ still corresponds to the weight of the first person in line. Similarly, $\hat{H}_P$ corresponds to the height of the tallest person in town, but $W_P$ still corresponds to the weight of the last person in line.  The approach presented by the CryptoNote reference code would then exclude all pairs $(\hat{H}_i, W_i)$ if the index $i$ is in the first $P/12$ or in the last $P/12$ indices.
+
+What data do we have remaining if we do this? We have $\hat{H}_{P/12 + 1}$, $\hat{H}_{P/12 + 2}$, $\ldots$, $\hat{H}_{5P/6}$, which correspond to the heights of the middle quantile. However, we also have $W_{P/12 + 1}, \ldots, W_{5P/6}$, which corresponds to the weights of the people \emph{who were in the middle of the line}. While our initial goal was to discard outliers based on their height and weight measurements, what we ended up doing was discarding outliers based on their position in line. While this may have yielded interesting statistical information about how rapidly overweight people make it into a queue, it is not relevant to relating height with weight. 
+
+With this simple description, we have demonstrated that the methods used in the CryptoNote reference code for discarding outliers are, at best, inappropriately applied.
+
+\subsection{Criticisms of the CryptoNote Reference Code: Lack of Equilibrium Solutions with Constant Hash Rate}\label{clockproblems}
+
+Our intuitive notion of difficulty says we should expect difficulty to go up when hash rate has gone up, and for difficulty to go down when hash rate has gone down. If hash rate has not changed, then difficulty should not change. Intuitively, the CryptoNote difficulty assessment formula, Equation \ref{diffFormInitial}, does a satisfactory job. Indeed, when hash rate doubles, the sample rate $\hat{\lambda}$ doubles, and so the difficulty of the next block will be approximately double the average of the sample blocks.  
+
+However, there is a critical problem with the formula, which has to do with equilibrium solutions to the difficulty equation. Again, our intuitive notion is that, if hash rate remains unchanged, then difficulty should remain unchanged, and vice versa. Of course, we may not directly measure hash rate, but we can directly measure rate of block arrivals. So what happens if blocks have been arriving on target? This would be the best possible indication that hash rate has remained unchanged.
+
+If blocks have been arriving on target, then $\hat{\lambda} \approx \lambda$ and so difficulty (approximately) goes up by one according to Equation \ref{diffFormInitial}.  On the other hand, if difficulty is being held (approximately) constant at equilibrium, then $\hat{d}_{721} \approx \overline{d}$ and
+\begin{align*}
+\hat{d}_{721} =& \frac{\hat{\lambda}}{\lambda} \overline{d} + 1\\
+\overline{d} \approx& \frac{\hat{\lambda}}{\lambda} \overline{d} + 1\\
+(1 - \frac{\hat{\lambda}}{\lambda})\overline{d} \approx& 1
+\end{align*}
+But then we must conclude that our sample rate, $\hat{\lambda}$, must be quite different from our target rate, $\lambda$, otherwise we would conclude $0 = 1$, which is absurd. Indeed, difficulty may never be held constant when blocks are arriving on target given Equation \ref{diffFormInitial}.
+
+This analysis of an equilibrium solution suggests that we have verified that $\hat{\lambda} = \lambda$ does not directly correspond with a static difficulty in the CryptoNote reference code, violating our intuition about difficulty.
+
+\section{Comparing Difficulty Adjustment Performance}\label{comp}
+
+In Appendix \ref{mySimulationCode}, we present Python code that takes as input a piecewise constant hashrate function and produces a stochastically generated blockchain in the form of a sequence of timestamp-difficulty ordered pairs. In this section, we use this code to compare the CryptoNote reference code difficulty equation (see Section \ref{refCode}) with the proposed difficulty equation (see Section \ref{myCode}). Recalling our interest in comparing the robustness of these equations against sudden changes in hash rate, and recalling that $\lambda = H/d$ in the model from Section \ref{modelDef}, we investigate several hash rate scenarios with piecewise constant hash rate functions. The overall goal of the difficulty equations is to keep observed block arrival rates, $\hat{\lambda}$, close to target block arrival rates, $\lambda^{*}$. Hence, this provides our metric for goodness of a difficulty equation: we compute the relative error between sample block arrival rate and target block arrival rate for each of our difficulty adjustment equations and for each hash rate function under investigation.
+
+We look into four hash rate scenarios. In the first hash rate scenario, we investigate a piecewise constant hash function of the form $H(t) =  \alpha^{\lfloor \beta t \rfloor}$ where $\lfloor x \rfloor$ denotes the usual floor function, i.e.\ $\lfloor x \rfloor = \max\left\{i \in \mathbb{Z} \mid i \leq x\right\}$, and where we take $\alpha > 1$ controls overall growth rate and $1/\beta > 0$ controls the period of time between hash rate jumps. This would be an exponential growth model. In the second hash rate scenario, we investigate a logistic growth model with $H(t) = K(1 + Ae^{-k\lfloor \beta t\rfloor })^{-1}$ where $K$ is the carrying capacity of the network, $k$ is the overall population growth rate, the constant $A$ satisfies $A = (K-H(0))/H(0)$, and where $1/\beta > 0$ represents the period of time between hash rate jumps. In the third hash rate scenario, we investigate a hash rate that starts small, jumps to a very large value for a sufficiently long enough period of time for difficulty to come to equilibrium, and then drops back to a very small value for the remainder of the simulation (i.e.\ a tophat function). This way, we can investigate two population-based hash rate models, and we can also investigate the scenario presented in Section \ref{hashrateattack}. In the final hash rate scenario, we investigate a square wave function to investigate what sort of effect periodic behavior in mining has upon network hash rate.
+
+\subsection{Exponential Growth in Hash Rate}\label{expComp}
+
+We find that, for large hash rates, the CryptoNote reference code tracks hash rate very well. However, the term $(\Delta T - 1)/\Delta T$ leads to a roughly linear increase in hash rate over time when hash rate is small or constant. This is true regardless of the choice of $\beta$, which controls how long hash rate is held constant before changing. In all investigations of exponential growth in hash rate, we find that the difficulty equation presented in Section \ref{myCode} does a better job of tracking true network hash rate compared to the CryptoNote reference code difficulty equation presented in Section \ref{refCode}. Not only is the relative error between observed block arrival rate and target block arrival rate generally smaller when we use our difficulty equation instead of the CryptoNote reference code, our difficulty equation appears to approach true network hash rate at a faster rate than the CryptoNote reference code.
+
+We numerically validate these claims by investigating two hash rate functions, $H_1(t) = 1.001^{\lfloor t/600 \rfloor}$, representing a $0.1\%$ increase in hash rate every ten minutes, and $H_2(t) = 1.005^{\lfloor t/120 \rfloor}$, representing a $0.5\%$ increase in hash rate every two minutes. Using a given hash rate function, we generate one blockchain using our new difficulty adjustment equation described in Section \ref{myCode}:
+\[\underline{\mathcal{B}}_{new} = (\mathcal{B}_{new,0}, \mathcal{B}_{new,1}, \mathcal{B}_{new,2}, \ldots, \mathcal{B}_{new,n_{new}-1})\]
+and we generate another blockchain using the CryptoNote reference code difficulty adjustment equation described in Section \ref{refCode}:
+\[\underline{\mathcal{B}}_{old} = (\mathcal{B}_{old,0}, \mathcal{B}_{old,1}, \mathcal{B}_{old,2}, \ldots, \mathcal{B}_{old,n_{old}-1})\]
+From these blockchain objects, we may compute the sample block arrival rates, $\hat{\lambda}$, at each new timestamp, and we may compare the relative error between $\hat{\lambda}$ and the target block arrival rate, $\lambda^{*}$ as a function of the latest timestamp (which may not be the top timestamp).
+
+\subsection{Logistic Growth in Hash Rate}\label{logComp}
+
+We again find that, for large hash rates, the CryptoNote reference code does a decent job. However, when hash rate is small (i.e.\ the beginning of a logistic population) or held constant (i.e.\ the end of a logistic population), the linear growth in the CryptoNote reference difficulty equation becomes steadily more apparent as time goes on; the relative error increases regularly over time. The effect of this linear term on the upswing portion of the logistic curve is difficult to detect, but is still present.  Not only is the relative error between observed block arrival rate and target block arrival rate generally smaller when we use our difficulty equation instead of the CryptoNote reference code, our difficulty equation appears to approach true network hash rate at a faster rate than the CryptoNote reference code.
+
+We numerically validate these claims by investigating two hash rate functions. We first investigate $H_1(t) = 10^\alpha/(1+A^{-k \lfloor \beta t \rfloor}$, where $A = (10^{\alpha} - 10^{\gamma})/10^{\gamma}$, representing a logistic growth model with carrying capacity $10^\alpha$, initial hash rate $10^{\gamma}$, growth rate $k$, and $1/\beta$ controls how often hash rate jumps. In particular, with a network carrying capacity of $1.0$ petahash per second ($\alpha = 15$) and an initial network hash rate of $1.0$ gigahash per second ($\gamma = 9$), with $\beta = 1/6 h^{-1}$ so that hash rate changes every six hours, and with a maximum growth rate of $k=1.0$ megahash per second per day.   We also investigate $H_2(t)$ of the same form with a terahash per second carrying capacity ($\alpha =12$), an initial hash rate of $1.0$ gigahash per second ($\gamma = 9$), and with $\beta = 1/6 h^{-1}$ so that hash rate changes every six hours, and with a maximum growht rate of $10^7$ hashes per second, a ten fold increase over $H_1(t)$.
+
+Using these given hash rate functions, we generate one blockchain using our new difficulty adjustment equation described in Section \ref{myCode}:
+\[\underline{\mathcal{B}}_{new} = (\mathcal{B}_{new,0}, \mathcal{B}_{new,1}, \mathcal{B}_{new,2}, \ldots, \mathcal{B}_{new,n_{new}-1})\]
+and we generate another blockchain using the CryptoNote reference code difficulty adjustment equation described in Section \ref{refCode}:
+\[\underline{\mathcal{B}}_{old} = (\mathcal{B}_{old,0}, \mathcal{B}_{old,1}, \mathcal{B}_{old,2}, \ldots, \mathcal{B}_{old,n_{old}-1})\]
+From these blockchain objects, we may compute the sample block arrival rates, $\hat{\lambda}$, at each new timestamp, and we may compare the relative error between $\hat{\lambda}$ and the target block arrival rate, $\lambda^{*}$ as a function of the latest timestamp (which may not be the top timestamp).
+
+\subsection{Big Swings in Hash Rate}\label{topHatComp}
+
+This is the fun, true test of the difficulty equations. We set a small constant $h_0 > 0$ and a large constant $h_1 > 0$. We choose a time interval $[t_1, t_2)$ on which network hash rate will blast up from $H(t) = h_0$ hashes per second to $H(t) = h_1$ hashes per second, and then back down to $H(t) = h_0$ again.
+\[H(t) = \begin{cases} h_0 & t \in [0,t_1)\\
+h_1 & t \in [t_1, t_2)\\
+h_0 & t > t_2\end{cases}\]
+
+Using this function, we can assess the rate at which sample block arrival rate approaches target block arrival rate after a large jump in network hash rate, and we can also assess how gracefully a difficulty equation can respond to the converse situation when hash rate drops. We investigate only values of $h_1, h_0$ such that $h_0/h_1 > 1/100$. This would correspond to a user who controls $99\%$ of the network turning off their equipment, and will lead to block arrival rates dropping by two orders of magnitude. Investigating smaller values of $h_0/h_1$ is unnecessary, for the consequence would be a large stall and no more blocks.
+
+We again see that our difficulty adjustment equation almost always has a smaller relative error between sample and target block arrival rates, and that these values approach each other more rapidly with our difficulty adjustment equation than with the CryptoNote reference code difficulty equation. 
+
+\subsubsection{Square Wave Hash Rate}\label{squareWaveComp}
+
+If hash rate behaves in a square wave, we may imagine this as applying a periodic forcing function to a difference equation. We investigate this scenario for several reasons. One reason is the possibility that resonance could cause a blowup in network hash rate. Indeed, if a difference equation has a natural frequency, $\omega$, and a periodic forcing function is applied with period $P$ that is an integer multiple of $\omega$, we run the risk of resonance.  Another reason to investigate periodic hash rates is because the CryptoNote reference code uses a dubious method of discarding outliers. Indeed, in the CryptoNote reference code, the top $720$ blocks are studied and the top and bottom $60$ blocks from that list are discarded. Hence, a miner could hop on the network, mine for $60$ blocks, and then hop off the network; such a miner would never personally experience an increase in difficulty as a response to their activity. This suggests some time delay properties between hash rates and difficulties which may be revealed with a periodic forcing function through the expressed phase angle.
+
+To these ends, we investigate periodic square wave functions. We choose $h_1 > h_0 > 0$ and a period, $P$. We choose some timepoint $0 \leq t^{*} < P$ to represent the time in which network activity is ``low.'' We then use a periodic extension of the function
+\[H_0(t) = \begin{cases} h_0 & t \in [0, t^*)\\
+h_1 & t \in [t^*, P)\end{cases}\]
+with period $P$ as our hash rate function. We investigate ranges of $1 > h_0/h_1 > 1/100$ as in the previous case, and we investigate several values for the period $P$ ranging from $P=1.0 s$ to $P=2m/\lambda^{*}$, which represents the amount of time it takes to receive $m$ blocks, where $m > \ell$ is the sample size used to estimate block arrival rates. If the frequency of the periodic hash rate is an integer multiple of difficulty adjustment period, or outlier discarding periods, or even block arrival rate sample size, this window should be wide enough to detect an effect.
+
+\subsection{Results}
+In response to exponential or logarithmic hash rate growth, the new difficulty adjustment equation performs uniformly better than the CryptoNote reference code; not only is relative error between sample block arrival rate and target block arrival rate usually smaller, it decays to zero more rapidly with the new difficulty equation. The CryptoNote reference code performs worse in the case of logistic growth, but sample block arrival rate is still usually kept to within a tolerance of BLAH BLAH INSERT NUMBER HERE compared to target block arrival rate.
+
+In response to the top hat function, the two difficulty equations perform nearly as well as one another, although the new difficulty equation performed moderately better in terms of responding to a large decrease in hash rate.  In response to the periodic function, we saw some interesting behaviors.
+
+The most surprising result is that the CryptoNote reference code, which has only a weak resemblance to our difficulty equation (which was derived directly from a blockchain growth model), it does a fair to good job of keeping arrival rates roughly constant in time. The primary concern about this algorithm, it turns out, is not the response to general hash rate trends. In fact, it is a far bigger problem that users may exploit the way that outliers are discarded in order to mine without seeing difficulty change due to their own activity. Nevertheless, our difficulty equation is still uniformly better than the reference code.
+
+\section{Further Questions}\label{furtherQuestions}
+
+In this section, we discuss several routes by which the above work could be expanded by interested and motivated readers. In general, it is to our benefit to produce many models and assess their precision and accuracy, and to compare and contrast multiple hypotheses to determine which model does the best job of representing the ground truth. In practice, exploring every possibility is unreasonable. We hope that others extend this work to develop yet better difficulty adjustment methods.
+
+\subsection{Time Series Models of Hash Rate}\label{timeSeries}
+In deriving our difficulty equation, we presumed hash rate of the future will match the average hash rate of the past. The holy grail of any adaptive algorithm is prediction. Time series analysis tools may be used to make statistical predictions of instantaneous hash rate estimate $\hat{H}_{n}$ based on previous observations $\hat{H}_{n-1}, \hat{H}_{n-2}, \ldots$. An ambitious user could seek a transformation of this time series that is covariance-stationary and use a SARIMA model, for example, to try to predict the next hash rate. 
+
+In this document, although we are tempted to use heavy statistical machinery to solve the problem, parsimony suggests that we seek a simpler solution before we seek a more complicated solution. Models such as SARIMA should only be used if ARIMA models are not doing particularly well, which should only be used if ARMA models are not doing very well, which should only be used if autoregressive (AR) or moving average (MA) models are insufficient. Furthermore, our difficulty adjustment algorithm will be unsupervised, whereas time series analysis problems usually require a modeler to actively tweak and play with data; time series analysis often does not perform well when fully automated and runs the risk of over-fitting. In our case, we decided upon using simple moving averages, but an efficient implementation of a fully automated SARIMA model that punishes overfitting would be a very interesting application of statistical analysis on a network.
+
+\subsection{Population Growth Models of Hash Rate}\label{popGrowth}
+
+In this document, we generated hash rate functions in a deterministic way; we picked formulae for the hash rate functions directly. We did this in order to see how difficulty equations responded to specific scenarios. However, we could generate hash rate functions stochastically using population models and various stochastic implementations of those models. 
+
+For example, we may investigate the logistic growth, which obeys the differential equation $H^{\prime} = kH(1-H/H_{max})$ where $k$ is maximum growth rate and $H_{max}$ is the carrying capacity of the network. The solution to this equation takes the form $H(t) = K/(1+Ae^{-kt})$ where $A = (K-H(0))/H(0)$, which is the form used in Section $\ref{logComp}$. However, rather than determining hash rate determinsitically from this function by plugging in the current time, we could implement a Gillespie algorithm from \cite{gillespie1977exact} to create a stochastic hash rate function whose expectation is the logistic solution above. This would allow a researcher to investigate how different assumptions about hash rate growth over time may influence blockchain growth.
+
+One may ask, ``why does the author recommend a complication of the hash rate function here, but in Section \ref{timeSeries} wishes to argue for parsimony?'' The answer to this is that in Section \ref{timeSeries}, we are discussing the way a user on the network estimates hash rate. This is the mechanism we are designing, a difficulty equation, which necessarily needs to be simple for computaitonal reasons. On the other hand, here, in Section \ref{popGrowth}, we are discussing which hash rate function we should use as input to our blockchain when testing a difficulty equation for robustness and accuracy. We are asking ``how should I test our designed mechanism?'' Testing our difficulty adjustment equation against a complicated hash rate is a stress test on the mechanism we have designed.
+
+\subsection{Graph Theoretic Modeling and Parent Coin Selection Rules}
+One other obvious route of extension is to model the cryptocurrency network itself and investigate the relationships between difficulty adjustment, parent coin selection rules, and network structure. One can representing a computer network with a weighted graph, where the weight between nodes represents a propagation delay. The choice of graph is important, for a five-node network with $1 s$ separation between each pair of connected nodes, can result in a maximum five second propagation delay (if the nodes are lined up serially), or could result in a maximum one second propagation delay (if the nodes are arranged as a pentagram in a complete graph with five nodes, $K_5$). Any graph choice must be justified; equating the average and standard deviation in propagation delay may be sufficient for some statistical purposes, but by no means capture the details of a complicated network.
+
+The presence of propagation delays will necessarily lead to competing chains, which necessitates a choice of parent coin selection rules, such as the ``longest chain'' rule first proposed by Nakamoto in \cite{nakamoto2008bitcoin}, and the so-called GHOST rule, first proposed by Sompolinsky in \cite{sompolinsky2013accelerating}. The GHOST rule selects the parent coin by seeking the heaviest observed sub-tree at each fork in the blocktree, and may exhibit certain advantages over the Nakamoto rule. It is possible that the choice of graph structure, parent coin selection rule, and difficulty adjustment formula are deeply inter-related. Given a particular graph structure and difficulty adjustment formula, a cryptocurrency network may prefer the GHOST rule over the Nakamoto rule, or vice versa. Given a particular graph structure and a particular parent coin selection rule, some difficulty adjustment formulae may make no sense, and some may perform very well. Finally, given a particular parent coin selection rule and difficulty adjustment formula, there may be a graph structure for which these choices perform optimally. These questions could lead to very efficient creations.
+
+\subsection{Stalling in the Event of Sudden Hash Rate Drop}
+
+The problem with stalling cryptocurrency networks in the event of a large hashrate drop can be mitigated in at least one of two ways.  The reason the network may stall is because the current estimate of difficulty remains unchanged until a block is added. The usual maximum likelihood estimate (MLE) of a Poisson rate doesn't use all the data we have available. The first way we recommend mitigating a stalling event involves exploiting the fact that we know how long it has been since the last block arrival. This is additional information we have available that is not yet being used. The second way we recommend mitigating a stalling event involves contact with a third party to obtain information about their own, distinct blockchain, whcih is also additional information we have available. Either or both of these avenues could present interesting lines of inquiry.
+
+Our first avenue is to incorporate the right-censored maximum likelihood estimate of a Poisson process rate. Indeed, in our derivation in Section \ref{myCode}, we assume network hash rate remains static and match our current estimate until more information arrives in the form of the next block to be added. However, at any given time, we actually have more information avaiable; that is to say, we know that some block \emph{will be arriving} at some point \emph{after the current time}. Assume that all users are reporting their timestamps honestly. Rather than declaring a blockchain to be a sequence of timestamp-difficulty ordered pairs together with a future difficulty of the form 
+\[(t_0, d_0), (t_1, d_1), \ldots, (t_{n-1}, d_{n-1}), d_n\] 
+and with no \emph{a priori} concept of time attached to the blockchain object (outside of it's timestamps), we could alternatively define a blockchain in the following way. Presuming some height $n$, we may define a blockchain to consist of a sequence of timestamp-difficulty ordered pairs together with a time, $t$, and a random variable, $\tau$, of the form
+\[(t_0, d_0), (t_1, d_1), \ldots, (t_{n-1}, d_{n-1}), (\tau, d_n)\]
+where the only information we have about $\tau$ is the distributional fact that $\tau > t$ with probability $1$. If we wish to incorporate data about timestamp manipulation, this may be stored in any distributional assumptions we place on $\tau$.
+
+The way we could implement this is to take our list of known timestamps, $t_0, t_1, \ldots, t_{n-1}$, include the current time, $t^*$, and sort the results into their ordered statistics, $t_{(0)} < t_{(1)} < \ldots < t_{(n)}$. We then simply use the MLE as usual from here. If all timestamps are being reported honestly on a network with no propagation delays, then we automatically have the ordering $t_0 < t_1 < t_2 < \cdots < t_{n-1} < t^*$. However, since timestamps may occur out of order, it is possible that $t_{(n-i)} < t^{*} < t_{(n)}$ for some $i=1, 2, \ldots$.
+
+One problem with this approach is that now every user may use their own time, $t^*$, to determine current difficulty. As a consequence, two users with the same copy of the blockchain may compute different values for ``next difficulty'' and there is no way to verify which, if any, are correct. Users with the same data will only compute the same difficulty if they also have synchronized clocks. To worsen matters, it seems this approach leaves open the possibility of a vulnerability. It will be in the users' best interest to obtain a smaller estimate of block arrival rates, if possible, in order to obtain a small estimate of hash rate, and therefore a smaller difficulty. Hence, using the right-censored maximum likelihood approach, each user would set their local clock very far ahead in the future to take advantage of low-difficulty mining. This would cause an upward drift in timestamps compared to the true time. This upward drift may be bounded and slow, in which case the risk of blocks being rejected by the rest of the network will prevent the behavior from getting out of control. On the other hand, the upward drift may be very fast or unbounded. Timestamps on the top of the blockchain will eventually be so far out of whack with reality that no honest miners will get their blocks accepted by relaying nodes.
+
+Our second avenue is to tie the estimate of block arrival rate on the network with the block arrival rates on other networks. For example, if a user has observed $n$ Bitcoin blocks validated since the last Monero block has been validated, she may reasonably presume that either (a) she has a connectivity issue with the Monero network, (b) the Monero network hash rate has suddenly dropped by a factor of $10n$, or (c) the Bitcoin network hash rate has suddenly increased by a factor of $10n$.  Due to the size of the Bitcoin network, option (c) is almost certainly false in general. This seems like an elegant solution, but any time we use a third party, verifiability and security become issues. Working out the details of either of these proposals could lead to an enormous improvement to the resilience of any cryptocurrency network.
 
 
-However, since our estimator is based only on $N$, our sample of the top few blocks, and the timestamps $t_{(H)}$ and $t_{(H-N+1)}$, we are using the timestamps from precisely two blocks to compute our spread of time. This leaves open the possibility that an attacker controls one or both of these timestamps, and hence an attacker partially controls the next block's difficulty. For example, a user with about $20\%$ hashing power has a $36\%$ chance of controlling at least one of these timestamps. This disproportionate control over timestamps is troubling. If the underlying processes are true Poisson processes, then we conclude manipulation of timestamps has not occurred and this issue is resolved.
+\subsection{Damping}
 
-\subsection{Testing for Poisson-icity}
-We learn the lessons of the previous section; we wish to ignore an estimate of hash rate under the condition that we believe it is untrustworthy.  Poisson processes with arrivals occurring at times $\left\{T_1, T_2, \ldots, T_{N+1}\right\}$ have inter-arrival times $S_{1} = T_2 - T_1, S_2 = T_3 - T_2, \ldots$. Furthermore, the maximum likelihood estimate matches the UMVUE of the underlying rate parameter $\lambda$, as previously mentioned, and is $\hat{\lambda} = N/(T_{(N)} - T_{(1)})$. This is, in fact, $1/\overline{S}$ where $\overline{S}$ is the sample mean of the inter-arrival times.
+We may wish to damp the amount of change we apply to our difficulty in the case that we believe that timestamps have recently been manipulated. To this end, recall the convenient property of Poisson processes with rate $\lambda$: the mean inter-arrival time is $1/\lambda$ and the variance in the inter-arrival times is $1/\lambda^2$. Hence, if mean inter-arrival time and standard deviation of inter-arrival times are very different, this suggests that timestamps have been manipulated and the process underlying the timestamp creation is not a homogeneous Poisson process. 
 
-A convenient property of Poisson processes with rate $\lambda$ is that the mean inter-arrival time is $1/\lambda$ and the variance in the inter-arrival times is $1/\lambda^2$. Hence, testing the squared mean against the variance is a test for whether a process is Poisson or not. Given a sequence of inter-arrival times, say $\left\{S_1, S_2, \ldots, S_n\right\}$, a test of whether these inter-arrival times came from a genuine Poisson process is to compare the squared sample mean $\overline{S}^2 = \left(n^{-1}\sum_{i=1}^{n} S_i\right)^2$ with the (unbiased) sample variance $\text{Var}(S) = (n-1)^{-1}\sum_{i=1}^{n} (S_i - \overline{S})^2$. Indeed, taking the whole Monero blockchain as a sample, we observe about a $22\%$ difference between mean and the standard deviation! This suggests that, throughout the history of Monero, quite a bit of timestamp manipulation has occurred, although at this point it is not clear to the Monero Research Lab how to quantify the amount or degree of manipulation.
+Hence, testing the squared mean against the variance is a test for whether a process is Poisson or not. Given a sequence of inter-arrival times, say $\left\{S_1, S_2, \ldots, S_n\right\}$, a test of whether these inter-arrival times came from a genuine Poisson process is to compare the squared sample mean $\overline{S}^2 = \left(n^{-1}\sum_{i=1}^{n} S_i\right)^2$ with the (unbiased) sample variance $\text{Var}(S) = (n-1)^{-1}\sum_{i=1}^{n} (S_i - \overline{S})^2$. Indeed, taking the whole Monero blockchain as a sample, we observe about a $22\%$ difference between mean and the standard deviation! This suggests that, throughout the history of Monero, quite a bit of timestamp manipulation has occurred, although at this point it is not clear to the Monero Research Lab how to quantify the amount or degree of manipulation.
 
-When the squared mean and variance are vastly different, we distrust the notion that the underlying process is Poisson.
+When the squared mean and variance are vastly different, we distrust the notion that the underlying process is Poisson. Furthermore, if this lack of Poisson-icity is due to an attacker manipulating timestamps, then the mean inter-arrival time, which only utilizes $T_{(1)}$ and $T_{(N)}$, the first and last times of arrival, is particularly vulnerable to a \textit{single} attacker changing their timestamp. On the other hand, the sample standard deviation utilizes all inter-arrival times equally in it's computation. Hence, rather than using $\hat{\lambda} = 1/\overline{S}$, another approach is to use $\hat{\lambda} = 1/\sqrt{\text{Var}(S)}$.
 
-
-\begin{comment} Furthermore, if this lack of Poisson-icity is due to an attacker manipulating timestamps, the mean inter-arrival time, which only utilizes $T_{(1)}$ and $T_{(N)}$, the first and last times of arrival, is particularly vulnerable to a \textit{single} attacker changing their timestamps. On the other hand, the sample standard deviation utilizes all inter-arrival times equally in it's computation. Hence, rather than using $\hat{\lambda} = 1/\overline{S}$, we shall use $\hat{\lambda} = 1/\sqrt{\text{Var}(S)}$. Furthermore, if variance and squared mean of inter-arrival times are drastically different, then we will choose to not trust our data at all and keep difficulty relatively constant. If variance spikes far above the squared mean, or plummets, this suggests a high degree of manipulation of timestamps in our recent data.
-\end{comment}
-
-Hence, rather than modifying $\mathcal{D}_{\text{old}}$ by a multiplicative factor $\frac{\hat{\lambda}}{\lambda}$, we will consider an equivalent additive change in difficulty, $\Delta \mathcal{D} = (\frac{\hat{\lambda}}{\lambda} - 1)\mathcal{D}_{\text{old}}$. Furthermore, we will scale this change in difficulty by a factor $\alpha$ which will depend on both the squared sample mean and the sample variance of inter-arrival times. If squared sample mean and sample variance are approximately equal, we will set $\alpha \approx 1$, and if sample meean and sample standard deviation are different, $\alpha \to 0$. To this end, let $\overline{S}^2$ denote the squared sample mean of inter-arrival times and denote $\text{Var}(S)$ denote the sample standard deviation. Many choices of $\alpha$ are reasonable. For example, we may choose
+Hence, rather than computing $d_{n+1}$ by a multiplicative factor $\overline{H}_n/\overline{H}_{n-1}$, consider an equivalent additive change in difficulty, $d_{n+1} - d_n = (\frac{\overline{H}_n - \overline{H}_{n-1}}{\overline{H}_{n-1}})d_n$. The idea of this approach is to scale  this change in difficulty by a factor $\alpha$ which will depend on both the squared sample mean and the sample variance of inter-arrival times. If squared sample mean and sample variance are approximately equal, we will set $\alpha \approx 1$, and if sample meean and sample standard deviation are different, $\alpha \to 0$. To this end, let $\overline{S}^2$ denote the squared sample mean of inter-arrival times and denote $\text{Var}(S)$ denote the sample standard deviation. Many choices of $\alpha$ are reasonable. For example, we may choose
 \[\alpha = \text{Exp}\left[-\left|\overline{S}^2-\text{Var}(S)\right|/\overline{S}^2\right]\]
 Notice that, since $\overline{S}^2$ is never zero, since all blocks must arrive with at least one second separating them, this expression is well defined. Also notice that $\alpha$ has the property that when $\overline{S}^2 = \text{Var}(S)$, we have that $\alpha = 1$, and whenever $\left|\overline{S}^2 - \text{Var}(S)\right|/\overline{S}^2 \to \infty$, we have that $\alpha \to 0$. The scaling in the denominator of the exponent provides a probabilistic guarantee that the momentum term $\alpha$ stays reasonable close to $1$ under manipulation-free circumstances.
 
-We can consider $\alpha$ a momentum term, where a close match between a true Poisson process and the observed inter-arrival times yields almost no momentum, or we can consider $\alpha$ a trust value, where a close match yields a high trust level. Using this interpretation, a $22\%$ variance between squared sample mean and sample variance would yield an $80.25\%$ trust rating.  On the other hand, a mere five percent difference between squared sample mean, $\overline{S}^2$, and sample variance, $\text{Var}(S)$, will provide a trust value of $0.95122$, or rather, about a $95\%$ trust rating. An unreasonably large difference between squared sample mean and sample variance, say $50\%$, would yield a trust value of $0.5 \leq -\ln(\alpha)$, or rather $\alpha \geq 0.606$ or a $60.6\%$ trust rating. As we have previously observed, there is, historically, a $22\%$ difference between squared sample mean, $\overline{S}^2$, and sample variance, $\text{Var}(S)$. This provides a good reference point for examining this momentum term. A $22\%$ difference would provide $-\ln(\alpha) \leq 0.22$, yielding $0.8025 \leq \alpha$. That is to say, our momentum term would cause our difficulty to adjust at around $80\%$ it's maximal rate.
-
-
-
-\subsection{Our Recommendation}
-
-We recommend that the difficulty of the next block be computed by setting
-\[\mathcal{D}_{H+1} = \overline{\mathcal{D}} + \Delta \mathcal{D}\]
-where $\overline{\mathcal{D}}$ denotes the sample mean of difficulty over the previous $N$ blocks (sticking with tradition, perhaps $N=600$) and where we use
-\begin{align}
-\Delta \mathcal{D} =& \alpha (\frac{\hat{\lambda}}{\lambda} -1)\overline{\mathcal{D}}\\
-\hat{\lambda}=& N\left(\sum_{i=1}^{n} S_i\right)^{-1} = N(T_{(n)} - T_{(1)})^{-1} = 1/\overline{S}\\
-\alpha =& \text{Exp}\left[-\left|\overline{S}^2-\text{Var}(S)\right|/\overline{S}^2 \right]
-\end{align}
-
-Recall that $\tau = 1/\lambda$, and so we may write $\Delta \mathcal{D} = \alpha (\tau \hat{\lambda} -1)\overline{\mathcal{D}}$ for some convenience. Notice that our adjustment method is protocol agnostic. This could be implemented in Bitcoin, Monero, or any other cryptocurrency. 
-
-We make no claims that our approach is the best possible one; although, any estimate of hash rate should (almost certainly) use the UMVUE of Poisson arrival rates. However, this approach brings several benefits compared to the reference CryptoNote difficulty adjustment. Consider the following.
-
-First, equilibrium occurs if and only if $\Delta \mathcal{D} = 0$, so difficulty is held approximately constant whenever $\Delta \mathcal{D} \approx 0$. This occurs whenever $\hat{\lambda} \approx \lambda$, which occurs if and only if $T_{(H+1)} - T_{(H-N+1)} \approx N\tau$, where $\tau$ is our target block arrival time and $N$ is our sample size. If the process is genuinely Poisson and $N$ is sufficiently large, this is true with high probability. If all blocks have arrived precisely on target, this is also true, but our sample variance is zero.  Regardless of block arrival rate, if the timestamps of blocks $\mathcal{B}_{H-1}, \ldots, \mathcal{B}_{H-N+2}$ stay bounded within the interval $\left[T_{(H-N+1)}, T_{(H)}\right]$, then the interior timestamps may vary in any way whatsoever without effecting the next block difficulty. Finally, we have the property that, as squared sample mean, $\overline{S}^2$, and squared sample variance, $\text{Var}(S)$, diverge from one another, $\alpha \to 0$. In this case, the process is wildly non-Poisson and so we don't change difficulty much.
-
-
-
-
-
-
-
-
-
-
-
-
-\section{Simulations}\label{simulations}
-
-We use Monte Carlo simulations to construct simulated blockchains given a known network hash rate. A simulated blockchain of height $H$ takes the form
-\[\texttt{blockChain}:=\left\{(t_{0}, \mathcal{D}_{0}), (t_{1}, \mathcal{D}_{1}), \ldots, (t_{H-1}, \mathcal{D}_{H-1}), (t_{H}, \mathcal{D}_{H})\right\}\]
-where each entry in each ordered pair of this array is an integer and we initialize all simulations with the initial condition $(t_0, \mathcal{D}_0) = (0, 1)$. The timepoints $t_i$ denote seconds that have elapsed since the genesis block and the difficulties $\mathcal{D}_i$ denote their associated difficulties. 
-
-Note that the Monero blockchain (as well as all other CryptoNote blockchains and most other cryptocurrencies) uses Epoch time and set the timestamp of the genesis block to the coordinated universal time (UTC), Thursday, 1 January 1970. Hence, for most cryptocurrencies, the ostensible timespan between the genesis block and the block of height one is over forty years, measured in seconds. Rather than specify a special rule for the inter-arrival time between block one and block zero to account for this, we simply ignore this convention; our simulations, which have a target block inter-arrival time of $60$ seconds, will not have a forty year gap between the first two blocks.
-
-Our simulations are designed to construct a local copy of the blockchain by an honest observer using a local clock, $t$, a known network hash rate, $N(t)$, and the assumption that blocks arrive in a non-homogeneous Poisson process dependent both on the hash rate and the difficulty of the top block, $\mathcal{D}_H$, which is deterministically computed from the blockchain. Our simulations depend critically on the memoryless property of Poisson processes. We first generate a known (integer) network hash rate, which is a function of time, and initialize time $t=0.0$. The ``true'' rate of arrival of the next block, in terms of the underlying Poisson process, is precisely $\lambda = N(t)/\mathcal{D}_H$ using the model presented previously in Section \ref{diffIsHash}. 
-
-We compute the next timepoint that the hash rate changes, $t_{\texttt{hash}}$. We generate a random inter-arrival time of the next possible block, $T_{\texttt{poss}}$, from an exponential distribution with ``true'' rate parameter $\lambda$. We check whether the next possible block arrives before or after the hash rate changes, i.e.\ we check whether $t + T_{\texttt{poss}} < t_{\texttt{hash}}$. 
-
-If the next block arrives before hash rate changes, that is to say, if $t + T_{\texttt{poss}} < t_{\texttt{hash}}$, then we choose a timepoint, $\hat{t}$, to be associated with that block. In general, $\hat{t}$ is a function of $t + T_{\texttt{poss}}$, as it is a timepoint based on the ``true'' time of arrival of the next block. Examples are easy to generate. If this function is the identity function, i.e.\ if $\hat{t} = t + T_{\texttt{poss}}$, then the node that discovered this block has reported its timestamp honestly. If $\hat{t} = t + T_{\texttt{poss}} + 18$, then the node that discovered this block has reported its timestamp $18$ seconds in the future; a node with a policy of reporting all timestamps $18$ seconds in the future almost certainly has an incorrect system time, and is ostensibly an honest user.  From $\hat{t}$ we compute the next difficulty, $\mathcal{D}_{\texttt{next}}$ to be associated with that block, and we append the ordered pair $(\hat{t},\mathcal{D})_{\texttt{next}})$ to $\texttt{blockChain}$. We then re-set our clock to $t = t + T_{\texttt{poss}}$ to reflect that time has rolled forward, and we begin again by generating a random inter-arrival time of the next possible block, $T_{\texttt{poss}}$.
-
-On the other hand, if the next block arrives after the hash rate changes, we ignore that next possible block arrival; we may do so due to the memoryless property of Poisson processes. We instead re-set our clock to $t = t_{\texttt{hash}}$, rollover our network hash rate to $N(t_{\texttt{hash}})$, recompute the ``true'' rate parameter $\lambda$, and re-compute the next timepoint that the hash rate changes, $t_{\texttt{hash}}$. We then begin again by generating another random inter-arrival time of the next possible block, $T_{\texttt{poss}}$.
-
-We proceed as described until $t \geq \texttt{MaxTime}$ and compare the sequence of ordered pairs represented by $\texttt{blockChain}$ against the known network hash rate, $N(t)$. A good difficulty algorithm will exhibit a close match between the ordered pairs $(t_i, \mathcal{D}_i)$ and $(t_i, N(t_i))$. Of course, this cannot be guaranteed by any algorithm since the timepoint in the ordered pair $(t_i, \mathcal{D}_i)$ is a possibly false timestamp, as previously described.
-
-\subsection{Generating Network Hash Rate}\label{NetHashRate}
-
-We generate six regimes of network hash rate behavior: both deterministic and probabilistic realizations of linear, exponential, and sigmoidal trends. We choose linear trends for simplicity, exponential trends to mimic simple population growth models, and sigmoidal trends to mimic more complicated population growth models that incorporate population carrying capacity. In all cases, we set $\texttt{maxTime} = 3\times 10^7$ to simulate approximately one year of data.
-
-In each deterministic case, we chose to run a one year simulation beginning with a one $kH/s$ network and terminating with a $100 GH/s$ network. The linear case takes the form $N(t)= mt + b$, the exponential case takes the form $N(t) = A e^{rt}$, and the sigmoidal case takes the form $N\left(K - N\right) = Ae^{rt}$ (this latter form is the solution of the differential equation modeling a population with carrying capacity, $N^{\prime} = rN(1-N/K)$). Presuming the sigmoidal case is approaching the carrying capacity of the Bitcoin network, currently in the $PH/s$ range, $K=10^15$ here. In order to meet our target of an initial condition of one $kH/s$ and ending after one year with $100 GH/s$, we have
-\begin{align*}
-\text{Linear:} m =& blah & b &= blah\\
-\text{Exp:} A=& blah & r =& blah\\
-\text{Sigmoid:} A=& blah & r =& blah
-\end{align*}
-
-
-
-
-
-
-
-
-
-
-
+We can consider $\alpha$ a momentum term, where a close match between a true Poisson process and the observed inter-arrival times yields almost no momentum, or we can consider $\alpha$ a trust value, where a close match yields a high trust level. Using this interpretation, a $22\%$ variance between squared sample mean and sample variance would yield an $80.25\%$ trust rating.  On the other hand, a mere five percent difference between squared sample mean, $\overline{S}^2$, and sample variance, $\text{Var}(S)$, will provide a trust value of $0.95122$, or rather, about a $95\%$ trust rating. An unreasonably large difference between squared sample mean and sample variance, say $50\%$, would yield a trust value of $0.5 \leq -\ln(\alpha)$, or rather $\alpha \geq 0.606$ or a $60.6\%$ trust rating. 
 
+As we have previously observed, there is, historically, a $22\%$ difference between squared sample mean, $\overline{S}^2$, and sample variance, $\text{Var}(S)$. This provides a good reference point for examining this momentum term. A $22\%$ difference would provide $-\ln(\alpha) \leq 0.22$, yielding $0.8025 \leq \alpha$. That is to say, our momentum term would cause our difficulty to adjust at around $80\%$ it's maximal rate.
 
+%%%%%%%%%%%%%%%%%%%%%%%%
 
 
 \newpage
@@ -985,6 +971,11 @@ for i in range( len( timestamps ) ):
 \end{lstlisting}
 
 
+\chapter{New Difficulty Reference Code}\label{referenceCode}
+
+\begin{lstlisting}[language=Python,basicstyle=\small,breaklines=true]
+
+\end{lstlisting}
 
 \end{appendices}