7.8 KiB
| layout | title | summary | tags | author | forum | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| post | Dev Diaries for the Week of 2015-03-02 | BlockchainDB progress update, performance considerations, and rationale; RPC->0MQ communication change progress update |
|
Riccardo Spagni (fluffypony) | https://forum.getmonero.org/1/news-announcements-and-editorials/195/monday-monero-missives-24-march-2nd-2015 |
To download the podcast directly please use this link to the MP3.
A brief summary of the points discussed follows, and a full transcription of the podcast is below (currently being completed, and it will be updated as it progresses).
In this week's podcast we restart dev diaries, and focus on two things:
-
Updates on blockchainDB, including rationale, performance considerations, and future steps
-
Moving away from the old RPC system for talking to the daemon, to a 0MQ-based IPC system
Technical note: in order to make this Missive a little more accessible, given its technical nature, we have taken some liberties in using the term "RPC" to refer to the JSON RPC API 2.0 over HTTP system used when currently communicating with the daemon, and "IPC" to refer to a complete replacement of that subsystem with 0MQ subsystem based on their Router/Dealer pattern, using the zmq_tcp transport for compatability.
External Projects has moved to be covered next week. Until then!
Podcast Transcription
Riccardo "fluffypony" Spagni
Hello! And welcome to our second Monero Missives podcast. I'm Riccardo, fluffypony.
Gingeropolous
And I'm Gingeropolous!
Riccardo "fluffypony" Spagni
Over the past week we've had some feedback on the Missive Podcast, we've had some feedback on the website. We're not going to talk much about the website - or at all. Instead, what we wanted to do, is get back to Dev Diaries, because obviously with the raw amount of content in last week's Missive we weren't able to talk about Dev Diaries at all.
So Dev Diaries is a Missive feature - a long-running Missive feature - where we discuss the more technical commits or things that have happened in the past week. First and foremost on everyone's mind: the move to BlockchainDB is something that seems to be a little bit misunderstood, especially in terms of where things are right now.
Just to clarify: the vast majority of the work on BlockchainDB has been completed. In fact, it was completed towards the end of last year. Plenty of us have been running BlockchainDB, or LMDB, instances of Monero for the almost two months now.
Gingeropolous
I can confirm that. I've been running the database version for a while now - I actually have a 2gb Linux box, it's just this crappy little old computer that I've managed to run the database build on for a while now. So it definitely works with low memory.
Riccardo "fluffypony" Spagni
No exactly, and it does compile on the various platforms that we support. So there's no platform with platform support, there's no holdup with anything, the main issue we've had over the past month or so has got to do with two things.
Firstly, with sync speed, so in other words...if you're catching up, maybe your @node has been down for a couple of days, or you're catching up from scratch, from the Genesis block. And the second issue that we've had to deal with is conversion. So you've got the current, in-memory @blockchain saved on disk, and you upgrade to the database version and you want to convert what you've got on disk - you don't need to sync up from scratch when you're pretty much caught up with the network.
So this really has to do with the speed at which we write to the embedded database. It's not really a performance issue in terms of the database that we're using, but it's really a product of the fact that the entire @blockchain, for example, needs to be kept in RAM and then you have to have the database version whilst it's dumping. Or, when you're syncing up from scratch, there's all these extra components in the database, like indexes, that need to get written as it's busy dumping @blocks and @transactions down as it verifies them.
It has been quite challenging to figure out ways of improving this, and the bulk of the really clever thinking and clever work has been done by warptangent, and we've had a lot of really good input from the LMDB author, Howard Chu. Without input from him I think it would've been a lot more challenging, a lot more difficult, to figure out how to take advantage of LMDB's speed.
But at the moment it's really coming along nicely, and I reckon we are no more than a couple of weeks away from a point at which we can merge it into upstream and it can be pulled out into a release, into 0.8.8.7, and then we'll be able to at least get a feel on a wider basis as to how things are performing.
Gingeropolous
So regarding the database, I've been watching the forums, and it's been mentioned that other CryptoNote coins have utilised other methods to get around the whole @blockchain being in memory. Is there a reason that we didn't choose to just clone those methods, or "why use LMDB?" I guess is the main question...why sink all this time into making something new vs taking something that works.
Riccardo "fluffypony" Spagni
That's a good question. So one of the things that we...one of the challenges that we faced is, when we really started going down the BlockchainDB route, there was some code added to the CryptoNote reference code, I believe Bytecoin added it initially. It was a version of the swapping between disk and memory that your operating system already does.
So just to explain what happens: if you've got a 1gb file, and a program needs to access that 1gb file, but it only needs to access a small portion in the middle...there's no point, generally, in loading the entire thing into memory. But because most programs have to deal with files of all sorts of size and shape and colour they can't say "oh, I only need this portion in the middle" preemptively. So what they'll do is they'll request the entire file to be loaded into memory.
But because the operating system knows that it only needs this portion at the beginning, and then later on this portion in the middle when it's requested, it won't actually load the whole thing into memory, but it'll page it into a temporary storage area on disk and then load portions into memory. So that's effectively what some of the other CryptoNote coins have used, this swapping.
We felt that there was limited value in doing the swapping that the operating system is doing already...in a manner that may not be as efficient as the operating system. And that's not a knock on the piece of code that's been written, we just felt that it was better to abstract the @blockchain class out, and to create something that is more performant and more extensible.
Obviously we had to balance that with the fact that the @blockchain was growing, and it was growing to a point where it was excessively big for many, and so we knew that there was this limited time that we had where we either had to get the database done, or we would have to fall back to some other solution. And we're still within that timeframe, and the database, as you know and as we've discussed, is pretty much done.
So using that other fake-swap system wouldn't have had value, and wouldn't have solved the problem, and at worst it would've made us lazy to finish the database because there would've been no pressure.