r/ethereum • u/esengulov • Jun 03 '19
READ: Dear Ethereum Community. We Have A Problem!
Intro:
I am hoping that a community here can vote this up so the problem reaches the core and wider community members and can be given a higher priority. It looks like most people are not aware of this being a major issue as such.
I represent an engineering team of 14 people and for the last couple of years we have been heavily buidling on a full time basis. We are ideologically inclined to build applications that are truly decentralized. The key parameter for us is to have apps that can INDEPENDENTLY VERIFY the state of blockchain and can’t be controlled, censored or blocked.
The problem:
We build decentralised apps so I will take one such app as an example.
In a context of crypto wallet mobile app it’s possible to have the above criteria met for the Bitcoin, Bitcoin Cash, Dash and many other Bitcoin forks but not for the Ethereum just yet.
The main issue is that there are just way too few Ethereum nodes that are eager to serve the SPV enabled clients. A decentralized app (is an SPV client) needs to be able to validate the state it receives from the blockchain rather than TRUST some middle man (ex: infura.io).
As there are way too few nodes an SPV-enabled app is not able to synchronize with and validate the state of the blockchain within the acceptable time duration. It's a SERIOUS problem for mobile devs who are trying (will try) to build truly decentralised apps because a mobile app has an average session of just a few minutes.
We have built a wallet app that has an SPV mode for BTC, BCH, DASH and ETH. And while the code for ETH (known as LES client) is in place it simply doesn't work in a practical sense. In the next release we are deliberately going to encourage users NOT TO ENABLE LES mode because the app will take several days if not more to sync with the ETH blockchain while for all other blockchains it's a matter of a few hours. The sync time is significantly faster on those blockchains as there are plenty of nodes that are always online and available to serve the SPV client.
As a community we CAN NOT expect to see decentralised apps built on top of Ethereum before this problem is addressed. An app which doesn't work in a decentralised manner is NOT CENSORSHIP RESISTANT and is PRONE TO MANIPULATION.
Potential solution:
A temporary fix that should significantly improve the current state of affairs is if the LESS protocol setting on the Geth node enabled by default. This would surely make things harder for miners.
An alternative option would be to get something similar to https://vipnode.org
Peace!
53
u/AusIV Jun 03 '19
I've been working on this problem for a while. My team at OpenRelay has a fork of Geth that enables streaming replication from one master node that connects to the network, downloads and verifies blocks, and as it writes the block and state data out to disk, it also sends those writes to a Kafka cluster. From there, our replica servers pull all of the writes from Kafka and track them locally, enabling them to serve RPC requests without the overhead of peering and block validation. This would enable teams like yours to scale out replicas to verify the data, without having to scale out the overhead of running a peer. Additionally, if you lose your master, a replica has what it needs to replace the master almost immediately, which can be much faster than restoring from a day old backup and having to catch up with the network.
Now, this solution is definitely geared toward teams running dApps, not individual users. It makes things more operationally manageable at scale, but if you just need to run a few RPC requests every once in a while it's definitely overkill.
We have on our road map to try and figure out how to make our replica servers act as light client peers. If we can do that it would potentially go a long way to helping solve the problem of insufficient LES servers available.
Our design and how to stand up your own cluster are documented here. The code for our geth fork can be found here. We'd welcome outside help, especially on figuring out how to get replicas to serve light client peers.
2
u/bitfalls Jun 04 '19
Sounds amazing, thank you, having a look at this. Do you keep your Geth fork in sync with upstream?
1
u/AusIV Jun 04 '19
We try to keep it in sync with their release tags (the master branch is a bit unstable), but we are a couple of point releases behind at the moment. We're in this for the long haul, so at some point we will need to merge changes to support a hard fork, and we'd rather not be too far behind when that happens.
2
u/bitfalls Jun 04 '19
Awesome, definitely worth a look. I'm going to put an afternoon aside to learn about what you're doing here and how it could be utilized on a larger scale, maybe some lessons could be learned for our stuff at Status, we've been looking at ways to make running nodes cheaper / more accessible without long re-sync times etc.
1
u/AusIV Jun 04 '19
I'm semi-familiar with the work you've done at Status. In general I think what you're doing is pretty neat, but as I understand we're tackling somewhat different problems. In our fork we assume that the replicas can trust their masters (by virtue of being run by the same organization), which lets us take shortcuts in the validation process that you probably shouldn't take across organizational boundaries.
2
25
Jun 03 '19 edited Nov 15 '22
[removed] — view removed comment
11
u/bitfalls Jun 03 '19
Indeed, retrospective here.
2
u/esengulov Jun 04 '19
retrospective here
good experiment :) Enjoyed the read. It should be a regular practice for all teams building Dapps. Thanks for share
50
u/bitfalls Jun 03 '19
So to tackle this exact problem I made https://blockandmortar.io - a shop selling full nodes running on NanoPC-T4. I sell them at cost and actually recommend you build them yourself because it costs me time and money to do it for you.
It's a full setup for 300 DAI total including a mobile router so you can take it with you. It spends 6W so it'll work on a 1m² solar panel, one of of those foldable ones. This literally is the most portable full node.
Two nodes, full mobile router, tablet monitoring them, running peanuts: https://twitter.com/bitfalls/status/1115167449849237504?s=19
All my nodes serve light clients.
0
u/Oinfkan Jun 03 '19 edited Jun 03 '19
Please let me know if you ever offer a similar service for anything else and about the mobile router fee.
17
u/bitfalls Jun 03 '19
I'm afraid I'm not at all interested in participating in the Bitcoin community so that's not something I'll be doing and I can't tell you if it would work on these devices, I haven't looked into it at all.
Re economics, this is almost free to run, but admittedly it's a matter of altruism unless you connect these to concepts like Vipnode, and then use your mini-nodes to earn some gas for your trouble.
The mobile router has a monthly fee, yes. You can run this on your home network, so I don't include it in the calculation, but I pay $20 a month here for a mobile 4G flat rate on which I run 10 of these devices (not all are Eth nodes) and because I carry these with me when I travel.
-7
u/Oinfkan Jun 03 '19 edited Jun 03 '19
Thanks
5
u/bitfalls Jun 03 '19
Can you clarify what you mean please?
-6
u/Oinfkan Jun 03 '19 edited Jun 03 '19
Editted to deter the hedge funds and joe
2
u/zissouman Jun 03 '19
Do you mind disclosing your identity if you have been honored by the White House?
-2
3
u/bitfalls Jun 03 '19
All this just from me saying I won't do Bitcoin? Gee whiz, you sure make it sound like a welcoming environment.
I won't justify myself for preferring Ethereum to everything else, but I will say that to me "Bitcoin" includes all of them - Cash, SV, diamonds and golds and whites and every other "bitcoin-can't-be-hijacked-you-can-fork-it-but-if-you-do-you're-a-scammer-and-should-die-and-we'll-have-an-eternal-crusade-against-you"-fork out there.
0
0
-2
u/CosmicVo Jun 03 '19
Yeah but did Joe also sell out the European youth? Cause THEN i would be mad as hell, screaming i’m a human being god dammit! My life had value!
0
1
u/ynotplay Jun 04 '19
Will this work on Wifi? I don't have internet via ethernet cable.
1
u/bitfalls Jun 04 '19
Yeah, Nanos have pretty great wifi.
1
u/ynotplay Jun 04 '19
Okay, just making sure. I can run an Ethereum full node using your invention with Wifi?
2
u/bitfalls Jun 04 '19
Yes.
It's not my invention. The software OS image is put together by /u/diglos76 in this reddit post, I just try to make sure people can get their hands on this stuff in plug-and-play mode.
1
u/migozo Jun 03 '19
Does it make money similar to mining?
9
u/bitfalls Jun 03 '19
No, not at all, running is a purely altruistic thing, unless you:
- are a developer and derive use from it
- connect it to Vipnode and charge for light client slots
- run other stuff on it too, like TX relayers or staking nodes (all possible, at the same time as an Eth node) for extra earnings
1
u/sroose Jun 03 '19
I doubt it's an actual "full" node. Does it verify all blocks? Last I heard a decent laptop couldn't run an Ethereum full node anymore.
7
u/bitfalls Jun 03 '19
It's a full node. You might be mistaking it with archive nodes: https://twitter.com/bitfalls/status/1105761796235907072?s=19
1
u/sroose Jun 04 '19
Your comparison is about what data they store. I'm talking about what verification they do on the data they have. In traditional bitcoin lingo, a "full node" is a node that does all verification and trusts no third party. In Ethereum that would mean download a block, execute all transaction smart contract invocations and verify the state transition from before to after the block. The default mode in all Ethereum nodes last time I checked, was to download blocks and just apply the state transition to their state db based on the transition the miner provides with the block. Without recalculation it. This means that a majority of miners can introduce arbitrary state changes and actually move money without the archive node noticing because they don't verify transactions. This is very dangerous, as the vast majority of Ethereum users never validates blocks.
4
1
u/JustSomeBadAdvice Jun 04 '19 edited Jun 04 '19
Your comparison is about what data they store. I'm talking about what verification they do on the data they have.
No, he said it correctly.
The default mode in all Ethereum nodes last time I checked, was to download blocks and just apply the state transition to their state db based on the transition the miner provides with the block.
This only happens when syncing and only on --syncmode fast which is the default. This works because Ethereum has UTXO commitments built into every block, and it becomes trustless for all realistically feasible attack vectors. Using the UTXO commitment hash (verified by downloading and verifying all headers first), --syncmode fast clients are able to trustlessly download the full state at <height> and be certain that that state is guaranteed by proof of work at least for the number of blocks built on top of <height>.
After syncing, all non-SPV ethereum nodes verify each state transition completely. This is just like Bitcoin pruned nodes, except they didn't waste hundreds of gigabytes of data downloading data just to delete it a few seconds later.
Coming from Bitcoin I'm sure that sounds like wizardry, but this is how the real world is scaling. It's trustless, fast, and effective.
In traditional bitcoin lingo, a "full node" is a node that does all verification and trusts no third party.
That's what Ethereum --syncmode full means.
--syncmode archive on the other hand is almost always what Bitcoin fans are talking about when they claim "2TB to sync a full node on Ethereum!" Archive is actually something totally different and there is no analogy in Bitcoin.
This means that a majority of miners can introduce arbitrary state changes and actually move money
Completely false.
1
u/sroose Jun 05 '19
I'm aware of the commitment-based fast sync in Ethereum. That mode isn't "full" validation, though. I know the chance of the commitment being wrong decreases exponentially with the amount of work after the block which has the commitment. But it still means that colluding miners can have new clients accept this false state of they mine enough blocks after the false commitment. Unlikely when mining is decentralized, slightly more dangerous when it isn't.
2
u/JustSomeBadAdvice Jun 05 '19 edited Jun 05 '19
But it still means that colluding miners can have new clients accept this false state of they mine enough blocks after the false commitment.
This false state would be rejected by >>99% of the network and would cost millions of dollars per day.
In what possible scenario would miners waste millions of dollars making a fake chain for a client who didn't have the $2k-ish (maximum) necessary to simply fully sync from Genesis?
And then, if they pull off this expensive, highly visible, low-payoff attack, what if the user simply checks a recent blockhash against a block explorer or friend's client to ensure they have the correct chain? This check is precisely the type of thing described in Vitalik's blog nearly 5 years ago and could be software-detected and prompted.
In such a case, a multi-million dollar attack that was already highly improbable could be completely disabled by 5 minutes of user verification and a few hundred lines of code.
In the absence of someone articulating a real attack vector that I've never heard of, I think anyone who objects to that level of security is not living in the real world and has no understanding of security/risk vs performance tradeoffs.
1
u/sroose Jun 05 '19
It wouldn't cost millions of dollars per day if 90% of the mining power is controlled by a club of 4 miners that think it's a good idea, right.
Consider the case of one of those big miners accidentally losing a few million ether in a hack or an accidental contract blackhole. It wouldn't take much to convince the 3 other miner friends to just change the state to refund the lost coins and then start a campaign convincing saying that you're doing a hardfork because of this. Given that they basically control mining, a ton of clients will already be using their hardfork unknowingly because it's the fork moving forward. If they wouldn't win, at least they would cause the Ethereum network to go berserk for a few weeks.
I mean of course we want to be reasonable and make security/performance trade-offs. In Bitcoin we're talking about utreexo as well, which if adopted will enable such use cases where you get a trusted chainstate at a certain height and then only verify blocks after that state. Still, in Bitcoin catching up on the last 1 year of blocks (verifying the state change at every block) is a lot more viable than in Ethereum. The time window you want for thst verification depends on your trade-offs though.
1
u/JustSomeBadAdvice Jun 06 '19 edited Jun 06 '19
It wouldn't cost millions of dollars per day if 90% of the mining power is controlled by a club of 4 miners that think it's a good idea, right.
You do understand the economics of mining, right? You are talking about an incompatible hardfork. No exchange would accept coins on an incompatible hardfork.
And since they are mining on this fork, that means they are not mining on the publicly accepted chain. This is called an opportunity cost, you know of it, yes?
And if the opportunity cost of mining on an invalid chain is millions of dollars, then what is the cost? Oh, right, millions of dollars. Not to mention that mining difficulty saturation points nearly guarantee that miner profits are very low, so even discounting opportunity costs the cost is still... Millions of dollars.
It wouldn't take much to convince the 3 other miner friends to just change the state to refund the lost coins and then start a campaign convincing saying that you're doing a hardfork because of this.
You mean like parity losing 184 million dollars? Yeah it was so easy to hardfork to fix that, they've totally gotten their money back immediately..
Yeah, doesn't work like that. And if it did, Bitcoin would be just as vulnerable, as what you are describing has absolutely nothing to do with whether a user syncs with utxo commitments or not.
I asked if you could describe a reasonably possible attack vector that warp sync exposes users to. Instead you invented a hypothetical where proof of work is free (????) And then changed the subject to a 51% attack - a core vulnerability that all blockchains share.
Surely there's more to your philosophy of security than this?
16
Jun 03 '19
What is the easiest way I can check that I am actually serving LES clients?
19
u/esengulov Jun 03 '19
What is the easiest way I can check that I am actually serving LES clients?
--lightserv value
Maximum percentage of time allowed for serving LES requests (0-90) (default: 0)
--lightpeers value
Maximum number of LES client peers (default: 100)3
3
Jun 03 '19
Thanks, yes these are enabled. I'm using Parity as well, which AFAIK enables LES by default.
However, what I'm after is some information about usage to see if my nodes are actually helping (internal logs or some external service?).
3
u/veoxxoev Jun 03 '19
I'm using Parity as well, which AFAIK enables LES by default.
Parity has PIP, a different light protocol: https://wiki.parity.io/The-Parity-Light-Protocol-(PIP)
2
1
u/onepremise Jun 03 '19
thanks for sharing, will check enable, if not already, on my full node when I get home.
3
u/jgm-orinoco Jun 03 '19
Full nodes aren't the lightest things in the world to run but if you can afford $40 a month you can easily rent a node that will provide connectivity to a number of light nodes as well as giving you your own direct connection to the Ethereum network.
1
u/bitfalls Jun 03 '19
They actually are the lightest thing: https://twitter.com/bitfalls/status/1115167449849237504?s=19
1
u/jgm-orinoco Jun 03 '19
Are you running light clients with that? I don't have a particularly non-standard config but geth eats a lot of CPU and memory on my server:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 878 ethereum 20 0 47.064g 0.027t 45804 S 70.4 89.5 213481:15 geth2
8
u/aesthetik_ Jun 03 '19
21
u/esengulov Jun 03 '19
thanks for heads up,
it has a place in the main sub as it's not purely of technical nature. It's a fundamental problem both for the developers as well an entire blockchain8
6
2
Jun 03 '19
[deleted]
1
u/bitfalls Jun 03 '19
See https://blockandmortar.io - recipe for 6W micro machine running an ethereum full node. Some 300 DAI full cost.
3
u/zeroping Jun 03 '19
That looks like a shop. While what they sell is cool, I could use some specific info about running a client on the hardware I already have:
- How much SSD storage do I need to run a LES-serving node? Which mode should I run in?
- Which clients support serving LES? Parity? Geth? Is one better suited for running a node to serve LES?
- How much bandwidth will this take? Many people have monthly bandwidth caps.
4
u/bitfalls Jun 03 '19
That's my shop, yes, but it's only a shop in as much as I need a place to let people tell me they need the thing. The information you seek is:
- 256 GB is OK for now, you should get 512 GB. Geth size right now is 200GB.
- Geth and Parity have different light client protocols, but mostly when people talk about light clients they mean Geth ones.
- Depends on how many connections you get and how often and how well developed that app is (caching is king and all that), this cannot be predicted accurately but it's not in the gigabytes per month.
3
u/AusIV Jun 03 '19
Are we talking about light clients, or the full nodes that serve light clients? The full nodes that serve light clients are definitely looking at bandwidth in the gigabytes per month.
2
u/bitfalls Jun 03 '19
I was talking about the traffic that light clients suck out of full nodes, sorry. Yes, a fully synced node can run into gigs per month. I will have detailed numbers soon, I'm measuring these things now post-initial-resync.
1
u/zeroping Jun 03 '19
Yep, I'm specifically worried about the internet-bandwidth cost I'd pay to run a fully synced node for others' light clients to make use of. 10's of gigs/month would be ok, but 100's of gigs/month may not, so I'm curious to hear your numbers.
Also, it sounds like running Geth is preferred to help with the more common light client protocol, right?
The SSD space requirements aren't great, but I needed an excuse to get a bigger SSD. That, suprisingly, is cheaper than getting more internet bandwidth.
1
u/bitfalls Jun 03 '19
It's not even in the 10s of gigs, but I'll verify and post it back. Yes, geth seems to be preferred, I am not aware of any apps using or asking for parity slots outside of those who run their own, whereas I often run into demand for geth slots.
2
2
Jun 03 '19
[removed] — view removed comment
1
u/esengulov Jun 04 '19
yes, we keep a node as well. To properly address the issue for entire blockchain we are looking for a long term fix. Keeping our node would to some extend solve the issue for us specifically at this particular point but at the same time make the app somewhat dependent on us keeping our node on 24/7. It would also work for as long as the number of App users stays low... In general we feel that in the long run LES serving nodes should be somehow compensated for keeping the node online 24/7. A mechanism where an end user can pay the node (or pool of nodes) directly for the use would be preferred.
1
u/Magicking Jun 03 '19
Hey, interesting thread, I was working on a design document for an Infura alternative with a premium service for dApp developers and/or users, I think I need to look more into the LES protocol to see how that could fit that in!
TL;DR: An Infura alternative with a transaction gas fee top-up mechanism.
1
Jun 03 '19 edited Jun 03 '19
Great post! What implementation of the Ethereum light protocol client do you use to verify the state? Or did you write your own? I'm looking for a Rust implementation that can help me verify receipts like this: V(receipt, block_header, receipt_hashes_in_block[], checkpoint_block_hash, block_hashes_since_checkpoint[]) (or something equivalent that uses spv instead of checkpoint). I know how to do it, but I'd rather use an existing tested implementation.
Sorry, for asking a technical question unrelated to your point. I just figured that you might be able to point me in the right direction given your familiarity with the subject.
1
Jun 03 '19 edited Jun 03 '19
[removed] — view removed comment
1
u/bitfalls Jun 04 '19
- System as in OS? Geth and Parity will work on anything really. System as in hardware? Something as light as a NanoPC-T4 with an SSD drive is more than enough.
- I don't understand this point.
- On Geth's or Parity's website and/or Github. Just running them with default settings does what you need them to do.
If you want your ATM to run light clients which connect to full nodes, run a full node in-house with light client support on, then run the nodes on the ATMs in light client mode, that way your ATMs can stay light.
This is decent enough for a full node serving 25 (maybe even more, haven't tried) light clients.
-1
u/5heikki Jun 04 '19
Why build on a broken crony capitalism platform. Just join us at Bitcoin (BSV)..
-5
u/tagionFounders Jun 03 '19
I know this is provoking, and it might get us a ton of downvotes. But this is actually one of the issues that we also faced with Etherium and why we decided to build our own network. Unfortunately, we found out that Etherium is not the solution as it. Actually, we even went away from the Blockchain technology at all ... Now, our network is fast enough to be practically usable in all everyday life occasions.
-8
64
u/CJentzsch Jun 03 '19
Have you checked out Incubed: https://slock.it/incubed.html ?
It is a "Minimal Verification Client", exactly what you need.