In a blockchain based cryptocurrency, maintaining consensus between the node implementations is a critical security requirement. If nodes don’t agree on the consensus rules, then a malicious actor can force a network split.
This requirement applies even more strongly to nodes used by miners. If a miner produces a block that is rejected, then the miner loses the block reward for the block and incurs a direct financial loss due to the loss of consensus. A network split could damage the reputation of the coin and the reduction in price of the coin reduces the miner’s future income.
Multi-Implementation Coin
In theory, a cryptocurrency can have multiple peer implementations of the consensus rules. No node implementation is more important than the others. The miners split their hashing power roughly equally between the implementations. It is assumed that the consensus rules are agreed between the clients. The procedure for changing the consensus rules need to balance minority veto rights with the right of a super-majority to move ahead with changes despite opposition. The problem of cryptocurrency governance is a major unsolved problem for cryptocurrencies.
Once the spec change is agreed, all the teams implement the changes. Each team must ensure that their implementation is consistent with the other clients. In the image, there are six node implementations. Each node team must run compatibility tests against the 5 other implementations. This is a significant amount of work for the node maintainers. Any change to the node needs a compatibility check. At minimum, they must verify that the change has no effect on consensus behavior.
Incompatibility Fork
An incompatibility between the implementations can lead to a fork of the chain. If a minority of the nodes consider one of the blocks in the longest chain to be invalid, this leads to a disagreement about which chain tip is the longest valid chain. A malicious actor who discovers this incompatibility can trigger the fork by producing a block (or transaction) that exposes the incompatibility.
In the image, the green implementation considers the black block to be invalid. The other 5 implementations consider the block to be valid. Miners using the green implementation extend the chain from the block before the black block. Since the other miners have a majority of the hashing power, their chain grows more quickly. The green miners do not consider the longest chain to be valid and they keep mining on their fork. If no manual action is taken to resolve the issue, then a permanent chain fork is the result.
An attacker could double spend their coins. Coins can be spent on both forks. A merchant using the green client may not be aware of transactions that are included in the other fork (and vice versa).
Preventing double spending is one of the fundamental purposes of the blockchain. This situation represents the system failing to meet this fundamental requirement.
If the situation is resolved, miners lose any rewards they earned on the disregarded fork.
This happened with Bitcoin in the March 2013 Chain Fork. It was caused by an upgrade to the UTXO set database library. The new database software was able to handle some blocks that the older software had difficulty processing. A block was produced by an upgraded node that the older database software couldn’t handle. The green node equivalents were the older nodes and they rejected the block. The new nodes processed it fine and continued to build on it. Once miners with a majority of the hashing power upgraded to the new software, the vulnerability was exposed.
The situation was resolved by the miners downgrading to the older version of the software and all nodes configuring their databases to handle larger blocks. In effect, most miners switched to the green software. The miners who lost money due to the fork were compensated by funds that were donated to be used for the benefit of Bitcoin. This eliminated the incentive for those miners to maintain their fork.
If a majority of miners consider the block to be invalid, a less serious situation results. The majority’s chain will simply fork off the blocks that they consider invalid.
In the image, the green miners produces blocks that the other nodes consider invalid. The other nodes refuse to build on those blocks and they are forked off the chain whenever they are produced. The green node continues to track the main chain, since it considers it valid. The green miners lose money due to this incompatibility. There is a double spend risk for merchants who use green nodes and wait for a single confirmation.
This situation damages the reputation of the green nodes and they would likely lose market share as a result.
Reference Client
The issues with multiple node implementations lead to one node becoming the dominant node. In most cases, this node is simply the first node and it remains dominant from the start. Once established, it is very difficult to dislodge. This implementation is referred to as the “reference client”. Coins which try to select the multi-implementation arrangement without a reference client are still subject to the forces that lead to a reference client appearing.
In the image, the blue node is the reference node. By using one node as reference, the complexity of compatibility testing is greatly reduced. Each node verifies that it is compatible with the reference implementation only. If all nodes are compatible with the reference client, then they are likely to be compatible with each other.
The reference client team need not do any compatibility testing with other nodes. By definition, their implementation defines the consensus rules. Testing that blocks produced by the latest version of the reference client are accepted by all older versions of the reference client is still crucial.
The reference client team must ensure a consistent product. Hard forks wipe the historical requirement for compatibility, since they are by definition not compatible. The reference client does not need to verify compatibility with versions before the most recent hard fork. Legacy testing involves testing that the specific historical chain can be processed by the reference client.
The compatibility testing load for the other implementations is reduced but not eliminated. In the multi-implementation situation, each team needs to check compatibility against five other nodes. There is an incentive for each team to reduce the number of other implementations that they do compatibility testing against and to move towards a de facto reference client. With a reference client, each team tests compatibility against the reference client only.
Miners are strongly incentivized to produce compatible blocks. The easiest way for a miner to reduce the risk of block incompatibility is to use the implementation that has the most usage by other miners. If almost all miners switch to the most used software then that software will be used for almost all blocks.
This is a stable equilibrium. The most used software is used by most miners because it is the most used software. Since it is used by most miners, it is by definition, the most used software, creating the stable loop.
These loops reinforce each other. If most miners use a particular implementation, then most node implementations will use that implementation as their reference to ensure compatibility with the chain that it being mined. If most implementations are using a specific implementation as reference, miners are more likely to assume that it is most used one.
Coin Governance
The general cryptocurrency government system is shown in the image. There are two fundamental rules. One rule is that coin governance is permissionless. You don’t need to ask anybody for permission to fork a coin. If a group thinks that it is worth forking the coin, then they can just do it.
The other rule is if there is broad consensus within the community for the coin, then any change is possible. The community acts as final court of appeal. This is inherent in the way blockchains work. No previously existing rule can stop a community if (almost) everyone upgrades to a new set of rules. The Ethereum DAO fork is an example of this in action.
Ethereum is designed to act on the basis of “the code is the law”. Contracts are enforced by the blockchain in accordance with their terms without any other considerations. The blockchain’s judgement for the DAO was, in effect, appealed to coin’s supreme court. The community decided, for their own reasons, that enforcing the contract, as written, was not desirable.
The consensus for this change wasn’t quite total. A portion of the community exercised the forking option in reverse and remained on the original chain.
Chain forks are normally extremely damaging to coins. The option to fork remains available at all times, but choosing it should not be the first choice. Two smaller communities are less effective that one larger community.
If the differences are truly irreconcilable and the conflict is damaging the coin, forking can increase value. It produces two internally cooperative communities rather than a larger community that is in conflict with itself. The benefit of better internal peace outweighs the cost of loss of a single large community. The loss of the single large community is still a loss though.
In a community that has a disagreement about something non-critical, a split is very likely to do more harm that good.
For changes where there is not broad consensus but which are not worth forking over, the coin’s specific governance system is used. In the image this where “magic happens”. Decision making for the coin depends on the structures and norms of the community associated with the coin.
Almost all decisions relating to consensus rules for a coin fall into this category. The existence of the forking option doesn’t eliminate the need for actual governance processes for issues that are not split worthy. Worse, the absence of such processes creates the possibility of an unnecessary coin split over a non-split worthy issue. When deciding if a split is worth it, the issue gets combined with leadership question.
Reference Client Governance
Once a reference client is established, the reference client’s developers become the de facto protocol developers. If the reference client has a specific project leader, then that person is de facto chief maintainer of the protocol. It doesn’t matter if this is desirable or not, it is simply the incentives of the parties playing out.
The leader of open software projects are often referred to as the benevolent dictator of the project. The potential for abuse of their position is counterbalanced by the ability of anyone to fork their project and produce a competing version. Tradition has it that the new project uses a new name, but can otherwise use the exact code. The new team has to build their own reputation. This allows a leader to be replaced if problems with his leadership arise.
This escape value exists in a much weaker form for cryptocurrencies. Forking the software is not enough. A significant portion of the value in embedded in the blockchain. To fork a cryptocurrency, the blockchain has to be forked too, but that has all the associated fork split damage. Both sides can’t agree to go their own way, since much of the value created by the community is embedded in the blockchain itself.
Reference client governance means that the reference client leader has significant influence on protocol modification decisions. This is not a binary thing where he has absolute power, but the reference client leader can normally tilt the decision in the direction of his choosing.
Self-Fulfilling Prophesy
When the rule change is contentious, the selection of consensus rules operates as a self-fulfilling prophesy. The rules selected are the rules that people believe are most likely to be selected. The selection is determined by what people think the selection is going to be.
In the event of a split, miners have to pick which fork to mine on. They pay the same in hashing for mining on both forks, but if they mine on the fork that loses, their block rewards will be worth much less. If the fork they pick simply dies, then the block reward they get is worth zero.
The same incentives apply to merchants. A merchant wants to accept the most popular coins. There is no point in supporting a coin that nobody is using. A merchant who accepts coins on the wrong fork ends up with coins that have a lower market price. When a merchant decides which fork to accept, picking the less used one causes direct financial harm.
Exchanges have to deal with a lot of complications when a fork happens. They have to handle two separate coins. Replay protection helps here, but that is only part of the complexity. They have to decide on assignment of the ticker symbol. Assignment of the ticker symbol could have legal consequences if customers feel that they were misled about what they were buying. Some exchanges split the ticker symbol into two longer symbols until one chain is the clear winner.
Exchanges can run future markets which allow traders to trade coins on each fork before the fork has happened. This gives direct information about which fork the traders think is more likely to be the one selected. It suffers from the same self fulfilling prophesy problem. The option that is most likely to be selected will rise in probability and the other ones will fall.
This feedback loop causes the most likely choice’s chance of being selected to be higher than people think and all the other options’ chances to be lower. If everyone thinks that the odds of proposal A being selected is 52-56% and the odds of proposal B is 44-48%, then proposal A’s odds of being selected could very easily be more than 90%. It is crucial for those advocating a proposal to convince as many participants as possible that their proposal is the one to be selected. If they can do that, their proposal is likely to be selected.
Challenging the Benevolent Dictator
If there is no formal method for deciding the protocol maintainer, then it falls back on the fundamental rules for governing cryptocurrencies. The challenger proposes his decision and the incumbent proposed his decision. Both claim that the other side is trying to split the coin. The split risk emerges as a fundamental consequence of the breakdown in consensus. Both sides are asserting their right to fork and the (potential) split is a consequence of it.
This situation has been likened to a game of chicken. Both sides are driving at each other and the first to swerve loses. The table below gives the payout matrix.
| Challenger Concedes | Challenger Commits | |
| Incumbent Concedes | Challenger Wins | |
| Incumbent Commits | Incumbent Wins | Chain Split (Both harmed) |
If neither side swerves, the coin is damaged due to a chain split and fracturing of the community. Ironically, all else being equal, the system gives power to the person or group that is most willing to damage the coin. This is a less than ideal criterion for selecting the leader.
The key to making decisions for a coin is to convince the community that a particular decision is the one that will be picked. This is where the incumbent has an advantage. Who is most likely to be the decision maker for the next decision, the person who made the final decision in all the previous occasions or some new guy?
The game is not a simple game of chicken, both sides are not balanced. The incumbent has a massive advantage. If neither side yields, the damage to the challenger is likely to be more than the damage to the incumbent. The incumbent is the most likely to win and most people know it. The feedback loop pushes that probability to near certainty. Even if there is a split, the incumbent’s fork is likely to be the most successful. The challenger’s fork has a high likelihood of simply dying.
Challenging the incumbent and failing has an effect on future disputes. If the incumbent has faced multiple split threats in the past and always prevailed, people will assume that he will prevail in future disputes. An easily defeated challenge has the potential to be beneficial to the incumbent.
Consequences for Strategy
Leadership by the incumbent is limited only by hyper-transfer to another client. Under hyper transfer, users suddenly switch to a competing product. When hyper-transfer happens, the incumbent starts as invincible and then suddenly, the challenger is obviously going to win. This means that the first strategy is to make it as easy as possible for the users to switch.
Miners are cautious and want a product that is safe to use. Loss of a block reward is an unacceptable cost for miners. The challenger must build up a reputation for producing reliable software.
Even if the software is bug free, it can still be incompatible with the reference client. Ideally, the challenger’s software should be identical to the reference client but include the proposed change. The challenger shouldn’t bundle lots of changes. Bundling forces the miner to evaluate a list of differences between the reference client and the challenger’s node to decide which is better. The easiest path for a miner is to skip the evaluation of all the changes and go with the reference client.
There is a paradox, simply building someone else’s software doesn’t demonstrate good software development skills. The development team could produce three versions of the software. First, they release a version of the reference client compiled by them. This demonstrates that they can produce an identical build environment to the reference client. The second is a version of the reference client with their proposed consensus rules and no other changes. The third is their full client with all the features that they offer. This means that a miner can run the second version to support the consensus change without having it bundled with all their other changes. Their full client shows that they have the technical skills to maintain a reference client in the future.
Only people who run nodes can be challengers. It is no good for someone to say that they want to challenge the incumbent, but don’t want to maintain a node. Miners aren’t going to run software that is planned to have a single release. They want a team that reliably releases software. If the reference client has such a team and the challenger does not, then miners will pick the reference client.
Since the choice selected is a self fulfilling prophesy, perception is a crucial part of making the decision. Even if all the whole community supports the challenger, it doesn’t matter if nobody knows about it. The challenger doesn’t win by convincing people to support him. Victory is achieved by demonstrating that he has more support than the incumbent. Miners who run his software in secret aren’t having much effect. They can switch back to the incumbent virtually instantly. Features and performance improvements are irrelevant to the miner when compared to actually staying in consensus. The miner’s top priority is protecting against any of his blocks being forked off.
Small defeats make it harder to win later. A challenger needs to bring overwhelming support once to cause a switch. Threatening to split the coin is perhaps more effective than actually doing it.
Technical features about the fork may also matter, though this matters more when the objective is to actually split the coin into two forks. The Bitcoin difficulty adjustment algorithm’s 2 week adjustment kills minority forks of Bitcoin and causes problems for other DSHA256 coins. Bitcoin Cash’s emergency difficulty adjustment and subsequent adjustment algorithm are specific responses to this fact. The EDA was an admission that Bitcoin Cash had a high likelihood of not being the majority chain, but miners could at least have confidence that it would not simply die.
Realistically, a challenger implementation faces a major uphill battle. It is extremely difficult to displace the incumbent. The incumbent needs to be sufficiently terrible to convince most people that he has little chance of success. The challenger implementation and their supporters always retain the right to fork the chain, but this is extremely damaging. It splits both the chain and the community. A community which has no cohesion will just keep fragmenting into smaller and smaller pieces and then into irrelevancy. Challengers must always weigh that risk against their views of the quality of the incumbent. If the challenger is not confident of victory, then the challenge may do more harm than good.
Role for Challenger Implementations
The reference client governing system operates with the reference client maintainer having de facto control over the protocol. This doesn’t mean that other implementations are completely irrelevant. Under this system, the challenger implementation play a critical role in moderating the actions of the incumbent. This is a thankless job but it is crucial to acting as a check on the incumbent. To act as this check, the alternative implementations must actually be capable of taking over as reference client.
The number of plausible alternative reference clients gives an indication of the level of decentralization for a coin. If there is a single reference client and none of the alternatives are viable alternatives, then there is not much of a check on the power of the reference client team. They know that the miners aren’t really going to switch to one of the other implementation.
The community must accept that there is a reference client. Denying reality doesn’t actually change the truth. It just makes it more difficult to make effective decisions. If the community wishes to replace the reference implementation, then the need to decide what they want afterwards. Do they want to shift to a different governing model completely or do they want to change the incumbent but retain the reference client governance system.
A multi-implementation governance system is not stable. Purely for efficiency’s sake, the system will evolve into a state where one implementation is being used as reference by the other implementations. Assertions that decisions are made by consensus don’t change anything. The general governance model is clear, if there is broad consensus, then it is easy to just implement the change.
A governance model needs to include a method to handle contentious changes. What happens if there is no broad consensus and everyone doesn’t agree. Is the result stasis? A reasonable answer is that the status quo wins for a while, but there should be a way to “end debate” and move forward against opposition.
Under the reference client system, the reference client leader has the power to end debate and make decisions about protocol updates. If they do, they risk a challenge and loss of reference client status, but unless the decision is clearly wrong, they will likely prevail. If there are no viable challenger nodes, then there is no check on the reference client leader’s power.