Recently, the Bitcoin community was split into two groups:
The groups were in disagreement over how to resolve the issue of delayed confirmations. The differences were mainly over the technical solution. However, there were ideological differences as well and things took a political turn. The divide was deep and resulted in creation of a new crypto-currency called BitcoinCash.
The controversy emerged because Bitcoin transactions were taking too long to get confirmed on the blockchain network. Bitcoin was not scalable.
Many transactions are clubbed together by a miner to form a block. To add a new block, miners need to solve a “problem”. The details of the problem, and its difficulty is explained here. The miners compete to solve the problem first and send a “block message” to the network as soon as the problem is solved. Other nodes accept the block into the blockchain based on the protocol rules. As transactions get included in a block, they get “confirmed”. When a new block gets added on top of the block containing the transaction, the number of confirmations increase by 1. It is advised that users must consider 6 or more confirmations on a transaction to be assured that the transaction is permanently accepted in the blockchain. The design is explained here: Bitcoin, Blockchain and the design elements explained.
Block message structure
|Field Size||Description||Data type||Comments|
|4||version||int32_t||Block version information (note, this is signed)|
|32||prev_block||char||The hash value of the previous block this particular block references|
|32||merkle_root||char||The reference to a Merkle tree collection which is a hash of all transactions related to this block|
|4||timestamp||uint32_t||A Unix timestamp recording when this block was created (Currently limited to dates before the year 2106!)|
|4||bits||uint32_t||The calculated difficulty target being used for this block|
|4||nonce||uint32_t||The nonce used to generate this block… to allow variations of the header and compute different hashes|
|?||txn_count||var_int||Number of transaction entries|
|?||txns||tx||Block transactions, in format of “tx” command|
There is a limit to the number of transactions that can be added to any block. The limit was such that the amount of data sent as a block cannot be greater than 1000000 bytes. The limit was set in:
This limit has been modified to 4000000 bytes. The new limit came into force with SegWit activation.
As more and more people started using Bitcoins, the number of transactions per second increased significantly. The limit on the size of a block was 1M bytes. The miners could not include all the submitted transactions in the new block. Thus, a lot of transactions were left unconfirmed during the creation of each new block. There was a build up of the number of unconfirmed transactions.
The current build up can be seen here: Unconfirmed transactions live. As blocks get added, there is a sudden drop in the number of unconfirmed transactions with each new block. This can be observed on the link provided if you keep it open in the browser long enough (possibly within a few minutes).
In the recent past the build up of unconfirmed transactions was so huge, that people could not get confirmations for hours and days (as claimed by some sites).
A number of solution approaches were considered. SegWit emerged with huge support, however there was a group which did not support SegWit and insisted on another approach. The two groups decided to part ways. In Blockchain, to part ways a hard fork is created. Thus BitcoinCash (User Activated Hard Fork) came into existence.
The Blockchain is a peer to peer network. All nodes have an equal say in whether to accept a block or not. All nodes follow the same rules for consensus. As there was a disagreement in the approach to solving the scalability problem, a group of nodes wanted to follow new consensus rules for creating and accepting a block. At the same time, these nodes had to accept all the previous blocks from the original Bitcoin Blockchain. They did not want all the investments made so far gone or be left on the Bitcoin blockchain. In order to achieve that, all the old transactions and blocks must be considered valid, however, new blocks and new transactions need to remain within the group of these nodes. This would lead to parted ways. The new transactions / blocks on either Blockchains must not spill into the other.
The block 478559 in the BitcoinCash network is called fork block.
The BitcoinCash approach was to increase the max block size to 8000000 bytes. This would cause a lot of transactions to be accommodated within each block. The miners would be benefited by the fees they received for mining transactions, and users would receive quick confirmations, hopefully within seconds. There were some changes needed to the consensus rules, so that the blockchain could have its own existence, and remain separated from the Bitcoin blockchain. The technical specification can be found here.
- The Bitcoin Cash was activated on Tue 1 Aug 2017, at 12:20:00 UTC. At that time the block size of Bitcoin was still 1MB.
- The fork block had to be greater than 1MB. As rest of the Bitcoin Blockchain was still on the 1MB limit, this prevented acceptance of that block in the original Bitcoin Blockchain.
- Once the fork is activated the transactions had to adhere to new way of signing the transaction. This needed a new signature implementation. The “fork id”, hashtype, and the hashing algorithm changes were implemented as explained by REQ-6-2 to REQ-6-4. The Bitcoin blockchain does not expect this, and hence will not accept the transactions from BitcoinCash.
- BitcoinCash would not accept old signature implementation on new transactions, thus new “Bitcoin” transactions will not spill over into the BitcoinCash Blockchain.
The other group argued that this would need much more processing power, and thus fewer miners would compete. This can lead to a situation where only a few powerful miners would be able to mine, and the users would have to trust only those. The group further argued that these “centers of power/control” is what Bitcoin was intended to remove.
SegWit: Segregated Witness
The SegWit solution tries to solve multiple problems.
- Transaction Malleability
- Makes Lightning networks possible
- Simplifying wallet software
When a transaction is created, a transaction id (txid) is generated. It is nothing more than a double hash of “version, inputs, outputs, lock time”.
The structure of a transaction was as follows:
|Version no||currently 1||4 bytes|
|In-counter||positive integer VI = VarInt||1 – 9 bytes|
|list of inputs||the first input of the first transaction is also called “coinbase” (its content was ignored in earlier versions)||<in-counter>-many inputs|
|Out-counter||positive integer VI = VarInt||1 – 9 bytes|
|list of outputs||the outputs of the first transaction spend the mined bitcoins for the block||<out-counter>-many outputs|
|lock_time||if non-zero and sequence numbers are < 0xFFFFFFFF: block height or timestamp when transaction is final||4 bytes|
|Previous Transaction hash||doubled SHA256–hashed of a (previous) to-be-used transaction||32 bytes|
|Previous Txout-index||non negative integer indexing an output of the to-be-used transaction||4 bytes|
|Txin-script length||non negative integer VI = VarInt||1 – 9 bytes|
|Txin-script / scriptSig||Script||<in-script length>-many bytes|
|sequence_no||normally 0xFFFFFFFF; irrelevant unless transaction’s lock_time is > 0||4 bytes|
|value||non negative integer giving the number of Satoshis(BTC/10^8) to be transfered||8 bytes|
|Txout-script length||non negative integer||1 – 9 bytes VI = VarInt|
|Txout-script / scriptPubKey||Script||<out-script length>-many bytes|
Put simply, the dynamics of the transaction are: who pays whom and how much. As a transaction gets propagated through the network, nodes in the middle can potentially change the contents of the script, without affecting the dynamics of the trade. A simple way to do this is to add a few inconsequential operations in the script (which was a part of each input).
The script usually has a signature, signed using a private key. Only a portion of the complete transaction is used for generating the signature. The portion which is not used for signing, can be modified without impacting the transaction and its dynamics. This will result in a new double hash of the transaction and thus the transaction ID is changed inflight. This was referred to as transaction malleability.
The node that created the transaction could not rely upon the txid it generated while submitting the transaction. It had to look for transactions in new blocks, and identify the one it submitted based on the inputs and outputs and not based on the txid.
To understand transactions and spending bitcoins using scripts, please read this link.
Simplifying the wallet software
With transaction malleability out of the way, wallet software can rely upon the fact that a transaction id that it generated (calculated) will not change. This eliminates the need for wallet software to write complicated code to verify if a transaction is confirmed or not. Also, the amount of processing required by the wallet software significantly reduces.
The current volume of transactions per day is very low on Bitcoin. The following link shows live data: Number of transactions per day! Even with this low volume we saw scalability issues. Bitcoin needs to improve its scalability by a factor of at least 1000 (as claimed by some popular websites). Clearly with the current way of handling transactions, Bitcoin cannot be used for micropayments. When micropayments are enabled, Bitcoin needs to handle Billions of transactions per day.
Lightning network has the potential to make this possible. Multiple people will come together to form payment channels. There would be transactions and updates to the transactions, and only the final settlement would need to get registered in the blockchain. In short there would be a private network of people who owe each other money. They would transact using such IOUs (simplication is deliberate here) and that will be settled with the last transaction, as finally someone pays the other the overall effect of these IOUs.
The following links are useful to understand the lightning networks:
The segwit implementation basically separates the transaction dynamics (who pays whom, how much) from the data that is needed to validate the transaction (signature and the script etc). The signature and the script used to be a part of the Transaction Input. Now they are stored in a separate field, which does not get used in transaction ID computation.
Serialization format for transaction with SegWit:
|4||version||int32_t||Transaction data format version|
|1||marker||char||Must be zero|
|1||flag||char||Must be nonzero|
|1+||txin_count||var_int||Number of transaction inputs|
|41+||txins||txin||A list of one or more transaction inputs|
|1+||txout_count||var_int||Number of transaction outputs|
|9+||txouts||txouts||A list of one or more transaction outputs|
|1+||script_witnesses||script_witnesses||The witness structure as a serialized byte array|
|4||lock_time||uint32_t||The block number or timestamp until which the transaction is locked|
Also, while calculating the size of the block, the witness data is not considered. This has reduced the size of each transaction and thus increased the overall capacity for transactions within a block. The block size limit has also been raised to 4000000 bytes.
Earlier to sign or to validate the signature on the transaction, the amount of bitcoins spent was not considered. This is because the value of bitcoins can always be extracted from the input transactions. The signing device would need to be online to calculate the amount spent, the commission for miners left etc, if it were to display the info.
Now, the amount of bitcoins spent is included in the signature digest algorithm. The following fields are considered to calculate the hash needed for signing or validating the signature.
Double SHA256 of the serialization of:
1. nVersion of the transaction (4-byte little endian)
2. hashPrevouts (32-byte hash)
3. hashSequence (32-byte hash)
4. outpoint (32-byte hash + 4-byte little endian)
5. scriptCode of the input (serialized as scripts inside CTxOuts)
6. value of the output spent by this input (8-byte little endian)
7. nSequence of the input (4-byte little endian)
8. hashOutputs (32-byte hash)
9. nLocktime of the transaction (4-byte little endian)
10. sighash type of the signature (4-byte little endian)
Due to the inclusion of “value”, IOUs (payment channels) have become possible.
It was not possible for SegWit to go live, until a sizable portion of the network signaled that it was ready for SegWit. If a new kind of transaction is to be incorporated, then it must be understood by the sender, receiver and the miners. With sizable portion of the network indicating readiness, it was possible for Bitcoin to activate the segwit implementation.
Segwit activation happened in a staged manner. First the network waited for atleast 95% of 2016 recent blocks to indicate that segwit would be supported if activated. This was called “lock-in”. Once Segwit was locked-in, the network waited for 2016 blocks (about 2 weeks) to enforce the new consensus rules.
The old mining software can potentially create blocks which will not be accepted by the new Segwit conforming nodes. This can lead to “fake” confirmations. There is an interesting history of such fake confirmations which can be read here.
The other group argued, that enabling Lightning Networks would mean creating small “clearing houses” which is what Bitcoin was supposed to remove!
Even if we ignore the economic or political outcome, it is evident that technologically the blockchain is very robust, and flexible. The ability to create a hard fork and part ways was successfully demonstrated. As well as the ability to expand the networks capacity and feature set, by altering the consensus rules was also demonstrated. Time will tell if Lightning nodes are capable of handling a Billion transactions per day, for now we are hopeful! 🙂