SegWit, BitcoinCash: Technical details explained

Recently, the Bitcoin community was split into two groups:

  • BitcoinCore
  • BitcoinCash

The groups were in disagreement over how to resolve the issue of delayed confirmations. The differences were mainly over the technical solution. However, there were ideological differences as well and things took a political turn. The divide was deep and resulted in creation of a new crypto-currency called BitcoinCash.

Scalability

The controversy emerged because Bitcoin transactions were taking too long to get confirmed on the blockchain network. Bitcoin was not scalable.

Many transactions are clubbed together by a miner to form a block. To add a new block, miners need to solve a “problem”. The details of the problem, and its difficulty is explained here. The miners compete to solve the problem first and send a “block message” to the network as soon as the problem is solved. Other nodes accept the block into the blockchain based on the protocol rules. As transactions get included in a block, they get “confirmed”. When a new block gets added on top of the block containing the transaction, the number of confirmations increase by 1. It is advised that users must consider 6 or more confirmations on a transaction to be assured that the transaction is permanently accepted in the blockchain. The design is explained here: Bitcoin, Blockchain and the design elements explained.

Block message structure

Field Size Description Data type Comments
4 version int32_t Block version information (note, this is signed)
32 prev_block char[32] The hash value of the previous block this particular block references
32 merkle_root char[32] The reference to a Merkle tree collection which is a hash of all transactions related to this block
4 timestamp uint32_t A Unix timestamp recording when this block was created (Currently limited to dates before the year 2106!)
4 bits uint32_t The calculated difficulty target being used for this block
4 nonce uint32_t The nonce used to generate this block… to allow variations of the header and compute different hashes
 ? txn_count var_int Number of transaction entries
 ? txns tx[] Block transactions, in format of “tx” command

There is a limit to the number of transactions that can be added to any block. The limit was such that the amount of data sent as a block cannot be greater than 1000000 bytes. The limit was set in:

https://github.com/bitcoin/…/src/consensus/consensus.h (MAX_BLOCK_SIZE of 1000000 – old commit)

This limit has been modified to 4000000 bytes. The new limit came into force with SegWit activation.

https://github.com/bitcoin/…/src/consensus/consensus.h (current commit)

As more and more people started using Bitcoins, the number of transactions per second increased significantly. The limit on the size of a block was 1M bytes. The miners could not include all the submitted transactions in the new block. Thus, a lot of transactions were left unconfirmed during the creation of each new block. There was a build up of the number of unconfirmed transactions.

The current build up can be seen here: Unconfirmed transactions live. As blocks get added, there is a sudden drop in the number of unconfirmed transactions with each new block. This can be observed on the link provided if you keep it open in the browser long enough (possibly within a few minutes).

In the recent past the build up of unconfirmed transactions was so huge, that people could not get confirmations for hours and days (as claimed by some sites).

Solution approaches

A number of solution approaches were considered. SegWit emerged with huge support, however there was a group which did not support SegWit and insisted on another approach. The two groups decided to part ways. In Blockchain, to part ways a hard fork is created. Thus BitcoinCash (User Activated Hard Fork) came into existence.

Hard fork

The Blockchain is a peer to peer network. All nodes have an equal say in whether to accept a block or not. All nodes follow the same rules for consensus. As there was a disagreement in the approach to solving the scalability problem, a group of nodes wanted to follow new consensus rules for creating and accepting a block. At the same time, these nodes had to accept all the previous blocks from the original Bitcoin Blockchain. They did not want all the investments made so far gone or be left on the Bitcoin blockchain. In order to achieve that, all the old transactions and blocks must be considered valid, however, new blocks and new transactions need to remain within the group of these nodes. This would lead to parted ways. The new transactions / blocks on either Blockchains must not spill into the other.

Hard_fork

The block 478559 in the BitcoinCash network is called fork block.

BitcoinCash

The BitcoinCash approach was to increase the max block size to 8000000 bytes. This would cause a lot of transactions to be accommodated within each block. The miners would be benefited by the fees they received for mining transactions, and users would receive quick confirmations, hopefully within seconds. There were some changes needed to the consensus rules, so that the blockchain could have its own existence, and remain separated from the Bitcoin blockchain. The technical specification can be found here.

  1. The Bitcoin Cash was activated on Tue 1 Aug 2017, at 12:20:00 UTC. At that time the block size of Bitcoin was still 1MB.
  2. The fork block had to be greater than 1MB. As rest of the Bitcoin Blockchain was still on the 1MB limit, this prevented acceptance of that block in the original Bitcoin Blockchain.
  3. Once the fork is activated the transactions had to adhere to new way of signing the transaction. This needed a new signature implementation. The “fork id”, hashtype, and the hashing algorithm changes were implemented as explained by REQ-6-2 to REQ-6-4. The Bitcoin blockchain does not expect this, and hence will not accept the transactions from BitcoinCash.
  4. BitcoinCash would not accept old signature implementation on new transactions, thus new “Bitcoin” transactions will not spill over into the BitcoinCash Blockchain.

The other group argued that this would need much more processing power, and thus fewer miners would compete. This can lead to a situation where only a few powerful miners would be able to mine, and the users would have to trust only those. The group further argued that these “centers of power/control” is what Bitcoin was intended to remove.

SegWit: Segregated Witness

The SegWit solution tries to solve multiple problems.

  1. Scalability
  2. Transaction Malleability
  3. Makes Lightning networks possible
  4. Simplifying wallet software

Transaction Malleability

When a transaction is created, a transaction id (txid) is generated. It is nothing more than a double hash of “version, inputs, outputs, lock time”.

The structure of a transaction was as follows:

Field Description Size
Version no currently 1 4 bytes
In-counter positive integer VI = VarInt 1 – 9 bytes
list of inputs the first input of the first transaction is also called “coinbase” (its content was ignored in earlier versions) <in-counter>-many inputs
Out-counter positive integer VI = VarInt 1 – 9 bytes
list of outputs the outputs of the first transaction spend the mined bitcoins for the block <out-counter>-many outputs
lock_time if non-zero and sequence numbers are < 0xFFFFFFFF: block height or timestamp when transaction is final 4 bytes

Each input:

Field Description Size
Previous Transaction hash doubled SHA256hashed of a (previous) to-be-used transaction 32 bytes
Previous Txout-index non negative integer indexing an output of the to-be-used transaction 4 bytes
Txin-script length non negative integer VI = VarInt 1 – 9 bytes
Txin-script / scriptSig Script <in-script length>-many bytes
sequence_no normally 0xFFFFFFFF; irrelevant unless transaction’s lock_time is > 0 4 bytes

Each output:

Field Description Size
value non negative integer giving the number of Satoshis(BTC/10^8) to be transfered 8 bytes
Txout-script length non negative integer 1 – 9 bytes VI = VarInt
Txout-script / scriptPubKey Script <out-script length>-many bytes

Put simply, the dynamics of the transaction are: who pays whom and how much. As a transaction gets propagated through the network, nodes in the middle can potentially change the contents of the script, without affecting the dynamics of the trade. A simple way to do this is to add a few inconsequential operations in the script (which was a part of each input).

The script usually has a signature, signed using a private key. Only a portion of the complete transaction is used for generating the signature. The portion which is not used for signing, can be modified without impacting the transaction and its dynamics. This will result in a new double hash of the transaction and thus the transaction ID is changed inflight. This was referred to as transaction malleability.

The node that created the transaction could not rely upon the txid it generated while submitting the transaction. It had to look for transactions in new blocks, and identify the one it submitted based on the inputs and outputs and not based on the txid.

To understand transactions and spending bitcoins using scripts, please read this link.

Simplifying the wallet software

With transaction malleability out of the way, wallet software can rely upon the fact that a transaction id that it generated (calculated) will not change. This eliminates the need for wallet software to write complicated code to verify if a transaction is confirmed or not. Also, the amount of processing required by the wallet software significantly reduces.

Lightning networks

The current volume of transactions per day is very low on Bitcoin. The following link shows live data: Number of transactions per day! Even with this low volume we saw scalability issues. Bitcoin needs to improve its scalability by a factor of at least 1000 (as claimed by some popular websites). Clearly with the current way of handling transactions, Bitcoin cannot be used for micropayments. When micropayments are enabled, Bitcoin needs to handle Billions of transactions per day.

Lightning network has the potential to make this possible. Multiple people will come together to form payment channels. There would be transactions and updates to the transactions, and only the final settlement would need to get registered in the blockchain. In short there would be a private network of people who owe each other money. They would transact using such IOUs (simplication is deliberate here) and that will be settled with the last transaction, as finally someone pays the other the overall effect of these IOUs.

The following links are useful to understand the lightning networks:

Lightning networks summary

Lightning networks paper

Lightning Networks Part I: Revocable Transactions

https://en.bitcoin.it/wiki/Lightning_Network

https://en.bitcoin.it/wiki/Hashed_Timelock_Contracts

 

Segwit implementation

The segwit implementation basically separates the transaction dynamics (who pays whom, how much) from the data that is needed to validate the transaction (signature and the script etc). The signature and the script used to be a part of the Transaction Input. Now they are stored in a separate field, which does not get used in transaction ID computation.

Serialization format for transaction with SegWit:

Field Size Name Type Description
4 version int32_t Transaction data format version
1 marker char Must be zero
1 flag char Must be nonzero
1+ txin_count var_int Number of transaction inputs
41+ txins txin[] A list of one or more transaction inputs
1+ txout_count var_int Number of transaction outputs
9+ txouts txouts[] A list of one or more transaction outputs
1+ script_witnesses script_witnesses[] The witness structure as a serialized byte array
4 lock_time uint32_t The block number or timestamp until which the transaction is locked

Also, while calculating the size of the block, the witness data is not considered. This has reduced the size of each transaction and thus increased the overall capacity for transactions within a block. The block size limit has also been raised to 4000000 bytes.

Signature Verification

Earlier to sign or to validate the signature on the transaction, the amount of bitcoins spent was not considered. This is because the value of bitcoins can always be extracted from the input transactions. The signing device would need to be online to calculate the amount spent, the commission for miners left etc, if it were to display the info.

Now, the amount of bitcoins spent is included in the signature digest algorithm. The following fields are considered to calculate the hash needed for signing or validating the signature.

Double SHA256 of the serialization of:
1. nVersion of the transaction (4-byte little endian)
2. hashPrevouts (32-byte hash)
3. hashSequence (32-byte hash)
4. outpoint (32-byte hash + 4-byte little endian)
5. scriptCode of the input (serialized as scripts inside CTxOuts)
6. value of the output spent by this input (8-byte little endian)
7. nSequence of the input (4-byte little endian)
8. hashOutputs (32-byte hash)
9. nLocktime of the transaction (4-byte little endian)
10. sighash type of the signature (4-byte little endian)

Details of signature verification.

Details of the segwit way of signature verification.

Due to the inclusion of “value”, IOUs (payment channels) have become possible.

Soft fork

It was not possible for SegWit to go live, until a sizable portion of the network signaled that it was ready for SegWit. If a new kind of transaction is to be incorporated, then it must be understood by the sender, receiver and the miners. With sizable portion of the network indicating readiness, it was possible for Bitcoin to activate the segwit implementation.

Segwit activation happened in a staged manner. First the network waited for atleast 95% of 2016 recent blocks to indicate that segwit would be supported if activated. This was called “lock-in”. Once Segwit was locked-in, the network waited for 2016 blocks (about 2 weeks) to enforce the new consensus rules.

The old mining software can potentially create blocks which will not be accepted by the new Segwit conforming nodes. This can lead to “fake” confirmations. There is an interesting history of such fake confirmations which can be read here.

The other group argued, that enabling Lightning Networks would mean creating small “clearing houses” which is what Bitcoin was supposed to remove!

Conclusion

Even if we ignore the economic or political outcome, it is evident that technologically the blockchain is very robust, and flexible. The ability to create a hard fork and part ways was successfully demonstrated. As well as the ability to expand the networks capacity and feature set, by altering the consensus rules was also demonstrated. Time will tell if Lightning nodes are capable of handling a Billion transactions per day, for now we are hopeful! 🙂

Advertisements

4 Comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s