Polkadot was designed with interoperability at heart, aiming to pave the way Blockchains can send, receive and interpret each other’s messages in a secure uniformly acceptable manner. This vision is what Polkadot set foot to achieve through XCM (cross-consensus message and formats).
But what is XCM? How does it work and is it different from the native blockchains messaging format? By the end of this article, you will have answers to that, plus know:
- Its architecture
- How the passes from one system to the other using XCMP
- How XCVM processes the message
- Use cases of XCM
Before we dive deep, let’s look into some historical background of the Internet. That will help us relate the complex subject to things we are already familiar with in hopes that we assimilate XCM better (at least that’s what helps me understand it). Thereafter, we will review the current state of blockchains so we can see how XCM fits in all this to solve the interoperability problem.
Earliest beginning of the internet
In the 70s, computers were more or less discrete machines with significant limitations in networking functionality. Even the then-available local networks were limited to government agencies, military institutions and universities. These private networks used what’s called ARPANET (advanced research projects agency network). Heightening demand in globalisation, coupled with the intrinsic constraints of ARPANET in effective data sharing, lead a team of developers thereof to come together and establish reformed set of rules that will enable those isolated local networks to connect with each other.
This collective effort gave birth to TCP/IP (transmission control protocol/Internet protocol), our modern day Internet. It became operational in 1983 and finally, computers can now communicate with universal language.
Midway into the decade, the domain name system was established. It uses the program Distributed Name Service (DNS) to easily name computers over the Internet. This largely replaced the rather complex and cumbersome IP address identification method, making it user-friendly.
Yet the universally accepted markup language then, SGML (standard generalized markup language), cannot interlink one resource to the other over the Internet.
Leveraging these breakthroughs, Tim Berners Lee in 1991 while working at CERN in Switzerland, came up with the idea of World-Wide-Web by inventing HTML (Hypertext Markup Language). With HTML, it’s easier to write web pages that can link up multiple web resources using hyperlinks. Thus named “web”. These hyperlinks use URI (Universal Resource Identifier), which uses a file-path system with junctions of ascension into the directory denoted by “/” to locate the linked resources. He also developed HTTP protocol to serve as a rail over which URI will carry HTML to the client.
Ever since, many modifications to his basic design are being made aimed at better web page UI designs and resource-fetching methods.
Yet, despite coming this far, client applications find it hard to manipulate data effectively on server databases running on different operating systems.
To tackle this problem, SOAP protocol was developed in 1998. APIs began using SOAP to enable client applications to fetch data servers irrespective of their OS. But still, SOAP only supports returned data in XML and strictly follows its underlying protocol.
This led Roy Fielding in 2001 to develop REST (Representational state transfer) as an alternative. REST is an architectural styling format rather than a protocol. In contrast to SOAP, it supports additional data types; JSON and htm. Being a format itself gave developers the ability to tailor it to best suit their business logic. Also, it abolished the need to conform to another protocol, but rather harness the already existing HTTP protocol. These properties make it universal and flexible.
For example;
GET https://example.com/document1:
will fetch document1 as located by URI
while:
POST https://example.com/document1:
will update document1.
By analogy, not that they are the same, XCM is to the Blockchains, what REST is to the RESTful services (keep this at the back of your mind).
How does this apply to the blockchain situation?
The current situation of Blockchains is analogous to the evolution of the Internet. Just that, we are yet to achieve true interoperability as the Internet does. There is still no universal lingua franca common to all Blockchains because almost each is powered by a different logic to the extent that forcing one Blockchain to read messages of the other may result in forking the chain.
- So how do we achieve “Internet” of blockchains?
- How do we ensure Blockchains can send messages to each other back and forth while they continuously upgrade their state engine without resulting in forking (forkless upgrade)?
This is where XCM comes in.
What is XCM?
Cross-consensus messaging format, abbreviated as XCM, is the architectural styling format that defines a standard language for communication between consensus systems. It dictates how the message should be interpreted by the receiving nodes, hence it is not a messaging protocol. The actual message interpretation is carried out by XCVM (cross-consensus virtual machine), which is version-sensitive and register-based, meaning XCM can only make sense to it if the version is known and well specified. It’s intentionally made that way because not every system is meant to interpret every message.
XCM is intended to be generic and extensible with the ability to interact with systems of unknown transaction format. This gives it an upper hand over the messaging format native to blockchain state transitions, in that it can still be supported by systems that decided to upgrade their logic. With that, one needs not worry about incompatibility because Polkadot got you covered by making its vision of scalability, security, and interoperability come true through XCM.
XCM transmission routes
For the sake of understanding, not that they are similar, this can be likened to how our conventional internet uses HTTP to transmit resources in a client-server relationship.
Polkadot uses two main routes for passing XCM messages:
- Vertical message passing (VMP): it involves passing messages between the relay chain itself and a parachain. Message data in both cases exist on the Relay-chain. It has two variants:
- Upward Message Passing (UMP): message passing from a parachain to the Relay-chain.
- Downward Message Passing (DMP): message passing from the Relay-chain to a parachain.
- Cross-Chain Message Passing (XCMP): used for secure message passing between parachains. It has two variants as well:
- Direct:- the message datagram passes directly between the parachains making it very scalable.
- Relayed:- the message datagram is sent first to the relay chain before en routing to the intended parachain, piggy-backing over VMP. Obviously, this is less scalable, and parathreads in particular may not receive messages due to excessive queue growth.
Note: at present, the relayed form of XCMP is used between parachains, also called HRMP (Horizontal Relay-routed Message Passing). The direct XCMP is still under development. HRMP is too demanding because it stores all the data on the relay chain reducing its efficiency. Since Polkadot aims to relieve the relay chain of all extra duties with time, as soon as direct XCMP is implemented, HRMP will be phased out in favor of it.
The four “A”s of XCM
The fact that XCM gives these Absolute guarantees allows it to be practically Asymmetric whereas other non-absolute protocols would find this difficult.
Being Agnostic means that XCM is not simply for messages between parachain(s) and/or the Relay-chain, but rather that XCM is suitable for messages between disparate chains connected through one or more bridge(s) and even for messages between smart contracts. Using XCM, all of the above may communicate with, or through, each other.
Architecture of XCM
To oversimplify the anatomy of XCM message, let us break it down into three parts:
MultilocationsMultiassets- Message
instructions
Multilocations
Going back to our little highlight on the history of internet evolution and how the different discoveries come together to provide a uniform way of assessing resources at different locations. We know how URI is used to precisely locate resources. Multilocations can be thought of to work in a similar way. It uses the same relative file-path system with junctional "/" to denote further ascension into the outer consensus just like how URI is written. A MultiLocation is a relative identifier, meaning that it can only be used to define the relative path between two locations, and cannot generally be used to refer to a location universally.
A MultiLocation is thus encoded as the pair of values:
parents: u8: The levels of consensus to ascend before interpreting the interior parameter.interior: InteriorMultiLocation: A location interior to the outer consensus system found by ascending from the local systemparentstimes
In figure above, for example, parachain 1000 wants to locate a user account on parachain 2000, ../ is referencing the parent, in this case, the Polkadot relay chain while "/" signifies the junction of ascension into parachain 2000 to further locate the user with AccountId(0x123…)
By definition, multilocation is an isolatable state machine held within a global consensus. Any of the following can be referenced and its non-exhaustive:
- layer-0 chain, e.g. the Polkadot Relay chain.
- layer-1 blockchain, e.g. the Bitcoin mainnet or a parachain.
- layer-2 smart contract, e.g. an ERC-20 on Ethereum.
- Sharded enclave e.g. SPREE
- A normal user account
Multiassets
This signifies the attributes of the specific asset in question. It helps with precisely identifying the asset while avoiding ambiguity or name collisions. To achieve this, two ways of identifying asset classes are proposed:
- Abstract identifiers
- Concrete identifiers
Each one has its own pros and cons which we are not going into in this article, but it’s good to know that Polkadot prefers concrete identifiers as they best work with relative patterns of multilocation without the need for a global registry, something that abstract identifiers need since they are meant to absolute and universal. Also, no need to worry about name collisions or ambiguity since the asset only has one identity, you just have to re-anchor it whenever being moved to a new consensus system
The typical format of multiassets represented by the SCALE-encoded pair of fields is:
class: AssetId: The asset class.fun: The fungibility of the asset, this is a tagged union with two possible variants:Fungible = 0 { amount: Compact }: In which case this is a fungible asset and theamountis expected to follow the 0 byte to identify this variant.NonFungible = 1 { instance: AssetInstance }: In which case this is a non-fungible asset and theinstanceis expected to follow the1byte to identify this variant.
For example, let’s say we want to display the multiassets format as well as multilocation of 100 DOTs on the relaychain as referenced from any parachain, it will look like this:
Multiasset {
id: concrete (
Multilocation (../)
),
fun: Fungible (100 [units]),
}
(I bet it makes sense, right!)
Message instructions:
Congratulations, with your detailed description and super efficient virtual GPS, you have successfully located the asset on the other consensus. Now, what do you want the XCVM of the respective consensus to do with the asset?
Just like how your four basic commands work in REST (remember it uses GET to display the data, PUT to insert new, POST to update it while DELETE to erase it). Similarly, XCM is encoded with the instructions set of what to do with the located asset. But in this case, it’s not four commands, it’s many commands each stacked over another to make logical sense to the XCVM executing it.
NB: this part of XCM is version-sensitive, hence if the XCM is differently versioned, it will be opaque and unreadable.
The instructions set are as follows:
WithdrawAssetReserveAssetDepositedReceiveTeleportedAssetQueryResponseTransferAssetTransferReserveAssetTransactHrmpNewChannelOpenRequestHrmpChannelAcceptedHrmpChannelClosingClearOriginDescendOriginReportErrorDepositAssetDepositReserveAssetExchangeAssetInitiateReserveWithdrawInitiateTeleportQueryHoldingBuyExecutionRefundSurplusSetErrorHandlerSetAppendixClearErrorClaimAssetTrapSubscribeVersionUnsubscribeVersion
For a detailed description of each one, please check out this xcm-format repo.
How does the actual message passing work?
First off, let’s get ourselves familiar with some terminologies used henceforth in this article as well as in Polkadot codebase documentation.
Ingress: input message on the inbound queue of receiving consensusEgress: output message on the outbound queue of dispatching consensus
Cross-chain transactions use a simple queuing mechanism based around a Merkle tree to ensure fidelity. Each channel of communication to another parachain is one-way (just like RESTful client-server relationship in which only the client initiates requests). But unlike in RESTful, the connected parachain may also want to send back messages to the other parachain, hence it opens another one-way channel to that parachain. This means that to reciprocate messages between two parachains, two channels of communication must be opened between the two at maximum.
Opening of every channel requires the deposition of a certain amount of DOT by the sender, and it only gets returned after closing the channel.
For better assimilation, let’s describe it in an example:
- Let us say, Alice using her account on parachain 1 wants to send some tokens to Bob’s account on parachain 2.
- Alice executes a smart contract on parachain 1 which initiates a new XCM; specifying the appropriate version, multilocation, multiasset and a timestamp of the transaction.
- The collator node of parachain 1 will place this new XCM on its egress queue.
- The collator node of parachain 2 while routinely checking all other collator nodes’ egress queues will find this new XCM meant for it and will add it to its ingress queue to be processed in the next block.
- At the same time, validators for the participating Parachains also see the XCM on their respective parachain queues. This is so that they will be able to verify the message transmission occurred.
- On its next block, the collator of parachain 2 will process the XCM it received on its ingress, this triggers the smart contract to read and complete the transfer.
- Now the collator hands over the block to its validator for verification, once everything is considered valid, the validator will add the block into the relay chain
Featured use-cases of XCM
- Sending requests for specific operations to occur on the recipient system
- Optionally including payment of fees on a target network for a requested operation
- Token transfer in either of the following models:
- Remote transfer: this means controlling an account on a remote chain, allowing the local chain to have an address on the remote chain for receiving funds and to eventually transfer those funds it controls into other accounts on that remote chain.
- Teleporting: this means that when moving an asset, one has the ability to destroy it on one side and create a clone of it on the other side.
- Reverse-based transfer: this comes in when there are two chains that want to nominate a third chain, where one includes a native asset that can be used as a reserve for that asset. Then, the derivative form of the asset on each of those chains would be fully backed, allowing the derivative asset to be exchanged for the underlying asset on the reserve chain backing it.
Stretch that neck I know it aches! See you on the next one. Check put these links for more details on the topic:
- Dr. Gavin Wood’s explainer articles: part 1, part 2 and part 3
- XCM: The backbone of multichain future. Stage presentation at Polkadot Decoded 2022
- Watch XCM AMA with Gavin Wood here