Debugging Tips and Tricks

Brief overview of how one can approach debugging CCIP dApps

Welcome to bonus chapter about debugging Chainlink CCIP dApps and troubleshooting issues with sending and receiving cross-chain messages. Writing code is just the beginning, the true skill often lies in effectively troubleshooting and resolving issues that arise.

Getting Started

To understand where the issue is or may arise, you first need to have bookmarked the whole transfer of cross-chain message flow, which you can find in the Processing CCIP Messages subsection.

The second most important thing is to understand how ABI encoding/decoding works in EVM. When you compile Solidity code, the two primary outputs are Application Binary Interface (ABI) and EVM bytecode.

EVM bytecode is the machine-level code that the Ethereum Virtual Machine executes. It is what gets deployed to the Ethereum blockchain. It's a low-level, hexadecimal-encoded instruction set that the EVM interprets and runs. The bytecode represents the actual logic of the smart contract in an executable form.

The ABI is essentially a JSON-formatted text file that describes the functions and variables in your smart contract. When you want to interact with a deployed contract (e.g., calling a function from a web application), the ABI is used to encode the function call into a format that the EVM can understand. It serves as an interface between your high-level application (like a JavaScript front-end) and the low-level bytecode running on Ethereum. The ABI includes details about each function's name, return type, visibility (public, private, etc.), and the types of its parameters.

Decode error messages

Anytime your message is not delivered due to error on the receiver side, CCIP Explorer will display the error message. However, if ABI is unknown to explorer (for example you haven't verified smart contract's source code on Block explorer) it will display its original content instead of human readable error message. However, if we know how ABI encoding/decoding works, that's still completely fine because there are plenty of tools that can help us.

So at first glance, when seeing this message, you can go into the panic mode and think how CCIP does not work or something similar. So let's decode the error message to see what went wrong.

So the error code is: 0x1c33fbee000000000000000000000000000000000000000000000000304611b6affba76a

This is esentially a hexadecimal value that mean a lot of stuff. But if we know the ABI of the Receiver smart contract it's simple to decode it.

Couple of tools that you can use:

• Some online decoder, this one is pretty decent: https://bia.is/tools/abi-decoder/

• Foundry's cast abi-decode

Let's use the https://bia.is/tools/abi-decoder/ online decoder for this example. We need to provide contract's ABI, the above error code and to hit decode to get a human readable error message.

The decoded output unequivocally tells us that the Solidity custom error ChainNotEnabled() was thrown, which most likely means that we forgot to call enableChain() function of the Receiver's smart contract in general case.

{
  "name": "ChainNotEnabled",
  "params": [
    {
      "name": "chainSelector",
      "value": "3478487238524512106",
      "type": "uint64"
    }
  ]
}

Use CCIP Revert Reason script

If you are a bit more technical, you can use a script present in the official CCIP GitHub repo https://github.com/smartcontractkit/ccip/blob/ccip-develop/core/scripts/ccip/ccip-revert-reason/main.go to accomplish the same thing when debugging, plus much more.

You will need to provide either an error code string or the chainId, txHash and txRequester alongside JSON RPC url in .env file (archive node preffered).

package main

import (
	"fmt"

	"github.com/ethereum/go-ethereum/ethclient"
	"github.com/joho/godotenv"

	"github.com/smartcontractkit/chainlink/core/scripts/ccip/revert-reason/handler"
	"github.com/smartcontractkit/chainlink/core/scripts/ccip/secrets"
)

// How to use
// Set either an error code string OR set the chainId, txHash and txRequester.
// Setting an error code allows the script to run offline and doesn't require any RPC
// endpoint. Using the chainId, txHash and txRequester requires an RPC endpoint, and if
// the tx is old, the node needs to run in archive mode.
//
// Set the variable(s) and run main.go. The script will try to match the error code to the
// ABIs of various CCIP contracts. If it finds a match, it will check if it's a CCIP wrapped error
// like ExecutionError and TokenRateLimitError, and if so, it will decode the inner error.
//
// To configure an RPC endpoint, set the RPC_<chain_id> environment variable to the RPC endpoint.
// e.g. RPC_420=https://rpc.<chain_id>.com
const (
	ErrorCodeString = "0x4e487b710000000000000000000000000000000000000000000000000000000000000032"

	// The following inputs are only used if ERROR_CODE_STRING is empty
	// Need a node URL
	// NOTE: this node needs to run in archive mode if the tx is old
	ChainId     = uint64(420)
	TxHash      = "0x97be8559164442595aba46b5f849c23257905b78e72ee43d9b998b28eee78b84"
	TxRequester = "0xe88ff73814fb891bb0e149f5578796fa41f20242"
	EnvFileName = ".env"
)

func main() {
	errorString, err := getErrorString()
	if err != nil {
		panic(err)
	}
	decodedError, err := handler.DecodeErrorStringFromABI(errorString)
	if err != nil {
		panic(err)
	}

	fmt.Println(decodedError)
}

func getErrorString() (string, error) {
	errorCodeString := ErrorCodeString

	if errorCodeString == "" {
		// Try to load env vars from .env file
		err := godotenv.Load(EnvFileName)
		if err != nil {
			fmt.Println("No .env file found, using env vars from shell")
		}

		ec, err := ethclient.Dial(secrets.GetRPC(ChainId))
		if err != nil {
			return "", err
		}
		errorCodeString, err = handler.GetErrorForTx(ec, TxHash, TxRequester)
		if err != nil {
			return "", err
		}
	}

	return errorCodeString, nil
}

And if you run go run main.go the output with these predefined values should be: If you access an array, bytesN or an array slice at an out-of-bounds or negative index (i.e. x[i] where i >= x.length or i < 0).

Which means that EVM Panic code 0x32 was thrown, which is actually really helpful error message.

Measure gas costs

Probably the most common error is the following one: ReceiverError. This may be due to an out of gas error on the destination chain. Error code: 0x

This most likely means that you've used extraArgs: "" syntax which defaults to 200_000 for gasLimit while your ccipReceive function consumes more.

To solve this error, you will most likely just need to, connect your wallet to CCIP Explorer, set the "Gas limit override" field to appropriate value and trigger manual execution of this function by clicking the blue button.

In future, to prevent this issue, you can use the following syntax, if you want to set the gas limit to 500_000 gas for example:

extraArgs: Client._argsToBytes(
    Client.EVMExtraArgsV1({gasLimit: 500_000})
)

Keep in mind that:

• If you are transferring only tokens, gasLimit should be set to 0 because there is no ccipReceive function

• The extraArgs should mutable if you want to make your dApp compatible with future CCIP upgrades (https://docs.chain.link/ccip/best-practices#using-extraargs)

Tenderly is your friend

Oftentimes Tenderly is a good choice when you want to debug or simulate transactions against live test networks. Full trace feature allows you to see where the function actually reverts, which is much helpful than a below error message from at Block explorer.

One of the biggest problems when working with unverified smart contracts is function tracing. This smart contract wasn't verified, but the function selector was still known, and that's why Polygonscan was able to display it. This is sometimes very helpful.

Function selectors are the first four bytes of the Keccak-256 hash of the function's signature. The function signature includes the function name and the parenthesized list of parameter types. When a function call is made, these first four bytes are used by the EVM to determine which specific function in the contract should be executed.

So the exact function call that failed was triggered by the unverified smart contract to the CCIP Router smart contract, and its function selector is 0x96f4e9f9. If we have the ABI of that smart contract finding the function selector will be extremely straightforward. But what if we don't? Well there is a https://openchain.xyz/signatures, website which has a database of all known function signatures so we can try searching for ours.

Looks like we found the function which reverted. Now we can take a look at its implementation and find out what went wrong. Although it was already clear from the Tenderly's UI that the issue is that CCIP Sender contract was not funded with tokens for covering CCIP fees - which is one of the most common User errors.