How to Research and Analyze EVM Mempool Data Using QuickNode

In the bustling ecosystem of blockchain transactions, a crucial part to understanding the underlying dynamics is the analysis of the Ethereum Virtual Machine (EVM) mempool, a holding space for transactions awaiting confirmation. Analyzing the data within this area can offer rich insights into the state of the network at any given time. Leveraging an RPC provider like QuickNode can greatly facilitate this analysis. In this article, we guide you step by step through researching and analyzing EVM mempool data using QuickNode.

Setting Up Your QuickNode Account

QuickNode is the best RPC provider and blockchain development platform on the market. It’s often praised for its comprehensive features, high performance, and dedicated support, making it a preferred choice for many developers and projects seeking reliable access to blockchain networks through its optimized nodes and user-friendly interface.

Head over to Quicknode.com to create a new free account and create a BSC Mainnet Endpoint. If you love the service (which I’m sure you will) use coupon code abqNVcwd to get $20 off if upgrade to the Build or Scale plan.

Also check out their Getting Started guide to ensure your making the most out of your QuickNode Subscription.

Understanding the Basic RPC Calls

To dissect EVM mempool data, acquaint yourself with essential JSON-RPC methods like txpool_content and eth_getTransactionByHash, pivotal to query the mempool effectively.

Getting Started with QuickNode

Step 1: Setting Up Your Development Environment

Equip your development environment with necessary tools like curl or Postman to facilitate HTTP POST requests to the Ethereum network via QuickNode.

Step 2: Making RPC Calls

Fetching Mempool Data with txpool_content

Initiate an HTTP POST request to your QuickNode URL using the txpool_content method to fetch mempool data.

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"txpool_content","params":[],"id":1}' YOUR_QUICKNODE_URL

The returned data typically comprises pending and queued transactions, encapsulating a wealth of information:

Structure of txpool_content Response:
Pending Transactions:
  • from: The sender’s address.
  • gas: The gas limit provided by the sender.
  • gasPrice: The price per unit of gas, specified in wei.
  • hash: The unique identifier of the transaction.
  • input: Encoded data representing contract method calls and arguments (if any).
  • nonce: The number of transactions sent from the sender’s address.
  • to: The recipient’s address (or null for contract creation transactions).
  • value: The amount of wei transferred in the transaction.
Queued Transactions:

Transactions that are in a waiting state due to reasons such as lower nonce or insufficient gas prices.

Fetching Specific Transaction Details with eth_getTransactionByHash

To retrieve details of a particular transaction, employ the eth_getTransactionByHash method.

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["TRANSACTION_HASH"],"id":1}' YOUR_QUICKNODE_URL

The response structure outlines various details like:

Structure of eth_getTransactionByHash Response:
  • blockHash: The hash of the block containing the transaction (null if pending).
  • blockNumber: The block’s number (null if pending).
  • from: The sender’s address.
  • gas: The gas limit set by the sender.
  • gasPrice: The gas price specified by the sender in wei.
  • hash: The transaction’s unique hash.
  • input: Encoded data showcasing contract method calls and parameters (for contract interactions).
  • nonce: The count of transactions sent from the sender’s address.
  • to: The recipient’s address (null for contract creations).
  • transactionIndex: The transaction’s position in the block (null if pending).
  • value: The number of wei transferred.
  • v, r, s: Cryptographic elements vital for verifying the transaction’s authenticity.

Step 3: Delving into Data Analysis

Analyzing Transaction Patterns

With the retrieved data, delve into transaction patterns like frequency, gas prices, and nonce to gauge the network’s current dynamics and forecast potential trends.

Fee Estimation

Leverage the data to dissect the ongoing fee market. This insight assists in approximating a fitting gas fee for swift transaction confirmations.

Step 4: Furthering Research

Network Congestion Analysis

Evaluate network congestion by scrutinizing the volume and magnitude of transactions in the mempool. This study aids in formulating strategies for optimal transaction timing.

Probing Transaction Strategies

Explore the transaction tactics utilized by various entities, offering a glimpse into how different players optimize their strategies.

Use Cases

Developing Dynamic Fee Algorithms

Harness the analyzed data to craft dynamic fee algorithms that adjust transaction fees based on the mempool’s current state, ensuring efficient transactions even during network congestion.

Crafting Transaction Monitoring Tools

Design tools that monitor transactions, furnishing users with real-time updates on transaction statuses and probable confirmation times.

Enhancing Research and Analysis

Utilize mempool data for deep analysis of network behavior, fostering the creation of optimization strategies and network improvement proposals.

Conclusion

Researching and analyzing EVM mempool data using QuickNode opens doors to a treasure trove of insights into the vibrant landscape of the Ethereum network. This data serves as a cornerstone for the development of advanced monitoring tools, dynamic fee algorithms, and an array of research opportunities. By delving deep into this data, developers and researchers can contribute significantly to the growing and dynamic blockchain ecosystem, fostering innovation and growth.

Demystifying EVM Internal Transactions and Analysis Using RPC Methods

If you’ve dabbled with Ethereum or any blockchain operating under the Ethereum Virtual Machine (EVM), you might have encountered the term “internal transactions.” These elusive components of the blockchain ecosystem often lead to confusion. Today, we’ll uncover what EVM internal transactions are and how you can analyze them using Remote Procedure Call (RPC) methods.

What Are EVM Internal Transactions?

At a high level, the Ethereum blockchain documents transactions made from one account to another. These are your typical “external transactions” – they’re visible, explicit, and you can find them in any Ethereum block explorer.

However, there’s another layer of transactions happening underneath, the so-called “internal transactions”. These are not explicit transfers of ETH, but are rather operations that occur due to smart contract execution.

For example, suppose you interact with a Decentralized Finance (DeFi) contract to swap one token for another. While the external transaction is your instruction to the contract, the internal transactions could be the contract sending you tokens, deducting fees, interacting with another contract, and so forth.

Analyzing Internal Transactions Using RPC Methods

Ethereum provides a set of RPC methods which developers can use to interact with the network. Some of these methods can be employed to dig deeper into internal transactions:

  1. eth_getTransactionByHash: Begin by retrieving the transaction details using its hash. This provides an overview of the external transaction, including the input data which is essential for decoding smart contract interactions.
  2. trace_replayTransaction: This is a part of the “trace” module, which may not be enabled in all Ethereum nodes. However, it’s invaluable for our purpose. By replaying the transaction, it returns a detailed breakdown of all the operations (including internal transactions) that happened during its execution.
  3. eth_getCode: If you want to delve deeper, this method retrieves the contract code of a given address. This can be vital for understanding what a particular contract is designed to do, especially when observing its internal transactions.
  4. Decoding Contract Input: The input data from our external transaction (retrieved in the first step) can often be decoded using the ABI of the respective contract. This will allow you to understand the specific function called and any parameters passed.
  5. Tools and Platforms: While RPC methods are powerful, there’s a learning curve. Platforms like Etherscan have made it simpler by offering a graphical interface to view internal transactions. However, understanding how to do it manually via RPC gives you flexibility and a deeper understanding.

Key Takeaways:

  • Internal transactions are operations triggered by contract executions. They are not explicitly recorded as regular transactions but are intrinsic to understanding on-chain activity.
  • RPC methods provide a gateway to retrieve and analyze these internal transactions. trace_replayTransaction is particularly useful for this purpose.
  • Familiarizing oneself with these methods and techniques enhances one’s grasp over Ethereum’s intricate workings, ensuring better clarity when developing or interacting with smart contracts.

In a decentralized world powered by smart contracts and blockchain, understanding the underpinnings of these technologies, like internal transactions, becomes invaluable. Embrace these tools and methods to unlock deeper insights into the Ethereum ecosystem.

Programmatically retrieving blockchain data – Part 3

Do you know how to retrieve data from EVM based chains but are lost with Solana? Consider this post as a primer to Solana for the developer already comfortable with EVM based chains. In this post I discuss various Ethereum Virtual Machine (EVM) based RPC methods and their equivalent counterparts in the Solana blockchain. This post compares and contrasts the methods in terms of their functionality and usage. I give step-by-step guidance and code examples to bridge the gap between the two ecosystems.

Common EVM based methods and their equivalent

eth_getBlockByNumber

The equivalent Solana RPC method for the Ethereum method eth_getBlockByNumber would be the Solana RPC method getBlock. However, it’s important to note that Solana and Ethereum have different structures and terminologies when it comes to their blockchains, so there might be some differences in the data returned and the way you interact with these methods.

In Ethereum, eth_getBlockByNumber allows you to retrieve a block’s information using its block number or block tag. Similarly, in Solana, getBlock allows you to retrieve block information by providing the block’s slot (which is analogous to the block number in Ethereum).

Here’s an example of how you might use the getBlock method in Solana:

# Using solana-cli
solana block <block_slot>

If you’re interacting with the Solana RPC API programmatically, you can use a library like solana-web3.js for JavaScript to call this method:

const web3 = require('@solana/web3.js');

const solanaRpcUrl = 'QUICKNODE_URL'; // Replace with the quicknode url
const connection = new web3.Connection(solanaRpcUrl, 'confirmed');

async function getSolanaBlock(slot) {
  const block = await connection.getBlock(slot);
  console.log(block);
}

const blockSlot = 12345678; // Replace with the desired block slot
getSolanaBlock(blockSlot);

eth_getTransactionReceipt

The Ethereum method eth_getTransactionReceipt retrieves the receipt of a transaction, including information about contract events and logs. In Solana, there isn’t a direct equivalent method because Solana and Ethereum have different transaction models and data structures. However, you can achieve similar functionality using a combination of Solana’s RPC methods and programming features.

To get similar information as eth_getTransactionReceipt in Solana, you would need to follow these steps:

  1. Use Solana’s RPC Methods to Get Transaction Information:
    1. Use the getTransaction Solana RPC method to retrieve transaction details.
  2. Decode Logs and Events:
    1. Solana doesn’t inherently have the concept of logs like Ethereum. Instead, you would need to design your Solana program to emit custom events or data that you’re interested in tracking.
    2. Parse and decode these custom events from the transaction’s transaction message or transaction logs.
  3. Construct a Custom Response:
    1. Based on the decoded events and information, you can construct a custom response object that resembles the information you would get from an Ethereum transaction receipt.

Here’s a simplified example of how you might approach this using JavaScript and Solana’s @solana/web3.js library:

const web3 = require('@solana/web3.js');

async function getSolanaTransactionReceipt(transactionSignature) {
  const solanaRpcUrl = 'QUICKNODE_URL'; // Replace with the quicknode url
  const connection = new web3.Connection(solanaRpcUrl, 'confirmed');

  const transaction = await connection.getTransaction(transactionSignature);

  if (!transaction) {
    return null; // Transaction not found
  }

  // Parse and decode custom events/logs from transaction.message
  const decodedEvents = parseAndDecodeEvents(transaction.message.logs);

  const receipt = {
    transactionHash: transactionSignature,
    blockNumber: transaction.slot, // Solana's equivalent of block number
    // Other relevant information based on your decoded events
    decodedEvents,
  };

  return receipt;
}

function parseAndDecodeEvents(logs) {
  // Implement logic to parse and decode logs into custom event data
  // This could involve iterating through logs and extracting data
  // Refer to Solana's documentation and your program's event structure

  return decodedEvents;
}

const transactionSignature = '...'; // Replace with the Solana transaction signature
getSolanaTransactionReceipt(transactionSignature)
  .then((receipt) => {
    console.log(receipt);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

Remember that Solana’s design and data structures are different from Ethereum’s, so you’ll need to adjust your approach accordingly. The example provided is a high-level guideline, and you’ll need to customize it based on your specific Solana program and use case.

eth_getCode

The Ethereum method eth_getCode is used to retrieve the bytecode of a smart contract deployed on the Ethereum blockchain. The comparable Solana RPC method for this is getAccountInfo, which can be used to retrieve information about an account, including the data associated with it.

Here’s how you can use the Solana getAccountInfo method in comparison to Ethereum’s eth_getCode:

Ethereum eth_getCode:

// Using web3.js for Ethereum
const Web3 = require('web3');
const web3 = new Web3('QUICKNDOE_URL'); // Replace with the quicknode url

async function getEthContractBytecode(address) {
  const bytecode = await web3.eth.getCode(address);
  return bytecode;
}

const contractAddress = '0x...'; // Replace with the Ethereum contract address
getEthContractBytecode(contractAddress)
  .then((bytecode) => {
    console.log(bytecode);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

Solana getAccountInfo:

const web3 = require('@solana/web3.js');

async function getSolanaContractData(address) {
  const solanaRpcUrl = 'QUICKNODE_URL'; // Replace with the quicknode url  
  const connection = new web3.Connection(solanaRpcUrl, 'confirmed');

  const publicKey = new web3.PublicKey(address);
  const accountInfo = await connection.getAccountInfo(publicKey);

  if (!accountInfo) {
    return null; // Account not found
  }

  const contractData = accountInfo.data;
  return contractData;
}

const contractAddress = '...'; // Replace with the Solana contract address
getSolanaContractData(contractAddress)
  .then((contractData) => {
    console.log(contractData);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

In the Solana example, getAccountInfo retrieves information about the specified account, and the data field of the account info object contains the bytecode or data associated with the contract. Keep in mind that Solana accounts can hold various types of data, not just bytecode, so the contents of the data field might need further processing based on your specific use case.

Adjust the examples according to your use case and the libraries you’re using. Solana’s design and terminology differ from Ethereum’s, so the way you handle contracts and their data will have some distinctions.

eth_call

The Ethereum method eth_call is used to execute a call to a contract’s function or to retrieve contract state without making an actual transaction on the Ethereum network. The comparable Solana RPC method for this is simulateTransaction, which allows you to simulate a transaction and obtain the result of executing a program (smart contract) on the Solana blockchain.

Here’s how you can use the Solana simulateTransaction method in comparison to Ethereum’s eth_call:

Ethereum eth_call:

// Using web3.js for Ethereum
const Web3 = require('web3');
const web3 = new Web3('QUICKNODE_URL');  // Replace with the quicknode url 

async function ethCall(contractAddress, data) {
  const result = await web3.eth.call({
    to: contractAddress,
    data: data,
  });
  return result;
}

const contractAddress = '0x...'; // Replace with the Ethereum contract address
const inputData = '0x...'; // Replace with the input data for the function call
ethCall(contractAddress, inputData)
  .then((result) => {
    console.log(result);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

Solana simulateTransaction:

const web3 = require('@solana/web3.js');

async function simulateSolanaTransaction(sender, programId, data) {
  const solanaRpcUrl = 'QUICKNODE_URL'; // Replace with the quicknode url  
  const connection = new web3.Connection(solanaRpcUrl, 'confirmed');

  const instruction = new web3.TransactionInstruction({
    keys: [{ pubkey: sender, isSigner: true, isWritable: true }],
    programId,
    data,
  });

  const simulatedTransactionResponse = await connection.simulateTransaction(
    new web3.Transaction().add(instruction)
  );

  const simulatedResult = simulatedTransactionResponse.value;
  return simulatedResult;
}

const senderPublicKey = new web3.PublicKey('...'); // Replace with the sender's public key
const programId = new web3.PublicKey('...'); // Replace with the Solana program ID
const inputData = Buffer.from('...'); // Replace with the input data for the program
simulateSolanaTransaction(senderPublicKey, programId, inputData)
  .then((simulatedResult) => {
    console.log(simulatedResult);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

In the Solana example, simulateTransaction allows you to simulate executing a program (smart contract) using a transaction instruction. Keep in mind that the setup of Solana programs and transactions is different from Ethereum’s, so you’ll need to adapt your code accordingly. Solana’s architecture and design may require additional considerations and adjustments compared to Ethereum’s EVM-based approach.

eth_getBalance

The Ethereum method eth_getBalance is used to retrieve the balance of a specific Ethereum address. The comparable Solana RPC method for this is getBalance, which allows you to retrieve the balance of a specific Solana wallet address.

Here’s how you can use the Solana getBalance method in comparison to Ethereum’s eth_getBalance:

Ethereum eth_getBalance:

// Using web3.js for Ethereum
const Web3 = require('web3');
const web3 = new Web3('QUICKNODE_URL'); // Replace with the quicknode url

async function getEthBalance(address) {
  const balance = await web3.eth.getBalance(address);
  return balance;
}

const ethAddress = '0x...'; // Replace with the Ethereum address
getEthBalance(ethAddress)
  .then((balance) => {
    console.log(balance);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

Solana getBalance:

const web3 = require('@solana/web3.js');

async function getSolanaBalance(address) {
  const solanaRpcUrl = 'QUICKNODE_URL'; // Replace with the quicknode url 
  const connection = new web3.Connection(solanaRpcUrl, 'confirmed');

  const publicKey = new web3.PublicKey(address);
  const balance = await connection.getBalance(publicKey);
  return balance;
}

const solanaAddress = '...'; // Replace with the Solana wallet address
getSolanaBalance(solanaAddress)
  .then((balance) => {
    console.log(balance);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

In both cases, you’re retrieving the balance of a specific address, but keep in mind that the address formats and network-specific details will differ between Ethereum and Solana. Solana uses the Ed25519 public key format for addresses, whereas Ethereum uses the hexadecimal format.

Please replace the placeholders with actual addresses and adapt the code according to the libraries and tools you are using. Solana and Ethereum have different architectures and designs, so the code and approach might vary in certain aspects.

debug_traceBlockByHash

Solana’s architecture and design are quite different from Ethereum’s, and while Ethereum provides the debug_traceBlockByHash method to trace and debug blocks, Solana’s RPC methods and tools might not have a direct equivalent with the same level of detail.

In Ethereum, debug_traceBlockByHash allows for detailed tracing and analysis of the execution and state changes within a block. Solana, on the other hand, primarily focuses on high throughput and performance, which might lead to differences in how debugging and tracing are implemented.

Solana’s RPC methods does not offer a direct equivalent for tracing blocks with the same level of detail as Ethereum’s debug_traceBlockByHash. Solana’s design prioritizes speed and scalability, and its focus on parallel processing might make tracing every detail in a block less practical.

However, for debugging and tracing purposes, you might find the following Solana tools and methods useful:

  • Transaction Explorer: Solana provides a transaction explorer on its block explorer websites (e.g., Solana Explorer). While it might not offer the same level of detail as Ethereum’s block tracing, it can help you visualize transactions and program execution within blocks.
  • Solana CLI and Logging: The Solana Command-Line Interface (CLI) provides commands to interact with the Solana network. While it might not provide the same tracing capability as Ethereum’s debugging methods, it does offer tools for monitoring transactions and program logs.
  • Program Logs: Solana programs can emit logs, and these logs can be retrieved using the getLogs RPC method. These logs can provide insights into the execution of Solana programs.
  • Developer Community: As Solana’s ecosystem evolves, new tools and techniques might emerge for debugging and tracing. It’s recommended to follow Solana’s official channels and developer community discussions for updates and best practices.

Conclusion

Bridging the conceptual and practical gap between EVM-based blockchains and Solana is paramount for developers aiming to diversify their expertise across these platforms. This post illuminated the equivalencies and disparities between Ethereum and Solana RPC methods, from block retrieval to transaction simulation. While Ethereum’s intricate methods offer depth, Solana, with its emphasis on high throughput and scalability, presents its own set of unique methods tailored to its architecture. For the discerning developer, the nuances in interaction, structure, and design between the two ecosystems should serve as a roadmap, guiding them in extracting maximum utility from both platforms. As the blockchain space continues to evolve, it’s imperative to stay updated, embracing new tools, techniques, and best practices that emerge within these dynamic ecosystems.

Programmatically retrieving blockchain data – Part 2

Introduction

In my last blog post Programmatically retrieving blockchain data – Part 1, I share a thorough guide on accessing and analyzing blockchain data programmatically. I emphasize the significance and advantages of this approach, highlighting benefits like real-time insights, transparency, trust, tailored analysis, automated auditing, smart contract oversight, innovation, research, and risk control. I walk readers through a detailed process of extracting transaction data from BSC Mainnet. This includes steps like registering with an RPC provider (where I particularly suggest QuickNode), procuring a BSC Mainnet Endpoint, fetching the most recent block number, sifting through that block for transactions, and then delving deep into the transaction details. Throughout the article, I’ve incorporated specific RPC methods and coding illustrations to facilitate a deeper grasp and practical application for my readers.

In this post I will walk through how to automate the process of getting transaction details using Node.js, a common library called web3, and using QuickNode as an RPC Provider.

Exercise

In this blog post, we’ll delve into a Node.js script that uses Web3 to fetch and display transaction details from a specific Ethereum-like blockchain.

Web3 Node.js Library

The web3.js library in Node.js is a powerful tool designed to facilitate interactions with the Ethereum blockchain. It enables developers to connect to local or remote Ethereum nodes, send transactions, deploy and interface with smart contracts, manage Ethereum accounts, and access blockchain data. With its modular design, web3.js abstracts the complexities of Ethereum, providing a streamlined JavaScript API for building decentralized applications and systems on the Ethereum network.

Source Code

https://github.com/lvandeyar/programmatically-retrieving-blockchain-data

Understanding the code

Let’s break down the provided code step by step:

1) Shebang Line

  #!/usr/bin/node

This line specifies the interpreter for the script. It indicates that the script should be run using the Node.js runtime.

2) Importing Web3

const { Web3 } = require('web3');

The code imports the Web3 class from the ‘web3’ package, allowing us to interact with Ethereum-compatible blockchains.

3) Blockchain Node URL

var quicknodeurl = 'YOUR_QUICKNODE_URL';

This variable holds the URL of a blockchain node. In this case, it’s a QuickNode URL, which serves as the connection point to the blockchain network.

4) Provider URL Setup

const providerUrl = quicknodeurl;
const web3 = new Web3(providerUrl);

Here, the providerUrl is set to the previously defined QuickNode URL, and a new instance of the Web3 class is created using this provider URL. This instance enables interaction with the blockchain.

5) Fetching Transaction Details

async function getTransactionDetails(transactionHash) {
    // ...
}

This function takes a transaction hash as a parameter and asynchronously fetches transaction details using the web3.eth.getTransaction() method. It then logs the retrieved transaction details to the console.

6) Fetching Transactions in a Block

async function getBlockTransactions(blockNumber) {
    // ...
}

This function takes a block number as input and fetches details about transactions within that block using the web3.eth.getBlock() method. It logs the block number and associated transaction hashes to the console. It also iterates through the transaction hashes and fetches details for each transaction using the getTransactionDetails() function.

7) Fetching All Blocks

async function fetchAllBlocks() {
    // ...
}

This function fetches the latest block number using web3.eth.getBlockNumber() and iterates through blocks from the latest to the genesis block (block number 0). For each block, it calls the getBlockTransactions() function to retrieve and display transaction details.

8) Executing the Fetching Process

fetchAllBlocks();

Finally, the fetchAllBlocks() function is called to initiate the process of fetching and displaying transaction details for all blocks.

Conclusion

In this blog post, we explored a Node.js script that interacts with an Ethereum-compatible blockchain using the Web3 library. The script fetches transaction details and logs them to the console, providing insights into the blockchain’s transaction history. This code can be a useful starting point for anyone looking to programmatically explore and analyze blockchain data. Whether you’re a developer, researcher, or blockchain enthusiast, this example demonstrates how to harness the power of Web3 and Node.js for blockchain interaction.

Programmatically retrieving blockchain data – Part 1

Introduction

Welcome to the first installment of this multi-part series, Programmatically Retrieving Blockchain Data. In this series, I will delve into the exciting world of blockchain data retrieval and explore its various aspects, applications, and advantages. Blockchain technology has revolutionized industries ranging from finance to supply chain management, and understanding how to programmatically access and analyze blockchain data can provide invaluable insights and opportunities. In this first part, i will focus on the benefits of programmatically retrieving blockchain data and why it matters in today’s digital landscape.

Benefits and use-cases for Programmatically Retrieving Blockchain Data

  • Real-Time Insights – Programmatically accessing blockchain data enables developers, analysts, and researchers to obtain real-time insights into transactions, smart contracts, and network activities. This up-to-the-minute information can empower decision-makers to make informed choices based on the most current data available.
  • Transparency and Trust – Blockchains are known for their transparency and immutability. By programmatically retrieving data, users can independently verify transactions, ensuring that information has not been tampered with. This level of transparency enhances trust among stakeholders and reduces the need for intermediaries.
  • Customized Analysis – Blockchain data can be vast and complex. By programmatically querying and analyzing specific aspects of the blockchain, users can extract customized insights tailored to their needs. This flexibility allows for targeted research, fraud detection, and trend identification.
  • Automated Auditing – Traditional auditing processes can be time-consuming and prone to human error. Programmatically retrieving blockchain data streamlines auditing procedures, making them more efficient and accurate. Auditors can verify transactions and compliance measures with greater ease.
  • Smart Contract Monitoring – Smart contracts are self-executing agreements with predefined conditions. Programmatic access to blockchain data enables real-time monitoring of smart contracts, ensuring that they function as intended and triggering alerts or actions when specific conditions are met.
  • Innovation and Research – Developers can leverage programmatically retrieved blockchain data to build new applications, services, and products. Researchers can study transaction patterns, user behaviors, and network dynamics to drive innovation in various industries.
  • Risk Management – Enterprises can use programmatically retrieved blockchain data to assess risks associated with specific transactions or addresses. This data-driven approach enhances risk management strategies and helps prevent fraudulent activities.

Exercise

In this post, we’ll be programmatically accessing transaction data from BSC Mainnet. A blockchain transaction is a digital record representing the movement of data, assets, or information between participants on a blockchain network, authenticated by cryptographic signatures, assigned a unique identifier (transaction hash), and confirmed through a consensus mechanism, thereby ensuring transparency, security, and immutability of the transaction’s details within the blockchain’s ledger.

We’ll be doing the following steps in this exercise:

  • Sign up for an RPC provider and obtain a BSC Mainnet Endpoint
  • Get the latest block number from BSC Mainnet
  • Parse that block for transaction
  • Return the transaction data for that transaction

Step 1 – Sign up for an RPC Provider and obtain a BSC Mainnet Endpoint

In order to interact with the blockchain programmatically, you need to access the RPC endpoint of the chain. You could use the shared public endpoint provided by the foundation or you sign up for an endpoint with an RPC Provider. A blockchain RPC provider is a software interface that allows external applications to communicate and interact with a blockchain network by making remote procedure calls to access data and services. Using a paid RPC provider offers enhanced reliability, performance, and dedicated support compared to public endpoints, ensuring consistent and high-quality access to blockchain data and services.

QuickNode

QuickNode is the best RPC provider and blockchain development platform on the market. It’s often praised for its comprehensive features, high performance, and dedicated support, making it a preferred choice for many developers and projects seeking reliable access to blockchain networks through its optimized nodes and user-friendly interface.

Head over to Quicknode.com to create a new free account and create a BSC Mainnet Endpoint. If you love the service (which I’m sure you will) use coupon code abqNVcwd to get $20 off if upgrade to the Build or Scale plan.

Also check out their Getting Started guide to ensure your making the most out of your QuickNode Subscription.

Step 2 – Get the latest block number

Blockchain transactions are individual records of data or value transfers, and they are grouped together in blocks, which are sequentially linked to form a chronological and immutable chain of transactions in a blockchain.

We are going to use a RPC method eth_blockNumber to retrieve the current and most recent block number.

curl $INSERT_QUICKNODE_BSC_ENDPOINT  \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}'

{"jsonrpc":"2.0","id":1,"result":"0xccb93d"}%

The payload from eth_blockNumber returns a hexadecimal string representing the current block number on the blockchain. This number indicates the position of the latest block in the blockchain’s sequence. In this case, the response payload is "0xccb93d", it corresponds to the decimal block number 13416765.

Step 3 – Parse block the block for transactions

We are going to use the RPC method eth_getBlockByNumber to retrieve detailed information about a specific block on the blockchain based on its block number or block tag. This method will return a comprehensive set of data associated with a particular block, including its hash, timestamp, transaction list, and other relevant details.

The payload from eth_getBlockByNumber returns the following key attributes.

  • number – The block number (as a hexadecimal string) or null if the block is not found.
  • hash – The block’s unique hash.
  • parentHash – The hash of the previous block.
  • nonce – A proof-of-work consensus algorithm value.
  • sha3Uncles – A hash of the uncles (stale blocks).
  • logsBloom – A bloom filter related to logs and transactions.
  • transactionsRoot – The root of the transaction trie.
  • stateRoot – The root of the state trie.
  • receiptsRoot – The root of the receipts trie.
  • miner – The address of the miner who created the block.
  • difficulty – The block’s difficulty value.
  • totalDifficulty – The cumulative difficulty of the chain up to this block.
  • extraData – Extra data associated with the block.
  • size – The size of the block in bytes.
  • gasLimit – The maximum gas allowed in the block.
  • gasUsed – The total gas used by all transactions in the block.
  • timestamp – The UNIX timestamp of when the block was mined.
  • transactions – An array of transaction objects included in the block.
  • uncles – An array of uncle blocks.
curl $INSERT_QUICKNODE_BSC_ENDPOINT \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"eth_getBlockByNumber","params":["0xccb93d",false],"id":1,"jsonrpc":"2.0"}'

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "number": "0xccb93d",
    "logsBloom": "0x3dff37f45b61fdfae68a3bddefe7419b3c0885c4be39d62fd5e646be77feb3d05378fe9fbf8435dfdffedabc2c2da542ab7588da78cab95f5b224c85fdbef7dfebfec6738fac217d3b5fff6dcc7da7b4fedbeae3f366fe8fb3fdfbc7cffb526affcefceaa366cbd766fbf3769ba3efd37b3929edaa75e7d676eff9f514d4611ab0ec97a7cc9fbd640aedb56dfd103db111a45dc5f97fb5ac3fb733c4cc18ff77e62fd9f276cd9f1fd8eb77484f6766f84ff7e59b32d6fac5cee04b9b34879aceeff15c8e337b01bb3c993fc965d7bed9ffedbe17e59b39b577ef5feff555ed6afbf537fdbdb0f9a8e965f314bf64dbe6d8cf2e1eff7f7a7fff77daac7f4fafad",
    "stateRoot": "0x2017ad7c1515b4dcf6fd9ef58970303abdc557862fc0d25cf215ce75fa1a3965",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "totalDifficulty": "0x1969e02",
    "receiptsRoot": "0x0c9c9c142ecccacd291bf5655b7d0ddf5c91dae15541526ff37d6418d995f8a0",
    "hash": "0x8779f0e31c5c3135273189cf58090c5831a23b80b3abd936d0d9f246ca35af74",
    "gasUsed": "0x3afec16",
    "timestamp": "0x61b633b0",
    "miner": "0xa6f79b60359f141df90a0c745125b131caaffd12",
    "difficulty": "0x2",
    "nonce": "0x0000000000000000",
    "size": "0xdfa7",
    "parentHash": "0xc82115cf6f2e42f3dcae41772677ba94d23f06d2544cc99b9b1913272e7ce366",
    "uncles": [],
    "gasLimit": "0x4e0a6df",
    "transactionsRoot": "0x34a104897ae467f78dc372d446000b3347961711915cd75629af40a3e914d783",
    "extraData": "0xd983010107846765746889676f312e31362e3130856c696e75780000c3167bdf878e0bace93cb22a5c4e2f32ed8e6f9631e9b2a1c581ad28709333d6f4e5ec8c0749b0d030a62e53e398932d120e59125166ab7f5d530b4dc5ffaffd4ce9679e01",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "transactions": [
      "0x5326574007e795ba4f51d09d4e79ee66ff1a3136b43b89e7f80da2c943612061",
      "..truncated"
    ]
  }
}

Step 4 – Get the transaction information

We’ll be examining the first transaction returned from eth_getBlockByNumber call. We’ll be using the eth_getTransactionByHash method to retrieve information about a specific transaction on the Ethereum blockchain by providing its unique transaction hash.

When you send a transaction on the Ethereum network, it gets assigned a unique hash, which serves as its identifier. This hash is derived from the transaction’s content, including details like sender address, recipient address, value, data, gas price, and more. It’s a cryptographic fingerprint of the transaction.

curl $INSERT_QUICKNODE_BSC_ENDPOINT \
-H "Content-Type: application/json" \
--data '{"method":"eth_getTransactionByHash","params":["0x5326574007e795ba4f51d09d4e79ee66ff1a3136b43b89e7f80da2c943612061"],"id":1,"jsonrpc":"2.0"}' 

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "hash": "0x5326574007e795ba4f51d09d4e79ee66ff1a3136b43b89e7f80da2c943612061",
    "to": "0x03ab98f5dc94996f8c33e15cd4468794d12d41f9",
    "r": "0xb204ebf87f50f638895d47bb05d73f04a924a820e07376363b2d519c0a6d4ae2",
    "chainId": "0x38",
    "from": "0xa14932d5312de6e6e2fc4848e46ca6e47921a72d",
    "blockHash": "0x8779f0e31c5c3135273189cf58090c5831a23b80b3abd936d0d9f246ca35af74",
    "gasPrice": "0x5d21dba00",
    "type": "0x0",
    "nonce": "0x1861",
    "input": "0x51cff8d900000000000000000000000022d5b727e9688e8b7203c60ecbafbee8808f8224",
    "s": "0x45240a3b02d7a44c3f0c9d5a50ff3c157a89b95ce65428c96d1503fcb67c4359",
    "v": "0x93",
    "blockNumber": "0xccb93d",
    "gas": "0x30d40",
    "value": "0x0",
    "transactionIndex": "0x0"
  }
}

By using the eth_getTransactionByHash method, you can query an Ethereum node for detailed information about a transaction based on its hash. The information returned typically includes:

  • Block Information – The block number and block hash in which the transaction was included.
  • Transaction Index – The position of the transaction within the block’s list of transactions.
  • Sender (From) Address – The Ethereum address that initiated the transaction.
  • Recipient (To) Address – The Ethereum address that received the transaction (if it’s a contract execution, this may be empty).
  • Transaction Value – The amount of Ether (ETH) transferred in the transaction.
  • Gas and Gas Price – The gas used by the transaction and the price per unit of gas.
  • Input Data – Additional data sent with the transaction (often used for smart contract interactions).
  • Transaction Hash – The unique identifier of the transaction.
  • Nonce – A nonce is a counter that prevents replay attacks and helps order transactions from a sender.
  • V, R, S Values – Cryptographic values related to the transaction’s signature.

This method is particularly useful for retrieving and verifying transaction details, checking the status of a transaction (whether it’s pending, mined, or failed), and for building tools, explorers, or applications that require access to specific transaction information on the Ethereum blockchain.

Closing

Programmatically retrieving blockchain data opens up a realm of possibilities for real-time insights, transparency, customization, and innovation. In this multi-part series, we will explore the technical aspects of accessing blockchain data programmatically, including APIs, libraries, and coding examples. Stay tuned for the upcoming parts, where we will delve deeper into the tools and techniques to harness the power of blockchain data for your projects and endeavors.

Setting up a Filecoin Saturn Node

The opinions expressed are solely my own and do not express the views or opinions of my employer 😀.

Summary

While in Lisbon at the end of 2022 I heard about the Filecoin Saturn launch and thought this was a pretty cool idea to create an on-chain CDN on-top of IPFS. The idea intrigued me to go through the process to setup a node. My setup notes were pretty involved so I figured I’d share it with community.

What is Filecoin Saturn?

Filecoin Saturn is an open-source, community-run Content Delivery Network (CDN) built on Filecoin.

Saturn is a Web3 CDN in Filecoin’s retrieval market. On one side of the network, websites buy fast, low-cost content delivery. On the other side, Saturn node operators earn Filecoin by fulfilling requests.

Prerequisites

Acquire a server with the recommended specifications

At the time of writing this (January 2023) the specifications are as follows:

Minimum Requirements Recommended Requirements
CPU with 6 cores CPU with 12+ cores
32GB RAM 128GB+ RAM
1TB SSD storage 4TB+ SSD Storage
10Gbps upload link 10Gbps+ upload link

Provider selection

This workload has a variable cost for data transfer and is based on the usage of the CDN service. Choosing a cloud provider that has less variable cost is more desirable for positive profit margins.

Baremetal Scenario

Some baremetal providers include 20TB of egress traffic with overages being $1 per TB. In this scenario your bandwidth costs would be $10.

Math ( (30-20) * 1 = 10 )

Cloud Scenario

AWS charges $0.09 per GB for egress costs with 100GB included for free monthly. In this scenario, your bandwidth costs would be $2961.

Math ( (30,000-100) * 0.09 = 2691 )

Create a Filecoin wallet

In order to receive your payouts via FIL you must own a Filecoin wallet. Filecoin wallets allow you to manage FIL, Filecoin’s native token. Wallets store the private keys that allow you to authorize Filecoin transactions, including paying for storage deals and sending FIL to other accounts.

Non-custodial Filecoin Wallets

File coin supports the following wallet implementations:

  • Glif Wallet – A lightweight web interface to send and receive Filecoin with a Ledger device (instructions).
  • Glif Safe – A multisig wallet with a lightweight web interface to send and receive Filecoin with a Ledger device (instructions). Check out the Glif Safe GitHub repo for more information.
  • Lotus – Includes a command-line wallet that can manage bls, sec1p256k1, multisig wallets, and supports Ledger integration.

Creating a Filecoin Wallet on Coinbase

At the time of writing this post, I’m not super bullish on FIL and as a result will be immediately converting my FIL to another currency. For this reason, I chose to create a Filecoin wallet on Coinbase.

To do so, you simply buy 1 FIL on Coinbase with your currency of choice. Then you click on “Send & Receive” and select FIL – The FIL address is your wallet address.

Coinbase will display your wallet address and QR code for sharing. Here is my QR code:

Seriously though, you might want to keep your wallet address private. All information regarding payouts, transfers, etc are public on the blockchain. Also, this data is commonly indexed and made publicly available via block explorers. Filfox is the popular block explorer for FIlecoin.

Preparing for deployment

From the above section you should have:

  • A server with the requirements and the ability to login into as root
  • A Filecoin Wallet

Now it’s time to prepare your deployment environment. In this example, I’ll be using my laptop as my deployment host but you could easily use a bastion host.

Ansible Installation

Ansible is an agent-less automation platform used to configure one or many nodes. I ran the following commands to install ansible:

brew update && brew upgrade
brew install ansible
ansible --version

Ansible Setup

The Filecoin foundation has prepared an Ansible playbook to fully configure a Saturn node however we have to prepare our deployment machine to do so.

  • Install a required Ansible Module
ansible-galaxy collection install community.docker
  • Clone the repo prepared by Filecoin and CD into it
  • Create a file called inventory.yaml and add your hosts
saturn:
  hosts:
    moon1:
      ansible_host: IP_ADDRESS
      ansible_user: root
      ansible_ssh_private_key_file: ~/.ssh/KEY
  • Note
    • In my inventory file i have a parent called “saturn” – this is for targeting all hosts
    • I could also target individuals hosts – in my inventory file they are called “moons”
  • Test end to end connectivity by running the Ansible Ping command – For me, this returns a success “pong”
ansible -vvv -i inventory.yaml saturn -m ping

moon1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "data": "pong"
        }
    },
    "ping": "pong"
}
  • Next we need to install the speedtest cli by ookla – for this I created an ansible playbook called playbooks/speedtest.yml
---
- name: Speedtest playbook
  tags: speedtst
  hosts: all
  gather_facts: false
  tasks:
    
   - name: Download speedtest script
     become: yes
     get_url:
       url: https://packagecloud.io/install/repositories/ookla/speedtest-cli/script.deb.sh
       dest: /home
       mode: 0777
       
   - name: Copy and Execute the script 
     command: bash /home/script.deb.sh

   - name: Install speedtest
     apt:
      name: speedtest
  • Run the speedtest ansible playbook to install speed test
ansible-playbook -i inventory.yaml playbooks/speedtest.yml -vvvv
  • Then we need to use speedtest to get a list of servers closest to us
ansible -i inventory.yaml saturn -m shell -a "speedtest -L"

moon1 | CHANGED | rc=0 >>
Closest servers:

   ID  Name                           Location             Country
==============================================================================
 45489  Smart City Telecom             Lake Buena Vista, FL United States
 53543  frantic.link                   Kissimmee, FL        United States
 42450  NexGen Communications          Orlando, FL          United States
 10161  CenturyLink                    Orlando, FL          United States
 28307  Summit Broadband               Orlando, FL          United States
 40227  Flash Fiber                    Orlando, FL          United States
 48319  Whitesky Communications LLC    Orlando, FL          United States
 36808  832communications              Orlando, FL          United States
 25760  The Villages                   The Villages, FL     United States
 17170  Spectrum                       Tampa, FL            United States
  • Add the additional parameters to your .env file
SATURN_NETWORK="main or test"
FIL_WALLET_ADDRESS="WalletAddress"
NODE_OPERATOR_EMAIL="YourEmail"
SPEEDTEST_SERVER_CONFIG="--server-id=PickIDFromLastCommand"
SATURN_HOME="/home/"
  • I removed the section around hardening the OS in the l1.yaml playbook because I have a playbook specifically around this which. I won’t share my playbook for security reasons but you can easily research one or create your own. Please use your own automation or best practice to harden your system. Below is the edited version of l1.yaml.
---
- name: Configure playbook
  tags: config
  hosts: all
  gather_facts: true
  tasks:
    - name: Add docker GPG key
      become: true
      become_user: root
      apt_key:
        url: https://download.docker.com/linux/{{ ansible_distribution | lower }}/gpg
    - name: Add docker repository (yum)
      become: true
      become_user: root
      # ref - https://www.ansiblepilot.com/articles/install-docker-in-redhat-like-systems-ansible-module-rpm_key-yum_repository-and-yum/
      ansible.builtin.yum_repository:
        name: docker
        description: docker repository to CentOS
        baseurl: "https://download.docker.com/linux/centos/$releasever/$basearch/stable"
        enabled: true
        gpgcheck: true
        gpgkey: "https://download.docker.com/linux/centos/gpg"
        state: present
      when: ansible_distribution == "CentOS"
    - name: Add docker repository (apt)
      become: true
      become_user: root
      ansible.builtin.apt_repository:
        filename: docker
        repo: deb [arch=amd64] https://download.docker.com/{{ ansible_system | lower }}/{{ ansible_distribution | lower }} {{ ansible_distribution_release }} stable
        state: present
      when: ansible_distribution in ["Debian", "Ubuntu"]
    - name: Install aptitude
      become: true
      become_user: root
      ansible.builtin.package:
        name:
          - aptitude
        update_cache: true
      when: ansible_distribution in ["Debian", "Ubuntu"]
    - name: Install dependencies
      become: true
      become_user: root
      ansible.builtin.package:
        name:
          - wget
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-compose-plugin
          - python3-pip
        state: latest
        update_cache: true
    - name: Install python packages
      become: true
      become_user: root
      ansible.builtin.pip:
        executable: pip3
        name:
          - docker>5.0.0
        state: present
    - name: Start docker
      become: true
      become_user: root
      ansible.builtin.service:
        name: docker
        enabled: true
        state: started

- name: Run playbook
  tags: run
  hosts: all
  vars:
    # this wasn't consistent between runs, so we need to store it
    homedir: "{{ ansible_env.HOME }}"
    saturn_home: "{{ saturn_root if saturn_root is defined else homedir }}"
    env_file: "{{ lookup('file', '../.env' ) }}"
  tasks:
    - name: Copy the .env file
      ansible.builtin.copy:
        src: ../.env
        dest: "{{ saturn_home }}"
    - name: Patch SATURN_HOME
      ansible.builtin.blockinfile:
        path: "{{ saturn_home }}/.env"
        block: SATURN_HOME="{{ saturn_home }}"
    - name: Get docker-compose.yml
      become: true
      become_user: root
      ansible.builtin.get_url:
        url: https://raw.githubusercontent.com/filecoin-saturn/L1-node/main/docker-compose.yml
        dest: "{{ saturn_home }}/docker-compose.yml"
    - name: Ensure we have the right $SATURN_HOME/shared permissions
      become: true
      become_user: root
      ansible.builtin.file:
        path: "{{ saturn_home }}/shared"
        mode: "0755"
        state: directory
    - name: Run docker compose
      become: true
      become_user: root
      ansible.builtin.command: docker compose up -d
      # ansible.builtin.command: docker compose restart watchtower
      # ansible.builtin.shell: (docker rm -f saturn-node || true) && (docker rm -f saturn-watchtower) && docker compose up -d
      args:
        chdir: "{{ saturn_home }}"
  • Run the l1.yaml playbook
ansible-playbook -i inventory.yaml playbooks/l1.yaml -vvvv
  • Once complete, check out the logs on the host for any errors by running the following command “docker logs –follow saturn-node”. You can also see in the logs the id of your node.
  • You can check the status of your wallet ID by visiting: https://dashboard.strn.network/address/
  • You can also check the status of your node by visiting the following url and filtering by your node id: https://dashboard.strn.network/stats

Closing

I will be posting a retrospective of my performance sometime in February of 2023. If you run into issues and need help during your setup, check out the Filecoin Foundation’s Slack: https://filecoin.io/slack.