As blockchain networks’ state growth continues to grow at an exponential rate, the challenge of efficiently accessing and querying data becomes increasingly complex. Traditional methods of querying blockchain data through RPC nodes are proving to be inadequate for the scale and speed required by bleeding-edge onchain applications. This post explores the current state of blockchain RPC nodes, existing solutions, and how innovative approaches like SQD are offering a compelling alternative with its decentralized data lake architecture.
The Current State of RPC Nodes
Remote Procedure Call (RPC) nodes have been the default method for interacting with blockchain networks. They serve as the interface between applications and the blockchain.
Depending on the blockchain, several hundred to many thousands of nodes store and compute and hold the same data. This redundant architecture allows for emergent properties such as permissionless and censorship resistance, but it is also an inefficient use of hardware resources. Unlike centralized data centers (or decentralized data lakes) that scale linearly with every new machine added, blockchains do not scale when new nodes join the network. One way to scale the blockchain is by requiring higher resource requirements on validators and full nodes running the network to validate transactions and sign blocks faster, which raises the operating costs. But If the cost of running a node continues to increase, fewer participants will run nodes.
As transactions, accounts, and smart contracts continue to grow, so does the blockchains’ state. State explosion exposes several limitations of RPC nodes:
- Scalability Issues: RPC nodes struggle to handle the ever-increasing volume of blockchain data. For example, POLYGON’s archival node requires 12 TB of storage, while NEAR’s archival node demands a staggering 55 TB and grows by 5-7 TB monthly.
- Performance Bottlenecks: RPC nodes were designed primarily for consensus, not for efficient querying. As the blockchain state expands, querying historical data becomes increasingly slow and resource-intensive.
- Reliability Concerns: RPC nodes are prone to failures, crashes, and slowdowns, especially when under heavy load. Every blockchain RPC startup has faced this at some time or another.
- Limited Query Capabilities: RPC interfaces often lack advanced query functionality, making complex data analysis challenging.
- Vertical Scaling Limitations: The node software was not designed for horizontal scaling, making it difficult to distribute the load across multiple machines.
- Centralization Risks: Relying on individual RPC nodes or RPC providers can introduce single points of failure.
Current Solutions and Their Limitations
To address the shortcomings of RPC nodes, several solutions have emerged:
- Custom Indexing Layers: Many developers build their own indexing layers on top of RPC nodes, which can be time-consuming and resource-intensive. For example, the wallet Rainbow.
- Centralized API Services: Providers like Alchemy or Infura create their own indexing layers and offer enhanced APIs for blockchain data access as a SaaS business, but these solutions often come with high costs and add dependencies on centralized providers.
- The Graph: A decentralized indexing protocol that allows querying blockchain data through GraphQL. While powerful, the data is only accessible through GraphQL, supports limited blockchains, and has long ways to go for improving on blocks indexed per second.
Enter SQD
SQD: A modern take to blockchain data powered by a decentralized data lake
SQD network pulls in raw data from across 170+ blockchains, regularly integrating support for more.
SQD offers a unique approach to blockchain data access by implementing a decentralized data lake architecture. This innovative solution addresses many of the limitations faced by traditional RPC nodes and existing indexing solutions.
Key Features of SQD:
- Efficient Data Format: SQD indexes historical data from RPC nodes and stores it in an efficient format using Parquet and a custom Arrow-based query execution, significantly reducing storage requirements and improving query performance.
- Decentralized Architecture: The data lake is distributed across multiple nodes, allowing for horizontal scaling and improved reliability.
- Customizable Indexing: SQD indexing software enables developers to create tailored data pipelines for their specific needs.
- GraphQL API: Indexed data is exposed through a GraphQL API, providing a flexible and powerful query interface.
- Extensibility: The data lake architecture allows for additional features like real-time events, webhooks, and SQL querying. Catering to different developer needs.
- Cost-Effective: By optimizing data storage and access, SQD significantly reduces costs compared to traditional RPC-based solutions. The more SQD tokens locked, the more throughput unlocked per epoch. The longer a developer (or Portal) stakes SQD tokens, the closer initial costs approach zero over time.
SQD leverages the Parquet columnar storage format to optimize data storage and retrieval. Parquet organizes data by columns rather than rows, which is beneficial for analytical queries common in blockchain analysis. It applies compression algorithms on a per-column basis, taking advantage of the homogeneity of data within each column. The results in higher compression ratios, reducing storage requirements and speeding up data retrieval. Also, Parquet allows for reading only specific columns, minimizing I/O operations and improving query performance. Especially useful when analyzing particular aspects of blockchain data without loading entire blocks or transactions. Arrow operates with in-memory columnar data, reducing data transfer overhead and enabling faster query execution. It supports parallel query execution, leveraging multi-core processors to speed up complex analytical tasks.
The potent combination of Parquet and Arrow creates a powerful system for blockchain data management. Parquet’s efficient storage format compliments Arrow’s fast query execution, allowing for quick access to specific blockchain data points. This setup enables SQD to horizontally scale its data lake, adding more nodes to handle increasing blockchain data volumes without being bound by single-machine limitations. The efficient compression and storage of Parquet in tandem with Arrow’s optimized query execution, significantly reduces the hardware resources needed from traditional RPC nodes.
| Data Storage Requirement Comparison | ||
| Blockchain | SQD Dataset Size | Archive Node RPC |
| Binance | 5.8 TB | 10 TB |
| Polygon | 5.1 TB | 10 TB |
| Ethereum | 2.0 TB | 12 TB |
| ZKsync | 1.5 TB | 3 TB + 8 TB PostgreSQL |
| Arbitrum | 878.9 GB | 9.7 TB |
| Base | 795.2 GB | 2 TB+ |
| Optimism | 691.3 GB | 5 TB |
By utilizing Parquet and Arrow, SQD creates a data lake that is more scalable, efficient, and performant. Each dataset is split and replicated across 10 to 100 of worker nodes, therefore providing the network 10 to 100x more throughput in comparison to a single RPC node. For most developers, SQD can effectively replace RPC nodes.
Future of Blockchain Data Access
As blockchains’ state continues to grow, the need for efficient, scalable, and cost-effective data access solutions becomes increasingly critical, even more so for multi-chain applications. SQD’s decentralized data lake approach represents a significant step forward in addressing these challenges.
By providing a horizontally scalable architecture that separates data storage from consensus mechanisms, SQD enables developers to build more complex and data-intensive decentralized applications without the limitations imposed by traditional RPC nodes.
While RPC nodes will still play a crucial role in transaction broadcasting and MEV use cases where millisecond-level latency is essential, the future of blockchain data access for most developers likely lies in solutions like SQD that prioritize scalability, performance, and cost-effectiveness.
As SQD becomes the standard, it unlocks new possibilities for value-added services. Just as real-time webhooks, subscriptions, NFT APIs, and other indexed APIs once relied on RPC nodes, these innovations can now leverage SQD’s data lake instead. The journey of SQD Adoption is only just now starting.
Nodies Contributions to SQD
Nodies contributes to SQD through industry insights as an SQD pilled infra provider. We also operate SQD workers. In the future, we will operate the first community portal and contribute additional software layers to help with traction, growth, and developer adoption.
We have scaled up to 13m+ worth of SQD, a total of 130 workers, in the last two months on the supply side and look forward to what we can do on the demand and adoption side!
If you’re interested in running SQD workers with us, visit our staking page!
– Nodies Team & special thanks to weeb3dev for coauthoring this post!