LevelDB read inefficiencies

Oct 13, 2025 by vinooalluri.com 15 views

Introduction

As a developer, testing the read API performance on a self-deployed Tron fullnode is crucial to ensure the smooth operation of the network. However, observing a peculiar behavior where the node shows low load and slow API performance when querying historical blocks, despite high load for recent blocks, can be puzzling. In this article, we will delve into the possible causes of this behavior and explore configuration tweaks and architectural optimizations to improve the read performance for historical data.

Environment Details

To better understand the issue, let's take a closer look at the environment details.

We are running two Tron nodes on a high-end server with the following specifications:
- 64 CPUs
- 256GB RAM
- SSD storage with a maximum read speed of 3GB/s
One node is running with normal p2p synchronization, fully synchronized, and holds both recent and historical data.
The other node is a read-only node that contains 70,000,000 historical blocks and is running with the --p2p-disable true flag.
Both nodes are started with JVM parameters: -Xmx64g -Xms32g -XX:+UseConcMarkSweepGC with 64Gb memory limitation, and no CPU limitation.
We are using the Jsonrpc to query block and transaction receipts, HTTP API for transactionInfo.

Observations

Based on the observations, we can see that:

When querying a range of recent blocks (e.g., 6 - 70,000,000), the node's CPU usage is high (4000%), and disk I/O reaches the disk limits (100%), with queries performing at high speed.
However, when querying older blocks (e.g., around block height 4,000,000), the node never reaches the same level of resource utilization—even with increased concurrency, the node ends up returning HTTP 504 errors.
This suggests that there is a performance bottleneck when reading historical data. Simply increasing concurrency only leads to timeouts.

Possible Causes

Based on the observations, there are several possible causes for this behavior:

Lower cache hit rates: When querying historical blocks, the cache hit rate may be lower due to the larger number of blocks being queried. This can lead to slower performance and increased CPU usage.
LevelDB read inefficiencies: LevelDB is the default storage engine for Tron nodes. However, it may not be optimized for reading historical data, leading to slower performance and increased CPU usage.
Issues with indexing for historical data: The indexing mechanism for historical data may not be optimized, leading to slower performance and increased CPU usage.

Configuration Tweaks

To improve the read performance for historical data, we can try the following configuration tweaks:

Increase the cache size: Increasing the cache size can help improve the cache hit rate, leading to faster performance and reduced CPU usage.
Optimize LevelDB settings: Optimizing LevelDB settings, such as increasing the read buffer size, can help improve the read performance for historical data.
Use a more efficient indexing mechanism: Using a more efficient indexing mechanism, such as a B-tree index, can help improve the read performance for historical data.

Architectural Optimizations

To further improve the read performance for historical data, we can try the following architectural optimizations:

Use a separate node for historical data: Using a separate node for historical data can help reduce the load on the main node and improve the read performance for historical data.
Use a more efficient storage engine: Using a more efficient storage engine, such as a column-store database, can help improve the read performance for historical data.
Use a distributed indexing mechanism: Using a distributed indexing mechanism, such as a distributed hash table, can help improve the read performance for historical data.

Conclusion

In conclusion, the low load and slow API performance when querying historical blocks, despite high load for recent blocks, can be caused by several factors, including lower cache hit rates, LevelDB read inefficiencies, and issues with indexing for historical data. By trying the configuration tweaks and architectural optimizations outlined in this article, we can improve the read performance for historical data and better utilize available resources.

Recommendations

Based on the analysis, we recommend the following:

Increase the cache size to improve the cache hit rate.
Optimize LevelDB settings to improve the read performance for historical data.
Use a more efficient indexing mechanism to improve the read performance for historical data.
Consider using a separate node for historical data to reduce the load on the main node.
Consider using a more efficient storage engine to improve the read performance for historical data.
Consider using a distributed indexing mechanism to improve the read performance for historical data.

Q: What are the possible causes of this behavior?

A: There are several possible causes for this behavior, including:

Lower cache hit rates: When querying historical blocks, the cache hit rate may be lower due to the larger number of blocks being queried. This can lead to slower performance and increased CPU usage.
LevelDB read inefficiencies: LevelDB is the default storage engine for Tron nodes. However, it may not be optimized for reading historical data, leading to slower performance and increased CPU usage.
Issues with indexing for historical data: The indexing mechanism for historical data may not be optimized, leading to slower performance and increased CPU usage.

Q: How can I increase the cache size to improve the cache hit rate?

A: To increase the cache size, you can modify the cache_size parameter in the tron-config.conf file. For example, you can set cache_size to 1024 to increase the cache size to 1024 MB.

Q: How can I optimize LevelDB settings to improve the read performance for historical data?

A: To optimize LevelDB settings, you can modify the leveldb section in the tron-config.conf file. For example, you can set read_buffer_size to 1024 to increase the read buffer size to 1024 MB.

Q: What are the benefits of using a more efficient indexing mechanism?

A: Using a more efficient indexing mechanism, such as a B-tree index, can help improve the read performance for historical data by reducing the time it takes to retrieve data from the index.

Q: How can I use a separate node for historical data to reduce the load on the main node?

A: To use a separate node for historical data, you can create a new node with the --p2p-disable true flag and configure it to store only historical data. This can help reduce the load on the main node and improve the read performance for historical data.

Q: What are the benefits of using a more efficient storage engine?

A: Using a more efficient storage engine, such as a column-store database, can help improve the read performance for historical data by reducing the time it takes to retrieve data from the database.

Q: How can I use a distributed indexing mechanism to improve the read performance for historical data?

A: To use a distributed indexing mechanism, such as a distributed hash table, you can configure multiple nodes to work together to index and retrieve data. This can help improve the read performance for historical data by distributing the load across multiple nodes.

Q: What are the best practices for optimizing the read performance for historical data?

A: The best practices for optimizing the read performance for historical data include:

Increasing the cache size to improve the cache hit rate.
Optimizing LevelDB settings to improve the read performance for historical data.
Using a more efficient indexing mechanism to improve the read performance for historical data.
Using a separate node for historical data to reduce the load on the main node.
Using a more efficient storage engine to improve the read performance for historical data.
Using a distributed indexing mechanism to improve the read performance for historical data.

Q: What are the common mistakes to avoid when optimizing the read performance for historical data?

A: The common mistakes to avoid when optimizing the read performance for historical data include:

Not increasing the cache size to improve the cache hit rate.
Not optimizing LevelDB settings to improve the read performance for historical data.
Not using a more efficient indexing mechanism to improve the read performance for historical data.
Not using a separate node for historical data to reduce the load on the main node.
Not using a more efficient storage engine to improve the read performance for historical data.
Not using a distributed indexing mechanism to improve the read performance for historical data.

By following these best practices and avoiding common mistakes, you can optimize the read performance for historical data and improve the overall performance of your Tron node.