Celeborn worker cannot start and an error occurred

Oct 13, 2025 by vinooalluri.com 21 views

[BUG] Errors When Configuring celeborn.storage.s3.dir with s3:// and s3a:// Schemes

Celeborn is a distributed data processing engine that provides a scalable and fault-tolerant way to process large datasets. However, when configuring the celeborn.storage.s3.dir property with the s3:// and s3a:// schemes, users may encounter errors that prevent the Celeborn worker from starting or cause issues during the shuffle phase. In this article, we will explore the bugs associated with these schemes and provide a step-by-step guide on how to reproduce and resolve them.

The bug occurs when the celeborn.storage.s3.dir property is configured with the s3:// or s3a:// scheme. The error message indicates that there is no file system for the specified scheme, which prevents the Celeborn worker from starting or causes issues during the shuffle phase.

Using `s3://` Scheme

To reproduce the bug using the s3:// scheme, follow these steps:

Configuration: Set the celeborn.storage.s3.dir property to s3://${bucket_name}, where ${bucket_name} is the name of your S3 bucket.
Celeborn worker cannot start and an error occurred: The Celeborn worker will fail to start, and an error message will be displayed indicating that there is no file system for the s3 scheme.

Error Message

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

Error Screenshot

Using `s3a://` Scheme

To reproduce the bug using the s3a:// scheme, follow these steps:

Configuration: Set the celeborn.storage.s3.dir property to s3a://${bucket_name}, where ${bucket_name} is the name of your S3 bucket.
Celeborn worker starts without issues, but an error occurs during the shuffle phase: The Celeborn worker will start without issues, but an error will occur during the shuffle phase, specifically in the createDiskFile method.

Error Message

org.apache.hadoop.fs.s3a.S3AIOException: Unable to create disk file

Error Screenshot

To resolve the bug, you can try the following solutions:

Use the s3n:// scheme: Instead of using the s3:// or s3a:// scheme, try using the s3n:// scheme, which is the recommended scheme for accessing S3 buckets.
Configure the fs.s3a.access.key and fs.s3a.secret.key properties: Make sure that the fs.s3a.access.key and fs.s3a.secret.key properties are configured correctly in your Celeborn configuration file.
Check the S3 bucket permissions: Ensure that the S3 bucket has the correct permissions to allow the Celeborn worker to access it.

In conclusion, the bugs associated with the s3:// and s3a:// schemes can be resolved by using the s3n:// scheme, configuring the fs.s3a.access.key and fs.s3a.secret.key properties, and checking the S3 bucket permissions. By following these steps, you can ensure that your Celeborn worker starts without issues and processes your data efficiently.

Check the Celeborn logs: The Celeborn logs can provide valuable information about the error that occurred. Check the logs to see if there are any error messages that can help you diagnose the issue.
Verify the S3 bucket configuration: Ensure that the S3 bucket is configured correctly and has the necessary permissions to allow the Celeborn worker to access it.
Check the Celeborn configuration file: Verify that the Celeborn configuration file is correctly configured and that the celeborn.storage.s3.dir property is set to the correct value.

Celeborn documentation: The Celeborn documentation provides detailed information about configuring the celeborn.storage.s3.dir property and troubleshooting common issues.
S3 documentation: The S3 documentation provides information about configuring S3 buckets and troubleshooting common issues.

Q: What is the recommended scheme for accessing S3 buckets? A: The recommended scheme for accessing S3 buckets is s3n://.
Q: How do I configure the fs.s3a.access.key and fs.s3a.secret.key properties? A: You can configure the fs.s3a.access.key and fs.s3a.secret.key properties in your Celeborn configuration file.
Q: How do I troubleshoot common issues with the celeborn.storage.s3.dir property? A: You can troubleshoot common issues with the celeborn.storage.s3.dir property by checking the Celeborn logs, verifying the S3 bucket configuration, and checking the Celeborn configuration file.
[BUG] Errors When Configuring celeborn.storage.s3.dir with s3:// and s3a:// Schemes: Q&A

In our previous article, we explored the bugs associated with the celeborn.storage.s3.dir property when configured with the s3:// and s3a:// schemes. We also provided a step-by-step guide on how to reproduce and resolve these issues. In this article, we will answer some frequently asked questions (FAQs) related to the celeborn.storage.s3.dir property and provide additional troubleshooting tips.

Q: What is the recommended scheme for accessing S3 buckets?

A: The recommended scheme for accessing S3 buckets is s3n://. This scheme is designed to provide a more secure and efficient way to access S3 buckets.

Q: How do I configure the `fs.s3a.access.key` and `fs.s3a.secret.key` properties?

A: You can configure the fs.s3a.access.key and fs.s3a.secret.key properties in your Celeborn configuration file. These properties are used to authenticate with the S3 bucket and access its contents.

Q: How do I troubleshoot common issues with the `celeborn.storage.s3.dir` property?

A: You can troubleshoot common issues with the celeborn.storage.s3.dir property by checking the Celeborn logs, verifying the S3 bucket configuration, and checking the Celeborn configuration file.

Q: What are some common errors that occur when using the `s3://` scheme?

A: Some common errors that occur when using the s3:// scheme include:

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
org.apache.hadoop.fs.s3a.S3AIOException: Unable to create disk file

Q: How do I resolve the `org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"` error?

A: To resolve this error, you can try the following:

Use the s3n:// scheme instead of s3://
Configure the fs.s3a.access.key and fs.s3a.secret.key properties correctly
Check the S3 bucket permissions to ensure that the Celeborn worker has the necessary access

Q: How do I resolve the `org.apache.hadoop.fs.s3a.S3AIOException: Unable to create disk file` error?

A: To resolve this error, you can try the following:

Check the Celeborn logs to see if there are any error messages that can help you diagnose the issue
Verify the S3 bucket configuration to ensure that it is correctly configured
Check the Celeborn configuration file to ensure that the celeborn.storage.s3.dir property is set to the correct value

Q: Can I use the `s3a://` scheme with the `fs.s3a.access.key` and `fs.s3a.secret.key` properties?

A: Yes, you can use the s3a:// scheme with the fs.s3a.access.key and fs.s3a.secret.key properties. However, you need to ensure that the fs.s3a.access.key and fs.s3a.secret.key properties are correctly configured and that the S3 bucket has the necessary permissions to allow the Celeborn worker to access it.

Check the Celeborn logs: The Celeborn logs can provide valuable information about the error that occurred. Check the logs to see if there are any error messages that can help you diagnose the issue.
Verify the S3 bucket configuration: Ensure that the S3 bucket is correctly configured and has the necessary permissions to allow the Celeborn worker to access it.
Check the Celeborn configuration file: Verify that the Celeborn configuration file is correctly configured and that the celeborn.storage.s3.dir property is set to the correct value.

Celeborn documentation: The Celeborn documentation provides detailed information about configuring the celeborn.storage.s3.dir property and troubleshooting common issues.
S3 documentation: The S3 documentation provides information about configuring S3 buckets and troubleshooting common issues.

In conclusion, the celeborn.storage.s3.dir property can be a complex configuration option, and errors can occur when using the s3:// and s3a:// schemes. By following the troubleshooting tips and FAQs provided in this article, you can resolve common issues and ensure that your Celeborn worker starts without issues and processes your data efficiently.