virtual reference error: "connection closed before message completed"

[nbren12]

What happened?

I am using icechunk to read data from an s3 compatible object store in a distributed context with ~100 workers spread across a few compute nodes. I will intermittently get errors like this

[rank12]: icechunk.IcechunkError:   x error fetching virtual reference
[rank12]:   |
[rank12]:   | context:
[rank12]:   |    0: icechunk::store::get
[rank12]:   |            with key="arr/c/3/3476/58/0/0" byte_range=From(0)
[rank12]:   |              at icechunk/src/store.rs:198
[rank12]:   |
[rank12]:   |-> error fetching virtual reference
[rank12]:   |-> dispatch failure
[rank12]:   |-> other
[rank12]:   `-> connection closed before message completed

Is this case handled by icechunks retry mechanism?

I'm not sure what the default settings of icechunk retry are since the docs just list None.

What did you expect to happen?

Error message saying failed after "x number of retries".

Anything else we need to know?

No response

Environment

[paraseba]

I would expect this to be retried. Defaults are poorly documented, sorry, this is what they are in S3:

max_tries: 10
initial_backoff_ms: 100
max_backoff_ms: 180,000

You could get more informatino by enabling debug or trace level logs, by either setting the env variable ICECHUNK_LOG=trace or calling icechunk.set_logs_filter("trace"). If you want less noise, there are also ways to enable specific loggers only.

[cchenzi]

@paraseba doing a quick research, it might be due to keep_alive_timeout differences between the hyper connection (being used under the hood from the aws sdk) vs the one on S3.

maybe a stress test passing this different config could be good to see if it's solvable

[paraseba]

Interesting! I wonder why aren't more people complaining about the Rust S3 SDK using the wrong timeout...

[nbren12]

Thanks. I'm just going to hammer on our object store a bit harder

        config = icechunk.Repository.fetch_config(storage)
        config.storage = icechunk.StorageSettings(
            retries=icechunk.StorageRetriesSettings(
                max_tries=10000, initial_backoff_ms=100, max_backoff_ms=2000
            )
        )

Maybe my workload is a bit different from the typical for icechunk. I'm loading data for ML training on 4 nodes of 8 gpus each * 4 cpu data workers per gpu fetching data with icechunk. Each node has a single network card. So basically a standard setup for data parallel ML training.

[paraseba]

Sounds good, please tell us how it goes.

[cchenzi]

@paraseba if it's from your interest, I think that

aws-smithy-http-client = { version = "1.1.4", features = ["default-client", "rustls-aws-lc"] }

    let http_client = aws_smithy_http_client::Builder::new()
        .pool_idle_timeout(std::time::Duration::from_secs(15))
        .tls_provider(aws_smithy_http_client::tls::Provider::Rustls(
            aws_smithy_http_client::tls::rustls_provider::CryptoMode::AwsLc,
        ))
        .build_https();

    let mut aws_config = aws_config::defaults(BehaviorVersion::v2025_08_07())
        .region(region)
        .app_name(app_name)
        .http_client(http_client);

at icechunk/src/storage/s3.rs::mk_client

could to the trick, at least to test if this behavior would persist.

[nbren12]

@paraseba tuning the retry settings seemed to work...no errors for 40 minutes so far, though i worry the pipeline is being slowed by the multiple retries.

[paraseba]

That's so weird, why are you getting so many errors from S3? I wonder if somehow in your workflow we are overwhelming that poor network card. Are you using multiple processes? That wasn't fully clear to me from your explanation of the setup. You may want to play with RepositoryConfig.max_concurrent_requests, default is 256 (per process if you are doing more than one). Maybe you can try lowering that and see what happens?

[nbren12]

My setup is 4 processors per gpu. 8 gpus per node. 4 nodes. So the total number of workers querying our object store is 128. Our object storage is s3 compatible but is on-prem, and generally fairly reliable.

Aggregating across the 4 workers feeding data to each GPU I only need about 100 MB/s which is far from saturating my pipes. The NIC's on these nodes are about 40-80 gbps and have handled significantly heavier workloadsin the past both in terms of bandwidth and number of chunks being loaded concurrently.

Within each process, I'm not sure what my concurrency is. Each processor is only querying for a single zarr chunk at a time (16 MB uncompressed). Does icechunk split a single chunk for concurrent downloads?

[nbren12]

I'm not sure if it's relevant, but the multiple processes are created by https://docs.pytorch.org/docs/stable/data.html. This is a common setup for distributed data parallel training of ML models. icechunk has been working quite nicely with data on a LUSTRE filesystem, and fsspec + zarr + async has worked well for our object store in the past, so I think there is something different with ice-chunk's s3 integration.

[paraseba]

It does split single requests. You can play with:

max_concurrent_requests_for_object default 12 MB
and
max_concurrent_requests_for_object default 18

But anyway that bandwidth is tiny. I have pushed single network cards EC2 instances close to 100 Gb/sec using Icechunk. Something else is wrong here.

[paraseba]

Sorry

max_concurrent_requests_for_object ideal_concurrent_request_size

[paraseba]

Any way you could get logs from the on-prem store and try to understand why it hates us?

[nbren12]

Yah something is up. I checked and my chunks are only 4MB compressed, so this shouldn't be splitting right? I'll try to bisect further. I definitely can't get systems level logs for the object store if that's what you meant, but I can try the ICECHUNK_LOG=trace option.

[paraseba]

Yeah, it shouldn't split for 4MB. Let's give trace or debug a try I guess ... if you get too much output there is some info on how to filter here