Cannot access a disposed object. Object name: 'SslStream'. #31254

samneefs · 2019-10-22T11:41:43Z

I have an issue with SendAsync in .NET Core 2.2.
When doing a lot of requests, I encounter the exception on some of them.
I have gotten the error on different requests, for example on a small GET request. However I do have to mention that asynchronously I'm also doing other requests, for example POST (with content smaller than 4MB).

This issue is similar to https://github.com/dotnet/corefx/issues/34033 , but not the same. Mine doesn't mention the handshake in the stacktrace and the other one should be fixed since Core 2.1 (and I use 2.2).

The operation was canceled.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at [details of own code omitted]

Inner exception:
The read operation failed, see inner exception.
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)

Inner Exception:
Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)

The text was updated successfully, but these errors were encountered:

scalablecory · 2019-10-22T12:19:20Z

Is it possible your requests are timing out? Does the same issue happen on .NET Core 3.0?

Plasma · 2019-10-22T14:13:07Z

Are you accidentally returning from an async method without awaiting within a using block?

samneefs · 2019-10-22T15:43:31Z

@scalablecory
Some sort of timeout seems likely:
when I do another request, that is one single long running request, to the same server I get a similar error after 10min, however, this error is not the same (and strangely it stays 10min after changing the timeout to something bigger).
However increasing the timeout on the client (to 20 min), had no effect: still got errors before the timeout.
On the server the executionTimeout had been increased tot 1800s (30min) and it also triggered before this.
About the original calls:
When doing all calls synchronously after each other (instead of async, multiple together), then I don't encounter the problem.
Maybe I created to much load on the server when I do it async and this breaks the connection?
I did await Task.WhenAll(tasks); maybe there is a timeout on how long I can await a task?

@Plasma
No, I receive a task for each request and await them all with await Task.WhenAll(tasks);

samneefs · 2019-10-22T15:47:39Z

@scalablecory
With the long running request, the exception is a bit different. Here are the details:

An error occurred while sending the request.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at SBB.Exact.Rest.Proxy.Handwritten.BaseExactRestApiProxy.GetAsync[T](String url, String applicationId, String division)
   at SBB.Exact.Rest.Proxy.Handwritten.BaseExactRestApiProxy.GetAllAsync[T](String applicationId, String division, String[] fields, String where, Boolean withPagination, String skiptoken, Int32 numberOfPages)
   at SBB.Exact.Rest.Proxy.Handwritten.ExactRestApiProxy.GetBulkTransactionLinesAsync(String applicationId, String division, String[] fields, String where)
   at Winbooks2Exact.Core.Tests.IntegrationTests.ExactRestIntegrationTest.GetAllTransactionLines(String dossierNr, String divisionId) in C:\Workspaces\SelectProjects\Winbooks3\Winbooks2Exact\Dev\SBB.Winbooks2Exact\Winbooks2Exact.Core.Tests\IntegrationTests\ExactRestIntegrationTest.cs:line 29
      
InnerException	
   Unable to read data from the transport connection: De externe host heeft een verbinding verbroken.
      at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
     
InnerException	
{System.Net.Sockets.SocketException (10054): De externe host heeft een verbinding verbroken}

scalablecory · 2019-10-22T15:59:20Z

Okay, can you reduce your code into a small repro? This will help diagnose the problem.

davidsh · 2019-10-22T17:17:05Z

@samneefs
I think if the only exception were this:

The operation was canceled.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at [details of own code omitted]

then this would be perceived as "normal" behavior. By "normal", I mean that the HTTP request was cancelled probably due to a timeout-related cancellation token expiring. The cancellation token could either be one you pass into the HttpClient APIs. Or the internal one used when HttpClient.Timeout is used. By default HttpClient.Timeout has a non-infinite timeout.

I think the "bug" here is that while cancelling the HTTP request, there are operations that get cancelled internally (such as on SslStream). And cancelling those I/O operation results in a bug where we are trying to reference the, now disposed, SslStream. That is probably why those other internal exceptions are occurring.

It would probably make sense for the HTTP stack to potentially ignore some of those exceptions since they are "expected" to occur while shutting down the connection. However, I think the SslStream ObjectDisposedException is something where we can fix that so that we don't try to access the SslStream after it has been disposed.

wfurt · 2020-03-05T04:43:22Z

I got same exception in the local build

      System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
      ---- System.TimeoutException : The operation was canceled.
      -------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
      ------------ System.Net.Http.HttpRequestException : Error while copying content to a stream.
      ---------------- System.ObjectDisposedException : Cannot access a disposed object.
      Object name: 'SslStream'.
      Stack Trace:
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(564,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/PostScenarioTest.cs(242,0): at System.Net.Http.Functional.Tests.PostScenarioTest.PostHelper(RemoteServer remoteServer, String requestBody, HttpContent requestContent, Boolean useContentLengthUpload, Boolean useChunkedEncodingUpload)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/PostScenarioTest.cs(157,0): at System.Net.Http.Functional.Tests.PostScenarioTest.PostLargeContentUsingContentLengthSemantics_Success(RemoteServer remoteServer, Int32 contentLength)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----

        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(695,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1008,0): at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(734,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(33,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTestBase.cs(104,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTestBase.VersionCheckerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(542,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(378,0): at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(759,0): at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(444,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(832,0): at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|140_0(ExceptionDispatchInfo e)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(822,0): at System.Net.Security.SslStream.ThrowIfExceptional()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(797,0): at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1212,0): at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1196,0): at System.Net.Http.HttpConnection.FlushAsync()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(982,0): at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(374,0): at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
  Finished:    System.Net.Http.Functional.Tests
Canceling due to test failure...
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 1831, Errors: 0, Failed: 2, Skipped: 54, Time: 204.718s

I was running tests under strace and everything is very slow. I did get several OperationCanceled exceptions before (and I fiddle with timeouts) but this is the first time when I get ObjectDisposedException.

wfurt · 2020-03-05T05:56:11Z

I can also get running current 5.0 master.

     Condition(s) not met: "IsWindows10Version1607OrGreater"
      System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
      ---- System.TimeoutException : The operation was canceled.
      -------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
      ------------ System.ObjectDisposedException : Cannot access a disposed object.
      Object name: 'System.Net.Sockets.NetworkStream'.
      Stack Trace:
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(564,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AutoRedirect.cs(359,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_AutoRedirect.GetAsync_MaxAutomaticRedirectionsNServerHops_ThrowsIfTooMany(Int32 maxHops, Int32 hops)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----

        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(695,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1008,0): at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(734,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(81,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(542,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(875,0): at System.Net.Sockets.NetworkStream.<ThrowIfDisposed>g__ThrowObjectDisposedException|63_0()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(872,0): at System.Net.Sockets.NetworkStream.ThrowIfDisposed()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(801,0): at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1212,0): at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1119,0): at System.Net.Http.HttpConnection.WriteBytesSlowAsync(Byte[] bytes)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(263,0): at System.Net.Http.HttpConnection.WriteHostHeaderAsync(Uri uri)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(402,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
  Finished:    System.Net.Http.Functional.Tests
Canceling due to test failure...
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 1729, Errors: 0, Failed: 3, Skipped: 54, Time: 196.215s

davidsh · 2020-03-05T12:47:02Z

I would say based on this stack trace, the only part we need to fix is the 'ObjectDisposedException'. We should be able to detect when the SslStream is being closed by upstack code (in this case due to a cancellation) and prevent SslStream from accessing its own objects while closing down.

karelz · 2020-04-16T15:57:57Z

Triage: We should see if we can detect it and avoid throwing. If not, then it is likely confusing to customers, in which case we may want to mask it / not include as inner exception of TaskCanceled.

Note: We have seen more customers running into this and being confused in the past.

kshyju · 2020-12-07T08:23:32Z

Any update on this issue @karelz ?

karelz · 2020-12-07T14:33:46Z

@kshyju not really. We just didn't get to it yet.
How does it impact you? (How often, how badly, etc.)

wfurt · 2020-12-07T17:52:20Z

Do you have way how to reproduce it @kshyj? Is is still happening with 5.0?

trympet · 2021-04-30T12:14:14Z

Is is still happening with 5.0?

Yep. "Just my code" needs to be disabled and the exception needs to be enabled to trigger a break in VS.

wfurt · 2021-04-30T16:15:49Z

Do you have any simple repro @trympet

trympet · 2021-04-30T17:25:12Z

@wfurt, sure. I'll email you a repro and memory dump.

trympet · 2021-04-30T17:39:52Z

@wfurt remember to disable "Just my code" and enable the IOException and ObjectDisposedException breakpoints.

wfurt · 2021-04-30T18:04:53Z

can it be reproduced without VS/Debugger @trympet?

trympet · 2021-05-03T14:29:02Z

@wfurt I was unable to reproduce the issue without attaching a debugger.

wfurt · 2021-05-07T05:08:50Z

HTTP/2 is whole different animal @trympet. Because of the multiplex, there is read task to pull incoming frames with race condition to disposal. However, as far as I can tell, this does not surface as that is expected and get eaten by the HttpClient. When you running under debugger, it allows you to see and intercept all the exception regarless if they are handled or not. So you would see it there but that should not impact the actual code execution.
The original issue is different as the exception would surface to the caller directly.

bhanupalsingh · 2024-02-06T19:43:01Z

Is there any update on this Issue , I am using net6.0 and I am also facing same error

   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1917
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 908
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action1 callback, TState& state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 314
   at System.Threading.QueueUserWorkItemCallback.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 924
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 790
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 107
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 106
 
  This exception was originally thrown at this call stack:
    System.Net.Security.SslStream.ThrowIfExceptional.__ThrowExceptional|138_0(System.Runtime.ExceptionServices.ExceptionDispatchInfo)
    System.Net.Security.SslStream.DecryptData(int)
    System.Net.Security.SslStream.ReadAsyncInternal<TIOAdapter>(TIOAdapter, System.Memory<byte>)
 
Inner Exception 1:
TaskCanceledException: The operation was canceled.
 
Inner Exception 2:
IOException: The read operation failed, see inner exception.
 
Inner Exception 3:
ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.`

wfurt · 2024-02-06T23:34:55Z

Do you have repro @bhanupalsingh? Perhaps you can also try 8.0 as current LTS.

MihaZupan · 2024-06-21T17:54:57Z

Since I didn't see it mentioned while skimming existing comments, this is almost certainly the result of our weird way of "cancelling" pending I/O in our HTTP/1.1 implementation.

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs

Lines 922 to 927 in ff0c538

    
           return cancellationToken.Register(static s => 
        
           { 
        
               var connection = (HttpConnection)s!; 
        
               if (NetEventSource.Log.IsEnabled()) connection.Trace("Cancellation requested. Disposing of the connection."); 
        
               connection.Dispose(); 
        
           }, this);

We're not actually passing the cancellation token down to ReadAsync/WriteAsync calls to the transport. Instead, we're disposing the transport stream on cancellation and expecting the disposal to abort any pending operations. It's thus no surprise that this sometimes raises ObjectDisposedExceptions.

wfurt · 2024-10-07T18:24:00Z

While SslStream has some provisions to support pending Read while disposed it will throw on Write. It seems like HttpClient should catch that exception and it should silently discard it if cancellation was requested ....

Or should we change the cancellation logic? I'm wondering if the existing code comes from time when cancellation on Socket did not quite work. Do you have any recollection here @stephentoub

Is this something we should try to investigate in 10? While the impact may be small it seems like this keeps coming over and over again ;(

antonfirsov · 2025-02-05T16:53:50Z

It's thus no surprise that this sometimes raises ObjectDisposedExceptions.

It seems like HttpClient should catch that exception and it should silently discard it if cancellation was requested

We have a mechanism to wrap these exceptions in OperationCanceledException where we first check the exception and the token with ShouldWrapInOperationCanceledException:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/CancellationHelper.cs

Lines 19 to 20 in c9af66c

    
           internal static bool ShouldWrapInOperationCanceledException(Exception exception, CancellationToken cancellationToken) => 
        
               !(exception is OperationCanceledException) && cancellationToken.IsCancellationRequested;

Then we wrap the exception to a TaskCanceledException:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/CancellationHelper.cs

Lines 26 to 27 in c9af66c

    
           internal static Exception CreateOperationCanceledException(Exception? innerException, CancellationToken cancellationToken) => 
        
               new TaskCanceledException(s_cancellationMessage, innerException, cancellationToken); // TCE for compatibility with other handlers that use TaskCompletionSource.TrySetCanceled()

We use this mechanism excessively in code that deals with the mentioned CancellationTokenRegistration-s, so I would assume that it's either leaky or the ODE is caused by something else than cancellation.

antonfirsov · 2025-02-10T18:04:45Z

This is a connection pooling bug where a disposed (canceled) connection is returned to the pool and picked up by the next request in the queue. (And that request fails with an HttpRequestException --> ObjectDisposedException, since there is no cancellation occuring during its' processing.)

In the case I managed to reproduce I see this happening in ChunkedEncodingReadStream.CopyToAsyncCore, but I think other code paths may also have the the same flaw:

cancelationToken is canceled at the moment the method is entered
RegisterCancelation runs the cancelation callback inline disposing the connection and the underlying stream. Since no OperationCanceledException is thrown, the execution of the method continues.
The response bytes are buffered already, the request can be fully served despite the cancellation. Calls to ReadChunkFromConnectionBuffer will parse out the response without touching the connections' stream.
The last call to ReadChunkFromConnectionBuffer eventually calls HttpConnection.CompleteResponse, which returns the connection to the pool since there are no guards against _disposed on that path.

I see two simple fixes to this bug:

Make sure that RegisterCancellation (its' call sites) throw a TaskCanceledException in case of inline cancelations aborting any further processing logic that may call CompleteResponse. Note that we don't have to throw here if we don't want to since the data might be buffered.
Acknowledge that calling CompleteResponse doesn't guarantee the connection is alive. Do not return it to the pool if it's _disposed.

MihaZupan · 2025-02-10T19:16:00Z

Nice catch.
We have code in place trying to handle this race condition:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ChunkedEncodingReadStream.cs

Lines 402 to 414 in cc74292

    
           // Dispose of the registration and then check whether cancellation has been 
        
           // requested. This is necessary to make deterministic a race condition between 
        
           // cancellation being requested and unregistering from the token.  Otherwise, 
        
           // it's possible cancellation could be requested just before we unregister and 
        
           // we then return a connection to the pool that has been or will be disposed 
        
           // (e.g. if a timer is used and has already queued its callback but the 
        
           // callback hasn't yet run). 
        
           cancellationRegistration.Dispose(); 
        
           CancellationHelper.ThrowIfCancellationRequested(cancellationRegistration.Token); 
        
           _state = ParsingState.Done; 
        
           _connection.CompleteResponse(); 
        
           _connection = null;

But CancellationTokenSource.Register hands back a non-initialized registration if the token was already cancelled, and we're passing around the registration instead of the token, so the above logic just no-ops instead in this case.

Another case where this check may fail is if the cancellation token passed to Read* methods changes.
We have a race condition where:

the first read succeeds, but the cancellation fires before we dispose the registration.
the second read uses a different cancellation token and tries to hand back the connection. The checks all no-op since the token isn't cancelled, and we return a disposed connection.

or a similar case where the second read happens as part of DrainAsync.

Solution-wise, we should ideally fix the underlying cause for all of this, which is using RegisterCancellation to dispose the stream instead of flowing the cancellation token around to ReadAsync/WriteAsync calls.
I think this would be simpler and easier to reason about.

Otherwise your proposed solution nr. 2 seems better.
With nr. 1 you wouldn't catch the other variations of this race conditions I mentioned.

antonfirsov · 2025-02-10T20:31:52Z

we should ideally fix the underlying cause for all of this, which is using RegisterCancellation to dispose the stream

I agree, but that's a relatively wide change that should be executed carefully. My preference would be to ship a minimal fix for this bug and open a separate one to track the HTTP/1.1 cancellation refactor.

msftgits transferred this issue from dotnet/corefx Feb 1, 2020

karelz added the untriaged New issue has not been triaged by the area owner label Mar 12, 2020

karelz added bug and removed untriaged New issue has not been triaged by the area owner labels Apr 16, 2020

karelz added this to the Future milestone Apr 16, 2020

This comment has been minimized.

Sign in to view

karelz added the test-run-core Test failures in .NET Core test runs label Apr 28, 2020

karelz changed the title ~~.NET Core 2.2: Cannot access a disposed object. Object name: 'SslStream'.~~ Cannot access a disposed object. Object name: 'SslStream'. Apr 28, 2020

This comment has been minimized.

Sign in to view

karelz added test-run-core Test failures in .NET Core test runs and removed test-run-core Test failures in .NET Core test runs labels May 11, 2021

ManickaP removed the test-run-core Test failures in .NET Core test runs label Sep 20, 2023

wfurt mentioned this issue Jun 20, 2024

Fix Http 1.1's read-ahead task management #103257

Merged

MihaZupan mentioned this issue Dec 10, 2024

ObjectDisposedException in HTTP/1.1 Stress #110577

Open

MihaZupan mentioned this issue Dec 30, 2024

Unobserved ObjectDisposedException when reading HTTPS stream #110998

Closed

antonfirsov self-assigned this Feb 6, 2025

antonfirsov linked a pull request Feb 10, 2025 that will close this issue

Prevent returning disposed HTTP/1.1 connections to the pool #112383

Open

dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Feb 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot access a disposed object. Object name: 'SslStream'. #31254

Cannot access a disposed object. Object name: 'SslStream'. #31254

samneefs commented Oct 22, 2019

scalablecory commented Oct 22, 2019

Plasma commented Oct 22, 2019

samneefs commented Oct 22, 2019

samneefs commented Oct 22, 2019 •

edited

Loading

scalablecory commented Oct 22, 2019

davidsh commented Oct 22, 2019 •

edited

Loading

wfurt commented Mar 5, 2020

wfurt commented Mar 5, 2020

davidsh commented Mar 5, 2020

karelz commented Apr 16, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kshyju commented Dec 7, 2020

karelz commented Dec 7, 2020

wfurt commented Dec 7, 2020

trympet commented Apr 30, 2021

wfurt commented Apr 30, 2021

trympet commented Apr 30, 2021

trympet commented Apr 30, 2021

wfurt commented Apr 30, 2021

trympet commented May 3, 2021

wfurt commented May 7, 2021

bhanupalsingh commented Feb 6, 2024 •

edited by antonfirsov

Loading

wfurt commented Feb 6, 2024

MihaZupan commented Jun 21, 2024

wfurt commented Oct 7, 2024

antonfirsov commented Feb 5, 2025 •

edited

Loading

antonfirsov commented Feb 10, 2025 •

edited

Loading

MihaZupan commented Feb 10, 2025 •

edited

Loading

antonfirsov commented Feb 10, 2025

Cannot access a disposed object. Object name: 'SslStream'. #31254

Cannot access a disposed object. Object name: 'SslStream'. #31254

Comments

samneefs commented Oct 22, 2019

scalablecory commented Oct 22, 2019

Plasma commented Oct 22, 2019

samneefs commented Oct 22, 2019

samneefs commented Oct 22, 2019 • edited Loading

scalablecory commented Oct 22, 2019

davidsh commented Oct 22, 2019 • edited Loading

wfurt commented Mar 5, 2020

wfurt commented Mar 5, 2020

davidsh commented Mar 5, 2020

karelz commented Apr 16, 2020 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kshyju commented Dec 7, 2020

karelz commented Dec 7, 2020

wfurt commented Dec 7, 2020

trympet commented Apr 30, 2021

wfurt commented Apr 30, 2021

trympet commented Apr 30, 2021

trympet commented Apr 30, 2021

wfurt commented Apr 30, 2021

trympet commented May 3, 2021

wfurt commented May 7, 2021

bhanupalsingh commented Feb 6, 2024 • edited by antonfirsov Loading

wfurt commented Feb 6, 2024

MihaZupan commented Jun 21, 2024

wfurt commented Oct 7, 2024

antonfirsov commented Feb 5, 2025 • edited Loading

antonfirsov commented Feb 10, 2025 • edited Loading

MihaZupan commented Feb 10, 2025 • edited Loading

antonfirsov commented Feb 10, 2025

samneefs commented Oct 22, 2019 •

edited

Loading

davidsh commented Oct 22, 2019 •

edited

Loading

karelz commented Apr 16, 2020 •

edited

Loading

bhanupalsingh commented Feb 6, 2024 •

edited by antonfirsov

Loading

antonfirsov commented Feb 5, 2025 •

edited

Loading

antonfirsov commented Feb 10, 2025 •

edited

Loading

MihaZupan commented Feb 10, 2025 •

edited

Loading