Skip to content

Sentry requests are pending indefinitely #6049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
antoantonydh opened this issue Oct 26, 2022 · 23 comments · Fixed by #7553
Closed
3 tasks done

Sentry requests are pending indefinitely #6049

antoantonydh opened this issue Oct 26, 2022 · 23 comments · Fixed by #7553
Assignees

Comments

@antoantonydh
Copy link

Is there an existing issue for this?

How do you use Sentry?

Self-hosted/on-premise

Which package are you using?

@sentry/browser

SDK Version

7.16.0

Framework Version

Angular 14.2.6

Link to Sentry event

No response

Steps to Reproduce

Sentry requests are pending indefinitely. It is not happening to all the requests. But to the few of them randomly. I checked if it is related to the payload issue by copying the requests from the chrome network tab to postman, and it is working fine without any issues. So, it is happening only from the browser.

Capture

I tried to revert to the older version of sentry lib v7.12.1 and seems like issue is solved. Is it related to this change #5697?

Expected Result

Sentry requests should either send to server or gives some error instead of pending state

Actual Result

Capture

@lobsterkatie
Copy link
Member

Hi, @antoantonydh.

Thanks for writing in! The change you linked does sound like a plausible cause. @Lms24, when you get a chance, would you mind giving this a look?

@lobsterkatie lobsterkatie added the Package: angular Issues related to the Sentry Angular SDK label Oct 27, 2022
@antoantonydh
Copy link
Author

Thanks @lobsterkatie. We noticed that, requests are in pending state usually have a large payload (38kB - 40kB). But if the size is the problem, it should not work when we sent the same request from the Postman right?

@Lms24
Copy link
Member

Lms24 commented Oct 27, 2022

Hi @antoantonydh.

This sounds strange to me and I can't recall that this came up before. I have a couple of questions:

  • You wrote that you're using @sentry/browser with Angular. Have you tried using @sentry/angular for first-class support with Angular? However, I'm quite certain that this won't change a lot in terms of this problem. Nevertheless, maybe it's worth a try?
  • Reading a little bit about the keepalive flag from fetch (which we use to not cancel our outgoing requests on navigations, as explained in feat(browser): Use fetch keepalive flag #5697), it seems to me as if it should work well in chromium-based browsers. There are problems in Firefox so I'm curious if you noticed any differences between browsers?
  • Are you using any libraries that modify fetch (e.g. monkey patch or instrument it in any way) other than the Sentry SDK?
  • Are you using Angular's HttpClient or any third-party Http request libraries?
  • Does this happen in special situations? E.g. when you navigate to a different page/route? Or does it also happen to requests when you don't navigate at all?
  • Have you noticed if it affects certain types of payloads? E.g. errors vs. transactions?

Let's see if we can narrow this down a little. I'm also curious if this only affects Angular users or if it happens in other frameworks as well (if readers of this issue are affected by the problem please feel free to comment. A last idea would be that this has to do something with Angular's HttpClient (assuming this behaviour only happens in Angular projects). However, I think it's unlikely, as our SDK doesn't interact with it at all.

@Lms24
Copy link
Member

Lms24 commented Oct 27, 2022

Reading this post (which I admit has little in common with your problem other than the pending state), can you verify if the requests that are pending actually made it to Sentry? I.e. do the events show up in Sentry (as issues or transactions)?

@antoantonydh
Copy link
Author

antoantonydh commented Oct 27, 2022

@Lms24 Thanks for the reply and looking into this.

You wrote that you're using @sentry/browser with Angular. Have you tried using @sentry/angular for first-class support with Angular? However, I'm quite certain that this won't change a lot in terms of this problem. Nevertheless, maybe it's worth a try?

I think, our project started using this initially and never changed the package to @sentry/angular. We have some logic, which we are trying to combine/merge the logs, after only we will call the sentry functions.

Reading a little bit about the keepalive flag from fetch (which we use to not cancel our outgoing requests on navigations, as explained in #5697), it seems to me as if it should work well in chromium-based browsers. There are problems in Firefox so I'm curious if you noticed any differences between browsers?

I tried in Firefox and there are no issues like this. I used the same env and same user for testing. But in chrome, we are getting pending requests and in the Firefox all are done.

Are you using any libraries that modify fetch (e.g. monkey patch or instrument it in any way) other than the Sentry SDK?

No, we are not using anything else. One thing, I noticed for the fetch API. Those are getting the angular polyfills by default.

Are you using Angular's HttpClient or any third-party Http request libraries?

Yes. only the HttpClient from Angular

Does this happen in special situations? E.g. when you navigate to a different page/route? Or does it also happen to requests when you don't navigate at all?

Not at all. It is happening in both scenarios.

Have you noticed if it affects certain types of payloads? E.g. errors vs. transactions?

I think it is mostly happening to events

{"type":"event"} {"message": "--------------------"}

@antoantonydh
Copy link
Author

Reading this post (which I admit has little in common with your problem other than the pending state), can you verify if the requests that are pending actually made it to Sentry? I.e. do the events show up in Sentry (as issues or transactions)?

We just checked it, and it never reached to the sentry. Somehow it is still stuck at the chrome

@jazlalli
Copy link

jazlalli commented Nov 1, 2022

if readers of this issue are affected by the problem please feel free to comment.

I'm not sure if this is strictly the same issue, but it seems too similar to be coincidental, so sharing my experience in case it helps.

I'm using the @sentry/react package v6.11.0. Some requests were getting through fine, e.g. an error in a React lifecycle method makes it to Sentry. But if I threw an error within an event handler (a regular async / await function), it remained pending and never made it to Sentry. This occurred in Chrome and Firefox, though each browser showed me different info.

Chrome
Screenshot 2022-11-01 at 17 27 04

Screenshot 2022-11-01 at 17 26 48

Firefox
Screenshot 2022-11-01 at 17 21 32

Screenshot 2022-11-01 at 17 28 29

but interestingly, a response (of sorts)

Screenshot 2022-11-01 at 17 28 45


Following the crumb that Firefox revealed (Payload Too Large), I inspected the payload of the pending request, and it contained a lot more than I was expecting. I'm running in dev mode, so there are lots of logs in the browser console, all of which were part of the payload that was attempting to be sent (I was surprised by this). I suppressed the console logs, and then the error was successfully sent to Sentry.

@antoantonydh might be worth exploring this route further in case it's the cause? Appreciate it might be unlikely given that you said the equivalent request via Postman worked for you, whereas the equivalent cURL request for me returned the same response (413 Payload Too Large).

@antoantonydh
Copy link
Author

@jazlalli Thanks for sharing the details. In my case, it is totally random, and even the payload is only less than 40kB. As mentioned, when I downgrade to the v7.12.1 it is not happening anymore. But I am yet to test it fully. I will confirm with in couple of days

@antoantonydh
Copy link
Author

antoantonydh commented Nov 3, 2022

@Lms24 we have deployed the downgraded version v7.12.1, and now we have logs coming in properly. So, I suspect either the issue was because of that keepAlive feature or some other recent changes happened.

@Lms24
Copy link
Member

Lms24 commented Nov 3, 2022

Hi everyone

@jazlalli thanks for chiming in but I think this is a separate issue. You're on version 6.11.0 of the SDK which for sure didn't contain any keepAlive changes. If what you're describing is still a problem, please open a separate issue. Thanks!

@antoantonydh

Our intention with adding the keepalive flag was to send requests successfully that would otherwise be cancelled. Can you confirm that the requests that are pending in the newer version are in fact the same ones that are successfully sent in 7.12.1? Or are they additional ones that weren't even sent before?

Thanks for the polyfill information. I'll look a little into the polyfills that Angular uses for fetch. Maybe they are not compatible with keepAlive as this is a relatively new option.

Apart from that, I'd like to make sure that this is indeed caused by #5697 and not by something else. We can try narrowing it down to a specific release: #5697 was introduced in 7.13.0. Would you mind trying that version?

Oh, one last thing:

I think, our project started using this initially and never changed the package to @sentry/angular. We have some logic, which we are trying to combine/merge the logs, after only we will call the sentry functions.

Maybe I'm reading too much into this but can you elaborate a bit on your setup? Are you working on a standard Angular SPA or is it a more compolex setup (e.g. SSR/Angular Universal?)

On an unrelated note: @sentry/angular comes with a few Angular-specific features, such as an Angular ErrorHandler or performance monitoring features. For example, we instrument the router to give you parameterized transaction names. It might be worth giving it a try

@antoantonydh
Copy link
Author

@Lms24 Thanks for the details.

Can you confirm that the requests that are pending in the newer version are in fact the same ones that are successfully sent in 7.12.1? Or are they additional ones that weren't even sent before?

Yes, same ones are now sending to the sentry. But it is totally random. Sometime these events will be sent to sentry and sometimes not. But downgrading to that version solves this problem.

Apart from that, I'd like to make sure that this is indeed caused by #5697 and not by something else. We can try narrowing it down to a specific release: #5697 was introduced in 7.13.0. Would you mind trying that version?

Sure. Let me try that.

Maybe I'm reading too much into this but can you elaborate a bit on your setup? Are you working on a standard Angular SPA or is it a more compolex setup (e.g. SSR/Angular Universal?)

I am working on a standard Angular SPA. However, we have some generic logger services in place for better controlling over different env's (staging, prod etc). So, we are calling the sentry SDK from this service.

On an unrelated note: @sentry/angular comes with a few Angular-specific features, such as an Angular ErrorHandler or performance monitoring features. For example, we instrument the router to give you parameterized transaction names. It might be worth giving it a try

Yeah, sure. We will use this package in the future.

@antoantonydh
Copy link
Author

antoantonydh commented Nov 8, 2022

image

@Lms24 v7.13.0 on the left side and v7.12.1 on the right side

@adarnon
Copy link

adarnon commented Nov 9, 2022

We are seeing a similar infinite pending behavior in our Next.js app (with Sentry). Gonna try downgrading Sentry and see if it keeps happening

@Lms24
Copy link
Member

Lms24 commented Nov 10, 2022

Hi @adarnon I would be very curious about the result of your investigation. If you can confirm that 7.12.1 did still work fine and 7.13.0 started to show this behaviour the problem around #5697 seems to be broader than just Angular.

It might be worthwile to investigate for a short time if both Next and Angular use polyfills thta aren't compatible but if this doesn't lead to anything, we should probably revert #5697 or make the keepAlive flag somehow optional per framework (or opt-in)

@adarnon
Copy link

adarnon commented Nov 11, 2022

We ended up staying with 7.18 but adding transportOptions: { fetchOptions: { keepalive: false } } to Sentry client init (which effectively turns off #5697). So far a full day without hanging requests, so to me looks like that was the culprit

@adarnon
Copy link

adarnon commented Nov 15, 2022

Confirming we haven't had this issue since turning off the keepalive flag in the client. So it's definitely the issue and the above solution is a workaround

@github-actions
Copy link
Contributor

github-actions bot commented Dec 7, 2022

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@adarnon
Copy link

adarnon commented Dec 7, 2022

This is a huge client-side bug, kind of upsetting no one has bothered responding... Please do not close

@Lms24
Copy link
Member

Lms24 commented Dec 7, 2022

Not going to close this, no worries ;)
We're quite busy at the moment with getting Sentry Session Replay stable. Unfortunately, we can only make long-term decisions about the keepAlive flag once Replay is in a more stable form because right now it needs this flag. Will make a note though to revisit this. Sorry for the inconvenience.

@Lms24 Lms24 removed the Package: angular Issues related to the Sentry Angular SDK label Dec 7, 2022
@ron23
Copy link

ron23 commented Dec 14, 2022

Seeing it with 7.26 and React using @senty-browser.
The workaround #6049 (comment) worked.
This is without even navigating between pages or anything like that. We're just sending around 10 events at the same time (somewhat a bug on our side) so I would assume something with the browser queueing those network requests isn't working.

@antoantonydh
Copy link
Author

I can also confirm that issue is not there after turning off the keepalive flag in the client(#6049 (comment)).

@lforst
Copy link
Member

lforst commented Dec 15, 2022

I am wondering if this is just some protection mechanism browsers employ when too many requests with keepalive are sent. I suspect this isn't an immediate problem with the SDK but rather the way keepalive works. Removing keepalive comes with tradeoffs:

+ Get rid of pending requests
- Errors that happen during navigation are getting clobbered

Considering that the pending requests are more or less just randomly sampled by the browser and not sending errors during navigation prevents an entire class of errors being sent to Sentry, I am leaning towards keeping the keepalive flag. But no definitive answer here.

@mydea
Copy link
Member

mydea commented Mar 21, 2023

Update: We have finally identified the underlying issue and have a fix pending to resolve this here: #7553

Thanks for the patience on this, it was rather tricky tracking this down, but we finally got to the bottom of it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
8 participants