Wrongly deleted sessions

[SebC99]

Issue Description

We do log a big number of INVALID_SESSION_TOKEN errors for our users, appearing without any apparent logic: sessions are not expired, the user hasn't changed the password, sometimes it happens multiple times in the same day, and we can't figure out why.
Is there any way for us to better detect the reason for this?
Are we the only one with this weird behavior?

I know there's multiple places in the code where Sessions are deleted (or session caches) but I can't see why it could explain that.

Any help would be appreciated :)

[mtrezza]

• Is there any pattern when you see these logs?
• Are you only getting these errors recently?

[SebC99]

Unfortunately not, we can't find any pattern
It's been a while, but maybe slightly more during last the last weeks, but we also had an increase of users coming back after a few weeks off because of the covid. But with a 1 year session token, that should not happen. We even have cases with users being disconnected each day (through FB login)

[dplewis]

@SebC99 @mtrezza Can I add you to the Parse Team? I would have sent an email sorry about that.

[mtrezza]

@dplewis Sure, please go ahead.

@SebC99 Could it be some external reason? I am thinking of an iOS setting that uninstalls apps automatically when they are not used for some time, and then restores them from cloud. Or auto-backup on Android, see parse-community/Parse-SDK-Android#970 (comment). If a user opens the app again after 2 months, maybe it gets restored and that causes some issues, hence you see more errors after covid?

[SebC99]

@dplewis not sure it was addressed to me, but if so yes for sure

@mtrezza We have seen the same on test devices where we know for sure this iOS setting is not activated... So I'm guessing it's not linked.
It seems to be somewhat proportional with the number of devices used by an account, but it's not even sure.

BTW we also know that we have an issue each time a user restore its phone as Parse always fails to restore the currentUser (we have activated the anonymous account, so it's hard to detect when this occurs). But I think that's an issue with Parse iOS SDK, and it's another subject.

[mtrezza]

I guess the first step would be determining whether the issue is server or client side. The error can be thrown in many different places on the server, can you tell from the stack trace where it is thrown?

[SebC99]

I know. A hint is that it seems to occur on both iOS and Android apps, so it seems to be server side.
But as we are 90% iOS, it's much more frequent on iOS and it's hard to be sure.

On client side, we only see the 209 Invalid Session Token error, but with AWS X-Ray we were able to see more details, and it seems it's always this trace:

Error: Invalid session token
at Object.getAuthForSessionToken (/var/app/current/node_modules/parse-server/lib/Auth.js:106)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97)

Which is why I supposed it was an issue with wrongly deleted sessions.

[mtrezza]

/lib/Auth.js:106

I assume thats this?

parse-server/src/Auth.js

Lines 104 to 109 in de79b70

	if (results.length !== 1 || !results[0]['user']) {
	throw new Parse.Error(
	Parse.Error.INVALID_SESSION_TOKEN,
	'Invalid session token'
	);
	}

[SebC99]

I think so...
And we have a unique index, and we always have the 'user' field, so it's really because of a deletion of the session object... but why?

[mtrezza]

That really seems like either the session was removed or the token was incorrect. Did you check whether the requests may be malicious, maybe trying to brute force the session token? See #6755

[SebC99]

We don't see any weird requests, and it also happens with our own devices / sessions.
We even found cases where all the sessions token of one user were removed without reasons.

[mtrezza]

Have you set a TTL index on the session collection expireAt field?

[mtrezza]

We even found cases where all the sessions token of one user were removed without reasons.

Interesting that you say "all the session tokens". There should ever only be one session object per user per installation. If all session objects of the user of different installations are deleted, that could point to an issue in your custom code.

If I understand this correctly, before a new session object is created, all existing sessions objects of that user/installation combination are destroyed:

parse-server/src/RestWrite.js

Line 975 in 288e746

RestWrite.prototype.destroyDuplicatedSessions = function() {

Interestingly, this is executed before runDatabaseOperation which runs the actual query. If creating the new session fails for some reason, all existing sessions would already be deleted and there would be not any session object left for that user/installation combination, if I understood the flow correctly. The new session is actually created before that in handleSession.

[SebC99]

No TTL index I think, and we have a lots of old session token, so I guess it wouldn't be the case with a TTL index. (It could be nice to have one haha)
All sessions tokens because or tests accounts are used on many devices, and we often logout/login from these accounts, and they are also used on simulators... so they do have lots of installationId :)

But the issue of deleted sessions is for all users.

[mtrezza]

(It could be nice to have one haha)

I added functionality to build a TTL index in #6748 so someone could add that feature easily after merge

All sessions tokens because or tests accounts are used on many devices, and we often logout/login from these accounts, and they are also used on simulators... so they do have lots of installationId :)

I cannot think of a debugging scenario in which a session would be deleted while it is still in use on another device, given that each device has its own installation ID. I also have never experienced that when debugging. Maybe the issue is related to your custom code where you manipulate sessions / installations / users?

Could it be that you try to execute some server calls on the client after the user logged out? I think that would cause the same error?

[SebC99]

I cannot think of a debugging scenario in which a session would be deleted while it is still in use on another device, given that each device has its own installation ID. I also have never experienced that when debugging. Maybe the issue is related to your custom code where you manipulate sessions / installations / users?

Agreed. But we don’t manipulate sessions/installations. We use standard logout/login, and the only thing we do is saving the user in the installation. We do have triggers on Users but that’s all.

Could it be that you try to execute some server calls on the client after the user logged out? I think that would cause the same error?
We do unfortunately as on the iOS SDK the notification for invalid session is not blocking. So other requests can be fired but they have the same error result.

Do you think we could patch the server code to add some debug logs or hints that could explain anything?
Thanks

[mtrezza]

Do you think we could patch the server code to add some debug logs or hints that could explain anything?

Definitely, it would be interesting to see which tokens are sent by the clients that are invalid. Now you don't want tokens in your logs, but these are invalid tokens anyway. Then you could log when a token gets deleted, in Parse Server or with even with a mongodb trigger on the DB level. Then you would know:

• Are the tokens real tokens sent by the clients, or does the client have an issue?
• When and why was the token deleted that the client used to auth?

You may want to log logout / password change / password reset operations as well; I don't know off the top of my head whether these trigger a session delete as well.

[SebC99]

Yes I'll try to add some logs thanks

[mtrezza]

Sure, let us know what you found.

[mtrezza]

@SebC99 Are there any new insights on this?

[SebC99]

Unfortunately no, we still have the issue, and we can't figure out why. Our logs in our app are not helping as we only see that people are "loosing" their session token at one point, and that those session tokens were valid just before (with an expiration date in 2021).
So we still don't know why those tokens are being deleted by the server

[mtrezza]

Interesting. I will also take a look into logs. For which platforms (SDKs) do you obverse this?

[SebC99]

We use the iOS, Android and JS SDK and we mostly see that on the iOS one, but that's also because most of our users are using iOS

[mtrezza]

So it does occur with all 3 SDKs?

Can you check that to be sure? For example for Android you'd want to make sure a user does not login with iOS as well and therefore the session is deleted somehow. In other words, if you have users who only use the Android SDK or only the JS SDK, whether they are affected as well.

Did you patch the server to get more logs as discussed?

[SebC99]

Really hard to say as it's seems to be random.
100% sure it occurs on iOS.
50% sure it occurs on Android (but there's other issues in the Android SDK so it might not be the same reason).
The web app can't be compared as there's no real local storage.

We have almost no users using both Android and iOS, but we do have lots of users with multiple iOS devices, and some of them are also using the web app on their computer.

As far as we can see, we see more of that issue with users with multiple devices.

As for the logs, not for now as I don't know what kind of logs to add in parse-server.... We did add logs in the clients app but it didn't help. We could add a beforeDelete hook in the cloud code, but I don't know what to display in the logs that could be helpful.

[mtrezza]

Let's analyze a bit more where we observe it and what we observe, so we don't overlook anything.

we only see that people are "loosing" their session token at one point

• By "loosing" I understand that you log INVALID_SESSION_TOKEN on the app?
• Do you also get user feedback about session issues or is that only something you technically observe?
• What are users usually doing before / after INVALID_SESSION_TOKEN? For example, if users log out manually in the app (maybe in the process when they delete their accounts), but some requests are still queued, these requests will fail and would log INVALID_SESSION_TOKEN.

50% sure it occurs on Android... there's other issues in the Android SDK so it might not be the same reason... The web app can't be compared as there's no real local storage.

• Can you add logs to the web app to see whether you log INVALID_SESSION_TOKEN there as well?
• Can you patch the server as discussed so we know which tokens exactly are considered invalid? First, we want to make sure that /lib/Auth.js:106 is actually

  parse-server/src/Auth.js

  Lines 104 to 109 in de79b70

  	if (results.length !== 1 || !results[0]['user']) {
  	throw new Parse.Error(
  	Parse.Error.INVALID_SESSION_TOKEN,
  	'Invalid session token'
  	);
  	}

  because the line numbers of lib and src are not the same and there are multiple places where INVALID_SESSION_TOKEN can be thrown. Maybe you can look into your deployment to see what exactly is at line 106. Then add some log lines there to write the whole session object to the logs (the token is invalid at this point anyway, so no security issue). We want to specifically look at the expiration date of the session and the auth method. Maybe there is a pattern and we only see this with a social login for example. Maybe users are deauthorizing the app on a social platform which may render their session token in your app invalid, depending on how you setup your auth logic.

[SebC99]

By "loosing" I understand that you log INVALID_SESSION_TOKEN on the app?
Exactly

Do you also get user feedback about session issues or is that only something you technically observe?
Yes they are disconnected without any reason and we have to reconnect them

What are users usually doing before / after INVALID_SESSION_TOKEN? For example, if users log out manually in the app (maybe in the process when they delete their accounts), but some requests are still queued, these requests will fail and would log INVALID_SESSION_TOKEN.
Most of the time they are just using the app, and that’s where is the real issue! They open the app and get that error randomly (it’s rare of course but it is clearly too often)

Can you add logs to the web app to see whether you log INVALID_SESSION_TOKEN there as well?
Sure!

First, we want to make sure that /lib/Auth.js:106 is actually
I think it is sure as there’s only one error in Auth.js with this comment ´Invalid session token´ (We can see the comment in IOS logs)

I’ll add those logs yes!
(But again, it appears to ourselves with our test devices where we don’t deauthorize anything)

Thanks !

[mtrezza]

Just for reference I'm adding #6726 because it is somewhat related.

[SebC99]

@mtrezza we might have found something...
On iOS, when we run saveInBackground() on the currentUser, it sometimes send the user's password with the request, as if it has been changed (which had not), and the transformUser method of the RestWrite.js file of Parse Server marks the request with generateNewSession: true, which will invalidate every sessions of that user, even the active one.
And it's not the case on the web as far as we can see.
Now we need to figure out why the password is still sent sometimes.

[mtrezza]

it sometimes send the user's password with the request, as if it has been changed (which had not)

Now that's interesting.

• Any idea why it would only send that sometimes?
• What other properties of a user are you changing before you call saveInBackground?
• Can you replicate this with a minimal example? If you can only observe this as part of your app logic, it could be related specifically to your app logic.

Parse Server marks the request with generateNewSession: true, which will invalidate every sessions of that user

Is it expected that changing a password causes invalidates all other sessions? I know some services do that (e.g. Google) while others do not (e.g. Instagram). I could see that as a feature, but it would have to be consistent across the SDKs of course. If it's currently inconsistent maybe we can take this as a opportunity to introduce a Parse Server configuration parameter for this, unless there is a best practice we should abide to.

[SebC99]

Well, on one device, when we do a login then just a save, it occurs each time (transforming the session from a login one to a signup one)
On another device, never...
We investigate!

(I think there’s already a setting in parse server for that: revokeSessionOnPasswordReset, and it's true by default)

[SebC99]

BTW, this code (in Auth.js) is weird:

RestWrite.prototype.createSessionToken = async function() {
  // cloud installationId from Cloud Code,
  // never create session tokens from there.
  if (this.auth.installationId && this.auth.installationId === 'cloud') {
    return;
  }
  const { sessionData, createSession } = Auth.createSession(this.config, {
    userId: this.objectId(),
    createdWith: {
      action: this.storage['authProvider'] ? 'login' : 'signup',
      authProvider: this.storage['authProvider'] || 'password',
    },
    installationId: this.auth.installationId,
  });
  if (this.response && this.response.response) {
    this.response.response.sessionToken = sessionData.sessionToken;
  }
  return createSession();
};

It means each time a new sessionToken is created (because of a save with a password, even the same as before), the action is set at signup. It shouldn't have any impact, but it's clearly not a signup when a user changes his password.

[mtrezza]

I think there’s already a setting in parse server for that: revokeSessionOnPasswordReset, and it's true by default

You are right, Parse Server never stops to amaze.

[mtrezza]

Can we close this issue or do you have any further info?

[SebC99]

No more info except it's still happening...

[mtrezza]

I'm reluctant to close this, because if it is indeed a bug, it may be an important one that affects user experience.

You wrote

Well, on one device, when we do a login then just a save, it occurs each time (transforming the session from a login one to a signup one)

Can you still observe this? Could you just debug the iOS app and Parse Server and step through the code to see why this is happening?

[SebC99]

Unfortunately as soon as we compiled a debug version on the device, it stopped on that one. So we're back with the bug happening randomly, with no clue for now

[nopol10]

Not sure if it is related to this issue but I've been experiencing a strange problem with invalid session tokens as well.
Here's what happens:

1. Login with Google Auth on my web app
2. App gets a session token, it matches with what's in the database. Login success.
3. Logout of Parse and google auth. No issues.

But if I do this

1. Refrsh the page and login again with google auth. Don't perform a logout
2. Refresh the page
3. Since I don't store the session token, I'll have to login again. I login with google again. No issues, apparently...
4. Try to logout. Invalid session token gets returned instead.
   So I go ahead and inspect the session token received in the login POST request vs the session token returned in getSessionToken in the User object from loginWith. Turns out they are different, the one in the POST request is what's in the database but the one returned to me by the function is the previous session token.
   Am I doing something wrong or is there an actual issue here?
   I'm on 4.2.0 btw as 4.3.0 has the issue which prevents google auth from working .

[oli107]

I am experiencing a similar issue and am uncertain if this is directly related, but does this only occur when directAccess is set to true? I've tracked down missing session tokens in my app (entirely using the JS SDK) to when there is an update of the user record and directAccess is enabled. It triggers a logout call which removes the session token. I can't quite work out how this is called? I've tried putting some logging into the transformUser method of RestWrite.js but that doesn't seem to get fired when direct access is enabled? Disabling direct access prevents the logout from being called and the sessions are not deleting. I've searched the code until my eyeballs bleed, could either of you @SebC99 or @mtrezza point me in the direction for where a logout call might be made after modifying the user object? It definitely seems to occur, by adding some logging into the ParseServerRESTController.js I can see that a POST to logout is called straight after a PUT to /classes/_User.

[mtrezza]

by adding some logging into the ParseServerRESTController.js I can see that a POST to logout is called straight after a PUT to /classes/_User.

@oli107 This is certainly an interesting observation and to my knowledge the only clearly reproducible case so far. Can you reproduce this with a clean local installation of Parse Server? If so, you could debug Parse Server while running into this scenario and find the stack trace that calls logout.

[mtrezza]

@oli107 @nopol10 @SebC99 Did anyone have any luck in reproducing this issue?

[nopol10]

Haven't had the time to reproduce in a brand new installation yet unfortunately. Not sure if this question

I am experiencing a similar issue and am uncertain if this is directly related, but does this only occur when directAccess is set to true?

was meant for me but I haven't explicity set directAccess so I guess it should default to false.

[mtrezza]

@SebC99 I stumbled upon this comment and it reminded me of this unsolved mystery - maybe related?

[SebC99]

@mtrezza the last comment reminds me of the issue we had with the password being sent again, which leads to the parse-server thinking it's a change of password and thus deleting all sessions

[mtrezza]

@SebC99 That's what I was thinking too, sounds remarkably related. Is this still an issue you're investigating?

[SebC99]

the password being sent was an issue on our side, so it has been fixed, but we still have the deleted session issues (just a bit less), and we have not a single clue about it.

[mtrezza]

@SebC99 could it have something to do with expiration of 3rd party tokens? Have you observed this only with FB access token for example? Do you have any customization related to the auth adapter or access token / Parse session handling?

[SebC99]

No customization on this, but I think there's room for improvement yes: typically, as parse-server only test the token on login, when the FB token expires, you can have an active Parse user (with a valid session) but with an invalid Facebook token. So if your app uses the FB token to retrieve data (like friends, profile info or anything), Facebook will show an alert (if you use the FB SDK) whereas it's still valid for parse-server.
I guess I would have check FB Token validity at each request on parse-server, invalidating parse sessions when the token is expired. But it's a difficult (and an other) subject.

But to answer your question, it's more often with FB access, but it is also happening with email access.
What we sometimes observed with FB access, is that if you are login with FB on the mobile app, and on the web app, and you logout from the web app, it sometimes log you out from the mobile app too. It seems to be less the case at the moment, but we never understood why. It seems like multiple sessions were deleted, but as it was hard to reproduce, it was hard to understand

[mtrezza]

I guess I would have check FB Token validity at each request on parse-server, invalidating parse sessions when the token is expired. But it's a difficult (and an other) subject.

I am not sure whether/how the parse session validity should be related to the 3rd party token validity. It would be interesting to see what's best practice.

But to answer your question, it's more often with FB access, but it is also happening with email access.

Then I'd still leave the issue open for now.

[mtrezza]

I'm closing this due to lack of information for further investigation.