Skip to content

Firestore indexDB persistence corrupted after user "Clear site data" #8593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
entrenadorhispano opened this issue Oct 24, 2024 · 10 comments · Fixed by #8871
Closed

Firestore indexDB persistence corrupted after user "Clear site data" #8593

entrenadorhispano opened this issue Oct 24, 2024 · 10 comments · Fixed by #8871
Assignees
@dconeybe
Labels
api: firestore bug

Comments

@entrenadorhispano
Copy link

Operating System

Windows 11, Chrome, Edge, Android, iOS

Environment (if applicable)

Chrome all major versions, Edge, Safari iOS tested on 16

Firebase SDK Version

10.14

Firebase SDK Product(s)

Firestore

Project Tooling

Vanilla JS.
Also tried on other web frameworks.

Detailed Problem Description

If Firestore persistence is active and a doc is cached. Then the user clears the site data, and and doc is updated on backend.
Cache gets corrupted and now the firestore listener returns NULL document FROM CACHE

I, leave some concrete steps for reproduction below.
(You have to put your own firebase_api_key for security)

Firebase.err.mp4

Steps and code to reproduce issue

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1.0"
    />
    <title>Firebase Cache Bug</title>
  </head>

  <script type="module">
    //For this sample Im using vanilla JS and firebase from CDN.
    //But the same error was tested on sveltekit + vite and firebase js sdk.

    import { initializeApp } from 'https://www.gstatic.com/firebasejs/10.14.1/firebase-app.js';
    import {
      initializeFirestore,
      CACHE_SIZE_UNLIMITED,
      persistentLocalCache,
      persistentMultipleTabManager,
      onSnapshot,
      doc,
      getDoc,
      clearIndexedDbPersistence,
    } from 'https://www.gstatic.com/firebasejs/10.14.1/firebase-firestore.js';

    import {
      signInWithEmailAndPassword,
      onAuthStateChanged,
    } from 'https://www.gstatic.com/firebasejs/10.14.1/firebase-auth.js';

    const firebaseConfig = {
      //Change this to your FIREBASE API KEYS
    };

    // Initialize Firebase
    const app = initializeApp(firebaseConfig);
    const db = initializeFirestore(app, {
      localCache: persistentLocalCache(),
    });

    // You can use this function to clear all the persistence on your IndexedDB.
    // This will fix the problem
    //
    //
    /*clearIndexedDbPersistence(db);*/

    //Getting the documentData
    const docRef = doc(db, 'tests', 'test_document');
    const unsub = onSnapshot(docRef, (doc) => {
      console.log('Current doc: ', doc);
      if (doc.exists()) {
        document.getElementById('display').innerHTML = doc.data().data;
      }
      document.getElementById('diplay-from-cache').innerHTML = JSON.stringify(
        doc.metadata,
        null,
        2
      );
    });
  </script>

  <body>
    <h1>Firebase BUG</h1>
    <blockquote>
      If user clear site data and document is modified from outside, firestore
      IndexedDB cache return null document from cache.
    </blockquote>
    <p>Setup:</p>
    <ul>
      <li>1. Change your FIREBASE API KEYS</li>
      <li>2. Create a document on tests/test_document {data:'Hello world'}</li>
    </ul>

    <hr />

    <ul>
      <li>1. Refresh page (To cache the doc)</li>
      <li>3. Clear site data</li>
      <li>4. Modify the document on firebase console</li>
      <li>5. Refresh</li>
      <li>6. Refresh Again... (Doc return blank from cache:true)</li>
    </ul>

    <div class="card">
      <h2 class="card-header">DocData:</h2>
      <p id="display"></p>
      <h2 class="card-header">Metadata:</h2>
      <pre id="diplay-from-cache"></pre>
    </div>
    <pre id="display-auth"></pre>
  </body>
</html>

<style>
  .card {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    padding: 1rem;
    border-radius: 0.5rem;
    box-shadow: 0 0 1rem rgba(0, 0, 0, 0.2);
  }
  .card-header {
    font-weight: bold;
  }
</style>
@entrenadorhispano entrenadorhispano added new A new issue that hasn't be categoirzed as question, bug or feature request question labels Oct 24, 2024
@google-oss-bot
Copy link
Contributor

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

@looptheloop88 looptheloop88 added api: firestore and removed needs-triage new A new issue that hasn't be categoirzed as question, bug or feature request labels Oct 24, 2024
@dconeybe dconeybe self-assigned this Oct 24, 2024
@looptheloop88
Copy link

Hi @entrenadorhispano, thank you for sharing a very detailed reproduction steps, video recording and the code snippet. I was able to easily reproduce same the behavior. I tried both the SDK version that you used and the latest 11.0.1 version. I also verified that adding clearIndexedDbPersistence(db) will fix the issue.

I will inform our engineering team of this issue. Please keep an eye out for any additional information they may provide.

@entrenadorhispano
Copy link
Author

Another Part of the Bug

I found another part of this bug.

Once the document cache is corrupted, even getDocFromServer() will fetch the cached, non-existent document.
So there's no way to create a fallback.

Severity of the Bug

I would also advise that this bug is severe and makes Firestore offline persistence unusable in production web apps.
(Perhaps some companies just haven't noticed yet.)

For example:
Production apps that depend on fetching user data to display content or redirect to "create user data" may easily overwrite and lose user documents if the user deletes their browser history (which users do frequently).

In JS Frameworks, It May Be Even Worse

I also tried the same process in frameworks like Svelte. There, you don't even have to modify data in the console. Simply clearing site data is enough to trigger this bug and start serving cached, non-existent documents.

Thank you very much for your support and time in dealing with this.

@aliechti
Copy link

I would like to share my experiences with persistence on this issue:

  • getDocFromServer() is also corrupted in my case.
  • I have been facing this problem for 2 years and haven't been able to find a fix other than letting users click a button to clear persistence manually, which is a really bad solution.
  • I have a specific reproducible case where clearing the browser cache isn't even required. Here's the flow:
    • Open a fresh browser session (e.g., incognito mode).
    • The user logs in using Google Login, creating a new auth user.
    • This triggers a user().onCreate function that creates two new documents.
    • There is a snapshot query on the client side that checks if the documents are created. Both documents are created successfully in Firestore, but only one of them is received by the frontend. The other document never resolves.
    • Even after refreshing the page, the missing document never resolves.

I want to thank @entrenadorhispano for the very detailed description of the problem.

This is a very serious issue and must be addressed urgently. It's incredibly difficult to debug unless you have a deep understanding of the inner workings of Firestore. Problems like this can drive developers away from Firebase. Personally, I am very close to abandoning Firebase entirely because of issues like this.

I didn't create an issue on my own earlier because I just couldn't pinpoint the problem and wasn't able to make a good description without that. Also, there were many cases where I thought it was my fault and probably fixed it somehow, but it the problem came back again.

Please prioritize a fix for this.

@entrenadorhispano
Copy link
Author

Hi guys, any news on this issue?

@dconeybe
Copy link
Contributor

Apologies for the delay. I have not had time to dig into it yet. I will ASAP though.

@garrett-wombat garrett-wombat mentioned this issue Mar 20, 2025
Open
@dconeybe
Copy link
Contributor

After investigation, what's happening is that when the IndexedDB database is wiped the database connection gets closed and Firestore's internals recover by transparently re-opening the database connection as if nothing had happened. But, the snapshot listener causes a bunch of state to be loaded from the IndexedDB database into memory, which unbeknownst to Firestore, becomes stale once the IndexedDB database is cleared. When an updated snapshot is received from the backend then Firestore tries to save it into the IndexedDB database, which had since been wiped, and the data it writes is not consistent because it assumes its in-memory cached data is still in the database. So when the website is refreshed in the browser it tries to load the partial data from the IndexedDB database, which is basically corrupted.

Hopefully that explanation made some sense.

As for a "fix", the best thing I can think of is to fail loudly if the IndexedDB database connection is unexpectedly closed, and disallow any writes to the IndexedDB database until a refresh. That way, no partial and/or inconsistent data can be written. Then, upon refreshing the web page, everything will work as if it were the first page load and the IndexedDB database were empty. After clearing site data, I presume that a page refresh would be the "natural" thing to do anyways. Do people generally expect that a web site continues to function after clearing site data? I know I wouldn't but I'm also a technical person and not an average user.

@dconeybe dconeybe mentioned this issue Mar 27, 2025
Merged
@dconeybe
Copy link
Contributor

I'm happy to report that I have a fix in progress: #8871. I'll update here once it's merged.

@entrenadorhispano
Copy link
Author

Hey, these are excellent news!

Thank you for your time and effort!

1. The problem isn't just local to our app.

Sometimes people clear their entire browsing history...

So, it's not just that they delete our app cache or site data. (In my tests, clearing the entire browsing history from another page also creates the bad Firestore state...
So, it's not a problem we can fix by simply listening to our app window event when a user 'clicks to clean our app site data').

2. Is there any way or method to detect when Firestore is broken?

I've tried to fetch documents explicitly from the server using getDocFromServer(reference)... and it fails when Firestore is already broken. (This may be another related bug.)

3. Simply refreshing the app may not solve the problem.

The only way I could reset the state is to use clearIndexedDbPersistence(db).
Simply refreshing the page will not solve it.

4. Regarding the current fix implementation... is it a patched version, or do we have to wait?

I saw the fix is merged.
Will it be available in a patch for us to use right now? Or do we have to wait for the entire Firestore release cycle?

THANKS A LOT!

@dconeybe
Copy link
Contributor

The fix has now been merged. Unfortunately, it didn't get merged in time for next week's release, so it will get included in the release after that, which is scheduled for end of April. If you'd like to test out the fix prior to the release just let me know and I'll put instructions here for building the Firebase JS SDK from source.

@google-oss-bot google-oss-bot mentioned this issue Apr 22, 2025
Merged
@firebase firebase locked and limited conversation to collaborators Apr 28, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dconeybe dconeybe

Labels
api: firestore bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

firestore: fix undefined document snapshot data after "clear site data" firebase/firebase-js-sdk
5 participants
@aliechti @google-oss-bot @dconeybe @looptheloop88 @entrenadorhispano