Support SQLite implementation for IStore

[theobouwman]

Hi,

We are currently using matrix js sdk in our react native app. As indexeddb is not supported in RN we currently use the memory store.

The problem is that when you open a notification from a room there is a loading state because nothin is stored on device. We are using TimelineWindow for our rooms with pagination etc https://matrix-org.github.io/matrix-js-sdk/classes/matrix.TimelineWindow.html.

So I have 2 questions:

• 1. What would some kind of solution be to store the 40 recent events of a room, and load it in the TimelineWindow?
• 2. as (1) would be quick workaround I am considering contributing to the sdk to implement a SQLite store implantation. Would this mean to just implement the IStore in https://github.com/matrix-org/matrix-js-sdk/blob/develop/src/store/index.ts methods to a SQLite implantation?

Thanks

[richvdh]

@theobouwman it's worth knowing that there are two stores in matrix-js-sdk: the IStore which you have found, but the crypto stack has its own, separate, store. If you want to support end-to-end encryption, you'll also need to update the crypto stack to support a different database implementation. matrix-org/matrix-rust-sdk-crypto-wasm#168 has some discussion about that.

[theobouwman]

@richvdh I know. I am first planning to implement the store without the end-to-end encryption.

But, by implementing the IStore (https://github.com/matrix-org/matrix-js-sdk/blob/develop/src/store/index.ts) should be a good start right?

And will the TimelineWindow class use the configured store as well?

[richvdh]

Fair enough.

I don't think TimelineWindow uses the store directly: it is a layer on top of other things that do. I'm pretty sure that if you implement IStore it will solve your problem.

[theobouwman]

Okay. I made a start.
Unfortunately I am getting a Caught /sync error [TypeError: undefined is not a function] on sync.

  storedFilters set filter:@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com:0 [{"definition": {"room": [Object]}, "filterId": "0", "roomFilter": [Object], "roomTimelineFilter": [Object], "userId": "@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com"}]
 LOG  set filter id by name with filterId FILTER_SYNC_@a09a125f-0346-4b98-a81c-5439d927b8bf:test.exploremomo.com 0
 DEBUG  Sending initial sync request...
 DEBUG  FetchHttpApi: --> GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&_cacheBuster=xxx
 DEBUG  Waiting for saved sync before starting sync processing...
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb/feature
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/c15f1d2f-9127-4802-a872-ffe4066e39fb/events/47f80386-a482-4237-b4db-beaffd58982c/attending
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c/messages/a5f3edf4-e338-42dc-ab79-848f0f3f8c44/details
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/1195db93-03b2-4b00-b388-6361dec59fae/groups/8a18e426-7cad-4763-bde6-a71175d3171c/messages/cd16aebc-62ec-4df6-9845-e9ddd41cbf12/details
 DEBUG  FetchHttpApi: <-- GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&_cacheBuster=xxx [165ms 200]
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/8a358d46-eb84-4b43-9ede-be7341410e94/attending
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/8a358d46-eb84-4b43-9ede-be7341410e94/messages/c044fcf6-458d-4814-865f-8c8c1c44e257/details
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/1e804092-3650-48f9-b9d2-d7bffa221293/details
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/e181d9db-374e-4a84-990f-ecdb3684b7a4/details
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/4dc2cfd3-d4f2-4b95-8be6-657ea1d49208/details
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/f1c06a44-16fc-4791-ba8d-f5a05b235538/events/66a45490-c7ce-47f4-98dc-12e984820053/messages/784c368f-492b-4866-b592-f917101bd21c/details
 LOG  get user @momo-admin:test.exploremomo.com {"_events": {}, "_eventsCount": 5, "currentlyActive": true, "displayName": "@momo-admin:test.exploremomo.com", "events": {"presence": {"content": [Object], "sender": "@momo-admin:test.exploremomo.com", "type": "m.presence"}}, "lastActiveAgo": 114914, "lastPresenceTs": 1733334816197, "modified": 1733334816197, "presence": "online", "rawDisplayName": "@momo-admin:test.exploremomo.com", "userId": "@momo-admin:test.exploremomo.com"}
 **ERROR  Caught /sync error [TypeError: undefined is not a function]**
 LOG  set sync data {"account_data": {"events": [[Object]]}, "device_one_time_keys_count": {}, "device_unused_fallback_key_types": [], "next_batch": "s838_4408_36_3019_614_3_1_9_0_1", "org.matrix.msc2732.device_unused_fallback_key_types": [], "presence": {"events": [[Object], [Object]]}, "rooms": {"join": {"!AlFHBMsEClBnOKiFLy:test.exploremomo.com": [Object]}}}
 LOG  sync PREPARED
 LOG  mmkv roomids []
 LOG  sync SYNCING
 LOG  mmkv roomids []
 INFO  Resuming queue after resumed sync
 DEBUG  Attempting to send queued to-device messages
 LOG  wants save true
 LOG  wants save true
 INFO  store:reallySave
 DEBUG  FetchHttpApi: --> GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&since=xxx
 DEBUG  All queued to-device messages sent
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/9223c358-1135-4e8d-91ea-683c29248ac2
 LOG  |HTTP|GET|200| http://192.168.178.166:8000/organization/9223c358-1135-4e8d-91ea-683c29248ac2/events/e56eacca-ed33-4248-8f8c-bd6e2ee6850e/messages/8357273f-1fc8-4ff0-8abe-5b629a6fe33c/details
 LOG  App was opened by a URL: null
 DEBUG  FetchHttpApi: <-- GET http://192.168.178.166:8008/_matrix/client/v3/sync?filter=xxx&timeout=xxx&since=xxx [266ms 200]
...

import { MatrixEvent, Room, User, Filter, IStateEventWithRoomId, IStartClientOpts, IEvent, ISyncResponse, MemoryStore, IStoredClientOpts, KnownMembership, RoomState, SyncAccumulator } from 'matrix-js-sdk';
import { RoomSummary } from 'matrix-js-sdk/lib/models/room-summary';
import { ToDeviceBatchWithTxnId, IndexedToDeviceBatch } from 'matrix-js-sdk/lib/models/ToDeviceMessage';
import { ISavedSync, IStore, UserCreator } from 'matrix-js-sdk/lib/store';
import { deepCopy, MapWithDefault } from 'matrix-js-sdk/lib/utils';
import { MMKVInstance, MMKVLoader } from 'react-native-mmkv-storage';

const WRITE_DELAY_MS = 1000 * 60 * 1; // once every 1 minutes

const isValidFilterId = (filterId?: string | number | null): boolean => {
    const isValidStr =
        typeof filterId === "string" &&
        !!filterId &&
        filterId !== "undefined" && // exclude these as we've serialized undefined in localStorage before
        filterId !== "null";

    return isValidStr || typeof filterId === "number";
}


class MatrixMMKVStore implements IStore {
    private storage: MMKVInstance;
    private userCreator: UserCreator | null = null;
    private readonly syncAccumulator: SyncAccumulator;
    accountData = new Map<string, MatrixEvent>();
    private syncTs = 0;

    constructor() {
        this.storage = new MMKVLoader().initialize();
        this.syncAccumulator = new SyncAccumulator();
    }

    private prefixKey(key: string): string {
        return `matrix:${key}`;
    }

    async isNewlyCreated(): Promise<boolean> {
        const flag = await this.storage.getBoolAsync(this.prefixKey('isNewlyCreated'));
        if (flag === null) {
            await this.storage.setBoolAsync(this.prefixKey('isNewlyCreated'), true);
            return Promise.resolve(true);
        }
        return Promise.resolve(false);
    }

    getSyncToken(): string | null {
        return this.storage.getString(this.prefixKey('syncToken')) as string | null;
    }

    setSyncToken(token: string = ''): void {
        this.storage.setString(this.prefixKey('syncToken'), token);
    }

    storeRoom(room: Room): void {
        this.storage.setMap(this.prefixKey(`room:${room.roomId}`), room);

        console.log('store room', room.roomId, room)

        const storedRoomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
        let newRoomIds = [...storedRoomIds, room.roomId]
        this.storage.setArray(this.prefixKey('roomIds'), newRoomIds)
    }

    setUserCreator(creator: UserCreator): void {
        this.userCreator = creator;
    }

    getRoom(roomId: string): Room | null {
        const room = this.storage.getMap<Room>(this.prefixKey(`room:${roomId}`));
        console.log('get room', roomId, room)
        return room
    }

    getRooms(): Room[] {
        const roomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
        console.log('mmkv roomids', roomIds)
        if (roomIds.length > 0) {
            const roomKeys = roomIds.map(r => `room:${r}`)
            const roomData = this.storage.getMultipleItems<Room>(roomKeys, 'map');
            const rooms = roomData
                .map((item) => item[1]!)
                .filter((room) => room);

            console.log('get rooms', rooms)
            return rooms
        }

        return []
    }

    removeRoom(roomId: string): void {
        console.log('remove room', roomId)
        this.storage.removeItem(this.prefixKey(`room:${roomId}`));
        const storedRoomIds = this.storage.getArray<string>(this.prefixKey('roomIds')) ?? []
        this.storage.setArray(this.prefixKey('roomIds'), storedRoomIds.filter(r => r !== roomId))
    }

    getRoomSummaries(): RoomSummary[] {
        console.log('get room summaries')
        const rooms = this.getRooms();
        return rooms.filter(room => room.summary !== null).map((room) => room.summary!);
    }

    storeUser(user: User): void {
        console.log('store user', user)
        this.storage.setMap(this.prefixKey(`user:${user.userId}`), user);
    }

    getUser(userId: string): User | null {
        const user = this.storage.getMap<User>(this.prefixKey(`user:${userId}`))
        console.log('get user', userId, user)
        return user
    }

    getUsers(): User[] {
        console.log('get users')
        const keys = this.storage.getAllMMKVInstanceIDs();
        const userKeys = keys.filter((key) => key.startsWith('matrix:user:'));
        const userData = this.storage.getMultipleItems<User>(userKeys, 'map');
        console.log('users:', userData)
        return userData
            .map((item) => item[1]!)
            .filter((user) => user);
    }

    scrollback(room: Room, limit: number): MatrixEvent[] {
        // Placeholder: implement logic for retrieving room scrollback
        console.log('scrollback', room, limit)
        return [];
    }

    storeEvents(room: Room, events: MatrixEvent[], token: string | null, toStart: boolean): void {
        // Placeholder: implement logic to store room events
        console.log('store events', room)
    }

    storeFilter(filter: Filter): void {
        if (!filter?.userId || !filter?.filterId) return;

        console.log('store filter', filter)

        const filterIdUserIdKey = `filter:${filter.userId}:${filter.filterId}`
        
        // let storedFilter = this.storage.getMap<Filter>(this.prefixKey(filterIdUserIdKey))
        // console.log('storedFilter', storedFilter)
        this.storage.setMap(this.prefixKey(filterIdUserIdKey), filter)
        // if (!storedFilter) {
        //     console.log('do store filter', filter)
        //     this.storage.setMap(this.prefixKey(filterIdUserIdKey), filter)
        // }

        let storedFilters = this.storage.getArray<Filter>(this.prefixKey('filters'))
        console.log('storedFilters array:', storedFilters)

        if (!storedFilters || storedFilters.length === 0) {
            storedFilters = []
            console.log('new stored filter')
        }

        storedFilters = storedFilters.filter(f => `filter:${f.userId}:${f.filterId}` !== filterIdUserIdKey)
        storedFilters.push(filter)
        console.log('storedFilters set', filterIdUserIdKey ,storedFilters)

        this.storage.setArray(this.prefixKey('filters'), storedFilters)
    }

    getFilter(userId: string, filterId: string): Filter | null {
        const filter = this.storage.getMap<object>(this.prefixKey(`filter:${userId}:${filterId}`));
        if (!filter) {
            console.log('get filter not found', userId, filterId)
            return null
        }

        const f = Filter.fromJson(userId, filterId, filter)

        console.log('get filter', userId, filterId, 'filterOBJ', f)
        
        return f
    }

    getFilterIdByName(filterName: string): string | null {
        try {
            const filterId = this.storage.getString(this.prefixKey(filterName.replace('FILTER_SYNC_', '')))
            if (isValidFilterId(filterId)) {
                console.log('get filter id by name', filterName, filterId)
                return filterId as string | null;
            }
        } catch {}

        console.log('get filter id by name', filterName, 'null-empty')

        return null;
    }

    setFilterIdByName(filterName: string, filterId?: string): void {
        console.log('set filter id by name with filterId', filterName, filterId)
        if (isValidFilterId(filterId)) {
            this.storage.setString(this.prefixKey(filterName.replace('FILTER_SYNC_', '')), filterId!);
        } else {
            this.storage.removeItem(this.prefixKey(filterName.replace('FILTER_SYNC_', '')));
        }
    }

    storeAccountDataEvents(events: MatrixEvent[]): void {
        console.log('store account data events', events.length)
        events.forEach((event) => {
            this.accountData.set(event.getType(), event);
        });
    }

    getAccountData(eventType: string): MatrixEvent | undefined {
        console.log('get account', eventType)
        return this.accountData.get(eventType);
    }

    async setSyncData(syncData: ISyncResponse): Promise<void> {
        this.storage.setMap(this.prefixKey('syncData'), syncData);
        console.log('set sync data', syncData)

        return Promise.resolve().then(() => {
            this.syncAccumulator.accumulate(syncData)
        })
    }

    wantsSave(): boolean {
        const now = Date.now();
        const want = now - this.syncTs > WRITE_DELAY_MS;
        console.log('wants save', want)
        return want
    }

    save(force?: boolean): Promise<void> {
        if (force || this.wantsSave()) {
            return this.reallySave();
        }
        return Promise.resolve();
    }

    reallySave(): Promise<void> {
        this.syncTs = Date.now(); // set now to guard against multi-writes

        console.info('store:reallySave')

        // work out changed users (this doesn't handle deletions but you
        // can't 'delete' users as they are just presence events).
        // const userTuples: [userId: string, presenceEvent: Partial<IEvent>][] = [];
        // for (const u of this.getUsers()) {
        //     if (this.userModifiedMap[u.userId] === u.getLastModifiedTime()) continue;
        //     if (!u.events.presence) continue;

        //     userTuples.push([u.userId, u.events.presence.event]);

        //     // note that we've saved this version of the user
        //     this.userModifiedMap[u.userId] = u.getLastModifiedTime();
        // }
        return Promise.resolve()

        // return this.backend.syncToDatabase(userTuples);
    }


    async startup(): Promise<void> {
        // No-op for MMKV initialization
        console.log('-------- store startup')
        
        const savedSync = await this.getSavedSync()

        console.log('savwd sunc startup', savedSync)

        if (savedSync) {
            console.log('-------- store startup sync accumulator')
            return this.syncAccumulator.accumulate({
                next_batch: savedSync.nextBatch,
                rooms: savedSync.roomsData,
                account_data: {
                    events: savedSync.accountData
                }
            }, true)
        }

        return Promise.resolve()
    }

    async getSavedSync(copy = true): Promise<ISavedSync | null> {
        const res = this.storage.getMap<ISavedSync|null>(this.prefixKey('syncData'))
        console.log('get saved sync', res)
        return Promise.resolve(res)

        // const data = this.syncAccumulator.getJSON();
        // if (!data.nextBatch) return Promise.resolve(null);
        // if (copy) {
        //     // We must deep copy the stored data so that the /sync processing code doesn't
        //     // corrupt the internal state of the sync accumulator (it adds non-clonable keys)
        //     return Promise.resolve(deepCopy(data));
        // } else {
        //     return Promise.resolve(data);
        // }
    }

    async getSavedSyncToken(): Promise<string | null> {
        const syncData = await this.getSavedSync();
        return syncData?.nextBatch || null;
    }

    async deleteAllData(): Promise<void> {
        this.storage.clearStore();
    }

    async getOutOfBandMembers(roomId: string): Promise<IStateEventWithRoomId[] | null> {
        console.log('get out of band members', roomId)
        const res = this.storage.getArray<IStateEventWithRoomId>(this.prefixKey(`oobMembers:${roomId}`));

        return Promise.resolve(res)
    }

    async setOutOfBandMembers(roomId: string, membershipEvents: IStateEventWithRoomId[]): Promise<void> {
        console.log('set out of band members', roomId)
        
        this.storage.setArray(`oobMembers:${roomId}`, membershipEvents);

        return Promise.resolve()
        
    }

    async clearOutOfBandMembers(roomId: string): Promise<void> {
        console.log('clear out of band members', roomId)
        this.storage.removeItem(this.prefixKey(`oobMembers:${roomId}`));
    }

    async getClientOptions(): Promise<IStartClientOpts | undefined> {
        return await this.storage.getMapAsync<IStartClientOpts>(this.prefixKey('clientOptions')) as IStartClientOpts | undefined;
    }

    async storeClientOptions(options: IStartClientOpts): Promise<void> {
        await this.storage.setMap(this.prefixKey('clientOptions'), options);
    }

    async getPendingEvents(roomId: string): Promise<Partial<IEvent>[]> {
        return await this.storage.getMapAsync<Partial<IEvent>[]>(this.prefixKey(`pendingEvents:${roomId}`)) || [];
    }

    async setPendingEvents(roomId: string, events: Partial<IEvent>[]): Promise<void> {
        await this.storage.setArray(this.prefixKey(`pendingEvents:${roomId}`), events);
    }

    async saveToDeviceBatches(batch: ToDeviceBatchWithTxnId[]): Promise<void> {
        await this.storage.setArrayAsync(this.prefixKey('toDeviceBatches'), batch);
    }

    async getOldestToDeviceBatch(): Promise<IndexedToDeviceBatch | null> {
        const batches = await this.storage.getArrayAsync<IndexedToDeviceBatch>(this.prefixKey('toDeviceBatches'));
        return batches?.[0] || null;
    }

    async removeToDeviceBatch(id: number): Promise<void> {
        const batches = await this.storage.getArray<IndexedToDeviceBatch>(this.prefixKey('toDeviceBatches'));
        if (batches) {
            const updatedBatches = batches.filter((batch) => batch.id !== id);
            await this.storage.setArrayAsync(this.prefixKey('toDeviceBatches'), updatedBatches);
        }
    }

    async destroy(): Promise<void> {
        await this.deleteAllData();
    }
}

export default MatrixMMKVStore;

[richvdh]

Related #657