wip: sqlite experiment

[OceanOak]

Changes made:

1. update the docker file to include sqlite3
2. add a new migrations file for SQLite
3. add Microsoft.Data.Sqlite library for .NET (provides a simple way to work with SQLite databases)
4. in the LibCloud/PackageManager add functions to save functions, types, and constants to the sqlite db
5. in LibPackageManager/PackageManager.fs, add a cache:
   • pre-fill the cache
   • then, just access it to retrieve types, functions, and constants
6. handle the TODO that is in both BuiltinCliHost/Libs/Cli.fs and backend/src/Cli/Cli.fs
// TODO: de-dupe with other Cli.fs
   so that the package manager is not called twice.
7. had to remove the /p:PublishTrimmed=true option to avoid MessagePack errors
8. embedded sqlite into the executable. The script to create the compiled SQLite shared libraries was copied from wip: include sqlite in darklang cli builds #5389. The only addition I made was to include SQLite compile flags related to size reduction https://www.sqlite.org/compile.html

[OceanOak]

Migration notes:

differences from PostgreSQL:

• No TIMESTAMPTZ - using TEXT for timestamps

• No ENUM types - using TEXT with CHECK constraints
  e.g. CREATE TYPE scheduling_rule_type AS ENUM ('pause', 'block'); -> CREATE TABLE scheduling_rules_v0 (rule_type TEXT NOT NULL CHECK (rule_type IN ('pause', 'block')))

• No JSONB - using TEXT for JSON

• No UUID type - using TEXT

• No GIN index

• Arrays (BIGINT[]) stored as TEXT

• No Now() using datetime('now')

• BYTEA is stored as BLOB

• instead of REFERENCES accounts_v0(id) use FOREIGN KEY(account_id) REFERENCES accounts_v0(id)

• BIGINT stored as an INTEGER

• INT stored as INTEGER

• CONSTRAINT user_data_key_uniq UNIQUE (canvas_id, table_tlid, dark_version, user_version, key) -> UNIQUE (canvas_id, table_tlid, dark_version, user_version, key)

• I was running migrations manually from the terminal using:
  sqlite3 sqlite-db/sqlitedb.db < backend/migrations/20251001_000000_migrationsSqlite.sqlite3

[OceanOak]

MessagePack error

DEBUG: Failed to deserialize type 0fcd5847-701c-4da7-9f24-adbe8b5eb397 (Exception+InternalException: error deserializing toplevel
 ---> MessagePack.MessagePackSerializationException: Failed to deserialize LibBinarySerialization.SerializedTypes+PackageType+PackageType value.
 ---> MessagePack.MessagePackSerializationException: Unexpected msgpack code 149 (fixarray) encountered.
   at MessagePack.MessagePackReader.ThrowInvalidCode(Byte code)
   at MessagePack.MessagePackReader.TryReadMapHeader(Int32& count)
   at MessagePack.MessagePackReader.ReadMapHeader()
   at MessagePack.Formatters.LibBinarySerialization_SerializedTypes\.PackageType\.PackageTypeFormatter1.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   --- End of inner exception stack trace ---
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](ReadOnlyMemory`1 buffer, MessagePackSerializerOptions options, CancellationToken cancellationToken)
   at LibBinarySerialization.BinarySerialization.PackageType.deserialize@76-1.Invoke(Unit unitVar0)
   at LibBinarySerialization.BinarySerialization.wrapSerializationException[a](String id, FSharpFunc`2 f)
   --- End of inner exception stack trace ---
   at LibBinarySerialization.BinarySerialization.wrapSerializationException[a](String id, FSharpFunc`2 f)
   at LibBinarySerialization.BinarySerialization.PackageType.deserialize(Guid uuid, Byte[] data)
   at [email protected](Unit unitVar)
   at Ply.TplPrimitives.tryWith[u](FSharpFunc`2 continuation, FSharpFunc`2 catch))
DEBUG: Failed to deserialize type cd64635d-6f94-4a2f-ab03-91e09d8e73da (Exception+InternalException: error deserializing toplevel
 ---> MessagePack.MessagePackSerializationException: Failed to deserialize LibBinarySerialization.SerializedTypes+PackageType+PackageType value.
 ---> MessagePack.MessagePackSerializationException: Unexpected msgpack code 149 (fixarray) encountered.
   at MessagePack.MessagePackReader.ThrowInvalidCode(Byte code)
   at MessagePack.MessagePackReader.TryReadMapHeader(Int32& count)
   at MessagePack.MessagePackReader.ReadMapHeader()
   at MessagePack.Formatters.LibBinarySerialization_SerializedTypes\.PackageType\.PackageTypeFormatter1.Deserialize(MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   --- End of inner exception stack trace ---
   
   ...

[OceanOak]

DotTrace results:
Running from the executable, reaching to the local packager manager, using sqlite:

Sampling

Timeline

Tracing

[OceanOak]

DotTrace results:
Running from the executable : no sqlite changes, no local PM (current main)

Sampling:

Timeline:

Tracing: