Integration with SQLite

[chadbrewbaker]

Any interest in a PR for a patch of SQLite that does this in the SQLite binary itself in the Unix OS file?

[phiresky]

I'm not sure what you mean? You can't do XHR requests from within webassembly, so that part has to be written in JS

[boramalper]

What @chadbrewbaker is asking, as I understand, is to develop a VFS implementation in C to let the original SQLite implementation to read databases on remote static hosts.

[phiresky]

Sure, that would make sense - but it would probably have completely separate code from this one since it would have to be written with libcurl or something instead of the browser APIs. So I don't think it would work as a PR, it would be a separate project.

[riyaz-ali]

Apologise for reviving an old issue, but I played around with implementing this idea as a native sqlite virtual file-system. You can checkout the result at - https://github.com/riyaz-ali/sqlite3.js

[phiresky]

Neat! It would probably be useful if you provided a more detailed comparison to the alternatives / similar projects that have popped up by now. Specifically wa-sqlite since that provides a generic wrapper of the SQLite VFS API from JS and implements multiple different VFSes using that - which is pretty good for flexibility since you don't have to touch the C code. You also loose the ability to use asyncify with removing the emscripten build system I guess which makes many kinds of VFSes impossible without requiring SharedArrayBuffer.

It would also be neat to have more detailed information about e.g. the bundle size and the tradeoffs. You mention it being leaner due to not needing the POSIX api but does that really matter? It's great to get rid of as much of emscripten as possible, but more specific numbers would be interesting to justify it. The VFS API has pretty much the same structure as POSIX anyways, so I doubt there's much overhead in there. You also don't mention anything about caching, which is pretty important since the network fetching dwarfs basically everything if you use the http vfs.

I also don't think a "normal" emscripten build of sqlite has any "pseudo-network stack" or anything with threading but I'm not sure there.

Lastly, I encourage you to post in this thread: sql-js/sql.js#447 which has 3-5 people that have all worked on very similar things.

[riyaz-ali]

Sorry my bad for making big claims without providing any significant metrics to back it 😓 I'd definitely like to follow up on it. I'll post my findings (maybe in a separate thread) later. The "pseudo-network stack" (and the threading module) I was referring to was Emscripten's shims for making tcp request and threading. Again, I'll try to gather up more details about it and collect it in one place.

Regarding caching, I haven't spend much time around it, but I plan to! Ideally, I'd like to build a layer of caching similar to one in lazyFile.ts

And thanks for the link to the thread 🙌 I'll go through it 👍

[phiresky]

No worries. Looking forward to your post!