Add createFile to JS API for wasmfs (#23667) #23668
Conversation
|
I've read "date" as "update", hope that's what you meant! |
test/fs/test_fs_js_api.c
Outdated
| @@ -469,6 +469,17 @@ void cleanup() { | |||
| remove("closetestfile"); | |||
| } | |||
|
|
|||
| #if WASMFS | |||
There was a problem hiding this comment.
This API is not specific to WASMFS is it? I see it exists in libfs.js too.
There was a problem hiding this comment.
Oh indeed it does, I had no idea. But this version of the test is wasm-specific since it uses a backend. It also looks like the libfs version ignores the "properties" field and has a different interface.
Should a different name than createFile be picked for the wasmfs version? createFileInBackend?
There was a problem hiding this comment.
I think the goal with the JS API is mostly feature parity with the existing/old FS. If possible I thin we should avoid extending the JS API.
But yes, if we decide we really do need a new API we should give it a new name.
Would you make making a version of createFile that is compatible with the old FS and testing that as part of this PR? Or as a separate PR?
There was a problem hiding this comment.
I've made WASMFS createFile compatible with the JS version. The JS version never used the third parameter, so I'm using it for the backend here. I also made it so the test (though a bit different) runs on both filesystems.
test/fs/test_fs_js_api.c
Outdated
| @@ -1,5 +1,5 @@ | |||
| /* | |||
| * Copyright 2023 The Emscripten Authors. All rights reserved. | |||
| * Copyright 2025 The Emscripten Authors. All rights reserved. | |||
There was a problem hiding this comment.
We don't update out copyright dates when we modify files
Although the unused "properties" argument now is used as the "backend" for the file. If null, wasmfs will figure out the right backend to use based on the path.
| ); | ||
| #else | ||
| EM_ASM( | ||
| var file = FS.createFile("/", "test.txt", {}, true, true); |
There was a problem hiding this comment.
Can we make sure that this simple usage (without a backend) is works with both the old FS and with wasmfs.
In fact, it might be best not to expose the backend directly here, since then the two APIs would not be compatible.
In fact I think the for in libfs.js which has createFile call through to FS.create should probably work fine under wasmfs too, so maybe just copy it verbatim and at a test for it.
Do we really need a new API for creating a backend-specific file?
There was a problem hiding this comment.
The fact that no other FS.xxx APIs for wasmfs take the backend as an argument suggest that perhaps this is not needed in this case either.
There was a problem hiding this comment.
It can’t be copied verbatim mainly because the JS version handles either a path or a node for the parent, but the wasm version doesn’t have the concept of a node. AFAIK there’s no way to get the path to a file descriptor/wasmfs file.
The reason the FS.apis in wasmfs don’t take a backend is presumably because they’re meant to mimic JS FS APIs which only support a 1:1 relationship between backends and mount points. But wasmfs backends aren’t necessarily mounted somewhere. Fetchfs, for example, has supported this on the C side (a fetchfs whose url is a specific file, meant to be used to create a file). The tests for the JS wasmfs backend show a similar usage. The wasmfs C API seems to use the metaphor of creating files and directories within specific backends, but we don’t have a way to express that in JS yet.
As for compatibility, in this case the third argument seems to have no meaning in the JS version (it’s unused), so it shouldn’t be a compatibility issue. But a different name can be picked if this is a problem.
There was a problem hiding this comment.
I think we have two different issue here perhaps.
The first issue is that WASMFS lacks the FS.createFile API that the old FS has. Step 1 would be to add this API and add tests for it. (it should work in both WASM FS and the old FS and it should operate on the root mountpoint just like the all the other APIs).
If you want to then extend the JS API to include backend-specific APIs that is anther question, perhaps one that @kripken and @tlively could weigh in on. Perhaps you could explain why there is a need to be able to manipulate backends that are not mounted anywhere in the filesytem?
There was a problem hiding this comment.
Maybe the tension here is that the only way I can see for JS code to use WASMFS is through the WASMFS port of the JS FS API, and I'm trying to extend that. If there were a "native" wasmfs API in JS, which exposed the same API as emscripten/wasmfs.h (with conveniences around strings for example) then JS code could use the WASMFS API in exactly the way C code can (wasmfs backends aren't necessarily mounted anywhere, they just get to create directories or create files on the backend at different paths on the filesystem).
I know there's a TODO on wasmfs_create_file saying that in the future only directories should be mounted, but that's inconsistent with the existing tests (test/wasmfs/wasmfs_jsfile.c, test/wasmfs/wasmfs_fetch.c) and it's convenient to be able to work with a backend that semantically is a single file. For now, that means I could achieve what I had talked about earlier with that "mapfs" or manifest idea purely on the JS side, by creating and mounting single-file backends where I need them.
There was a problem hiding this comment.
Yeah. I want to use the fetch backend to fetch files from many distinct paths on a web server into one folder (with specific file names) in the filesystem. I had proposed a manifest parameter for fetchfs to support this kind of thing:
https://server/path1/file1.txt => dir/fileA
https://server/path2/file2.txt => dir/fileB
If it’s hard to implement or slow to land a mapping backend, I guess I could change my server to use a common root url (these are DB records grouped as a result of a query so this is nontrivial), but currently I solve it by making a (small) number of single file fetchfs backends and use create_file for each.
There was a problem hiding this comment.
In the above example could we not have have fetchfs mounted at dir with two files withing it?
Is that idea that each file gets its own backend/mount point?
If so it doesn't seems like maybe better design would be to allow a single fetch backend to have any number of files under a common root (dir in the example above). Maybe that right-hand-side entries in the manifest could be relative to where the backend is mounted rather than relative to where the actual root FS.
There was a problem hiding this comment.
fetchfs doesn’t include manifest support right now (I pulled it out in favor of a hypothetical future mapfs that maps one path to another path); it’s true that the manifest would actually be /fileA, /fileB.
As it is, because fetchfs uses the path relative to the mount point as the URL relative to the baseURL, if I didn’t have create_file I would either need to be ok with a filesystem like dir/path1/file1.txt, dir/path2/file2.txt (using a baseURL of /), change my server (nontrivial due to the rationale above), or implement mapfs to map a path like fileA->/fetch/path1/file1 and fileB->/fetch/path2/file2 and mount my fetchfs backend at /fetch.
I do actually like the mapfs idea, but it’s a bigger change and I’m not sure when I’ll have bandwidth to do it. Even if you don’t want to commit to including createFile in the JS api, leaving it in the C api for a few versions would give me time to try and build and land a PR for mapfs.
As a separate question, is it possible to write out-of-tree wasmfs backend implementations? What would compiling them and linking them into a build look like since they’d potentially provide both C and JS symbols?
There was a problem hiding this comment.
I got a little obsessed with this problem so I experimented and ended up with #23808, though I haven't fully explored it for my use case yet I think it should probably work.
There was a problem hiding this comment.
It also occurs to me that wasmfs fetch backend as far back as I’m aware requires users to create individual files before reading them. I don’t know how to (as in, I haven’t looked into) make the backend support creating files on read/stat (which feels like a better user experience anyway).
Per #23667, add a createFile function to the WASMFS JS API that calls out to
wasmfs_create_file.