Skip to content

Commit 04bb61d

Browse files
authored
feat: add custom backend parameter (#63)
1 parent 40bf0bf commit 04bb61d

File tree

4 files changed

+314
-196
lines changed

4 files changed

+314
-196
lines changed

README.md

+73-10
Original file line numberDiff line numberDiff line change
@@ -59,16 +59,17 @@ const fs = new FS("testfs")
5959

6060
Options object:
6161

62-
| Param | Type [= default] | Description |
63-
| --------------- | ------------------ | --------------------------------------------------------------------- |
64-
| `wipe` | boolean = false | Delete the database and start with an empty filesystem |
65-
| `url` | string = undefined | Let `readFile` requests fall back to an HTTP request to this base URL |
66-
| `urlauto` | boolean = false | Fall back to HTTP for every read of a missing file, even if unbacked |
67-
| `fileDbName` | string | Customize the database name |
68-
| `fileStoreName` | string | Customize the store name |
69-
| `lockDbName` | string | Customize the database name for the lock mutex |
70-
| `lockStoreName` | string | Customize the store name for the lock mutex |
71-
| `defer` | boolean = false | If true, avoids mutex contention during initialization |
62+
| Param | Type [= default] | Description |
63+
| --------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
64+
| `wipe` | boolean = false | Delete the database and start with an empty filesystem |
65+
| `url` | string = undefined | Let `readFile` requests fall back to an HTTP request to this base URL |
66+
| `urlauto` | boolean = false | Fall back to HTTP for every read of a missing file, even if unbacked |
67+
| `fileDbName` | string | Customize the database name |
68+
| `fileStoreName` | string | Customize the store name |
69+
| `lockDbName` | string | Customize the database name for the lock mutex |
70+
| `lockStoreName` | string | Customize the store name for the lock mutex |
71+
| `defer` | boolean = false | If true, avoids mutex contention during initialization |
72+
| `backend` | IBackend | If present, none of the other arguments (except `defer`) have any effect, and instead of using the normal LightningFS stuff, LightningFS acts as a wrapper around the provided custom backend. |
7273

7374
#### Advanced usage
7475

@@ -191,6 +192,68 @@ Returns the size of a file or directory in bytes.
191192

192193
All the same functions as above, but instead of passing a callback they return a promise.
193194

195+
## Providing a custom `backend` (advanced usage)
196+
197+
There are only two reasons I can think of that you would want to do this:
198+
199+
1. The `fs` module is normally a singleton. LightningFS allows you to safely(ish) hotswap between various data sources by calling `init` multiple times with different options. (It keeps track of file system operations in flight and waits until there's an idle moment to do the switch.)
200+
201+
2. LightningFS normalizes all the lovely variations of node's `fs` arguments:
202+
203+
- `fs.writeFile('filename.txt', 'Hello', cb)`
204+
- `fs.writeFile('filename.txt', 'Hello', 'utf8', cb)`
205+
- `fs.writeFile('filename.txt', 'Hello', { encoding: 'utf8' }, cb)`
206+
- `fs.promises.writeFile('filename.txt', 'Hello')`
207+
- `fs.promises.writeFile('filename.txt', 'Hello', 'utf8')`
208+
- `fs.promises.writeFile('filename.txt', 'Hello', { encoding: 'utf8' })`
209+
210+
And it normalizes filepaths. And will convert plain `StatLike` objects into `Stat` objects with methods like `isFile`, `isDirectory`, etc.
211+
212+
If that fits your needs, then you can provide a `backend` option and LightningFS will use that. Implement as few/many methods as you need for your application to work.
213+
214+
**Note:** If you use a custom backend, you are responsible for managing multi-threaded access - there are no magic mutexes included by default.
215+
216+
Note: throwing an error with the correct `.code` property for any given situation is often important for utilities like `mkdirp` and `rimraf` to work.
217+
218+
```tsx
219+
220+
type EncodingOpts = {
221+
encoding?: 'utf8';
222+
}
223+
224+
type StatLike = {
225+
type: 'file' | 'dir' | 'symlink';
226+
mode: number;
227+
size: number;
228+
ino: number | string | BigInt;
229+
mtimeMs: number;
230+
ctimeMs?: number;
231+
}
232+
233+
interface IBackend {
234+
// highly recommended - usually necessary for apps to work
235+
readFile(filepath: string, opts: EncodingOpts): Awaited<Uint8Array | string>; // throws ENOENT
236+
writeFile(filepath: string, data: Uint8Array | string, opts: EncodingOpts): void; // throws ENOENT
237+
unlink(filepath: string, opts: any): void; // throws ENOENT
238+
readdir(filepath: string, opts: any): Awaited<string[]>; // throws ENOENT, ENOTDIR
239+
mkdir(filepath: string, opts: any): void; // throws ENOENT, EEXIST
240+
rmdir(filepath: string, opts: any): void; // throws ENOENT, ENOTDIR, ENOTEMPTY
241+
242+
// recommended - often necessary for apps to work
243+
stat(filepath: string, opts: any): Awaited<StatLike>; // throws ENOENT
244+
lstat(filepath: string, opts: any): Awaited<StatLike>; // throws ENOENT
245+
246+
// suggested - used occasionally by apps
247+
rename(oldFilepath: string, newFilepath: string): void; // throws ENOENT
248+
readlink(filepath: string, opts: any): Awaited<string>; // throws ENOENT
249+
symlink(target: string, filepath: string): void; // throws ENOENT
250+
251+
// bonus - not part of the standard `fs` module
252+
backFile(filepath: string, opts: any): void;
253+
du(filepath: string): Awaited<number>;
254+
}
255+
```
256+
194257
## License
195258

196259
MIT

src/DefaultBackend.js

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
const { encode, decode } = require("isomorphic-textencoder");
2+
const debounce = require("just-debounce-it");
3+
4+
const CacheFS = require("./CacheFS.js");
5+
const { ENOENT, ENOTEMPTY, ETIMEDOUT } = require("./errors.js");
6+
const IdbBackend = require("./IdbBackend.js");
7+
const HttpBackend = require("./HttpBackend.js")
8+
const Mutex = require("./Mutex.js");
9+
const Mutex2 = require("./Mutex2.js");
10+
11+
const path = require("./path.js");
12+
13+
module.exports = class DefaultBackend {
14+
constructor() {
15+
this.saveSuperblock = debounce(() => {
16+
this._saveSuperblock();
17+
}, 500);
18+
}
19+
async init (name, {
20+
wipe,
21+
url,
22+
urlauto,
23+
fileDbName = name,
24+
fileStoreName = name + "_files",
25+
lockDbName = name + "_lock",
26+
lockStoreName = name + "_lock",
27+
} = {}) {
28+
this._name = name
29+
this._idb = new IdbBackend(fileDbName, fileStoreName);
30+
this._mutex = navigator.locks ? new Mutex2(name) : new Mutex(lockDbName, lockStoreName);
31+
this._cache = new CacheFS(name);
32+
this._opts = { wipe, url };
33+
this._needsWipe = !!wipe;
34+
if (url) {
35+
this._http = new HttpBackend(url)
36+
this._urlauto = !!urlauto
37+
}
38+
}
39+
async activate() {
40+
if (this._cache.activated) return
41+
// Wipe IDB if requested
42+
if (this._needsWipe) {
43+
this._needsWipe = false;
44+
await this._idb.wipe()
45+
await this._mutex.release({ force: true })
46+
}
47+
if (!(await this._mutex.has())) await this._mutex.wait()
48+
// Attempt to load FS from IDB backend
49+
const root = await this._idb.loadSuperblock()
50+
if (root) {
51+
this._cache.activate(root);
52+
} else if (this._http) {
53+
// If that failed, attempt to load FS from HTTP backend
54+
const text = await this._http.loadSuperblock()
55+
this._cache.activate(text)
56+
await this._saveSuperblock();
57+
} else {
58+
// If there is no HTTP backend, start with an empty filesystem
59+
this._cache.activate()
60+
}
61+
if (await this._mutex.has()) {
62+
return
63+
} else {
64+
throw new ETIMEDOUT()
65+
}
66+
}
67+
async deactivate() {
68+
if (await this._mutex.has()) {
69+
await this._saveSuperblock()
70+
}
71+
this._cache.deactivate()
72+
try {
73+
await this._mutex.release()
74+
} catch (e) {
75+
console.log(e)
76+
}
77+
await this._idb.close()
78+
}
79+
async _saveSuperblock() {
80+
if (this._cache.activated) {
81+
this._lastSavedAt = Date.now()
82+
await this._idb.saveSuperblock(this._cache._root);
83+
}
84+
}
85+
_writeStat(filepath, size, opts) {
86+
let dirparts = path.split(path.dirname(filepath))
87+
let dir = dirparts.shift()
88+
for (let dirpart of dirparts) {
89+
dir = path.join(dir, dirpart)
90+
try {
91+
this._cache.mkdir(dir, { mode: 0o777 })
92+
} catch (e) {}
93+
}
94+
return this._cache.writeStat(filepath, size, opts)
95+
}
96+
async readFile(filepath, opts) {
97+
const { encoding } = opts;
98+
if (encoding && encoding !== 'utf8') throw new Error('Only "utf8" encoding is supported in readFile');
99+
let data = null, stat = null
100+
try {
101+
stat = this._cache.stat(filepath);
102+
data = await this._idb.readFile(stat.ino)
103+
} catch (e) {
104+
if (!this._urlauto) throw e
105+
}
106+
if (!data && this._http) {
107+
let lstat = this._cache.lstat(filepath)
108+
while (lstat.type === 'symlink') {
109+
filepath = path.resolve(path.dirname(filepath), lstat.target)
110+
lstat = this._cache.lstat(filepath)
111+
}
112+
data = await this._http.readFile(filepath)
113+
}
114+
if (data) {
115+
if (!stat || stat.size != data.byteLength) {
116+
stat = await this._writeStat(filepath, data.byteLength, { mode: stat ? stat.mode : 0o666 })
117+
this.saveSuperblock() // debounced
118+
}
119+
if (encoding === "utf8") {
120+
data = decode(data);
121+
}
122+
}
123+
if (!stat) throw new ENOENT(filepath)
124+
return data;
125+
}
126+
async writeFile(filepath, data, opts) {
127+
const { mode, encoding = "utf8" } = opts;
128+
if (typeof data === "string") {
129+
if (encoding !== "utf8") {
130+
throw new Error('Only "utf8" encoding is supported in writeFile');
131+
}
132+
data = encode(data);
133+
}
134+
const stat = await this._cache.writeStat(filepath, data.byteLength, { mode });
135+
await this._idb.writeFile(stat.ino, data)
136+
}
137+
async unlink(filepath, opts) {
138+
const stat = this._cache.lstat(filepath);
139+
this._cache.unlink(filepath);
140+
if (stat.type !== 'symlink') {
141+
await this._idb.unlink(stat.ino)
142+
}
143+
}
144+
readdir(filepath, opts) {
145+
return this._cache.readdir(filepath);
146+
}
147+
mkdir(filepath, opts) {
148+
const { mode = 0o777 } = opts;
149+
this._cache.mkdir(filepath, { mode });
150+
}
151+
rmdir(filepath, opts) {
152+
// Never allow deleting the root directory.
153+
if (filepath === "/") {
154+
throw new ENOTEMPTY();
155+
}
156+
this._cache.rmdir(filepath);
157+
}
158+
rename(oldFilepath, newFilepath) {
159+
this._cache.rename(oldFilepath, newFilepath);
160+
}
161+
stat(filepath, opts) {
162+
return this._cache.stat(filepath);
163+
}
164+
lstat(filepath, opts) {
165+
return this._cache.lstat(filepath);
166+
}
167+
readlink(filepath, opts) {
168+
return this._cache.readlink(filepath);
169+
}
170+
symlink(target, filepath) {
171+
this._cache.symlink(target, filepath);
172+
}
173+
async backFile(filepath, opts) {
174+
let size = await this._http.sizeFile(filepath)
175+
await this._writeStat(filepath, size, opts)
176+
}
177+
du(filepath) {
178+
return this._cache.du(filepath);
179+
}
180+
}

0 commit comments

Comments
 (0)