Skip to content

Commit 3426b43

Browse files
anthony-murphyjzaffiroJosmithr
authored
Enable Synchronous Child Datastore Creation (#23143)
## Overview This feature introduces a new pattern for creating datastores synchronously within the Fluid Framework. It allows for the synchronous creation of a child datastore from an existing datastore, provided that the child datastore is available synchronously via the existing datastore's registry and that the child's factory supports synchronous creation. This method also ensures strong typing for the consumer. In this context, "child" refers specifically to the organization of factories and registries, not to any hierarchical or hosting relationship between datastores. The parent datastore does not control the runtime behaviors of the child datastore beyond its creation. The synchronous creation of child datastores enhances the flexibility of datastore management within the Fluid Framework. It ensures type safety and provides a different way to manage datastores within a container. However, it is important to consider the overhead associated with datastores, as they are stored, summarized, garbage collected, loaded, and referenced independently. This overhead should be justified by the scenario's requirements. Datastores offer increased capabilities, such as the ability to reference them via handles, allowing multiple references to exist and enabling those references to be moved, swapped, or changed. Additionally, datastores are garbage collected after becoming unreferenced, which can simplify final cleanup across clients. This is in contrast to subdirectories in a shared directory, which do not have native capabilities for referencing or garbage collection but are very low overhead to create. Synchronous creation relies on both the factory and the datastore to support it. This means that asynchronous operations, such as resolving handles, some browser API calls, consensus-based operations, or other asynchronous tasks, cannot be performed during the creation flow. Therefore, synchronous child datastore creation is best limited to scenarios where the existing asynchronous process cannot be used, such as when a new datastore must be created in direct response to synchronous user input. ## Key Benefits - **Synchronous Creation**: Allows for the immediate creation of child datastores without waiting for asynchronous operations. - **Strong Typing**: Ensures type safety and better developer experience by leveraging TypeScript's type system. ## Use Cases ### Example 1: Creating a Child Datastore In this example, we demonstrate how to support creating a child datastore synchronously from a parent datastore. ```typescript /** * This is the parent DataObject, which is also a datastore. It has a * synchronous method to create child datastores, which could be called * in response to synchronous user input, like a key press. */ class ParentDataObject extends DataObject { get ParentDataObject() { return this; } protected override async initializingFirstTime(): Promise<void> { // create synchronously during initialization this.createChild("parentCreation"); } createChild(name: string): ChildDataStore { assert( this.context.createChildDataStore !== undefined, "this.context.createChildDataStore", ); // creates a detached context with a factory who's package path is the same // as the current datastore, but with another copy of its own type. const { entrypoint } = this.context.createChildDataStore( ChildDataStoreFactory.instance, ); const dir = this.root.createSubDirectory("children"); dir.set(name, entrypoint.handle); entrypoint.setProperty("childValue", name); return entrypoint; } getChild(name: string): IFluidHandle<ChildDataStore> | undefined { const dir = this.root.getSubDirectory("children"); return dir?.get<IFluidHandle<ChildDataStore>>(name); } } ``` For a complete example see the follow test: https://github.com/microsoft/FluidFramework/blob/main/packages/test/local-server-tests/src/test/synchronousDataStoreCreation.spec.ts --------- Co-authored-by: jzaffiro <[email protected]> Co-authored-by: Joshua Smithrud <[email protected]>
1 parent c11bc16 commit 3426b43

File tree

10 files changed

+555
-99
lines changed

10 files changed

+555
-99
lines changed

.changeset/solid-keys-agree.md

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
"@fluidframework/container-runtime": minor
3+
"@fluidframework/runtime-definitions": minor
4+
---
5+
---
6+
"section": feature
7+
---
8+
9+
Enable Synchronous Child Datastore Creation
10+
11+
## Overview
12+
13+
This feature introduces a new pattern for creating datastores synchronously within the Fluid Framework. It allows for the synchronous creation of a child datastore from an existing datastore, provided that the child datastore is available synchronously via the existing datastore's registry and that the child's factory supports synchronous creation. This method also ensures strong typing for the consumer.
14+
15+
In this context, "child" refers specifically to the organization of factories and registries, not to any hierarchical or hosting relationship between datastores. The parent datastore does not control the runtime behaviors of the child datastore beyond its creation.
16+
17+
The synchronous creation of child datastores enhances the flexibility of datastore management within the Fluid Framework. It ensures type safety and provides a different way to manage datastores within a container. However, it is important to consider the overhead associated with datastores, as they are stored, summarized, garbage collected, loaded, and referenced independently. This overhead should be justified by the scenario's requirements.
18+
19+
Datastores offer increased capabilities, such as the ability to reference them via handles, allowing multiple references to exist and enabling those references to be moved, swapped, or changed. Additionally, datastores are garbage collected after becoming unreferenced, which can simplify final cleanup across clients. This is in contrast to subdirectories in a shared directory, which do not have native capabilities for referencing or garbage collection but are very low overhead to create.
20+
21+
Synchronous creation relies on both the factory and the datastore to support it. This means that asynchronous operations, such as resolving handles, some browser API calls, consensus-based operations, or other asynchronous tasks, cannot be performed during the creation flow. Therefore, synchronous child datastore creation is best limited to scenarios where the existing asynchronous process cannot be used, such as when a new datastore must be created in direct response to synchronous user input.
22+
23+
## Key Benefits
24+
25+
- **Synchronous Creation**: Allows for the immediate creation of child datastores without waiting for asynchronous operations.
26+
- **Strong Typing**: Ensures type safety and better developer experience by leveraging TypeScript's type system.
27+
28+
## Use Cases
29+
30+
### Example 1: Creating a Child Datastore
31+
32+
In this example, we demonstrate how to support creating a child datastore synchronously from a parent datastore.
33+
34+
```typescript
35+
/**
36+
* This is the parent DataObject, which is also a datastore. It has a
37+
* synchronous method to create child datastores, which could be called
38+
* in response to synchronous user input, like a key press.
39+
*/
40+
class ParentDataObject extends DataObject {
41+
createChild(name: string): ChildDataStore {
42+
assert(
43+
this.context.createChildDataStore !== undefined,
44+
"this.context.createChildDataStore",
45+
);
46+
47+
const { entrypoint } = this.context.createChildDataStore(
48+
ChildDataStoreFactory.instance,
49+
);
50+
const dir = this.root.createSubDirectory("children");
51+
dir.set(name, entrypoint.handle);
52+
entrypoint.setProperty("childValue", name);
53+
54+
return entrypoint;
55+
}
56+
57+
getChild(name: string): IFluidHandle<ChildDataStore> | undefined {
58+
const dir = this.root.getSubDirectory("children");
59+
return dir?.get<IFluidHandle<ChildDataStore>>(name);
60+
}
61+
}
62+
```
63+
64+
For a complete example see the following test:
65+
https://github.com/microsoft/FluidFramework/blob/main/packages/test/local-server-tests/src/test/synchronousDataStoreCreation.spec.ts

packages/runtime/container-runtime/src/dataStoreContext.ts

+57-1
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ import {
5555
IInboundSignalMessage,
5656
type IPendingMessagesState,
5757
type IRuntimeMessageCollection,
58+
type IFluidDataStoreFactory,
5859
} from "@fluidframework/runtime-definitions/internal";
5960
import {
6061
addBlobToSummary,
@@ -65,6 +66,7 @@ import {
6566
LoggingError,
6667
MonitoringContext,
6768
ThresholdCounter,
69+
UsageError,
6870
createChildMonitoringContext,
6971
extractSafePropertiesFromMessage,
7072
generateStack,
@@ -496,7 +498,7 @@ export abstract class FluidDataStoreContext
496498
this.rejectDeferredRealize("No registry for package", lastPkg, packages);
497499
}
498500
lastPkg = pkg;
499-
entry = await registry.get(pkg);
501+
entry = registry.getSync?.(pkg) ?? (await registry.get(pkg));
500502
if (!entry) {
501503
this.rejectDeferredRealize(
502504
"Registry does not contain entry for the package",
@@ -517,6 +519,42 @@ export abstract class FluidDataStoreContext
517519
return factory;
518520
}
519521

522+
createChildDataStore<T extends IFluidDataStoreFactory>(
523+
childFactory: T,
524+
): ReturnType<Exclude<T["createDataStore"], undefined>> {
525+
const maybe = this.registry?.getSync?.(childFactory.type);
526+
527+
const isUndefined = maybe === undefined;
528+
const diffInstance = maybe?.IFluidDataStoreFactory !== childFactory;
529+
530+
if (isUndefined || diffInstance) {
531+
throw new UsageError(
532+
"The provided factory instance must be synchronously available as a child of this datastore",
533+
{ isUndefined, diffInstance },
534+
);
535+
}
536+
if (childFactory?.createDataStore === undefined) {
537+
throw new UsageError("createDataStore must exist on the provided factory", {
538+
noCreateDataStore: true,
539+
});
540+
}
541+
542+
const context = this._containerRuntime.createDetachedDataStore([
543+
...this.packagePath,
544+
childFactory.type,
545+
]);
546+
assert(
547+
context instanceof LocalDetachedFluidDataStoreContext,
548+
"must be a LocalDetachedFluidDataStoreContext",
549+
);
550+
551+
const created = childFactory.createDataStore(context) as ReturnType<
552+
Exclude<T["createDataStore"], undefined>
553+
>;
554+
context.unsafe_AttachRuntimeSync(created.runtime);
555+
return created;
556+
}
557+
520558
private async realizeCore(existing: boolean) {
521559
const details = await this.getInitialSnapshotDetails();
522560
// Base snapshot is the baseline where pending ops are applied to.
@@ -1428,6 +1466,24 @@ export class LocalDetachedFluidDataStoreContext
14281466
return this.channelToDataStoreFn(await this.channelP);
14291467
}
14301468

1469+
/**
1470+
* This method provides a synchronous path for binding a runtime to the context.
1471+
*
1472+
* Due to its synchronous nature, it is unable to validate that the runtime
1473+
* represents a datastore which is instantiable by remote clients. This could
1474+
* happen if the runtime's package path does not return a factory when looked up
1475+
* in the container runtime's registry, or if the runtime's entrypoint is not
1476+
* properly initialized. As both of these validation's are asynchronous to preform.
1477+
*
1478+
* If used incorrectly, this function can result in permanent data corruption.
1479+
*/
1480+
public unsafe_AttachRuntimeSync(channel: IFluidDataStoreChannel) {
1481+
this.channelP = Promise.resolve(channel);
1482+
this.processPendingOps(channel);
1483+
this.completeBindingRuntime(channel);
1484+
return this.channelToDataStoreFn(channel);
1485+
}
1486+
14311487
public async getInitialSnapshotDetails(): Promise<ISnapshotDetails> {
14321488
if (this.detachedRuntimeCreation) {
14331489
throw new Error(

packages/runtime/container-runtime/src/dataStoreRegistry.ts

+10
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
* Licensed under the MIT License.
44
*/
55

6+
import { isPromiseLike } from "@fluidframework/core-utils/internal";
67
import {
78
FluidDataStoreRegistryEntry,
89
IFluidDataStoreRegistry,
@@ -40,4 +41,13 @@ export class FluidDataStoreRegistry implements IFluidDataStoreRegistry {
4041

4142
return undefined;
4243
}
44+
45+
public getSync(name: string): FluidDataStoreRegistryEntry | undefined {
46+
const entry = this.map.get(name);
47+
if (!isPromiseLike(entry)) {
48+
return entry;
49+
}
50+
51+
return undefined;
52+
}
4353
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
/*!
2+
* Copyright (c) Microsoft Corporation and contributors. All rights reserved.
3+
* Licensed under the MIT License.
4+
*/
5+
6+
import { strict as assert } from "assert";
7+
8+
import { FluidErrorTypes } from "@fluidframework/core-interfaces/internal";
9+
import { isPromiseLike, LazyPromise } from "@fluidframework/core-utils/internal";
10+
import { IDocumentStorageService } from "@fluidframework/driver-definitions/internal";
11+
import {
12+
IFluidDataStoreChannel,
13+
IFluidDataStoreFactory,
14+
IFluidDataStoreRegistry,
15+
IFluidParentContext,
16+
type NamedFluidDataStoreRegistryEntries,
17+
type IContainerRuntimeBase,
18+
type ISummarizerNodeWithGC,
19+
} from "@fluidframework/runtime-definitions/internal";
20+
import { isFluidError } from "@fluidframework/telemetry-utils/internal";
21+
import { MockFluidDataStoreRuntime } from "@fluidframework/test-runtime-utils/internal";
22+
23+
import {
24+
FluidDataStoreContext,
25+
LocalDetachedFluidDataStoreContext,
26+
} from "../dataStoreContext.js";
27+
28+
describe("createChildDataStore", () => {
29+
const throwNYI = () => {
30+
throw new Error("Method not implemented.");
31+
};
32+
const testContext = class TestContext extends FluidDataStoreContext {
33+
protected pkg = ["ParentDataStore"];
34+
public registry: IFluidDataStoreRegistry | undefined;
35+
public getInitialSnapshotDetails = throwNYI;
36+
public setAttachState = throwNYI;
37+
public getAttachSummary = throwNYI;
38+
public getAttachGCData = throwNYI;
39+
protected channel = new Proxy({} as any as IFluidDataStoreChannel, { get: throwNYI });
40+
protected channelP = new LazyPromise(async () => this.channel);
41+
};
42+
43+
const createRegistry = (
44+
namedEntries?: NamedFluidDataStoreRegistryEntries,
45+
): IFluidDataStoreRegistry => ({
46+
get IFluidDataStoreRegistry() {
47+
return this;
48+
},
49+
async get(name) {
50+
return new Map(namedEntries).get(name);
51+
},
52+
getSync(name) {
53+
const entry = new Map(namedEntries).get(name);
54+
return isPromiseLike(entry) ? undefined : entry;
55+
},
56+
});
57+
58+
const createContext = (namedEntries?: NamedFluidDataStoreRegistryEntries) => {
59+
const registry = createRegistry(namedEntries);
60+
const createSummarizerNodeFn = () =>
61+
new Proxy({} as any as ISummarizerNodeWithGC, { get: throwNYI });
62+
const storage = new Proxy({} as any as IDocumentStorageService, { get: throwNYI });
63+
64+
const parentContext = {
65+
clientDetails: {
66+
capabilities: { interactive: true },
67+
},
68+
containerRuntime: {
69+
createDetachedDataStore(pkg, loadingGroupId) {
70+
return new LocalDetachedFluidDataStoreContext({
71+
channelToDataStoreFn: (channel) => ({
72+
entryPoint: channel.entryPoint,
73+
trySetAlias: throwNYI,
74+
}),
75+
createSummarizerNodeFn,
76+
id: "child",
77+
makeLocallyVisibleFn: throwNYI,
78+
parentContext,
79+
pkg,
80+
scope: {},
81+
snapshotTree: undefined,
82+
storage,
83+
loadingGroupId,
84+
});
85+
},
86+
} satisfies Partial<IContainerRuntimeBase> as unknown as IContainerRuntimeBase,
87+
} satisfies Partial<IFluidParentContext> as unknown as IFluidParentContext;
88+
89+
const context = new testContext(
90+
{
91+
createSummarizerNodeFn,
92+
id: "parent",
93+
parentContext,
94+
scope: {},
95+
storage,
96+
},
97+
false,
98+
false,
99+
throwNYI,
100+
);
101+
context.registry = registry;
102+
return context;
103+
};
104+
105+
const createFactory = (
106+
createDataStore?: IFluidDataStoreFactory["createDataStore"],
107+
): IFluidDataStoreFactory => ({
108+
type: "ChildDataStore",
109+
get IFluidDataStoreFactory() {
110+
return this;
111+
},
112+
instantiateDataStore: throwNYI,
113+
createDataStore,
114+
});
115+
116+
it("Child factory does not support synchronous creation", async () => {
117+
const factory = createFactory();
118+
const context = createContext([[factory.type, factory]]);
119+
try {
120+
context.createChildDataStore(factory);
121+
assert.fail("should fail");
122+
} catch (e) {
123+
assert(isFluidError(e));
124+
assert(e.errorType === FluidErrorTypes.usageError);
125+
assert(e.getTelemetryProperties().noCreateDataStore === true);
126+
}
127+
});
128+
129+
it("Child factory not registered", async () => {
130+
const factory = createFactory();
131+
const context = createContext();
132+
try {
133+
context.createChildDataStore(factory);
134+
assert.fail("should fail");
135+
} catch (e) {
136+
assert(isFluidError(e));
137+
assert(e.errorType === FluidErrorTypes.usageError);
138+
assert(e.getTelemetryProperties().isUndefined === true);
139+
}
140+
});
141+
142+
it("Child factory is a different instance", async () => {
143+
const factory = createFactory();
144+
const context = createContext([[factory.type, createFactory()]]);
145+
146+
try {
147+
context.createChildDataStore(factory);
148+
assert.fail("should fail");
149+
} catch (e) {
150+
assert(isFluidError(e));
151+
assert(e.errorType === FluidErrorTypes.usageError);
152+
assert(e.getTelemetryProperties().diffInstance === true);
153+
}
154+
});
155+
156+
it("createChildDataStore", async () => {
157+
const factory = createFactory(() => ({ runtime: new MockFluidDataStoreRuntime() }));
158+
const context = createContext([[factory.type, factory]]);
159+
context.createChildDataStore(factory);
160+
});
161+
});

packages/runtime/runtime-definitions/api-report/runtime-definitions.legacy.alpha.api.md

+12-2
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,7 @@ export interface IFluidDataStoreChannel extends IDisposable {
150150
export interface IFluidDataStoreContext extends IFluidParentContext {
151151
// (undocumented)
152152
readonly baseSnapshot: ISnapshotTree | undefined;
153+
createChildDataStore?<T extends IFluidDataStoreFactory>(childFactory: T): ReturnType<Exclude<T["createDataStore"], undefined>>;
153154
// @deprecated (undocumented)
154155
readonly createProps?: any;
155156
// @deprecated (undocumented)
@@ -170,6 +171,9 @@ export const IFluidDataStoreFactory: keyof IProvideFluidDataStoreFactory;
170171

171172
// @alpha
172173
export interface IFluidDataStoreFactory extends IProvideFluidDataStoreFactory {
174+
createDataStore?(context: IFluidDataStoreContext): {
175+
readonly runtime: IFluidDataStoreChannel;
176+
};
173177
instantiateDataStore(context: IFluidDataStoreContext, existing: boolean): Promise<IFluidDataStoreChannel>;
174178
type: string;
175179
}
@@ -179,8 +183,8 @@ export const IFluidDataStoreRegistry: keyof IProvideFluidDataStoreRegistry;
179183

180184
// @alpha
181185
export interface IFluidDataStoreRegistry extends IProvideFluidDataStoreRegistry {
182-
// (undocumented)
183186
get(name: string): Promise<FluidDataStoreRegistryEntry | undefined>;
187+
getSync?(name: string): FluidDataStoreRegistryEntry | undefined;
184188
}
185189

186190
// @alpha
@@ -375,11 +379,17 @@ export interface LocalAttributionKey {
375379
}
376380

377381
// @alpha
378-
export type NamedFluidDataStoreRegistryEntries = Iterable<NamedFluidDataStoreRegistryEntry>;
382+
export type NamedFluidDataStoreRegistryEntries = Iterable<NamedFluidDataStoreRegistryEntry2>;
379383

380384
// @alpha
381385
export type NamedFluidDataStoreRegistryEntry = [string, Promise<FluidDataStoreRegistryEntry>];
382386

387+
// @alpha
388+
export type NamedFluidDataStoreRegistryEntry2 = [
389+
string,
390+
Promise<FluidDataStoreRegistryEntry> | FluidDataStoreRegistryEntry
391+
];
392+
383393
// @alpha
384394
export interface OpAttributionKey {
385395
seq: number;

0 commit comments

Comments
 (0)