Enabling testing of 2.0 file format features (#20109)

The issue we have in our tests (mostly end-to-end tests) is that we are not testing new capabilities that are either enabled or will be enabled by customers as part of 2.0 timeframe. Example of the features are:
1. Id Compressor
2. op compression
3. op chunking
4. op grouping

Successful solution here will have these properties:
1. We will enable all the features in our tests that are possible to enable, in order to get best coverage
- i.e. in tests that use only 2.0, all the recent features will be enabled.
3. At the same time, we will reduce usage of such features in tests to a level that the minimum runtime version understands.
- I.e. in tests that uses 1.3 directly, or mixes 1.3 and 2.0, we will disable all the features that 1.3 does not understand.

The obvious question to ask - are we losing too much of coverage for when these features are off?
I'd say no, as
- Any test combos that use a mix of 1.3 & 2.0 still test it
- When options are off, we are (ideally, not guaranteed) should get to well know state that was already tested (before such features were added). There are no guarantees here, but there are even less guarantees for these features to be working when they are On, as that's a lot of new code.

Please note that this question is orthogonal to a question - should each feature in question here be properly tested by dedicated UT with on & off states. The answer is - absolutely yes, but that's not enough.

Here are bugs found and already fixed thanks to this work (I also hit first two bugs independently while stress testing unrelated piece of code that used ID compressor):
1. #20089
2. #20080
3. #20111
4. #20115

As for results:
- Initial commit (for this PR) has comments describing how many tests are failing. That was 2 weeks back or so.
- Rerunning it today (with merging rebasing to latest main, which includes first 2 commits/PRs mentioned above) shows smaller number of failures. I expect that the PRs I've mentioned above (that I merged this week) are addressing some of the issues we are seeing here (but I have not validated that).
- I'll keep adding PRs to the list with bugfixes as I identify issues and fixes.

This is just a prototype.
I think the next steps are:
1. Investigate all the failures and fix all the production / test bugs that this direction finds.
   - Or possibly - learn that direction has a flaw.
2. Expose by container runtime package a default config & max config for testing
5. Test packages use above if available, and not hard code options in test code.
6. We need to fix AzureClient not to use Immediate Flush mode, and approach it in exactly same way as the rest of runtime options (in terms of testing / flighting / making it eventually a reality)

Author: vladsud

examples/data-objects/webflow/src/test/layout.spec.ts

@@ -45,7 +45,7 @@ describeCompat("Layout", "LoaderCompat", (getTestObjectProvider) => {
 
 	let provider: ITestObjectProvider;
 	before(async function () {
-		// Initialization of 'compat LTS ^1.3.4 - old loader' compat variant can take a couple seconds.
+		// Initialization of 'compat LTS ^1.3.x - old loader' compat variant can take a couple seconds.
 		this.timeout(10000);
 
 		provider = getTestObjectProvider({ resetAfterEach: false });

packages/runtime/container-runtime/src/containerRuntime.ts

@@ -416,7 +416,7 @@ export interface IContainerRuntimeOptions {
 	 * If specified, when in FlushMode.TurnBased, if the size of the ops between JS turns exceeds this value,
 	 * an error will be thrown and the container will close.
 	 *
-	 * If unspecified, the limit is 950 * 1024.
+	 * If unspecified, the limit is 700Kb.
 	 *
 	 * 'Infinity' will disable any limit.
 	 *

packages/test/test-end-to-end-tests/src/test/batchBreakRegression.spec.ts

@@ -18,7 +18,7 @@ import {
 	ISequencedDocumentMessage,
 	ISequencedDocumentSystemMessage,
 } from "@fluidframework/protocol-definitions";
-import { IContainerRuntimeOptions } from "@fluidframework/container-runtime";
+import { IContainerRuntimeOptions, CompressionAlgorithms } from "@fluidframework/container-runtime";
 import { FluidErrorTypes } from "@fluidframework/core-interfaces";
 import { wrapObjectAndOverride } from "../mocking.js";
 
@@ -149,6 +149,11 @@ describeCompat("Batching failures", "NoCompat", (getTestObjectProvider) => {
 
 			await runAndValidateBatch(provider, proxyDsf, this.timeout(), {
 				enableGroupedBatching,
+				chunkSizeInBytes: Number.POSITIVE_INFINITY, // disable
+				compressionOptions: {
+					minimumBatchSizeInBytes: Number.POSITIVE_INFINITY, // disable
+					compressionAlgorithm: CompressionAlgorithms.lz4,
+				},
 			});
 			assert.strictEqual(batchesSent, 1, "expected only a single batch to be sent");
 

packages/test/test-end-to-end-tests/src/test/detachedContainerTests.spec.ts

@@ -478,6 +478,9 @@ describeCompat("Detached Container", "FullCompat", (getTestObjectProvider, apis)
 			if (runtimeMessage === false) {
 				return;
 			}
+			if (message.type === ContainerMessageType.IdAllocation) {
+				return;
+			}
 			try {
 				assert.strictEqual(
 					message.type,

packages/test/test-end-to-end-tests/src/test/fewerBatches.spec.ts

@@ -58,6 +58,13 @@ describeCompat("Fewer batches", "NoCompat", (getTestObjectProvider, apis) => {
 		const configWithFeatureGates = {
 			...containerConfig,
 			loaderProps: { configProvider: configProvider(featureGates) },
+			// This test counts number of ops and observes them at the container level.
+			// It has certain assumptions about count and shape of those ops.
+			// Disable op chunking to make sure test have full control over op stream, and thus can rely on those assumptions.
+			runtimeOptions: {
+				chunkSizeInBytes: Number.POSITIVE_INFINITY, // disable
+				...containerConfig.runtimeOptions,
+			},
 		};
 
 		// Create a Container for the first client.
@@ -99,6 +106,7 @@ describeCompat("Fewer batches", "NoCompat", (getTestObjectProvider, apis) => {
 				...testContainerConfig,
 				runtimeOptions: {
 					flushMode: test.flushMode,
+					chunkSizeInBytes: Number.POSITIVE_INFINITY, // disable
 				},
 			});
 

packages/test/test-end-to-end-tests/src/test/idCompressor.spec.ts

@@ -144,6 +144,9 @@ describeCompat("Runtime IdCompressor", "NoCompat", (getTestObjectProvider, apis)
 		],
 		fluidDataObjectType: DataObjectFactoryType.Test,
 		loaderProps: {},
+		runtimeOptions: {
+			enableRuntimeIdCompressor: "off",
+		},
 	};
 
 	const containerConfigWithCompressor: ITestContainerConfig = {

packages/test/test-end-to-end-tests/src/test/stashedOps.spec.ts

@@ -44,6 +44,7 @@ import {
 	ContainerRuntime,
 	DefaultSummaryConfiguration,
 	type RecentlyAddedContainerRuntimeMessageDetails,
+	CompressionAlgorithms,
 } from "@fluidframework/container-runtime";
 import { IDocumentServiceFactory } from "@fluidframework/driver-definitions";
 import { wrapObjectAndOverride } from "../mocking.js";
@@ -89,6 +90,11 @@ describeCompat("stashed ops", "NoCompat", (getTestObjectProvider, apis) => {
 		fluidDataObjectType: DataObjectFactoryType.Test,
 		registry,
 		runtimeOptions: {
+			chunkSizeInBytes: Number.POSITIVE_INFINITY, // disable
+			compressionOptions: {
+				minimumBatchSizeInBytes: Number.POSITIVE_INFINITY,
+				compressionAlgorithm: CompressionAlgorithms.lz4,
+			},
 			summaryOptions: {
 				summaryConfigOverrides: {
 					...DefaultSummaryConfiguration,

packages/test/test-version-utils/src/compatConfig.ts

@@ -23,7 +23,7 @@ import {
 	codeVersion,
 	testBaseVersion,
 } from "./baseVersion.js";
-import { getRequestedVersion } from "./versionUtils.js";
+import { getRequestedVersion, resolveVersion } from "./versionUtils.js";
 
 /**
  * Represents a previous major release of a package based on the provided delta. For example, if the base version is 2.X and
@@ -70,7 +70,7 @@ const defaultCompatVersions = {
 	// N and N - 1
 	currentVersionDeltas: [0, -1],
 	// we are currently supporting 1.3.X long-term
-	ltsVersions: ["^1.3.4"],
+	ltsVersions: [resolveVersion("^1.3", false)],
 };
 
 // This indicates the number of versions above 2.0.0.internal.1.y.z that we want to support for back compat.

packages/test/test-version-utils/src/compatUtils.ts

@@ -3,6 +3,7 @@
  * Licensed under the MIT License.
  */
 
+import * as semver from "semver";
 import { FluidTestDriverConfig, createFluidTestDriver } from "@fluid-private/test-drivers";
 import {
 	FluidObject,
@@ -28,6 +29,12 @@ import {
 } from "@fluidframework/test-utils";
 import { TestDriverTypes } from "@fluidframework/test-driver-definitions";
 import { mixinAttributor } from "@fluid-experimental/attributor";
+import {
+	IContainerRuntimeOptions,
+	DefaultSummaryConfiguration,
+	CompressionAlgorithms,
+	ICompressionRuntimeOptions,
+} from "@fluidframework/container-runtime";
 import { pkgVersion } from "./packageVersion.js";
 import {
 	getLoaderApi,
@@ -42,6 +49,96 @@ import {
  */
 export const TestDataObjectType = "@fluid-example/test-dataStore";
 
+/**
+ * This function modifies container runtime options according to a version of runtime used.
+ * If a version of runtime does not support some options, they are removed.
+ * If a version runtime supports some options, such options are enabled to increase a chance of
+ * hitting feature set controlled by such options, and thus increase chances of finding product bugs.
+ *
+ * @param version - a version of container runtime to be used in test
+ * @param optionsArg - input runtime options (optional)
+ * @returns - runtime options that should be used with a given version of container runtime
+ * @internal
+ */
+function filterRuntimeOptionsForVersion(
+	version: string,
+	optionsArg: IContainerRuntimeOptions = {
+		summaryOptions: {
+			summaryConfigOverrides: {
+				...DefaultSummaryConfiguration,
+				...{
+					initialSummarizerDelayMs: 0,
+				},
+			},
+		},
+	},
+) {
+	let options = { ...optionsArg };
+
+	// No test fails with this option, it allows us to validate properly expectations and
+	// implementation of services
+	options.loadSequenceNumberVerification = "close";
+
+	const compressorDisabled: ICompressionRuntimeOptions = {
+		minimumBatchSizeInBytes: Number.POSITIVE_INFINITY,
+		compressionAlgorithm: CompressionAlgorithms.lz4,
+	};
+
+	// These is the "maximum" config.
+	const {
+		compressionOptions = {
+			minimumBatchSizeInBytes: 200,
+			compressionAlgorithm: CompressionAlgorithms.lz4,
+		},
+		enableGroupedBatching = true,
+		enableRuntimeIdCompressor = "on",
+		// chunkSizeInBytes = 200,
+	} = options;
+
+	if (version.startsWith("1.")) {
+		options = {
+			...options,
+			// None of these features are supported by 1.3
+			compressionOptions: undefined,
+			enableGroupedBatching: false,
+			enableRuntimeIdCompressor: undefined,
+			chunkSizeInBytes: Number.POSITIVE_INFINITY, // disabled
+		};
+	} else if (version.startsWith("2.0.0-rc.1.")) {
+		options = {
+			compressionOptions: compressorDisabled, // Can't use compression, need https://github.com/microsoft/FluidFramework/pull/20111 fix
+			enableGroupedBatching,
+			enableRuntimeIdCompressor: undefined, // it was boolean in RC1, switched to enum in RC2
+			chunkSizeInBytes: Number.POSITIVE_INFINITY, // disabled, need https://github.com/microsoft/FluidFramework/pull/20115 fix
+			...options,
+		};
+	} else if (version.startsWith("2.0.0-rc.2.")) {
+		options = {
+			compressionOptions: compressorDisabled, // Can't use compression, need https://github.com/microsoft/FluidFramework/pull/20111 fix
+			enableGroupedBatching,
+			// Can't track it down, but enabling Id Compressor for this config results in small number of t9s tests to timeout.
+			// This is very likely related to one of these bugfixes that missed that release:
+			// https://github.com/microsoft/FluidFramework/pull/20089
+			// https://github.com/microsoft/FluidFramework/pull/20080
+			enableRuntimeIdCompressor: undefined,
+			chunkSizeInBytes: Number.POSITIVE_INFINITY, // disabled, need https://github.com/microsoft/FluidFramework/pull/20115 fix
+			...options,
+		};
+	} else {
+		// "2.0.0-rc.3." ++
+		options = {
+			compressionOptions,
+			enableGroupedBatching,
+			// need to investigate - some small number of t9s tests time out with this option on.
+			// chunkSizeInBytes,
+			enableRuntimeIdCompressor,
+			...options,
+		};
+	}
+
+	return options;
+}
+
 /**
  * @internal
  */
@@ -138,7 +235,10 @@ export async function getVersionedTestObjectProviderFromApis(
 		return new factoryCtor(
 			TestDataObjectType,
 			dataStoreFactory,
-			containerOptions?.runtimeOptions,
+			filterRuntimeOptionsForVersion(
+				apis.containerRuntime.version,
+				containerOptions?.runtimeOptions,
+			),
 		);
 	};
 
@@ -207,6 +307,28 @@ export async function getCompatVersionedTestObjectProviderFromApis(
 		apis.dataRuntimeForLoading,
 	);
 
+	// We want to ensure that we are testing all latest runtime features, but only if both runtimes
+	// (one that creates containers and one that loads them) support them.
+	//
+	// Theoretically it should be fine to use config for apis.containerRuntimeForLoading?.version.
+	// If it's higher then apis.containerRuntime, then unknown to lower version of apis.containerRuntime
+	// would be ignored.
+	//
+	// But TestObjectProviderWithVersionedLoad.createLoader() implementation is dumb - it resets this.useCreateApi
+	// on first call and thus uses apis.containerRuntimeForLoading for any container created after.
+	// Many use non-first container instance to send ops, so that screws things up.
+	//
+	// As result, we absolutly need to use the min between two versions!
+	const versionForCreating = apis.containerRuntime?.version;
+	assert(versionForCreating !== undefined, "versionForCreating");
+	const versionForLoading = apis.containerRuntimeForLoading?.version;
+	assert(versionForLoading !== undefined, "versionForLoading");
+
+	const minVersion =
+		semver.compare(versionForCreating, versionForLoading) < 0
+			? versionForCreating
+			: versionForLoading;
+
 	const createContainerFactoryFn = (containerOptions?: ITestContainerConfig) => {
 		const dataStoreFactory = getDataStoreFactoryFn(containerOptions);
 		const factoryCtor = createTestContainerRuntimeFactory(
@@ -215,7 +337,7 @@ export async function getCompatVersionedTestObjectProviderFromApis(
 		return new factoryCtor(
 			TestDataObjectType,
 			dataStoreFactory,
-			containerOptions?.runtimeOptions,
+			filterRuntimeOptionsForVersion(minVersion, containerOptions?.runtimeOptions),
 			[innerRequestHandler],
 		);
 	};
@@ -235,7 +357,7 @@ export async function getCompatVersionedTestObjectProviderFromApis(
 		return new factoryCtor(
 			TestDataObjectType,
 			dataStoreFactory,
-			containerOptions?.runtimeOptions,
+			filterRuntimeOptionsForVersion(minVersion, containerOptions?.runtimeOptions),
 			[innerRequestHandler],
 		);
 	};