Performance of `dotnet format` is very bad compared to `dotnet-format`

[bradwilson]

Versions used

dotnet format via .NET SDK 8.0.200
dotnet-format NuGet package version 5.1.250801

Describe the bug

The NuGet package has been "deprecated" in favor of built-in dotnet format, and I'm blocked from upgrading because of very bad performance.

As part of the normal build process, today I run dotnet dotnet-format -f --check to ensure people are properly formatting source files before submitting PRs (or they will also fail to build during the PR build process). I've been doing this for quite a while, with good result.

I ran across a new C# feature recently that dotnet-format does not understand (makes sense, since it's a few years old at this point), so I went to go see what the state of dotnet format is. In the past I rejected it because it was too slow. It's still too slow.

In the xunit project on the v2 branch, here are the timing differences:

2.4 seconds: measure-command { dotnet dotnet-format --check --folder --verbosity minimal --exclude src/common/AssemblyResolution/Microsoft.DotNet.PlatformAbstractions --exclude src/common/AssemblyResolution/Microsoft.Extensions.DependencyModel } (this is the command I run by default today)

27.9 seconds: measure-command { dotnet format --verify-no-changes --verbosity minimal --exclude src/common/AssemblyResolution/Microsoft.DotNet.PlatformAbstractions src/common/AssemblyResolution/Microsoft.Extensions.DependencyModel }

I felt like spending 2.4 seconds to verify formatting was an acceptable cost from a command line build, but nearly 30 seconds is excessive.

If I take away the --folder option from dotnet-format, the time rises to 11.5 seconds. That's why I use the --folder option, because it bypasses a ton of the stuff I don't care about. There is no such option any more. Why not? I might be able to accept 2.4 seconds becoming 5 (even though that is not great, either), but definitely not 30.

[jaredpar]

As a quick check I ran dotnet format --binarylog format.binlog on the xunit repo. One thing that stood out was the following:

Essentially the annotating reference assembly task is running even on design time builds and it's taking up the bulk of time here (34 seconds in the evaluation, 21 spent on annotate assembly references). I'm not familiar enough with that package to say if there is a better way of doing that or not. Given that it's mutating assemblies and updating the reference set of compilations it likely needs to occur in some form though so that format can faithfully re-create compilations for analysis.

Note: didn't do any profiling of the actual format work, just took a quick look at the binlog
Note2: :I was looking at main when I did this.

[bradwilson]

The main branch (with a simpler command, because no third party source) is worse:

3.2 seconds: measure-command { dotnet dotnet-format --check --folder --verbosity minimal }
50.6 seconds: measure-command { dotnet format --verify-no-changes --verbosity minimal }

There's also this in the output:

Warnings were encountered while loading the workspace. Set the verbosity option to the 'diagnostic' level to log warnings.
Required references did not load for xunit.v3.assert.all-off(netstandard1.1) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-off(netstandard1.2) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-off(netstandard1.3) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-off(netstandard1.4) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-off(netstandard1.5) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-off(netstandard1.6) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-on(netstandard1.1) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.immutable or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.1) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.2) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.3) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.4) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.5) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.nullable-mixed(netstandard1.6) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.span or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.tests(net472) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.all-on(netstandard1.1) or referenced project. Run `dotnet restore` prior to formatting.
Required references did not load for xunit.v3.assert.tests(net472) or referenced project. Run `dotnet restore` prior to formatting.

I have obviously performed dotnet restore before running the command.

I expect some of the reason this got worse is because I have a lot of projects that are including the source from other projects (all the xunit.v3.assert.* projects, which are re-building the assertion library source with different DEFINEs, different target frameworks, and different language versions). This is yet another reason I'd vastly prefer a "just analyze the source, stop caring about the projects" mode.

[timheuer]

@sharwell FYI (not sure if we should transfer to https://github.com/dotnet/format

[bradwilson]

@timheuer I logged it here because someone told me dotnet/format was being merged into dotnet/sdk. 😄

[jaredpar]

I logged it here because someone told me dotnet/format was being merged into dotnet/sdk. 😄

s/someone/jaredpar 😄

[arunchndr]

Starting with @sharwell

[JoeRobich]

3.2 seconds: measure-command { dotnet dotnet-format --check --folder --verbosity minimal }
50.6 seconds: measure-command { dotnet format --verify-no-changes --verbosity minimal }

The whitespace subcommand supports the --folder flag. measure-command { dotnet format whitespace --folder --verify-no-changes --verbosity minimal } would be more apples to apples as the 5.x dotnet-format only ran whitespace formatting by default.

[bradwilson]

The whitespace subcommand supports the --folder flag. measure-command { dotnet format whitespace --verify-no-changes --verbosity minimal } would be more apples to apples as the 5.x dotnet-format only ran whitespace formatting by default.

Thanks, that does give better results:

Measure-Command { dotnet format whitespace --folder } => 3.388s
Measure-Command { dotnet dotnet-format --folder } => 2.927s

That's a non-meaningful difference. 🎉