Request for Benchmarks

[the-database]

I'm collecting benchmarks for different hardware configurations on the wiki here: https://github.com/the-database/mpv-upscale-2x_animejanai/wiki/Benchmarks

If you would like to contribute your benchmarks, please try downloading the prerelease archive 1.0.1 here: https://github.com/the-database/mpv-upscale-2x_animejanai/releases/tag/1.0.1

To run the benchmark, extract the release archive and then run the bat file at: mpv-upscale-2x_animejanai\portable_config\shaders\animejanai_v2_benchmark_all.bat. If running the bat just opens and closes the console Window immediately, an error has occurred. If that happens, please open a command window, navigate to the mpv-upscale-2x_animejanai\portable_config\shaders directory, and then run this command: .\animejanai_v2_benchmark_all.bat. Any errors that occurred should be printed to the console, please share them here.

[foxbox93]

Unfortunately, I think my computer's performance is not good enough to test benchmarking
As shown in the picture, the graphics card is broken, so I can't proceed

Only Visual Studio 2022 and cmd are on
Tested with mpv-upscale-2x_animejanai-1.0.1 you posted.
The cpu used is i5 13600k
gpu is Geforce RTX 4070 ti

In my case, it would be better to lower the .onnx specifications.

**Additionally #9 updated the best combination in my environment
#9 (comment)

Also, I tested rife_cuda, and it seems that it can be used if it is the same as the existing my 60fps setting

Best setting results for my environment

rife_cuda test

custom file ( i5 13600k + 4070 ti )

custom.zip

[your-eggcellency]

NVIDIA GeForce RTX 4090
Intel Core i9-13900K

benchmark.txt

[hooke007]

7950x(No PBO) + 5600Mhz DDR5(2x) + RTX4080
It's interesting that one of the scores is higher than 4090.

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	345.79 fps	167.99 fps	126.77 fps	62.67 fps	27.93 fps
2x (2x_UltraCompact)	671.05 fps	323.08 fps	242.30 fps	117.99 fps	52.73 fps
2x (2x_SuperUltraCompact)	1840.96 fps	827.06 fps	571.67 fps	275.27 fps	118.53 fps
4x (2x_Compact+2x_Compact)	66.89 fps	37.35 fps	28.49 fps	19.40 fps
4x (2x_Compact+2x_UltraCompact)	107.56 fps	57.28 fps	45.09 fps	28.78 fps
4x (2x_Compact+2x_SuperUltraCompact)	203.77 fps	102.28 fps	77.05 fps	43.98 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	283.85 fps	144.57 fps	104.65 fps	65.30 fps
4x (2x_UltraCompact+2x_UltraCompact)	126.33 fps	70.66 fps	54.11 fps	36.45 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	308.08 fps	174.35 fps	119.94 fps	85.43 fps

[your-eggcellency]

@hooke007

It's interesting that one of the scores is higher than 4090.

Are you referring to 480x360 2x (2x_SuperUltraCompact) 1520 fps vs 1840 fps? Framerate is extremely high here so the difference is likely due to CPU and other factors, not GPU itself. This benchmark runs in under 1 second so there's huge variability. I tried a few runs and it's giving me results ranging from 1450 to 1750.

[hooke007]

Yes of course CPU and RAM speed would affect the speed. And you can check the wiki page to see more output.

[the-database]

Thanks to everyone who has been sharing benchmarks, please keep them coming.

Based on some of those results it seems like it might be useful to have another model with performance in between UltraCompact and SuperUltraCompact. For example the 3070 and 3070 Ti are barely unable to run UltraCompact on 23.796 fps 1080p video smoothly in mpv but they can run SuperUltraCompact easily. I'll look into it to see if it can be done easily.

[foxbox93]

Based on 3070 and 3070 ti, it would be good to have the following rules.

If the 3070 is aiming to play 1080p60 I think it would be better to design the model assuming resize from the start.

Since the 1080p60 video comes out at 52.65 fps even with SuperUltraCompact anyway, you should also think about the situation of resizing.

I recommend designing assuming 1080p is resized to 720p.

[720p-3070]

• UltraCopact : 54.16 fps
• SuperUltraCompact : 121.76 fps

If you lower the performance slightly on UltraCompact, it will work smoothly at 60fps.

The following up-scaling methods are likely to be possible in 3070.

30fps
1080p : V2_Ultra_Compact (x2)
1080p : resize720 + V2_Ultra_Compact + V2_Ultra_Compact (x4)
720p : V1_Compact (x2)
720p : V2_Ultra_Compact + V2_Ultra_Compact (x4)

60fps
1080p : resize720 + V2_Ultra_Compact (x2)
720p : V2_Ultra_Compact (x2)

[your-eggcellency]

@the-database, I think it would be useful to add 6x 480x360 (or 8x) benchmarks for those people who are going to upscale old/low resolution anime into 4K. I personally have 4K screen so I might encounter such situations. Perhaps you're expecting to upscale 4x and then rely on more simple upscaling algorithms?

[RoyalQX]

NVIDIA GeForce RTX 3070 TI
AMD Ryzen 5 5600
benchmark.txt

[animeojisan]

NVIDIA GeForce RTX 4070 TI
AMD Ryzen 5 5600G

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	254.65 fps	132.56 fps	93.55 fps	45.74 fps	20.27 fps
2x (2x_UltraCompact)	485.64 fps	226.51 fps	160.02 fps	77.83 fps	34.48 fps
2x (2x_SuperUltraCompact)	557.97 fps	326.24 fps	229.27 fps	108.35 fps	48.22 fps
4x (2x_Compact+2x_Compact)	49.13 fps	27.47 fps	20.98 fps	14.08 fps
4x (2x_Compact+2x_UltraCompact)	74.24 fps	42.19 fps	30.87 fps	19.68 fps
4x (2x_Compact+2x_SuperUltraCompact)	92.84 fps	52.17 fps	38.13 fps	23.43 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	111.77 fps	59.93 fps	46.04 fps	29.86 fps
4x (2x_UltraCompact+2x_UltraCompact)	85.29 fps	46.37 fps	35.81 fps	23.96 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	115.01 fps	65.29 fps	50.32 fps	33.36 fps

[Kimi0n]

RTX 3060 12GB
Ryzen 5 5600X with PBO enabled
32GB DDR4 3200Mhz Ram (4x8GB)

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	113.86 fps	66.14 fps	46.68 fps	20.60 fps	9.27 fps
2x (2x_UltraCompact)	211.46 fps	123.16 fps	86.52 fps	37.88 fps	18.28 fps
2x (2x_SuperUltraCompact)	628.99 fps	367.28 fps	260.99 fps	110.11 fps	57.36 fps
4x (2x_Compact+2x_Compact)	23.78 fps	13.55 fps	10.44 fps	6.89 fps
4x (2x_Compact+2x_UltraCompact)	37.52 fps	21.42 fps	16.23 fps	9.97 fps
4x (2x_Compact+2x_SuperUltraCompact)	67.93 fps	39.39 fps	28.96 fps	15.65 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	93.88 fps	53.87 fps	40.53 fps	23.68 fps
4x (2x_UltraCompact+2x_UltraCompact)	44.14 fps	25.23 fps	19.40 fps	11.77 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	133.48 fps	76.20 fps	59.18 fps	33.98 fps

[AlteriaX]

Intel i5-10500H
16GB DDR4 2933MHz RAM (2 x 8GB)
RTX 3060 Laptop 6GB (105W)

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	107.59 fps	61.59 fps	42.79 fps	21.91 fps	9.60 fps
2x (2x_UltraCompact)	177.20 fps	99.31 fps	68.37 fps	34.68 fps	15.43 fps
2x (2x_SuperUltraCompact)	574.37 fps	331.01 fps	229.27 fps	113.49 fps	50.10 fps
4x (2x_Compact+2x_Compact)	22.12 fps	12.71 fps	9.58 fps	6.81 fps
4x (2x_Compact+2x_UltraCompact)	31.51 fps	17.67 fps	13.36 fps	9.28 fps
4x (2x_Compact+2x_SuperUltraCompact)	62.11 fps	35.11 fps	25.45 fps	15.49 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	80.94 fps	44.88 fps	32.93 fps	20.57 fps
4x (2x_UltraCompact+2x_UltraCompact)	35.64 fps	19.85 fps	15.19 fps	10.90 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	118.80 fps	65.62 fps	50.30 fps	34.95 fps

[Galahahad]

i5 12600k
RTX 3080 12G
16GB DDR4 3600MHz RAM (2 x 8GB)

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	240.89 fps	143.15 fps	101.94 fps	51.05 fps	23.06 fps
2x (2x_UltraCompact)	404.59 fps	239.46 fps	169.87 fps	84.12 fps	38.05 fps
2x (2x_SuperUltraCompact)	896.09 fps	536.44 fps	391.25 fps	194.05 fps	88.24 fps
4x (2x_Compact+2x_Compact)	52.35 fps	30.23 fps	23.31 fps	16.03 fps
4x (2x_Compact+2x_UltraCompact)	76.19 fps	44.05 fps	33.60 fps	21.82 fps
4x (2x_Compact+2x_SuperUltraCompact)	124.32 fps	72.34 fps	54.23 fps	32.39 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	156.80 fps	90.88 fps	68.65 fps	42.78 fps
4x (2x_UltraCompact+2x_UltraCompact)	87.26 fps	50.18 fps	38.78 fps	26.20 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	198.58 fps	115.12 fps	88.64 fps	60.41 fps

i5 12600k
RTX 3080 12G
32GB DDR4 3600MHz RAM (4 x 8GB)

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	239.51 fps	142.09 fps	101.09 fps	50.71 fps	22.85 fps
2x (2x_UltraCompact)	405.23 fps	239.25 fps	169.49 fps	84.25 fps	38.02 fps
2x (2x_SuperUltraCompact)	889.07 fps	543.97 fps	387.56 fps	194.30 fps	88.14 fps
4x (2x_Compact+2x_Compact)	52.05 fps	30.06 fps	23.21 fps	16.06 fps
4x (2x_Compact+2x_UltraCompact)	76.02 fps	43.92 fps	33.53 fps	21.87 fps
4x (2x_Compact+2x_SuperUltraCompact)	123.95 fps	72.14 fps	53.89 fps	32.19 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	157.29 fps	90.93 fps	68.56 fps	42.85 fps
4x (2x_UltraCompact+2x_UltraCompact)	87.05 fps	50.06 fps	38.46 fps	26.26 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	198.75 fps	115.03 fps	87.39 fps	60.48 fps

[mitch1887]

NVIDIA GeForce RTX 2080 Ti
AMD Ryzen 9 5900X
32GB DDR4 3800MHz RAM (2 x 16GB)

	480x360	640x480	768x576	1280x720	1920x1080
2x (2x_Compact)	197.70 fps	115.51 fps	81.54 fps	40.39 fps	18.11 fps
2x (2x_UltraCompact)	366.62 fps	214.35 fps	151.26 fps	75.05 fps	33.63 fps
2x (2x_SuperUltraCompact)	759.26 fps	451.13 fps	321.36 fps	160.86 fps	72.56 fps
4x (2x_Compact+2x_Compact)	41.90 fps	24.03 fps	18.47 fps	12.61 fps
4x (2x_Compact+2x_UltraCompact)	65.91 fps	37.86 fps	28.78 fps	18.40 fps
4x (2x_Compact+2x_SuperUltraCompact)	102.40 fps	59.11 fps	43.97 fps	25.95 fps
4x (2x_UltraCompact+2x_SuperUltraCompact)	134.48 fps	77.38 fps	58.56 fps	36.84 fps
4x (2x_UltraCompact+2x_UltraCompact)	77.86 fps	44.60 fps	34.36 fps	23.23 fps
4x (2x_SuperUltraCompact+2x_SuperUltraCompact)	165.85 fps	95.57 fps	73.74 fps	50.04 fps