PCI: Release unused bridge resources during resize #6653
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resizing BARs can be blocked when a device in the bridge hierarchy
itself consumes resources from the resized range. This scenario is
common with Intel Arc DG2 GPUs where the following is a typical
topology:
+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380]
\-04.0-[61]----00.0 Intel Corporation DG2 Audio Controller
Here the system BIOS has provided a large 64bit, prefetchable window:
pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window]
But only a small portion is programmed into the root port aperture:
pci 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
The upstream port then provides the following aperture:
pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
With the missing range found to be consumed by the switch port itself:
pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]
The downstream port above the GPU provides the same aperture as upstream:
pci 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
Which is entirely consumed by the GPU:
pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]
In summary, iomem reports the following:
b000000000-bfffffffff : PCI Bus 0000:5d
bfe0000000-bff07fffff : PCI Bus 0000:5e
bfe0000000-bfefffffff : PCI Bus 0000:5f
bfe0000000-bfefffffff : PCI Bus 0000:60
bfe0000000-bfefffffff : 0000:60:00.0
bff0000000-bff07fffff : 0000:5e:00.0
The GPU at 0000:60:00.0 supports a Resizable BAR:
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
However when attempting a resize we get -ENOSPC:
pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: can't assign; no space
pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: failed to assign
pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
pcieport 0000:5d:00.0: bridge window [mem 0xb9000000-0xba0fffff]
pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
pcieport 0000:5e:00.0: PCI bridge to [bus 5f-61]
pcieport 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
pcieport 0000:5f:01.0: PCI bridge to [bus 60]
pcieport 0000:5f:01.0: bridge window [mem 0xb9000000-0xb9ffffff]
pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: assigned
In this example we need to resize all the way up to the root port
aperture, but we refuse to change the root port aperture while resources
are allocated for the upstream port BAR.
The solution proposed here builds on the idea in commit 91fa127
("PCI: Expose PCIe Resizable BAR support via sysfs") where the BAR can
be resized while there is no driver attached. In this case, when there
is no driver bound to the upstream switch port we'll release resources
of the bridge which match the reallocation. Therefore we can achieve
the below successful resize operation by unbinding 0000:5e:00.0 from the
pcieport driver before invoking the resource2_resize interface on the
GPU at 0000:60:00.0.
pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]: releasing
pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]: releasing
pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]: assigned
pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
pci 0000:5e:00.0: BAR 0 [mem 0xb200000000-0xb2007fffff 64bit pref]: assigned
pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
pci 0000:60:00.0: BAR 2 [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
pci 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
pcieport 0000:5d:00.0: bridge window [mem 0xb9000000-0xba0fffff]
pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]
pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
pci 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
pcieport 0000:5f:01.0: PCI bridge to [bus 60]
pcieport 0000:5f:01.0: bridge window [mem 0xb9000000-0xb9ffffff]
pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
Link: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Signed-off-by: Alex Williamson <[email protected]>
Signed-off-by: Jonathan Bell <[email protected]>
P33M
force-pushed
the
bar_resize_bridge
branch
from
08662e5 to
26e9a9d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Take the suggested patch which frees upstream bridge resources if a BAR is being resized.