TLB resource management #43
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cfriedt
reviewed
Feb 28, 2025
There was a problem hiding this comment.
Looks good to me - I defer to @alewycky-tenstorrent for approval, but am looking forward to using this.
This is the first part of moving TLB window management into the driver. Add the necessary interfaces and data structures for TLB window allocation, configuration, and ownership tracking: - Add bitmap and refcount tracking in chardev_private for per-fd TLB ownership - Define new IOCTLs for TLB allocation, freeing, and configuration - Add TLB management callback interfaces to device class - Add tlb.h
Implement the TLB management interfaces introduced in the previous commit: - Add TLB allocation logic using bitmap tracking at the device level - Add TLB allocation, freeing, and configuration IOCTL handlers - Add cleanup of TLBs on file descriptor close - Extend mmap offset space to accommodate TLB windows - Add tlb.c implementing core TLB management functions The allocation strategy supports multiple TLB kinds of different sizes, selecting the smallest TLB that can accommodate the requested size. The actual TLB configuration is delegated to device-specific implementations via the device class callbacks.
Implement memory mapping support for TLB windows: - Add map_tlb_window function to handle TLB-specific mmap requests - Add vma_close() handler to track TLB mapping reference counts - Wire up TLB window mapping in tenstorrent_mmap() The implementation uses reference counting to track active mappings of each TLB window, ensuring we don't free TLBs that are still mapped into user address spaces. Window mappings are backed by PCI BAR regions.
- Support both 2M and 4G window sizes - Add separate configuration handlers for each size - Add TLB descriptor implementation - Convert kernel TLB window usage to new interface The Blackhole chip provides 202 2M windows and 8 4G windows, mapped through BAR0 and BAR4 respectively.
joelsmithTT
force-pushed
the
joelsmith/tlb-resource-admin
branch
from
12c9241 to
24902be
Compare
joelsmithTT
force-pushed
the
joelsmith/tlb-resource-admin
branch
2 times, most recently
from
0709e42 to
3b3fc9d
Compare
- Support 1M, 2M, and 16M window sizes - Constructs register config for all TLB kinds into a u64 - Sanity check for address alignment and width Wormhole supports 156 1M windows, 10 2M windows, and 20 16M windows, all in BAR0.
joelsmithTT
force-pushed
the
joelsmith/tlb-resource-admin
branch
from
3b3fc9d to
420f2d9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce TLB resource management to KMD.
https://tenstorrent.atlassian.net/wiki/spaces/syseng/pages/537100642/Inbound+PCIe+TLBs
Key features:
Implementation details:
tlb.h/tlb.cFuture work: