Skip to content

Commit 137e667

Browse files
IgorBagnuckiIgorBagnucki
authored
Merge branch 'main' into istep_8_optimizations
2 parents 976d758 + e1da80e commit 137e667

31 files changed

+6337
-12103
lines changed

‎LICENSE

Lines changed: 451 additions & 21 deletions
Large diffs are not rendered by default.

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ programming guides:
3434
- [POWER9 Registers vol3](https://ibm.ent.box.com/s/flt3hs6eiwd9glq3yzzff0flnup2j7p0)
3535
- [POWER ISA v3.0B](https://ibm.ent.box.com/s/1hzcwkwf8rbju5h9iyf44wm94amnlcrv)
3636
- [POWER9 processor errata](https://ibm.ent.box.com/s/0ixfserqjzjmt3q6vabotz9arxzs59md)
37+
- [POWER9 IPL flow](https://wiki.raptorcs.com/w/images/b/bd/IPL-Flow-POWER9.pdf)
3738

3839
Other useful information extracted form documents aboive may be found in
3940
[ppc.md](devnotes/ppc.md).

‎devnotes/RAM_addressing.ods

18.9 KB
Binary file not shown.

‎devnotes/isteps/13_dram_training/01_host_disable_memvolt.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## host_disable_memvolt: Disable VDDR on Warm Reboots (13.1)
2+
3+
> a) Power off dram - VDDR and vPP. Must drop VDDR first, then VPP.
4+
> - Turned off here to handle reconfig loop for dimm failure
5+
> - Only really issued if VDDR/VPP is on
6+
7+
No-op for Nimbus (why IPL list doesn't say so?), enable pins driven by FPGA, not
8+
configurable.

‎devnotes/isteps/13_dram_training/02_mem_pll_reset.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
## mem_pll_reset: Reset PLL for MCAs in async (13.2)
2+
3+
> a) p9_mem_pll_reset.C (proc chip)
4+
> - This step is a no-op on cumulus as the centaur is already has its PLLs setup in step 11
5+
> - This step is a no-op if memory is running in synchronous mode since the MCAs are using the nest PLL, HWP detect
6+
> and exits
7+
> - If in async mode then this HWP will put the PLL into bypass, reset mode
8+
> - Disable listen_to_sync for MEM chiplet, whenever MEM is not in sync to NEST
9+
10+
```
11+
For each functional Proc:
12+
> if ( !ATTR_MC_SYNC_MODE )
13+
For each functional MC(BIST?):
14+
// Assert endpoint reset
15+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WOR) // 0x070F0042
16+
[all] 0
17+
[1] PCB_EP_RESET = 1
18+
19+
// Mask PLL unlock error in PCB slave
20+
TP.TPCHIP.NET.PCBSLMC01.SLAVE_CONFIG_REG // 0x070F001E
21+
[12] (part of) ERROR_MASK = 1
22+
23+
// Move MC PLL into reset state (3 separate writes, no delays between them)
24+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WOR) // 0x070F0042
25+
[all] 0
26+
[5] PLL_BYPASS = 1
27+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WOR) // 0x070F0042
28+
[all] 0
29+
[4] PLL_RESET = 1
30+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WOR) // 0x070F0042
31+
[all] 0
32+
[3] PLL_TEST_EN = 1
33+
34+
// Assert MEM PLDY and DCC bypass
35+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL1 (WOR) // 0x070F0046
36+
[all] 0
37+
[1] CLK_DCC_BYPASS_EN = 1
38+
[2] CLK_PDLY_BYPASS_EN = 1
39+
40+
// Drop endpoint reset
41+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WAND) // 0x070F0041
42+
[all] 1
43+
[1] PCB_EP_RESET = 0
44+
45+
// Disable listen to sync pulse to MC chiplet, when MEM is not in sync to nest
46+
TP.TCMC01.MCSLOW.SYNC_CONFIG // 0x07030000
47+
[4] LISTEN_TO_SYNC_PULSE_DIS = 1
48+
49+
// Initialize OPCG_ALIGN register
50+
TP.TCMC01.MCSLOW.OPCG_ALIGN // 0x07030001
51+
[all] 0
52+
[0-3] INOP_ALIGN = 5 // 8:1
53+
[12-19] INOP_WAIT = 0
54+
[47-51] SCAN_RATIO = 0 // 1:1
55+
[52-63] OPCG_WAIT_CYCLES = 0x20
56+
57+
// scan0 flush PLL boundary ring
58+
TP.TCMC01.MCSLOW.CLK_REGION // 0x07030006
59+
[all] 0
60+
[14] CLOCK_REGION_UNIT10 = 1
61+
[48] SEL_THOLD_SL = 1
62+
[49] SEL_THOLD_NSL = 1
63+
[50] SEL_THOLD_ARY = 1
64+
TP.TCMC01.MCSLOW.SCAN_REGION_TYPE // 0x07030005
65+
[all] 0
66+
[14] SCAN_REGION_UNIT10 = 1
67+
[56] SCAN_TYPE_BNDY = 1
68+
TP.TCMC01.MCSLOW.OPCG_REG0 // 0x07030002
69+
[0] RUNN_MODE = 0
70+
// Separate write, but don't have to read again
71+
TP.TCMC01.MCSLOW.OPCG_REG0 // 0x07030002
72+
[2] RUN_SCAN0 = 1
73+
74+
timeout(200 * 16us):
75+
TP.TCMC01.MCSLOW.CPLT_STAT0 // 0x07000100
76+
if (([8] CC_CTRL_OPCG_DONE_DC) == 1) break
77+
delay(16us)
78+
79+
// Cleanup
80+
TP.TCMC01.MCSLOW.CLK_REGION // 0x07030006
81+
[all] 0
82+
TP.TCMC01.MCSLOW.SCAN_REGION_TYPE // 0x07030005
83+
[all] 0
84+
```

‎devnotes/isteps/13_dram_training/03_mem_pll_initf.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
## mem_pll_initf: PLL Initfile for MBAs (13.3)
2+
3+
> a) p9_mem_pll_initf.C (proc chip)
4+
> - This step is a no-op on cumulus
5+
> - This step is a no-op if memory is running in synchronous mode since the MCAs are using the nest PLL, HWP detect
6+
> and exits
7+
> - MCA PLL setup
8+
> - Note that Hostboot doesn't support twiddling bits, Looks up which "bucket" (ring) to use from attributes set
9+
> during mss_freq
10+
> - Then request the SBE to scan ringId with setPulse
11+
> - SBE needs to support 5 RS4 images
12+
> - Data is stored as a ring image in the SBE that is frequency specific
13+
> - 5 different frequencies (1866, 2133, 2400, 2667, EXP)
14+
15+
```
16+
For each functional Proc:
17+
> if ( !ATTR_MC_SYNC_MODE )
18+
For each functional MCBIST
19+
- fapi2::putRing(mbist, ring_id(depends on RAM freq), RING_MODE_SET_PULSE_NSL)
20+
// FIXME: depending on whether putRing() is used anywhere else, we may implement this as a function or directly
21+
- /src/user/scan/scandd.C:169 sbeScanPerformOp()
22+
```

‎devnotes/isteps/13_dram_training/04_mem_pll_setup.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
## mem_pll_setup: Setup PLL for MBAs (13.4)
2+
3+
> a) p9_mem_pll_setup.C (proc chip)
4+
> - This step is a no-op on cumulus
5+
> - This step is a no-op if memory is running in synchronous mode since the MCAs are using the nest PLL, HWP detect
6+
> and exits
7+
> - MCA PLL setup
8+
> - Moved PLL out of bypass (just DDR)
9+
> - Performs PLL checking
10+
11+
```
12+
For each functional Proc:
13+
> if ( !ATTR_MC_SYNC_MODE )
14+
For each functional MC(BIST?):
15+
// Drop PLDY bypass of Progdelay logic
16+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL1 (WAND) // 0x070F0045
17+
[all] 1
18+
[2] CLK_PDLY_BYPASS_EN = 0
19+
20+
// Drop DCC bypass of DCC logic
21+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL1 (WAND) // 0x070F0045
22+
[all] 1
23+
[1] CLK_DCC_BYPASS_EN = 0
24+
25+
// Attribute description: "Skip the locking sequence and check for lock of NEST/MEM/XBUS/OBUS/PCI PLLs".
26+
// Can't find where it is set, assuming 0.
27+
> if (ATTR_NEST_MEM_X_O_PCI_BYPASS == 0)
28+
// Drop PLL test enable
29+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WAND) // 0x070F0041
30+
[all] 1
31+
[3] PLL_TEST_EN = 0
32+
33+
// Drop PLL reset
34+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WAND) // 0x070F0041
35+
[all] 1
36+
[4] PLL_RESET = 0
37+
38+
delay(5ms)
39+
40+
// Check PLL lock
41+
TP.TPCHIP.NET.PCBSLMC01.PLL_LOCK_REG // 0x070F0019
42+
assert([0] (reserved) == 1)
43+
44+
// Drop PLL Bypass
45+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WAND) // 0x070F0041
46+
[all] 1
47+
[5] PLL_BYPASS = 0
48+
49+
// Set scan ratio to 4:1
50+
TP.TCMC01.MCSLOW.OPCG_ALIGN // 0x07030001
51+
[47-51] SCAN_RATIO = 3 // 4:1
52+
53+
> end if
54+
55+
// Reset PCB Slave error register
56+
TP.TPCHIP.NET.PCBSLMC01.ERROR_REG // 0x070F001F
57+
[all] 1 // Write 1 to clear
58+
59+
// Unmask PLL unlock error in PCB slave
60+
TP.TPCHIP.NET.PCBSLMC01.SLAVE_CONFIG_REG // 0x070F001E
61+
[12] (part of) ERROR_MASK = 0
62+
```

‎devnotes/isteps/13_dram_training/05_proc_mcs_skewadjust.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## proc_mcs_skewadjust: Update clock mesh deskew (13.5)
2+
3+
> a) This step is a no-op

‎devnotes/isteps/13_dram_training/06_mem_startclocks.md

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
## mem_startclocks: Start clocks on MBA/MCAs (13.6)
2+
3+
> a) p9_mem_startclocks.C (proc chip)
4+
> - This step is a no-op on cumulus
5+
> - This step is a no-op if memory is running in synchronous mode since the MCAs are using the nest PLL, HWP detect
6+
> and exits
7+
> - Drop fences and tholds on MBA/MCAs to start the functional clocks
8+
9+
```
10+
For each functional Proc:
11+
pg_vector = bitmap of ATTR_CHIP_UNIT_POS of functional Perv children of Proc, bit numbers come from Perl + XML
12+
- /src/import/chips/p9/procedures/hwp/perv/p9_sbe_common.C:692
13+
> if(!ATTR_MC_SYNC_MODE)
14+
For each functional MC(BIST?):
15+
// Call p9_mem_startclocks_cplt_ctrl_action_function for Mc chiplets
16+
p9_mem_startclocks_cplt_ctrl_action_function():
17+
// Drop partial good fences
18+
TP.TCMC01.MCSLOW.CPLT_CTRL1 (WO_CLEAR) // 0x07000021
19+
[all] 0
20+
[3] TC_VITL_REGION_FENCE = ~ATTR_PG[3] // TODO: where does ATTR_PG value come from?
21+
[4-14] TC_REGION{1-3}_FENCE, UNUSED_{8-14}B = ~ATTR_PG[4-14]
22+
// Reset abistclk_muxsel and syncclk_muxsel
23+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_CLEAR) // 0x07000020
24+
[all] 0
25+
[0] CTRL_CC_ABSTCLK_MUXSEL_DC = 1
26+
[1] TC_UNIT_SYNCCLK_MUXSEL_DC = 1
27+
28+
// Call module align chiplets for Mc chiplets
29+
p9_sbe_common_align_chiplets():
30+
// Exit flush
31+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_OR) // 0x07000010
32+
[all] 0
33+
[2] CTRL_CC_FLUSHMODE_INH_DC = 1
34+
35+
// Enable alignement
36+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_OR) // 0x07000010
37+
[all] 0
38+
[3] CTRL_CC_FORCE_ALIGN_DC = 1
39+
40+
// Clear chiplet is aligned
41+
TP.TCMC01.MCSLOW.SYNC_CONFIG // 0x07030000
42+
[7] CLEAR_CHIPLET_IS_ALIGNED = 1
43+
44+
// Unset Clear chiplet is aligned
45+
// No delay, but Hostboot does a second read, doesn't reuse previously written value
46+
TP.TCMC01.MCSLOW.SYNC_CONFIG // 0x07030000
47+
[7] CLEAR_CHIPLET_IS_ALIGNED = 0
48+
49+
delay(100us)
50+
51+
// Line below copied from Hostboot, but it mentions wrong bit
52+
// Poll OPCG done bit to check for run-N completeness
53+
timeout(10*100us):
54+
TP.TCMC01.MCSLOW.CPLT_STAT0 // 0x07000100
55+
if (([9] CC_CTRL_CHIPLET_IS_ALIGNED_DC) == 1) break
56+
delay(100us)
57+
58+
// Disable alignment
59+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_CLEAR) // 0x07000020
60+
[all] 0
61+
[3] CTRL_CC_FORCE_ALIGN_DC = 1
62+
63+
// Call module clock start stop for MC01, MC23
64+
p9_sbe_common_clock_start_stop(l_trgt_chplt, CLOCK_CMD = 1,
65+
DONT_STARTSLAVE = 0, DONT_STARTMASTER = 0, l_clock_regions,
66+
CLOCK_TYPES = 0x7):
67+
// Chiplet exit flush
68+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_OR) // 0x07000010
69+
[all] 0
70+
[2] CTRL_CC_FLUSHMODE_INH_DC = 1
71+
72+
// Clear Scan region type register
73+
TP.TCMC01.MCSLOW.SCAN_REGION_TYPE // 0x07030005
74+
[all] 0
75+
76+
// Setup all Clock Domains and Clock Types
77+
TP.TCMC01.MCSLOW.CLK_REGION // 0x07030006
78+
[0-1] CLOCK_CMD = 1 // start
79+
[2] SLAVE_MODE = 0
80+
[3] MASTER_MODE = 0
81+
[4-14] CLOCK_REGION_* = ATTR_PG[4-14]
82+
[48] SEL_THOLD_SL = 1
83+
[49] SEL_THOLD_NSL = 1
84+
[50] SEL_THOLD_ARY = 1
85+
86+
// Poll OPCG done bit to check for completeness
87+
timeout(10*100us):
88+
TP.TCMC01.MCSLOW.CPLT_STAT0 // 0x07000100
89+
if (([8] CC_CTRL_OPCG_DONE_DC) == 1) break
90+
delay(100us)
91+
92+
// Here Hostboot calculates what is expected clock status, based on previous values and requested command. It is done
93+
// by generic functions, but because we know exactly which clocks were to be started, we can test just for those.
94+
TP.TCMC01.MCSLOW.CLOCK_STAT_SL // 0x07030008
95+
TP.TCMC01.MCSLOW.CLOCK_STAT_NSL // 0x07030009
96+
TP.TCMC01.MCSLOW.CLOCK_STAT_ARY // 0x0703000A
97+
assert(([4-14] & ATTR_PG[4-14]) == ATTR_PG[4-14])
98+
99+
// Call p9_mem_startclocks_fence_setup_function for Mc chiplets
100+
p9_mem_startclocks_fence_setup_function():
101+
// Hostboot does it based on pg_vector. I have no idea what exactly pg_vector represents. I also don't know if it is
102+
// possible to have a functional MCBIST for which we don't want to drop the fence (functional MCBIST with nonfunctional
103+
// PERV?). In any case, further code tries to configure all functional MCBISTs, so perhaps it is better to drop more
104+
// fences than necessary than to forget about one.
105+
> if ((MC.ATTR_CHIP_UNIT_POS == 0x07 && pg_vector[5]) ||
106+
> (MC.ATTR_CHIP_UNIT_POS == 0x08 && pg_vector[3]))
107+
> {
108+
// Drop chiplet fence
109+
TP.TPCHIP.NET.PCBSLMC01.NET_CTRL0 (WAND) // 0x070F0041
110+
[all] 1
111+
[18] FENCE_EN = 0
112+
> }
113+
114+
// Call p9_mem_startclocks_flushmode for Mc chiplets
115+
p9_mem_startclocks_flushmode():
116+
// Clear flush_inhibit to go in to flush mode
117+
TP.TCMC01.MCSLOW.CPLT_CTRL0 (WO_CLEAR) // 0x07000020
118+
[all] 0
119+
[2] CTRL_CC_FLUSHMODE_INH_DC = 1
120+
121+
// Call p9_sbe_common_configure_chiplet_FIR for MC chiplets
122+
p9_sbe_common_configure_chiplet_FIR():
123+
// reset pervasive FIR
124+
TP.TCMC01.MCSLOW.LOCAL_FIR // 0x0704000A
125+
[all] 0
126+
127+
// configure pervasive FIR action/mask
128+
TP.TCMC01.MCSLOW.LOCAL_FIR_ACTION0 // 0x07040010
129+
[all] 0
130+
TP.TCMC01.MCSLOW.LOCAL_FIR_ACTION1 // 0x07040011
131+
[all] 0
132+
[0-3] 0xF
133+
TP.TCMC01.MCSLOW.LOCAL_FIR_MASK // 0x0704000D
134+
// 0x0FFFFFFFFFC00000
135+
[all] 0
136+
[4-41] 0x3FFFFFFFFF
137+
138+
// reset XFIR
139+
TP.TCMC01.MCSLOW.XFIR // 0x07040000
140+
[all] 0
141+
142+
// configure XFIR mask
143+
TP.TCMC01.MCSLOW.FIR_MASK // 0x07040002
144+
[all] 0
145+
146+
// Reset FBC chiplet configuration
147+
TP.TCMC01.MCSLOW.CPLT_CONF0 // 0x07000008
148+
[48-51] TC_UNIT_GROUP_ID_DC = ATTR_PROC_FABRIC_GROUP_ID // Where do these come from?
149+
[52-54] TC_UNIT_CHIP_ID_DC = ATTR_PROC_FABRIC_CHIP_ID
150+
[56-60] TC_UNIT_SYS_ID_DC = ATTR_PROC_FABRIC_SYSTEM_ID
151+
152+
// Add to Multicast Group
153+
// avoid setting if register is already set, i.e. != p9SbeChipletReset::MCGR_CNFG_SETTING_EMPTY
154+
TP.TPCHIP.NET.PCBSLMC01.MULTICAST_GROUP_1 // 0x070F0001
155+
[3-5] MULTICAST1_GROUP: if 7 then set to 0
156+
[16-23] (not described): if [3-5] == 7 then set to 0x1C // No clue why Hostboot modifies these bits
157+
TP.TPCHIP.NET.PCBSLMC01.MULTICAST_GROUP_2 // 0x070F0002
158+
[3-5] MULTICAST1_GROUP: if 7 then set to 2
159+
[16-23] (not described): if [3-5] == 7 then set to 0x1C
160+
```

‎devnotes/isteps/13_dram_training/07_host_enable_memvolt.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## host_enable_memvolt: Enable the VDDR3 Voltage Rail (13.7)
2+
3+
> a) Bring power to dram rails VDDR and VPP. VPP must be enabled prior to VDDR
4+
> - BMC based systems - this is a no-op
5+
> - Send message to FSP to turn on voltages
6+
> - Message must have accounted for voltage/current tweaking based on number of plugged dimms (Dynamic VID)
7+
> - Pulled from HWPF attributes per voltage rail
8+
> - FSP
9+
> - Trigger voltage ramp to DPSS via I2C
10+
> - Wait for min 200 ms ramp, must be stable 500us after DPSS claims Pgood
11+
> - Wait for ack message from FSP - confirms that voltage is on and ready

0 commit comments

Comments
 (0)