[SmartSwitch] Enhance PCIe device check to skip the warning log, if device is in detaching mode (#546)

vvolam · oleksandrivantsiv · gpunathilell · web-flow · commit 7a0813a0d110 · 2025-02-05T21:10:45.000-08:00
* Skip logging the warning, if device is in detaching mode * Add detach_info table and unittests * Fix unit tests * Increase code coverage * Remove unused header import * Fix dict get values * Increase code coverage * Increase test coverage * [SmartSwitch] Extend implementation of the DPU chassis daemon. (#563) * Addition of DPU Chassis for thermalctld (#564) * [stormond] Added new dynamic field 'last_sync_time' to STATE_DB (#535) * Added new dynamic field 'last_sync_time' that shows when STORAGE_INFO for disk was last synced to STATE_DB * Moved 'start' message to actual starting point of the daemon * Added functions for formatted and epoch time for user friendly time display * Made changes per prgeor review comments * Pivot to SysLogger for all logging * Increased log level so that they are seen in syslogs * Code coverage improvement * [lag_id] Add lagid to free_list when LC absent for 30 minutes (#542) When LC is absent for 30 minutes, the database cleanup kicks in. When LagId is released, it needs to be appended to the SYSTEM_LAG_IDS_FREE_LIST This PR works with the following 2 PRs: sonic-net/sonic-swss#3303 sonic-net/sonic-buildimage#20369 Signed-off-by: mlok <marty.lok@nokia.com> * Fixed bug in chassisd causing incorrect number of ASICs in CHASSIS_STATE_DB (#560) Fixed the bug in chassisd due to which incorrect number of ASICs were being pushed to CHASSIS_STATE_DB. * thermalctld: Add support for fans on non-CPU modules (#555) * thermalctld: Add support for fans on non-CPU modules * Add module fan to unit tests * Advanced Azure pipeline to Bookworm (#572) Description This PR advances the azure pipeline on sonic_platform_daemons from bullseye to bookworm. This fixes the issue where sonic-platform-daemons azp is having some issues due to upgrade to bookworm. See Pipelines - Run 20241210.8 logs for details. * Take non-CMIS xcvrs out of lpmode in SFF Manager (#565) Description Fix non-CMIS transceivers in down state by bringing them out of low power mode in the SFF Manager Task. This is intended to work together with the change in sonic-net/sonic-buildimage#20886. Motivation and Context Non-CMIS transceivers were not functioning correctly when put into Low Power mode. So XCVRD now brings them out of lpmode. How Has This Been Tested? Loaded an image containing this change alongside the change from sonic-net/sonic-buildimage#20886 on an Arista chassis containing a Clearwater2 linecard. Verified that without this image some interfaces were in a down state but with the image all interfaces came up as expected. * Added SmartSwitch support in chassisd and enabling chassisd (#467) Added SmartSwitch support in chassisd and enabling chassisd * [chassis][psud] Move the PSU parent information generation to the loop run function from the initialization function (#576) Description Move the PSU parent information generation to the loop run function from the initialization function Motivation and Context Fixes #575 How Has This Been Tested? Tested on Cisco chassis, the PHYSICAL_ENTITY_INFO|PSU * can be re-inserted after thermalctld restart. And monitored the stated db for memory for hours, works well: * [chassisd] Address the chassisd crash issue and add UT for it (#573) Description On Nokia platform, slot name of Supervisor is string "A" instead of a number. Using "int" to convert it could cause issue backtrace. We should use slot value to any checking without any conversion. This will fixes sonic-net/sonic-buildimage#21131 Motivation and Context Modify the _get_module_info not to convert "slot" to a string value. And also modify the code not to convert slot value to an to do any checking. Just directly use the returned value of get_slot(). Also add UT test_moduleupdater_check_slot_string() to valid it. How Has This Been Tested? Tested on 202405 branch Signed-off-by: mlok <marty.lok@nokia.com> * Fix a comment --------- Signed-off-by: mlok <marty.lok@nokia.com> Co-authored-by: Oleksandr Ivantsiv <oivantsiv@nvidia.com> Co-authored-by: Gagan Punathil Ellath <gpunathilell@nvidia.com> Co-authored-by: Ashwin Srinivasan <93744978+assrinivasan@users.noreply.github.com> Co-authored-by: Marty Y. Lok <76118573+mlok-nokia@users.noreply.github.com> Co-authored-by: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Co-authored-by: Patrick MacArthur <pmacarthur@arista.com> Co-authored-by: Peter Bailey <peterbailey@arista.com> Co-authored-by: rameshraghupathy <43161235+rameshraghupathy@users.noreply.github.com> Co-authored-by: Jianquan Ye <jianquanye@microsoft.com>
diff --git a/sonic-pcied/scripts/pcied b/sonic-pcied/scripts/pcied
@@ -27,6 +27,10 @@ SYSLOG_IDENTIFIER = "pcied"
 PCIE_RESULT_REGEX = "PCIe Device Checking All Test"
 PCIE_DEVICE_TABLE_NAME = "PCIE_DEVICE"
 PCIE_STATUS_TABLE_NAME = "PCIE_DEVICES"
+PCIE_DETACH_INFO_TABLE = "PCIE_DETACH_INFO"
+
+PCIE_DETACH_BUS_INFO_FIELD = "bus_info"
+PCIE_DETACH_DPU_STATE_FIELD = "dpu_state"
 
 PCIED_MAIN_THREAD_SLEEP_SECS = 60
 
@@ -92,6 +96,7 @@ class DaemonPcied(daemon_base.DaemonBase):
         self.state_db = daemon_base.db_connect("STATE_DB")
         self.device_table = swsscommon.Table(self.state_db, PCIE_DEVICE_TABLE_NAME)
         self.status_table = swsscommon.Table(self.state_db, PCIE_STATUS_TABLE_NAME)
+        self.detach_info = swsscommon.Table(self.state_db, PCIE_DETACH_INFO_TABLE)
 
     def __del__(self):
         if self.device_table:
@@ -102,6 +107,10 @@ class DaemonPcied(daemon_base.DaemonBase):
             stable_keys = self.status_table.getKeys()
             for stk in stable_keys:
                 self.status_table._del(stk)
+        if self.detach_info:
+            detach_info_keys = self.detach_info.getKeys()
+            for dk in detach_info_keys:
+                self.detach_info._del(dk)
 
     # load aer-fields into statedb
     def update_aer_to_statedb(self):
@@ -151,6 +160,28 @@ class DaemonPcied(daemon_base.DaemonBase):
 
         self.status_table.set("status", fvs)
 
+    # Check if any PCI interface is in detaching mode by querying the state_db
+    def is_dpu_in_detaching_mode(self, pcie_dev):
+        # Ensure detach_info is not None
+        if self.detach_info is None:
+            self.log_debug("detach_info is None")
+            return False
+
+        # Query the state_db for the device detaching status
+        detach_info_keys = list(self.detach_info.getKeys())
+        if not detach_info_keys:
+            return False
+
+        for key in detach_info_keys:
+            dpu_info = self.detach_info.get(key)
+            if dpu_info:
+                bus_info = dpu_info.get(PCIE_DETACH_BUS_INFO_FIELD)
+                dpu_state = dpu_info.get(PCIE_DETACH_DPU_STATE_FIELD)
+                if bus_info == pcie_dev and dpu_state == "detaching":
+                    return True
+
+        return False
+
     # Check the PCIe devices
     def check_pcie_devices(self):
         self.resultInfo = platform_pcieutil.get_pcie_check()
@@ -160,6 +191,14 @@ class DaemonPcied(daemon_base.DaemonBase):
 
         for result in self.resultInfo:
             if result["result"] == "Failed":
+                # Convert bus, device, and function to a bus_info format like "0000:03:00.0"
+                pcie_dev = "0000:{int(result['bus'], 16):02x}:{int(result['dev'], 16):02x}.{int(result['fn'], 16)}"
+
+                # Check if the device is in detaching mode
+                if device_info.is_smartswitch() and self.is_dpu_in_detaching_mode(pcie_dev):
+                    self.log_debug("PCIe Device: {} is in detaching mode, skipping warning.".format(pcie_dev))
+                    continue
+
                 self.log_warning("PCIe Device: " + result["name"] + " Not Found")
                 err += 1
             else:
diff --git a/sonic-pcied/tests/test_DaemonPcied.py b/sonic-pcied/tests/test_DaemonPcied.py
@@ -143,17 +143,86 @@ def test_run(self):
         daemon_pcied.run()
         assert daemon_pcied.check_pcie_devices.call_count == 1
 
+    @mock.patch('pcied.load_platform_pcieutil', mock.MagicMock())
+    def test_is_dpu_in_detaching_mode(self):
+        daemon_pcied = pcied.DaemonPcied(SYSLOG_IDENTIFIER)
+        daemon_pcied.detach_info = mock.MagicMock()
+        daemon_pcied.detach_info.getKeys = mock.MagicMock(return_value=['DPU_0', 'DPU_1'])
+        daemon_pcied.detach_info.get = mock.MagicMock(
+            side_effect=lambda key: {
+                'DPU_0': {'bus_info': '0000:03:00.1', 'dpu_state': 'detaching'},
+                'DPU_1': {'bus_info': '0000:03:00.2', 'dpu_state': 'attached'}
+            }.get(key, None)
+        )
+
+        # Test when the device is in detaching mode
+        assert daemon_pcied.is_dpu_in_detaching_mode('0000:03:00.1') == True
+
+        # Test when the device is not in detaching mode
+        assert daemon_pcied.is_dpu_in_detaching_mode('0000:03:00.2') == False
+
+        # Test when the device does not exist in detach_info
+        assert daemon_pcied.is_dpu_in_detaching_mode('0000:03:00.3') == False
+
+        # Test when detach_info is None
+        daemon_pcied.detach_info = None
+        assert daemon_pcied.is_dpu_in_detaching_mode('0000:03:00.1') == False
+
+        # Test when detach_info has no keys
+        daemon_pcied.detach_info = mock.MagicMock()
+        daemon_pcied.detach_info.getKeys.return_value = []
+        assert daemon_pcied.is_dpu_in_detaching_mode('0000:03:00.1') == False
+
+    @mock.patch('pcied.device_info.is_smartswitch', mock.MagicMock(return_value=False))
+    @mock.patch('pcied.DaemonPcied.is_dpu_in_detaching_mode', mock.MagicMock(return_value=False))
     @mock.patch('pcied.load_platform_pcieutil', mock.MagicMock())
     def test_check_pcie_devices(self):
         daemon_pcied = pcied.DaemonPcied(SYSLOG_IDENTIFIER)
         daemon_pcied.update_pcie_devices_status_db = mock.MagicMock()
         daemon_pcied.check_n_update_pcie_aer_stats = mock.MagicMock()
-        pcied.platform_pcieutil.get_pcie_check = mock.MagicMock()
+        pcied.platform_pcieutil.get_pcie_check = mock.MagicMock(
+            return_value=[
+                {"result": "Failed", "bus": "03", "dev": "00", "fn": "1", "name": "PCIe Device 1"},
+            ]
+        )
 
         daemon_pcied.check_pcie_devices()
         assert daemon_pcied.update_pcie_devices_status_db.call_count == 1
         assert daemon_pcied.check_n_update_pcie_aer_stats.call_count == 0
 
+    @mock.patch('pcied.device_info.is_smartswitch', mock.MagicMock(return_value=False))
+    @mock.patch('pcied.DaemonPcied.is_dpu_in_detaching_mode', mock.MagicMock(return_value=False))
+    @mock.patch('pcied.load_platform_pcieutil', mock.MagicMock())
+    def test_check_pcie_devices_update_aer(self):
+        daemon_pcied = pcied.DaemonPcied(SYSLOG_IDENTIFIER)
+        daemon_pcied.update_pcie_devices_status_db = mock.MagicMock()
+        daemon_pcied.check_n_update_pcie_aer_stats = mock.MagicMock()
+        pcied.platform_pcieutil.get_pcie_check = mock.MagicMock(
+            return_value=[
+                {"result": "Passed", "bus": "03", "dev": "00", "fn": "1", "name": "PCIe Device 1"},
+            ]
+        )
+
+        daemon_pcied.check_pcie_devices()
+        assert daemon_pcied.update_pcie_devices_status_db.call_count == 1
+        assert daemon_pcied.check_n_update_pcie_aer_stats.call_count == 1
+
+    @mock.patch('pcied.device_info.is_smartswitch', mock.MagicMock(return_value=True))
+    @mock.patch('pcied.DaemonPcied.is_dpu_in_detaching_mode', mock.MagicMock(return_value=True))
+    @mock.patch('pcied.load_platform_pcieutil', mock.MagicMock())
+    def test_check_pcie_devices_detaching(self):
+        daemon_pcied = pcied.DaemonPcied(SYSLOG_IDENTIFIER)
+        daemon_pcied.update_pcie_devices_status_db = mock.MagicMock()
+        daemon_pcied.check_n_update_pcie_aer_stats = mock.MagicMock()
+        pcied.platform_pcieutil.get_pcie_check = mock.MagicMock(
+            return_value=[
+                {"result": "Failed", "bus": "03", "dev": "00", "fn": "1", "name": "PCIe Device 1"},
+            ]
+        )
+
+        daemon_pcied.check_pcie_devices()
+        assert daemon_pcied.update_pcie_devices_status_db.call_count == 1
+        assert daemon_pcied.check_n_update_pcie_aer_stats.call_count == 0
 
     @mock.patch('pcied.load_platform_pcieutil', mock.MagicMock())
     def test_update_pcie_devices_status_db(self):
@@ -210,5 +279,5 @@ def test_update_aer_to_statedb(self):
              ])
         """
 
-        daemon_pcied.update_aer_to_statedb() 
+        daemon_pcied.update_aer_to_statedb()
         assert daemon_pcied.log_debug.call_count == 0
