HLD for diagnostic monitoring of CMIS based transceivers #1828

mihirpat1 · 2024-10-11T06:26:42Z

This HLD while provide an overview of how SONiC reads and stores the various diagnostic parameters read from a CMIS based transceiver.

Proposed changes for existing tables in SONiC

Removed CCMIS specific VDM data from DOM related tables and moved to VDM related tables
Not all VDM thresholds are currently shown.
With the proposed changes, I am planning to change the name of existing fields to be in line with the CMIS spec.
Moved txfault, txlos, txcdrlol, rxlos and rxcdrlol from TRANSCEIVER_STATUS to TRANSCEIVER_DOM_FLAG table.

PR title	state	context
Move DomInfoUpdateTask class to a separate file
Add VDM and Status related cmis fields for onboarding xcvr diagnostic features
Add diagnostic monitoring APIs for DOM, VDM and status access for CMIS modules
[xcvrd] Enable periodic polling of VDM relevant data
Create is_transceiver_vdm_supported API for CMIS transceivers

Signed-off-by: Mihir Patel <[email protected]>

mihirpat1 · 2024-10-11T06:27:35Z

@qinchuanares - It will be great if you can help in reviewing this PR.

mihirpat1 · 2024-10-11T15:45:05Z

@Junchao-Mellanox @keboliu - It will be great if you can help in reviewing this PR.

qinchuanares · 2024-10-11T16:53:59Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+
+### 4.2 Dynamic Diagnostic Information
+
+The `DomInfoUpdateTask` thread is responsible for updating the dynamic diagnostic information for all the transceivers in the system. The `DomInfoUpdateTask` thread is triggered by a timer (`DOM_INFO_UPDATE_PERIOD_SECS`), which is set to 60 seconds by default. The `DomInfoUpdateTask` thread reads the diagnostic information from the transceiver and updates the relevant tables in `redis-db`.


Curious if there is any plan for making this alarm logging interrupt based?

@prgeor Can you please help in clarifying this?

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

prgeor

@mihirpat1 I don't see how the timestamp, count for each alarm will be updated so that these times are as closer to the actual reporting time by the module

Junchao-Mellanox · 2024-10-14T01:50:52Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+    3. Read the transceiver firmware information from the module and update the `TRANSCEIVER_FIRMWARE_INFO` table.
+    4. Read the transceiver DOM sensor data from the module and update the `TRANSCEIVER_DOM_SENSOR` table.
+    5. Read the transceiver DOM flag data from the module, record the timestamp, and update the `TRANSCEIVER_DOM_FLAG` table.
+    6. Analyze the transceiver DOM flag data by comparing the current flag data with the previous flag data and update the `TRANSCEIVER_DOM_FLAG_CHANGE_COUNT`, `TRANSCEIVER_DOM_FLAG_TIME_SET`, and `TRANSCEIVER_DOM_FLAG_TIME_CLEAR` tables.


should we also record the flag set/clear event to syslog?

@Junchao-Mellanox Sure, will plan to record the set/clear event to syslog as a debug. Making it as a notice will make the syslog chatty.

Junchao-Mellanox · 2024-10-14T01:53:47Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+    6. Analyze the transceiver DOM flag data by comparing the current flag data with the previous flag data and update the `TRANSCEIVER_DOM_FLAG_CHANGE_COUNT`, `TRANSCEIVER_DOM_FLAG_TIME_SET`, and `TRANSCEIVER_DOM_FLAG_TIME_CLEAR` tables.
+    7. Read the transceiver status data from the module and update the `TRANSCEIVER_STATUS` table.
+    8. If the transceiver supports VDM monitoring, perform the following steps:
+        1. Freeze the statistics by calling the CMIS API and wait for `FreezeDone`. Once the statistics are frozen, record the timestamp and copy the VDM and PM statistics from the transceiver.


Could you please specify which API will be used to "freeze" the statistic? I guess it will write something to module EEPROM, and it is only applicable when module is under software control.

@Junchao-Mellanox I have now updated the API name now. Can you please elaborate on what do you mean by "it is only applicable when module is under software control"?

Junchao-Mellanox · 2024-10-14T02:57:30Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+-----------  ---------------  --------     --------       --------      --------
+
+Example:
+admin@sonic#show interfaces transceiver vdm flag Ethernet1


is it possible to add a filter to get abnormal status only?

@prgeor do we want to add filter as part of current enhancement?

@Junchao-Mellanox We are now displaying the output after applying the filter. A --detail option has been added to allow user to dump data for all lanes irrespective of the flag being set.

Junchao-Mellanox · 2024-10-14T03:01:02Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+**Tables Used for Flag Analysis:**
+
+- `TRANSCEIVER_VDM_FLAG`: This table stores flags indicating the status of various VDM parameters.
+- `TRANSCEIVER_VDM_FLAG_CHANGE_COUNT`: This table keeps a count of how many times each VDM flag has changed.


If xcvrd stopped, will the change count be reset to 0? The same question to time set and time clear. Also, should we record the change count to syslog?

@Junchao-Mellanox Yes - the change count and the set/clear time in redis-db tables will be removed when xcvrd is stopped. We can log the change count to syslog before deleting the table.

mihirpat1 · 2024-10-16T03:06:28Z

@mihirpat1 I don't see how the timestamp, count for each alarm will be updated so that these times are as closer to the actual reporting time by the module

@prgeor I have now added Diagnostic Information Update During Link Down Event section to address this scenario.

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

mihirpat1 · 2024-10-14T18:37:14Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+    3. Read the transceiver firmware information from the module and update the `TRANSCEIVER_FIRMWARE_INFO` table.
+    4. Read the transceiver DOM sensor data from the module and update the `TRANSCEIVER_DOM_SENSOR` table.
+    5. Read the transceiver DOM flag data from the module, record the timestamp, and update the `TRANSCEIVER_DOM_FLAG` table.
+    6. Analyze the transceiver DOM flag data by comparing the current flag data with the previous flag data and update the `TRANSCEIVER_DOM_FLAG_CHANGE_COUNT`, `TRANSCEIVER_DOM_FLAG_TIME_SET`, and `TRANSCEIVER_DOM_FLAG_TIME_CLEAR` tables.


@Junchao-Mellanox Sure, will plan to record the set/clear event to syslog as a debug. Making it as a notice will make the syslog chatty.

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

mihirpat1 · 2024-10-15T23:44:05Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+-----------  ---------------  --------     --------       --------      --------
+
+Example:
+admin@sonic#show interfaces transceiver vdm flag Ethernet1


@prgeor do we want to add filter as part of current enhancement?

mihirpat1 · 2024-10-16T00:18:18Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+    6. Analyze the transceiver DOM flag data by comparing the current flag data with the previous flag data and update the `TRANSCEIVER_DOM_FLAG_CHANGE_COUNT`, `TRANSCEIVER_DOM_FLAG_TIME_SET`, and `TRANSCEIVER_DOM_FLAG_TIME_CLEAR` tables.
+    7. Read the transceiver status data from the module and update the `TRANSCEIVER_STATUS` table.
+    8. If the transceiver supports VDM monitoring, perform the following steps:
+        1. Freeze the statistics by calling the CMIS API and wait for `FreezeDone`. Once the statistics are frozen, record the timestamp and copy the VDM and PM statistics from the transceiver.


@Junchao-Mellanox I have now updated the API name now. Can you please elaborate on what do you mean by "it is only applicable when module is under software control"?

mihirpat1 · 2024-10-16T01:00:36Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+**Tables Used for Flag Analysis:**
+
+- `TRANSCEIVER_VDM_FLAG`: This table stores flags indicating the status of various VDM parameters.
+- `TRANSCEIVER_VDM_FLAG_CHANGE_COUNT`: This table keeps a count of how many times each VDM flag has changed.


@Junchao-Mellanox Yes - the change count and the set/clear time in redis-db tables will be removed when xcvrd is stopped. We can log the change count to syslog before deleting the table.

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

qinchuanares · 2024-10-16T17:32:10Z

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

+    ; Defines Transceiver PM information for a port
+    key                          = TRANSCEIVER_PM|ifname            ; information of PM on port
+    ; field                      = value 
+    prefec_ber_avg               = FLOAT                            ; prefec ber avg


we could expand this to media and host prefec BER and uncorr frames, even if this is C-CMIS only.

Media BER and UC frames info comes from page 34h, host BER and UC frames come from page 3Ah

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

prgeor

@mihirpat1 can you add a requirement section detailing what are the goals we need to meet and what are the NON-goals or future work

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md

mssonicbld · 2025-02-05T23:54:02Z

/azp run

azure-pipelines · 2025-02-05T23:54:08Z

No pipelines are associated with this pull request.

…logic

mssonicbld · 2025-02-06T20:18:00Z

/azp run

azure-pipelines · 2025-02-06T20:18:07Z

No pipelines are associated with this pull request.

mihirpat1 · 2025-02-07T00:44:55Z

@mihirpat1 Capture the FreezeDone and UnfreezeDone time value defined by SONiC

mssonicbld · 2025-02-07T05:09:31Z

/azp run

azure-pipelines · 2025-02-07T05:09:37Z

No pipelines are associated with this pull request.

…nge detection

mssonicbld · 2025-02-07T20:27:46Z

/azp run

azure-pipelines · 2025-02-07T20:27:52Z

No pipelines are associated with this pull request.

mssonicbld · 2025-02-09T16:52:21Z

/azp run

azure-pipelines · 2025-02-09T16:52:26Z

No pipelines are associated with this pull request.

…INFO table

mssonicbld · 2025-02-11T07:00:00Z

/azp run

azure-pipelines · 2025-02-11T07:01:33Z

No pipelines are associated with this pull request.

HLD for diagnostic monitoring of CMIS based transceivers

d9b409e

Signed-off-by: Mihir Patel <[email protected]>

mihirpat1 requested a review from prgeor October 11, 2024 06:26

mihirpat1 requested review from keboliu and Junchao-Mellanox October 11, 2024 15:44

qinchuanares reviewed Oct 11, 2024

View reviewed changes

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Outdated Show resolved Hide resolved

prgeor reviewed Oct 13, 2024

View reviewed changes

Junchao-Mellanox reviewed Oct 14, 2024

View reviewed changes

Addressed review comments

4efaeff

mihirpat1 commented Oct 16, 2024

View reviewed changes

qinchuanares reviewed Oct 16, 2024

View reviewed changes

doc/platform_api/CMIS_Diagnostic_Monitoring_Overview_in_SONiC.md Show resolved Hide resolved