Improve nvme_device_critical_warnings_state alert when drive's warranty from the manufacturer is over

[andy108369]

Hi,

I've noticed this alert:

nvme_device_critical_warnings_state

Instance: nvme.device_nvme1n1_critical_warnings_state
Desc: NVMe device nvme1n1 has critical warnings

DB LOOKUP:
max of all values of chart nvme.device_nvme1n1_critical_warnings_state, starting 30 seconds ago and up to now, with options unaligned

On the server:

root@node1:~# smartctl --all /dev/nvme1
...
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
- NVM subsystem reliability has been degraded

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x04
Temperature:                        39 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    133%
Data Units Read:                    1,049,223,056 [537 TB]
Data Units Written:                 1,594,868,738 [816 TB]
Host Read Commands:                 23,625,989,252
Host Write Commands:                6,223,712,056
Controller Busy Time:               69,256
Power Cycles:                       17
Power On Hours:                     7,241
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               39 Celsius
Temperature Sensor 2:               45 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

However, Hetzner (DC owner) said it's false alarm:

The message 'Critical Warning: 0x04' is caused by "Percentage Used" being above 100%. This only means that the drive's warranty from the manufacturer is over. But as long as 'Available Spare' is greater than 'Available Spare Threshold', you can safely ignore this message.

My question is how do I address the alert, considering Hetzner's response that it's a false alarm due to the "Percentage Used" exceeding 100% indicating the drive's warranty period has ended?

Can I suppress this error either in SMART (smartctl) or the Netdata Agent for my case?

Also, I don't want to disable it entirely since there could be a different critical alert/warning come related to the disk malfunction or similar... I'd like to only disable it for this particular warranty-related case or improve Netdata Agent's judgment in its regards.

[ilyam8]

@andy108369 hey. Can you share

sudo nvme smart-log /dev/nvme1

And I see you have

SMART overall-health self-assessment test result: FAILED!
- NVM subsystem reliability has been degraded

which is likely the reason for the alarm and not the percentage used.

This is the nvme.device_critical_warnings_state chart description (formatted for better readability):

Critical warnings for the status of the controller. Status active if set to 1.

- AvailableSpare - the available spare capacity is below the threshold. 
- TempThreshold - the composite temperature is greater than or equal to an over temperature threshold or less than or equal to an under temperature threshold. 
- NvmSubsystemReliability - the NVM subsystem reliability is degraded due to excessive media or internal errors. 
- ReadOnly - media is placed in read-only mode. 
- VolatileMemBackupFailed - the volatile memory backup device has failed. 
- PersistentMemoryReadOnly - the Persistent Memory Region has become read-only or unreliable.

You can change "group by" to "dimension" + "label:device" to find out which critical warning status is active for your device.

[andy108369]

Hey @ilyam8 👋

• NVM subsystem reliability has been degraded

I've also noticed that.. will check with Hetzner.

Can you share
sudo nvme smart-log /dev/nvme1

root@node1:~# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning			: 0x4
temperature				: 38 C (311 Kelvin)
available_spare				: 100%
available_spare_threshold		: 10%
percentage_used				: 133%
endurance group critical warning summary: 0x4
data_units_read				: 1049223676
data_units_written			: 1594997390
host_read_commands			: 23626001802
host_write_commands			: 6224436225
controller_busy_time			: 69269
power_cycles				: 17
power_on_hours				: 7242
unsafe_shutdowns			: 3
media_errors				: 0
num_err_log_entries			: 0
Warning Temperature Time		: 0
Critical Composite Temperature Time	: 0
Temperature Sensor 1           : 38 C (311 Kelvin)
Temperature Sensor 2           : 45 C (318 Kelvin)
Thermal Management T1 Trans Count	: 0
Thermal Management T2 Trans Count	: 0
Thermal Management T1 Total Time	: 0
Thermal Management T2 Total Time	: 0

You can change "group by" to "dimension" + "label:device" to find out which critical warning status is active for your device.

Thanks! I'll give it a shot next time I see this alert.

For some reason it's gone (saying No data) which I find strange since smartctl still reports it. I was also reinstalling the netdata agent ~2 hours ago. I'm wondering if it can be bounced in some way to get re-triggered by the Netdata Agent / web UI?

Thanks for your prompt response!

[Kofl]

Same issue, available spare is fine, seems related to that kernel bug report filling up the smart error log, which is not related to NvmSubsystemReliability:
https://bugzilla.kernel.org/show_bug.cgi?id=217445

Is there any way to change that behaviour, as the NVMe seems fine and it will mask really upcoming issues?

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!

• NVM subsystem reliability has been degraded

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x04
Temperature: 30 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 167%
Data Units Read: 2,669,542,568 [1.36 PB]
Data Units Written: 681,023,034 [348 TB]
Host Read Commands: 40,023,172,407
Host Write Commands: 10,036,858,394
Controller Busy Time: 3,194,702,995
Power Cycles: 14
Power On Hours: 34,791
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 95
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 30 Celsius
Temperature Sensor 2: 30 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS
0 95 0 0x901d 0x4004 - 0 1 -
1 94 0 0x700a 0x4004 - 0 1 -
2 93 0 0x3008 0x4004 - 0 1 -
3 92 0 0x2007 0x4004 - 0 1 -
4 91 0 0x9008 0x4004 - 0 1 -
5 90 0 0x0010 0x4004 - 0 0 -

root@h1pve:/etc/netdata# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning : 0x4
temperature : 31°C (304 Kelvin)
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 167%
endurance group critical warning summary: 0
Data Units Read : 2,669,542,569 (1.37 PB)
Data Units Written : 681,023,501 (348.68 TB)
host_read_commands : 40,023,172,502
host_write_commands : 10,036,873,564
controller_busy_time : 3,194,702,995
power_cycles : 14
power_on_hours : 34,791
unsafe_shutdowns : 1
media_errors : 0
num_err_log_entries : 96
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 31°C (304 Kelvin)
Temperature Sensor 2 : 31°C (304 Kelvin)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0

root@h1pve:/etc/netdata# nvme error-log /dev/nvme0n1 |head -n 30 Error Log Entries for device:nvme0n1 entries:64 ................. Entry[ 0] ................. error_count : 95 sqid : 0 cmdid : 0xd00c status_field : 0x2002(Invalid Field in Command: A reserved coded value or an unsupported value in a defined field) phase_tag : 0 parm_err_loc : 0xffff lba : 0 nsid : 0x1 vs : 0 trtype : The transport type is not indicated or the error is not transport related. cs : 0 trtype_spec_info: 0 ................. Entry[ 1] ................. error_count : 94 sqid : 0 cmdid : 0x700e status_field : 0x2002(Invalid Field in Command: A reserved coded value or an unsupported value in a defined field) phase_tag : 0 parm_err_loc : 0xffff lba : 0 nsid : 0x1 vs : 0 trtype : The transport type is not indicated or the error is not transport related.