Ubuntu 20.04 crashes: An ECC error or L2 poison was detected

Ubuntu 20.04 crashes randomly at different times. Unable to point to a specific event.

uname -a 
Linux ubuntu 5.11.0-051100-generic #202102142330 
SMP Sun Feb 14 23:33:21 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Crashes with the following signal:

 kernel:[19849.215258] [Hardware Error]: Uncorrected, software restartable error.

 kernel:[19849.215259] [Hardware Error]: CPU:22 (19:21:0) MC0_STATUS[-|UE|MiscV|AddrV|-|-|-|-|Poison|-]: 0xbc00080001010135

 kernel:[19849.215263] [Hardware Error]: Error Addr: 0x000000076bed1c00

 kernel:[19849.215264] [Hardware Error]: IPID: 0x001000b000000000

 kernel:[19849.215266] [Hardware Error]: Load Store Unit Ext. Error Code: 1, An ECC error or L2 poison was detected on a data cache read by a load.

 kernel:[19849.215269] [Hardware Error]: cache level: L1, tx: DATA, mem-tx: DRD

Hardware info:

### CPU
  Architecture:                    x86_64
  CPU op-mode(s):                  32-bit, 64-bit
  Byte Order:                      Little Endian
  Address sizes:                   48 bits physical, 48 bits virtual
  CPU(s):                          24
  On-line CPU(s) list:             0-23
  Thread(s) per core:              2
  Core(s) per socket:              12
  Socket(s):                       1
  NUMA node(s):                    1
  Vendor ID:                       AuthenticAMD
  CPU family:                      25
  Model:                           33
  Model name:                      AMD Ryzen 9 5900X 12-Core Processor
  Stepping:                        0
  Frequency boost:                 enabled
  CPU MHz:                         2200.000
  CPU max MHz:                     6442.4800
  CPU min MHz:                     2200.0000

### Base Board Information
  Manufacturer: ASRock
  Product Name: X570 Taichi

### Memory:
G Skill Trident Z Neo DDR4 - 3600Mhz 32GB (2 x 16GB)

What are the suggested ways of finding out the root cause? How do I enable more logging or if the log already exists where can I find them etc. Any guidance will be appreciated. Thanks!

This isn't technically an answer, but ...

The ECC error or L2 poison was detected on a data cache read by a load message points to a memory problem, either with the RAM itself or the cache on the CPU. Neither are great, but you can test the system RAM with the following process:

1. Restart your system
2. Press and hold the Shift key to bring up the GRUB menu
3. Select "Ubuntu, memtest86+" and press Enter
   The memory test will run until the end of time or until you press the Esc key. Let the machine complete at least one test before escaping.

Based on reports around the web, this issue seems to be seen only with the higher-end AMD Ryzen processors. Reading through this long thread on AMD's community site revealed this interesting bit:

I replaced memory and the computer has been rock solid now for a few days. Hopefully this helps you out as it helped me out. Previous memory was Gskill 3600mhz memory... new memory is 3200 memory from Corsair.

Your question does not state what sort of memory you have installed but, if it's a higher-frequency set of modules, there may be something between the RAM and the CPU that is causing an instability. If the memory test fails and you happen to have some compatible 3200MHz RAM available (even if it's just one DIMM), consider swapping it out and performing the memory test again.

BIOS

ASRock X570 Taichi

The BIOS is current at version P4.30.

MEMORY

G Skill Trident Z Neo DDR4 - 3600Mhz 32GB (2 x 16GB), product: F4-3600C16-16GTZNC

AMD Ryzen 9 5900X 12-Core Processor

Ryzen processors are very fussy about RAM.

These DIMMs don't appear on the memory supported list, as seen here.

memtest passed all tests.

When we look at sudo lshw -C memory we see that the DIMMs may be installed into incorrect slot locations. When using 2 equal size DIMMs, they should be installed into slots A2 and B2. Here's an image of the board layout, and the memory slots... taken from the User Manual at here... so just verify this...

Based on the suggestion from @heynnema, I was able to find out that the model of DIMMs installed on my computer is not listed in their compatibility list. Here's the steps followed:

1. Visit the CPU Supported list of the ASRock x570 Taichi Website. Find out the core type. In my case it was Vermeer
2. Find out the model of the DIMMs installed on the system by running sudo lshw -C memory (it was F4-3600C16-16GTZNC)
3. Navigate to the Memory Supported List for Vermeer and see if it is supported. Unfortunately its not in the list! perhaps that is the cause for the inconsistent crashes. I will try out a supported version of DIMMs to see if the crashes occur again and update this answer accordingly.

 *-firmware
       description: BIOS
       vendor: American Megatrends Inc.
       physical id: 0
       version: P4.30
       date: 04/14/2021
       size: 64KiB
       capacity: 16MiB
       capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
  *-memory
       description: System Memory
       physical id: e
       slot: System board or motherboard
       size: 32GiB
     *-bank:0
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: F4-3600C16-16GTZNC
          vendor: Unknown
          physical id: 0
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
     *-bank:1
          description: Project-Id-Version: lshwReport-Msgid-Bugs-To: FULL NAME <EMAIL@ADDRESS>PO-Revision-Date: 2012-02-02 13:04+0000Last-Translator: Joel Addison <[email protected]>Language-Team: English (Australia) <[email protected]>MIME-Version: 1.0Content-Type: text/plain; charset=UTF-8Content-Transfer-Encoding: 8bitX-Launchpad-Export-Date: 2021-01-21 18:43+0000X-Generator: Launchpad (build 2d1d5e352f0d063d660df2300e31f66bed027fa5)Project-Id-Version: lshwReport-Msgid-Bugs-To: FULL NAME <EMAIL@ADDRESS>PO-Revision-Date: 2012-02-02 13:04+0000Last-Translator: Joel Addison <[email protected]>Language-Team: English (Australia) <[email protected]>MIME-Version: 1.0Content-Type: text/plain; charset=UTF-8Content-Transfer-Encoding: 8bitX-Launchpad-Export-Date: 2021-01-21 18:43+0000X-Generator: Launchpad (build 2d1d5e352f0d063d660df2300e31f66bed027fa5) [empty]
          product: Unknown
          vendor: Unknown
          physical id: 1
          serial: Unknown
          slot: DIMM 1
     *-bank:2
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: F4-3600C16-16GTZNC
          vendor: Unknown
          physical id: 2
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
     *-bank:3
          description: Project-Id-Version: lshwReport-Msgid-Bugs-To: FULL NAME <EMAIL@ADDRESS>PO-Revision-Date: 2012-02-02 13:04+0000Last-Translator: Joel Addison <[email protected]>Language-Team: English (Australia) <[email protected]>MIME-Version: 1.0Content-Type: text/plain; charset=UTF-8Content-Transfer-Encoding: 8bitX-Launchpad-Export-Date: 2021-01-21 18:43+0000X-Generator: Launchpad (build 2d1d5e352f0d063d660df2300e31f66bed027fa5)Project-Id-Version: lshwReport-Msgid-Bugs-To: FULL NAME <EMAIL@ADDRESS>PO-Revision-Date: 2012-02-02 13:04+0000Last-Translator: Joel Addison <[email protected]>Language-Team: English (Australia) <[email protected]>MIME-Version: 1.0Content-Type: text/plain; charset=UTF-8Content-Transfer-Encoding: 8bitX-Launchpad-Export-Date: 2021-01-21 18:43+0000X-Generator: Launchpad (build 2d1d5e352f0d063d660df2300e31f66bed027fa5) [empty]
          product: Unknown
          vendor: Unknown
          physical id: 3
          serial: Unknown
          slot: DIMM 1
  *-cache:0
       description: L1 cache
       physical id: 11
       slot: L1 - Cache
       size: 768KiB
       capacity: 768KiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=1
  *-cache:1
       description: L2 cache
       physical id: 12
       slot: L2 - Cache
       size: 6MiB
       capacity: 6MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=2
  *-cache:2
       description: L3 cache
       physical id: 13
       slot: L3 - Cache
       size: 64MiB
       capacity: 64MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=3