Score:1

Black Screen, PCIe-Error with AMD GPU and server motherboard

cn flag

We have a workstation in our group that has a KNPA-U16 motherboard, and an AMD RX560 GPU which works most of the times. The Operating System is Kubuntu 20.04 and the kernel version 5.8.0-59.

Now the problem appears when we do not use the workstation for some time and than try to access it in front of it. Behaviour is as following: The screen appears for a short period of time (varies between ~0-10s) and than goes black. As said the time varies, I once was even able to login before the screen went dark. There is no possibility anymore to wake it up after that event. Its is reachable via ssh though.

Kernel Log shows the following:

09:27:51 PC3 kernel: [165861.461855] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4  
09:27:51 PC3 kernel: [165861.461858] {1}[Hardware Error]: event severity: info  
09:27:51 PC3 kernel: [165861.461860] {1}[Hardware Error]:  Error 0, type: fatal  
09:27:51 PC3 kernel: [165861.461861] {1}[Hardware Error]:  fru_text: PcieError  
09:27:51 PC3 kernel: [165861.461862] {1}[Hardware Error]:   section_type: PCIe error  
09:27:51 PC3 kernel: [165861.461863] {1}[Hardware Error]:   port_type: 4, root port  
09:27:51 PC3 kernel: [165861.461864] {1}[Hardware Error]:   version: 0.2  
09:27:51 PC3 kernel: [165861.461866] {1}[Hardware Error]:   command: 0x0407, status: 0x0010  
09:27:51 PC3 kernel: [165861.461867] {1}[Hardware Error]:   device_id: 0000:20:03.1  
09:27:51 PC3 kernel: [165861.461868] {1}[Hardware Error]:   slot: 7  
09:27:51 PC3 kernel: [165861.461868] {1}[Hardware Error]:   secondary_bus: 0x23  
09:27:51 PC3 kernel: [165861.461869] {1}[Hardware Error]:   vendor_id: 0x1022, device_id: 0x1453  
09:27:51 PC3 kernel: [165861.461870] {1}[Hardware Error]:   class_code: 060400  
09:27:51 PC3 kernel: [165861.461871] {1}[Hardware Error]:   bridge: secondary_status: 0x2000, control: 0x001a  
09:27:51 PC3 kernel: [165861.461872] {1}[Hardware Error]:   aer_uncor_status: 0x00000000, aer_uncor_mask: 0x04500000  
09:27:51 PC3 kernel: [165861.461873] {1}[Hardware Error]:   aer_uncor_severity: 0x004e2030  
09:27:51 PC3 kernel: [165861.461874] {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000  
09:27:51 PC3 kernel: [165861.461933] pcieport 0000:20:03.1: AER: aer_status: 0x00000000, aer_mask: 0x04500000  
09:27:51 PC3 kernel: [165861.461939] pcieport 0000:20:03.1: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID  
09:27:51 PC3 kernel: [165861.461941] pcieport 0000:20:03.1: AER: aer_uncor_severity: 0x004e2030  
09:27:51 PC3 kernel: [165861.461945] amdgpu 0000:23:00.0: AER: can't recover (no error_detected callback)  
09:27:51 PC3 kernel: [165861.461947] snd_hda_intel 0000:23:00.1: AER: can't recover (no error_detected callback)  
09:27:52 PC3 kernel: [165862.485806] pcieport 0000:20:03.1: AER: Root Port link has been reset  
09:27:52 PC3 kernel: [165862.485854] pcieport 0000:20:03.1: AER: device recovery successful   
09:28:02 PC3 kernel: [165866.837702] [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fences timed out!  
09:28:02 PC3 kernel: [165872.219438] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=841906, emitted seq=841908  
09:28:02 PC3 kernel: [165872.219526] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process sddm-greeter pid 88965 thread sddm-greet:cs0 pid 88969  
09:28:02 PC3 kernel: [165872.219534] amdgpu 0000:23:00.0: amdgpu: GPU reset begin!  
09:28:02 PC3 kernel: [165872.219865] amdgpu:   
09:28:02 PC3 kernel: [165872.219865]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219870] amdgpu:   
09:28:02 PC3 kernel: [165872.219870]  failed to send message 281 ret is 65535   
09:28:02 PC3 kernel: [165872.219879] amdgpu:   
09:28:02 PC3 kernel: [165872.219879]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219883] amdgpu:   
09:28:02 PC3 kernel: [165872.219883]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219887] amdgpu:   
09:28:02 PC3 kernel: [165872.219887]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219890] amdgpu:   
09:28:02 PC3 kernel: [165872.219890]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219894] amdgpu:   
09:28:02 PC3 kernel: [165872.219894]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219897] amdgpu:   
09:28:02 PC3 kernel: [165872.219897]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219901] amdgpu:   
09:28:02 PC3 kernel: [165872.219901]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219905] amdgpu:   
09:28:02 PC3 kernel: [165872.219905]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219909] amdgpu:   
09:28:02 PC3 kernel: [165872.219909]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219912] amdgpu:   
09:28:02 PC3 kernel: [165872.219912]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219916] amdgpu:   
09:28:02 PC3 kernel: [165872.219916]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219919] amdgpu:   
09:28:02 PC3 kernel: [165872.219919]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219923] amdgpu:   
09:28:02 PC3 kernel: [165872.219923]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219926] amdgpu:   
09:28:02 PC3 kernel: [165872.219926]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219930] amdgpu:   
09:28:02 PC3 kernel: [165872.219930]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219933] amdgpu:   
09:28:02 PC3 kernel: [165872.219933]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219937] amdgpu:   
09:28:02 PC3 kernel: [165872.219937]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219940] amdgpu:   
09:28:02 PC3 kernel: [165872.219940]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219944] amdgpu:   
09:28:02 PC3 kernel: [165872.219944]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219947] amdgpu:   
09:28:02 PC3 kernel: [165872.219947]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219951] amdgpu:   
09:28:02 PC3 kernel: [165872.219951]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219954] amdgpu:   
09:28:02 PC3 kernel: [165872.219954]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219958] amdgpu: 
09:28:02 PC3 kernel: [165872.219958]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219961] amdgpu:   
09:28:02 PC3 kernel: [165872.219961]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219965] amdgpu:   
09:28:02 PC3 kernel: [165872.219965]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219968] amdgpu:   
09:28:02 PC3 kernel: [165872.219968]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219972] amdgpu:   
09:28:02 PC3 kernel: [165872.219972]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219975] amdgpu:   
09:28:02 PC3 kernel: [165872.219975]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219978] amdgpu:   
09:28:02 PC3 kernel: [165872.219978]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219981] amdgpu:   
09:28:02 PC3 kernel: [165872.219981]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219985] amdgpu:   
09:28:02 PC3 kernel: [165872.219985]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219988] amdgpu:   
09:28:02 PC3 kernel: [165872.219988]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.219992] amdgpu:   
09:28:02 PC3 kernel: [165872.219992]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.219995] amdgpu:   
09:28:02 PC3 kernel: [165872.219995]  failed to send message 261 ret is 65535   
09:28:02 PC3 kernel: [165872.220169] amdgpu:   
09:28:02 PC3 kernel: [165872.220169]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220173] amdgpu:   
09:28:02 PC3 kernel: [165872.220173]  failed to send message 306 ret is 65535   
09:28:02 PC3 kernel: [165872.220175] amdgpu:   
09:28:02 PC3 kernel: [165872.220175]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220179] amdgpu:   
09:28:02 PC3 kernel: [165872.220179]  failed to send message 5e ret is 65535   
09:28:02 PC3 kernel: [165872.220183] amdgpu:   
09:28:02 PC3 kernel: [165872.220183]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220186] amdgpu:   
09:28:02 PC3 kernel: [165872.220186]  failed to send message 145 ret is 65535   
09:28:02 PC3 kernel: [165872.220190] amdgpu:   
09:28:02 PC3 kernel: [165872.220190]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220195] amdgpu:   
09:28:02 PC3 kernel: [165872.220195]  failed to send message 146 ret is 65535   
09:28:02 PC3 kernel: [165872.220200] amdgpu:   
09:28:02 PC3 kernel: [165872.220200]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220203] amdgpu:   
09:28:02 PC3 kernel: [165872.220203]  failed to send message 148 ret is 65535   
09:28:02 PC3 kernel: [165872.220207] amdgpu:   
09:28:02 PC3 kernel: [165872.220207]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220210] amdgpu:   
09:28:02 PC3 kernel: [165872.220210]  failed to send message 145 ret is 65535   
09:28:02 PC3 kernel: [165872.220215] amdgpu:   
09:28:02 PC3 kernel: [165872.220215]  last message was failed ret is 65535  
09:28:02 PC3 kernel: [165872.220219] amdgpu:   
09:28:02 PC3 kernel: [165872.220219]  failed to send message 146 ret is 65535   
09:28:22 PC3 kernel: [165892.248439] [drm:atom_op_jump [amdgpu]] *ERROR* atombios stuck in loop for more than 20secs aborting  
09:28:22 PC3 kernel: [165892.248505] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing D8DE (len 824, WS 0, PS 0) @ 0xDA5E  
09:28:22 PC3 kernel: [165892.248569] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing D798 (len 326, WS 0, PS 0) @ 0xD888  
09:28:22 PC3 kernel: [165892.248664] [drm:dce110_link_encoder_disable_output [amdgpu]] *ERROR* dce110_link_encoder_disable_output: Failed to execute VBIOS command table!  

We had the problem for a long time now (~ 1 year, visible in SMBIOS event log) and tried some things:

  • Taking the Graphics card out of the PCIe-Slot and plugging it in again
  • Installing a new kernel
  • Installing the amdgpu proprietary drivers
  • Turning off all sleep settings in Kubuntu we could find
  • Using pcie_aspm=off
  • Using a different screen (DVI / Display port)
  • Changing some BIOS Settings related to PCIe

However nothing we do seems to change anything. What makes this problem so hard to tinker with is that it only appears if the PC is not used for some time. So it is really a pain to test anything.

Does anybody have an idea what the error could be or where we could start looking based on the log?


Update we tested the graphics card in a different PC with Windows and it worked fine. Afterwards we installed Win 10 on the same PC and it is also working without problems. So it seems to be the combination GPU + motherboard + KDE neon. Linux seems to allow some kind energy saving state that doesn't go well with the mainboard. However we have searched a lot and did not find any hibernating option we did not turn off.

nobody avatar
gh flag
`lspci -nn | grep 1453` please. is it really one of your graphics card? Please try bootoption `mce=off` #Disable machine check but only if the Computer is runnung without error in use. for security check your ram.
amh23 avatar
cn flag
thanks, this is the output: `00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453] 20:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453] 20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453] 60:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]` However as far as we know from the other IDs involved this is the PCIe-Bridge of the GPU
amh23 avatar
cn flag
We have ECC-Ram and recently checked that it is fine with edac-util . Why do you thinkg `mce=off` could help? As far as I understand this is a cpu related setting. Do you think it could have something to do with heat?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.