I am upgrading a server with 2 SAS cards from Debian 11 -> Debian 12
I have come upon an interesting problem. At first it didn't load any SAS cards and I got to initramfs.
Increase disk detection timeout at boot with Linux/Systemd
Then I used the fix in here and used rootdelay=180 and then 1 SAS card loaded itself, but the second one didnt come online.
I have added a picture aswell:
1_sas_card_detected
I have added the output of dmesg -T | grep sas aswell
[Wed Aug 9 12:50:24 2023] mpt3sas version 43.100.00.00 loaded
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (3988540 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: MSI-X vectors supported: 96
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: 0 16 16
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: High IOPs queues : disabled
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 250
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 252
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 254
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 256
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 258
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 260
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 262
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 264
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 266
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 268
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 270
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 272
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 274
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 276
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 278
[Wed Aug 9 12:50:24 2023] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 280
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: iomem(0x00000000e0e40000), mapped(0x000000004222aca5), size(65536)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: ioport(0x000000000000b000), size(256)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: sending message unit reset !!
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: message unit reset: SUCCESS
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: request pool(0x0000000051d56977) - dma(0x938a3a000): depth(3200), frame_size(128), pool_size(400 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: sense pool(0x000000009c61e8a2) - dma(0x93f7ba000): depth(2939), element_size(96), pool_size (275 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: reply pool(0x00000000c769fa7c) - dma(0x93f83a000): depth(3264), frame_size(128), pool_size(408 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: config page(0x000000008a8c65b6) - dma(0x93f72a000): size(512)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: Allocated physical memory: size(8887 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: Current Controller Queue Depth(2936),Max Controller Queue Depth(3072)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: _base_display_fwpkg_version: complete
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.10.00), ChipRevision(0x02), BiosVersion(08.37.00.00)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: sending port enable !!
[Wed Aug 9 12:50:24 2023] mpt3sas_cm1: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (3988540 kB)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: hba_port entry: 000000006a2e3bf6, port: 255 is added to hba_port list
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500304801bd55b02), phys(8)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0x9) sas_address(0x4433221100000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xa) sas_address(0x4433221101000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xb) sas_address(0x4433221102000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xc) sas_address(0x4433221103000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xd) sas_address(0x4433221104000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xe) sas_address(0x4433221105000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0xf) sas_address(0x4433221106000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: handle(0x10) sas_address(0x4433221107000000) port_type(0x1)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm0: port enable: SUCCESS
[Wed Aug 9 12:50:24 2023] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x5002538f024576d9)
[Wed Aug 9 12:50:24 2023] end_device-0:0: add: handle(0x0009), sas_addr(0x4433221100000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x5002538f024576dc)
[Wed Aug 9 12:50:24 2023] end_device-0:1: add: handle(0x000a), sas_addr(0x4433221101000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221102000000), phy(2), device_name(0x5002538f024576d8)
[Wed Aug 9 12:50:24 2023] end_device-0:2: add: handle(0x000b), sas_addr(0x4433221102000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221103000000), phy(3), device_name(0x5002538f024576d7)
[Wed Aug 9 12:50:24 2023] end_device-0:3: add: handle(0x000c), sas_addr(0x4433221103000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221104000000), phy(4), device_name(0x5002538f024576e0)
[Wed Aug 9 12:50:24 2023] end_device-0:4: add: handle(0x000d), sas_addr(0x4433221104000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:5:0: SATA: handle(0x000e), sas_addr(0x4433221105000000), phy(5), device_name(0x5002538f024576e1)
[Wed Aug 9 12:50:24 2023] end_device-0:5: add: handle(0x000e), sas_addr(0x4433221105000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:6:0: SATA: handle(0x000f), sas_addr(0x4433221106000000), phy(6), device_name(0x5002538f0245768d)
[Wed Aug 9 12:50:24 2023] end_device-0:6: add: handle(0x000f), sas_addr(0x4433221106000000)
[Wed Aug 9 12:50:24 2023] scsi 0:0:7:0: SATA: handle(0x0010), sas_addr(0x4433221107000000), phy(7), device_name(0x5002538f02457689)
[Wed Aug 9 12:50:24 2023] mpt3sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug 9 12:50:24 2023] mpt3sas_cm1: MSI-X vectors supported: 96
[Wed Aug 9 12:50:24 2023] mpt3sas_cm1: 0 16 16
[Wed Aug 9 12:50:24 2023] mpt3sas 0000:b3:00.0: xen map irq failed -22 for 32752 domain
[Wed Aug 9 12:50:24 2023] end_device-0:7: add: handle(0x0010), sas_addr(0x4433221107000000)
[Wed Aug 9 12:50:24 2023] mpt3sas_base_map_resources.cold+0x296/0x718 [mpt3sas]
[Wed Aug 9 12:50:24 2023] mpt3sas_base_attach+0xeb/0x4c0 [mpt3sas]
[Wed Aug 9 12:50:24 2023] _scsih_probe+0x6e9/0x8f0 [mpt3sas]
[Wed Aug 9 12:50:24 2023] _mpt3sas_init+0x198/0x1000 [mpt3sas]
[Wed Aug 9 12:50:24 2023] Modules linked in: ahci libahci xhci_pci i40e mpt3sas(+) libata xhci_hcd crct10dif_pclmul crct10dif_common crc32_pclmul raid_class scsi_transport_sas crc32c_intel usbcore scsi_mod ptp i2c_i801 pps_core i2c_smbus lpc_ich usb_common scsi_common
EDIT:
I have added normal dmesg -T logs it seems to be a kernel error.
[Wed Aug 9 16:13:21 2023] mpt3sas_cm1: MSI-X vectors supported: 96
[Wed Aug 9 16:13:21 2023] no of cores: 16, max_msix_vectors: -1
[Wed Aug 9 16:13:21 2023] mpt3sas_cm1: 0 16 16
[Wed Aug 9 16:13:21 2023] mpt3sas 0000:b3:00.0: xen map irq failed -22 for 32752 domain
[Wed Aug 9 16:13:21 2023] BUG: kernel NULL pointer dereference, address: 000000000000004d
[Wed Aug 9 16:13:21 2023] #PF: supervisor read access in kernel mode
[Wed Aug 9 16:13:21 2023] #PF: error_code(0x0000) - not-present page
[Wed Aug 9 16:13:21 2023] PGD 0 P4D 0
[Wed Aug 9 16:13:21 2023] Oops: 0000 [#1] PREEMPT SMP NOPTI
[Wed Aug 9 16:13:21 2023] CPU: 0 PID: 230 Comm: (udev-worker) Not tainted 6.1.0-10-amd64 #1 Debian 6.1.38-2
[Wed Aug 9 16:13:21 2023] Hardware name: Supermicro SYS-1019P-WTR/X11SPW-TF, BIOS 3.9 03/15/2023
[Wed Aug 9 16:13:21 2023] RIP: e030:xen_destroy_irq+0x98/0x110
[Wed Aug 9 16:13:21 2023] Code: c7 80 ff ba 82 e8 f8 e8 3d 00 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 75 7d 48 83 c4 10 44 89 e0 5b 5d 41 5c c3 cc cc cc cc <f6> 45 4d 04 75 c1 0f b7 45 48 bf 0e 00 00 00 48 89 e6 89 44 24 04
[Wed Aug 9 16:13:21 2023] RSP: e02b:ffffc90040c2b8d0 EFLAGS: 00010202
[Wed Aug 9 16:13:21 2023] RAX: 0000000000000001 RBX: 000000000000013a RCX: 0000000000000000
[Wed Aug 9 16:13:21 2023] RDX: ffff888118460000 RSI: ffff888100520490 RDI: ffffffff82baff80
[Wed Aug 9 16:13:21 2023] RBP: 0000000000000000 R08: ffff888100520688 R09: ffffffff82ad47c0
[Wed Aug 9 16:13:21 2023] end_device-0:7: add: handle(0x0010), sas_addr(0x4433221107000000)
[Wed Aug 9 16:13:21 2023] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88810276d0d0
[Wed Aug 9 16:13:21 2023] R13: 0000000000000010 R14: ffffc90040239000 R15: ffff88810276d000
[Wed Aug 9 16:13:21 2023] FS: 00007f8c107fa8c0(0000) GS:ffff888195000000(0000) knlGS:0000000000000000
[Wed Aug 9 16:13:21 2023] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[Wed Aug 9 16:13:21 2023] CR2: 000000000000004d CR3: 0000000118104000 CR4: 0000000000050660
[Wed Aug 9 16:13:21 2023] Call Trace:
[Wed Aug 9 16:13:21 2023] <TASK>
[Wed Aug 9 16:13:21 2023] ? __die_body.cold+0x1a/0x1f
[Wed Aug 9 16:13:21 2023] ? page_fault_oops+0xd2/0x2b0
[Wed Aug 9 16:13:21 2023] ? exc_page_fault+0x70/0x170
[Wed Aug 9 16:13:21 2023] ? asm_exc_page_fault+0x22/0x30
[Wed Aug 9 16:13:21 2023] ? xen_destroy_irq+0x98/0x110
[Wed Aug 9 16:13:21 2023] xen_teardown_msi_irqs+0x3b/0x70
[Wed Aug 9 16:13:21 2023] msi_domain_free_irqs_descs_locked+0x18/0x40
[Wed Aug 9 16:13:21 2023] pci_msi_teardown_msi_irqs+0x3e/0x40
[Wed Aug 9 16:13:21 2023] __pci_enable_msix_range+0x33f/0x4e0
[Wed Aug 9 16:13:21 2023] pci_alloc_irq_vectors_affinity+0xa9/0x110
[Wed Aug 9 16:13:21 2023] mpt3sas_base_map_resources.cold+0x296/0x718 [mpt3sas]
[Wed Aug 9 16:13:21 2023] ? ncpus_cmp_func+0x10/0x10
[Wed Aug 9 16:13:21 2023] mpt3sas_base_attach+0xeb/0x4c0 [mpt3sas]
[Wed Aug 9 16:13:21 2023] _scsih_probe+0x6e9/0x8f0 [mpt3sas]
[Wed Aug 9 16:13:21 2023] local_pci_probe+0x3e/0x80
[Wed Aug 9 16:13:21 2023] pci_device_probe+0xc3/0x230
[Wed Aug 9 16:13:21 2023] really_probe+0xdb/0x380
[Wed Aug 9 16:13:21 2023] ? pm_runtime_barrier+0x50/0x90
[Wed Aug 9 16:13:21 2023] __driver_probe_device+0x78/0x120
[Wed Aug 9 16:13:21 2023] driver_probe_device+0x1f/0x90
[Wed Aug 9 16:13:21 2023] __driver_attach+0xce/0x1c0
[Wed Aug 9 16:13:21 2023] ? __device_attach_driver+0x110/0x110
[Wed Aug 9 16:13:21 2023] bus_for_each_dev+0x84/0xd0
[Wed Aug 9 16:13:21 2023] bus_add_driver+0x1ae/0x200
[Wed Aug 9 16:13:21 2023] driver_register+0x89/0xe0
[Wed Aug 9 16:13:21 2023] ? 0xffffffffc06a2000
[Wed Aug 9 16:13:21 2023] _mpt3sas_init+0x198/0x1000 [mpt3sas]
[Wed Aug 9 16:13:21 2023] ? 0xffffffffc06a2000
[Wed Aug 9 16:13:21 2023] do_one_initcall+0x56/0x220
[Wed Aug 9 16:13:21 2023] do_init_module+0x4a/0x200
[Wed Aug 9 16:13:21 2023] __do_sys_finit_module+0xac/0x120
[Wed Aug 9 16:13:21 2023] do_syscall_64+0x58/0xc0
[Wed Aug 9 16:13:21 2023] ? fpregs_assert_state_consistent+0x22/0x50
[Wed Aug 9 16:13:21 2023] ? exit_to_user_mode_prepare+0x40/0x1d0
[Wed Aug 9 16:13:21 2023] ? syscall_exit_to_user_mode+0x17/0x40
[Wed Aug 9 16:13:21 2023] ? do_syscall_64+0x67/0xc0
[Wed Aug 9 16:13:21 2023] ? check_preemption_disabled+0x31/0xf0
[Wed Aug 9 16:13:21 2023] ? fpregs_assert_state_consistent+0x22/0x50
[Wed Aug 9 16:13:21 2023] usb 1-7: New USB device found, idVendor=0557, idProduct=7000, bcdDevice= 0.00
[Wed Aug 9 16:13:21 2023] ? exit_to_user_mode_prepare+0x40/0x1d0
[Wed Aug 9 16:13:21 2023] usb 1-7: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[Wed Aug 9 16:13:21 2023] ? syscall_exit_to_user_mode+0x17/0x40
[Wed Aug 9 16:13:21 2023] ? do_syscall_64+0x67/0xc0
[Wed Aug 9 16:13:21 2023] ? do_syscall_64+0x67/0xc0
[Wed Aug 9 16:13:21 2023] ? do_syscall_64+0x67/0xc0
[Wed Aug 9 16:13:21 2023] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[Wed Aug 9 16:13:21 2023] RIP: 0033:0x7f8c10f06539
[Wed Aug 9 16:13:21 2023] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 08 0d 00 f7 d8 64 89 01 48
