I have an embedded device running Ansgstrom Linux on a Colibri VF61 board. The kernel version is: 4.4.59-2.7.2+g7cfa321
In some occasions the device becomes completely inacessible (does not respond to ping or to other services running on other ports).
The leds were still blinking and disconnecting/reconnecting the cable was sufficient to reestablish connectivity. During the outage all the services, apart the network seem to have worked properly.
On of the outages occurred this morning approx 7am, at around 8am, the cable was disconnected/reconnected. Here is what I can see in journalctl log
Jun 07 06:38:31 AlbenV3_0480 systemd[1]: Time has been changed
Jun 07 06:38:31 AlbenV3_0480 systemd[479]: Time has been changed
Jun 07 07:59:05 AlbenV3_0480 systemd-journald[28488]: System journal (/var/log/journal/) is 16.0M, max 16.0M, 0B free.
Jun 07 07:59:06 AlbenV3_0480 systemd-journald[28488]: Missed 125536 kernel messages
Jun 07 07:59:06 AlbenV3_0480 kernel: [<803bb3e8>] (fec_enet_rx_napi) from [<804cb9b4>] (net_rx_action+0x1d8/0x2b4)
Jun 07 07:59:06 AlbenV3_0480 kernel: r10:09417509 r9:8de7fe30 r8:0000012c r7:8084ee80 r6:00000040 r5:00000001
Jun 07 07:59:06 AlbenV3_0480 kernel: r4:8e5a6d40
Jun 07 07:59:06 AlbenV3_0480 kernel: [<804cb7dc>] (net_rx_action) from [<800249dc>] (__do_softirq+0xf4/0x25c)
Jun 07 07:59:06 AlbenV3_0480 kernel: r10:40000000 r9:8087a90c r8:00000101 r7:8de7e000 r6:8087a90c r5:00000003
Jun 07 07:59:06 AlbenV3_0480 kernel: r4:00000004
Jun 07 07:59:06 AlbenV3_0480 kernel: [<800248e8>] (__do_softirq) from [<80024bbc>] (do_softirq.part.2+0x34/0x3c)
Jun 07 07:59:06 AlbenV3_0480 kernel: r10:00000004 r9:8e138900 r8:00000000 r7:808720d8 r6:00000000 r5:ffffe000
Jun 07 07:59:06 AlbenV3_0480 kernel: r4:200b0013
Jun 07 07:59:06 AlbenV3_0480 kernel: [<80024b88>] (do_softirq.part.2) from [<80024c80>] (__local_bh_enable_ip+0xbc/0xcc)
Jun 07 07:59:06 AlbenV3_0480 kernel: r5:ffffe000 r4:000001ff
Jun 07 07:59:06 AlbenV3_0480 kernel: [<80024bc4>] (__local_bh_enable_ip) from [<8056f310>] (unix_create1+0x160/0x19c)
Jun 07 07:59:06 AlbenV3_0480 kernel: r5:808a12d0 r4:8da2c6c0
Jun 07 07:59:06 AlbenV3_0480 kernel: [<8056f1b0>] (unix_create1) from [<8056f3a8>] (unix_create+0x5c/0xa0)
Jun 07 07:59:06 AlbenV3_0480 kernel: r9:00000000 r8:808720d8 r7:80846c6c r6:8e138900 r5:8056f34c r4:8065f4d0
Jun 07 07:59:06 AlbenV3_0480 kernel: [<8056f34c>] (unix_create) from [<804b2dfc>] (__sock_create+0xec/0x188)
Jun 07 07:59:06 AlbenV3_0480 kernel: [<804b2d10>] (__sock_create) from [<804b3ddc>] (SyS_socket+0x5c/0xf0)
Jun 07 07:59:06 AlbenV3_0480 kernel: r10:00000000 r9:8de7e000 r8:8000f944 r7:00000119 r6:8de7ff84 r5:00080002
Jun 07 07:59:06 AlbenV3_0480 kernel: r4:00000000 r3:00000000
Jun 07 07:59:06 AlbenV3_0480 kernel: [<804b3d80>] (SyS_socket) from [<8000f780>] (ret_fast_syscall+0x0/0x3c)
Jun 07 07:59:06 AlbenV3_0480 kernel: r7:00000119 r6:54aff820 r5:00000001 r4:7ef0bfa3
Jun 07 07:59:06 AlbenV3_0480 kernel: Mem-Info:
Jun 07 07:59:06 AlbenV3_0480 kernel: active_anon:2586 inactive_anon:46 isolated_anon:0
active_file:25114 inactive_file:26725 isolated_file:6
unevictable:0 dirty:15 writeback:0 unstable:0
slab_reclaimable:2563 slab_unreclaimable:1595
mapped:1035 shmem:97 pagetables:184 bounce:0
free:997 free_pcp:25 free_cma:829
Jun 07 07:59:06 AlbenV3_0480 kernel: Normal free:3988kB min:1928kB low:2408kB high:2892kB active_anon:10344kB inactive_anon:184kB active_file:100456kB inactive_file:106900kB unevictable:0kB isolated(anon):0kB isolated(file):24kB present:260096kB managed:249040kB mlocked:0kB dirty:60kB writeback:0kB mapped:4140kB shmem:388kB slab_reclaimable:10252kB slab_unreclaimable:6380kB kernel_stack:832kB pagetables:736kB unstable:0kB bounce:0kB free_pcp:100kB local_pcp:100kB free_cma:3316kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 07 07:59:06 AlbenV3_0480 kernel: lowmem_reserve[]: 0 0
Jun 07 07:59:06 AlbenV3_0480 kernel: Normal: 85*4kB (UEC) 40*8kB (UEC) 44*16kB (UC) 22*32kB (C) 6*64kB (C) 2*128kB (C) 1*256kB (C) 0*512kB 1*1024kB (C) 0*2048kB 0*4096kB = 3988kB
Jun 07 07:59:06 AlbenV3_0480 kernel: 51941 total pagecache pages
Jun 07 07:59:06 AlbenV3_0480 kernel: 0 pages in swap cache
Jun 07 07:59:06 AlbenV3_0480 kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 07 07:59:06 AlbenV3_0480 kernel: Free swap = 0kB
Jun 07 07:59:06 AlbenV3_0480 kernel: Total swap = 0kB
Jun 07 07:59:06 AlbenV3_0480 kernel: 65024 pages RAM
Jun 07 07:59:06 AlbenV3_0480 kernel: 0 pages HighMem/MovableOnly
Jun 07 07:59:06 AlbenV3_0480 kernel: 2764 pages reserved
Jun 07 07:59:06 AlbenV3_0480 kernel: 4096 pages cma reserved
Jun 07 07:59:06 AlbenV3_0480 kernel: systemd-logind: page allocation failure: order:0, mode:0x2080020
[...]
Jun 07 07:59:09 AlbenV3_0480 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
Jun 07 07:59:09 AlbenV3_0480 kernel: ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
Jun 07 07:59:09 AlbenV3_0480 kernel: mmcblk0: error -84 transferring data, sector 18133984, nr 512, cmd response 0x900, card status 0xb00
Jun 07 07:59:09 AlbenV3_0480 kernel: ftdi_sio 2-1:1.0: device disconnected
Does anybody have any idea of what happened and what can be done to prevent it from happening again?
On of the things I was thinking is that we are using the USB bus to acquire data. Since the network is also using the same bus afaik, could it be that it was unable to process the data and somehow crashed? But I would expect some entry in the logs in that case...
Thanks!
