I have 1 box that seems to not want to run chef, while I have 5 other identical boxes that are working just fine using the same rhel and chef versions. On the non-working box, Chef runs correctly when the runbook is empty, but it fails when I add something to the runbook and try again.
Here is some of the error that is returned
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529: [BUG] rb_sys_fail_path_in(rb_file_s_rename, (/tmp/chef-rest20230427-6900-1ip3kea, /var/chef/cache/cookbooks/selinux_policy/resources/fcontext.rb)) - errno == 0
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
-- Control frame information ---------------------------------------------
c:0014 p:---- s:0081 e:000080 CFUNC :rename
c:0013 p:0062 s:0075 e:000074 BLOCK /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529
c:0012 p:0061 s:0069 e:000068 BLOCK /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1557
c:0011 p:0098 s:0064 e:000063 METHOD /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1573
c:0010 p:0009 s:0057 e:000056 METHOD /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1555
c:0009 p:0066 s:0051 e:000050 METHOD /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:520
c:0008 p:0128 s:0040 e:000039 METHOD /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/file_cache.rb:83
c:0007 p:0056 s:0032 e:000031 METHOD /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:318
c:0006 p:0073 s:0025 e:000024 METHOD /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:290
c:0005 p:0007 s:0019 e:000018 BLOCK /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:177
c:0004 p:0028 s:0014 e:000013 BLOCK /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/util/threaded_job_queue.rb:52 [FINISH]
c:0003 p:---- s:0010 e:000009 CFUNC :loop
c:0002 p:0006 s:0006 e:000005 BLOCK /opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/util/threaded_job_queue.rb:50 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/util/threaded_job_queue.rb:50:in `block (2 levels) in process'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/util/threaded_job_queue.rb:50:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/util/threaded_job_queue.rb:52:in `block (3 levels) in process'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:177:in `block (2 levels) in sync_cookbooks'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:290:in `sync_file'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/cookbook/synchronizer.rb:318:in `download_file'
/opt/chef/embedded/lib/ruby/gems/2.6.0/gems/chef-15.12.22/lib/chef/file_cache.rb:83:in `move_to'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:520:in `mv'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1555:in `fu_each_src_dest'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1573:in `fu_each_src_dest0'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1557:in `block in fu_each_src_dest'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529:in `block in mv'
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529:in `rename'
-- Other runtime information ---------------------------------------------
* Loaded script: /bin/chef-client
* Loaded features:
0 enumerator.so
1 thread.rb
2 rational.so
3 complex.so
4 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so
5 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so
6 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/rbconfig.rb
7 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/compatibility.rb
8 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/defaults.rb
9 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/deprecate.rb
10 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/errors.rb
11 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/version.rb
12 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/requirement.rb
13 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/platform.rb
14 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/basic_specification.rb
15 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/stub_specification.rb
16 /opt/chef/embedded/lib/ruby/2.6.0/delegate.rb
17 /opt/chef/embedded/lib/ruby/2.6.0/uri/rfc2396_parser.rb
18 /opt/chef/embedded/lib/ruby/2.6.0/uri/rfc3986_parser.rb
19 /opt/chef/embedded/lib/ruby/2.6.0/uri/common.rb
20 /opt/chef/embedded/lib/ruby/2.6.0/uri/generic.rb
21 /opt/chef/embedded/lib/ruby/2.6.0/uri/file.rb
22 /opt/chef/embedded/lib/ruby/2.6.0/uri/ftp.rb
23 /opt/chef/embedded/lib/ruby/2.6.0/uri/http.rb
24 /opt/chef/embedded/lib/ruby/2.6.0/uri/https.rb
25 /opt/chef/embedded/lib/ruby/2.6.0/uri/ldap.rb
26 /opt/chef/embedded/lib/ruby/2.6.0/uri/ldaps.rb
27 /opt/chef/embedded/lib/ruby/2.6.0/uri/mailto.rb
28 /opt/chef/embedded/lib/ruby/2.6.0/uri.rb
29 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/specification_policy.rb
30 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/util/list.rb
31 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so
32 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/specification.rb
33 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/exceptions.rb
34 /opt/chef/embedded/lib/ruby/2.6.0/rubygems/util.rb
...
7f8d2f10a000-7f8d2f309000 ---p 00001000 fd:05 8640350 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/fcntl.so
7f8d2f309000-7f8d2f30a000 r--p 00000000 fd:05 8640350 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/fcntl.so
7f8d2f30a000-7f8d2f30b000 rw-p 00001000 fd:05 8640350 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/fcntl.so
7f8d2f30b000-7f8d2f311000 r-xp 00000000 fd:05 8640349 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/etc.so
7f8d2f311000-7f8d2f510000 ---p 00006000 fd:05 8640349 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/etc.so
7f8d2f510000-7f8d2f511000 r--p 00005000 fd:05 8640349 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/etc.so
7f8d2f511000-7f8d2f512000 rw-p 00006000 fd:05 8640349 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/etc.so
7f8d2f512000-7f8d2f519000 r-xp 00000000 fd:05 8640359 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/pathname.so
7f8d2f519000-7f8d2f718000 ---p 00007000 fd:05 8640359 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/pathname.so
7f8d2f718000-7f8d2f719000 r--p 00006000 fd:05 8640359 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/pathname.so
7f8d2f719000-7f8d2f71a000 rw-p 00007000 fd:05 8640359 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/pathname.so
7f8d2f71a000-7f8d2f722000 r-xp 00000000 fd:05 8640366 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so
7f8d2f722000-7f8d2f921000 ---p 00008000 fd:05 8640366 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so
7f8d2f921000-7f8d2f922000 r--p 00007000 fd:05 8640366 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so
7f8d2f922000-7f8d2f923000 rw-p 00008000 fd:05 8640366 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so
7f8d2f923000-7f8d2f925000 r-xp 00000000 fd:05 12613225 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so
7f8d2f925000-7f8d2fb25000 ---p 00002000 fd:05 12613225 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so
7f8d2fb25000-7f8d2fb26000 r--p 00002000 fd:05 12613225 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so
7f8d2fb26000-7f8d2fb27000 rw-p 00003000 fd:05 12613225 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so
7f8d2fb27000-7f8d2fb29000 r-xp 00000000 fd:05 8640313 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so
7f8d2fb29000-7f8d2fd28000 ---p 00002000 fd:05 8640313 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so
7f8d2fd28000-7f8d2fd29000 r--p 00001000 fd:05 8640313 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so
7f8d2fd29000-7f8d2fd2a000 rw-p 00002000 fd:05 8640313 /opt/chef/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so
7f8d2fd2a000-7f8d31d33000 rw-p 00000000 00:00 0
7f8d31d33000-7f8d31eef000 r-xp 00000000 fd:02 4195575 /usr/lib64/libc-2.28.so
7f8d31eef000-7f8d320ee000 ---p 001bc000 fd:02 4195575 /usr/lib64/libc-2.28.so
7f8d320ee000-7f8d320f2000 r--p 001bb000 fd:02 4195575 /usr/lib64/libc-2.28.so
7f8d320f2000-7f8d320f4000 rw-p 001bf000 fd:02 4195575 /usr/lib64/libc-2.28.so
7f8d320f4000-7f8d320f8000 rw-p 00000000 00:00 0
7f8d320f8000-7f8d32279000 r-xp 00000000 fd:02 4636839 /usr/lib64/libm-2.28.so
7f8d32279000-7f8d32478000 ---p 00181000 fd:02 4636839 /usr/lib64/libm-2.28.so
7f8d32478000-7f8d32479000 r--p 00180000 fd:02 4636839 /usr/lib64/libm-2.28.so
7f8d32479000-7f8d3247a000 rw-p 00181000 fd:02 4636839 /usr/lib64/libm-2.28.so
7f8d3247a000-7f8d3249a000 r-xp 00000000 fd:02 4196587 /usr/lib64/libcrypt.so.1.1.0
7f8d3249a000-7f8d32699000 ---p 00020000 fd:02 4196587 /usr/lib64/libcrypt.so.1.1.0
7f8d32699000-7f8d3269a000 r--p 0001f000 fd:02 4196587 /usr/lib64/libcrypt.so.1.1.0
7f8d3269a000-7f8d326a3000 rw-p 00000000 00:00 0
7f8d326a3000-7f8d326a6000 r-xp 00000000 fd:02 4636838 /usr/lib64/libdl-2.28.so
7f8d326a6000-7f8d328a5000 ---p 00003000 fd:02 4636838 /usr/lib64/libdl-2.28.so
7f8d328a5000-7f8d328a6000 r--p 00002000 fd:02 4636838 /usr/lib64/libdl-2.28.so
7f8d328a6000-7f8d328a7000 rw-p 00003000 fd:02 4636838 /usr/lib64/libdl-2.28.so
7f8d328a7000-7f8d328ae000 r-xp 00000000 fd:02 4636844 /usr/lib64/librt-2.28.so
7f8d328ae000-7f8d32aad000 ---p 00007000 fd:02 4636844 /usr/lib64/librt-2.28.so
7f8d32aad000-7f8d32aae000 r--p 00006000 fd:02 4636844 /usr/lib64/librt-2.28.so
7f8d32aae000-7f8d32aaf000 rw-p 00007000 fd:02 4636844 /usr/lib64/librt-2.28.so
7f8d32aaf000-7f8d32aca000 r-xp 00000000 fd:02 4636842 /usr/lib64/libpthread-2.28.so
7f8d32aca000-7f8d32cc9000 ---p 0001b000 fd:02 4636842 /usr/lib64/libpthread-2.28.so
7f8d32cc9000-7f8d32cca000 r--p 0001a000 fd:02 4636842 /usr/lib64/libpthread-2.28.so
7f8d32cca000-7f8d32ccb000 rw-p 0001b000 fd:02 4636842 /usr/lib64/libpthread-2.28.so
7f8d32ccb000-7f8d32ccf000 rw-p 00000000 00:00 0
7f8d32ccf000-7f8d32ce5000 r-xp 00000000 fd:05 4194641 /opt/chef/embedded/lib/libz.so.1.2.11
7f8d32ce5000-7f8d32ee4000 ---p 00016000 fd:05 4194641 /opt/chef/embedded/lib/libz.so.1.2.11
7f8d32ee4000-7f8d32ee5000 r--p 00015000 fd:05 4194641 /opt/chef/embedded/lib/libz.so.1.2.11
7f8d32ee5000-7f8d32ee6000 rw-p 00016000 fd:05 4194641 /opt/chef/embedded/lib/libz.so.1.2.11
7f8d32ee6000-7f8d3322c000 r-xp 00000000 fd:05 4194555 /opt/chef/embedded/lib/libruby.so.2.6.6
7f8d3322c000-7f8d3342b000 ---p 00346000 fd:05 4194555 /opt/chef/embedded/lib/libruby.so.2.6.6
7f8d3342b000-7f8d33431000 r--p 00345000 fd:05 4194555 /opt/chef/embedded/lib/libruby.so.2.6.6
7f8d33431000-7f8d33434000 rw-p 0034b000 fd:05 4194555 /opt/chef/embedded/lib/libruby.so.2.6.6
7f8d33434000-7f8d33445000 rw-p 00000000 00:00 0
7f8d33445000-7f8d33472000 r-xp 00000000 fd:02 5173330 /usr/lib64/ld-2.28.so
7f8d3348e000-7f8d33610000 rw-p 00000000 00:00 0
7f8d33610000-7f8d33663000 r--p 00000000 fd:02 8409611 /usr/lib/locale/en_US.utf8/LC_CTYPE
7f8d33663000-7f8d33668000 rw-p 00000000 00:00 0
7f8d33669000-7f8d33670000 r--s 00000000 fd:02 8427126 /usr/lib64/gconv/gconv-modules.cache
7f8d33670000-7f8d33672000 rw-p 00000000 00:00 0
7f8d33672000-7f8d33673000 r--p 0002d000 fd:02 5173330 /usr/lib64/ld-2.28.so
7f8d33673000-7f8d33675000 rw-p 0002e000 fd:02 5173330 /usr/lib64/ld-2.28.so
7fffbe4c0000-7fffbecbf000 rw-p 00000000 00:00 0 [stack]
7fffbede4000-7fffbede8000 r--p 00000000 00:00 0 [vvar]
7fffbede8000-7fffbedea000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: https://www.ruby-lang.org/bugreport.html
I tracked the error down to chef attempting to move files from a temp directory to it's cache directory:
ex:
from: /tmp/chef-rest20230427-6900-1ip3kea
to: /var/chef/cache/cookbooks/selinux_policy/resources/fcontext.rb
I'm able to move the files via terminal/command line as root, but when I attempt to use ruby to perform the same change, I get the same error:
[root@myserver myuser]# /opt/chef/embedded/bin/irb
irb(main):001:0> require 'fileutils'
=> true
irb(main):002:0> FileUtils.mv('/tmp/chef-rest20230427-6900-1ip3kea', '/var/chef/cache/cookbooks/selinux_policy/resources/fcontext.rb')
/opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529: [BUG] rb_sys_fail_path_in(rb_file_s_rename, (/tmp/chef-rest20230427-6900-1ip3kea, /var/chef/cache/cookbooks/selinux_policy/resources/fcontext.rb)) - errno == 0
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
-- Control frame information ---------------------------------------------
c:0014 p:---- s:0081 e:000080 CFUNC :rename
c:0013 p:0062 s:0075 e:000074 BLOCK /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:529
c:0012 p:0061 s:0069 e:000068 BLOCK /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1557
c:0011 p:0098 s:0064 e:000063 METHOD /opt/chef/embedded/lib/ruby/2.6.0/fileutils.rb:1573
I also get errors when moving files into the /var/ and the /var/chef/cache/cookbooks/s directory using ruby.
That being said, the same ruby command works when I move files around my home directory
[root@mycomputer myuser]# /opt/chef/embedded/bin/irb
irb(main):001:0> require 'fileutils'
=> true
irb(main):002:0> FileUtils.mv('/home/myuser/fileIcreated.txt', '/home/myuser/chef-fileIcreated2.txt')
=> 0
I've checked the permissions between my working server and my non-working server and they look to be the same:
notWorking:
[root@notWorkingComputer cookbooks]# ls -lah /tmp
total 12K
drwxrwxrwt. 12 root root 4.0K Apr 28 14:18 .
dr-xr-xr-x. 18 root root 255 Apr 27 14:14 ..
drwx------ 3 root root 22 Apr 27 16:32 BES
-rw------- 1 root root 0 Apr 27 17:01 chef-rest20230427-9210-17z2ovo
-rw------- 1 root root 0 Apr 27 17:01 chef-rest20230427-9210-19d788s
-rw------- 1 root root 0 Apr 27 17:01 chef-rest20230427-9210-1jxgrj2
-rw------- 1 root root 0 Apr 27 17:01 chef-rest20230427-9210-1p8iyjg
-rw------- 1 root root 0 Apr 27 17:01 chef-rest20230427-9210-nzj3gb
-rw------- 1 root root 395 Apr 27 17:01 chef-rest20230427-9210-p2zib8
-rw------- 1 root root 0 Apr 28 13:23 chef-rest20230428-63218-go0u2k
-rw------- 1 root root 0 Apr 28 13:23 chef-rest20230428-63218-kem2pc
-rw------- 1 root root 1.9K Apr 28 13:23 chef-rest20230428-63218-w7bx67
-rw------- 1 root root 0 Apr 28 13:23 chef-rest20230428-63218-wmjcd7
drwxrwxrwt 2 root root 6 Apr 27 14:14 .font-unix
drwxrwxrwt 2 root root 6 Apr 27 14:14 .ICE-unix
drwxr-xr-x 4 root root 33 Oct 7 2022 kheaders-4.18.0-372.32.1.el8_6.x86_64
drwx------ 3 root root 17 Apr 27 16:32 systemd-private-4cc96e5e60104fb1a9ab1726725d7a78-chronyd.service-bYOmR8
drwxrwxrwt 2 root root 6 Apr 27 14:14 .Test-unix
drwx------ 2 root root 6 Apr 27 14:14 vmware-root_1201-3979774027
drwx------ 2 root root 6 Apr 27 16:32 vmware-root_1206-2730693522
drwxrwxrwt. 2 root root 6 Mar 14 09:09 .X11-unix
drwxrwxrwt 2 root root 6 Apr 27 14:14 .XIM-unix
working
[root@workingComputer tmp]# ls -lah
total 0
drwxrwxrwt. 8 root root 284 Apr 28 14:16 .
dr-xr-xr-x. 18 root root 235 Dec 16 11:29 ..
drwx------. 2 myUser domain users 77 Apr 25 15:24 install.sh.504060
drwx------. 2 myUser domain users 78 Apr 25 15:27 install.sh.504563
srwxr-xr-x. 1 redis redis 0 Apr 27 12:44 redis.sock
drwx------. 3 root root 17 Mar 30 23:35 systemd-private-88977748067140c78f032c795561768f-chronyd.service-Z9CLKA
drwx------. 3 root root 17 Apr 26 10:00 systemd-private-88977748067140c78f032c795561768f-postfix.service-Qzco75
drwx------. 2 root root 6 Apr 22 03:53 vmware-root_1195-3988621783
drwxrwxrwt. 2 root root 6 Dec 16 05:36 .X11-unix
We have carbon black installed on both boxes, but I had it disabled on the non-working box, and that did not help, so I don't believe the problem is related to carbon black.
At this point I'm out of ideas, is there anything for me to look at that could help me figure out what's going on here?
Thanks,
Alex
