Join Log in
Score:0
avatar Server

How can I skip an async task that is already running?

br flag
Vladimir Botka

(Credit @Kerrick Staley)

I would like to create an Ansible playbook with an async task followed by an async_status task that lets me do the following:

  1. I run the playbook on my laptop. It starts the async task in the background and starts polling it with the async_status task.

  2. I reboot my laptop. The async task continues running on the server.

  3. I run the playbook again. It recognizes that the async task is already running on the server and goes straight to the async_status task, polling the task that was started in step 1.

Does Ansible support this? How can I create a playbook that does it?

431
2 + 0
Score:1
avatar Server
br flag
Vladimir Botka

In the first play start the processes asynchronously, create the job ID files var/run//jid, and write log to var/log/async.log. In the second play wait for the processes to finish, write the log, and remove the job ID files.

Example of a complete project for testing

shell> tree .
.
├── ansible.cfg
├── group_vars
│   └── all
│       └── async_common.yml
├── hosts
├── pb-start.yml
├── pb-status.yml
├── proc01.yml
└── var
    ├── log
    └── run

7 directories, 7 files

shell> cat group_vars/all/async_common.yml 
var_run: "{{ playbook_dir }}/var/run"
var_log: "{{ playbook_dir }}/var/log"
async_log: async.log

shel> cat hosts
test_11
test_13

Declare the processes. Two processes will be started on each host

shell> cat proc01.yml 
cmds:
  - cmd: sleep 30
    async: 45
  - cmd: sleep 45
    async: 60

Start the processes, create job ID, and write log

shell> cat pb-start.yml
- hosts: all

  vars:

    results_dict: "{{ dict(ansible_play_hosts|
                           zip(ansible_play_hosts|
                               map('extract', hostvars, 'async_results'))) }}"

  tasks:

    - name: Start processes
      block:

        - command: "{{ item.cmd }}"
          async: "{{ item.async }}"
          poll: 0
          loop: "{{ cmds }}"
          register: async_results
        - debug:
            var: async_results
          when: debug|d(false)|bool

    - name: Record processes and write log
      block:

        - file:
            state: directory
            path: "{{ item }}"
          loop:
            - "{{ var_run }}"
            - "{{ var_log }}"
        - file:
            state: directory
            path: "{{ var_run }}/{{ item }}"
          loop: "{{ ansible_play_hosts }}"

        - copy:
            dest: "{{ var_run }}/{{ item.0.key }}/{{ item.1.ansible_job_id }}"
            content: "{{ item.1.results_file }}"
          loop: "{{ results_dict|dict2items|subelements('value.results') }}"
          loop_control:
            label: "{{ item.0.key }} {{ item.1.ansible_job_id }}"

        - lineinfile:
            create: true
            dest: "{{ var_log }}/{{ async_log }}"
            line: >-
              {{ '%Y-%m-%d %H:%M:%S'|strftime }}
              [start]  {{ item.0.key }}
              s:{{ item.1.started }}
              f:{{ item.1.finished }}
              {{ item.1.ansible_job_id }}
          loop: "{{ results_dict|dict2items|subelements('value.results') }}"
          loop_control:
            label: "{{ item.0.key }} {{ item.1.ansible_job_id }}"

      run_once: true
      delegate_to: localhost

gives

shell> ansible-playbook -e @proc01.yml pb-start.yml

PLAY [all] ***********************************************************************************

TASK [command] *******************************************************************************
changed: [test_11] => (item={'cmd': 'sleep 30', 'async': 45})
changed: [test_13] => (item={'cmd': 'sleep 30', 'async': 45})
changed: [test_11] => (item={'cmd': 'sleep 45', 'async': 60})
changed: [test_13] => (item={'cmd': 'sleep 45', 'async': 60})

TASK [debug] *********************************************************************************
skipping: [test_11]
skipping: [test_13]

TASK [file] **********************************************************************************
ok: [test_11 -> localhost] => (item=/export/scratch/tmp7/test-265/var/run)
ok: [test_11 -> localhost] => (item=/export/scratch/tmp7/test-265/var/log)

TASK [file] **********************************************************************************
ok: [test_11 -> localhost] => (item=test_11)
ok: [test_11 -> localhost] => (item=test_13)

TASK [copy] **********************************************************************************
changed: [test_11 -> localhost] => (item=test_11 571415057700.84160)
changed: [test_11 -> localhost] => (item=test_11 924759903126.84193)
changed: [test_11 -> localhost] => (item=test_13 551498199552.84159)
changed: [test_11 -> localhost] => (item=test_13 976946831378.84194)

TASK [lineinfile] ****************************************************************************
changed: [test_11 -> localhost] => (item=test_11 571415057700.84160)
changed: [test_11 -> localhost] => (item=test_11 924759903126.84193)
changed: [test_11 -> localhost] => (item=test_13 551498199552.84159)
changed: [test_11 -> localhost] => (item=test_13 976946831378.84194)

PLAY RECAP ***********************************************************************************
test_11: ok=5    changed=3    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
test_13: ok=1    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

shell> cat var/log/async.log 
2023-03-21 05:43:36 [start]  test_11 s:1 f:0 571415057700.84160
2023-03-21 05:43:36 [start]  test_11 s:1 f:0 924759903126.84193
2023-03-21 05:43:36 [start]  test_13 s:1 f:0 551498199552.84159
2023-03-21 05:43:37 [start]  test_13 s:1 f:0 976946831378.84194

shell> tree var/run/
var/run/
├── test_11
│   ├── 571415057700.84160
│   └── 924759903126.84193
└── test_13
    ├── 551498199552.84159
    └── 976946831378.84194

2 directories, 4 files

Wait for the processes to finish, remove the job ID, and write the log

shell> cat pb-status.yml
- hosts: localhost

  vars:

    procs_cmd: "cd {{ var_run }}; ls -1 *"
    procs_dict: "{{ dict(lookup('ansible.builtin.pipe', procs_cmd)|
                         community.general.jc('ls')|
                         groupby('parent')) }}"

# Optionally, select and/or deny job IDs
#                        selectattr('filename', 'in' , jid_allow|d([]))|
#                        rejectattr('filename', 'in' , jid_deny|d([]))|

  tasks:
    - debug:
        var: procs_dict

    - name: Wait for processes to finish and write log
      block:

        - debug:
            msg: |
              host: {{ item.0.key }} jid: {{ item.1.filename }}
          loop: "{{ procs_dict|dict2items|subelements('value') }}"
          loop_control:
            label: "{{ item.0.key }}"
          when: debug|d(false)|bool

        - async_status:
            jid: "{{ item.1.filename }}"
          loop: "{{ procs_dict|dict2items|subelements('value') }}"
          loop_control:
            label: "{{ item.0.key }}"
          delegate_to: "{{ item.0.key }}"
          register: async_poll
          until: async_poll.finished
          retries: 999
        - debug:
            var: async_poll
          when: debug|d(false)|bool

        - lineinfile:
            create: true
            dest: "{{ var_log }}/{{ async_log }}"
            line: >-
              {{ '%Y-%m-%d %H:%M:%S'|strftime }}
              [status] {{ item.item.0.key }}
              s:{{ item.started }}
              f:{{ item.finished }}
              {{ item.ansible_job_id }}
          loop: "{{ async_poll.results }}"
          loop_control:
            label: "{{ item.item.1.parent }} {{ item.item.1.filename }}"

    - name: Remove finished processes and write log
      block:

        - file:
            state: absent
            path: "{{ var_run }}/{{ item.item.0.key }}/{{ item.ansible_job_id }}"
          loop: "{{ async_poll.results|selectattr('finished', 'eq', 1) }}"
          loop_control:
            label: "{{ item.item.1.parent }} {{ item.item.1.filename }}"

        - lineinfile:
            create: true
            dest: "{{ var_log }}/{{ async_log }}"
            line: >-
              {{ '%Y-%m-%d %H:%M:%S'|strftime }}
              [remove] {{ item.item.0.key }}
              s:{{ item.started }}
              f:{{ item.finished }}
              {{ item.ansible_job_id }}
          loop: "{{ async_poll.results|selectattr('finished', 'eq', 1) }}"
          loop_control:
            label: "{{ item.item.1.parent }} {{ item.item.1.filename }}"

      when: remove_finished|d(false)|bool

gives

shell> ansible-playbook pb-status.yml -e remove_finished=true

PLAY [localhost] *****************************************************************************

TASK [debug] *********************************************************************************
skipping: [localhost] => (item=test_11) 
skipping: [localhost] => (item=test_11) 
skipping: [localhost] => (item=test_13) 
skipping: [localhost] => (item=test_13) 
skipping: [localhost]

TASK [async_status] **************************************************************************
changed: [localhost -> test_11] => (item=test_11)
changed: [localhost -> test_11] => (item=test_11)
changed: [localhost -> test_13] => (item=test_13)
changed: [localhost -> test_13] => (item=test_13)

TASK [debug] *********************************************************************************
skipping: [localhost]

TASK [lineinfile] ****************************************************************************
changed: [localhost] => (item=test_11 571415057700.84160)
changed: [localhost] => (item=test_11 924759903126.84193)
changed: [localhost] => (item=test_13 551498199552.84159)
changed: [localhost] => (item=test_13 976946831378.84194)

TASK [file] **********************************************************************************
changed: [localhost] => (item=test_11 571415057700.84160)
changed: [localhost] => (item=test_11 924759903126.84193)
changed: [localhost] => (item=test_13 551498199552.84159)
changed: [localhost] => (item=test_13 976946831378.84194)

TASK [lineinfile] ****************************************************************************
changed: [localhost] => (item=test_11 571415057700.84160)
changed: [localhost] => (item=test_11 924759903126.84193)
changed: [localhost] => (item=test_13 551498199552.84159)
changed: [localhost] => (item=test_13 976946831378.84194)

PLAY RECAP ***********************************************************************************
localhost: ok=4    changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

shell> cat var/log/async.log 
2023-03-21 05:43:36 [start]  test_11 s:1 f:0 571415057700.84160
2023-03-21 05:43:36 [start]  test_11 s:1 f:0 924759903126.84193
2023-03-21 05:43:36 [start]  test_13 s:1 f:0 551498199552.84159
2023-03-21 05:43:37 [start]  test_13 s:1 f:0 976946831378.84194
2023-03-21 05:46:58 [status] test_11 s:1 f:1 571415057700.84160
2023-03-21 05:46:58 [status] test_11 s:1 f:1 924759903126.84193
2023-03-21 05:46:58 [status] test_13 s:1 f:1 551498199552.84159
2023-03-21 05:46:58 [status] test_13 s:1 f:1 976946831378.84194
2023-03-21 05:47:00 [remove] test_11 s:1 f:1 571415057700.84160
2023-03-21 05:47:00 [remove] test_11 s:1 f:1 924759903126.84193
2023-03-21 05:47:00 [remove] test_13 s:1 f:1 551498199552.84159
2023-03-21 05:47:00 [remove] test_13 s:1 f:1 976946831378.84194

shell> tree var/run/
var/run/
├── test_11
└── test_13

2 directories, 0 files

You can selectively deny IDs. For example, create a list

shell> cat jid_deny.yml 
jid_deny:
  - '424790592066.84404'
  - '828365727638.84439'

and update the declaration of the dictionary

    procs_dict: "{{ dict(lookup('ansible.builtin.pipe', procs_cmd)|
                         community.general.jc('ls')|
                         rejectattr('filename', 'in' , jid_deny|d([]))|
                         groupby('parent')) }}"

Then use the list in the run string

shell> ansible-playbook pb-status.yml -e remove_finished=true -e @jid_deny.yml

...

In the same way, you can selectively allow IDs. Create a list jid_allow and update the declaration

    procs_dict: "{{ dict(lookup('ansible.builtin.pipe', procs_cmd)|
                         community.general.jc('ls')|
                         selectattr('filename', 'in' , jid_allow|d([]))|
                         rejectattr('filename', 'in' , jid_deny|d([]))|
                         groupby('parent')) }}"
+ 0
Score:0
Matt Blaha avatar Server
dj flag
Matt Blaha

It's a little hard to say without some code and more info about your environment, but assuming the remote host is a modern Linux distro, I'd look at letting systemd do much of the work for me.

Have a look at one-shot services:

https://www.redhat.com/sysadmin/systemd-oneshot-service

Ansible can just do the work of managing the service, and check on its status asynchronously.

+ 1
br flag
Vladimir Botka
Thank you for the link. systemd would be an overkill here. There are also modern BSD distros that don't use systemd. The goal is to keep the list of async processes on clients as simple as possible.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.