Ansible: How to merge list of dictionary objects with the same key getting one dictionary element with a list of thouse different values grouped?

Juan M.Victoria

11/21/23, 8:08 PM

I have been struggled with Ansible for a time and finally requesting for any help if possible,

Having a set of Nifi policies:

"nifi_raw_policies": [
    {
        "action": "read",
        "group": "8c052e6c-0184-1000-0000-000072a0bb44",
        "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
    },
    {
        "action": "read",
        "group": "8c0536d8-0184-1000-0000-000018ba98a9",
        "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
    }
]

I have the need to group them to obtain:

"nifi_grouped_policies": [
    {
        "policy": "read/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa",
        "groups": [ "8c052e6c-0184-1000-0000-000072a0bb44","8c0536d8-0184-1000-0000-000018ba98a9" ]
    }
]

But until now I'm still unable to group by "action"+"resource" key getting a list of related "group" in a new field "groups". I have managed to obtain only a list of dictionaries in the form { "action"+"resource" : "group1" }, ..., { "action"+"resource" : "groupN" }

Related ansible task code:

- name: Declare policies (simplest)
  set_fact:
    nifi_raw_policies: [
        {
            "action": "read",
            "group": "8c052e6c-0184-1000-0000-000072a0bb44",
            "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
        },
        {
            "action": "read",
            "group": "8c0536d8-0184-1000-0000-000018ba98a9",
            "resource": "/data/process-groups/8b6b5c9f-0184-1000-57e3-f27fc56dd4aa"
        }
    ]

- name: Declare policies - debug
  debug: var=nifi_raw_policies

- name: Combine action and resource
  set_fact:
    nifi_policies: >-
      {{ nifi_policies|default([])
        | union([{
            item.action + item.resource : item.group
            }])
      }}
  with_items: "{{ nifi_raw_policies }}"

- name: Combine action and resource - debug
  debug: var=nifi_policies

After several tries using combine, union, groupby ansible filters without success.. any insigth would be highly appreciated.

Best regards,

1717

1 + 0

ansible

ansible-playbook

Score:1

Server

Vladimir Botka

11/21/23, 11:26 PM

Create the structure by Jinja. For example, given the mre simplified data

  raw:
    - action: read
      group: 1
      resource: data
    - action: read
      group: 2
      resource: data

declare the dictionary how to rename the keys

    keys:
      group: groups
      resource: policy

and declare the structure

  grouped: |
    {% for i in raw|groupby('action') %}
    {{ i.0 }}:
    {% for k in i.1|json_query('[].keys(@)|[]')|unique %}
    {% if k != 'action' %}
      {{ keys[k] }}: {{ i.1|map(attribute=k)|unique }}
    {% endif %}
    {% endfor %}
    {% endfor %}

gives

  grouped: |-
    read:
      groups: [1, 2]
      policy: ['data']

You can get the values only

  grouped_vals: "{{ grouped|from_yaml|dict2items|map(attribute='value')|list }}"

gives

  grouped_vals:
    - groups: [1, 2]
      policy: [data]

The first item from the list is what you're looking for. Or, you can get it from the dictionary

  nifi_grouped_policies: "{{ (grouped|from_yaml).read }}"

gives

  nifi_grouped_policies:
    groups: [1, 2]
    policy: [data]

^{Example of a complete playbook for testing
- hosts: localhost

vars:

raw:
- action: read
group: 1
resource: data
- action: read
group: 2
resource: data

keys:
group: groups
resource: policy
grouped: |
{% for i in raw|groupby('action') %}
{{ i.0 }}:
{% for k in i.1|json_query('[].keys(@)|[]')|unique %}
{% if k != 'action' %}
{{ keys[k] }}: {{ i.1|map(attribute=k)|unique }}
{% endif %}
{% endfor %}
{% endfor %}
grouped_vals: "{{ grouped|from_yaml|dict2items|map(attribute='value')|list }}"
nifi_grouped_policies: "{{ (grouped|from_yaml).read }}"

tasks:

- debug:
var: grouped
- debug:
var: grouped_vals|to_yaml
- debug:
var: nifi_grouped_policies|to_yaml}

+ 1

Juan M.Victoria

11/22/23, 11:43 AM

Great Vladimir! I hadn't even considered doing the transformation in Jinja2 directly. Much thanks by your quick and elightening response.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Ansible: How to merge list of dictionary objects with the same key getting one dictionary element with a list of thouse different values grouped?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.