Best way to create yaml files that have a variable number sections based on host

I am trying to use ansible to deploy configuration files out to hundreds of machines in which different machines will have multiple iterations of specific configuration snippets. Specifically I am using the promtail log parser and different machines will have different log file locations to parse with different labels. Ideally I want to keep the ansible configuration pretty simply so I can just use pull requests to make changes to the various sections.

Initially I was going to use group_vars and have each log file location being defined in the group_var. Which works fine as long as I am only building a single log location. Once I need multiple log locations, it breaks as I will only have one value returned from group_vars.

To illustrate.

hosts:
    
    LOGFILE1:
      hosts:
        app[15:16].qa2.example.com
    LOGFILE2:
      hosts
        app[16:17].qa2.example.com



GROUP_VARS/LOGFILE1
GROUP_VARS/LOGFILE2

I could just possibly look to iterate through each group and then append the the output to the config file but I don't see a way to do that with the template function. Ideally I could just iterate through all of the log file locations but I'm not sure how to do that.

Or maybe I could use an external variable file and then use a conditional of some sort to determine which hosts get which configuration?

Same data in the group_vars...

file: /opt/tomcat/fxcts/logs/gxxss.log
comp: TX_Tomcat
app: TX
module: GXX
pipeline_regex:  None 
pipeline_vars:
  - None
drop_expression: None
Multiline: None

Here is the jinja template

scrape_configs:
- job_name: {{ module }}
    pipeline_stages:
        - regex:
            expression: {{ pipeline_regex }}
        - labels:
            {% for labels in pipeline_vars -%}
            {{ labels }}:
            {% endfor %}
{#  This is a test #}
        - timestamp:
            source: date
            format: 2006-01-01 15:00:00.000000
        - drop:
            expression: {{ drop_expression }}
        - multiline:
            firstline: ""
            max_wait_time: 3s
            static_configs:
    static_configs:
    - targets:
        - localhost
      labels:
        app: {{ app }}
        host: {{ ansible_hostname }}
        component: {{ comp }}
        __path__: {{ file }}

Here is a sample of an actual yaml config. As I said the different log locations can vary by host.

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://host:3100/loki/api/v1/push
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      host: ${HOSTNAME}
      __path__: /var/log/*log
- job_name: apps_ssi
  static_configs:
  - targets:
      - localhost
    labels:
      job: ssi
      host: ${HOSTNAME}
      __path__: /opt/tomcat/ssi/logs/*log
- job_name: apps_fxcts
  static_configs:
  - targets:
      - localhost
    labels:
      job: fxcts
      host: ${HOSTNAME}
      __path__: /opt/tomcat/fxcts/logs/*log
- job_name: journal
  journal:
    json: false
    max_age: 12h
    labels:
      job: systemd-journal
      host: ${HOSTNAME}
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'

Again, not sure I follow exactly, but if Roman's answer doesn't work, then I'd suggest using host vars in a folder structure in your inventory and simply creating a variable with a map of path/labels. Then you can loop through that variable in your jinja template.

If there's so many variations of paths and hosts that this is infeasible given the number of machines, then I suggest you look at either trying to standardize locations/labels, or use a tool other than promtail that is able to handle missing files. That way you can just manage one large list of logs to scrape and not worry about changing anything.

The final option I can suggest is to reverse the approach: find a tool that allows you to use a folder for configuration and drop in files for each tool that needs monitoring. What I mean by this is that you'd have a folder /etc/logscraper/conf.d which is included in the config. Then each tool would create a file inside that e.g. /etc/logscraper/conf.d/10-tool.conf, and it would define how to parse the logs. That way you can deploy that with the tool itself, not the logging tool. This has the added benefit of keeping configuration related to a product/tool (e.g. apache) within the playbook that is deploying the product.

I'm sorry but it's just too much to read through and completely grasp but in essence I get that you need to push different config to different hosts based on some log file locations to parse.

My solution to this - to catch them all - would be to loop through all the paths you need to monitor via promtail log parser and adjust config accordingly:

---
- name: "playbook to add specific config files"
  hosts: localhost


  tasks:
  
  - name: Block for promtail log parser conf files
    block:
      - name: Check if log path exists
        stat:
          path: "{{ item.path }}"
        register: reg_path
      - name: Push conf file if path exists
        template:
          src: "{{ item.templ }}"
          dest: "/etc/promtal/whatever.d/"
        when: reg_path.stat.exists
    loop:
      - { name: 'syslog', path: '/var/log/syslog', templ: 'syslog.cong.j2'}
      - { name: 'messages', path: '/var/log/messages', templ: 'messages.cong.j2'}
      - { name: 'apache-acl', path: '/var/log/httpd/access.log', templ: 'apache-acl.cong.j2'}

This YAML code is not tested and promtail config may need to be manipulated differently but hopefully you get my point.