Score:0

Grafana Loki, AlertManager - unable to read rule dir, open /tmp/loki/rules/fake: no such file or directory

jp flag

I've deployed promtail, Grafana, Loki and AlertManager using Helm charts on k3d cluster on my local machine. I would like to have some rules in Loki such that if something will happen, AlertManager should be informed. Now I tried only with some simple rule, just to check if it works.

My Loki version: {"version":"2.6.1","revision":"6bd05c9a4","branch":"HEAD","buildUser":"root@ea1e89b8da02","buildDate":"2022-07-18T08:49:07Z","goVersion":""}

My Grafana version:

enter image description here

And the Loki configuration:

loki:
  # should loki be deployed on cluster?
  enabled: true

  image:
    repository: grafana/loki
    pullPolicy: Always
    pullSecrets:
      - registry
  priorityClassName: normal
  resources:
    limits:
      memory: 3Gi
      cpu: 0
    requests:
      memory: 0
      cpu: 0
  config:
    chunk_store_config:
      max_look_back_period: 30d
    table_manager:
      retention_deletes_enabled: true
      retention_period: 30d
    query_range:
      split_queries_by_interval: 0
      parallelise_shardable_queries: false
    querier:
      max_concurrent: 2048
    frontend:
      max_outstanding_per_tenant: 4096
      compress_responses: true
    ingester:
      wal:
        enabled: true
        dir: /tmp/wal
    schema_config:
      configs:
        - from: 2022-12-05
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      boltdb_shipper:
        active_index_directory: /tmp/loki/boltdb-shipper-active
        cache_location: /tmp/loki/boltdb-shipper-cache
        cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space 
        shared_store: filesystem
      filesystem:
        directory: /tmp/loki/chunks
    compactor:
      working_directory: /tmp/loki/boltdb-shipper-compactor
      shared_store: filesystem
    ruler:
      storage:
        type: local
        local:
          directory: /tmp/loki/rules/
      ring:
        kvstore:
          store: inmemory
      rule_path: /tmp/loki/rules-temp
      alertmanager_url: http://onprem-kube-prometheus-alertmanager.svc.mylocal-monitoring:9093
      enable_api: true
      enable_alertmanager_v2: true
  write:
    extraVolumeMounts:
      - name: rules-config
        mountPath: /tmp/loki/rules/fake/
    extraVolumes:
      - name: rules-config
        configMap:
          name: rules-cfgmap
          items:
            - key: "rules.yaml"
              path: "rules.yaml"
  read:
    extraVolumeMounts:
      - name: rules-config
        mountPath: /tmp/loki/rules/fake/
    extraVolumes:
      - name: rules-config
        configMap:
          name: rules-cfgmap
          items:
            - key: "rules.yaml"
              path: "rules.yaml"

promtail:
  image:
    registry: docker
    pullPolicy: Always
  imagePullSecrets:
    - name: registry
  priorityClassName: normal
  resources:
    limits:
      memory: 256Mi
      cpu: 0
    requests:
      memory: 0
      cpu: 0
  livenessProbe:
    failureThreshold: 5
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  config:
    snippets:
      pipelineStages:
        - cri: {}
      common:
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: node_name
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - action: replace
          replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__
        - action: replace
          replacement: /var/log/pods/*$1/*.log
          regex: true/(.*)
          separator: /
          source_labels:
            - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
            - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
            - __meta_kubernetes_pod_container_name
          target_label: __path__

monitoring:
  enabled: false

networkPolicies:
  enabled: false

The problem is, when I want to check the rules, doing curl -X GET localhost:3100/loki/api/v1/rules it shows me: unable to read rule dir /tmp/loki/rules/fake: open /tmp/loki/rules/fake: no such file or directory.

So it seems it can't find the rules file.

I also tried to change the config like this:

write:
    extraVolumeMounts:
      - name: rules-conf
        mountPath: /tmp/loki/rules/fake/rules.yaml
    extraVolumes:
      - name: rules-conf
  read:
    extraVolumeMounts:
      - name: rules-conf
        mountPath: /tmp/loki/rules/fake/rules.yaml
    extraVolumes:
      - name: rules-conf

And my configmap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rules-cfgmap
  namespace: mylocal-monitoring
data:
  rules.yaml: |
    groups:
      - name: PrometheusAlertsGroup
    rules:
      - alert: test1
      expr: |
        1 > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: TEST: testing test
          description: test

And the rules file:

groups:
  - name: PrometheusAlertsGroup
  rules:
    - alert: test1
      expr: |
        1 > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: TEST: testing test
          description: test

But the problem is the same. Any ideas?

It finally worked when I created /tmp/loki/rules/fake/rules.yaml manually, but thats not the point to create it manually.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.