Score:0

Grafana "Data source connected, but no labels received. Verify that Loki and Promtail is configured properly."

mv flag

I'm trying to install loki so that I can read logs on Grafana, but I keep receiving Data source connected, but no labels received. Verify that Loki and Promtail is configured properly. (nothing from the logs).

I'm using docker images for each service (grafana, loki, promtail) on a Raspberry Pi 4 8Gb.

Each service are working properly. I can connect to Grafana (localhost:3000) and see node_exporter data. I can connect to loki itself and see the metrics (localhost:3100/metrics), and I can also connect to promtail (localhost:9080/targets).

On Grafana, I tried pointing loki to 10.88.0.1:3100/metrics (since Grafana is a docker container, I have to use 10.88.0.1 to connect to the host) but I get a new error Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Here is what I get from the logs for that error:

logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204508553Z level=error msg="Failed to call resource" error="404 page not found\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-02-12T13:21:35.204758623Z level=error msg="Request Completed" method=GET path=/api/datasources/2/resources/labels status=500 remote_addr=192.168.1.20 time_ms=7 duration=7.563973ms size=51 referer=http://raspberry.local:3000/datasources/edit/sgEZvn14k handler=/api/datasources/:id/resources/*

It's weird because without the /metrics Grafana does find loki.

I'm using ansible to provision my rasp :

- name: Grafana - Run container using podman
  containers.podman.podman_container:
      name: grafana
      image: grafana/grafana-oss:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3000:3000"
      memory: "2048m"
      net:
          - host
      volume:
          - "grafana-storage:/var/lib/grafana"

- name: Loki - Run container using podman
  containers.podman.podman_container:
      name: loki
      image: grafana/loki:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports: "3100:3100"
      memory: "2048m"
      net:
          - host
      volume:
          - "loki-storage:/loki"

- name: Promtail - Run container using podman
  containers.podman.podman_container:
      name: promtail
      image: grafana/promtail:latest
      state: started
      recreate: true
      restart_policy: on-failure
      ports:
          - "9080:9080"
          - "1514:1514"
      memory: "2048m"
      net:
          - host
      volume:
          - "{{ lokiData }}/promtail.yml:/etc/promtail/promtail-config.yml"
      command: -config.file=/etc/promtail/promtail-config.yml

My promtail config:

server:
    http_listen_port: 9080
    grpc_listen_port: 0

positions:
    filename: /tmp/positions.yaml

clients:
    - url: http://10.88.0.1:3100/loki/api/v1/push

scrape_configs:
    - job_name: syslog
      syslog:
        listen_address: 0.0.0.0:1514
        labels:
          job: "syslog"
      relabel_configs:
        - source_labels: ['__syslog_message_hostname']
          target_label: 'host'

my loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

I really don't know what to do now, I'm new to Grafana, so maybe I'm missing something. I googled the error, but I found nothing interesting (mainly config mistakes).

Score:0
mv flag

I forgot to add the journal/syslogs volumes. In fact, everything was working, but promtail had no data to send. Make sure to read the positions.yaml file (located by promtail's config).

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.