I am trying to configure a Prometheus agent with the 'remote_write' function. The Prometheus agent configuration file has these parameters.
global: scrape_interval: 30s external_labels:
environment: roble
remote_write:
- url: http://dominio.com/api/v1/write
name: hostname
write_relabel_configs:
- source_labels: [__address__]
regex: expensive*
action: keep
scrape_configs:
- job_name: 'agente-telegraf'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9273']
The metrics I get for now are: cpu, mem, disk and I get them with a Telegraf client. With the 'outputs.file' plugin I can see that they are generated without problems.
The problem that I detect is that when these metrics are sent to the frontend, the requests arrive but with nothing, I can see it in this log line that it indicates that the request arrived empty (0 bytes).
X.X.X.X - - [17/Feb/2023:19:34:57 -0300] "POST /api/v1/write HTTP/1.1" 204 -
The agent log says this and I don't really see that it's reporting any errors:
ts=2023-02-17T23:19:50.512Z caller=dedupe.go:112 component=remote level=debug remote_name=hostname url=http://dominio.com/api/v1/write msg=QueueManager.calculateDesiredShards dataInRate=21.12 dataOutRate=35.2 dataKeptRatio=1 dataPendingRate=-14.080000000000002 dataPending=105.60000000000001 dataOutDuration=0.0021454153 timePerSample=6.094929829545454e-05 desiredShards=0.001609061475 highestSent=1.676675982e+09 highestRecv=1.676675987e+09
ts=2023-02-17T23:19:50.512Z caller=dedupe.go:112 component=remote level=debug remote_name=hostname url=http://dominio.com/api/v1/write msg=QueueManager.updateShardsLoop lowerBound=0.7 desiredShards=0.001609061475 upperBound=1.3
Can you guide me regarding the configuration of the prometheus agent please. This is my first prometheus adventure and surely there is something wrong with the prometheus agent configuration file.
Thank you very much in advance.
