0

In a setup with promtail, loki and grafana, no data shows up in grafana explore. All three parts are running on the same machine.

Question is: where to look for an error in the setup?

Promtail

promtail.yaml:

server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /var/db/promtail/positions.yaml clients: - url: http://127.0.0.1:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: varlogs __path__: /var/log/**/*log

This seems to be fine. Testing:

promtail --config.file=promtail.yaml --log.level=info -print-config-stderr

--- # Loki Config # (version=, branch=, revision=unknown) global: file_watch_config: min_poll_frequency: 250ms max_poll_frequency: 250ms server: http_listen_network: tcp http_listen_address: "" http_listen_port: 9080 [...] level=info ts=2023-12-13T16:00:24.711575585Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=169d8cc02367d196d7ba24694ea90c5d level=info ts=2023-12-13T16:00:24.712927375Z caller=server.go:322 http=[::]:9080 grpc=[::]:32729 msg="server listening on addresses" level=info ts=2023-12-13T16:00:24.713264874Z caller=main.go:174 msg="Starting Promtail" version="(version=, branch=, revision=unknown)" level=warn ts=2023-12-13T16:00:24.713474024Z caller=promtail.go:263 msg="enable watchConfig" level=info ts=2023-12-13T16:00:29.714075081Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/**/*log:{job=\"varlogs\"}" level=info ts=2023-12-13T16:00:29.716662721Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/loki level=info ts=2023-12-13T16:00:29.716703277Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/promtail level=info ts=2023-12-13T16:00:29.716812088Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/gitea level=info ts=2023-12-13T16:00:29.716857702Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/grafana level=info ts=2023-12-13T16:00:29.717132151Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/redis level=info ts=2023-12-13T16:00:29.717201082Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/caddy level=info ts=2023-12-13T16:00:29.717280021Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log ts=2023-12-13T16:00:29.720424446Z caller=log.go:168 level=info msg="Seeked /var/log/caddy/caddy.log - &{Offset:606936482 Whence:0}" level=info ts=2023-12-13T16:00:29.720534804Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/caddy/caddy.log ts=2023-12-13T16:00:29.720631282Z caller=log.go:168 level=info msg="Seeked /var/log/gitea/gitea.log - &{Offset:8194 Whence:0}" level=info ts=2023-12-13T16:00:29.72066379Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/gitea/gitea.log ts=2023-12-13T16:00:29.720728343Z caller=log.go:168 level=info msg="Seeked /var/log/grafana/grafana.log - &{Offset:38901 Whence:0}" level=info ts=2023-12-13T16:00:29.720756759Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/grafana/grafana.log [...]

Loki

loki.yaml

auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 common: instance_addr: 127.0.0.1 path_prefix: /var/db/loki storage: filesystem: chunks_directory: /var/db/loki/chunks rules_directory: /var/db/loki/rules replication_factor: 1 ring: kvstore: store: inmemory query_range: results_cache: cache: embedded_cache: enabled: true max_size_mb: 100 schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h

The loki application / service is up and running. Anything else I could be testing here?

Grafana

Data source for loki created with:

  • Type: Loki
  • URL: http://127.0.0.1:3100 (everything else: left at default settings)

Looking at grafana's /explore only leads to no data being shown, even without any filter criteria. Pressing Run query just doesn't seem to have any effect.

The grafana log looks good to me:

tail /var/log/grafana/grafana.log

logger=grafana.update.checker t=2023-12-13T16:31:16.759743533+01:00 level=info msg="Update check succeeded" duration=486.885735ms logger=cleanup t=2023-12-13T16:41:16.24610575+01:00 level=info msg="Completed cleanup jobs" duration=13.483743ms logger=plugins.update.checker t=2023-12-13T16:41:16.617294501+01:00 level=info msg="Update check succeeded" duration=334.848905ms logger=grafana.update.checker t=2023-12-13T16:41:16.657803737+01:00 level=info msg="Update check succeeded" duration=387.307337ms logger=cleanup t=2023-12-13T16:51:16.240282272+01:00 level=info msg="Completed cleanup jobs" duration=7.750284ms logger=grafana.update.checker t=2023-12-13T16:51:16.292110778+01:00 level=info msg="Update check succeeded" duration=21.261137ms logger=plugins.update.checker t=2023-12-13T16:51:16.313163122+01:00 level=info msg="Update check succeeded" duration=31.295959ms logger=cleanup t=2023-12-13T17:01:16.220217724+01:00 level=info msg="Completed cleanup jobs" duration=4.195714ms logger=plugins.update.checker t=2023-12-13T17:01:16.312587838+01:00 level=info msg="Update check succeeded" duration=30.754461ms logger=grafana.update.checker t=2023-12-13T17:01:16.333881656+01:00 level=info msg="Update check succeeded" duration=63.366615ms

Pointers on what might be missing here are appreciated.

Improve this question

1 Answer 1

Reset to default
1

Explore for Loki without any query doesn't suppose to provide anything: you are expected to provide a valid query.

You can try something like {host=~".+"}, as a starting point.

See this two examples:

Improve this answer
6
  • Thanks, very good pointer. Also, the example link shows it pretty well. "My" grafana requires at least 1 label filter, though. However, the respective filter dropdown only shows "No options found". Fow now, I don't see how to make that label filter optional. But as play.grafana shows, that seems to be possible. Need to find more time to invest. Commented Dec 18, 2023 at 10:41
  • @Windowlicker, what do you mean? Any Loki requires at least on label in a selector. You cannot use selector without labels (or no selector at all): that what demos supposed to show. Commented Dec 18, 2023 at 11:13
  • Oh, labels are required? (Maybe, Loki isn't as clean and simple as expected. Maybe for this "a couple of services on a single server"-setup over here, a mere "tail -f" and "grep" are a valid solution.) Commented Dec 18, 2023 at 21:43
  • Well, this is only for you to decide. On the other hand you'll have a couple labels like host and probably filename anyway, so I don't see how selector like {host=~".+"} (or {filename=~".+"}) might be a problem. @Windowlicker Commented Dec 18, 2023 at 21:47
  • Valid point. According to the Grafana over here, Loki doesn't have any labels. Or at least there is no results for =~".+" on any possible label. As the Promtail output above looks good (to me), a look into what's received by Loki might be suitable. Any chance of doing that? Besides staring at Grafana's /explore? Commented Dec 19, 2023 at 22:30

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.