I have difficulties understanding their product lineup and roadmap. Everything is named Grafana something. Grafana the company, Grafana the monitoring Frontend, Grafana Agent, Grafana Agent Flow, Grafana Agent Operator, Grafana Operator. It is easy to mix things up especially if you talk to someone who is not in the Grafana world.

They also seem to change their mind quiet fast on how users are supposed to send metrics/logs. First there was the loki, prometheus clients, then Grafana Agent Operator, then Grafana Agent Flow mode and now they announced alloy at kubecon, immediately deprecating other solutions. All of wich do more or less the same - as far as I know.

These iterations are too fast for companies to develop trust in their solution. Even in private test clusters I have trouble catching up. And I have to confess the the new alloy solution did not make a good first impression. I dont't know why it replaces the Agent Flow mode and even worse, instead of sticking to k8s standards it uses its own custom configuration language that does not seem to have any advantages.

I hope they figure something out that works for base tasks and improve that solution over time instead of doing something new every year.

One thing I have found is that certain parts of grafana aren’t really designed for very large k8s clusters. For example if I want to utilize the k8s_sd_config for scraping metrics from pods the agent needs to watch ALL pods in the cluster which for clusters with hundreds of thousands of pods is a no go.

I ended up recently writing my own daemonset agent which queries the kubelet API to get pods on the local node and uses that information to set up the scrape routines. There are other solutions such as scrape managers which do the watching and then tell the agents what endpoints to scrape, but that seems overly complex, and those features are in beta and not in the happy path.

Another thing I would like to do more easily is have different relabel configs for different pods/targets. If you are running a multitennant k8s cluster it is useful to allow tenants to define their own scrape configs because they may want to relabel/rename/drop their metrics. Again, I found the grafana solutions either too complex or they don’t solve my problem so I wrote into my own solution where tenants define their scrape configs in k8s config maps and then the agent consumes those configmaps to set up scraping.