Carlos Aguni

Highly motivated self-taught IT analyst. Always learning and ready to explore new skills. An eternal apprentice.


Prometheus Out of Order Remote Write

14 Jun 2022 »

Prometheus update 14feb24

[tsdb] Ingest out of order samples and samples from a few hours ago #8535 https://github.com/prometheus/prometheus/issues/8535

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1948

Mimir

https://grafana.com/blog/2022/09/07/new-in-grafana-mimir-introducing-out-of-order-sample-ingestion/

Mirabelle

https://www.mcorbin.fr/posts/2022-03-28-mirabelle-0-10-0/

https://mirabelle.mcorbin.fr/api/

https://mirabelle.mcorbin.fr/howto/stream/

solve out of order error

ts=2022-05-14T03:27:27.753Z caller=write_handler.go:102 level=error component=web msg="Out of order sample from remote write" err="out of order sample" series="{__name__=\"cpu_min\", thread=\"t06\"}" timestamp=1652498774953
ts=2022-05-14T03:27:27.757Z caller=write_handler.go:102 level=error component=web msg="Out of order sample from remote write" err="out of order sample" series="{__name__=\"cpu_min\", thread=\"t03\"}" timestamp=1652498774953
ts=2022-05-14T03:27:27.767Z caller=write_handler.go:102 level=error component=web msg="Out of order sample from remote write" err="out of order sample" series="{__name__=\"cpu_min\", thread=\"t07\"}" timestamp=1652498753917

https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent

Please download vmutils-* archive from releases page

./vmagent-prod -remoteWrite.url=http://localhost:9090/api/v1/write -remoteWrite.queues=1