others-how to solve the problem of `prometheus alertmanager failed to start `?

Sep 5, 2022

1. Purpose

problem

prometheus alertmanager failed to start

^C[bswen@iZ235g0vxqjZ alertmanager]$ tail -f nohup.out
ts=2022-08-31T07:33:46.444Z caller=main.go:232 level=info build_context="(go=go1.17.8, user=root@265f14f5c6fc, date=20220325-09:31:33)"
ts=2022-08-31T07:33:46.445Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=10.117.134.201 port=9094
ts=2022-08-31T07:33:46.455Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2022-08-31T07:33:46.495Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=./alertmanager.yml
ts=2022-08-31T07:33:46.496Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=./alertmanager.yml
ts=2022-08-31T07:33:46.499Z caller=main.go:535 level=info msg=Listening address=:9093
ts=2022-08-31T07:33:46.499Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
ts=2022-08-31T07:33:48.456Z caller=cluster.go:705 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000595828s
ts=2022-08-31T07:33:49.955Z caller=main.go:574 level=info msg="Received SIGTERM, exiting gracefully..."
ts=2022-08-31T07:33:49.955Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=1 elapsed=3.500199861s

start.sh

nohup /opt/alertmanager/alertmanager-0.24.0.linux-amd64/alertmanager --config.file=./alertmanager.yml > nohup.out &

2. Solution

Here is the solution: disable alertmanager cluster mode:

Turn off high availability If running Alertmanager in high availability mode is not desired, setting –cluster.listen-address= prevents Alertmanager from listening to incoming peer requests.

the final start.sh

nohup /opt/alertmanager/alertmanager-0.24.0.linux-amd64/alertmanager --config.file=./alertmanager.yml --cluster.listen-address= > nohup.out &

now alertmanager started ok:

[bswen@iZ235g0vxqjZ alertmanager]$ ./start.sh
[bswen@iZ235g0vxqjZ alertmanager]$ nohup: redirecting stderr to stdout

[bswen@iZ235g0vxqjZ alertmanager]$
[bswen@iZ235g0vxqjZ alertmanager]$ tail -f nohup.out
ts=2022-08-31T07:37:19.252Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=HEAD, revision=f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)"
ts=2022-08-31T07:37:19.252Z caller=main.go:232 level=info build_context="(go=go1.17.8, user=root@265f14f5c6fc, date=20220325-09:31:33)"
ts=2022-08-31T07:37:19.300Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=./alertmanager.yml
ts=2022-08-31T07:37:19.300Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=./alertmanager.yml
ts=2022-08-31T07:37:19.304Z caller=main.go:535 level=info msg=Listening address=:9093
ts=2022-08-31T07:37:19.305Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false

3. Summary

In this post, I demonstrated how to solve the startup problem of prometheus alertmanager. That’s it, thanks for your reading.