How to send OpenShift alerts to Gmail
Lately I’ve been exploring new monitoring capabilities which are built-in the OpenShift product — without the need for a 3rd party monitoring tool. Even though most of the times 3rd party companies develop more advanced monitoring tools, the purpose of this blog is to show off one of the many out of the box monitoring features that come with OpenShift.
In this blog, I’ll be going step by step through the procedure of integrating the OpenShift monitoring stack to a Gmail account.
Why would I do that?
I think that the reasons for integrating a platform with a an external and independent source are quite obvious -
- Gmail will aggregate alerts from the past and the present. Making it easier to trace a reoccurring error.
- Alerts that were sent during off-work time will be saved in the mailbox, while they can disappear in the OpenShift console.
- An external source does not depend on the stability of the OpenShift cluster. The cluster can be down, but the administrator could still debug the errors received by the Alert Manager.
- A central Gmail account can manage multiple Alert Managers. Meaning that you could connect multiple OpenShift clusters to it, or even standalone Alert Managers that perform non-OpenShift related monitoring.
NOTE: while this blog is about connecting an OpenShift Alert Manager to Gmail, it is important to note that you can integrate Alert Manager with many other systems, like Telegram, Discord, AWS SNS and more. To read about Alert Manager integrations go to — https://prometheus.io/docs/operating/integrations/
Let’s get to it
Now that we know why would we need to integrate Alert Manager with a Gmail account. Let’s go through the configuration needed for it to happen!
Prerequisites —
- OpenShift 4.4+.
- Internet connection.
- oc CLI tool.
Procedure —
- We will need to create a Gmail account that Alert Manager will use to send emails on its behalf. Since it will not be too secure to store the account’s password in OpenShift’s Alert Manager configuration, it is better to generate an app password for Alert Manager at — https://support.google.com/accounts/answer/185833?hl=en
- Now that we have an account, lets generate Alert Manager’s configuration.
- smtp_from, smtp_auth_username and smtp_auth_password describe the source user’s information. Alert Manager uses the credentials to send an email through the mentioned account.
- The to field describes the destination email to which alert manager will forward to alerts.
- The severity field describes the severity of the alerts that will be send to Gmail. In my case, I’m using critical, but warning can be used as well.
- Now let’s apply the configuration onto the Alert Manager. Use the file created in the previous step.
- Wait for Alert Manager to reload it’s configuration
And you are done! Alert Manager should now redirect critical errors to your Gmail account.
Testing —
NOTE: DO NOT PERFORM THIS TEST IN PRODUCTION ENVIRONMENTS
To test the integration, let’s simulate a scenario in which a filesystem runs out of space. To perform the test, I will create a large file in the /tmp directory on one of the nodes in my OpenShift cluster. Creating a large file will trigger a NodeFilesystemAlmostOutOfSpace Alert in OpenShift, which will get sent to my Gmail mailbox.
- Access the console of one of the worker nodes in the OpenShift cluster —
- Create a large file in the /tmp directory (leave less than 3% in the filesystem to trigger the alert) —
- Check that the filesystem is indeed full —
- Make sure that an alert has been triggered in the OpenShift Web console (Using an administrative user, navigate to Monitoring -> Alerting) —
- Log into your Gmail mailbox, and you will be able to see the alert!
Conclusion
In the article we went through some of the basics of OpenShift’s monitoring stack. You should now be able to integrate OpenShift’s Alert Manager with your Gmail account or even your organization’s mailing server.
Make sure to check out the rest of OpenShift’s monitoring features at —
https://docs.openshift.com/container-platform/4.6/monitoring
And along the way, make sure familiarize yourself with Prometheus, to fully understand how monitoring in OpenShift works out of the box!
