private GitLab instance for personal projects

  • Matt Bostock's avatar
    Send AlertManager notifications to Elasticsearch · 407da025
    Matt Bostock authored
    Add a new service, `alertmanager2es`, which receives [HTTP webhook
    notifications][] from [AlertManager][] and inserts them into an
    [Elasticsearch][] index for searching and analysis.
    
    Notifications are received on the `/webhook` endpoint.
    
    Prometheus metrics are exposed on `/metrics`.
    
    Note that this service will not capture silenced or inhibited alerts
    currently.
    
    I used a custom type for `groupKey` so that the `groupKey` field can be
    marshalled into JSON as a string, since Elasticsearch does not currently
    support unsigned integers:
    
    https://github.com/elastic/elasticsearch/issues/13951
    
    I tested extensively to try to map the integer to a string using
    Elasticsearch's mappings, which I succeeded in doing, however Kibana
    then failed to display the documents because Elasticsearch's
    highlighting could not handle the unsigned integer stored in `_source`:
    
        RemoteTransportException[[Eternal Brain][172.17.0.2:9300][indices:data/read/search[phase/query+fetch]]]; nested: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
        Caused by: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
                at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:123)
                at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:140)
                at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:188)
                at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:490)
                at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:392)
                at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:389)
                at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
                at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
                at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
    
    The dependencies for this code are vendored using gvt.
    
    [AlertManager]: https://github.com/prometheus/alertmanager
    [Elasticsearch]: https://www.elastic.co/products/elasticsearch
    [HTTP webhook]: https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>
    407da025
This project is licensed under the Apache License 2.0. Learn more