private GitLab instance for personal projects

  1. 24 Oct, 2019 1 commit
  2. 01 Oct, 2018 2 commits
  3. 06 Mar, 2018 1 commit
  4. 10 May, 2017 4 commits
  5. 13 Apr, 2017 1 commit
    • Matt Bostock's avatar
      Send AlertManager notifications to Elasticsearch · 407da025
      Matt Bostock authored
      Add a new service, `alertmanager2es`, which receives [HTTP webhook
      notifications][] from [AlertManager][] and inserts them into an
      [Elasticsearch][] index for searching and analysis.
      
      Notifications are received on the `/webhook` endpoint.
      
      Prometheus metrics are exposed on `/metrics`.
      
      Note that this service will not capture silenced or inhibited alerts
      currently.
      
      I used a custom type for `groupKey` so that the `groupKey` field can be
      marshalled into JSON as a string, since Elasticsearch does not currently
      support unsigned integers:
      
      https://github.com/elastic/elasticsearch/issues/13951
      
      I tested extensively to try to map the integer to a string using
      Elasticsearch's mappings, which I succeeded in doing, however Kibana
      then failed to display the documents because Elasticsearch's
      highlighting could not handle the unsigned integer stored in `_source`:
      
          RemoteTransportException[[Eternal Brain][172.17.0.2:9300][indices:data/read/search[phase/query+fetch]]]; nested: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
          Caused by: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
                  at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:123)
                  at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:140)
                  at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:188)
                  at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:490)
                  at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:392)
                  at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:389)
                  at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
                  at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
                  at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
                  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
      
      The dependencies for this code are vendored using gvt.
      
      [AlertManager]: https://github.com/prometheus/alertmanager
      [Elasticsearch]: https://www.elastic.co/products/elasticsearch
      [HTTP webhook]: https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>
      407da025