Kelson Martins Blog

Introduction

Fluentd is an open-source log collector that is getting an immense traction recently, mostly due to its ability to centralize log flow through its vast amount of plugins.
Saying that on this post a docker-compose containing EFK images will be deployed to explore fluentd usage as a replacement for Logstash in a scenario aimed at Docker logging.
Before proceeding, ensure that docker and docker-compose are installed.
I am assuming that readers of this particular post have basic knowledge of docker-compose but if you are new to the technology or need a refresh on its concepts, this post from CodeShip.com has a great summarized introduction to it.

Step 1) Overview of docker-compose.yml

First, let’s analyze our docker-compose.yml, which as mentioned contains the images for the EFK Stack components.
version: '2'
services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION      
    volumes:
      - /var/lib/elasticsearch:/usr/share/elasticsearch/data      
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"      
    networks:
      - efk         
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: efk.elasticsearch
    depends_on:
      - fluentd        
  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION        
    volumes:
      - ./kibana/config/:/usr/share/kibana/config
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - efk
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: efk.kibana  
  fluentd:
    build:
      context: ./fluentd
      args:
        FLUENTD_VERSION: $FLUENTD_VERSION      
    volumes:
      - ./fluentd/conf:/fluentd/etc
    networks:
      - efk
    ports:
      - "24224:24224"
      - "24224:24224/udp"
networks:
  efk:
    driver: bridge

With our docker-compose, the following .env file exists to define the image versions to be used (you may change to match any available version):

ELK_VERSION=6.0.0
FLUENTD_VERSION=v0.14
The point of interest here is the logging statement on Kibana and Elasticserach. By default, docker uses syslog logging driver and our compose is overriding original docker logging settings to use fluentd as logging driver.
Log events will be dispatched with the stated tags for each container, where this is helpful when we want to match and filter logs from different containers in fluentd.
Port 24224 is the default port that fluentd uses in the communication with docker daemon.
If you want to confirm th default docker logging driver, you can perform the following on your docker host:
[root@localhost elk_fluentd]# docker info | grep Logging
Logging Driver: syslog
Also, note that for Kibana and Elasticsearch containers we are mounting volumes to point to their respective configuration files. As these files are out of the scope of this article, they will not be approached here but you can find the complete code available here.

Step 2) Overview of the Dockerfiles

As for the Dockerfile for each component, we will be using official images for Elasticsearch and Kibana, while having a slightly customized Dockerfile for the official fluentd image, as we require the installation of the fluent-plugin-elasticsearch which is not included in fluentd by default.
Please refer to the Elasticsearch, Kibana and Fluentd Dockerfiles respectively:
#Elasticsearch Dockerfile
ARG ELK_VERSION
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:${ELK_VERSION}
#Kibana Dockerfile
ARG ELK_VERSION
FROM docker.elastic.co/kibana/kibana-oss:${ELK_VERSION}
#Fluentd Dockerifle
ARG FLUENTD_VERSION
FROM fluent/fluentd:${FLUENTD_VERSION}
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri", "--version", "2.4.0"]
As mentioned, the point deserving special mention is the installation of the fluent-plugin-elasticsearch plugin version 2.4.0, which is the current latest version at the time that this is being written.

Step 3) Overview of the Fluentd configuration file

Let’s now analyze our fluentd conf file, where we have 2sections, one to receive logs from the docker logging driver (port 24224) and the second to forward these events to Elasticsearch.
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.*>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix logstash
    logstash_dateformat %Y-%m-%d
    include_tag_key true
  </store>
  <store>
    @type stdout
  </store>
</match>
On our particular case, we are matching all events with the <match *.*> statement but we could be able to forward events from Kibana and Elasticsearch to different destinations using the tags mentioned in step 1. If this was the use case, we would match logs from Kibana and Elasticsearch containers using <match efk.kibana> and <match efk.elasticsearch> match statements respectively.

Step 4) Starting docker-compose

With our configuration properly set up, we can start our docker-compose.
[root@localhost]# docker-compose up

Once the containers are started, you will see messages similar to:

elasticsearch_1  | WARNING: no logs are available with the 'fluentd' log driver
kibana_1         | WARNING: no logs are available with the 'fluentd' log driver
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: starting fluentd-0.14.25 pid=5 ruby="2.3.5"
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: gem 'fluentd' version '0.14.25'
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: adding match pattern="*.*" type="copy"
fluentd_1        | 2017-12-07 08:33:35 +0000 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
fluentd_1        | 2017-12-07 08:33:36 +0000 [info]: adding source type="forward"
The above messages are interesting, note that for the Elasticsearch and Kibana containers, we do not receive logs on STDOUT. The reason for that is that in our docker-compose we have overwritten the logging driver for these 2 containers to fluentd, thus we do not have the ability to visualize them in STDOUT.
As for the fluentd container, its logs are still being written to STDOUT so we are able to see them on screen.

Step 5) Acessing Kibana

With our containers up an running, you may access Kibana on http://localhost:5601/app/kibana#/, add the fluentd-* index and view your valuable container logs.
To add the index, once you access Kibana, click in “Management” > “Index Patterns” > “Create Index Pattern”. On the textField, insert fluentd-*, which is the name of the index that we defined when outputting events from fluentd to elasticsearch (see step 3) and finally press the Create button.
Kibana fluentd index creation

Kibana fluentd index creation

Now, move to the “Discover” tab, where you will be able to visualize the Kibana and Elasticsearch container logs, as shown below:

Kibana fluentd data flow

Kibana fluentd data flow

Important Notes

An important point to mention that Docker containers do not go up in a scenario where the configured remote logging is not properly up and running.
To explain in other words and based on this post article, if the fluentd container was not the first container to go up, Elasticserach and Kibana containers would fail to go up.
If that was the case, you would see an error similar to the following when starting docker-compose:
ERROR: for elasticsearch  Cannot start service elasticsearch: failed to initialize logging driver: dial tcp [::1]:24224: getsockopt: connection refused
ERROR: Encountered errors while bringing up the project.
As explained, see that Elasticsearch container tried to go up but failed to communicate with the remote logging driver.
To solve that, in this example, we defined Elasticsearch to depend on fluentd, as shown in the following snippet from the Elasticserach portion of our docker-compose.yml
  logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: efk.elasticsearch
    depends_on:
      - fluentd
Other solutions would include fluentd to be installed as a service from their official packages, explained here.

Conclusion

This post provides an introduction to the use of fluentd in a context of Docker logging. For that, the use of a docker-compose contemplating the components of the EFK stack was used.
I have been a longtime user of Logstash and while exploring fluentd, I can provide my personal opinion that fluentd is also a good fit for any logging layer.
The full code for this article is available here.

Software engineer, geek, traveler, wannabe athlete and a lifelong learner. Works at @IBM

Next Post