Implement Filebeat as SideCar to export logs to Elastic
Exporting Logs inside a clustered environments is very important. When multiple instances of the same application are running it can be really hard to track down what exactly went down with the application. And Logging can be an unpleasant experience in Kubernetes.
In this post, I expect that your ELK setup is already in place. I will try to explain how we can implement Filebeat as a sidecar alongside your Kubernetes pods.
Why set up Filebeat as SideCar and not inside the same container?
While you can set up Filebeat inside the same container, but by doing that you are dishonoring the microservice architecture. Also exporting logs is an infrastructural component, and it is generally a good idea to separate the infrastructural components from your business applications.
Let's move on to the implementation
To keep it simple I will export success and error logs from a simple NGINX container. The image that I will be using for this is nginxdemos/hello
. This is a simple hello world app by Nginx.
You can find all the Kubernetes config objects in this git repo.
Filebeat Config
Filebeat is a log shipper it’s one of the many shippers provided by Elastic Stack. Filebeat offers various modules that can allow fast and simple log exporting for popular applications. So we will make use of the Nginx module. In this configuration, we need to provide two things. The path to the log files (inside the container, see: deployment.yml file below) and the ElasticSearch host. Let's look at the filebeat.yml
file
Within the module section, we have defined the log file path for the access and error log files.
You can head on to Filebeat page it’s well written to find out how to write the filebeat.yml file.
Applying configuration with ConfigMap
To apply the filebeat.yml inside our SideCar we will make use of the Kubernetes ConfigMap Object. ConfigMap lets you save non-confidential data in your NameSpace. You can read more about what is a ConfigMap here.
We will create a ConfigMap named filebeat-configmap
with the content of filebeat.yml
file.
Deploying Nginx with Sidecar
We will write the deployment file for our deployment in a moment. But before that, we need our sidecar to have access to the log files. Since the Nginx logs are generated in a different container we need to share the log files with our Sidecar.
To do that we can create a Volume Object in Kubernetes and Mount this volume in both our Nginx application and our SideCar. By doing so we will be sharing the log directory between the two containers. Whenever our Nginx container will write inside the log directory it will get written to the volume mount. And our sidecar will be able to read the logs from there.
We also need to create a Volume Object (filebeat-config
) to apply the filebeat-configmap
we generated just now.
Let’s see the deployment.yaml
file.
Conclusion
With this, we have successfully written a deployment for Nginx with a Filebeat sidecar. Here I have chosen a very simple example, but I believe this will give you an idea of how a sidecar is implemented. For more details on how to implement Filebeat for your application please visit the Filebeat page. Please feel free to drop a comment. 🙂