This is the story abouthow to practically use the Fluent Bit tail input plugin. You mightwonder about the existence of an official documentation on this. Yes, there isone and here is the link. But I’llguarantee this post will explain the content in a simpler and clearer manner.
Before going in to theconfiguration, let me tell you about the versions and the environment that Ihave tried these tests out.
· Fluent Bitversion : 1.6.10
fluent-bit — version
· OS version :Ubuntu 20.04.1 LTS
cat /etc/os-release
Fluent Bit tailplugin is similar to the tail command you encounter in Unix, Unix-likesystems, FreeDOS and MSX-DOS. Therefore in order to begin, we need a file toread. We can use a default system file. But I prefer to create my own file,even though it is out of the scope of this topic. Below shell script will reada file and write to another file at every second-delay. By that, we have bettercontrol of what is written and what is missed.
#!/bin/bash
: > OutputFile.txt
input="my_log_file.log"
while IFS= read -r line
do
echo"$line" >> OutputFile.txt
sleep 1
done < "$input"
Let’s begin the journeywith the simplest configuration file. I have named the above script as ReadWrite.shand content of my working directory is as below.
my_log_file.log
example.conf
ReadWrite.sh
First I ran the scriptto write to the file.
./ReadWrite.sh
Then I executed theFluent Bit configuration file.
fluent-bit -cexample.conf
My configuration filelooks like below.
[INPUT]
Name tail
Path ./OutputFile.txt
[OUTPUT]
Name stdout
Match *
If you are copying andpasting, please ensure you indent the code properly. Otherwise you will get “Invalidindentation level” error. Above code is the simplest configuration you canhave with tail input plugin. The output will be something like below.
[3] tail.0:[1609993104.104741564, {"log"=>"some text here"}]
In a more generalisedform;
[number] tail.0: [Unixepoch time, {"log"=>"line content"}]
Now we shall try tomodify the configuration file.
[INPUT]
Name tail
Path./OutputFile.txt
Tag mytag
Then the output willlook like below;
[number] mytag: [Unixepoch time, {"log"=>"line content"}]
What has happened hereis that tail.0 has been replaced by the tag we introduced.
Before explaininganother configuration parameter, I advise you to try the following scenario.
· First startthe script that writes to a file
· Then run theFluent Bit with above configuration file
· Stop theFluent Bit process for about a minute and re-run it
· Note downthe last log written in the first run of Fluent Bit and first log line writtenin the rerun session
· Compare itwith your original log file where your shell script is reading the logs. Youcan see that Fluent Bit has missed some log lines during the time it was notrunning.
What happens is, withonly the above configuration parameters, Fluent Bit is reading the log linethat is written after the Fluent Bit process has started. It doesn’t keep trackof the last read line of the file. In order to address this issue, they haveintroduced a parameter call DB. You can update your configurationfile as below.
[INPUT]
Name tail
Path ./OutputFile.txt
Tag mytag
DB ./file_status.db
You will be able to seea database file that has been created in the same location where yourconfiguration file is. You can open that file using DB Browser for SQLiteapp.
The offset valueis the place where Fluent Bit has last read the OutputFile.txt document.You can compare the last line written in the Fluent Bit log and open the OutputFile.txtfile using the vim command and run the below command.
:goto offset_value
If you restart theFluent Bit service, it will read from the very next line, which will solve theissue I described earlier.
What I am going toexplain next is a little complex to recreate. In a real world scenario, if wetry to add another log file with some content already written, will the FluentBit be able to read that file from the beginning?
Follow below stepsthoroughly;
· Edit thefluent-bit configuration file as below.
[INPUT]
Name tail
Path./OutputFile*.txt
Tag mytag
DB./file_status.db
What I have changed isthe Path configuration parameter in order to read all the files that match thegiven pattern.
· I assume thedatabase file created above is already exists, where it has the last readlocation of OutputFile.txt
· Start ReadWrite2.shwhich writes to OutputFile2.txt. Content of ReadWrite2.sh isalmost similar as the ReadWrite.sh that I used at the very beginning. Ihave changed the output file name and added a prefix for the log lines thatwrite to that file in order to separate those in Fluent Bit logs. For yourconvenience, I have highlighted the changes below.
#!/bin/bash
: > OutputFile2.txt
input="my_log_file.log"
while IFS= read -r line
do
echo "2$line" >> OutputFile2.txt
sleep 1
done < "$input"
· Stop theabove process after about ten seconds
· Next, start ReadWrite.shwhich writes to OutputFile.txt
· Start FluentBit service
· Modify the ReadWrite2.shin order to identify new lines that writing next.
echo "2 2$line" >> OutputFile2.txt
· Restart ReadWrite2.sh
· Checkwhether Fluent Bit has read the very first line of the OutputFile2.txt
What you are able to seeis, that Fluent Bit has started reading from the new line written after theprocess has started. To solve that issue, Fluent Bit has used the Read_from_Headconfiguration parameter.
Let’s modify ourconfiguration file like below.
[INPUT]
Name tail
Path ./OutputFile*.txt
Tag mytag
DB ./file_status.db
Read_from_Headon
Okay, now you have thesolution for that issue as well. I have shown you how to create the environmentthat you can test by yourself. We will meet with a new post to discuss how tosend logs to a particular Splunk index using Fluent Bit, soon.