Fluent-bit

Fluent Bit is a lightweight log data collector that is used to collect and process data. By installing Flentbit as a sidecar in your application, you parse the application's logs and forward them to O

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

1. Fluent-bit Setting

1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

2. Application Setting

1) Logging - Copy the token of the application created in Application Management

2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

fluent-bit.conf

[SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf


[INPUT]
        Name         tail
        Path         [The path to the log file where the log will be loaded]
        
[OUTPUT]
        Name         stdout
        Match        *

[FILTER]
        Name parser
        Match *
        Key_Name log
        Parser nginx
        # Setting to preserve existing log messages before parsing (true: preservation)
        Preserve_Key true
        Reserve_Data true

# instrumentation_scope, service.name.. in body
[FILTER]
        Name        modify
        Match       *
        Add         service.name 'Application Name'       

# Handled by user by inserting script
[FILTER]
        Name        lua
        Match       *
        script      rewrite.lua
        call        rewrite_tag
        

[OUTPUT]
        Name         opentelemetry
        Match        *
        # Host	     log-agent-cocktail-log-agent.cocktail-addon
        Host         'log-agent Service Address'
        Port         4318
        metrics_uri  /v1/metrics
        logs_uri     /v1/logs
        traces_uri   /v1/traces
        header       app_token 'Token'
        header	     app_name  'Application Name'
        Log_response_payload True
        tls          off
        tls.verify   off
        logs_body_key_attributes true

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S 

Application logs create a label called 'level' to provide users with the ability to filter by level. The following is an example of converting nginx's code value to level when the user's application does not have a value called level.

rewrite.lua

# A script that adds the value INFO to the value level if the code value is 200, and the value ERROR if it is not 200.
function rewrite_tag(tag, timestamp, record)
    if record["code"] == "200" then
        record["level"] = "INFO"
    else
        record["level"] = "ERROR"
    end
    return 1, timestamp, record
end

Once the config map creation is complete, return to the application to create the volume.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Config Map

Volume Name : Custom Name

Config Map : User-created ConfigMap name

Permission : 644

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : Log Data - Directory path where logs are stored (eg. /var/log)

Container Path : Fluent-bit -conf -fluent-bit configuration file path (eg. /fluent-bit/etc)

When the fluent-bit container does not operate properly

Troubleshooting | Cocktail Cloud Online

4. Check Application Log

1) Logging - Application Log - Search for the application you set in the application list.