1 of 9

Application logging

We recommend that you select a method that suits your environment.

1. Automatic instrumentation of container logs

Collect container logs from workloads using automated instrumentation.

2. Manual instrumentation of file logs (SDK)

The SDK method is used when you want to collect logs from a specific service through Logger settings.

3. Manual measurement of file logs (Sidecar)

The sidecar method is used when you want to read and collect log files in a specific directory.

Automatic instrumentation of container logs

Using Opentelemetry-operateor, you can simply log using container logs with annotations. Since container logs are collected only in the namespace unit set by the user, the method of collecting them fo

You must install the cocktail-log-operator add-on to use it.

1. Java

2. Python

Java

You can leverage Custom Resources to configure the OpenTelemetry auto-instrumentation library and add annotations to your workloads to easily collect logs.

1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages in the same namespace.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: {Instrumentation name}
spec:
  exporter:
    endpoint: {log-agent Service address}:4318
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.0.0
    env:
      - name: OTEL_LOGS_EXPORTER
        value: otlp
      - name: OTEL_METRICS_EXPORTER
        value: none
      - name: OTEL_TRACES_EXPORTER
        value: none
        
      # Used when you want to collect all Java applications for which the anotation value of the namespace is 'true'..                
      - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
        value: {log-agent Service address}:4318/v1/logs

log-agent Service address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

By adding an environment variable for each individual application rather than all applications in the namespace through CRD, you can only collect logs for a specific application.

2) Add annotations to the individual applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

instrumentation.opentelemetry.io/inject-java: 'true'

3) Add service name and token value to environment variables

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:

	# Application name
	- name: OTEL_SERVICE_NAME
          value: {Application name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application token},app_name={Application name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...

4. Check Application log

1) Logging - Application Log - Search for the application you set in the application list.

Python

1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages in the same namespace.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: {Instrumentation name}
spec:
  exporter:
    endpoint: {log-agent Service address}:4318
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
    env:
      - name: OTEL_LOGS_EXPORTER
        value: otlp_proto_http
      - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
        value: {log-agent Service address}:4318/v1/logs
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
        value: 'true'
        
      # Used when you want to collect all Java applications for which the anotation value of the namespace is 'true'.
      - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
        value: app_token={Application Token},app_name={Application Name}

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

2) Add annotations to the applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

instrumentation.opentelemetry.io/inject-python: 'true'

3) Add service name and token value to environment variables.

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:

	# Application name
	- name: OTEL_SERVICE_NAME
          value: {Application name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application token},app_name={application name}     
        
        image: {python-application image}
        imagePullPolicy: Always
        ...

4. Check Application log

1) Logging - Application Log - Search for the application you set in the application list.

Manual measurement of file logs(SDK)

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

1. Java

2. Python

Java

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

This guide is for existing JAVA applications that have been build on Cocktail Cloud.

1. Log Appender Setting

Log Appender is an interface provided by a logging framework or library that provides the ability to collect and process log messages. OpenTelemetry interacts with Log Appender through the Log Bridge API to collect log messages and associate them with tracking data from OpenTelemetry. Therefore, log appenders can be used to collect and integrate log data from OpenTelemetry.

We introduce how to collect data using logback and log4j, which are representative loggers.

1) logback

1-1) dependency addition

1-2) logback.xml setting

2) log4J

2-1) dependency addition

2-2) log4j.xml setting

3) Logger setting

2. Image creation via SDK

1) Build/Pipeline - Build details of the Java application for which you want to collect logs from the build.

2) Click “Image Build Task” to edit

3) Setting environment variables and downloading SDK of Opentelemetry-java-instrumentation

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log Check

1) Logging - Application Log - Search for the application you set in the application list.

Python

This is a method of installing into an existing application using the SDK provided by Opentelemetry. Logging in Python is currently under development at opentelemetry.

This guide is for existing Python applications that have been build on Cocktail Cloud.

Additionally, the Python application in this guide was created based on 'Flask'

1. Image creation via SDK

1) Build/Pipeline - Build details of the Python application for which you want to collect logs from the build.

2) Click on the image build task to edit

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log

1) Logging - Application Log - Search for the application you set in the application list

Manual measurement of file logs (Sidecar)

The advantage of using the open source fluent-bit is that the user can handle it by reading log files stored in the directory.

Fluent-bit

Fluent Bit is a lightweight log data collector that is used to collect and process data. By installing Flentbit as a sidecar in your application, you parse the application's logs and forward them to O

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

1. Fluent-bit Setting

1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

2. Application Setting

1) Logging - Copy the token of the application created in Application Management

2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

fluent-bit.conf

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

Application logs create a label called 'level' to provide users with the ability to filter by level. The following is an example of converting nginx's code value to level when the user's application does not have a value called level.

rewrite.lua

Once the config map creation is complete, return to the application to create the volume.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Config Map

Volume Name : Custom Name

Config Map : User-created ConfigMap name

Permission : 644

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : Log Data - Directory path where logs are stored (eg. /var/log)

Container Path : Fluent-bit -conf -fluent-bit configuration file path (eg. /fluent-bit/etc)

When the fluent-bit container does not operate properly

4. Check Application Log

1) Logging - Application Log - Search for the application you set in the application list.

Java

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

This guide is for existing JAVA applications that have been build on Cocktail Cloud.

1. Log Appender Setting

We introduce how to collect data using logback and log4j, which are representative loggers.

1) logback

1-1) dependency addition

1-2) logback.xml setting

2) log4J

2-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-appender-2.17</artifactId>
    <version>OPENTELEMETRY_VERSION</version>
    <scope>runtime</scope>
  </dependency>
</dependencies>

runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:OPENTELEMETRY_VERSION")

2-2) log4j.xml setting

<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout
          pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
    </Console>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root>
      <AppenderRef ref="OpenTelemetryAppender" level="All"/>
      <AppenderRef ref="Console" level="All"/>
    </Root>
  </Loggers>
</Configuration>

3) Logger setting

private static final Logger logger = LoggerFactory.getLogger("OTLP");
// Logging with a logger called OTLP. This can be changed
// If you set it to 'ALL' in the above document, you can log with console and opentelemetry.

2. Image creation via SDK

1) Build/Pipeline - Build details of the Java application for which you want to collect logs from the build.

2) Click “Image Build Task” to edit

3) Setting environment variables and downloading SDK of Opentelemetry-java-instrumentation

...

# Required settings to export log
ENV OTEL_EXPORTER_OTLP_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=$OTEL_EXPORTER_OTLP_LOGS_HEADERS
ENV OTEL_LOGS_EXPORTER=$OTEL_LOGS_EXPORTER
ENV OTEL_SERVICE_NAME=$OTEL_SERVICE_NAME

# None processing to not collect metrics and traces
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_TRACES_EXPORTER=none


# Download sdk
RUN wget -q https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.0.0/opentelemetry-javaagent.jar
ENTRYPOINT ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "app.jar"]

...

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:
	# opentelemetry collector svc address (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318)
	- name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: {log-agent Service address}:4318

	# Log service address of opentelemetry collector (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318/v1/logs)
        - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
          value: {log-agent Service address}:4318/v1/logs

	# Settings using opelemetry protocol
        - name: OTEL_LOGS_EXPORTER
          value: otlp
        
	# Application Name
	- name: OTEL_SERVICE_NAME
          value: {Application Name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application Token},app_name={Application Name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log Check

1) Logging - Application Log - Search for the application you set in the application list.

Fluent-bit

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

1. Fluent-bit Setting

1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

2. Application Setting

1) Logging - Copy the token of the application created in Application Management

2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

fluent-bit.conf

[SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf


[INPUT]
        Name         tail
        Path         [The path to the log file where the log will be loaded]
        
[OUTPUT]
        Name         stdout
        Match        *

[FILTER]
        Name parser
        Match *
        Key_Name log
        Parser nginx
        # Setting to preserve existing log messages before parsing (true: preservation)
        Preserve_Key true
        Reserve_Data true

# instrumentation_scope, service.name.. in body
[FILTER]
        Name        modify
        Match       *
        Add         service.name 'Application Name'       

# Handled by user by inserting script
[FILTER]
        Name        lua
        Match       *
        script      rewrite.lua
        call        rewrite_tag
        

[OUTPUT]
        Name         opentelemetry
        Match        *
        # Host	     log-agent-cocktail-log-agent.cocktail-addon
        Host         'log-agent Service Address'
        Port         4318
        metrics_uri  /v1/metrics
        logs_uri     /v1/logs
        traces_uri   /v1/traces
        header       app_token 'Token'
        header	     app_name  'Application Name'
        Log_response_payload True
        tls          off
        tls.verify   off
        logs_body_key_attributes true

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S

rewrite.lua

# A script that adds the value INFO to the value level if the code value is 200, and the value ERROR if it is not 200.
function rewrite_tag(tag, timestamp, record)
    if record["code"] == "200" then
        record["level"] = "INFO"
    else
        record["level"] = "ERROR"
    end
    return 1, timestamp, record
end