1 of 11

Application Logs

‘Application’ in Cocktail Log Service means ‘Workload that exists in the Service map'.

To check the application log, you must first register in the application management tab.

1. Inquiry Log

Logging - Application Log - Select an application from the list of applications to view.

View By Hour : You can search logs from the last 5 minutes to 48 hours ago, and check logs from up to 2 weeks ago.

Application: You can view a list of all applications that exist in the cluster.

View More : Get logs since the last time in the list.

Current number of logs / Total number of logs : This refers to the total number of logs viewed at that time and the maximum number of logs.

The maximum number of logs that can be viewed at one time is 5000. The inquiry period is up to 7 days.

The Show More button is displayed when the total number of logs exceeds 5000.

2. View log details

Click the link button for the log viewed by time to check detailed information about the log.

Log Message : You can check the contents of the log that actually occurred.

Label Information : Click the + button to expand and view label information.

Label information: You can close the expanded label information by clicking the - button.

3. View logs by time

If you click on each graph once, you can check the log for that time, and if you click again, you can check the log for the entire time again.

Graph Select : You can see that there are 60 logs of the current time.

4. Inquiry Label

Click the arrow button on the right to see the set of labels for the logs present at that time.

label list key : Indicates a list of labels for viewed logs. You can check the label value by clicking the label button.

label list values : Click the label value to search for the log you want to find by adding the conditions you want to search through AND search.

Selected label value : When you add a label condition, the condition is added to the top of the graph, and you can search by clicking the X button to remove the condition.

5. Search Log

Enter the keyword you want to search for and click the search button to view the log for the search term.

Search word : You can search logs where the string exists regardless of case.

6. Download Log

Click the “Download” button at the top of the graph to download the log.

Download: You can download log data for up to 5,000 searched logs in Excel file format. Each column contains a label value.

Application Management

The log service can perform logging depending on various applications, and the process of registering to enable logging is explained.

1. Application registration

Logging - Application Management - Click the Registration button to register.

Name : Actual application service name

Description : Description field to distinguish

Cluster : Cluster with application

Namespace : Namespace where the application resides

Developing Language : Application development language

Log Service Information : Log service information useful for the current platform

2. Application modification

The log service can collect and search logs of authenticated applications through tokens.

A token is automatically issued when you first register an application, and a new token can be issued through renewal.

Even if the token is renewed, the token applied to the application is not automatically renewed, so you must renew it manually to collect logs.

You can also click the "Action" button to disable the application to stop logging, or delete it from the application list.

Application logging

We recommend that you select a method that suits your environment.

1. Automatic instrumentation of container logs

Collect container logs from workloads using automated instrumentation.

2. Manual instrumentation of file logs (SDK)

The SDK method is used when you want to collect logs from a specific service through Logger settings.

3. Manual measurement of file logs (Sidecar)

The sidecar method is used when you want to read and collect log files in a specific directory.

Automatic instrumentation of container logs

Using Opentelemetry-operateor, you can simply log using container logs with annotations. Since container logs are collected only in the namespace unit set by the user, the method of collecting them fo

You must install the cocktail-log-operator add-on to use it.

1. Java

2. Python

Java

You can leverage Custom Resources to configure the OpenTelemetry auto-instrumentation library and add annotations to your workloads to easily collect logs.

1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages in the same namespace.

log-agent Service address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

By adding an environment variable for each individual application rather than all applications in the namespace through CRD, you can only collect logs for a specific application.

2) Add annotations to the individual applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

3) Add service name and token value to environment variables

4. Check Application log

1) Logging - Application Log - Search for the application you set in the application list.

Python

1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages in the same namespace.

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

2) Add annotations to the applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

3) Add service name and token value to environment variables.

4. Check Application log

1) Logging - Application Log - Search for the application you set in the application list.

Manual measurement of file logs(SDK)

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

1. Java

2. Python

Java

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

This guide is for existing JAVA applications that have been build on Cocktail Cloud.

1. Log Appender Setting

Log Appender is an interface provided by a logging framework or library that provides the ability to collect and process log messages. OpenTelemetry interacts with Log Appender through the Log Bridge API to collect log messages and associate them with tracking data from OpenTelemetry. Therefore, log appenders can be used to collect and integrate log data from OpenTelemetry.

We introduce how to collect data using logback and log4j, which are representative loggers.

1) logback

1-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.0.0-alpha</version>
    <scope>runtime</scope>
</dependency>

runtimeOnly group: 'io.opentelemetry.instrumentation', name: 'opentelemetry-logback-appender-1.0', version: '2.0.0-alpha'

1-2) logback.xml setting

    ...

    <appender name="OpenTelemetry"
              class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    </appender>

    ...
 
    <logger name="OTLP" additivity="false">
        <appender-ref ref="OpenTelemetry"/>
        <level value="INFO"/>
    </logger>

2) log4J

2-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-appender-2.17</artifactId>
    <version>OPENTELEMETRY_VERSION</version>
    <scope>runtime</scope>
  </dependency>
</dependencies>

runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:OPENTELEMETRY_VERSION")

2-2) log4j.xml setting

<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout
          pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
    </Console>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root>
      <AppenderRef ref="OpenTelemetryAppender" level="All"/>
      <AppenderRef ref="Console" level="All"/>
    </Root>
  </Loggers>
</Configuration>

3) Logger setting

private static final Logger logger = LoggerFactory.getLogger("OTLP");
// Logging with a logger called OTLP. This can be changed
// If you set it to 'ALL' in the above document, you can log with console and opentelemetry.

2. Image creation via SDK

1) Build/Pipeline - Build details of the Java application for which you want to collect logs from the build.

2) Click “Image Build Task” to edit

3) Setting environment variables and downloading SDK of Opentelemetry-java-instrumentation

...

# Required settings to export log
ENV OTEL_EXPORTER_OTLP_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=$OTEL_EXPORTER_OTLP_LOGS_HEADERS
ENV OTEL_LOGS_EXPORTER=$OTEL_LOGS_EXPORTER
ENV OTEL_SERVICE_NAME=$OTEL_SERVICE_NAME

# None processing to not collect metrics and traces
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_TRACES_EXPORTER=none


# Download sdk
RUN wget -q https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.0.0/opentelemetry-javaagent.jar
ENTRYPOINT ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "app.jar"]

...

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:
	# opentelemetry collector svc address (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318)
	- name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: {log-agent Service address}:4318

	# Log service address of opentelemetry collector (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318/v1/logs)
        - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
          value: {log-agent Service address}:4318/v1/logs

	# Settings using opelemetry protocol
        - name: OTEL_LOGS_EXPORTER
          value: otlp
        
	# Application Name
	- name: OTEL_SERVICE_NAME
          value: {Application Name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application Token},app_name={Application Name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log Check

1) Logging - Application Log - Search for the application you set in the application list.

Python

This is a method of installing into an existing application using the SDK provided by Opentelemetry. Logging in Python is currently under development at opentelemetry.

This guide is for existing Python applications that have been build on Cocktail Cloud.

Additionally, the Python application in this guide was created based on 'Flask'

1. Image creation via SDK

1) Build/Pipeline - Build details of the Python application for which you want to collect logs from the build.

2) Click on the image build task to edit

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log

1) Logging - Application Log - Search for the application you set in the application list

Manual measurement of file logs (Sidecar)

The advantage of using the open source fluent-bit is that the user can handle it by reading log files stored in the directory.

Fluent-bit

Fluent Bit is a lightweight log data collector that is used to collect and process data. By installing Flentbit as a sidecar in your application, you parse the application's logs and forward them to O

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

1. Fluent-bit Setting

1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

2. Application Setting

1) Logging - Copy the token of the application created in Application Management

2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

fluent-bit.conf

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

Application logs create a label called 'level' to provide users with the ability to filter by level. The following is an example of converting nginx's code value to level when the user's application does not have a value called level.

rewrite.lua

Once the config map creation is complete, return to the application to create the volume.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Config Map

Volume Name : Custom Name

Config Map : User-created ConfigMap name

Permission : 644

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : Log Data - Directory path where logs are stored (eg. /var/log)

Container Path : Fluent-bit -conf -fluent-bit configuration file path (eg. /fluent-bit/etc)

When the fluent-bit container does not operate properly

4. Check Application Log

1) Logging - Application Log - Search for the application you set in the application list.

Java

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

This guide is for existing JAVA applications that have been build on Cocktail Cloud.

1. Log Appender Setting

We introduce how to collect data using logback and log4j, which are representative loggers.

1) logback

1-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.0.0-alpha</version>
    <scope>runtime</scope>
</dependency>

runtimeOnly group: 'io.opentelemetry.instrumentation', name: 'opentelemetry-logback-appender-1.0', version: '2.0.0-alpha'

1-2) logback.xml setting

    ...

    <appender name="OpenTelemetry"
              class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    </appender>

    ...
 
    <logger name="OTLP" additivity="false">
        <appender-ref ref="OpenTelemetry"/>
        <level value="INFO"/>
    </logger>

2) log4J

2-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-appender-2.17</artifactId>
    <version>OPENTELEMETRY_VERSION</version>
    <scope>runtime</scope>
  </dependency>
</dependencies>

runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:OPENTELEMETRY_VERSION")

2-2) log4j.xml setting

<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout
          pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
    </Console>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root>
      <AppenderRef ref="OpenTelemetryAppender" level="All"/>
      <AppenderRef ref="Console" level="All"/>
    </Root>
  </Loggers>
</Configuration>

3) Logger setting

private static final Logger logger = LoggerFactory.getLogger("OTLP");
// Logging with a logger called OTLP. This can be changed
// If you set it to 'ALL' in the above document, you can log with console and opentelemetry.

2. Image creation via SDK

1) Build/Pipeline - Build details of the Java application for which you want to collect logs from the build.

2) Click “Image Build Task” to edit

3) Setting environment variables and downloading SDK of Opentelemetry-java-instrumentation

...

# Required settings to export log
ENV OTEL_EXPORTER_OTLP_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=$OTEL_EXPORTER_OTLP_LOGS_HEADERS
ENV OTEL_LOGS_EXPORTER=$OTEL_LOGS_EXPORTER
ENV OTEL_SERVICE_NAME=$OTEL_SERVICE_NAME

# None processing to not collect metrics and traces
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_TRACES_EXPORTER=none


# Download sdk
RUN wget -q https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.0.0/opentelemetry-javaagent.jar
ENTRYPOINT ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "app.jar"]

...

3. Container Setting

1) Logging - Copy the token of the application created in Application Management

2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:
	# opentelemetry collector svc address (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318)
	- name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: {log-agent Service address}:4318

	# Log service address of opentelemetry collector (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318/v1/logs)
        - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
          value: {log-agent Service address}:4318/v1/logs

	# Settings using opelemetry protocol
        - name: OTEL_LOGS_EXPORTER
          value: otlp
        
	# Application Name
	- name: OTEL_SERVICE_NAME
          value: {Application Name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application Token},app_name={Application Name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

4. Check Application Log Check

1) Logging - Application Log - Search for the application you set in the application list.

Fluent-bit

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

1. Fluent-bit Setting

1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

2. Application Setting

1) Logging - Copy the token of the application created in Application Management

2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

fluent-bit.conf

[SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf


[INPUT]
        Name         tail
        Path         [The path to the log file where the log will be loaded]
        
[OUTPUT]
        Name         stdout
        Match        *

[FILTER]
        Name parser
        Match *
        Key_Name log
        Parser nginx
        # Setting to preserve existing log messages before parsing (true: preservation)
        Preserve_Key true
        Reserve_Data true

# instrumentation_scope, service.name.. in body
[FILTER]
        Name        modify
        Match       *
        Add         service.name 'Application Name'       

# Handled by user by inserting script
[FILTER]
        Name        lua
        Match       *
        script      rewrite.lua
        call        rewrite_tag
        

[OUTPUT]
        Name         opentelemetry
        Match        *
        # Host	     log-agent-cocktail-log-agent.cocktail-addon
        Host         'log-agent Service Address'
        Port         4318
        metrics_uri  /v1/metrics
        logs_uri     /v1/logs
        traces_uri   /v1/traces
        header       app_token 'Token'
        header	     app_name  'Application Name'
        Log_response_payload True
        tls          off
        tls.verify   off
        logs_body_key_attributes true

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S

rewrite.lua

# A script that adds the value INFO to the value level if the code value is 200, and the value ERROR if it is not 200.
function rewrite_tag(tag, timestamp, record)
    if record["code"] == "200" then
        record["level"] = "INFO"
    else
        record["level"] = "ERROR"
    end
    return 1, timestamp, record
end