arrow-left

All pages
gitbookPowered by GitBook
1 of 11

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Java

You can leverage Custom Resources to configure the OpenTelemetry auto-instrumentation library and add annotations to your workloads to easily collect logs.

hashtag
1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

circle-info

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages ​​in the same namespace.

log-agent Service address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

circle-info

By adding an environment variable for each individual application rather than all applications in the namespace through CRD, you can only collect logs for a specific application.

hashtag
2) Add annotations to the individual applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

hashtag
3) Add service name and token value to environment variables

hashtag
4. Check Application log

hashtag
1) Logging - Application Log - Search for the application you set in the application list.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: {Instrumentation name}
spec:
  exporter:
    endpoint: {log-agent Service address}:4318
  java:
    env:
      - name: OTEL_LOGS_EXPORTER
        value: otlp
      - name: OTEL_METRICS_EXPORTER
        value: none
      - name: OTEL_TRACES_EXPORTER
        value: otlp              
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
        
      # Used when you want to collect all Java applications for which the anotation value of the namespace is 'true'..                
      - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
        value: {log-agent Service address}:4318/v1/logs
instrumentation.opentelemetry.io/inject-java: 'true'
apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:

	# Application name
	- name: OTEL_SERVICE_NAME
          value: {Application name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application token},app_name={Application name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...

Application logging

We recommend that you select a method that suits your environment.

hashtag
1. Automatic instrumentation of container logs

Collect container logs from workloads using automated instrumentation.

Automatic instrumentation of container logschevron-right

hashtag
2. Manual instrumentation of file logs (SDK)

The SDK method is used when you want to collect logs from a specific service through Logger settings.

hashtag
3. Manual measurement of file logs (Sidecar)

The sidecar method is used when you want to read and collect log files in a specific directory.

Manual measurement of file logs(SDK)chevron-right
Manual measurement of file logs (Sidecar)chevron-right

Automatic instrumentation of container logs

Using Opentelemetry-operateor, you can simply log using container logs with annotations. Since container logs are collected only in the namespace unit set by the user, the method of collecting them fo

circle-info

You must install the cocktail-log-operator add-on to use it.

hashtag
1. Java

hashtag
2. Python

Javachevron-right
Pythonchevron-right

Application Management

The log service can perform logging depending on various applications, and the process of registering to enable logging is explained.

hashtag
1. Application registration

Logging - Application Management - Click the Registration button to register.

Name : Actual application service name

Description : Description field to distinguish

Cluster : Cluster with application

Namespace : Namespace where the application resides

Developing Language : Application development language

Log Service Information : Log service information useful for the current platform

hashtag
2. Application modification

The log service can collect and search logs of authenticated applications through tokens.

A token is automatically issued when you first register an application, and a new token can be issued through renewal.

circle-exclamation

Even if the token is renewed, the token applied to the application is not automatically renewed, so you must renew it manually to collect logs.

You can also click the "Action" button to disable the application to stop logging, or delete it from the application list.

Python

hashtag
1) CRD installation

You can create one by searching for Infrastructure - Custom Resources - 'instrumentations'.

You can create it by clicking the Create button, selecting the namespace where you want to collect logs, and modifying the form below.

circle-info

The above CRD is applied on a namespace basis, and automatic container logs can be collected for other languages ​​in the same namespace.

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

hashtag
2) Add annotations to the applications you want to collect

Add annotations to the workloads in the namespace for which you want to collect logs.

Application - Service Map - Service Map to collect logs - Workload - Select the application to collect logs - Click the "Settings" button.

Change the Yaml view and add the following annotations to the template - metadata - annotations section.

hashtag
3) Add service name and token value to environment variables.

hashtag
4. Check Application log

hashtag
1) Logging - Application Log - Search for the application you set in the application list.

Manual measurement of file logs (Sidecar)

The advantage of using the open source fluent-bit is that the user can handle it by reading log files stored in the directory.

Fluent-bitchevron-right
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: {Instrumentation name}
spec:
  exporter:
    endpoint: {log-agent Service address}:4318
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
    env:
      - name: OTEL_LOGS_EXPORTER
        value: otlp_proto_http
      - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
        value: {log-agent Service address}:4318/v1/logs
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
        value: 'true'
        
      # Used when you want to collect all Java applications for which the anotation value of the namespace is 'true'.
      - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
        value: app_token={Application Token},app_name={Application Name}
instrumentation.opentelemetry.io/inject-python: 'true'
apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:

	# Application name
	- name: OTEL_SERVICE_NAME
          value: {Application name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application token},app_name={application name}     
        
        image: {python-application image}
        imagePullPolicy: Always
        ...

Java

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

circle-exclamation

This guide is for existing JAVA applications that have been build on Cocktail Cloud.

hashtag

Python

This is a method of installing into an existing application using the SDK provided by Opentelemetry. Logging in Python is currently under development at opentelemetry.

circle-exclamation

This guide is for existing Python applications that have been build on Cocktail Cloud.

circle-info

1. Log Appender Setting

Log Appender is an interface provided by a logging framework or library that provides the ability to collect and process log messages. OpenTelemetry interacts with Log Appender through the Log Bridge API to collect log messages and associate them with tracking data from OpenTelemetry. Therefore, log appenders can be used to collect and integrate log data from OpenTelemetry.

We introduce how to collect data using logback and log4j, which are representative loggers.

hashtag
1) logback

hashtag
1-1) dependency addition

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.0.0-alpha</version>
    <scope>runtime</scope>
</dependency>
runtimeOnly group: 'io.opentelemetry.instrumentation', name: 'opentelemetry-logback-appender-1.0', version: '2.0.0-alpha'

hashtag
1-2) logback.xml setting

hashtag
2) log4J

hashtag
2-1) dependency addition

hashtag
2-2) log4j.xml setting

hashtag
3) Logger setting

hashtag
2. Image creation via SDK

hashtag
1) Build/Pipeline - Build details of the Java application for which you want to collect logs from the build.

hashtag
2) Click “Image Build Task” to edit

hashtag
3) Setting environment variables and downloading SDK of Opentelemetry-java-instrumentation

hashtag
3. Container Setting

hashtag
1) Logging - Copy the token of the application created in Application Management

hashtag
2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

hashtag
4. Check Application Log Check

hashtag
1) Logging - Application Log - Search for the application you set in the application list.

Additionally, the Python application in this guide was created based on 'Flask'

hashtag
1. Image creation via SDK

hashtag
1) Build/Pipeline - Build details of the Python application for which you want to collect logs from the build.

hashtag
2) Click on the image build task to edit

hashtag
3. Container Setting

hashtag
1) Logging - Copy the token of the application created in Application Management

hashtag
2) In Container Details - Settings - Environment Variables tab, set environment variables as follows

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

hashtag
4. Check Application Log

hashtag
1) Logging - Application Log - Search for the application you set in the application list

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-appender-2.17</artifactId>
    <version>OPENTELEMETRY_VERSION</version>
    <scope>runtime</scope>
  </dependency>
</dependencies>
runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:OPENTELEMETRY_VERSION")
    ...

    <appender name="OpenTelemetry"
              class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    </appender>

    ...
 
    <logger name="OTLP" additivity="false">
        <appender-ref ref="OpenTelemetry"/>
        <level value="INFO"/>
    </logger>
<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout
          pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
    </Console>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root>
      <AppenderRef ref="OpenTelemetryAppender" level="All"/>
      <AppenderRef ref="Console" level="All"/>
    </Root>
  </Loggers>
</Configuration>
private static final Logger logger = LoggerFactory.getLogger("OTLP");
// Logging with a logger called OTLP. This can be changed
// If you set it to 'ALL' in the above document, you can log with console and opentelemetry.
...

# Required settings to export log
ENV OTEL_EXPORTER_OTLP_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=$OTEL_EXPORTER_OTLP_LOGS_HEADERS
ENV OTEL_LOGS_EXPORTER=$OTEL_LOGS_EXPORTER
ENV OTEL_SERVICE_NAME=$OTEL_SERVICE_NAME

# None processing to not collect metrics and traces
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_TRACES_EXPORTER=none


# Download sdk
RUN wget -q https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.0.0/opentelemetry-javaagent.jar
ENTRYPOINT ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "app.jar"]

...
apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:
	# opentelemetry collector svc address (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318)
	- name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: {log-agent Service address}:4318

	# Log service address of opentelemetry collector (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318/v1/logs)
        - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
          value: {log-agent Service address}:4318/v1/logs

	# Settings using opelemetry protocol
        - name: OTEL_LOGS_EXPORTER
          value: otlp
        
	# Application Name
	- name: OTEL_SERVICE_NAME
          value: {Application Name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application Token},app_name={Application Name}        
        
        image: {java-application image}
        imagePullPolicy: Always
        ...
...

# flask 애플리케이션용 opentelemetry Sdk 다운로드
RUN pip install --upgrade pip && pip install opentelemetry-distro && opentelemetry-bootstrap -a install && pip install flask

# Log를 Export할 필수설정
ENV OTEL_EXPORTER_OTLP_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=$OTEL_EXPORTER_OTLP_LOGS_HEADERS
ENV OTEL_SERVICE_NAME=$OTEL_SERVICE_NAME
ENV OTEL_LOGS_EXPORTER=$OTEL_LOGS_EXPORTER

# Python 애플리케이션 로깅을 위한 필수 설정
ENV OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

# metric과 trace를 수집하지 않기 위한 none 처리
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_TRACES_EXPORTER=none

...
apiVersion: apps/v1
kind: Deployment
...
spec:
...
  template:
    spec:
      containers:
      - env:
	# opentelemetry collector svc address (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318)
	- name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: {log-agent Service address}:4318

	# Log service address of opentelemetry collector (eg. http://log-agent-cocktail-log-agent.cocktail-addon:4318/v1/logs)
        - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
          value: {log-agent Service address}:4318/v1/logs

	# Settings using opelemetry protocol
        - name: OTEL_LOGS_EXPORTER
          value: otlp_proto_http
        
	# Application Name
	- name: OTEL_SERVICE_NAME
          value: {Application Name}
        
        # Settings for authentication
        - name: OTEL_EXPORTER_OTLP_LOGS_HEADERS
          value: app_token={Application Token},app_name={Application Name}        
        
        image: {python-application image}
        imagePullPolicy: Always
        ...

Fluent-bit

Fluent Bit is a lightweight log data collector that is used to collect and process data. By installing Flentbit as a sidecar in your application, you parse the application's logs and forward them to O

circle-info

The above guide explains how the application stores logs in the /var/log directory. Please modify the directory and log pattern to suit your environment.

hashtag
1. Fluent-bit Setting

hashtag
1) Create Container

Application to collect logs - Settings - Container Click the "Add" button to create a container as follows.

Image: fluent/fluent-bit:3.0.0

When you press the save button, the container runs in the existing application in fluent-bit sidecar format.

hashtag
2. Application Setting

hashtag
1) Logging - Copy the token of the application created in Application Management

hashtag
2) Volume Mount - Log

Logs are stored in the path set in Log Appender, so you need to create a volume in the container and mount it.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Empty Dir

Volume Name : custom name

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : File path set in Log Appender (eg. /var/log)

hashtag
3) Volume Mount - Fluent-bit

The container must mount the directory path where it stores the logs before it can read the file and parse the logs.

You can also add labels or change the label name through Config provided by fluent-bit.

hashtag
Create fluent-bit Config Map

Service map to collect logs - Configuration information - Click the "Create" button to create a configuration map.

Name : The name of the config map you want to set.

Description : Additionally, a description of the config map to be specified by the user.

Click the “Add” button to add the config file.

circle-info

The following config file is not absolute. The location where the log is loaded or the log pattern may vary, so please set it according to your environment.

hashtag
fluent-bit.conf

log-agent Service Address : Infrastructure - Cluster - Add-ons - Click 'log-agent' and check the service name.

( http port = 4318 , grpc port = 4317)

parsers.conf

circle-check

Application logs create a label called 'level' to provide users with the ability to filter by level. The following is an example of converting nginx's code value to level when the user's application does not have a value called level.

rewrite.lua

Once the config map creation is complete, return to the application to create the volume.

Application to collect logs - Settings - Volume - Click the "Create" button to create a volume as follows.

Volume Type : Config Map

Volume Name : Custom Name

Config Map : User-created ConfigMap name

Permission : 644

The following is the process of mounting the created volume.

Application to collect logs - Settings - Volume mount - Click the "Add" button to mount the volume with the following settings.

Container Path : Log Data - Directory path where logs are stored (eg. /var/log)

Container Path : Fluent-bit -conf -fluent-bit configuration file path (eg. /fluent-bit/etc)

When the fluent-bit container does not operate properly

hashtag
4. Check Application Log

hashtag
1) Logging - Application Log - Search for the application you set in the application list.

[SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf


[INPUT]
        Name         tail
        Path         [The path to the log file where the log will be loaded]
        
[OUTPUT]
        Name         stdout
        Match        *

[FILTER]
        Name parser
        Match *
        Key_Name log
        Parser nginx
        # Setting to preserve existing log messages before parsing (true: preservation)
        Preserve_Key true
        Reserve_Data true

# instrumentation_scope, service.name.. in body
[FILTER]
        Name        modify
        Match       *
        Add         service.name 'Application Name'       

# Handled by user by inserting script
[FILTER]
        Name        lua
        Match       *
        script      rewrite.lua
        call        rewrite_tag
        

[OUTPUT]
        Name         opentelemetry
        Match        *
        # Host	     log-agent-cocktail-log-agent.cocktail-addon
        Host         'log-agent Service Address'
        Port         4318
        metrics_uri  /v1/metrics
        logs_uri     /v1/logs
        traces_uri   /v1/traces
        header       app_token 'Token'
        header	     app_name  'Application Name'
        Log_response_payload True
        tls          off
        tls.verify   off
        logs_body_key_attributes true
[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S 
# A script that adds the value INFO to the value level if the code value is 200, and the value ERROR if it is not 200.
function rewrite_tag(tag, timestamp, record)
    if record["code"] == "200" then
        record["level"] = "INFO"
    else
        record["level"] = "ERROR"
    end
    return 1, timestamp, record
end
Troubleshooting | Cocktail Cloud Onlinecocktailcloud.gitbook.iochevron-right
Logo

Manual measurement of file logs(SDK)

This is a method of installing into an existing application using the SDK provided by Opentelemetry.

hashtag
1. Java

hashtag
2. Python

Javachevron-right
Pythonchevron-right

Application Logs

hashtag
‘Application’ in Cocktail Log Service means ‘Workload that exists in the Service map'.

circle-info

To check the application log, you must first register in the application management tab.

hashtag
1. Inquiry Log

Logging - Application Log - Select an application from the list of applications to view.

View By Hour : You can search logs from the last 5 minutes to 48 hours ago, and check logs from up to 2 weeks ago.

Application: You can view a list of all applications that exist in the cluster.

View More : Get logs since the last time in the list.

Current number of logs / Total number of logs : This refers to the total number of logs viewed at that time and the maximum number of logs.

circle-info

The maximum number of logs that can be viewed at one time is 5000. The inquiry period is up to 7 days.

circle-info

The Show More button is displayed when the total number of logs exceeds 5000.

hashtag
2. View log details

Click the link button for the log viewed by time to check detailed information about the log.

Log Message : You can check the contents of the log that actually occurred.

Label Information : Click the + button to expand and view label information.

Label information: You can close the expanded label information by clicking the - button.

hashtag
3. View logs by time

If you click on each graph once, you can check the log for that time, and if you click again, you can check the log for the entire time again.

Graph Select : You can see that there are 60 logs of the current time.

hashtag
4. Inquiry Label

Click the arrow button on the right to see the set of labels for the logs present at that time.

label list key : Indicates a list of labels for viewed logs. You can check the label value by clicking the label button.

label list values : Click the label value to search for the log you want to find by adding the conditions you want to search through AND search.

Selected label value : When you add a label condition, the condition is added to the top of the graph, and you can search by clicking the X button to remove the condition.

hashtag
5. Search Log

Enter the keyword you want to search for and click the search button to view the log for the search term.

Search word : You can search logs where the string exists regardless of case.

hashtag
6. Download Log

Click the “Download” button at the top of the graph to download the log.

Download: You can download log data for up to 5,000 searched logs in Excel file format. Each column contains a label value.

Application Managementchevron-right