Simple Log Service:Collect stdout and stderr from a self-managed cluster in DaemonSet mode (old version)

CRD - AliyunPipelineConfig (recommended)

To create a Logtail configuration, you need to only create a Custom Resource (CR) from the AliyunPipelineConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Modify the parameters in the following YAML example as needed, copy and paste it into the template, then click Create.

   Note

   You can use the Logtail configuration generator to create a target scenario YAML script. This tool helps you quickly complete the configuration and reduces manual operations.

   The example YAML file below captures standard output from Pods labeled with app: ^(.*test.*)$ within the default namespace, using multi-line text mode, and forwards it to a logstore called k8s-stdout, which is automatically created within a project named k8s-log-<YOUR_CLUSTER_ID>. Adjust the parameters in the YAML as needed:

   • project: Log on to the Simple Log Service Console, and identify the project name created by the Logtail you installed, typically in the format k8s-log-<YOUR_CLUSTER_ID>.

   • IncludeK8sLabel: Used to filter the labels of the target pod. For example, app: ^(.*test.*)$ indicates that the label key is app, and it will collect pods with values that include test.

   • Endpoint and Region: For example, ap-southeast-1.log.aliyuncs.com and ap-southeast-1.

   For more information on config in the YAML file, such as supported inputs, outputs, processing plug-in types, and container filtering methods, see PipelineConfig. For a comprehensive list of YAML parameters, see CR parameters.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   # Create a ClusterAliyunPipelineConfig.
   kind: ClusterAliyunPipelineConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. This name is also the name of the Logtail configuration created. 
     name: example-k8s-stdout
   spec:
     # Specify the target project.
     project:
       name: k8s-log-<YOUR_CLUSTER_ID>
     # Create a logstore for storing logs.
     logstores:
       - name: k8s-stdout
     # Define the Logtail configuration.
     config:
       # Sample log (optional)
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Define input plug-ins.
       inputs:
             # Use the service_docker_stdout plug-in to collect text logs inside the container.
         - Type: service_docker_stdout
           Stdout: true
           Stderr: true
           # Configure container information filter conditions. Multiple options are in an "and" relationship.
           # Specify the namespace to which the pod containing the container to be collected belongs. Supports regular expression matching.
           K8sNamespaceRegex: "^(default)$"
           # Enable container metadata preview.
           CollectContainersFlag: true
           # Collect containers that meet the Pod label conditions. Multiple entries are in an "or" relationship.
           IncludeK8sLabel:
             app: ^(.*test.*)$
           # Configure multi-line chunk configuration. Invalid configuration for single-line log collection.
           # Configure the regular expression for the beginning of the line.
           BeginLineRegex: \d+-\d+-\d+.*
       # Define output plug-ins
       flushers:
         # Use the flusher_sls plug-in to send logs to the specified logstore.
         - Type: flusher_sls
           # Make sure that the logstore exists.
           Logstore: k8s-stdout
           # Make sure that the endpoint is valid.
           Endpoint: ap-southeast-1.log.aliyuncs.com
           Region: ap-southeast-1
           TelemetryType: logs

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container stdout and stderr to Simple Log Service.

CRD - AliyunLogConfig

To create a Logtail configuration, you need to only create a CR from the AliyunLogConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   This YAML script creates a Logtail configuration named simple-stdout-example that collects stdout and stderr in multi-line mode from all containers whose names start with app in the cluster, and sends the logs to the k8s-stdout Logstore in the project named k8s-log-<your_cluster_id>.

   For more information about the logtailConfig item in the YAML file, including supported input and output plugins, processing plugin types, and container filtering methods, see AliyunLogConfigDetail. For more information about all YAML parameters, see CR parameters.

   # Stdout configuration
   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster.
     name: simple-stdout-example
   spec:
     # Specify the name of the project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # The name of the Logstore. If the Logstore that you specify does not exist, Simple Log Service automatically creates a Logstore.
     logstore: k8s-stdout
     # Configure the Logtail configuration.
     logtailConfig:
       # The type of the data source. If you want to collect stdout and stderr logs, you must set the value to plugin.
       inputType: plugin
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name.
       configName: simple-stdout-example
       inputDetail:
         plugin:
           inputs:
             - type: service_docker_stdout
               detail:
                 # The settings that allow Logtail to collect both stdout and stderr logs.
                 Stdout: true
                 Stderr: true
                 # Specify the namespace of the pods to which the required containers belong. Regular expression matching is supported.
                 K8sNamespaceRegex: "^(default)$"
                 # Specify the name of the required containers. Regular expression matching is supported.
                 K8sContainerRegex: "^(app.*)$"
                 # Configure settings for multi-line log collection
                 # Specify the regular expression that is used to match the beginning of the first line of a log.
                 BeginLineRegex: \d+-\d+-\d+.*
                 

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container stdout and stderr to Simple Log Service.

Simple Log Service console

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you use to install Logtail components. Example: k8s-log-<your_cluster_id>. On the page that appears, click the Logstore that you want to manage, and then click Logtail Configurations. On the Logtail Configuration page, click Add Logtail Configuration. In the Quick Data Import dialog box, find the K8S - Stdout And Stderr - Old Version card and click Integrate Now.

3. In the Create Machine Group step of the Import Data wizard, click Use Existing Machine Groups.

4. In the Machine Group Configurations step of the Import Data wizard, set the Scenario parameter to Kubernetes Clusters and the Deployment Method parameter to ACK Daemonset, select the k8s-group-${your_k8s_cluster_id} machine group and click the > icon to move the machine group from the Source Machine Group section to the Applied Server Groups section, and then click Next.

5. Create a Logtail configuration. In the Logtail Configuration step of the Import Data wizard, configure the required parameters and click Next. Approximately 1 minute is required to create a Logtail configuration.

   You must configure the parameter in the Global Configurations section. For more information, see Create a Logtail configuration.

   • Global Configurations

     In the Global Configurations section, configure the Configuration Name parameter.

     

6. Create Indexes and Preview Data. By default, full-text indexing is enabled for Simple Log Service. In this case, full-text indexes are created. You can query all fields in logs based on the indexes. You can also manually create indexes for fields based on the collected logs. Alternatively, you can click Automatic Index Generation. Then, Simple Log Service generates indexes for fields. You can query data in an accurate manner based on field indexes. This reduces indexing costs and improves query efficiency. For more information, see Create indexes.