Business, application, or system logs that are scattered across different servers are difficult to retrieve, monitor, and analyze centrally. You can use LoongCollector (Logtail) from Simple Log Service to collect text logs in real time and incrementally from ECS instances, self-managed data centers, or hosts from other cloud providers. The logs are sent to a Logstore for centralized management and analysis. To collect full logs, you can import historical log files.
Scope
Supported operating systems and architectures
LoongCollector currently supports only Linux systems. For Windows hosts, you must use Logtail. We recommend using LoongCollector for new collection scenarios.
LoongCollector is the new-generation log collection agent for Simple Log Service and is an upgrade to Logtail. You only need to install either LoongCollector or Logtail. Do not install both.
Computing resource requirements
CPU: A minimum of 0.4 cores.
Memory: A minimum of 300 MB.
Recommended usage: To ensure stable operation, keep the actual resource usage of LoongCollector (Logtail) below 80% of the limit. Actual usage depends on factors such as collection speed, the number of monitored directories and files, and any sending blockages.
Permission requirements
If you use a RAM user, you must grant the
AliyunLogFullAccessandAliyunECSFullAccesspermissions. For more fine-grained control, see Appendix: Custom Permission Policies.
Collection configuration process
Create a Project and a Logstore: A Project is a resource management unit used to isolate logs from different services. A Logstore is the basic unit for log storage.
Configure a machine group (Install LoongCollector): Install LoongCollector based on the server type and add the server to a machine group. Machine groups are used to manage collection nodes, distribute configurations, and monitor server status.
Create and configure log collection rules:
Step 1: Global and Input Configuration: Defines the name of the collection configuration, the log source, and the collection scope.
Step 2: Log processing and structuring: Configure data processing based on the log format.
Multi-line logs: This applies to single log entries that span multiple lines, such as Java exception stack traces or Python tracebacks. Use a regular expression to match the first line of each log entry.
Structured parsing: Configure a parsing plugin, such as a regular expression, delimiter, or NGINX pattern plugin, to extract raw strings into structured key-value pairs. This facilitates subsequent queries and analysis.
Step 3: Log Filtering and Classification: You can configure topics, blacklists, and content filtering rules to enable fine-grained log classification and filtering. This lets you effectively isolate sensitive information and reduce redundant data, which in turn lowers storage costs.
Step 4: Configure queries and analysis: By default, the full-text index is enabled for keyword searches. We recommend that you also enable the field index to perform term queries and analysis on structured fields and improve retrieval efficiency.
Verification and troubleshooting: After the configuration is complete, verify that logs are being collected. For troubleshooting issues such as no data collection, heartbeat failures, or parsing errors, see Troubleshooting FAQ.
1. Create a project and a Logstore
Before you can collect logs, you must plan and create a project and a Logstore to manage and store them.
Project: A resource management unit of Simple Log Service, used to isolate and manage logs for different projects or services.
Logstore: A storage unit for logs.
If you have already created a machine group, you can skip this step and proceed directly to Configure a machine group (Install LoongCollector).
Log on to the Simple Log Service console.
Click Create Project and configure the following parameters:
Region: Select a region based on your log source. This parameter cannot be changed after the project is created.
Project Name: The name must be globally unique within Alibaba Cloud and cannot be changed after the project is created.
Leave the other parameters at their default values and click Create. For more information about the other parameters, see Manage a project.
Click the project name to go to the project details page.
In the navigation pane on the left, select
, and click +.On the Create Logstore page, configure the following parameters:
Logstore Name: A unique name for the Logstore within the Project that cannot be changed after creation.
Logstore Type: Select Standard or Query based on their specifications.
Billing Method:
Pay-by-feature: Resources such as storage, indexes, and read/write operations are billed independently. This billing method is suitable for small-scale scenarios or scenarios with unpredictable usage.
Pay-by-ingested-data: You are charged only for the volume of raw data written. This billing method includes a 30-day free storage period and free features, such as data transformation and delivery. This simple cost model is suitable for scenarios where the storage period is approximately 30 days or where the data processing pipeline is complex.
Data Retention Period: The number of days to retain logs. The value can be an integer from 1 to 3650. A value of 3650 indicates permanent storage. The default value is 30 days.
Leave the other parameters at their default values and click OK. For more information about other parameters, see Manage a Logstore.
2. Configure a machine group (install LoongCollector)
After you create a Project and a Logstore, install LoongCollector on your servers and add them to a machine group.
Click the target project. On the Logstores
All text log access templates differ only in their parsing plugins. The rest of the configuration process is the same, and you can modify the plugins later. |
|
page:
icon next to the name of the target Logstore to expand the details.
after Data Ingestion,
Scenario 1: Alibaba Cloud ECS instances
This method applies when the log source is an Alibaba Cloud ECS instance that is in the same Alibaba Cloud account and region as the Simple Log Service project. If the ECS instance and the project are not in the same account or region, see Scenario 2: Self-managed servers (Linux/Windows) to perform a manual installation.
Procedure
On the Machine Group Configuration page, configure the following parameters:
Scenario: Host Scenario
Installation Environment: ECS
Configure Machine Group: Based on the LoongCollector installation status and machine group configuration of the target server, choose one of the following operations:
If a machine group already has LoongCollector installed, you can select it from the Source Machine Group list and add it to the Application Machine Group list. You do not need to create a new one.
If LoongCollector is not installed, click Create Machine Group:
The following steps guide you through the automatic installation of LoongCollector and the creation of a new machine group.
The system automatically lists the ECS instances that are in the same region as the project. Select one or more instances from which to collect logs.
Click Install And Create As Machine Group. The system automatically installs LoongCollector on the selected ECS instances.
Configure the machine group Name and click OK.
NoteIf the installation fails or remains in a waiting state, verify that the ECS instance is in the same region as the project.
To add a server with LoongCollector installed to an existing machine group, see the FAQ: How do I add a server to an existing machine group?
Scenario 2: Self-managed servers (Linux/Windows)
If your log source is a self-managed server, such as a physical server in a local data center or a host from another cloud provider, you must install LoongCollector manually and configure a machine group to connect to Simple Log Service. Follow the steps below based on your operating system and network environment.
Procedure
On the Machine Group Configuration page, configure the following parameters:
Scenario: Host Scenario
Installation Environment: Select Self-managed Machine - Linux or Self-managed Machine - Windows
Configure Machine Group: Based on the LoongCollector installation status and machine group configuration of the target server, choose one of the following operations:
If LoongCollector is already installed and has been added to a machine group, you can directly select the machine group from the Source Machine Group list and add it to the Application Machine Group list. You do not need to create the machine group again.
If LoongCollector is not installed, click Create Machine Group:
The following steps guide you through the manual installation of LoongCollector and the creation of a new machine group.
Install LoongCollector (Logtail):
Self-managed Machine - Linux
Copy and run the installation command based on your network environment:
Alibaba Cloud Internal Network: Suitable for scenarios where you connect to the internal network from your self-managed servers.
Internet: This option is suitable for most scenarios and uploads log data to the target Logstore over the Internet.
Global Accelerator: If your server is in a mainland China region and your Simple Log Service project is in an overseas region, we recommend that you select transfer acceleration to avoid issues such as high network latency and instability when you transfer data over the Internet. However, additional charges apply to the traffic.
Before you run the Global Accelerator installation command, you must first enable the cross-domain log transfer acceleration feature for the project.
After the installation is complete, run the following command to check the status:
sudo /etc/init.d/loongcollectord statusIf the command returns
loongcollector is running, the startup was successful.
Self-managed Machine - Windows
Download the installation package for your region.
Unzip
logtail_installer.zipto the current directory.Run Windows PowerShell or cmd as an administrator and navigate to the
logtail_installerdirectory.Copy and run the installation command for your network type:
Alibaba Cloud Internal Network: Suitable for scenarios where you connect to the internal network from your self-managed servers.
Internet: This option is suitable for most scenarios and uploads log data to the target Logstore over the Internet.
Global Accelerator: If your server and Simple Log Service project are located in regions in mainland China and outside China, we recommend that you select transfer acceleration to avoid issues such as high latency and instability over the Internet. However, traffic is billed separately.
Before you run the Global Accelerator installation command, you must first enable the cross-domain log transfer acceleration feature for the project.
Configure the user identifier (AliUid): Obtain the Alibaba Cloud account ID that owns the Simple Log Service project. Create a file named after the account ID in the specified directory.
Replace ${AliUid} with your Alibaba Cloud account ID. If the specified directory does not exist, you must create it manually.
Linux:
/etc/ilogtail/users/${AliUid}Windows:
C:\LogtailData\users\${AliUid}
Configure a custom identifier: To ensure precise machine group binding, we recommend that you use a custom identifier for machine group identification.
Write a custom string to the specified file. If the directory does not exist, you must create it manually. The file path and name are fixed by Simple Log Service and cannot be customized.
NoteA machine group cannot contain both Linux and Windows servers. Do not configure the same custom identifier on both Linux and Windows servers. A server can be configured with multiple custom identifiers, separated by line feeds.
For Linux: write a custom string to the
/etc/ilogtail/user_defined_idfile, such asuser-defined-test-1.echo "user-defined-test-1" > /etc/ilogtail/user_defined_idWindows: In
C:\LogtailData\user_defined_id, write a custom string, such asuser-defined-test-1.
Configure a machine group: In the Machine Group Settings section of the console, configure the following:
Name: The name of the machine group, which must be unique within the project. The name must start and end with a lowercase letter or a digit, contain only lowercase letters, digits, hyphens (-), and underscores (_), and be 3 to 128 characters in length.
Machine Group Identifier: You can use a Custom Identifier or an IP Address. We recommend that you use a custom identifier. The identifier must match the custom string in the
user_defined_idfile on the server.If you want to configure an IP address as the machine group identifier, see the FAQ How do I obtain the server's IP address to use as a machine group identifier?.
Click Next. The Machine Group Heartbeat Status page appears. If the Heartbeat status is OK, the machine group is connected. Click Next to proceed to the Logtail configuration page.
If the status is FAIL, the initial heartbeat may take some time to establish. Wait for about two minutes and then refresh the heartbeat status. If the status is still FAIL after you refresh, see the FAQ topic Machine group heartbeat connection fails for troubleshooting.
3. Create and configure log collection rules
After you complete the LoongCollector installation and machine group configuration, go to the Logtail Configuration page to define log collection and processing rules.
Step 1: Global and input configuration
In this step, you define the name of the collection configuration and the source and scope of log collection.
Global Configurations:
Configuration Name: The name must be unique within the project and cannot be modified after creation. Naming conventions:
Can contain only lowercase letters, digits, hyphens (-), and underscores (_).
Must start and end with a lowercase letter or a digit.
Input Configuration:
Type: Text Log Collection.
File Path: The path from which to collect logs.
Linux: Must start with a forward slash (/). For example,
/data/mylogs/**/*.logmatches all files with the .log extension in the/data/mylogsdirectory.Windows: Must start with a drive letter. For example,
C:\Program Files\Intel\**\*.Log.
Maximum Directory Monitoring Depth: The maximum depth of subdirectories that the
**wildcard character in the File Path can match. The default value is 0, which means that only the current directory is monitored.
Step 2: Log processing and structuring
By configuring log processing rules, you can transform raw, unstructured logs into structured, searchable data. This improves the efficiency of log queries and analysis.
Before you configure processing rules, we recommend that you add a sample of the log that you want to collect. A log sample helps you accurately identify the log format and set processing parameters, such as the regular expression for multi-line logs and parsing plugins. This simplifies the configuration process. To add a sample log, perform the following operations:
On the Logtail Configuration page, in the Processing Configuration area, click Add Sample Log and enter a sample log.
Scenario 1: Process multi-line logs (such as Java stack traces)
Problem: Logs such as Java exception stack traces and JSON objects often span multiple lines. The default single-line collection mode splits them into multiple incomplete logs, which causes a loss of context.
Solution: Enable Multi-line Mode to merge multiple lines that belong to the same log entry into a single entry using a regular expression to match the first line.
Configuration steps: In the Processing Configuration area on the Logtail Configuration page, do the following:
| Raw log: |
Single-line mode: Each line is treated as a separate log. The stack trace is broken up and the context is lost.
| |
Multi-line mode: A regular expression for the first line identifies the complete log and preserves the full semantic structure. |
Scenario 2: Structure logs (such as NGINX access logs)
Problem: Although logs such as NGINX access logs are single-line, all the information is packed into a single string. This makes it difficult to perform precise queries based on fields such as IP address or status code.
Solution: Use a parsing plugin, such as a regular expression, JSON, or delimiter plugin, to extract the string into key-value pairs.
Procedure (NGINX example):
| Raw log: |
Parse into key-value pairs based on the |
Other parsing plugins: For logs that are not in the Nginx format, you can configure plugins such as regular expression parsing, separator-based parsing, and JSON parsing based on their actual format. For detailed configuration instructions, see Appendix: Native parsing plug-ins in detail.
Step 3: Log filtering and categorization
Use fine-grained controls to collect and store only the data you need. This ensures security and reduces costs.
Configure log topics
Problem: When multiple applications or instances have logs with the same format but different paths, such as /apps/app-A/run.log and /apps/app-B/run.log, it is difficult to distinguish their sources after they are collected together.
Solution: Configure log topics to flexibly distinguish log data from different machine groups, directory paths, or business sources within the same Logstore. This improves log management efficiency and query and analysis capabilities.
Procedure:
: Select the method for generating topics. The following three types are supported:
Type | Description |
Machine Group Topic | When a collection configuration is applied to multiple machine groups, LoongCollector automatically uses the name of the server's machine group as the value for the |
Custom | The format is |
File Path Extraction | Extracts key information from the full path of a log file to dynamically tag the log source. This is suitable for cases where multiple users or applications share the same log file name but have different paths. |
Details of file path extraction
When multiple users or services write logs to different top-level directories but the subdirectory paths and file names are the same, you cannot distinguish the source by file name alone. For example:
/data/logs ├── userA │ └── serviceA │ └── service.log ├── userB │ └── serviceA │ └── service.log └── userC └── serviceA └── service.logIn this case, you can configure File Path Extraction and use a regular expression to extract key information from the full path. The matched result is then uploaded to the Logstore as the log topic.
Extraction rules: Based on regular expression capturing groups
When you configure a regular expression, the system automatically determines the output field format based on the number and naming of capturing groups. The rules are as follows:
Single capturing group: Generates the
__topic__field.Named multiple capturing groups: Generates a tag field
__tag__:{name}.Multiple unnamed capturing groups: generates the tag field
__tag__:__topic_{i}__, where{i}is the ordinal number of the capturing group.
In the regular expression for the file path, you must escape the forward slash (/).
Capturing group type | Scenario | Regex example | Matching path example | Generated field |
Single capturing group (only one | Only one dimension is needed to distinguish the source (such as username or environment). |
|
|
|
Multiple unnamed capturing groups (multiple | Multiple dimensions are needed, but no semantic labels. |
|
|
|
Multiple named capturing groups ( | Multiple dimensions are needed, and you want the field meanings to be clear for easy querying and analysis. |
|
|
|
Reduce costs by filtering content
Problem: The volume of DEBUG or INFO level logs is huge, but daily troubleshooting requires only WARNING or ERROR level logs. Collecting all logs creates unnecessary costs.
Solution: Add a Data Filtering plugin to collect only logs that meet specified conditions.
Procedure
Click Add Processor, and select :
| Raw log: |
Filter logs: Set the Field Name to |
Control collection scope with a blacklist
Problem: In a production environment, some directories or files may contain debug logs, temporary data, or sensitive information such as user privacy or configuration files. Collecting this content indiscriminately leads to log redundancy, increased storage costs, and the risk of a data breach.
Solution: Enable the collection blacklist feature to proactively exclude specified directories or files. This prevents irrelevant or sensitive logs from being uploaded.
Procedure:
: Enable Collection Blacklist, click Add, and configure the blacklist.
Supports full matching and wildcard matching for directories and filenames. Wildcard characters only support the asterisk (*) and the question mark (?).
File Path Blacklist: The file path to be ignored. Example:
/home/admin/private*.log: During collection, ignore all files in the/home/admin/directory that start with "private" and end with ".log"./home/admin/private*/*_inner.log: During collection, ignore files ending with "_inner.log" within directories that start with "private" under the/home/admin/directory.
File Blacklist: Configure the filenames to be ignored during collection. Example:
app_inner.log: During collection, ignore all files namedapp_inner.log.
Directory Blacklist: The directory path cannot end with a forward slash (/). Example:
/home/admin/dir1/: The directory blacklist will not take effect./home/admin/dir*: During collection, ignore files in subdirectories that start with "dir" under the/home/admin/directory./home/admin/*/dir: During collection, ignore all files in subdirectories named "dir" at the second level under the/home/admin/directory. For example, files in the/home/admin/a/dirdirectory are ignored, while files in the/home/admin/a/b/dirdirectory are collected.
Step 4: Configure query and analysis
After you configure log processing and plug-ins, click Next. The Query And Analysis Configuration page appears.
By default, the full-text index is enabled. This lets you search raw log content by keyword.
To perform term queries by field, click Auto Generate Index after the Preview Data loads. Simple Log Service then generates a field index based on the first log entry in the preview data.
After you complete the configuration, click Next to finish the collection setup.
4. Verification and troubleshooting
After you complete the configuration, apply it to the machine group and save it. Wait a few moments, and then use the following checklist to verify the setup.
Verification checklist
Confirm that new content is added to the log file: LoongCollector collects only incremental logs. Run the
tail -f /path/to/your/log/filecommand and trigger a business operation to make sure that new logs are being written.Check the LoongCollector status:
sudo /etc/init.d/loongcollectord status.Check the machine group heartbeat: Go to the
Resources > Machine Groups page. Click the name of the target machine group. In the section, check the Heartbeat status.If the heartbeat status is OK, the machine group is connected to the Simple Log Service project.
If the heartbeat status is FAIL, see The machine group heartbeat is FAIL for troubleshooting.
Query logs: Go to the query and analysis page of the target Logstore to check if new data is reported.
FAQ
The machine group heartbeat is FAIL
Check the user identifier: If your server is not an ECS instance, or if the ECS instance and the project belong to different Alibaba Cloud accounts, check the specified directory for the correct user identifier as shown in the following table.
System
Specified directory
Solution
Linux
/etc/ilogtail/users/Run the
cd /etc/ilogtail/users/ && touch <uid>command to create the user identifier file.Windows
C:\LogtailData\users\Go to the
C:\LogtailData\users\directory and create an empty file named<uid>.If a file named after the Alibaba Cloud account ID of the project exists in the specified path, the user identifier is configured correctly.
Check the machine group identifier: If you use a machine group with a custom identifier, check if the
user_defined_idfile exists in the specified directory. If the file exists, check if its content matches the custom ID configured for the machine group.System
Specified directory
Solution
Linux
/etc/ilogtail/user_defined_id# Configure the custom identifier. If the directory does not exist, create it manually. echo "user-defined-1" > /etc/ilogtail/user_defined_idWindows
C:\LogtailData\user_defined_idIn the
C:\LogtailDatadirectory, create auser_defined_idfile and write the custom identifier in it. If the directory does not exist, create it manually.If both the user identifier and the machine group identifier are configured correctly, see Troubleshoot LoongCollector (Logtail) machine group issues for more information.
No data is collected from logs
Check for incremental logs: After you configure LoongCollector (Logtail) for collection, it does not collect data from a log file if no new logs are added.
Check the machine group heartbeat status: Go to the
Resources > Machine Groups page. Click the name of the target machine group. In the section, check the Heartbeat status.If the heartbeat status is OK, the machine group is connected to the Simple Log Service project.
If the heartbeat status is FAIL, see The machine group heartbeat is FAIL for troubleshooting.
Confirm that the LoongCollector (Logtail) collection configuration is applied to the machine group: Logs cannot be collected if the LoongCollector (Logtail) collection configuration is not applied to the machine group, even if the configuration is created.
Go to the
Resources > Machine Groups page. Click the name of the target machine group to go to the Machine Group Settings page.On the page, find Manage Configurations. The All Logtail Configurations list is on the left, and the Applied Logtail Configurations list is on the right. If the target LoongCollector (Logtail) collection configuration is in the list on the right, it is successfully applied to the machine group.
If the target LoongCollector (Logtail) collection configuration is not in the list on the right, click Modify. In the All Logtail Configurations list on the left, select the target LoongCollector (Logtail) configuration. Click
to move it to the list on the right, then click Save.
Log collection errors or incorrect format
Troubleshooting: This type of issue indicates that your network connectivity and basic configuration are correct. The problem is usually a mismatch between the log content and the parsing rules. Check the specific error message to identify the cause of the problem:
On the Logtail Configurations page, click the name of the LoongCollector (Logtail) configuration that has errors. On the Log Collection Errors tab, click Time Range to set the time range for the query.
In the area, view the Alarm Metric for the error log and find the corresponding solution in Common error types for data collection.
Quotas and limits
Limit | Description |
Single log length | The default limit is 512 KB. You can adjust this limit using the max_read_buffer_size startup parameter. The maximum size cannot exceed 8 MB. For more information, see Logtail network types, startup parameters, and configuration files. After a multi-line log is split by the regular expression that matches the start of a line, each resulting log entry is still limited to 512 KB. If a log entry exceeds 512 KB, it is split into multiple entries for collection. For example, if a log entry is 1025 KB, Logtail processes 512 KB, then another 512 KB, and finally 1 KB. This results in multiple incomplete log entries. |
File encoding | Logtail supports log files encoded in UTF-8 or GBK. Use UTF-8 encoding for better performance. Warning If a log file uses a different encoding format, issues such as garbled characters and data loss may occur. |
Log file rotation | The default size of the log rotation queue is 20. You can adjust this size using the logreader_max_rotate_queue_size startup parameter. For more information, see Logtail network types, startup parameters, and configuration files. You can set the collection path to either Important Do not mix these two formats in the same Logtail instance. This can cause a single file to match multiple Logtail collection configurations, which results in duplicate data collection. If more than 20 files are waiting to be processed, newly generated logs are lost. If this happens, first check whether the Logstore shard write quota is exceeded. Then, adjust the Logtail concurrency level. For more information, see Logtail network types, startup parameters, and configuration files. |
Collection behavior when log parsing is blocked | If log parsing is blocked, Logtail keeps the log file descriptor open. This prevents the file from being deleted and avoids data loss. If log files are rotated while parsing is blocked, Logtail adds the files to the rotation queue. |
Regular expression | Logtail supports Perl-compatible regular expressions. |
JSON | Logtail fully supports standard JSON (RFC7159 and ECMA-404). Non-standard JSON, such as |
File open behavior | Logtail keeps collected files and files in the rotation queue open to ensure data integrity. A file is closed in the following situations:
To release a file handle within a specific time after the file is deleted, use the force_release_deleted_file_fd_timeout startup parameter to set a timeout. This occurs regardless of whether collection is complete or logs are still being written to the file. For more information, see Logtail network types, startup parameters, and configuration files. |
Initial log collection behavior | Logtail collects only incremental changes to log files. When Logtail detects a modified file for the first time, it starts collection from a specific point. If the file is larger than 1 MB (512 KB for container standard output), collection starts from the last 1 MB of the file. Otherwise, collection starts from the beginning of the file. You can adjust the initial collection size for new files using the tail_size_kb parameter in the Logtail configuration. For more information, see Logtail configuration (Legacy). If a log file is not modified after you apply a Logtail collection configuration, Logtail does not collect data from that file. To collect data from historical files, see Import historical log files. |
Behavior when a file is overwritten | Logtail identifies a file by its inode and a hash of its first 1024 bytes. If a file is overwritten and its inode or the hash of its first 1024 bytes changes, Logtail treats it as a new file and collects it from the beginning. Otherwise, the file is not collected. |
Behavior when a file is moved | If a file is moved, Logtail treats it as a new file and collects it from the beginning if it matches a Logtail collection configuration that has not previously matched the file. Otherwise, the file is not collected. |
File collection history | Logtail stores the collection progress for files in memory. This ensures that only incremental changes are collected when a file is modified. If data is written to a file that is outside the retention scope, duplicate collection occurs.
|
Non-standard text logs | For log entries that contain the |
Billing
Installing LoongCollector or Logtail is free of charge.
You are charged for log writing, storage, indexing, querying, processing, and delivery based on the Logstore billing model.
If you use the Global Accelerator feature during installation or configuration, you incur additional traffic fees for data transmitted over the accelerated network.
FAQ
How do I send logs from an ECS server to a Project in another Alibaba Cloud account?
If you have not installed LoongCollector, see Install LoongCollector (Linux) and select the appropriate cross-account scenario for installation.
If you have already installed LoongCollector, follow these steps to configure a custom identifier. This identifier grants the server permission to be accessed by the Alibaba Cloud account that owns the Simple Log Service project, which allows log collection.
You only need to configure a custom identifier when collecting logs from an ECS instance that is not under your account, a self-managed IDC, or a server from another cloud provider.
Copy the ID of the Alibaba Cloud account that owns Simple Log Service. Hover your mouse over your profile picture in the upper-right corner. You can view and copy the account ID from the tab that appears.
Log on to the server from which you want to collect logs and create an Alibaba Cloud account ID file to configure the custom identifier:
touch /etc/ilogtail/users/{Alibaba Cloud account ID} # If the /etc/ilogtail/users folder does not exist, create it manually. The custom identifier configuration file only needs a file name, not a file extension.
How do I send logs from an ECS server to a Project in a different region under the same account?
If you have not installed LoongCollector, see Install LoongCollector (Linux) and select the appropriate cross-region scenario for installation.
If LoongCollector is already installed, you need to modify its configuration.
Run the
sudo /etc/init.d/ilogtaild stopcommand to stop LoongCollector.Modify the LoongCollector startup configuration file
ilogtail_config.json. Choose one of the following two methods based on your network requirements:Configuration file path:
/usr/local/ilogtail/ilogtail_config.jsonMethod 1: Use the Internet for transmission
See Region IDs and replace the region in the configuration file with the region where Simple Log Service is located. The fields to modify include the following:
primary_regionThe region part in
config_serversThe
regionand the region part ofendpoint_listindata_servers
Method 2: Use transfer acceleration
Replace the endpoint line in the data_server_list parameter with
log-global.aliyuncs.com. For the file path, see Logtail network types, startup parameters, and configuration files.
Run the
sudo /etc/init.d/ilogtaild startcommand to start LoongCollector.
How do I add a server to an existing machine group?
If you have a configured machine group and want to add a new server, such as a newly deployed ECS instance or a self-managed server, to inherit its collection configurations, you can attach it by following these steps.
Prerequisites:
A configured machine group already exists.
LoongCollector is installed on the new server. For more information, see Install LoongCollector.
Procedure:
View the identifier of the target machine group:
On the page of the target Project, in the navigation pane on the left, click
.On the Machine Groups page, click the name of the target machine group.
On the machine group configuration page, view the machine group identifier.
Perform the corresponding operation based on the identifier type:
NoteA machine group cannot contain both Linux and Windows servers. Do not configure the same custom identifier on both Linux and Windows servers. A server can have multiple custom identifiers, separated by line feeds.
Type 1: The machine group identifier is an IP address
On the server, run the following command to open the
app_info.jsonfile and view theipvalue.cat /usr/local/ilogtail/app_info.jsonOn the target machine group configuration page, click Modify. Enter the IP address of the server. Separate multiple IP addresses with line feeds.
After the configuration is complete, click Save and confirm the heartbeat status. If the status is OK, the server automatically applies the collection configurations of the machine group.
If the heartbeat status is FAIL, see the FAQ The heartbeat of a machine group is FAIL for troubleshooting.
Type 2: The machine group identifier is a custom identifier
Based on the operating system, write the custom identifier string that matches the target machine group to the specified file:
If the folder does not exist, you must create it manually. The file path and name are fixed by Simple Log Service and cannot be customized.
Linux: Write the custom string to the
/etc/ilogtail/user_defined_idfile.Windows: Write the custom string to
C:\LogtailData\user_defined_id.
How do I import collection configurations from another Project?
After you create a Project and a Logstore and configure a machine group, you can quickly import collection configurations from an existing project to the current Logstore. This avoids reconfiguration and improves efficiency.
Procedure:
After you configure the machine group, click Next to go to the Logtail Configuration page.
In the upper-right corner of the page, click Import Other Configurations.
Select the Project and the collection configuration under that Project to import.
Click OK. The system automatically loads the selected configuration.
After you check that the imported configuration information is correct, click Next to go to the Query and Analysis Configuration page and complete the subsequent configurations.
How do I get the IP address of a server to use as a machine group identifier?
On a server where LoongCollector (Logtail) is installed, open the /usr/local/ilogtail/app_info.json file to view the ip value.
The server IP address that Logtail automatically obtains is recorded in the ip field of the app_info.json file, as shown below.
If you have multiple servers, you must manually enter their IP addresses. Separate the IP addresses with line feeds.
A machine group cannot contain both Linux and Windows servers. Do not add the IP addresses of both Windows and Linux servers to the same Machine Group.
How can I have the same log file collected by multiple collection configurations at the same time?
By default, to avoid data duplication, Simple Log Service allows a text log file to be collected by only one Logtail configuration. To have the same log file collected by multiple configurations simultaneously, you must manually enable the feature that allows a file to be collected multiple times.
Procedure:
When a file is collected multiple times, the file read I/O, computing resources, and network I/O all increase linearly.
Log on to the Simple Log Service console and go to the target Project.
In the navigation pane on the left, choose
Logstores and find the target Logstore.Click
next to its name to expand the Logstore.Click Logtail Configurations. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.
On the Logtail configuration page, click Edit:
Choose and enable Allow Multiple Collections For A File.
After the configuration is complete, click Save.
Why is the last log segment reported after a long delay? Why is it sometimes truncated?
Cause analysis: Log truncation usually occurs when a log file is missing a line feed at the end, or when a multi-line log, such as an exception stack, has not been fully written. Because the collector cannot determine whether the log entry is complete, the last segment might be split prematurely or reported with a delay. Different versions of LoongCollector (Logtail) handle this differently:
Versions earlier than 1.8:
If the last line of a log does not have a line feed (carriage return), or if a multi-line log paragraph is not finished, the collector waits for the next write to trigger an output. This can cause the last log entry to be held for a long time and not sent until a new log is written.Versions 1.8 and later:
A timeout refresh mechanism was introduced to prevent logs from getting stuck. When an incomplete log line is detected, the system starts a timer. After the timeout, it automatically submits the current content to ensure the log is eventually collected.Default timeout: 60 seconds (This ensures integrity in most scenarios).
You can adjust this value as needed, but setting it to 0 is not recommended because doing so may cause log truncation or partial data loss.
Solution:
You can extend the waiting time to ensure the complete log is written before it is collected:
Log on to the Simple Log Service console and go to the target Project.
In the navigation pane on the left, choose
Logstores and find the target Logstore.Click
next to its name to expand the Logstore.Click Logtail Configurations. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.
On the Logtail configuration page, click Edit:
Choose . Add the following JSON configuration to customize the timeout period:
{ "FlushTimeoutSecs": 1 }Default value: Determined by the startup parameter
default_reader_flush_timeout(usually a few seconds).Unit: Seconds.
Recommended value: ≥1 second. Setting it to 0 is not recommended because it may cause log truncation or partial data loss.
After the configuration is complete, click Save.
Why does LoongCollector (Logtail) switch from an internal endpoint to a public endpoint during operation? Can it automatically switch back?
During operation, if LoongCollector (Logtail) detects an issue with communication over the internal endpoint, such as a network failure or connection timeout, it automatically switches to the public endpoint to send data. This switch ensures the continuity and reliability of log collection and prevents logs from stacking up or being lost.
LoongCollector: Automatically switches back to the internal network after it recovers.
Logtail: Does not automatically switch back. You must restart it manually to resume communication over the internal network.
Appendix: Native parsing plug-ins in detail
On the Logtail Configuration page, in the Processor Configurations section, you can add processing plugins to structure raw logs. To add a processing plugin to an existing collection configuration, perform the following steps:
In the navigation pane on the left, choose
Logstores and find the destination Logstore.Click
next to its name to expand the Logstore.Click Logtail Configurations. In the list of configurations, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.
On the Logtail Configuration page, click Edit.
This section introduces common processing plug-ins for typical log processing scenarios. For more features, see Extension processing plug-ins.
Rules for combining plug-ins (for LoongCollector / Logtail 2.0 and later):
Native processing plug-ins and extension processing plug-ins can be used independently or combined as needed.
Prioritize native processing plug-ins because they offer better performance and greater stability.
If native features do not meet your needs, append extension processing plug-ins after the configured native plug-ins for additional processing.
Order constraint:
All plug-ins form a processing chain and are executed in the order they are configured. Note: All native processing plug-ins must precede any extension processing plug-ins. After you add an extension processing plug-in, you can no longer add native processing plug-ins.
Regular expression parsing
Use regular expressions to extract log fields and parse logs into key-value pairs.
Configure . To add one, click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Custom regular expression parsing: Regular expression |
Separator parsing
Use a separator to structure log content and parse it into multiple key-value pairs. Single-character and multi-character separators are supported.
Configure . To add one, click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Split fields by the specified character |
Standard JSON parsing
Structure Object-type JSON logs and parse them into key-value pairs.
Configure . To add one, click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Automatic extraction of standard JSON key-value pairs: |
Nested JSON parsing
Parse nested JSON logs into key-value pairs by specifying an expansion depth.
Click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Unfold depth: 0, using the unfold depth as the prefix. | |
Unfold depth: 1, using the unfold depth as the prefix. |
JSON array parsing
Use the json_extract function to extract JSON objects from a JSON array.
Switch Processing Method to SPL:
| Raw log: |
Extracted JSON array structure: |
Apache log parsing
Structure log content and parse it into multiple key-value pairs based on the definitions in the Apache log configuration file.
Configure . To add one, click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Apache Common Log Format |
IIS log parsing
Structure log content and parse it into multiple key-value pairs based on the IIS log format definition.
Configure . To add one, click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Adaptation for Microsoft IIS server-specific format: |
Data masking
Mask sensitive data in logs.
Click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Masking result: |
Time parsing
Parse the time field in a log and set the parsing result as the log's __time__ field.
Click Add Processing Plug-in and choose . The parameters are described as follows:
| Raw log: |
Time parsing:
|
Appendix: Access policy reference
Alibaba Cloud account: By default, an Alibaba Cloud account has full permissions and can perform all operations.
RAM user: An Alibaba Cloud account must grant a corresponding access policy to the RAM user.
Custom policies (fine-grained control)
If system policies do not meet the principle of least privilege, you can create a custom policy. Use the following example policy, which includes the following primary permissions:
View projects: View the list of projects and the details of a specific project.
Manage Logstores: Create, modify, or delete Logstores in a project.
Manage collection configurations: Create, delete, and modify collection configurations.
View logs: Query and analyze data in a specific Logstore within a specific project.
Replace${regionName},${uid},${projectName}, and${logstoreName}with the actual region, Alibaba Cloud account ID, target project, and Logstore name.
Permission | Operations | Resource |
Read-only access to projects |
|
|
Get a specific project |
|
|
Manage Logstores |
|
|
Manage Logtail data collection |
|
|
Query saved searches |
|
|
Query dashboards |
|
|
Query logs of a specific Logstore |
|
|
ECS operation permissions |
|
|
OOS operation permissions (Optional) Required only to automatically install Logtail using OOS when the Simple Log Service project and the ECS instance are in the same account and region. |
|
|
System policies
When using system-defined policies, add the following permissions:
AliyunLogFullAccess: Grants permissions to manage Simple Log Service.AliyunECSFullAccess: Grants permissions to manage ECS.(Optional)
AliyunOOSFullAccess: Required to install Logtail from OOS with a single click.
More information
Processor configuration parameters
Parameter | Description |
Log Sample | A sample of the log to be collected. Make sure to use a log from a real-world scenario. The log sample helps you configure log processing parameters and simplifies the configuration. You can add multiple samples. The total length cannot exceed 1500 characters. |
Multi-line Mode |
|
Processing Method | Processors, including Native Processing Plugins and Extended Processing Plugins. For more information about processing plugins, see Using native and extended processing plugins. Important For the limits on processing plugins, refer to the prompts on the console page.
|
