Issue
Commerce Cloud offers customers to view and search logs in dedicated instance of AWS OpenSearch. Disk quota there is limited, so after some time logs are gone there, but you can find them still available dumped in your dedicated Azure Blob Storage.
You can find that on your Cloud Portal: Environments > [given environment] : Cloud Storage > logs
Unfortunately these files are hard to use as blob entries are impossible to filter by creation date and also by content.
Solution
In order to filter files I have used java code to connect to Azure Blob Storage and then filter roughly by date.
Download Azure Blob Files
In below example I wanted to filter our files which has been created from 11-Aug-2023, 5 AM UTC to 12-Aug-2023 11:59 PM UTC:
package pl.com.like.zip; import com.azure.identity.*; import com.azure.storage.blob.*; import com.azure.storage.blob.models.*; import java.io.*; public class BlobLogFetch { public static void main(String[] args) throws IOException { var connectionString = "AccountName=XXXXXXXXXXX;AccountKey=YYYYYYYYYYYYY;EndpointSuffix=core.windows.net;DefaultEndpointsProtocol=https;"; var container = "commerce-logs-separated"; var containerClient = new BlobContainerClientBuilder().connectionString(connectionString) .containerName(container).buildClient(); System.out.println(containerClient.exists()); var from = OffsetDateTime.of(2023, 8, 11, 5, 0, 0, 0, ZoneOffset.UTC); var to = OffsetDateTime.of(2023, 8, 12, 23, 59, 0, 0, ZoneOffset.UTC); containerClient.listBlobs().stream().filter(i -> i.getProperties().getCreationTime().compareTo(from) > 0) .filter(item -> item.getProperties().getCreationTime().compareTo(to) < 0).forEach(i -> { var blob = containerClient.getBlobClient(i.getName()); var path = Paths.get(i.getName()).normalize().toAbsolutePath(); // create dirs path.toFile().getParentFile().mkdirs(); blob.downloadToFile(path.toString(), Boolean.TRUE); Above code will list all blob files, filter by file properties if meets date range requirements. If creation date is fine, file is downloaded.
Install OpenSearch
Then, we would like to be able to store file content where data are easily searchable.
There are few options on table, but if we have access to SAP Commerce Cloud, we naturally have used Amazon OpenSearch, so we will follow same tooling locally.
It is possible to find OpenSearch on Docker Hub searching for opensearchproject/opensearch.
I have used linked docker-compose.yml as it contains additionally Dashboard.
version: '3' services: opensearch-node1: image: opensearchproject/opensearch:latest container_name: opensearch-node1 environment: - cluster.name=opensearch-cluster - node.name=opensearch-node1 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data1:/usr/share/opensearch/data ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net opensearch-node2: image: opensearchproject/opensearch:latest container_name: opensearch-node2 environment: - cluster.name=opensearch-cluster - node.name=opensearch-node2 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data2:/usr/share/opensearch/data networks: - opensearch-net opensearch-dashboards: image: opensearchproject/opensearch-dashboards:latest container_name: opensearch-dashboards ports: - 5601:5601 expose: - "5601" environment: OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' networks: - opensearch-net volumes: opensearch-data1: opensearch-data2: networks: opensearch-net: To run it locally just download file and run:
docker-compose up -d OpenSearch Configuration
When docker compose is started, you should be able to find OpenSearch Dashboard available on http://localhost:5601/ and logging with credentials admin:admin.
Initially instance will not have any data. We need to create an index first. To do that, click on hamburger menu: OpenSearch Plugins > Index Management, there go to: Index Management > Indices and create index. On my instance I have created index called commerce.
During index creation you can set up data mapping. Below you can find mapping which I have used:
{ "properties": { "@timestamp": { "type": "date" }, "_p": { "type": "text" }, "kubernetes": { "type": "object", "properties": { "annotations": { "type": "object", "properties": { "ccv2_cx_sap_com_build-code": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_deployment-id": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cni_projectcalico_org_containerID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cni_projectcalico_org_podIP": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cni_projectcalico_org_podIPs": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "data-ingest_dynatrace_com_injected": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "dynakube_dynatrace_com_injected": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "fluentbit_io_parser": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "oneagent_dynatrace_com_injected": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "platform-security-configmap-hash": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "container_hash": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "container_image": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "container_name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "docker_id": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "host": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "labels": { "type": "object", "properties": { "app_kubernetes_io_component": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "app_kubernetes_io_managed-by": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "app_kubernetes_io_name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "app_kubernetes_io_part-of": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_build-code": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_deployment-id": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_platform-aspect": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_service-name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "ccv2_cx_sap_com_service-version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "controller-revision-hash": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "controller-uid": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "job-name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "pod-template-hash": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "statefulset_kubernetes_io_pod-name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "update_ccv2_cx_sap_com_mode": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "namespace_name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "pod_id": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "pod_name": { "type": "text" } } }, "log": { "type": "text" }, "logs": { "type": "object", "properties": { "@timestamp": { "type": "date" }, "@version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "buildDate": { "type": "date" }, "container": { "type": "object", "properties": { "image": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "content": { "type": "object", "properties": { "apiToken": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "apiUrl": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "clusterID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "hasHost": { "type": "boolean" }, "hostGroup": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "initialConnectRetry": { "type": "long" }, "monitoringNodes": { "type": "object", "properties": { "aks-ey4c7ymiac-68098534-vmss000012": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "aks-ey4c7ymiac-68098534-vmss000013": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "aks-ey4c7ymiac-68098534-vmss000016": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "aks-ey4c7ymiac-68098534-vmss000017": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "aks-ey4c7ymiac-68098534-vmss000019": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "aks-ey4c7ymiac-68098534-vmss00001a": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "networkZone": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "paasToken": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "proxy": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "skipCertCheck": { "type": "boolean" }, "tenantUUID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "tlsCert": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "trustedCAs": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "contextMap": { "type": "object", "properties": { "CronJob": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "RemoteAddr": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "Tenant": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "audit": { "type": "object", "properties": { "actionId": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "actionName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "data": { "type": "object", "properties": { "PK": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "changeType": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cronJob": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cronJobResult": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "cronJobStatus": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "gyIsActive": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "gyUID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "idVerified": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "idVerifiedAt": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "job": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "newUser": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "prevUser": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "residenceVerified": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "residenceVerifiedAt": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "typePK": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "user": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "sessionId": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "user": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "bytesSent": { "type": "long" }, "commitMillis": { "type": "long" }, "connectionStatus": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "myid": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "processMillis": { "type": "long" }, "remoteHost": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "requestLine": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "sourceClassName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "sourceMethodName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "statusCode": { "type": "long" } } }, "endOfBatch": { "type": "boolean" }, "env": { "type": "object", "properties": { "containers": { "type": "object", "properties": { "image": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "dataIngestInjected": { "type": "boolean" }, "failurePolicy": { "type": "boolean" }, "installPath": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "installVersion": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "installerFlavor": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "installerTech": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "installerUrl": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8BasePodName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8BasePodUID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8ClusterID": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8Namespace": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8NodeName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "k8PodName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "mode": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "oneAgentInjected": { "type": "boolean" }, "workloadKind": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "workloadName": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "gitCommit": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "goVersion": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "hostTenant": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "instant": { "type": "object", "properties": { "epochSecond": { "type": "text" }, "nanoOfSecond": { "type": "text" } } }, "level": { "type": "text" }, "level_value": { "type": "long" }, "logger": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "loggerFqcn": { "type": "text" }, "loggerName": { "type": "text" }, "logger_name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "marker": { "type": "object", "properties": { "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "message": { "type": "text" }, "msg": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "origin": { "type": "text" }, "platform": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "thread": { "type": "text" }, "threadId": { "type": "integer" }, "threadPriority": { "type": "integer" }, "thread_name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "thrown": { "type": "object", "properties": { "cause": { "type": "object", "properties": { "cause": { "type": "object", "properties": { "cause": { "type": "object", "properties": { "commonElementCount": { "type": "long" }, "extendedStackTrace": { "type": "object", "properties": { "class": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "exact": { "type": "boolean" }, "file": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "line": { "type": "long" }, "location": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "method": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "localizedMessage": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "message": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "commonElementCount": { "type": "long" }, "extendedStackTrace": { "type": "object", "properties": { "class": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "exact": { "type": "boolean" }, "file": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "line": { "type": "long" }, "location": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "method": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "localizedMessage": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "message": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "commonElementCount": { "type": "long" }, "extendedStackTrace": { "type": "object", "properties": { "class": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "exact": { "type": "boolean" }, "file": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "line": { "type": "long" }, "location": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "method": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "localizedMessage": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "message": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "commonElementCount": { "type": "long" }, "extendedStackTrace": { "type": "object", "properties": { "class": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "exact": { "type": "boolean" }, "file": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "line": { "type": "long" }, "location": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "method": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "localizedMessage": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "message": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" }, "name": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "timeMillis": { "type": "long" }, "ts": { "type": "date" }, "version": { "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "type": "text" } } }, "record_date": { "type": "long" }, "stream": { "type": "text" }, "time": { "type": "date" } } } Feeding Search Index
Now is the time to upload data from local files to index. Data in blob items is already nicely prepared and stored in JSON format as 1-liner.
Additionally I have added code, which ignores default unsigned SSL certificate, which by default is used locally:
import groovy.json.JsonOutput import javax.net.ssl.HostnameVerifier; import javax.net.ssl.SSLSession; import javax.net.ssl.HttpsURLConnection import javax.net.ssl.SSLContext import javax.net.ssl.SSLEngine import javax.net.ssl.SSLSocketFactory import javax.net.ssl.TrustManager import javax.net.ssl.X509ExtendedTrustManager import java.security.cert.CertificateException import java.security.cert.X509Certificate import java.util.Base64; class UnsafeTrustManager extends X509ExtendedTrustManager { @Override void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {} @Override void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {} @Override void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {} @Override void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {} @Override void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {} @Override void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {} @Override X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0] } } def handleError(httpResponseException) { println "Request failed with status code: ${httpResponseException.statusCode}" println "Error message: ${httpResponseException.message}" } def dir = new File('./') // List all files in the directory def postmanPost = new URL('https://localhost:9200/commerce/_doc') def v = new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { return true; } }; def sslContext = SSLContext.getInstance("TLS") sslContext.init(null, new TrustManager[]{new UnsafeTrustManager()}, null) def sslSocketFactory = sslContext.getSocketFactory() def userCredentials = "admin:admin" String basicAuth = Base64.getEncoder().encodeToString(userCredentials.getBytes()); HttpsURLConnection.setDefaultHostnameVerifier(v); dir.eachFile { file -> if (file.name.endsWith(".log")) { println "Sending file ${file.name}" def inputFile = new File(dir, file.name); // Read file line by line inputFile.eachLine { line -> try { HttpsURLConnection conn = (HttpsURLConnection) postmanPost.openConnection(); HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { return true; } }); conn.setSSLSocketFactory(sslSocketFactory); conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8"); conn.setRequestProperty ("Authorization", "Basic ${basicAuth}"); conn.setDoOutput(true); conn.setDoInput(true); conn.setRequestMethod("POST"); conn.setHostnameVerifier(v); def os = conn.getOutputStream(); os.write(line.trim().getBytes("UTF-8")); os.close(); if (conn.getResponseCode() >= 300) { println("Failed : HTTP error code : ${conn.getResponseCode()} for ${line}"); } } catch(Exception e) { println("Error: ${e}, for line ${line}"); } } } } Run script by calling:
groovy [script-name].groovy After that, you will be able to see data in you Dashboard.
Top comments (0)