Skip to content

Commit ce98afa

Browse files
mykola-elasticmykola-elastic
authored
[airflow] Make Airflow package GA (#15287)
1 parent 5811423 commit ce98afa

File tree

15 files changed

+1937
-1022
lines changed

15 files changed

+1937
-1022
lines changed

‎packages/airflow/_dev/build/build.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
dependencies:
22
ecs:
3-
reference: "git@v8.11.0"
3+
reference: "git@v8.17.0"

‎packages/airflow/_dev/build/docs/README.md‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ Use the Airflow integration to:
1010
- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
1111
- Monitor your workflows' performance and status in real-time.
1212

13-
## Data streams
13+
## Compatibility
14+
15+
The Airflow package is tested with Airflow `2.4.3`. It should work with any `2.*` version.
16+
17+
## What data does this integration collect?
1418

1519
The Airflow integration gathers metric data.
1620

@@ -23,19 +27,13 @@ Data stream:
2327
Note:
2428
- Users can monitor and view metrics within the ingested documents for Airflow in the `metrics-*` index pattern from `Discover`.
2529

26-
## Compatibility
27-
28-
The Airflow module is tested with Airflow `2.4.0`. It should work with versions `2.0.0` and later.
29-
3030
## What do I need to use this integration?
3131

3232
Users require Elasticsearch to store and search user data, and Kibana to visualize and manage it. They can utilize the hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on their own hardware.
3333

34-
To ingest data from Airflow, users must have [StatsD](https://github.com/statsd/statsd) to receive the same.
35-
36-
## Setup
34+
## How do I deploy this integration?
3735

38-
For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.
36+
For step-by-step instructions on how to set up an integration, refer to the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.
3937

4038
## Steps to Setup Airflow
4139

‎packages/airflow/_dev/deploy/docker/sample_logs/test-airflow.log‎

Lines changed: 95 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,100 @@
1-
dag_processing.total_parse_time:1.066219636995811|g
2-
dagbag_size:0|g
1+
triggers.running.b33243cee4eb:0|g
2+
triggers.running:0|g
3+
triggerer.capacity_left.b33243cee4eb:1000|g
4+
triggerer.capacity_left:1000|g
5+
scheduler.critical_section_query_duration:1.459139|ms
6+
scheduler.tasks.starving:0|g
7+
scheduler.tasks.executable:0|g
8+
scheduler.critical_section_duration:6.649818|ms
9+
executor.open_slots:32|g
10+
executor.queued_tasks:0|g
11+
executor.running_tasks:0|g
12+
scheduler.scheduler_loop_duration:26.456103|ms
13+
dag_processing.file_path_queue_size:0|g
14+
dag_processing.file_path_queue_update_count:1|c
15+
dag_processing.total_parse_time:0.0017347259999951348|g
16+
dagbag_size:81|g
17+
dag_processing.import_errors:0|g
18+
dag_processing.file_path_queue_size:0|g
19+
dag_processing.file_path_queue_update_count:1|c
20+
dag_processing.total_parse_time:0.0015908499999568448|g
21+
dagbag_size:81|g
22+
dag_processing.import_errors:0|g
23+
triggers.running.b33243cee4eb:0|g
24+
triggers.running:0|g
25+
triggerer.capacity_left.b33243cee4eb:1000|g
26+
triggerer.capacity_left:1000|g
27+
scheduler.critical_section_query_duration:2.268273|ms
28+
scheduler.tasks.starving:0|g
29+
scheduler.tasks.executable:0|g
30+
scheduler.critical_section_duration:8.062624|ms
31+
executor.open_slots:32|g
32+
executor.queued_tasks:0|g
33+
executor.running_tasks:0|g
34+
scheduler.scheduler_loop_duration:41.562299|ms
35+
dag_processing.file_path_queue_size:0|g
36+
dag_processing.file_path_queue_update_count:1|c
37+
dag_processing.total_parse_time:0.0028237789997547225|g
38+
dagbag_size:81|g
39+
dag_processing.import_errors:0|g
40+
dag_processing.file_path_queue_size:0|g
41+
dag_processing.file_path_queue_update_count:1|c
42+
dag_processing.total_parse_time:0.0037441220001710462|g
43+
dagbag_size:81|g
44+
dag_processing.import_errors:0|g
45+
scheduler.critical_section_query_duration:2.240898|ms
46+
scheduler.tasks.starving:0|g
47+
scheduler.tasks.executable:0|g
48+
scheduler.critical_section_duration:7.354492|ms
49+
executor.open_slots:32|g
50+
executor.queued_tasks:0|g
51+
executor.running_tasks:0|g
52+
scheduler.scheduler_loop_duration:31.234402|ms
53+
triggers.running.b33243cee4eb:0|g
54+
triggers.running:0|g
55+
triggerer.capacity_left.b33243cee4eb:1000|g
56+
triggerer.capacity_left:1000|g
57+
scheduler.critical_section_query_duration:1.584433|ms
58+
scheduler.tasks.starving:0|g
59+
scheduler.tasks.executable:0|g
60+
scheduler.critical_section_duration:3.518036|ms
61+
executor.open_slots:32|g
62+
executor.queued_tasks:0|g
63+
executor.running_tasks:0|g
64+
scheduler_heartbeat:1|c
65+
pool.open_slots.default_pool:128|g
66+
pool.queued_slots.default_pool:0|g
67+
pool.running_slots.default_pool:0|g
68+
pool.deferred_slots.default_pool:0|g
69+
pool.scheduled_slots.default_pool:0|g
70+
pool.open_slots:128|g
71+
pool.queued_slots:0|g
72+
pool.running_slots:0|g
73+
pool.deferred_slots:0|g
74+
pool.scheduled_slots:0|g
75+
scheduler.scheduler_loop_duration:26.222143|ms
76+
dag_processing.file_path_queue_size:0|g
77+
dag_processing.file_path_queue_update_count:1|c
78+
dag_processing.total_parse_time:0.002773403999981383|g
79+
dagbag_size:81|g
80+
dag_processing.import_errors:0|g
81+
dag_processing.file_path_queue_size:0|g
82+
dag_processing.file_path_queue_update_count:1|c
83+
dag_processing.total_parse_time:0.002388358000189328|g
84+
dagbag_size:81|g
385
dag_processing.import_errors:0|g
86+
triggers.running.b33243cee4eb:0|g
87+
triggers.running:0|g
88+
triggerer.capacity_left.b33243cee4eb:1000|g
89+
triggerer.capacity_left:1000|g
90+
scheduler.critical_section_query_duration:1.903853|ms
491
scheduler.tasks.starving:0|g
592
scheduler.tasks.executable:0|g
6-
scheduler.critical_section_duration:7.831787|ms
93+
scheduler.critical_section_duration:5.178595|ms
794
executor.open_slots:32|g
895
executor.queued_tasks:0|g
9-
executor.running_tasks:0|g
96+
executor.running_tasks:0|g
97+
scheduler.scheduler_loop_duration:26.217601|ms
98+
dag_processing.file_path_queue_size:0|g
99+
dag_processing.file_path_queue_update_count:1|c
100+
dag_processing.total_parse_time:0.0029031969997959095|g

‎packages/airflow/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.0.0"
3+
changes:
4+
- description: Make Airflow package GA.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/15287
27
- version: "0.11.0"
38
changes:
49
- description: Improve documentation

‎packages/airflow/data_stream/statsd/_dev/test/system/test-default-config.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
service_notify_signal: SIGHUP
2-
vars:
3-
host: 0.0.0.0
4-
port: 8125
2+
vars: ~
3+
data_stream:
4+
vars:
5+
host: 0.0.0.0
6+
port: 8125

‎packages/airflow/data_stream/statsd/agent/stream/stream.yml.hbs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
metricsets: ["server"]
22
host: {{host}}
33
port: {{port}}
4+
ttl: 60s
45
statsd.mappings:
56
- metric: '<job_name>_start'
67
labels:

‎packages/airflow/data_stream/statsd/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ description: Pipeline for processing airflow data
33
processors:
44
- set:
55
field: ecs.version
6-
value: "8.11.0"
6+
value: "8.17.0"
77
- rename:
88
field: statsd
99
target_field: airflow

‎packages/airflow/data_stream/statsd/manifest.yml‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,24 @@
11
type: metrics
22
title: Airflow metrics
3-
release: experimental
43
streams:
54
- input: statsd/metrics
65
enabled: true
76
title: Airflow metrics
87
description: Collect Airflow metrics
8+
vars:
9+
- name: host
10+
type: text
11+
title: Host
12+
multi: false
13+
required: true
14+
show_user: true
15+
default: localhost
16+
- name: port
17+
type: text
18+
title: Port
19+
multi: false
20+
required: true
21+
show_user: true
22+
default: "8125"
923
elasticsearch:
1024
index_mode: "time_series"

‎packages/airflow/data_stream/statsd/sample_event.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
"type": "metrics"
2525
},
2626
"ecs": {
27-
"version": "8.11.0"
27+
"version": "8.17.0"
2828
},
2929
"elastic_agent": {
3030
"id": "97400795-188c-4140-a1ee-0002078c785d",

‎packages/airflow/docs/README.md‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ Use the Airflow integration to:
1010
- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
1111
- Monitor your workflows' performance and status in real-time.
1212

13-
## Data streams
13+
## Compatibility
14+
15+
The Airflow package is tested with Airflow `2.4.3`. It should work with any `2.*` version.
16+
17+
## What data does this integration collect?
1418

1519
The Airflow integration gathers metric data.
1620

@@ -23,19 +27,13 @@ Data stream:
2327
Note:
2428
- Users can monitor and view metrics within the ingested documents for Airflow in the `metrics-*` index pattern from `Discover`.
2529

26-
## Compatibility
27-
28-
The Airflow module is tested with Airflow `2.4.0`. It should work with versions `2.0.0` and later.
29-
3030
## What do I need to use this integration?
3131

3232
Users require Elasticsearch to store and search user data, and Kibana to visualize and manage it. They can utilize the hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on their own hardware.
3333

34-
To ingest data from Airflow, users must have [StatsD](https://github.com/statsd/statsd) to receive the same.
35-
36-
## Setup
34+
## How do I deploy this integration?
3735

38-
For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.
36+
For step-by-step instructions on how to set up an integration, refer to the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.
3937

4038
## Steps to Setup Airflow
4139

@@ -95,7 +93,7 @@ An example event for `statsd` looks as following:
9593
"type": "metrics"
9694
},
9795
"ecs": {
98-
"version": "8.11.0"
96+
"version": "8.17.0"
9997
},
10098
"elastic_agent": {
10199
"id": "97400795-188c-4140-a1ee-0002078c785d",

0 commit comments

Comments
 (0)