Skip to content

Commit 031d5ec

Browse files
milan-elasticmilan-elastic
authored
[O11y][Ceph] Resolve the conflicts in host.ip field (#7491)
* resolve host.ip field conflict * update pr link in changelog.yml * resolving review comments
1 parent 939c278 commit 031d5ec

File tree

11 files changed

+215
-1
lines changed

11 files changed

+215
-1
lines changed

‎packages/ceph/_dev/build/docs/README.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,100 @@ In the above JSON, please consider `api` as API User and value of `52dffd92-a103
8787

8888
After the integration is successfully configured, clicking on the Assets tab of the Ceph Integration should display a list of available dashboards. Click on the dashboard available for your configured data stream. It should be populated with the required data.
8989

90+
### Troubleshooting
91+
92+
If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Cluster Disk``, ``Cluster Health``, ``Cluster Status``, ``OSD Performance``, ``OSD Pool Stats``, ``OSD Tree`` and ``Pool Disk`` data stream's indices.
93+
To reindex the data, the following steps must be performed.
94+
95+
1. Stop the data stream by going to `Integrations -> Ceph -> Integration policies` open the configuration of Ceph and disable the `Collect Ceph metrics` toggle to reindex logs data streams and save the integration.
96+
97+
2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.
98+
99+
```
100+
POST _reindex
101+
{
102+
"source": {
103+
"index": "<index_name>"
104+
},
105+
"dest": {
106+
"index": "temp_index"
107+
}
108+
}
109+
```
110+
Example:
111+
```
112+
POST _reindex
113+
{
114+
"source": {
115+
"index": "logs-ceph.cluster_disk-default"
116+
},
117+
"dest": {
118+
"index": "temp_index"
119+
}
120+
}
121+
```
122+
123+
```
124+
DELETE /_data_stream/<data_stream>
125+
```
126+
Example:
127+
```
128+
DELETE /_data_stream/logs-ceph.cluster_disk-default
129+
```
130+
131+
```
132+
DELETE _index_template/<index_template>
133+
```
134+
Example:
135+
```
136+
DELETE _index_template/logs-ceph.cluster_disk
137+
```
138+
3. Go to `Integrations -> Ceph -> Settings` and click on `Reinstall Ceph`.
139+
140+
4. Copy data from temporary index to new index by performing the following steps in the Dev tools.
141+
142+
```
143+
POST _reindex
144+
{
145+
"conflicts": "proceed",
146+
"source": {
147+
"index": "temp_index"
148+
},
149+
"dest": {
150+
"index": "<index_name>",
151+
"op_type": "create"
152+
153+
}
154+
}
155+
```
156+
Example:
157+
```
158+
POST _reindex
159+
{
160+
"conflicts": "proceed",
161+
"source": {
162+
"index": "temp_index"
163+
},
164+
"dest": {
165+
"index": "logs-ceph.cluster_disk-default",
166+
"op_type": "create"
167+
168+
}
169+
}
170+
```
171+
172+
5. Verify data is reindexed completely.
173+
174+
6. Start the data stream by going to the `Integrations -> Ceph -> Integration policies` and open configuration of integration and enable the `Collect Ceph metrics` toggle and save the integration.
175+
176+
7. Delete temporary index by performing the following step in the Dev tools.
177+
178+
```
179+
DELETE temp_index
180+
```
181+
182+
More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).
183+
90184
## Metrics reference
91185

92186
### Cluster Disk

‎packages/ceph/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.10.1"
3+
changes:
4+
- description: Resolve host.ip field conflict.
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/7491
27
- version: "0.10.0"
38
changes:
49
- description: Rename ownership from obs-service-integrations to obs-infraobs-integrations

‎packages/ceph/data_stream/cluster_disk/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/cluster_health/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/cluster_status/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/osd_performance/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/osd_pool_stats/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/osd_tree/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/data_stream/pool_disk/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,7 @@
1818
name: event.original
1919
- external: ecs
2020
name: event.type
21+
- external: ecs
22+
name: host.ip
2123
- external: ecs
2224
name: service.address

‎packages/ceph/docs/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,100 @@ In the above JSON, please consider `api` as API User and value of `52dffd92-a103
8787

8888
After the integration is successfully configured, clicking on the Assets tab of the Ceph Integration should display a list of available dashboards. Click on the dashboard available for your configured data stream. It should be populated with the required data.
8989

90+
### Troubleshooting
91+
92+
If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Cluster Disk``, ``Cluster Health``, ``Cluster Status``, ``OSD Performance``, ``OSD Pool Stats``, ``OSD Tree`` and ``Pool Disk`` data stream's indices.
93+
To reindex the data, the following steps must be performed.
94+
95+
1. Stop the data stream by going to `Integrations -> Ceph -> Integration policies` open the configuration of Ceph and disable the `Collect Ceph metrics` toggle to reindex logs data streams and save the integration.
96+
97+
2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.
98+
99+
```
100+
POST _reindex
101+
{
102+
"source": {
103+
"index": "<index_name>"
104+
},
105+
"dest": {
106+
"index": "temp_index"
107+
}
108+
}
109+
```
110+
Example:
111+
```
112+
POST _reindex
113+
{
114+
"source": {
115+
"index": "logs-ceph.cluster_disk-default"
116+
},
117+
"dest": {
118+
"index": "temp_index"
119+
}
120+
}
121+
```
122+
123+
```
124+
DELETE /_data_stream/<data_stream>
125+
```
126+
Example:
127+
```
128+
DELETE /_data_stream/logs-ceph.cluster_disk-default
129+
```
130+
131+
```
132+
DELETE _index_template/<index_template>
133+
```
134+
Example:
135+
```
136+
DELETE _index_template/logs-ceph.cluster_disk
137+
```
138+
3. Go to `Integrations -> Ceph -> Settings` and click on `Reinstall Ceph`.
139+
140+
4. Copy data from temporary index to new index by performing the following steps in the Dev tools.
141+
142+
```
143+
POST _reindex
144+
{
145+
"conflicts": "proceed",
146+
"source": {
147+
"index": "temp_index"
148+
},
149+
"dest": {
150+
"index": "<index_name>",
151+
"op_type": "create"
152+
153+
}
154+
}
155+
```
156+
Example:
157+
```
158+
POST _reindex
159+
{
160+
"conflicts": "proceed",
161+
"source": {
162+
"index": "temp_index"
163+
},
164+
"dest": {
165+
"index": "logs-ceph.cluster_disk-default",
166+
"op_type": "create"
167+
168+
}
169+
}
170+
```
171+
172+
5. Verify data is reindexed completely.
173+
174+
6. Start the data stream by going to the `Integrations -> Ceph -> Integration policies` and open configuration of integration and enable the `Collect Ceph metrics` toggle and save the integration.
175+
176+
7. Delete temporary index by performing the following step in the Dev tools.
177+
178+
```
179+
DELETE temp_index
180+
```
181+
182+
More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).
183+
90184
## Metrics reference
91185

92186
### Cluster Disk
@@ -182,6 +276,7 @@ An example event for `cluster_disk` looks as following:
182276
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | |
183277
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | | |
184278
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | |
279+
| host.ip | Host ip addresses. | ip | | |
185280
| input.type | Type of Filebeat input. | keyword | | |
186281
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
187282
| tags | List of keywords used to tag each event. | keyword | | |
@@ -272,6 +367,7 @@ An example event for `cluster_health` looks as following:
272367
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | |
273368
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | |
274369
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | |
370+
| host.ip | Host ip addresses. | ip | |
275371
| input.type | Type of Filebeat input. | keyword | |
276372
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
277373
| tags | List of keywords used to tag each event. | keyword | |
@@ -453,6 +549,7 @@ An example event for `cluster_status` looks as following:
453549
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | |
454550
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | | |
455551
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | |
552+
| host.ip | Host ip addresses. | ip | | |
456553
| input.type | Type of Filebeat input. | keyword | | |
457554
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
458555
| tags | List of keywords used to tag each event. | keyword | | |
@@ -547,6 +644,7 @@ An example event for `osd_performance` looks as following:
547644
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | |
548645
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | | |
549646
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | |
647+
| host.ip | Host ip addresses. | ip | | |
550648
| input.type | Type of Filebeat input. | keyword | | |
551649
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
552650
| tags | List of keywords used to tag each event. | keyword | | |
@@ -649,6 +747,7 @@ An example event for `osd_pool_stats` looks as following:
649747
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | |
650748
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | | |
651749
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | |
750+
| host.ip | Host ip addresses. | ip | | |
652751
| input.type | Type of Filebeat input. | keyword | | |
653752
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
654753
| tags | List of keywords used to tag each event. | keyword | | |
@@ -758,6 +857,7 @@ An example event for `osd_tree` looks as following:
758857
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | |
759858
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | |
760859
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | |
860+
| host.ip | Host ip addresses. | ip | |
761861
| input.type | Type of Filebeat input. | keyword | |
762862
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
763863
| tags | List of keywords used to tag each event. | keyword | |
@@ -862,6 +962,7 @@ An example event for `pool_disk` looks as following:
862962
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword | | |
863963
| event.original | Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. If users wish to override this and index this field, please see `Field data types` in the `Elasticsearch Reference`. | keyword | | |
864964
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword | | |
965+
| host.ip | Host ip addresses. | ip | | |
865966
| input.type | Type of Filebeat input. | keyword | | |
866967
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
867968
| tags | List of keywords used to tag each event. | keyword | | |

0 commit comments

Comments
 (0)