[New Integration] ProxySG

[mjwolf]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes [New Integration] Blue Coat ProxySG #9511

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[mjwolf]

This PR (currently) only supports the default main access log format from proxysg. Support for other default logs formats will be added in the future, in other follow-up PRs. The full list of default access log formats is here

[taylor-swanson]

Seems like there are more ECS mappings we can add, particularly with client_to_server

[mjwolf]

I've more ECS mappings, specifically some more for client_to_server fields.

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[taylor-swanson]

I just realized that there's a screenshot of the dashboard, but there's no dashboard included in this PR. Are you planning on adding one in this PR or a follow-up PR?

[mjwolf]

I just realized that there's a screenshot of the dashboard, but there's no dashboard included in this PR. Are you planning on adding one in this PR or a follow-up PR?

I've added a dashboard now

[taylor-swanson]

I'm stilling seeing a lot of missed fields for ECS (working on compiling a list).

A bigger issue is that it appears the dashboard isn't working. I loaded the dashboard on my local stack and saw these errors. Dashboards should be standalone, any and all resources must be included with the integration. Likewise, all visualizations should be built using Lens, which as far as I know should be entirely self-contained.

[taylor-swanson]

Remaining tasks/questions/comments:

• proxysg.time_taken -> event.duration (ensure value is converted to nanoseconds)
• proxysg.server_to_client.status -> http.response.status_code (if this is actually a proxy status, then we probably don't want to copy it to the ECS field. That's what I did for Squid)
• proxysg.client_to_server.method -> http.request.method
• proxysg.client_to_server.host -> url.domain
• client.user.id -> client.user.name (since the value is a short name, .name is a better fit)
• Copy IPs to related.ip (i.e., source.ip, destination.ip, IPs in vendor fields, etc)
• Copy hosts to related.hosts (i.e.,url.domain, ideally the referrer host as well, but it's contained within a url)
• Copy user name to related.user
• Run registered_domain processor against url.domain
• Run user_agent processor against user_agent.original
• Run geoip against source.ip and destination.ip

Regarding the timestamp, I don't see any information on time zone. What is the time supposed to represent here? Is it the local time of the proxy, is it UTC, etc? If it's the local time, then we need to be able to provide a time zone configuration for the user. If we need the local time zone, we can add the add_locale processor to the agent configuration, but note that this the local time zone of the agent, not necessarily the time zone of the proxy itself (this is why we provide the time zone configuration option as well).

Additionally, I usually remove entries that are -, since these indicate empty values. This isn't critical and doesn't need to be done as part of this change. It's a bit more difficult to do as the values extracted from the message are put into a tree structure instead of a flat structure. For squid, I had extracted all values to a flat structure and then used a small script to remove any values that were -.

[mjwolf]

@taylor-swanson thanks, I've made all the suggestions now, except for removing '-'. I agree it's better to remove those though, I'll try to remove them later.

For the timezone, the time from proxysg is defined to always be GMT in the source logs, so I don't think it needs any other conversion.

I don't know what was wrong with the dashboard, but I completely rebuilt it, and verified that it works in a separate cluster, so I think it should be ok now.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b4a244f

History

• 💚 Build #15856 succeeded 4b3d6b2
• 💔 Build #15855 failed a9280d2
• 💚 Build #15849 succeeded 1aa71c5
• 💚 Build #15543 succeeded f8731df
• 💔 Build #14807 failed 51ba330
• 💔 Build #14710 failed 7894655

[taylor-swanson]

@taylor-swanson thanks, I've made all the suggestions now, except for removing '-'. I agree it's better to remove those though, I'll try to remove them later.

No worries, I have no issues with following up with that later.

For the timezone, the time from proxysg is defined to always be GMT in the source logs, so I don't think it needs any other conversion.

Sounds good. Unix timestamps also follow a similar pattern and don't need extra processing.

[taylor-swanson]

LGTM!

[elasticmachine]

Package proxysg - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=proxysg