fix: blocking issues of runtime communicator

[pkoutsovasilis]

What does this PR do?

This PR fixes multiple deadlock conditions in the runtime communicator logic and improves overall lifecycle handling. Specifically, it:

• Handles potential deadlock when sending the initial observed message to the runtime if the communicator is destroyed.
• Handles deadlock when the runtime calls CheckinExpected (init expected check-in) and the communicator is already destroyed.
• Handles deadlock when the runtime calls CheckinExpected (init expected check-in) and the server has been disconnected.
• Although highly unlikely due to existing synchronisation primitives, it doesn't block against multiple init checkin messages that arrive after the first that completed the init checkin process.
• Removes a redundant goroutine in the checkin method, which also allows returning accurate gRPC status codes back to the client. (PS: this optimisation can be also applied for the actions handling)
• Adds comprehensive unit tests covering all the above scenarios to prevent regressions.

Why is it important?

These fixes ensure the runtime communicator behaves predictably and safely across lifecycle boundaries like shutdown, reconnection, and concurrent access. Without these changes, users could experience hangs or lost check-in signals in rare but critical failure modes.

You can see that the previous implementation fails to complete these scenarios under test in CI here

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This is a bug-fix for internal lifecycle logic and is not expected to introduce changes in behaviour or configuration requirements for end users.

How to test this PR locally

mage unitTest 

Related issues

• Relates Elastic Agent not serving policy to Endpoint #7944

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

Looks really good. I like all the additional testing. That will hopefully keep this from dead locks in the future as well.

I do have one comment that is inline. See if it make sense.

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: b211ef1

Failed CI Steps

• x86_64:sudo: standalone-upgrade

History

• 💚 Build #23731 succeeded edc74ff
• 💔 Build #23728 failed 7fdbca2

cc @pkoutsovasilis

[blakerouse]

Thanks. With the latest change, this looks good.

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

[mergify]

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

• #8944 [8.17] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.17
• #8945 [8.18] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.18
• #8946 [8.19] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.19
• #8947 [9.0] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 9.0
• #8948 [9.1] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 9.1