Retry saving Agent information file (`fleet.enc`) on Windows #9224

ycombinator · 2025-08-01T13:59:16Z

What does this PR do?

This PR retries saving the Agent information file, fleet.enc, on Windows. The retries are done every 50ms for up to 2 seconds.

Why is it important?

Saving the Agent information file involves renaming/moving a file to its final destination. However, on Windows, it is sometimes not possible to rename/move a file to its destination file because the destination file is locked by another process (e.g. antivirus software). For such situations, we now retry the save operation on Windows.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
~~I have added an integration test or an E2E test~~

Disruptive User Impact

None.

Related issues

Closes [Windows] Access is denied error when rotating fleet.enc file #5862

mergify · 2025-08-01T14:00:02Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

ycombinator · 2025-08-01T14:03:15Z

@pchila @swiatekm Following the feedback on elastic/elastic-agent-libs#333, I'm replacing that PR with this one. At the moment, I've sketched out two alternate implementations in this PR — one that specifically retries saving the fleet.enc file when the policy is being handled from Fleet and the other that's a bit more generic / at a lower-level where it retries saving any file using the storage package. In both implementations, the retries are only done if the OS is Windows and error from the save/rotate operation is ACCESS_DENIED.

Let me know which of these approaches you prefer and I'll get rid of the other one + add tests. Or if you prefer a totally different approach to solving this bug, that's fine too — just let me know. Thanks!

pchila

I would start by touching only the handler_action_policy_change with a retry and see how that goes (ideally we would need a test that reproduces the issue but as I understand we don't have one yet)

Please have a look at my comment below

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go

elasticmachine · 2025-08-04T22:41:20Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pchila

I think we can start with retrying only at the handler_policy_change level (without the additional retry loop at EncryptedStore.
There are also a couple of comments about the usage of the backoff library

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go

ycombinator · 2025-08-05T16:48:14Z

I think we can start with retrying only at the handler_policy_change level (without the additional retry loop at EncryptedStore

Can you clarify what you mean by this? In other words, what would you like to see done differently than the current implementation in this PR?

pchila · 2025-08-06T07:32:20Z

I think we can start with retrying only at the handler_policy_change level (without the additional retry loop at EncryptedStore

Can you clarify what you mean by this? In other words, what would you like to see done differently than the current implementation in this PR?

I would do without the encrypted store retry (I am referring to this part that duplicate the original implementation of the handler retry).

The bug we are trying to fix is relative to not being able to write a new fleet.enc on windows because of Access is denied error so I would keep the fix focused on that operation (unless there is concrete evidence that the bug originates in the *diskstore implementation).

ycombinator · 2025-08-07T20:33:29Z

I would do without the encrypted store retry (I am referring to this part that duplicate the original implementation of the handler retry).

The bug we are trying to fix is relative to not being able to write a new fleet.enc on windows because of Access is denied error so I would keep the fix focused on that operation (unless there is concrete evidence that the bug originates in the *diskstore implementation).

@pchila My bad, I just forgot to remove the previous implementation (the one you linked to) which is why I was unsure as to what I was missing 🤦. Thanks for the clarification. Removed the previous implementation in 866d457.

ycombinator · 2025-08-08T02:50:00Z

The TestSaveConfigToStore unit test introduced in this PR is failing.

PS C:\Users\Shaunak Kashyap\development\github\elastic-agent> go test -v ./internal/pkg/agent/application/actions/handlers/... -run TestSaveConfigToStore -count 10 === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.55s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.55s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.51s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.51s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.51s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.55s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.56s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.52s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.55s) === RUN TestSaveConfigToStore --- PASS: TestSaveConfigToStore (1.51s) PASS ok github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers 15.461s PS C:\Users\Shaunak Kashyap\development\github\elastic-agent>

Moving PR to draft while I investigate this issue.

pchila

Just a couple of nitpicks, nothing blocking. Please have a look.

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go

internal/pkg/agent/application/actions/handlers/handler_helpers_other.go

pchila

A small change in the debug logs, not blocking

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go

elastic-sonarqube · 2025-08-20T23:37:02Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
71.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

github-actions · 2025-08-21T00:02:39Z

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

elasticmachine · 2025-08-21T00:02:48Z

💚 Build Succeeded

Buildkite Build
Commit: e20c50b

History

💔 Build #25581 failed 55c1777
💔 Build #25569 failed 5b96517
💛 Build #25109 was flaky ccade71
💛 Build #25103 was flaky 6d70a49
💛 Build #25089 was flaky eb05684

cc @ycombinator

mergify · 2025-08-21T00:02:51Z

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

#9507 [8.17] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows has been created for branch 8.17
#9508 [8.18] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows has been created for branch 8.18
#9509 [8.19] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows has been created for branch 8.19
#9510 [9.0] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows has been created for branch 9.0
#9511 [9.1] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows has been created for branch 9.1

* Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249)

…9507) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249) Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>

…9510) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249) Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>

…9508) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249) Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>

…9511) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249) Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>

…9509) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes (cherry picked from commit ead9249) Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>

…#9224) * Retry saving Agent information file on Windows * Implement retries on saving fleet.enc file * Remove previous implementation * Adding CHANGELOG file * Running mage fmt * Remove unnecessary changes

mergify bot assigned ycombinator Aug 1, 2025

pchila reviewed Aug 4, 2025

View reviewed changes

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go Outdated Show resolved Hide resolved

ycombinator requested a review from pchila August 4, 2025 22:40

ycombinator added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-active-all Automated backport with mergify to all the active branches labels Aug 4, 2025

ycombinator changed the title ~~Retry fleet.enc file rotation on Windows~~ Retry saving Agent information file (fleet.enc) on Windows Aug 4, 2025

ycombinator marked this pull request as ready for review August 4, 2025 22:41

ycombinator requested a review from a team as a code owner August 4, 2025 22:41

ycombinator requested a review from blakerouse August 4, 2025 22:41

ycombinator force-pushed the fleet-enc-save-retries branch from d835648 to fd1c7dc Compare August 4, 2025 22:41

pchila reviewed Aug 5, 2025

View reviewed changes

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go Outdated Show resolved Hide resolved

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go Outdated Show resolved Hide resolved

ycombinator requested a review from pchila August 5, 2025 17:11

ycombinator force-pushed the fleet-enc-save-retries branch from 0d1c73f to 866d457 Compare August 7, 2025 20:32

ycombinator marked this pull request as draft August 8, 2025 02:50

ycombinator force-pushed the fleet-enc-save-retries branch from e52e8c3 to 7d79815 Compare August 11, 2025 21:35

ycombinator marked this pull request as ready for review August 11, 2025 21:37

pchila previously approved these changes Aug 12, 2025

View reviewed changes

internal/pkg/agent/application/actions/handlers/handler_helpers_windows.go Outdated Show resolved Hide resolved

internal/pkg/agent/application/actions/handlers/handler_helpers_other.go Outdated Show resolved Hide resolved

ycombinator dismissed pchila’s stale review via e730242 August 12, 2025 10:23

ycombinator requested a review from pchila August 12, 2025 10:38

pchila previously approved these changes Aug 12, 2025

View reviewed changes

ycombinator requested a review from pchila August 12, 2025 12:25

ycombinator enabled auto-merge (squash) August 12, 2025 12:26

ycombinator dismissed pchila’s stale review via ccade71 August 12, 2025 13:04

ycombinator added 4 commits August 20, 2025 14:57

Implement retries on saving fleet.enc file

ec0029a

Remove previous implementation

395a9de

Adding CHANGELOG file

fe62acd

Running mage fmt

553678b

ycombinator force-pushed the fleet-enc-save-retries branch from c2e79a5 to 553678b Compare August 20, 2025 21:57

Remove unnecessary changes

e20c50b

cmacknz approved these changes Aug 20, 2025

View reviewed changes

ycombinator enabled auto-merge (squash) August 20, 2025 22:53

ycombinator merged commit ead9249 into elastic:main Aug 21, 2025
19 checks passed

This was referenced Aug 21, 2025

[8.17] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows #9507

Merged

[8.18] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows #9508

Merged

mergify bot mentioned this pull request Aug 21, 2025

[8.19] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows #9509

Merged

8 tasks

This was referenced Aug 21, 2025

[9.0] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows #9510

Merged

[9.1] (backport #9224) Retry saving Agent information file (fleet.enc) on Windows #9511

Merged

ycombinator deleted the fleet-enc-save-retries branch August 21, 2025 00:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Retry saving Agent information file (`fleet.enc`) on Windows #9224

Retry saving Agent information file (`fleet.enc`) on Windows #9224

Uh oh!

ycombinator commented Aug 1, 2025 •

edited

Loading

mergify bot commented Aug 1, 2025

ycombinator commented Aug 1, 2025 •

edited

Loading

pchila left a comment

Uh oh!

elasticmachine commented Aug 4, 2025

pchila left a comment

Uh oh!

Uh oh!

ycombinator commented Aug 5, 2025 •

edited

Loading

pchila commented Aug 6, 2025

ycombinator commented Aug 7, 2025

ycombinator commented Aug 8, 2025 •

edited

Loading

pchila left a comment

Uh oh!

Uh oh!

pchila left a comment

Uh oh!

Uh oh!

Uh oh!

elastic-sonarqube bot commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

elasticmachine commented Aug 21, 2025

mergify bot commented Aug 21, 2025 •

edited

Loading

Labels

5 participants

Retry saving Agent information file (fleet.enc) on Windows #9224

Retry saving Agent information file (fleet.enc) on Windows #9224

Uh oh!

Conversation

ycombinator commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

Related issues

mergify bot commented Aug 1, 2025

ycombinator commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 4, 2025

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ycombinator commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pchila commented Aug 6, 2025

ycombinator commented Aug 7, 2025

ycombinator commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elastic-sonarqube bot commented Aug 20, 2025

Quality Gate passed

Uh oh!

github-actions bot commented Aug 21, 2025

elasticmachine commented Aug 21, 2025

💚 Build Succeeded

History

mergify bot commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Labels

5 participants

Retry saving Agent information file (`fleet.enc`) on Windows #9224

Retry saving Agent information file (`fleet.enc`) on Windows #9224

ycombinator commented Aug 1, 2025 •

edited

Loading

ycombinator commented Aug 1, 2025 •

edited

Loading

ycombinator commented Aug 5, 2025 •

edited

Loading

ycombinator commented Aug 8, 2025 •

edited

Loading

mergify bot commented Aug 21, 2025 •

edited

Loading