fix(pool): wip, pool reauth should not interfere with handoff #3547

ndyakov · 2025-10-14T18:02:55Z

No description provided.

Copilot

Pull Request Overview

This PR addresses a concurrency issue where pool re-authentication could interfere with connection handoff operations during maintenance notifications. The fix introduces connection state management using Usable and Used flags to coordinate between background operations (re-authentication) and handoff processes.

Key changes:

Added connection-level credentials listener caching to prevent duplicate listeners per connection
Introduced Used flag alongside existing Usable flag to prevent concurrent command execution on connections
Modified re-authentication flow to wait for connections to be available before proceeding

Reviewed Changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
redis.go	Added credentials listener management with caching per connection and modified re-authentication callbacks to accept connection parameter
internal/pool/conn.go	Added `Used` flag and reorganized connection state fields, modified handoff marking logic to acquire connection before state changes
internal/pool/pool.go	Updated connection usage tracking with `Used` flag in pool operations
internal/pool/pool_single.go	Added `Used` flag management for single connection pools
auth/conn_reauth_credentials_listener.go	New file implementing connection-aware credentials listener with timeout-based waiting for connection availability
error.go	Added `ErrConnUnusableTimeout` to bad connection error handling
redis_test.go	Added mutex protection to mock streaming provider
sentinel_test.go	Renamed test function for clarity
internal/pool/pool_test.go	Adjusted test expectations for dial retry behavior
maintnotifications/handoff_worker.go	Added formatting and clarifying comments
internal/pool/buffer_size_test.go	Updated unsafe pointer field layout to match new connection structure

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

redis.go

internal/pool/conn.go

internal/auth/conn_reauth_credentials_listener.go

internal/pool/buffer_size_test.go

Copilot

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

internal/auth/cred_listeners.go

Copilot · 2025-10-15T16:24:23Z

internal/pool/conn.go

+// the delay will be 1, 2, 4... up to 512 microseconds
+// Moving this to the top of the loop to avoid "continue" without delay
+if attempt > 0 && attempt < maxRetries-1 {
+delay := baseDelay * time.Duration(1<<uint(attempt%10)) // Cap exponential growth


The magic number 10 for capping exponential growth is unexplained. Add a named constant like maxExponentShift = 10 with a comment explaining why this value prevents excessive delays while maintaining backoff effectiveness.

Copilot · 2025-10-15T16:24:23Z

internal/auth/conn_reauth_credentials_listener.go

+err = pool.ErrConnUnusableTimeout
+default:
+// small sleep to avoid busy looping
+time.Sleep(100 * time.Microsecond)


The sleep duration 100 * time.Microsecond is duplicated and hard-coded. Extract to a named constant like checkUsablePollInterval = 100 * time.Microsecond for better maintainability and to make the polling strategy more explicit.

Copilot · 2025-10-15T16:24:23Z

internal/auth/conn_reauth_credentials_listener.go

+conn: conn,
+reAuth: reAuth,
+onErr: onErr,
+checkUsableTimeout: 1 * time.Second,


The default timeout of 1 * time.Second is hard-coded. Extract to a named constant like defaultCheckUsableTimeout = 1 * time.Second to make it easier to maintain and document the default behavior.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

internal/pool/buffer_size_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ndyakov · 2025-10-17T15:01:30Z

internal/auth/streaming/pool_hook.go

+r.shouldReAuthLock.Lock()
+delete(r.shouldReAuth, conn.GetID())
+r.shouldReAuthLock.Unlock()
+r.ClearReAuthMark(conn.GetID())


ndyakov · 2025-10-17T15:01:40Z

internal/auth/streaming/pool_hook.go

+r.scheduledLock.RUnlock()
+// has scheduled reauth, reject the connection
+if ok && hasScheduled {
+// simply reject the connection, it will be re-authenticated in OnPut


incorrect comment

Copilot

Pull Request Overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

internal/pool/pool.go

internal/auth/streaming/pool_hook.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

internal/pool/pool.go

internal/pool/buffer_size_test.go

Copilot · 2025-10-17T15:05:56Z

internal/auth/streaming/pool_hook.go

+} else {
+// Release Usable and retry
+conn.Usable.Store(true)
+time.Sleep(time.Millisecond)


Fixed 1ms sleep in tight retry loop can cause unnecessary delays and poor performance. Consider using exponential backoff or a shorter initial delay (e.g., 10-100 microseconds) with gradual increase, similar to the pattern used in MarkQueuedForHandoff at line 490.

Copilot · 2025-10-17T15:05:57Z

internal/auth/streaming/pool_hook.go

+time.Sleep(time.Millisecond)
+}
+} else {
+time.Sleep(time.Millisecond)


Fixed 1ms sleep in tight retry loop can cause unnecessary delays and poor performance. This should use exponential backoff similar to the retry pattern in MarkQueuedForHandoff to reduce contention and improve responsiveness.

Copilot

Pull Request Overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-17T17:02:38Z

internal/pool/pool_single.go

 if p.stickyErr != nil {
 return nil, p.stickyErr
 }
+if p.cn == nil {
+return nil, ErrClosed
+}


The nil check for p.cn is performed after checking p.stickyErr, which means if both p.cn is nil and p.stickyErr is set, the function will return the sticky error instead of ErrClosed. This could mask the fact that the connection is nil. Consider checking p.cn for nil before checking p.stickyErr.

Suggested change

if p.stickyErr != nil {

return nil, p.stickyErr

}

if p.cn == nil {

return nil, ErrClosed

}

if p.cn == nil {

return nil, ErrClosed

}

if p.stickyErr != nil {

return nil, p.stickyErr

}

Copilot · 2025-10-17T17:02:39Z

internal/pool/buffer_size_test.go

@@ -1,9 +1,8 @@
 package pool_test

 import (


The import of net package was removed, but net.Conn is still referenced in the comment at line 142 as 'net.Conn'. While this is just a comment, it could cause confusion. Consider either keeping the import or updating the comment to remove the package reference.

Copilot · 2025-10-17T17:02:39Z

internal/auth/streaming/pool_hook.go

+} else {
+// Release Usable and retry
+conn.Usable.Store(true)
+time.Sleep(time.Millisecond)


The busy-wait loop with a fixed 1ms sleep could cause unnecessary CPU usage and delays. Consider using exponential backoff similar to the pattern used in MarkQueuedForHandoff() in conn.go, starting with shorter intervals and increasing gradually.

ndyakov force-pushed the ndyakov/pool-reauth branch from ef20907 to a65a724 Compare October 14, 2025 18:04

fix(pool): wip, pool reauth should not interfere with handoff

5fe0bfa

ndyakov force-pushed the ndyakov/pool-reauth branch from a65a724 to 5fe0bfa Compare October 14, 2025 18:06

ndyakov added 2 commits October 14, 2025 21:18

fix credListeners map

d39da69

fix race in tests

8a629fb

ndyakov force-pushed the ndyakov/pool-reauth branch from c9e9e6c to 8a629fb Compare October 14, 2025 19:45

ndyakov and others added 8 commits October 14, 2025 22:57

Merge branch 'master' into ndyakov/pool-reauth

6c54ab5

better conn usable timeout

90bfdb3

add design decision comment

07283ec

few small improvements

1bbf2e6

update marked as queued

1428068

add Used to clarify the state of the conn

e7dc339

rename test

77c0c73

fix(test): fix flaky test

011ef96

ndyakov requested review from Copilot and htemelski-redis October 15, 2025 15:57

Copilot AI reviewed Oct 15, 2025

View reviewed changes

ndyakov added the bug label Oct 15, 2025

ndyakov added 2 commits October 15, 2025 19:14

lock inside the listeners collection

e03396e

address pr comments

391b6c5

ndyakov requested a review from Copilot October 15, 2025 16:23

Copilot AI reviewed Oct 15, 2025

View reviewed changes

Update internal/auth/cred_listeners.go

6ad9a67

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ndyakov requested a review from Copilot October 15, 2025 16:28

Copilot AI reviewed Oct 15, 2025

View reviewed changes

internal/pool/buffer_size_test.go Outdated Show resolved Hide resolved

Update internal/pool/buffer_size_test.go

0c4f8fb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ndyakov force-pushed the ndyakov/pool-reauth branch from 5f8a0b9 to 0b2baa1 Compare October 17, 2025 09:39

wip refactor entraid

acb55d8

ndyakov force-pushed the ndyakov/pool-reauth branch from 0b2baa1 to acb55d8 Compare October 17, 2025 09:43

fix maintnotif pool hook

d74671b

ndyakov force-pushed the ndyakov/pool-reauth branch from 6cbbdd5 to d74671b Compare October 17, 2025 10:47

fix mocks

0e10cd7

ndyakov force-pushed the ndyakov/pool-reauth branch from be6c8b6 to 4eac8f8 Compare October 17, 2025 11:30

fix nil listener

afba8c2

ndyakov force-pushed the ndyakov/pool-reauth branch from 4eac8f8 to afba8c2 Compare October 17, 2025 12:17

ndyakov added 6 commits October 17, 2025 15:53

sync and async reauth based on conn lifecycle

4bc6d33

be able to reject connection OnGet

3020e3a

pass hooks so the tests can observe reauth

f886775

give some time for the background to execute commands

72cf74a

fix tests

c715185

only async reauth

f14095b

ndyakov force-pushed the ndyakov/pool-reauth branch from 11afe91 to f14095b Compare October 17, 2025 14:39

Merge branch 'master' into ndyakov/pool-reauth

e94cc9f

ndyakov commented Oct 17, 2025

View reviewed changes

ndyakov requested a review from Copilot October 17, 2025 15:02

Copilot AI reviewed Oct 17, 2025

View reviewed changes

internal/pool/pool.go Outdated Show resolved Hide resolved

internal/auth/streaming/pool_hook.go Outdated Show resolved Hide resolved

ndyakov and others added 2 commits October 17, 2025 18:03

Update internal/pool/pool.go

19f4080

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/auth/streaming/pool_hook.go

4049d5e

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ndyakov requested a review from Copilot October 17, 2025 15:04

Copilot AI reviewed Oct 17, 2025

View reviewed changes

ndyakov requested a review from Copilot October 17, 2025 17:01

Copilot AI reviewed Oct 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(pool): wip, pool reauth should not interfere with handoff #3547

fix(pool): wip, pool reauth should not interfere with handoff #3547

Uh oh!

ndyakov commented Oct 14, 2025

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 15, 2025

Copilot AI Oct 15, 2025

Copilot AI Oct 15, 2025

Copilot AI left a comment

Uh oh!

ndyakov Oct 17, 2025

ndyakov Oct 17, 2025

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Copilot AI Oct 17, 2025

Copilot AI left a comment

Copilot AI Oct 17, 2025

Copilot AI Oct 17, 2025

Copilot AI Oct 17, 2025

Labels

1 participant

fix(pool): wip, pool reauth should not interfere with handoff #3547

Are you sure you want to change the base?

fix(pool): wip, pool reauth should not interfere with handoff #3547

Uh oh!

Conversation

ndyakov commented Oct 14, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

ndyakov Oct 17, 2025

Choose a reason for hiding this comment

ndyakov Oct 17, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Labels

1 participant