Fix data race caused by sharing sd.Event between observers

[toffaletti]

sd/internal/instance/cache.go causes sd.Event.Instances to be shared between multiple observers:

https://github.com/go-kit/kit/blob/master/sd/internal/instance/cache.go#L71

func (r registry) broadcast(event sd.Event) {	for c := range r {	c <- event	} } 

but observers frequently directly modify the sd.Event.Instances they receive, for example sd/endpoint_cache.go contains this:

https://github.com/go-kit/kit/blob/master/sd/endpoint_cache.go#L54

...	// Happy path.	if event.Err == nil {	c.updateCache(event.Instances) ... func (c *endpointCache) updateCache(instances []string) {	// Deterministic order (for later).	sort.Strings(instances) 

The fix is to deep copy sd.Event every time it is shared. This requirement should probably be called out in the documentation as well if this is the expected behavior.

Test output before the fix:

WARNING: DATA RACE Read at 0x00c000126350 by goroutine 13: sort.(*StringSlice).Less() /usr/local/go/src/sort/sort.go:298 +0x92 sort.doPivot() /usr/local/go/src/sort/sort.go:117 +0x564 sort.quickSort() /usr/local/go/src/sort/sort.go:190 +0xa7 sort.Sort() /usr/local/go/src/sort/sort.go:218 +0x8a github.com/go-kit/kit/sd.(*endpointCache).updateCache() /usr/local/go/src/sort/sort.go:314 +0x88 github.com/go-kit/kit/sd.(*endpointCache).Update() /Users/jason/go/src/github.com/go-kit/kit/sd/endpoint_cache.go:54 +0x3f5 github.com/go-kit/kit/sd.(*DefaultEndpointer).receive() /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:77 +0x132 Previous write at 0x00c000126350 by goroutine 12: sort.(*StringSlice).Swap() /usr/local/go/src/sort/sort.go:299 +0x144 sort.medianOfThree() /usr/local/go/src/sort/sort.go:77 +0x14b sort.doPivot() /usr/local/go/src/sort/sort.go:105 +0x9b sort.quickSort() /usr/local/go/src/sort/sort.go:190 +0xa7 sort.quickSort() /usr/local/go/src/sort/sort.go:197 +0x140 sort.Sort() /usr/local/go/src/sort/sort.go:218 +0x8a github.com/go-kit/kit/sd.(*endpointCache).updateCache() /usr/local/go/src/sort/sort.go:314 +0x88 github.com/go-kit/kit/sd.(*endpointCache).Update() /Users/jason/go/src/github.com/go-kit/kit/sd/endpoint_cache.go:54 +0x3f5 github.com/go-kit/kit/sd.(*DefaultEndpointer).receive() /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:77 +0x132 Goroutine 13 (running) created at: github.com/go-kit/kit/sd.NewEndpointer() /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:38 +0x33c github.com/go-kit/kit/sd/internal/instance.TestDataRace() /Users/jason/go/src/github.com/go-kit/kit/sd/internal/instance/cache_test.go:109 +0x44c testing.tRunner() /usr/local/go/src/testing/testing.go:865 +0x163 Goroutine 12 (running) created at: github.com/go-kit/kit/sd.NewEndpointer() /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:38 +0x33c github.com/go-kit/kit/sd/internal/instance.TestDataRace() /Users/jason/go/src/github.com/go-kit/kit/sd/internal/instance/cache_test.go:108 +0x3fc testing.tRunner() /usr/local/go/src/testing/testing.go:865 +0x163 

[peterbourgon]

Can you rebase this on master?

[toffaletti]

@peterbourgon done

[peterbourgon]

Vielen Dank!