Unit testing patterns for concurrent code

Unit testing patterns for concurrent code Dror Helper drorh@oz-code.com | @dhelper | http://blog.drorhelper.com Examples: https://github.com/dhelper/ConcurrentUnitTesting

About.ME • Senior consultant @CodeValue • Developing software (professionally) since 2002 • Mocking code since 2008 • Test Driven Developer • Blogger: http://blog.drorhelper.com

We live in a concurrent world! The free lunch is over! • Multi-core CPUs are the new standard • New(er) language constructs • New(ish) languages

Meanwhile in the unit testing “world” [Test] public void AddTest() { var cut = new Calculator(); var result = cut.Add(2, 3); Assert.AreEqual(5, result); }

The dark art of concurrent code Several actions at the same time Hard to follow code path Non deterministic execution

Good unit tests must be: • Trustworthy – Consistent results – Only fail due to bug or requirement change • Maintainable – Robust – Easy to refactor – Simple to update • Readable

Concurrency test “smells” × Inconsistent results × Untraceable fail × Long running tests × Test freeze

How can we test this method public void Start() { _worker = new Thread(() => { while (_isAlive) { Thread.Sleep(1000); var msg = _messageProvider.GetNextMessage(); //Do stuff LastMessage = msg; } }); _worker.Start(); }

Testing start take #1 [TestMethod] public void ArrivingMessagePublishedTest() { var fakeMessageProvider = A.Fake<IMessageProvider>(); A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!"); var server = new Server(fakeMessageProvider); server.Start(); Thread.Sleep(2000); Assert.AreEqual("Hello!", server.LastMessage); }

Test smell - “Sleep” in test × Time based - fail/pass inconsistently × Test runs for too long × Hard to investigate failures

“In concurrent programming if something can happen, then sooner or later it will, probably at the most inconvenient moment” Paul Butcher – Seven concurrency models in seven weeks

Testing start take #2 [TestMethod] public async Task ArrivingMessagePublishedTest() { var fakeMessageProvider = A.Fake<IMessageProvider>(); A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!"); var server = new Server(fakeMessageProvider); server.Start(); await Task.Delay(2000); Assert.AreEqual("Hello!", server.LastMessage); }

Solution: avoid concurrent code!

Pattern #1 – humble object pattern We extract all the logic from the hard-to-test component into a component that is testable via synchronous tests. http://xunitpatterns.com/Humble%20Object.html Perform Action Code under Test Start Humble object Async Perform action Assert Result Production Code

public void Start() { _worker = new Thread(() => { while (_isAlive) { Thread.Sleep(1000); var msg = _messageProvider.GetNextMessage(); //Do stuff LastMessage = msg; } }); _worker.Start(); }

public void Start() { _worker = new Thread(() => { while (_isAlive) { Thread.Sleep(1000); _messageHandler.HandleNextMessage(); } }); _worker.Start(); }

And finally – the test [TestMethod] public void ArrivingMessagePublishedTest() { var fakeMessageProvider = A.Fake<IMessageProvider>(); A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!"); var messageHandler = new MessageHandler(fakeMessageProvider); messageHandler.HandleNextMessage(); Assert.AreEqual("Hello!", messageHandler.LastMessage); }

Concurrency as part of program flow public class MessageManager { private IMesseageQueue _messeageQueue; public void CreateMessage(string message) { // Here Be Code! _messeageQueue.Enqueue(message); } } public class MessageClient { private IMesseageQueue _messeageQueue; public string LastMessage { get; set; } private void OnMessage(object sender, EventArgs e) { // Here Be Code! LastMessage = e.Message; } }

Test before – Test After Start Code under test Async Logic2 Assert results Logic1 Start Logic1 Fake Assert Results Start Fake Logic2 Assert Results

Testing Flow part 1 [TestMethod] public void AddNewMessageProcessedMessageInQueue() { var messeageQueue = new AsyncMesseageQueue(); var manager = new MessageManager(messeageQueue); manager.CreateNewMessage("a new message"); Assert.AreEqual(1, messeageQueue.Count); }

Testing Flow part 2 [TestMethod] public void QueueRaisedNewMessageEventClientProcessEvent() { var messeageQueue = new AsyncMesseageQueue(); var client = new MessageClient(messeageQueue); client.OnMessage(null, new MessageEventArgs("A new message")); Assert.AreEqual("A new message", client.LastMessage); }

Avoid concurrency Patterns The best possible solution No concurrency == no problems × Do not test some of the code × Not applicable in every scenario

How can we test this class? public class ClassWithTimer { private Timer _timer; public ClassWithTimer(Timer timer) { _timer = timer; _timer.Elapsed += OnTimerElapsed; _timer.Start(); } private void OnTimerElapsed(object sender, ElapsedEventArgs e) { SomethingImportantHappened = true; } public bool SomethingImportantHappened { get; private set; } }

Not a good idea [TestMethod] public void ThisIsABadTest() { var timer = new Timer(1); var cut = new ClassWithTimer(timer); Thread.Sleep(100); Assert.IsTrue(cut.SomethingImportantHappened); }

Set timeout/interval to 1 Also seen with a very small number (or zero) Usually done when need to wait for next tick/timeout × Time based == fragile/inconsistent test × Hard to investigate failures × Usually comes with Thread.Sleep

Fake & Sync Test Code under Test Fake Logic Assert Results

Using Typemock Isolator to fake Timer [TestMethod, Isolated] public void ThisIsAGoodTest() { var fakeTimer = Isolate.Fake.Instance<Timer>(); var cut = new ClassWithTimer(fakeTimer); var fakeEventArgs = Isolate.Fake.Instance<ElapsedEventArgs>(); Isolate.Invoke.Event( () => fakeTimer.Elapsed += null, this, fakeEventArgs); Assert.IsTrue(cut.SomethingImportantHappened); }

Unfortunately not everything can be faked • Mocking tool limitation (example: inheritance based) • Programming language attributes • Special cases (example: MSCorlib) Solution – wrap the unfakeable × Problem – it requires code change

Wrapping Threadpool public class ThreadPoolWrapper { public static void QueueUserWorkItem(WaitCallback callback) { ThreadPool.QueueUserWorkItem(callback); } }

Wrapping Threadpool public class ClassWithWrappedThreadpool { public void RunInThread() { ThreadPool.QueueUserWorkItem(_ => { SomethingImportantHappened = true; }); } public bool SomethingImportantHappened { get; private set; } } ThreadPoolWrapper.QueueUserWorkItem(_ =>

Testing with new ThreadpoolWrapper [TestClass] public class WorkingWithThreadpool { [TestMethod, Isolated] public void UsingWrapperTest() { Isolate.WhenCalled(() => ThreadPoolWrapper.QueueUserWorkItem(null)) .DoInstead(ctx => ((WaitCallback)ctx.Parameters[0]).Invoke(null)); var cut = new ClassWithWrappedThreadpool(); cut.RunInThread(); Assert.IsTrue(cut.SomethingImportantHappened); } }

How can we test that an asynchronous operation never happens?

Test Code Under Test Is Test Async Deterministic Assert Results Run in sync

Another day – another class to test public void Start() { _cancellationTokenSource = new CancellationTokenSource(); Task.Run(() => { var message = _messageBus.GetNextMessage(); if(message == null) return; // Do work if (OnNewMessage != null) { OnNewMessage(this, EventArgs.Empty); } }, _cancellationTokenSource.Token); }

Switching code to “test mode” • Dependency injection • Preprocessor directives • Pass delegate • Other

Run in single thread patterns • Fake & Sync • Async code  sync test

Synchronized run patterns When you have to run a concurrent test in a predictable way

The Signaled pattern Start Run Code under test Signal Fake object Call Wait Assert result

Using the Signaled pattern public void DiffcultCalcAsync(int a, int b) { Task.Run(() => { Result = a + b; _otherClass.DoSomething(Result); }); }

Using the Signaled pattern [TestMethod] public void TestUsingSignal() { var waitHandle = new ManualResetEventSlim(false); var fakeOtherClass = A.Fake<IOtherClass>(); A.CallTo(() => fakeOtherClass.DoSomething(A<int>._)).Invokes(waitHandle.Set); var cut = new ClassWithAsyncOperation(fakeOtherClass); cut.DiffcultCalcAsync(2, 3); var wasCalled = waitHandle.Wait(10000); Assert.IsTrue(wasCalled, "OtherClass.DoSomething was never called"); Assert.AreEqual(5, cut.Result); }

Busy Assert Start Run Code under test Assert or Timeout

Busy assertion [TestMethod] public void DifficultCalculationTest() { var cut = new ClassWithAsyncOperation(); cut.RunAsync(2, 3); AssertHelper.BusyAssert(() => cut.Result == 5, 50, 100, "fail!"); }

Synchronized test patterns × Harder to investigate failures × Cannot test that a call was not made Test runs for too long but only when it fails  Use if other patterns are not applicable

Unit testing for concurrency issues Because concurrency (and async) needs to be tested as well

Test for async Start Method under test Fake object wait Assert results

Test for Deadlock Start Execute Code under test Thread WWaaitit Assert result Call Thread

[TestMethod, Timeout(5000)] public void CheckForDeadlock() { var fakeDependency1 = A.Fake<IDependency>(); var fakeDependency2 = A.Fake<IDependency>(); var waitHandle = new ManualResetEvent(false); var cut = new ClassWithDeadlock(fakeDependency1, fakeDependency2); A.CallTo(() => fakeDependency1.Call()).Invokes(() => waitHandle.WaitOne()); A.CallTo(() => fakeDependency2.Call()).Invokes(() => waitHandle.WaitOne()); var t1 = RunInOtherThread(() => cut.Call1Then2()); var t2 = RunInOtherThread(() => cut.Call2Then1()); waitHandle.Set(); t1.Join(); t2.Join(); } T1 T2 L1 L2

Avoid Concurrency Humble object Test before – test after Run in single thread Fake & Sync Async in production - sync in test Synchronize test The Signaled pattern Busy assertion Concurrency tests Test for Deadlock Test for non blocking Concurrent unit testing patterns

Conclusion It is possible to test concurrent code Avoid concurrency Run in single thread Synchronize test

Dror Helper C: 972.50.7668543 e: drorh@codevalue.net B: blog.drorhelper.com w: www.codevalue.net

Unit testing patterns for concurrent code

In this document