Sunday, June 28, 2015

.NET Asynchronous Batch Processor

The .NET Framework offers a series of Thread-Safe Collections that allows you to consume collections across threads. Processing the contents of these collections still requires a thread, and while there is a BlockingCollection there is unfortunately no such class to support this in an asynchronous fashion. (Please note that the always awesome Stephen Cleary did actually implement an AsyncCollection.)

What if you want to handle dynamically sized batches of data in an asynchronous manner?

You could use a series of Dataflow blocks, or if you are looking for a simple solution you can write a small class that uses an async loop to process a ConcurrentQueue. Below is an abstract base class that can help you implement this:

Base Class

public abstract class BatchProcessorBase<T> : IDisposable
    protected readonly int MaxBatchSize;
    private readonly ConcurrentQueue<T> _queue;
    private readonly CancellationTokenSource _cancelSource;
    private readonly object _queueTaskLock;
    private Task _queueTask;
    private bool _isDiposed;
    protected BatchProcessorBase(int maxBatchSize)
        MaxBatchSize = maxBatchSize;
        _queue = new ConcurrentQueue<T>();
        _cancelSource = new CancellationTokenSource();
        _queueTaskLock = new object();
        _queueTask = Task.FromResult(true);
    public void Enqueue(T item)
    public void Dispose()
        if (_isDiposed)
        _isDiposed = true;
    protected abstract Task ProcessBatchAsync(
        IList<T> list, 
        CancellationToken cancelToken);
    private void TryStartProcessLoop()
        // Lock so only one thread can manipulate the queue task.
        lock (_queueTaskLock)
            // If cancellationhas been requested, do not start.
            if (_cancelSource.IsCancellationRequested)
            // If the loop is still active, do not start.
            if (!_queueTask.IsCompleted)
            // If the queue is empty, do not start.
            if (_queue.Count == 0)
            // Start a new task to process the queue.
            _queueTask = Task.Run(() => ProcessQueue(), _cancelSource.Token);
            // When the process queue task completes, check to see if
            // the queue has been populated again and needs to restart.
            _queueTask.ContinueWith(t => TryStartProcessLoop());
    private async Task ProcessQueue()
        // Stay alive until the queue is empty or cancellation is requested.
        while (!_cancelSource.IsCancellationRequested && _queue.Count > 0)
            var list = new List<T>();
            T item;
            // Dequeue up to a full batch from the queue.
            while (list.Count < MaxBatchSize && _queue.TryDequeue(out item))
            // Process the dequeued items.
            await ProcessBatchAsync(list, _cancelSource.Token);

Wednesday, June 24, 2015

Capture xUnit Test Output with NLog and Common Logging

I recently blogged about How To Capture Test Output in xUnit 2.0. This is great, but how can we pass the ITestOutputHelper into our code to capture log output?

You could just wrap the xUnit helper in an ILog or ILogger, but we can also take it a step further and get all of the NLog features too! By creating an NLog target that wraps the ITestOutputHelper we can enable ourselves to use multiple targets, layouts, variables, verbosity levels, and more.

Sample Unit Test

public class NLogTests : IDisposable
    private readonly ILogger _logger;
    public NLogTests(ITestOutputHelper outputHelper)
        _logger = outputHelper.GetNLogLogger();
    public void Dispose()
    public void Hello()
        _logger.Trace("World Trace");
        _logger.Debug("World Debug");
        _logger.Warn("World Warn");
        _logger.Error("World Error");

Monday, June 22, 2015

How To: Kill child process when parent process is killed

Killing child all child process spawned by a parent process is an extremely useful trick that is not directly supported by the .NET framework. Fortunately the windows operating system, more specifically Kernel32.dll, does support the ability to link one process to another on shutdown. A huge thanks to Matt Howell for sharing this solution on Stack Overflow!

I took the liberty of cleaning up a few small things in the code and creating a demo:

Saturday, May 30, 2015

.NET Generic Overloads: How to support T and IList

What happens when you want to overload a generic method where the first method accepts a single object and the other accepts an IList of that type?

It will only work when specifically try to pass in an IList. If you try to pass in a List it will fail because the compiler will identify the generic parameter overload and fail before trying to use the implicit cast to an IList.

This is because a C# compiler tries to identify overloaded methods it checks for matching parameters in the following order:

  1. Explicit type matches.
  2. Generic parameters.
  3. Implicit type matches.

Let's look at an example!

Example of what DOES NOT work.

public class OverloadSample1
    public void Test()
        // Matches Method A - Good
        ISample sample = new Sample();
        // Matches Method B - Good
        IList<ISample> samples1 = new List<ISample>();
        // Matches Method A - BAD!
        List<ISample> samples2 = new List<ISample>();
    // A
    public void Method<T>(T sample)
        where T : ISample
    // B
    public void Method<T>(IList<T> sample)
        where T : ISample
}, how do we solve this problem?

Saturday, May 23, 2015

How To Capture Test Output in xUnit 2.0

As of xUnit 2.0 the test framework no longer captures any native outputs, this is by design. Now the question is: where do you write your test output?

You now have to write all test output to an interface, ITestOutputHelper, that is injected into your test's constructor. This design seems to have polarized developers a bit, as not everyone enjoys being require to add a constructor to your test classes. Personally, as a huge fan of dependency injection, I really like this solution.

Spoilers: My next blog post will be about how to combine this with Common.Logging and NLog.

Sample Test

using System.Threading.Tasks;
using Xunit;
using Xunit.Abstractions;
namespace XUnitDemo
    public class OutputTests
        private readonly ITestOutputHelper _testOutput;
        public OutputTests(ITestOutputHelper testOutput)
            _testOutput = testOutput;
        public async Task WriteLine()
            await Task.Delay(1000);


Thursday, May 14, 2015

Split your App.config into Multiple Files

What do you do when any file in your project becomes too big? You break it apart into multiple files!

Did you know that you can split your .NET configuration files into multiple files as well? Here is how...


<?xml version="1.0" encoding="utf-8" ?>
    <section name="customSection" 
    type="ConfigSectionDemo.CustomConfigurationSection, ConfigSectionDemo" />
  <appSettings configSource="AppSettings.config" />
  <customSection configSource="CustomSection.config" />
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />


  <add key="Hello" value="World"/>


<customSection goodnight="Moon">

Thursday, April 30, 2015

Persistent Cache Repository using SQLite

A while back I blogged about an ICacheRepository. I still really like that concept, although admittedly there a few updates that I should make to the implementation.

The basic implementation that I shared before was for an in memory cache. There are plenty of document databases that I would recommend for you to use as a persistent cache mechanism, and they would also have the added benefit of being distributed across services.

What do you do if you need a persistent cache, but you don't have any of those document databases available? Just use files! By which I mean SQLite.

Would this stand up under load? Probably not.
Would I recommend that production systems use this? No.
Is it super simple to setup? Yes, yes it is!


Real Time Web Analytics