Posted on Jul 1

Revisiting My Old Neural Network Project in Go

A few years ago, I built a minimal neural network toolkit in Go: Rolade – a minimal neural network library written in Go.

It was messy, naïve, and mostly forgotten in my GitHub. The goal was to better understand the internals of forward and backpropagation by implementing everything manually, without relying on external ML libraries. The code worked, but it was very basic and lacked in performance.

Now, I’m revisiting this project to refactor the code and improve the overall design. Here’s a summary of what I’ve improved:

What Was Already Working

Modular organization

The codebase already had a good separation of concerns: activation functions, loss, optimizers, and network logic were neatly organized into their own modules. This made it relatively easy to extend the library with new features.

The Improvement : Parallel Training

To improve training efficiency, I implemented parallel batch training.

Training data is split into multiple batches, each processed in a separate goroutine using sync.WaitGroup.
I used sync.Mutex to safely collect training errors and weight deltas from all batches.
After all batches are processed, the deltas are merged using a custom mergeDeltas() function and applied once at the end of the epoch.

Lessons Learned

Race Condition. Initial versions caused data races when multiple goroutines accessed or modified shared weight structures.

Fix: Delay all updates until after parallel computation finishes.

Locking Can Kill Parallelism. Putting locks around weight updates inside each batch makes the process sequential. It removes the benefit of parallelism.

Fix: Do all mutation after goroutines have joined.

Unstable Training Without Averaging. Summing raw deltas led to unstable error gradient.

Fix: Add delta average in mergeDeltas() to stabilize learning.