SIMD acceleration usually creates a portability problem. The fastest kernel for an AMD64 server may use AVX2 or AVX-512, while an Apple Silicon laptop needs NEON, SVE, or SME, and a constrained deployment may need a pure scalar fallback. Without an abstraction layer, each hot loop becomes a matrix of architecture-specific files, build tags, handwritten assembly, and duplicated tests.
Portable SIMD dispatch solves that problem by splitting the kernel into three layers: a vector-shaped algorithm, generated or handwritten target implementations, and a small dispatcher that selects the best implementation available at build time or runtime. The Go ecosystem historically pushed teams toward scalar code or handwritten assembly, but Go 1.26 introduces an experimental architecture-specific SIMD package for AMD64, and libraries such as go-highway show how to wrap that low-level access in a higher-level, Highway-inspired API.
The transferable technique is not a single library API. It is the design pattern of writing data-parallel kernels once, generating target-specific variants for AVX2, AVX-512, NEON, SVE, SME, and fallback paths, then testing those variants against the same scalar reference. This matters for local inference engines, embedding models, vector search, tokenizers, image processing, compression, and any Go service that needs CPU acceleration without handing the hot path to C or Rust.
A SIMD-friendly algorithm processes contiguous lanes with the same operation. Instead of thinking about one float at a time, the implementation thinks in vector registers: load a block, transform every lane, store the block, and finish the tail with a fallback path. The algorithm should minimize per-lane branches, prefer structure-of-arrays layouts, and expose reductions explicitly.
Each target maps the same logical operations to the best available backend. On AMD64, Go 1.26’s experimental simd/archsimd package exposes architecture-specific vector operations behind GOEXPERIMENT=simd. On ARM64, a library may need assembly wrappers or generated assembly because the standard experimental package does not cover every ARM SIMD family. A scalar fallback remains mandatory for portability and testing.
The public function should not force callers to choose AVX2, NEON, or fallback manually. A dispatcher can bind a package-level function variable during initialization, use build tags to select compiled variants, or perform runtime CPU-feature detection. The caller invokes one stable function while the implementation routes to the fastest safe kernel.
The core engineering goal is semantic unity. Every target variant must implement the same mathematical function, including edge cases, precision expectations, tail handling, and error behavior. Performance changes by target; semantics do not.
A portable SIMD layer exposes operations such as Load, Store, Add, Mul, FMA, ReduceSum, masks, and shuffles. The source algorithm uses that vocabulary instead of calling architecture intrinsics directly. Code generation then emits concrete implementations for the chosen targets.
package kernels
// Logical kernel: multiply each lane by scale and accumulate the sum.
// A generator can lower Load, Mul, Store, and ReduceSum to each target.
func ScaleAndSum(input, output []float32, scale float32) float32 {
acc := hwy.Zero[float32]()
factor := hwy.Set[float32](scale)
for i := 0; i+int(hwy.NumLanes[float32]()) <= len(input); i += int(hwy.NumLanes[float32]()) {
v := hwy.Load(input[i:])
y := hwy.Mul(v, factor)
hwy.Store(output[i:], y)
acc = hwy.Add(acc, y)
}
return hwy.ReduceSum(acc)
}A generator can emit files for avx2, avx512, neon, and fallback. The AMD64 variants may call Go’s experimental architecture-specific SIMD operations. ARM64 variants may call assembly wrappers or generated Go assembly, especially for compute-heavy kernels where per-vector function-call overhead would dominate the work.
# Generate concrete kernels and a dispatcher from one source file.
go build -o bin/hwygen ./cmd/hwygen
./bin/hwygen -input scale_sum.go -output . -targets avx2,avx512,neon,fallbackThe generated package can expose a generic function for convenience and type-specific function variables for hot paths. During initialization, the package binds those variables to the best supported implementation. Tests can force the fallback path to verify correctness and can run target-specific benchmarks when hardware support is present.
var scaleAndSumFloat32 func([]float32, []float32, float32) float32
func init() {
switch {
case cpu.HasAVX512:
scaleAndSumFloat32 = scaleAndSumAVX512Float32
case cpu.HasAVX2:
scaleAndSumFloat32 = scaleAndSumAVX2Float32
case cpu.HasNEON:
scaleAndSumFloat32 = scaleAndSumNEONFloat32
default:
scaleAndSumFloat32 = scaleAndSumFallbackFloat32
}
}
func ScaleAndSumFloat32(input, output []float32, scale float32) float32 {
return scaleAndSumFloat32(input, output, scale)
}Embedding models and rerankers spend substantial CPU time in vector math, activation functions, normalization, and matrix-vector operations. Portable SIMD lets a Go inference engine accelerate those kernels on developer laptops, cloud AMD64 machines, and ARM64 edge devices without maintaining separate public APIs.
ANN search systems rely on dot products, L2 distance, popcount, quantization, dequantization, and bit-packing loops. These operations are naturally data-parallel. A portable dispatch layer allows the same search engine to use AVX-512 on high-end servers, NEON on mobile or Apple Silicon, and scalar fallback in test or unsupported environments.
SIMD is useful beyond neural networks. Codecs, varint decoders, JSON scanners, UTF-8 validation, image filters, and wavelet transforms all benefit from vectorized loops. A Go service can keep those kernels in-process and memory-safe at the API boundary while still using target-specific instructions internally.
Start with a simple scalar reference implementation. Every generated target should be tested against that reference using the same inputs, including empty slices, short slices, tails that are not a multiple of vector width, NaNs, infinities, denormals, and alignment-sensitive offsets. The reference path is not just a fallback; it is the semantic oracle.
func TestScaleAndSumTargets(t *testing.T) {
input := []float32{1, -2, 3.5, 4, 9, 0.25, -8}
wantOut := make([]float32, len(input))
gotOut := make([]float32, len(input))
want := scaleAndSumFallbackFloat32(input, wantOut, 0.5)
got := ScaleAndSumFloat32(input, gotOut, 0.5)
if !nearlyEqual(want, got, 1e-5) {
t.Fatalf("sum mismatch: fallback=%v dispatched=%v", want, got)
}
for i := range wantOut {
if !nearlyEqual(wantOut[i], gotOut[i], 1e-5) {
t.Fatalf("lane %d mismatch: fallback=%v dispatched=%v", i, wantOut[i], gotOut[i])
}
}
}Vector registers operate on fixed lane counts. Real slices rarely align perfectly with that width. Use a scalar tail loop, masked loads and stores, or generated tail code. Silent overreads and overwrites are unacceptable in Go libraries because callers expect slice bounds to be respected.
Keep target-specific code behind narrow functions. The rest of the package should not know whether a kernel came from archsimd, generated assembly, or scalar Go. This preserves testability and makes it possible to add a new target without rewriting callers.
0, 1, lanes-1, lanes, and lanes+1.