Super dangerous to benchmark lock performance using microbenchmarks. If you have a tiny benchmark, then you're putting the CPU and memory into a very specific and unusual state (everything is quiet other than the lock itself).
The real world story for locks is usually that you're not rage-contending 100% of the time, but that you have some contention combined with CPUs doing some real work and some real memory accesses.
What I've found is that in those more real scenarios, the locks that perform best in microbenchmarks fall apart compared to completely different and unexpected algorithms.
If you're expecting heavy contention, and there's no risk of any of the threads being descheduled, then FIFO spinlocks are probably best.
In a FIFO threads register themselves into a linked list, and the thread calling unlock() directly wakes the next. It's possible to have e.g. 20 threads in this case all spinning on their own cache lines (their private node), rather than a shared one (the lock head).
This can be coherence protocol optimal.
A dumb test and set spinlock, or variant thereof, is going to degrade quickly as all the cores are spinning on the same cacheline causing a lot of coherence traffic between cores (transitions between shared, exclusive and modified states)
Spinlocks are unsuitable for situations where you can be involuntarily context switched (the vast majority of userspace programs). Probably worth mentioning that.
I genuinely had not heard of anyone actually using a spinlock in production code until I started using LMAX Disruptor a few years ago.
I was always told that they were an anti-pattern, and I think that generally that is a pretty good rule of thumb, but I guess like most stuff in CS: there are always exceptions to "good rules of thumb".
I still haven't actually explicitly written a spinlock for anything in production, but Disruptor has shown me that there are cases for it.
If you have an application where your threads are pinned to dedicated cores, and those cores are all isolated from general OS scheduling, then it's the lowest latency means to synchronize arbitrary things between threads
Entering the kernel with a futex wait or wake under contention costs a couple of microseconds, whereas a spinlock will cost you double digit to low triple digit nanos depending on cores/sockets etc
To be really pedantic, it's a spin wait, not a spin lock in disruptor. You are waiting for a sequence, not mutually excluding some resource. Many threads can watch the same volatile at the same time without blocking each other.
This would have different answers depending on if it ran on a machine with a more closely-shared cache, right? For example on an Intel efficiency core cluster where 4 cores share an L2.
Super dangerous to benchmark lock performance using microbenchmarks. If you have a tiny benchmark, then you're putting the CPU and memory into a very specific and unusual state (everything is quiet other than the lock itself).
The real world story for locks is usually that you're not rage-contending 100% of the time, but that you have some contention combined with CPUs doing some real work and some real memory accesses.
What I've found is that in those more real scenarios, the locks that perform best in microbenchmarks fall apart compared to completely different and unexpected algorithms.
If contention is expected, would it be better to first perform a relaxed read before the exchange? For example:
If you're expecting heavy contention, and there's no risk of any of the threads being descheduled, then FIFO spinlocks are probably best.
In a FIFO threads register themselves into a linked list, and the thread calling unlock() directly wakes the next. It's possible to have e.g. 20 threads in this case all spinning on their own cache lines (their private node), rather than a shared one (the lock head).
This can be coherence protocol optimal.
A dumb test and set spinlock, or variant thereof, is going to degrade quickly as all the cores are spinning on the same cacheline causing a lot of coherence traffic between cores (transitions between shared, exclusive and modified states)
Thanks for sharing! Happy to get feedback :)
Note that I don't recommend spinlock for most cases, only when there is a 1:1 mapping between threads and phsycal CPU cores, and only after measuring
Spinlocks are unsuitable for situations where you can be involuntarily context switched (the vast majority of userspace programs). Probably worth mentioning that.
I genuinely had not heard of anyone actually using a spinlock in production code until I started using LMAX Disruptor a few years ago.
I was always told that they were an anti-pattern, and I think that generally that is a pretty good rule of thumb, but I guess like most stuff in CS: there are always exceptions to "good rules of thumb".
I still haven't actually explicitly written a spinlock for anything in production, but Disruptor has shown me that there are cases for it.
If you have an application where your threads are pinned to dedicated cores, and those cores are all isolated from general OS scheduling, then it's the lowest latency means to synchronize arbitrary things between threads
Entering the kernel with a futex wait or wake under contention costs a couple of microseconds, whereas a spinlock will cost you double digit to low triple digit nanos depending on cores/sockets etc
To be really pedantic, it's a spin wait, not a spin lock in disruptor. You are waiting for a sequence, not mutually excluding some resource. Many threads can watch the same volatile at the same time without blocking each other.
I think Linus says it well: https://www.realworldtech.com/forum/?threadid=189711&curpost...
It’s one of the secret ingredients to avoid a Big Kernel Lock™.
> had not heard of anyone actually using a spinlock in production code
Go stdlib sync.Mutex uses spins: https://victoriametrics.com/blog/go-sync-mutex / https://archive.vn/BIb7F
Optimistically spinning for a bit before falling back to futex or equivalent is very different from a spinlock.
not all architectures have atomic cas
Real architectures you'd run more than a single thread on? Such as?
This would have different answers depending on if it ran on a machine with a more closely-shared cache, right? For example on an Intel efficiency core cluster where 4 cores share an L2.