Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem.
bet it gets fixed realy fast as those grid guys definitly get listened to.
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
> It really was years of detective work
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
https://news.ycombinator.com/item?id=50314617
It had both ended up in the second chance pool.
Thanks for sharing, very cool!
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:
etc.That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
Systemd timers do this natively btw, no need to hardcode And imho are also friendlier.
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem. bet it gets fixed realy fast as those grid guys definitly get listened to.
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
I believe this is what you're referring to.
https://news.ycombinator.com/item?id=50153229&hl=en-US
Yes, thanks for allowing me to offload my RAM to yours!