There are two aspects: first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered, and second that there is a much shorter window between discovery and large-scale exploitation.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.
> People are saying the pace will settle down once LLMs find and fix all the bugs
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
I have often whined about the quality (or lack, thereof) of today's software, but I certainly have no room to be smug. I recently turned an LLM loose on a number of mature packages that I wrote, and would have sworn were "perfect," and it found issues (thankfully, none, serious, but issues, nonetheless).
I think LLMs are a gamechanger. I was just talking to a friend of mine, last night, and he pointed out that it would have been quite difficult for a human (on their own) to find the PHP2Shell issue, but he thinks that multiple LLMs probably found it (it seems that some people knew about it before it was initially "discovered").
I think that we, as programmers, need to run LLMs on our own code -no matter what we think of them, for authoring the code- and do everything we can, to batten down the hatches.
There is tons of old software still around you can use your LLM to poke at. You'll see that it was crap too. Software has always sucked, very little of it has been formally verified or built.
I remember accidentally finding buffer overflows in Netware in school but I had no idea what that actually was at the time.
Everything I've read indicates that yes, this is the new normal.
When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.
The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
Can you explain the asymmetry you are claiming that exists where some models would be excellent at finding faults in code/systems and exploiting them often through coding the required tools to exploit them... but at the same time they would be bad at avoiding this kind of faults when coding the system itself?
Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.
Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.
Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...
Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell
Let’s not forget the AI industry developed the technology and infrastructure for that security nightmare to exists. And no, none of that was inevitable. At every single step the AI leadership took decisions that makes it more likely that we face the current threats
> The rate at which we're churning out AI-generated code is going to make this worse
Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.
This is the new playing field, but this game can't last forever. Most software is not churning through a software factory of endless patches and releases. Most software, once mature, is mostly done.
I see a world in which we all get some generous Dependabot 2.0 on large forges because leaving so much software insecure otherwise is untenable, but once the software is not iterating a billion times a week, you're not introducing new classes of bugs just sitting there.
You can have a frontier security model pentesting your program every night while you sleep and you can wake up knowing that whatever happens, you did the best you could.
Alas, our brave new LLM world is making iterating one billion releases per week all too easy.
And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.
The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.
But at least somebody is maintaining that software.
It already is a thing since I believe April as I follow cybersec.
Last year May, June, July counts of sec bulbs was order of magnitude lower than this year.
But idea is that it will not last forever. New code will be vuln checked by AI as well. So now we are in transition period that is awful but it won’t last.
What confuses me the most is that nobody talks about the nature on how to exploit this vulnerability.
This has been a well known vulnerability for years but considered too complicated to exploit. It is the type of work that would have taken an expert weeks to hack into a single system and the idea that an AI can speed this is up is not unfounded but currently without evidence.
Yes, the attack surface is huge but when compared to Log4J this is materially more difficult to execute and the countless PoC that have been seen were AI slop.
As a security expert I see why you would not want to downplay this, but I think the involved actors have all the incentive to use this for their typical messaging.
As a tech community we have a greater responsibility to not be driven by hype but understand what is going on and let make ourselves some tools and be swayed by the messaging and instead talk about this in a more nuanced way. Perhaps I am wrong because I am not a cyber person myself but this all strikes me as similar to the whole Specter outrage.
> Who knew a regex engine was such a security issue
Anyone who has ever implemented a regex engine, I would imagine. Regex has always been a security nightmare, especially due to persistent attempts to use regexes to validate untrusted input
The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?
It is obviously true that there can only be a finite number of serious security bugs in any given software. For a given level of LLM capability, there is a finite subset of these bugs that is actually discoverable and exploitable. Once this finite subset is exhausted, the tide of new vulnerabilities will stop.
Perhaps we will see a few more tides like this, as LLM capabilities advance further. But there will be an end.
Yes, the PHP risks have been known, and partly assumed. I watched a group from MIT ignore sage advice on avoiding PHP but built their administration interface with it anyways. Now this device is deployed in heavy industry, in mining, in transportation, in healthcare, it's deployed everywhere and in multiple continents and in multiple huge cities. It's concerning that the recommendations we made were eshewed and they went ahead with this anyways and now we are in this situation. This group from MIT had formed a company, and they were bought by a large network device manufacturer. It's deeply deeply concerning.
I'm not in a position to know. But Jason Clinton seemed to approximate "18 months" as an upper bound for the crisis period:
>as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
Big embarrassing network-exploitable bugs in major open-source projects happen sometimes, and network-scanning attack botnets have been around forever. Small shady CAs mess things up sometimes. People upload malware to PyPI and NPM all the time, and "move fast and break things" software-deployment startups get breached all the time, to the point where it doesn't even really feel like news to me when those things happen again.
It's unfortunate that a few of those events all happened at around the same time. But, like, even if that's not a coincidence and these events are actually happening more frequently, you should have been preparing for them anyway!
I can't help but wonder if we're going to see the OS and browser maintainers begin streamlining code and reducing complexity. Software these days almost seems like a fractal of insecurity, the only way to make the problem more tractable is to reduce attack surface.
For now yes. When LLMs can iterate and test at speeds humans cannot, it is going to continue to find holes.
An interesting side effect will be whether we accelerate moving towards memory safe languages (and development of more of those languages) in more sectors than just 'low-level'.
Not forgetting that this is only one mitigation of cyber-attacks and exploits and potentially removes a whole category of attacks developed by analyzing *source code*
My concern is that while we could absolutely eliminate most security bugs through LLM-assisted scanning of source code written in deterministic languages, we are shifting away from deterministic languages because of LLMs themselves. Instead of having purpose-built programs to carry out computational tasks that have a tree of logic that you can trace out and formally prove, we're expecting a black box to guess what we want done and we can't prove what it will do.
It really depends on your gut feeling about how deep the rabbit holes in terms of number of game changer vulnerabilities.
How many RCEs are in PHP of this kind? Many people would've said zero before. After the fixes, it may be zero again. It could be thousands, it could be eighty-six, it could be four, it could be zero. But even if you don't think it's zero again, most people will find themselves that there's probably not many left.
Extend this to other world ecosystem - how many high impact devastating security bugs are there? Dozens, hundreds, thousands, millions? At least when it comes to open source security, there has to be some low upper bound given how countable the number of projects there are where impact could be high and most have taken some care for their quality (despite obvious hardships). And while closed source software is harder to reason about (also on the ecosystem level), if there's a low upper bound for open source software surely the bound for closed source software can't be many orders of magnitude higher.
We now have new tools that allows us to discover these. Both the discovery as well as changes in the development process most are taken should bring these numbers closer to zero again. The main question is - are we working through a small backlog of issues, or a very deep one with no practical end? I'm still inclined to believe it's a manageable backlog, but I'm an optimist and everybody's intuitions differ on this.
People running old versions and not applying patches rigorously is also normal. Its been normal for decades, nobody likes doing maintenance so it gets forgotten or ignored.
> Even though this was found by “responsible security researchers” at “respectable companies”, by the time the security patch actually got released, it was already being exploited in the wild.
I am really not pleased with the "let's delay" strategy. IMO, you should without any delay, let everyone know of security-related issues. This "try to keep it hidden so that malicious actors can not exploit" is incredibly unfair to everyone else. Why would I want to support that? I want to know the exploit, I want it explained and then, in that order, I also expect that it can be fixed eventually. This closed-source thinking is very annoying. If one LLM can find an exploit, so can others, so what's with the delay?
There was a literal PoC video that started the panic, then multiple writeups. It's not hard (for people who know what they're doing) to exploit the PHP vuln. It's just not one exploit to rule every PHP installation anywhere, since this is a low-level exploit that needs to bypass ASLR, instead of a logic bug - so you need to tweak your exploit for every architecture, system, configuration, etc. And you need to know a fair bit about your target.
The response was overblown, but your dismissive reaction is too far in the other direction.
PHP made a mistake by trying to hold back the fix. That's really all it was. It's not okay to do that. They literally chose to keep it unpatchable while it was being exploited in the wild. That's on them.
No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.
> The NHS approach: close source everything.
Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.
Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.
That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.
Somewhat. I take heart in the fact that people actually took the "website down now" messaging seriously and that there was a quick patch to address the issue. It'll still be an arms race between red team and blue team, just faster paced.
I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.
> At the time of writing (23rd Oct), eMudhra has announced that it will be retiring from the certificate authority space. In unrelated news, a farmer has announced that he has shut his stable door, following reports of an escaped horse.
Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.
The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.
Allegedly AIs sometimes even forget to implement authentication for APIs, in the way anyone can just call them. So no, bugs will continue to exist, we've just magically got tons more supply of economical labor.
There are two aspects: first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered, and second that there is a much shorter window between discovery and large-scale exploitation.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.
> People are saying the pace will settle down once LLMs find and fix all the bugs
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
AI skynet is killing humans. We know that already though.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
It's called job security :)
> first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered
The slowest that will ever be.
> second that there is a much shorter window between discovery and large-scale exploitation.
The longest that will ever be.
Aren’t exponentials beautiful?
I have often whined about the quality (or lack, thereof) of today's software, but I certainly have no room to be smug. I recently turned an LLM loose on a number of mature packages that I wrote, and would have sworn were "perfect," and it found issues (thankfully, none, serious, but issues, nonetheless).
I think LLMs are a gamechanger. I was just talking to a friend of mine, last night, and he pointed out that it would have been quite difficult for a human (on their own) to find the PHP2Shell issue, but he thinks that multiple LLMs probably found it (it seems that some people knew about it before it was initially "discovered").
I think that we, as programmers, need to run LLMs on our own code -no matter what we think of them, for authoring the code- and do everything we can, to batten down the hatches.
>of today's software
There is tons of old software still around you can use your LLM to poke at. You'll see that it was crap too. Software has always sucked, very little of it has been formally verified or built.
I remember accidentally finding buffer overflows in Netware in school but I had no idea what that actually was at the time.
Everything I've read indicates that yes, this is the new normal.
When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.
The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
_Is_ AI generated code notoriously terrible at security?
My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.
Are we comparing to old models generating code? Or mythos-class models?
Can you explain the asymmetry you are claiming that exists where some models would be excellent at finding faults in code/systems and exploiting them often through coding the required tools to exploit them... but at the same time they would be bad at avoiding this kind of faults when coding the system itself?
Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?
You just need to use AI to test your AI code.
(What a wonderful new world we live in)
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.
Agreed. Shouldn't continuous pen testing be a thing by now?
> because AI-generated code is kind of notoriously terrible at security.
and vaccines notoriously cause autism. But you might investigate why that belief is so widespread and you might not believe it that much anymore.
Can you point at something specifically that indicates this statement is true?
Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.
Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...
Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell
Let’s not forget the AI industry developed the technology and infrastructure for that security nightmare to exists. And no, none of that was inevitable. At every single step the AI leadership took decisions that makes it more likely that we face the current threats
"New normal"? Is that phrase back again? It was heavily pushed in 2020.
> The rate at which we're churning out AI-generated code is going to make this worse
Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.
Rule #2 - do not blindly trust Ai output
https://susam.net/inverse-laws-of-robotics.html
This is the new playing field, but this game can't last forever. Most software is not churning through a software factory of endless patches and releases. Most software, once mature, is mostly done.
I see a world in which we all get some generous Dependabot 2.0 on large forges because leaving so much software insecure otherwise is untenable, but once the software is not iterating a billion times a week, you're not introducing new classes of bugs just sitting there.
You can have a frontier security model pentesting your program every night while you sleep and you can wake up knowing that whatever happens, you did the best you could.
Alas, our brave new LLM world is making iterating one billion releases per week all too easy.
And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.
> Most software, once mature, is mostly done.
The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.
But at least somebody is maintaining that software.
Not new normal.
It already is a thing since I believe April as I follow cybersec.
Last year May, June, July counts of sec bulbs was order of magnitude lower than this year.
But idea is that it will not last forever. New code will be vuln checked by AI as well. So now we are in transition period that is awful but it won’t last.
What confuses me the most is that nobody talks about the nature on how to exploit this vulnerability.
This has been a well known vulnerability for years but considered too complicated to exploit. It is the type of work that would have taken an expert weeks to hack into a single system and the idea that an AI can speed this is up is not unfounded but currently without evidence.
Yes, the attack surface is huge but when compared to Log4J this is materially more difficult to execute and the countless PoC that have been seen were AI slop.
As a security expert I see why you would not want to downplay this, but I think the involved actors have all the incentive to use this for their typical messaging.
As a tech community we have a greater responsibility to not be driven by hype but understand what is going on and let make ourselves some tools and be swayed by the messaging and instead talk about this in a more nuanced way. Perhaps I am wrong because I am not a cyber person myself but this all strikes me as similar to the whole Specter outrage.
> Who knew a regex engine was such a security issue
Anyone who has ever implemented a regex engine, I would imagine. Regex has always been a security nightmare, especially due to persistent attempts to use regexes to validate untrusted input
> especially due to persistent attempts to use regexes to validate untrusted input
This is not the case here, it's an untrusted regex which can be the issue.
> already being exploited in the wild [0]
The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?
[0]: https://ellio.tech/en/blog/php2shell-what-we-observed-and-wh...
This situation is temporary.
It is obviously true that there can only be a finite number of serious security bugs in any given software. For a given level of LLM capability, there is a finite subset of these bugs that is actually discoverable and exploitable. Once this finite subset is exhausted, the tide of new vulnerabilities will stop.
Perhaps we will see a few more tides like this, as LLM capabilities advance further. But there will be an end.
Yes, the PHP risks have been known, and partly assumed. I watched a group from MIT ignore sage advice on avoiding PHP but built their administration interface with it anyways. Now this device is deployed in heavy industry, in mining, in transportation, in healthcare, it's deployed everywhere and in multiple continents and in multiple huge cities. It's concerning that the recommendations we made were eshewed and they went ahead with this anyways and now we are in this situation. This group from MIT had formed a company, and they were bought by a large network device manufacturer. It's deeply deeply concerning.
I can tell you, if it is, I'm going to be radically changing the software architectures and patterns that I am recommending to people.
Data privacy and security must not be sacrificed.
A shift is needed towards simplification and minimization of attack surfaces.
I've entirely removed several dependencies which provide some pretty nice features, but unfortunately make things almost unmaintainable.
I want to go much further though to simplify architectures themselves even if it makes far worse ux.
I'm not in a position to know. But Jason Clinton seemed to approximate "18 months" as an upper bound for the crisis period:
>as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
https://x.com/JasonDClinton/status/2113325263157395456
Is this the new normal, or just, uh, normal?
Big embarrassing network-exploitable bugs in major open-source projects happen sometimes, and network-scanning attack botnets have been around forever. Small shady CAs mess things up sometimes. People upload malware to PyPI and NPM all the time, and "move fast and break things" software-deployment startups get breached all the time, to the point where it doesn't even really feel like news to me when those things happen again.
It's unfortunate that a few of those events all happened at around the same time. But, like, even if that's not a coincidence and these events are actually happening more frequently, you should have been preparing for them anyway!
I can't help but wonder if we're going to see the OS and browser maintainers begin streamlining code and reducing complexity. Software these days almost seems like a fractal of insecurity, the only way to make the problem more tractable is to reduce attack surface.
For now yes. When LLMs can iterate and test at speeds humans cannot, it is going to continue to find holes.
An interesting side effect will be whether we accelerate moving towards memory safe languages (and development of more of those languages) in more sectors than just 'low-level'.
Not forgetting that this is only one mitigation of cyber-attacks and exploits and potentially removes a whole category of attacks developed by analyzing *source code*
My concern is that while we could absolutely eliminate most security bugs through LLM-assisted scanning of source code written in deterministic languages, we are shifting away from deterministic languages because of LLMs themselves. Instead of having purpose-built programs to carry out computational tasks that have a tree of logic that you can trace out and formally prove, we're expecting a black box to guess what we want done and we can't prove what it will do.
It really depends on your gut feeling about how deep the rabbit holes in terms of number of game changer vulnerabilities.
How many RCEs are in PHP of this kind? Many people would've said zero before. After the fixes, it may be zero again. It could be thousands, it could be eighty-six, it could be four, it could be zero. But even if you don't think it's zero again, most people will find themselves that there's probably not many left.
Extend this to other world ecosystem - how many high impact devastating security bugs are there? Dozens, hundreds, thousands, millions? At least when it comes to open source security, there has to be some low upper bound given how countable the number of projects there are where impact could be high and most have taken some care for their quality (despite obvious hardships). And while closed source software is harder to reason about (also on the ecosystem level), if there's a low upper bound for open source software surely the bound for closed source software can't be many orders of magnitude higher.
We now have new tools that allows us to discover these. Both the discovery as well as changes in the development process most are taken should bring these numbers closer to zero again. The main question is - are we working through a small backlog of issues, or a very deep one with no practical end? I'm still inclined to believe it's a manageable backlog, but I'm an optimist and everybody's intuitions differ on this.
No. This is just the start. It's going to get way worse before it gets better.
People running old versions and not applying patches rigorously is also normal. Its been normal for decades, nobody likes doing maintenance so it gets forgotten or ignored.
> Even though this was found by “responsible security researchers” at “respectable companies”, by the time the security patch actually got released, it was already being exploited in the wild.
I am really not pleased with the "let's delay" strategy. IMO, you should without any delay, let everyone know of security-related issues. This "try to keep it hidden so that malicious actors can not exploit" is incredibly unfair to everyone else. Why would I want to support that? I want to know the exploit, I want it explained and then, in that order, I also expect that it can be fixed eventually. This closed-source thinking is very annoying. If one LLM can find an exploit, so can others, so what's with the delay?
It’s mostly hype. Days in now and nobody can exploit the PHP vuln. Yawn.
That's not true. The vuln was exploited and utilized within minutes of the disclosure.
There was a literal PoC video that started the panic, then multiple writeups. It's not hard (for people who know what they're doing) to exploit the PHP vuln. It's just not one exploit to rule every PHP installation anywhere, since this is a low-level exploit that needs to bypass ASLR, instead of a logic bug - so you need to tweak your exploit for every architecture, system, configuration, etc. And you need to know a fair bit about your target.
The response was overblown, but your dismissive reaction is too far in the other direction.
LiveOverflow has exploited the PHP vulnerability, it’s even linked in the article.
PHP made a mistake by trying to hold back the fix. That's really all it was. It's not okay to do that. They literally chose to keep it unpatchable while it was being exploited in the wild. That's on them.
> Is this the new cyber normal?
No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.
> The NHS approach: close source everything.
Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
[0] Could take a while.
> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.
Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.
That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.
>It isn't realistic to expect new security vulnerabilities to be found at this rate
I wouldn't challenge fate like this.
+1. There aren't an infinite amount of serious security vulnerabilities out there.
> > The NHS approach: close source everything.
> Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
Me too. The agents are much faster if it has access to source. NHS would have been easy hunting grounds. Now at least they don't stand out.
"a system should be secure, even if everything about the system—except the key—is public knowledge"
Should indeed, but we've all been caught with our pants down.
I hope not. A lot of LLM assisted vulnerability stuff has been noise, slop, fluff, or fake/false data. Drowns out the real issues.
Am I the only one that hates seeing "cyber" when refering to cybersecurity?
The same with "crypto" to refer to cryptocoins instead of cryptography.
Somewhat. I take heart in the fact that people actually took the "website down now" messaging seriously and that there was a quick patch to address the issue. It'll still be an arms race between red team and blue team, just faster paced.
By generating the entire website in AI, yes, you are contributing to the new normal
I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.
Not for me. I’m taking my shit offline. Out of harms way.
> At the time of writing (23rd Oct), eMudhra has announced that it will be retiring from the certificate authority space. In unrelated news, a farmer has announced that he has shut his stable door, following reports of an escaped horse.
Ah, wit is alive and strong!
LLMs seem to be very good at detecting vulnerabilities caused by using C and other forms of human stupidity.
So yes.
These are all human-generated software bugs. At some point when AI writes all code, the bugs should all but disappear. Almost.
Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.
The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.
Allegedly AIs sometimes even forget to implement authentication for APIs, in the way anyone can just call them. So no, bugs will continue to exist, we've just magically got tons more supply of economical labor.
I think that’s a fallacy, see https://news.ycombinator.com/item?id=50207540
1. The average human software-generator sucks at security. 2. LLMs are trained on human-generated software. 3. Therefore, LLMs suck at security.