So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents. This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor.
Imagine the potential impact. You're a single mother, fighting for custody of your kids. Your lawyer has some documentation of something that happened to you, that wasn't your fault, but would look bad if brought up in court. Suddenly you receive a phone call - it's a mysterious voice, demanding $10,000 or they will send the documents to the opposition. Neither of them knows each other; someone just found a trove of documents in an open back door and wanted to make a quick buck.
This is exactly what a software building code would address (if we had one!). Just like you can't open a new storefront in a new building without it being inspected, you should not be able to process millions of sensitive files without having your software's building inspected. The safety and privacy of all of us shouldn't be optional.
but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...
And those humans would be looking for a new job or face other consequences. An AI model can merrily do this with zero consequences because no meaningful consequences can be visited upon it.
Just like if any human employee publicly sexually harassed his female CEO, he'd be out of a job and would find it very hard to find a new one. But Grok can do it and it's the CEO who ends up quitting.
90% of human devs can fit more than 3-5 files into their short-term memory.
They also know not to, say, temporarily disable auth to be able to look at the changes they've made on a page hidden behind auth, which is what I observed Gemini 3 Pro doing just yesterday.
Ok, and that’s your prediction for 2 years from now? It’d be quite remarkable if humans had a bigger short term memory than LLMs in 2 years. Or that the kind of dumb security mistakes LLMs make today don’t trigger major, rapid improvements.
Do you understand what the term "context window" means? Have you ever tried using an LLM to program anything even remotely complex? Have you observed how the quality of the output drastically reduces the longer the coversation gets?
That's what makes it bad at security. It cannot comprehend more than a floppy drive worth of data before it reverts to absolute gibberish.
There's about a dozen workarounds around context limits, agents being one of them, MCP servers being another one, AGENTS.md being the third one, but none of them actually solve the issue of a context window being so small that it's useless for anything even remotely complex.
Let's imagine a codebase that can fit onto a revolutionary piece of technology known as a floppy drive. As we all know, a floppy drive can store <2 megabytes of storage. But a 100k tokens is only about 400 kilobytes. So, to process the whole codebase that can fit onto a floppy drive, you need 5 agents plus the sixth "parent process" that those 5 agents will report to.
Those five agents can report "no security issues found" in their own little chunk of the codebase to the parent process, and that parent process will still be none the wiser about how those different chunks interact with each other.
In one instance it could not even describe why a test is bad unit test (asserting true is equal to true), which doesn’t even require context or multi file reasoning.
Its almost as if it has additional problems beyond the context limits :)
You may want to try using it, anecdotes often differ from theories, especially when they are being sold to you for profit. It takes maybe a few days to see a pattern of ignoring simple instructions even when context is clean. Or one prompt fixes one issue and causes new issues, rinse and repeat. It requires human guidance in practice.
Strongman: LLMs aren't a tool, they're fuzzy automation.
And what keeps security problems from making it into prod in the real world?
Code review, testing, static and dynamic code scanning, and fuzzing.
Why aren't these things done?
Because there isn't enough people-time and expertise.
So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and dynamic code scanning, and fuzzing.
It seems very unlikely those forms of automation won't be improved in the near future by even the dumbest form of LLMs.
And if you offered CISOs a "pay to scan" service that actually worked cross-language and -platform (in contrast to most "only supported languages" scanners), they'd jump at it.
And that buys you what, exactly? Your point is 100% correct and why LLMs are no where near able to manage / build complete simple systems and surely not complex ones.
Why? Context. LLMs, today, go off the rails fairly easily. As I've mentioned in prior comments I've been working a lot with different models and agentic coding systems. When a code base starts to approach 5k lines (building the entire codebase with an agent) things start to get very rough. First of all, the agent cannot wrap it's context (it has no brain) around the code in a complete way. Even when everything is very well documented as part of the build and outlined so the LLM has indicators of where to pull in code - it almost always cannot keep schemas, requirements, or patterns in line. I've had instances where APIs that were being developed were to follow a specific schema, should require specific tests and should abide by specific constraints for integration. Almost always, in that relatively small codebase, the agentic system gets something wrong - but because of sycophancy - it gleefully informs me all the work is done and everything is A-OK! The kicker here is that when you show it why / where it's wrong you're continuously in a loop of burning tokens trying to put that train back on the track. LLMs can't be efficient with new(ish) code bases because they're always having to go lookup new documentation and burning through more context beyond what it's targeting to build / update / refactor / etc.
So, sure. You can "call an LLM multiple times". But this is hugely missing the point with how these systems work. Because when you actually start to use them you'll find these issues almost immediately.
To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too. It’s not more reliable than just taking a random person off the street of SF and giving them instructions on what to copy paste without any context. They might also change unrelated things or get sidetracked when they encounter friction. My point is that when you try to compensate by prompting repeatedly, you are just adding more chances for entropy to leak in — so I am agreeing with you.
> To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too.
Spot on. If we look at, historically, "AI" (pre-LLM) the data sets were much more curated, cleaned and labeled. Look at CV, for example. Computer Vision is a prime example of how AI can easily go off the rails with respect to 1) garbage input data 2) biased input data. LLMs have these two as inputs in spades and in vast quantities. Has everyone forgotten about Google's classification of African American people in images [0]? Or, more hilariously - the fix [1]? Most people I talk to who are using LLMs think that the data being strung into these models has been fine tuned, hand picked, etc. In some cases for small models that were explicitly curated, sure. But in the context (no pun) of all the popular frontier models: no way in hell.
The one thing I'm really surprised nobody is talking about is the system prompt. Not in the manner of jailbreaking it or even extracting it. But I can't imagine that these system prompts aren't collecting mass tech debt at this point. I'm sure there's band aid after band aid of simple fixes to nudge the model in ever so different directions based on things that are, ultimately, out of the control of such a large culmination of random data. I can't wait to see how these long term issues crop and and duct taped for the quick fixes these tech behemoths are becoming known for.
Agreed, but "vibe coding will be better at security" is not one of them. Better by which metric, against which threat model, with which stakes? What security even means for greenfield projects is inherently different than for hardened systems. Vibe coding is sufficient for security today because it's not used for anything that matters.
It'll play a role in both securing and security research I'm sure, but I'm not confident it'll be better.
But also, you'd need to have some metrics - how good are developers at security already? What if the bar is on the floor and LLM code generators are already better?
I've seen a lot of job ads (Canva) lately that mandate AI use or AI experience, and as an AI company if they wanted that I think they would have put it in the ad.
For the record I think I may be fine with the insincerity of selling AI but not using it!
In 90% of the cases. And if you don't know how to spot that other 10%, you are still screwed, cause someone else will found that (and you don't even need to be an elite black hat to find it).
But money is a way to motivate the people who created AI to create better AI. Because if it doesn't perform as expected, either people won't use it or they'll turn to a competitor next time they need to do something. And these companies need recurring revenue.
If we're saying the way to ensure competency is to instill fear of not getting money tomorrow as a consequence of failure, then AI companies and humans are on equal footing.
You can run multiple chatbots in parallel. Use different models and different setups.
It's like having multiple people audit your systems. Even if everyone only catches 90%, as long as they don't catch exactly the same 90%, this parallel effort helps.
Basically what happened in the Vastaamo case in Finland [1]. Except of course it wasn't individual phone calls – it was mass extortion of 30,000 people at once via email.
if I remember correctly the attacker got caught in such a silly way
he wanted to demonstrate that he indeed has the private data. But he fucked up the tar command and it ended up having his username in the directory names, a username he used in other places on the internet
http-only makes it also very easy to sniff for LE if they decide to. This allows them to get knowledge about cases. Like, they could be scanning it with their own AI tool for all we know. In a free country with proper LE, this would neither be legal nor happening. But I am not sure the USA is remaining one, given the leader is a convicted felon with very dubious moral standards.
The problem here however is that they get away with their sloppiness as long as the security researcher who found this is a whitehat, and the regular news won't pick it up. Once regular media pick this news up (and the local ones should), their name is tarnished and they may regret their sloppiness. Which is a good way to ensure they won't make the same mistake. After all, money talks.
All the big tech companies are in the news every week. Everybody knows how bad they are. Their names are tarnished and yet everyone is still using their junk and they face zero repercussions when fucking up. I dont think things in the media would do any harm.
In the news, sure. But negatively? I consider myself included in 'everyone', and I am not using junk from all the big tech companies. More than once, I've successfully quit using certain ones, and Signal has become much more popular in my country ever since Trump II took office. Meta had to change the name of their company (Facebook) since it had such a bad name, and Zuck started a charm offensive.
This is HN. We understood exactly what “exposed … confidential files” meant before reading your overly dramatic scenario. As overdone as it is, it’s not even realistic. A likely single mother is likely tiny potatoes in comparison to deep-pocketed legal firms or large corporations.
The story is an example of the market self-correcting, but out comes this “building code” hobby horse anyway. All a software “building code” will do is ossify certain current practices, not even necessarily the best ones. It will tilt the playing field in favor of large existing players and to the disadvantage of innovative startups.
The model fails to apply in multiple ways. Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software. Local city workers inspecting by the local municipality’s code at least has clear jurisdiction because of where the physical fixed location is. Who will write the “building code”? Who will be the inspectors?
This is HN. Of all places, I’d expect to see this presented as an opportunity for new startups, not calls for slovenly bureaucracy and more coercion. The private market is perfectly capable of performing this function. E&O and professional liability insurers if they don’t already will be soon motivated after seeing lawsuits to demand regular pentests.
The reported incident is a great reminder of caveat emptor.
> Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software.
I don't...think this is true? Google has no problems shipping complex software projects, their London HQ is years behind schedule and vastly over budget.
Construction is really complex. These can be mega-projects with tens of thousands of people involved, where the consequences of failure are injury or even death. When software failure does have those consequences - things like aviation control software, or medical device firmware - engineers are held to a considerably higher standard.
> The private market is perfectly capable of performing this function
But it's totally not! There are so many examples in the construction space of private markets being wholly unable to perform quality control because there are financial incentives not to.
The reason building codes exist and are enforced by municipalities is because the private market is incapable of doing so.
The bigwigs at my company want to build out a document management suite. After talking to VP of technology about requirements I ask about security as well as what the regulatory requirements are and all I get is a blank stare.
I used to think developers had to be supremely incompetent to end up with vulnerabilities like this.
But now I understand it’s not the developers who are incompetent…
Oh, I should have been more careful in my formulation:
There are organisations that are generally competent, and there are places that are less competent. It's not all that uncommon for the whole organisation to be generally incompetent.
The saddest places (for me) are those where almost every individual you talk to seems generally competent, but judging by their output the company might as well be stuffed by idiots. Something in the way they are organised suppresses the competence. (I worked at one such company.)
> Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.
It's very important before you start any new job to suss out how competent people and the organisation are. Ideally, you probably want to work for a competent company. But at least you want to know what you are getting into.
There's a bit of luck involved, if you go in blindly, but you can also use skill and elbow-grease to investigate.
Yes, it's hard, and I'm not sure there are general strategies that always work.
It's fundamentally the same problem that the company is trying to solve when they interview you, just the other way 'round.
Some ideas: observe and ask in the interviews and hiring process in general. See what you can find out about the company from friends, contacts and even strangers. Network! Do some online research, too.
Btw, lots of the cliché interview questions ("What are your greatest weaknesses?" etc) actually make decent questions you can ask about the company and team you are about to join.
Something I've found useful is just reaching out to past employees. Usually folks that don't work there anymore will be more transparent. Only challenge is getting someone to respond to you, but you'd be surprised how many folks will talk if you don't come off like you're trying to sell them something or a bot.
Not only does the Peter principle generally show more incompetence the higher up a structure you move, but the outsized influence those positions have make for a very noticeably higher level of “things fucked up by incompetence“ coming from the C suite compared to the rest of the structure.
There’s definitely plenty of incompetence regardless. But I’ve never seen a company where the incompetence was more noteworthy in the cog positions than “leadership”.
I've had the same. Ask them to come up with a ToS and they're like "we'll talk about that in an upcoming meeting" it's been a few years now with nothing.
I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a <1hr fix plus the time for QA testing to make sure it didn't break anything.
Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emails get so much spam it's really hard to separate the noise from the legit signal? I'm genuinely curious.
In my experience, it comes down to project management and organizational structure problems.
Companies hire a "security team" and put them behind the security@ email, then decide they'll figure out how to handle issues later.
When an issue comes in, the security team tries to forward the security issue to the team that owns the project so it can be fixed. This is where complicated org charts and difficult incentive structures can get in the way.
Determining which team actually owns the code containing the bug can be very hard, depending on the company. Many security team people I've worked with were smart, but not software developers by trade. So they start trying to navigate the org chart to figure out who can even fix the issue. This can take weeks of dead-ends and "I'm busy until Tuesday next week at 3:30PM, let's schedule a meeting then" delays.
Even when you find the right team, it can be difficult to get them to schedule the fix. In companies where roadmaps are planned 3 quarters in advance, everyone is focused on their KPIs and other acronyms, and bonuses are paid out according to your ticket velocity and on-time delivery stats (despite PMs telling you they're not), getting a team to pick up the bug and work on it is hard. Again, it can become a wall of "Our next 3 sprints are already full with urgent work from VP so-and-so, but we'll see if we can fit it in after that"
Then legal wants to be involved, too. So before you even respond to reports you have to flag the corporate counsel, who is already busy and doesn't want to hear it right now.
So half or more of the job of the security team becomes navigating corporate bureaucracy and slicing through all of the incentive structures to inject this urgent priority somewhere.
Smart companies recognize this problem and will empower security teams to prioritize urgent things. This can cause another problem where less-than-great security teams start wielding their power to force everyone to work on not-urgent issues that get spammed to the security@ email all day long demanding bug bounties, which burns everyone out. Good security teams will use good judgment, though.
Oh man this is so true. In this sort of org, getting something fixed out-of-band takes a huge political effort (even a critical issue like having your client database exposed to the world).
While there were numerous problems with the big corporate structures I worked in decades ago where everything was done by silos of specialists, there were huge advantages. No matter where there was a security, performance, network, hardware, etc. issue, the internal support infrastructure had the specialist’s pagers and for a problem like this, the people fixing it would have been on a conference call until it was fixed. There was always a team of specialists to diagnose and test fixes, always available developers with the expertise to write fixes if necessary, always ops to monitor and execute things, always a person in charge to make sure it all got done, and everybody knew which department it was and how to reach them 24/7.
Now if you needed to develop something not-urgent that involved, say, the performance department, database department, and your own, hope you’ve got a few months to blow on conference calls and procedure documents.
Their job was specifically managing server resource allocation— as an IT role and not a dev role— in a completely standardized environment. Most applications were given a standard allotment of resources, and they only got involved if something was running out of ram, disk access was too slow, or something just seemed to be taking a lot longer than usual. If it seemed to be a network problem, or just a program crash, for example, they were never involved unless troubleshooting indicated it involved them. More often than not, I’d get a phone call telling me the system I was working on seemed to be heavy on the disk access or something, and they had already allotted it more to keep it stable, but I should check to make sure we weren’t doing something stupid.
Now that I think of it, I’ll bet a lot of companies have a system similar to this for their infrastructure… they just outsource it to AWS, Azure, Google, etc. and comparatively fly by the seat of their pants on the dev side. You could only scale that system down so much, I imagine.
> Many security team people I've worked with were smart, but not software developers by trade.
A lot are people who cannot code at all, cannot administer - they just fill tables and check boxes, maybe from some automated suite.
They dont know what http and https is, because they are just paper pushers what is far from real security, but more like security in name only.
A lot of the time it’s less “nobody checked the security inbox” and more “the one person who understands that part of the system is juggling twelve other fires.” Security fixes are often a one-hour patch wrapped in two weeks of internal routing, approvals, and “who even owns this code?” archaeology. Holiday schedules and spam filters don’t help, but organizational entropy is usually the real culprit.
> A lot of the time it’s less “nobody checked the security inbox” and more “the one person who understands that part of the system is juggling twelve other fires.”
At my past employers it was "The VP of such-and-such said we need to ship this feature as our top priority, no exceptions"
I've once had a whole sector of a fintech go down because one DevOps person ignored daily warning emails for three months that an API key was about to expire and needed reset.
And of course nobody remembered the setup, and logging was only accessible by the same person, so figuring out also took weeks.
I'm currently on the other side of this trying to convince management that the maintenance that should have been done 3 years ago needs to get done. They need "justification".
Write a short memo that saying you are very concerned, and describe a range of things that may happen (from "not much" over medium to maximum scare - lawsuits, brand/customer trust destroyed etc.).
Email the memo to a decision maker with the important flag on and CC: another person as a witness.
If you have been saying it for a long time and nobody has taken any action, you may use the word "escalation" as part of the subject line.
If things hit the fan, it will also make sure that what drops from the fan falls on the right people, and not on you.
It could also be someone "practicing good time management."
They have a specific time of day, when they check their email, and they only give 30 minutes to that time, and they check emails from most recent, down.
The email comes in, two hours earlier, and, by the time they check their email, it's been buried under 50 spams, and near-spams; each of which needs to be checked, so they run out of 30 minutes, before they get to it. The next day, by email check time, another 400 spams have been thrown on top.
Think I'm kidding?
Many folks that have worked for large companies (or bureaucracies) have seen exactly this.
security@ emails do get a lot of spam. It doesn't get talked about very much unless you're monitoring one yourself, but there's a fairly constant stream of people begging for bug bounty money for things like the Secure flag not being set on a cookie.
That said, in my experience this spam is still a few emails a day at the most, I don't think there's any excuse for not immediately patching something like that. I guess maybe someone's on holiday like you said.
There is so much spam from random people about meaningless issues in our docs. AI has made the problem worse. Determining the meaningful from the meaningless is a full time job.
This is where “managed” bug bounty programs like BugCrowd or HackerOne deliver value: only telling you when there is something real. It can be a full time job to separate the wheat from the chaff.
It’s made worse by the incentive of the reporters to make everything sound like a P1 hair-on-fire issue.
My favorite one is the "We've identified a security hole in your website"... and I always respond quickly that my website is statically generated, nothing dynamic and immutable on cloudflare pages. For some odd reason, I never hear back from them.
Well we have 600 people in the global response center I work at. And the priority issue count is currently 26000. That means its serious enough that its been assigned to some one. There are tens of thousands of unassigned issues cuz the traige teams are swamped. People dont realize as systems get more complex issues increase. They never decrease. And the chimp troupes response has always been a Story - we can handle it.
The security@ inbox has so much junk these days with someone reporting that if you paste alert('hacked') into devtools then it makes the website hacked!
I reckon only 1% of reports are valid.
LLM's can now make a plausible looking exploit report ('there is a use after free bug in your server side implementation of X library which allows shell access to your server if you time these two API calls correctly'), but the LLM has made the whole thing up. That can easily waste hours of an experts time for a total falsehood.
I can completely see why some companies decide it'll be an office-hours-only task to go through all the reports every day.
My favorite was "we can trigger your website to initiate a connection to the server we control". They were running their own mail servers and were creating a new accounts on our website. Of course someone needs to initiate a TCP connection to deliver an email message!
Of course this could be a real vulnerability if it would disclose the real server IP behind cloudflare. This was not the case, we were sending via AWS email gateway
Not every organization prioritizes being able to ship a code change at the drop of a hat. This often requires organizational dedication to heavy automated testing a CI, which small companies often aren't set up to do.
I can't believe that any company takes a month to ship something. Even if they don't have CI, surely they'd prefer to break the app (maybe even completely) than risk all their legal documents exfiltrated.
> I can't believe that any company takes a month to ship something.
Outside of startups and big tech, it's not uncommon to have release cycles that are months long. Especially common if there is any legal or regulatory involvement.
I remember heartbleed dropping shortly after a deployment and not being allowed to patch for like ten months because the fix wasn't "validated". This was despite insurers stating this issue could cost coverage and legal getting involved.
It’d be pretty reasonable to take the whole API down in this scenario, and put it back up once it’s patched. They’d lose tons of cash but avoid being liable for extreme amounts of damages.
> October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed
I have unfortunately seen way worse. If it will take more than an hour and the wrong people are in charge of the money, you can go a pretty long time with glaring vulnerabilities.
I call that one of the worrisome outcomes from "Marketing Driven Development" where the business people don't let you do technical debt "Stories" because you REALLY need to do work that justifies their existence in the project.
Another aspect to consider: when you reduce the amount of permission anything has (like here the returned token), you risk breaking something.
In a complex system it can be very hard to understand what will break, if anything. In a less complex system, it can still be hard to understand if the person who knows the security model very well isn't available.
> October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed
There is always the simple answer, these are lawyers so they are probably scrambling internally to write a response that covers themselves legaly also trying to figure out how fucked they are.
I'm a bit conflicted about what responsible disclosure should be, but in many cases it seems like these conditions hold:
1) the hack is straightforward to do;
2) it can do a lot of damage (get PII or other confidential info in most cases);
3) downtime of the service wouldn't hurt anyone, especially if we compare it to the risk of the damage.
But, instead of insisting on the immediate shutting down of the affected service, we give companies weeks or months to fix the issue while notifying no one in the process and continuing with business as usual.
I've submitted 3 very easy exploits to 3 different companies the past year and, thankfully, they fixed them in about a week every time. Yet, the exploits were trivial (as I'm not good enough to find the hard ones, I admit). Mostly IDORs, like changing id=123456 to id=1 all the way up to id=123455 and seeing a lot medical data that doesn't belong to me. All 3 cases were medical labs because I had to have some tests done and wanted to see how secure my data was.
Sadly, in all 3 cases I had to send a follow-up e-mail after ~1 week, saying that I'll make the exploit public if they don't fix it ASAP. What happened was, again, in all 3 cases, the exploit was fixed within 1-2 days.
If I'd given them a month, I feel they would've fixed the issue after a month. If I'd given then a year - after a year.
And it's not like there aren't 10 different labs in my city. It's not like online access to results is critical, either. You can get a printed result or call them to write them down. Yes, it would be tedious, but more secure.
So I should've said from the beginning something like:
> I found this trivial exploit that gives me access to medical data of thousands of people. If you don't want it public, shut down your online service until you fix it, because it's highly likely someone else figured it out before me. If you don't, I'll make it public and ruin your reputation.
Now, would I make it public if they don't fix it within a few days? Probably not, but I'm not sure. But shutting down their service until the fix is in seems important. If it was some hard-to-do hack chaining several exploits, including a 0-day, it would be likely that I'd be the first one to find it and it wouldn't be found for a while by someone else afterwards. But ID enumerations? Come on.
So does the standard "responsible disclosure", at least in the scenario I've given (easy to do; not critical if the service is shut down), help the affected parties (the customers) or the businesses? Why should I care about a company worth $X losing $Y if it's their fault?
I think in the future I'll anonymously contact companies with way more strict deadlines if their customers (or others) are in serious risk. I'll lose the ability to brag with my real name, but I can live with it.
As to the other comments talking about how spammed their security@ mail is - that's the cost of doing business. It doesn't seem like a valid excuse to me. Security isn't one of hundreds random things a business should care about. It's one of the most important ones. So just assign more people to review your mail. If you can't, why are you handling people's PII?
I understand you think you are doing the right thing but be aware that by shutting down a medical communication services there's a non-trivial chance someone will die because of slower test results.
Your responsibility is responsible disclosure.
Their responsibility is how to handle it. Don't try to decide that for them.
> I think in the future I'll anonymously contact companies with way more strict deadlines if their customers (or others) are in serious risk. I'll lose the ability to brag with my real name, but I can live with it.
What you're describing is likely a crime. The sad reality is most businesses don't view protection of customers' data as a sacred duty, but simply another of the innumerable risks to be managed in the course of doing business. If they can say "we were working on fixing it!" their asses are likely covered even if someone does leverage the exploit first—and worst-case, they'll just pay a fine and move on.
Precisely - they view security as just one part of many of their business, instead of viewing it as one of the most important parts. They've insured themselves against a breach, so it's not a big deal for them. But it should be.
The more casualties, the more media attention -> the more likely they, and others in their field, will take security more seriously in the future.
If we let them do nothing for a month, they'll eventually fix it, but in the mean time malicious hackers may gain access to the PII. They might not make it public, but sell that PII via black markets. The company may not get the negative publicity it deserves and likely won't learn to fix their systems in time and to adopt adequate security measures. The sale of the PII and the breach itself might become public knowledge months after the fact, while the company has had a chance to grow in the meantime, and make more security mistakes that may be exploited later on.
And yes, I know it may be a crime - that's why I said I'd report it anonymously from now on. But if the company sits on their asses for a month, shouldn't that count as a crime, as well? The current definition of responsible disclosure gives companies too much leeway, in my opinion.
If I knew I operated a service that was trivial to exploit and was hosting people's PII, I'd shut it down until I fixed it. People won't die if I make everything in my power to provide the test results (in my example of medical labs) to doctors and patients via other means, such as via paper or phone. And if people do die, it would be devastating, of course, but it would mean society has put too much trust into a single system without making sure it's not vulnerable to the most basic of attacks. So it would happen sooner or later, anyway. Although I can't imagine someone dying because their doctor had to make a phone call to the lab instead of typing in a URL.
The same argument about people dying due to the disruption of the medical communications system could be made about too-big-to-fail companies that are entrenched into society because a lot of pension funds have invested in them. If the company goes under, the innocent people dependent on the pension fund's finances would suffer. While they would suffer, which would be awful, of course, would the alternative be to not let such companies go bankrupt? Or would it be better for such funds to not rely so much on one specific company in the first place? That is to say, in both cases (security or stocks in general) the reality is that currently people are too dependent on a few singular entities, while they shouldn't be. That has to change, and the change has to begin somewhere.
SOC2 is mainly to check boxes, and forces you to think about a few things. There’s no real / actual audit, and in my experience the pen tests are very much a money grab. You’re paying way too much money for some “pentesting” automated suite to run.
The auditors themselves pretty much only care that you answered all questions, they don’t really care what the answers are and absolutely aren’t going to dig any deeper.
When I worked for a consulting firm some years back I randomly got put on a project that dealt with payment information. I had never had to deal with payment information before so I was a bit nervous about being compliant. I was pointed to SOC2 compliance which sounded scary. Much to my relief (and surprise), the SOC2 questionnaire was literally just what amounted to a survey monkey form. I answered as truthfully as I could and at the end it just said "congrats you're compliant!" or something to that effect.
I asked my my manager if that's all that was required and he said yes, just make sure you do it again next year. I spent the rest of my time worrying that we missed something. I genuinely didn't believe him until your comment.
Soc2 and most other certifications are akin to the tsa, security theater. After seeing the info sec security space from the inside i can only say that it blows my mind how abhorrent the security space is. Prod db creds in code? A ok. Not using some stupid vendors “pen testing” software on each mr, blasphemy?
Unless im missing something, they replied stating they would look into it and then its totally vague when they patched, with Alex apparently randomly testing later and telling them in a "follow up" that it was fixed.
I dont at all get why there is a paragraph thanking their communication if that is the case.
The time to fix isn't really important, assuming that they took the system offline in the mean time... but we all know they didn't, because that would cost to much.
According to the timeline it took more than a week just for Filevine to respond saying they would review and fix the vulnerability. It was 24 days after initial disclosure when he confirmed the fix was in place.
Given that the author describes the company as prompt, communicative and professional, I think it’s fair to assume there was more contact than the four events in the top of the article.
If they have a billion dollar valuation, this fairly basic (and irresponsible) vulnerability could have cost them a billion dollars. If someone with malice had been in your shoes, in that industry, this probably wouldn't have been recoverable. Imagine a firm's entire client communications and discovery posted online.
They could have sold this to a ransomare group or affiliate for 5-6 figures and then the ransomware group could have exfil'd the data and attempted to extort the company for millions.
Then if they didnt pay and the ransomware group leaked the info to the public, they'd likely have to spend millions on lawsuits and fines anyways.
They should have paid this dude 5-6 figures for this find. It's scenarios like this that lead people to sell these vulns on the gray/black market instead of traditional bug bounty whitehat routes.
I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y.
My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it.
This article demonstrates that, but it does sort of beg the question as to why not trust one vs the other when they both promise the same safeguards.
While the FileVine service is indeed a Legal AI tool, I don't see the connection between this particular blunder and AI itself. It sure seems like any company with an inexperienced development team and thoughtless security posture could build a system with the same issues.
Specifically, it does not appear that AI is invoked in any way at the search endpoint - it is clearly piping results from some Box API.
There is none. Filevine is not even an "AI" company. They are a pretty standard SaaS that has some AI features nowadays. But the hive mind needs its food, and AI bad as we all know.
Because it's the Cloud and we're told the cloud is better and more secure.
In truth the company forced our hand by pricing us out of the on-premise solution and will do that again with the other on-premise we use, which is set to sunset in five years or so.
Probably has more to do with responsibility outsourcing: if SaaS has security breach AND they tell in the contract that they’re secure, then you’re not responsible. Sure, there may be reputational damage for you, but it’s a gamble with good odds in most cases.
Storing lots of legal data doesn’t seem to be one of these cases though.
Selling an on-premise service requires customer support, engineering, and duplication of effort if you’re pushing to the cloud as well. Then you get the temptations and lock in of cloud-only tooling and an army of certified consultant drones whose resumes really really need time on AWS-doc-solution-2035, so the on premise becomes a constant weight on management.
SaaS and the cloud is great for some things some of the time, but often you’re just staring at the marketing playbook of MS or Amazon come to life like a golem.
SaaS is now a "solved problem"; almost all vendors will try to get SOX/SOC2 compliance (and more for sensitive workloads). Although... its hard to see how these certifications would have prevented something like this :melting_face:.
Does SaaS X/Cloud offer IAM capabilities? Or going further, do they dogfood their own access via the identity and access policies? If so, and you construct your own access policy, you have relative peace of mind.
If SaaS Y just says "Give me your data and it will be secure", that's where it gets suspect.
> My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it.
The funny thing is that this exploit (from the OP) has nothing to do with AI and could be <insert any SaaS company> that integrates into another service.
And nobody seems to pay attention to the fact that modern copiers cache copies on a local disk and if the machines are leased and swapped out the next party that takes possession has access to those copies if nobody bothered to address it.
The first thing that comes to my mind is SOC2 HIPAA and the whole security theater.
I am one of the engineers that had to suffer through countless screenshots and forms to get these because they show that you are compliant and safe. While the real impactful things are ignored
SemiAnalysis made this a base requirement for being appropriately ranked on their ClusterMAX report, telling me it is akin to FAA certifications, and then getting hacked themselves for not enforcing simple security controls.
You have to start somewhere though. Security theater sucks, and it's not like compliance is a silver bullet, but at least it's something. Having been through implementing standards compliance, it did help the company in some areas. Was it perfect? Definitely not. Was it driven by financial goals? Absolutely. It did tighten up some weak spots though.
If the options mainly consist of "trust me bro" vs "we can demonstrate that we put in some effort", the latter seems more preferable, even if it's not perfect.
This is the collision between two cultures that were never meant to share the same data: "move fast and duct-tape APIs together" startup engineering, and "if this leaks we ruin people's lives" legal/medical confidentiality.
What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals. This is a 2010-level bug pattern wrapped in 2025 AI hype. The only truly "AI" part is that centralizing all documents for model training drastically raises the blast radius when you screw up.
The economic incentive is obvious: if your pitch deck is "we'll ingest everything your firm has ever touched and make it searchable/AI-ready", you win deals by saying yes to data access and integrations, not by saying no. Least privilege, token scoping, and proper isolation are friction in the sales process, so they get bolted on later, if at all.
The scary bit is that lawyers are being sold "AI assistant" but what they're actually buying is "unvetted third party root access to your institutional memory". At that point, the interesting question isn't whether there are more bugs like this, it's how many of these systems would survive a serious red-team exercise by anyone more motivated than a curious blogger.
First, as an organization, do all this cybersecurity theatre, and then create an MCP/LLM wormhole that bypasses it all.
All because non-technical folks wave their hands about AI and not understanding the most fundamental reality about LLM software being fundamentally so different than all the software before it that it becomes an unavoidable black hole.
I'm also a little pleased I used two space analogies, something I can't expect LLMs to do because they have to go large with their language or go home.
My first reaction to the announcement of MCP was that I must be missing something. Surely giving an LLM unlimited access to protected data is going to introduce security holes?
Assuming a 101 security program past the quality bar, there are a number of reason why this can still happen at companies.
Summarized as - security is about risk acceptance, not removal. There’s massive business pressure to risk accept AI. Risk acceptance usually means some sort of supplemental control that’s not the ideal but manages. There are very little of these with AI tools however - small vendors, they’re not really service accounts but IMO best way to monitor them probably is that, integrations are easy, eng companies hate devs losing admin of some kind but if you have that random AI on endpoints becomes very likely.
I’m ignoring a lot of nuance but solid sec program blown open by LLM vendors is going to be common, let alone bad sec programs. Many sec teams I think are just waiting for the other shoe to drop for some evidentiary support while managing heavy pressure to go full bore AI integration until then.
And then folks can gasp and faint like goats and pretend they didn’t know.
It reminds me of the time I met an IT manager who dint have an IT background. Outsourced hilarity ensued through sales people who were also non-technical.
Nitpick, but wormholes and black holes aren't limited to space! (unless you go with the Rick & Morty definition where "there's literally everything in space")
Maybe this is the key takeaway of GenAI: that some access to data, even partially hallucinated data, is better than the hoops that the security theatre puts in place that prevents average Joe doing their job.
This might just be a golden age for getting access to the data you need for getting the job done.
Next security will catch up and there'll be a good balance between access and control.
Then, as always security goes to far and nobody can get anything done.
"GenAI" is nothing new. "AI" is just software. It's not intelligent, or alive, or sentient, or aware. People can scifi sentimentalize it if they want.
It might simulate parts of things, hopefully more reliably.
It's however a different category of software which requires management that doesn't exist yet how it should.
Cybersecurity security theatre for me is using a web browser to secure and administer what was previously already done and creating new security holes from a web interface.
Then, bypassing it to allow unmanaged MCP access to internal data moats creating it's own universe of security vulnerabilities, full stop. In a secured and contained environment, using an MCP to access data to unlock insight is one thing.
It doesn't mean dont' use MCPs. It means the AI won't figure out what the user doesn't know about security around securing MCPs which is a far more massive vulnerability because users of AI have delegated their thinking to a statistics formula ("GenAI"), because it is so impressive on the surface, but no one is checking the work to make sure it stays that way. Managing quality however, is improving.
My comment is calling out effectively letting external paths have unadulterated access to your private and corporate data.
Data is the new moat. Not UI/UX/Software.
A wormhole that exposes your data makes it available for someone to put it into their data moat far too commonly, and also for it to be mis-interpretted.
I'm less and less sure that when a billion-dollar company screws up this bad, the right thing to do is privately disclose it and let them fix it. This kind of thing just allows companies to go on taking people's money without facing the consequences of their mistakes.
It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.
Right, that's the problem. There need to be standards that govern what can ever be released to customers/the public in the first place. When violations of those are discovered, the penalties should be based on time from release, so the longer it was out in the wild, the greater the penalty.
But you can't remove something from the internet once it's there, so once it's released, it's expected that it always will be.
It's also impossible to guarantee a 100% secure infrastructure, no matter how good your product team is.
In the grey is a term of art: "best efforts."
If data is leaking, and it wasn't because hackers bypassed a bunch of safeguards, if it can be shown that you didn't use Best Efforts to secure said data, there is liability.
A charitable way of interpreting "best effort" is that it's similar to what I said: we need standards. But the problems with our notion of "best effort" are:
1. The standards aren't clearly defined (i.e., you must specifically do this).
2. They are defined in terms of efforts rather than effects. It is like saying "every car sold must be made of steel" rather than "every car sold must be capable of withstanding an impact against a concrete wall at 60mph with X amount of deformation, etc." We want the rules to determine what level of threat is protected against, not just what motions the company went through. In the case in the article, it wasn't because hackers bypassed a bunch of safeguards; the company didn't protect against even basic threats.
3. It's not enough to have "liability". That puts the onus on individuals to sue the company for their specific damages. We need criminal penalties that are designed to punish companies (and the individuals who direct them) for the harm they do to society by the overall process of rushing ahead selling things instead of slowing down and being careful. We need large-scale enforcement so that companies actually stop doing these things because the cost of doing them becomes too enormous.
4. Our laws do not adequately take account of the differential power of those who cut corners, and the differential gains reaped. We frequently find small operators on the wrong end of painful lawsuits and onerous criminal penalties, while the biggest companies and wealthiest individuals use their position to avoid consequences. Laws need to explicitly take this into account, lowering the standard of proof for penalties against larger, wealthier, and more powerful companies and individuals, and also making those penalties exponentially higher.
So is that true if they find out when the public does too? It seems that disclosing it privately has some upside (protecting the users) and no downside.
What would you suggest the right thing to do would be?
Edit: I agree with you that we shouldnt let companies like this get away with what amounts to a slap on the wrist. But everything else seems irresponsible as well.
I guess if I imagine the ideal world, it would be that you report it to the authorities and they impose penalties on the offender that are large enough that the company winds up significantly worse off than if they had just grown more slowly. In other words the punishment for moving fast and breaking things needs to be bad enough to outweigh the gains of doing so.
In the current world, I dunno. I guess it depends on what the company is. If it's something like a hedge fund or a fossil fuel company I think I'd be fine with some kind of wikileaks-like avenue for exposing it in such a way that it results in the company being totally destroyed.
I am at a loss for words. This wasn't a sophisticated attack.
I'd love to know who filevine uses for penetration testing (which they do, according to their website) because holy shit, how do you miss this? I mean, they list their bug bounty program under a pentesting heading, so I guess it's just nice internet people.
This was my impression after reading the article too. I have no doubt that the team at Filevine attempted to secure their systems and have probably thwarted other attackers, but got their foot stuck in what is an unsophisticated attack. It only takes one chain vulnerability to bring down the site.
Security reminds me of the Anna Karenina principle: All happy families are alike; each unhappy family is unhappy in its own way.
Man. Hopefully their remediation steps included a full audit of their Box's account.
One could only imagine that if OP wasn't the first to discover it, people could've generated tons of shared links for all kinds of folders, for instance, which would remain active even if they invalidated the API token.
Old school blue chip type of companies are like this too. They’ve thrown all the process and caution they used to have to the wind so that they can… apply AI to their IT org which isn’t even their core business?
It's so great that they allowed him to publish a technical blog post. I once discovered a big vulnerability in a listed consumer tech company -- exposing users' private messages and also allowing to impersonate any user. The company didn't allow me to write a public blogpost.
Up until Van Buren v. United States in 2020, ToS violations were sometimes prosecuted as unauthorized access under the CFAA. I suspect there are other jurisdictions that still do the equivalent to that.
Presumably they'll threaten to sue you and/or file a criminal complaint, which can be pretty hard to deal with depending on the jurisdiction. At that point you'll probably start asking yourself if it's worth publishing a blog post for some internet points.
Through that API, the frontend is handed a high-privilege API key for Box. Not only was that a huge blunder on the backend, but it reveals what passes for architecture these days. Should our application's backend speak to the super sensitive file store? No, we should hand over the keys to that to the React app, because it literally did not occur to them that there's anything that physically could be driven by the frontend that shouldn't be.
My apologies to the frontend engineers out there who know what they're doing.
I don't disagree with the sentiment. But let's also be honest. There is a lot of improvement to be made in security software, in terms of ease of use and overcomplicating things.
I worked at Google and then at Meta. Man, the amount of "nonsense" of the ACL system was insane. I write nonsense in quotes because for sure from a security point of view it all made a lot of sense. But there is exactly zero chance that such a system can be used in a less technical company. It took me 4 years to understand how it worked...
So I'll take this as another data point to create a startup that simplifies security... Seems a lot more complicated than AI
Lawyers can and will send cease and desist letters to people whether or not there is any legal basis for it. Often the threat of a lawsuit, even a meritless one, is enough to keep people quiet.
FYI, a "cease and desist" carries the same legal weight as me sending a one-liner saying "Knock it off".
They are strongly worded requests from a legal point of view. The only real message they send is that the sender is serious enough about the issue to have involved a lawyer, unless of course you write it yourself, which is something that literally anyone can do.
If you want to actually force an action, you need a court order of some type.
NB for the actual lawyers: I'm oversimplifying, since they can be used in court to prove that you tried to get the other party to stop, and tried to resolve the issue outside of court.
Given the absurd amount startups I see lately that have the words "healthcare" and "AI", I'm actually incredibly concerned that in just a couple of months we're going to have an multiple, enormous HIPAA-data disasters
Hey I think I've just found a new marketing stunt for a new vibe-coding platform:
"Worried your vibe-coded app is about to be broadcast on the internet’s biggest billboard? Chill. ACME AI now wraps it in “NSA-grade” security armor."
I've never thought that there will be multiple billion-dollar-AI-features that fixes all the monkey patching problems that no one saw them coming from the older billion-dollar-AI-features that fixes all the monkey patching problems that no one saw them coming from...
-The Filevine team was responsive, professional, and took the findings seriously throughout the disclosure process. They acknowledged the severity, worked to remediate the issues, allowed responsible disclosure, and maintained clear communication. This is another great example of how organizations should handle security disclosures.
In the same tenure I think that a professional etical hacker or a curious fellow that is poking around with no harm intent, shouldn't disclose the name of the company that had a security issue if they resolve it professionally.
You can write the same blog post without mentioning that it was Filevine.
If they didn't take care of the incident that's a different story...
This is a very standard part of responsible disclosure. Hacker finds bugs -> discloses them to the vendor -> (hopefully) the vendor communicates with them and remediates -> both sides publish the technical details. It also helps to demonstrate to the rest of the security world which companies will take reports seriously and which ones won’t, which is very useful information to have.
That's not how ethical disclosure works. Both parties should publish and we, the wider tech industry should see this as a good thing both for the hacker and the company that worked with them.
Eh, with something this horrendously egregious I think their customers have a right to know how carelessly their data was handled, regardless of the remediation steps taken after disclosure; that aside, who knows how many other AI SaaS vendors might stumble across this article and realize they've made a similarly boneheaded error, and save both themselves and their customers a huge amount of pain . . .
That doesn't surprise me one bit. Just think about all the confidential information that people post into their Chatgpt and Claude sessions. You could probably keep the legal system busy for the next century on a couple of days of that.
i recall reading a silly article like half a year ago about using leetspeak and setting the prompt up to emulate House the tv show or something to get around restrictions
This might be off topic since we are in topic of AI tool and on HackerNews.
I've been pondering a long time how does one build a startup company in domain they are not familiar with but ... Just have this urge to 'crave a pie' in this space. For the longest time, I had this dream of starting or building a 'AI Legal Tech Company' -- big issue is, I don't work in legal space at all. I did some cold reach on lawfirm related forums which did not take any traction.
I later searched around and came across the term, 'case management software'. From what I know, this is what Cilo fundamentally is and make millions if not billion.
This was close to two years or 1.5 years ago and since then, I stopped thinking about it because of this understanding or belief I have, "how can I do a startup in legal when I don't work in this domain" But when I look around, I have seen people who start companies in totally unrelated industry. From starting a 'dental tech's company to, if I'm not mistaken, the founder of hugging face doesn't seem to have PHD in AI/ML and yet founded HuggingFace.
Given all said, how does one start a company in unrelated domain? Say I want to start another case management system or attempt to clone FileVine, do I first read up what case management software is or do I cold reach to potential lawfirm who would partner up to built a SAAS from scratch? Other school of thought goes like, "find customer before you have a product to validate what you want to build", how does this realistically work?
I think if you have no domain expertise or unique insight it will be quite hard to find a real pain point to solve, deliver a winning solution, and have the ability to sell it.
Not impossible, but very hard. And starting a company is hard enough as it is.
So 9/10 times the answer will be to partner with someone who understands the space and pain point, preferably one who has lived it, or find an easier problem to solve.
1. Compliancy with relevant standards. HIPAA, GDPR, ISO, military, legal, etc. Realistically you're going to outsource this or hire someone who knows how to build it, and then you're going to pay an agency to confirm that you're compliant. You also need to consider whether the incumbent solution is a trust-based solution, like the old "nobody gets fired for buying Intel".
2. Domain expertise is always easier if you have a domain expert. Big companies also outsource market research. They'll go to a firm like GLG, pay for some expert's time or commission a survey.
It seems like table stakes to do some basic research on your own to see what software (or solutions) exist and why everyone uses them, and why competitors failed. That should cost you nothing but time, and maybe expense if you buy some software. In a lot of fields even browsing some forums or Reddit is enough. The difference is if you have a working product that's generic enough to be useful to other domains, but you're not sure. Then you might be able to arrange some sort of quid pro quo like a trial where the partner gets to keep some output/analysis, and you get some real-world testing and feedback.
I think it comes down to, having some insight about the customer need and how you would solve it. Having prior experience in the same domain is helpful but is neither a guarantee nor a blocker, towards having a customer insight (lots of people might work in a domain but have no idea how to improve it; alternatively an outsider might see something that the "domain experts" have been overlooking).
I just randomly happened to read about the story of, some surgeons asking a Formula 1 team to help improve its surgical processes, with spectacular results in the long term... The F1 team had zero medical background, but they assessed the surgical processes and found huge issues with communication and lack of clarity, people reaching over each other to get to tools, or too many people jumping to fix something like a hose coming loose (when you just need 1 person to do that 1 thing). F1 teams were very good at designing hyper efficient and reliable processes to get complex pit stops done extremely quickly, and the surgeons benefitted a lot from those process engineering insights, even though it had nothing specifically to do with medical/surgical domain knowledge.
Anyways, back to your main question -- I find that it helps to start small... Are you someone who is good at using analogies to explain concepts in one domain, to a layperson outside that domain? Or even better, to use analogies that would help a domain expert from domain A, to instantly recognize an analogous situation or opportunity in domain B (of which they are not an expert)? I personally have found a lot of benefit, from both being naturally curious about learning/teaching through analogies, finding the act of making analogies to be a fun hobby just because, and also honing it professionally to help me be useful in cross-domain contexts. I think you don't need to blow this up in your head as some big grand mystery with some big secret cheat code to unlock how to be a founder in a domain you're not familiar with -- I think you can start very small, and just practice making analogies with your friends or peers, see if you can find fun ways of explaining things across domains with them (either you explain to them with an analogy, or they explain something to you and you try to analogize it from your POV).
"Companies often have a demo environment that is open" - huh?
And... Margolis allowed this open demo environment to connect to their ENTIRE Box drive of millions of super sensitive documents?
HUH???!
Before you get to the terrible security practices of the vendor, you have to place a massive amount of blame on the IT team of Margolis for allowing the above.
No amount of AI hype excuses that kind of professional misjudgement.
I don't think we have enough information to conclude exactly what happened. But my read is the researcher was looking for demo.filevine.com and found margolis.filevine.com instead. The implication is that many other customers may have been vulnerable in the same way.
Ah, I see now that I read too quickly - the "open demo environment" was clearly referencing the idea that the vendor (Filevine) would have a live demo, NOT that each client wanted an open playground demo account that is linked to a subset of their data (which would be utterly insane).
When liability of a corporation and its owners is limited, does it benefit their business to be dilligent in every step or have a mentality of move fast and break things?
I mean... in what world would you send a customers private root key to a web browsing client. Like even if the user was authenticated why would they need this? This sort of secret shouldn't even be in an environment variable or database but stored with encryption at rest. There could easily have been a proxy service between client and box if the purpose is to search or download files. It's very bad, even for a prototype... this researcher deserves a bounty!
> Who is Margolis, and are they happy that OP publicly announced accessing all their confidential files?
Google tells me they are a NY law firm specializing in Real Estate and Immigration law. There are other firms with Margolis in the name too. Kinda doesn't matter; see below.
I doubt that they are thrilled to have their name involved in this, but that is covered by the US constitution's protections on free press.
My thing is, even ingesting the BOK should have been done in phases, to avoid having all your virtual eggs in one basket or nest at any ONE time. Staggering tokens to these compartments would not have cost them anything at all . I always say, whatever convenience you enjoy yourself, will be highly appreciated by bad actors... WHEN, not if.. they get thru.
For all the talk in the blog of how "super-professional" their team was (probably just a courtesy on the side of the author, I don´t think he believes his own words either)... I have noticed using AI to produce some kind of API -OR- use a 3rd party point with integration into frontend, is almost guaranteed to produce code in which the frontend either exposes the API secrets directly in the frontend code (literally injecting it into a variable as string), or if you ask it for authentication, it will build some half-built lazy solution which makes no sense. So I imagine their "super-professional" team built this with AI, blindly trusting, probably even allowing it to commit and merge changes itself because if you are not merging 10K LoC a day with all this "great" technology, what are you even doing, right? It is not super-professional to work effectively with a blindfold on, I´d argue.
Attorneys are ethically obligated to follow very stringent rules to protect their client's confidential information. Having been a practicing litigator for 40+ years, I can confidently state I came across very few attorneys who truly understood their obligations.
Things were easier when I first began practicing in the 1970s. There weren't too many ways confidential materials in our files could be compromised. Leaving my open file spread out on the conference room table while I went to lunch while attorneys arriving for a deposition on my partner's case were one by one seated into the conference room. That's the kind of thing we had to keep an eye on.
But things soon got complicated. Computers. Digital copies of files that didn't disappear into an external site for storage like physical files. Then email. What were our obligations to know what could - and could not - be intercepted while email traveled the internet.
Then most dangerous of all. Digital storage that was outside our physical domain. How could we now know if the cloud vendor had access to our confidential data? Where were the backups stored? How exactly was the data securely compartmentalized by a cloud vendor? Did we need our own IT experts to control the data located on the external cloud? What did the contracts with the cloud vendor say about the fact we were a law firm and that we, as the lawyers responsible for our clients confidential information, needed to know that they - the cloud vendor - understood the legal obligations and that they - the cloud vendor - would hire lawyers to oversee the manner in which the cloud vendor blocked all access to the legal data located on their own servers. And so on and so forth.
I'm no longer in active practice but these issues were a big part of my practice my last few years at a Fortune 500 insurance company that used in-house attorneys nationwide to represent insureds in litigation - and the corporation was in engaged in signing onto a cloud service to hold all of the corporate data - including the legal departments across all 50 states. It was a nightmare. I'm confident it still is.
I'll be honest... I'm not at all surprised that this happened. Purely because it seems like everyone who wants to implement AI just forgot all of the institutional knowledge that cybersecurity has acquired over the last 30-40 years. When you "forget" all of that because you want to rush out something really fast, well, you know what they say: play stupid games, win stupid prizes and all that.
Personally, I'd just use common sense and good judgment. At the end of the day, would you want someone to hand your address, and other private data to OpenAI just like that? Probably not. So don't paste customer data into it if you can avoid it.
On the other hand, minified code is literally published by the company. Everyone can see it and do with it as they please. So handing that over to an AI to un-minify is not really your problem, since you're not the developer working on the tool internally.
I got downvoted, so maybe that means someone thinks un-minifying code is not advised for dealing with security issues? But on reflection surely you can just use the 'format code' command in the ide? I am no expert but surely it's ok to use AI to help track down and identify security issues with the usual caveats of 'don't believe it blindly, do your double checking and risk assessing.'
That comment didn't read like AI generated content to me. It made useful points and explained them well. I would not expect even the best of the current batch of LLMs to produce an argument that coherent.
This sentence in particular seems outside of what an LLM that was fed the linked article might produce:
> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.
The users' comment history does read like generic LLM output. Look at the first lines of different comments:
> Interesting point about Cranelift! I've been following its development for a while, and it seems like there's always something new popping up.
> Interesting point about the color analysis! It kinda reminds me of how album art used to be such a significant part of music culture.
> Interesting point about the ESP32 and music playback! I've been tinkering with similar projects, and it’s wild how much potential these little devices have.
> We used to own tools that made us productive. Now we rent tools that make someone else profitable. Subscriptions are not about recurring value but recurring billing
> Meshtastic is interesting because it's basically "LoRa-first networking" instead of "internet with some radios attached." Most consumer radios are still stuck in the mental model of walkie-talkies, while Meshtastic treats RF as an IP-like transport layer you can script, automate, and extend. That flips the stack:
> This is the collision between two cultures that were never meant to share the same data: "move fast and duct-tape APIs together" startup engineering, and "if this leaks we ruin people's lives" legal/medical confidentiality.
The repeated prefixes (Interesting point about!) and the classic it's-this-not-that LLM pattern are definitely triggering my LLM suspicions.
I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.
Or maybe these are people who learned from a LLM that English is supposed to sound like this if you want to be permitted to communicate a.k.a. "to be taken into consideration"! Which is wrong and also kinda sucks, but also it sucks and is wrong for a kinda non-obvious reason.
Or, bear with me there, maybe things aren't so far downhill yet, these users just learned how English is supposed to sound, from the same place where the LLMs learned how English is supposed to sound! Which is just the Internet.
AI hype is already ridiculous; the whole "are you using an AI to write your posts for you" paranoia is even more absurd. So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere. Just like most non-AI-generated thoughts, except perhaps the one which leads to the fridge.
Or maybe the 2 month old account posting repetitive comments and using the exact patterns common to AI generated comment is, actually, posting LLM generated content.
> So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere.
FYI, spammers love LLM generated posting because it allows them to "season" accounts on sites like Hacker News and Reddit without much effort. Post enough plausible-sounding comments without getting caught and you have another account to use for your upvote army, which is a service you can now sell to desperate marketing people who promised their boss they'd get on the front page of HN. This was already a problem with manual accounts but it took a lot of work to generate the comments and content.
> I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.
Yes, if this is LLM then it definitely wouldn't be zero-shot. I'm still on the fence myself as I've seen similar writing patterns with Asperger's (specifically what used to be called Asperger's; not general autism spectrum) but those comments don't appear to show any of the other tells to me, so I'm not particularly confident one way or the other.
That's ye olde memetic "immune system" of the "onlygroup" (encapsulated ingroup kept unaware it's just an ingroup). "It don't sound like how we're taught, so we have no idea what it mean or why it there! Go back to Uncanny Valley!"
It's always enlightening to remember where Hans Asperger worked, and under what sociocultural circumstances that absolutely proverbial syndrome was first conceived.
GP evidently has some very subtle sort of expectations as to what authentic human expression must look like, which however seem to extend only as far as things like word choice and word order. (If that's all you ever notice about words, congrats, you're either a replicant or have a bad case of "learned literacy in USA" syndrome.)
This makes me want to point out that neither the means nor the purpose of the kind of communication which GP seems to implicitly expect (from random strangers) are even considered to be a real thing in many places and by many people.
I do happen to find that sort of thing way more coughinterestingcough than the whole "howdy stranger, are you AI or just a pseud" routine that HN posters seem to get such a huge kick out of.
Sure looks like one of the most basic moves of ideological manipulation: how about we solved the Turing Test "the wrong way around" by reducing the tester's ability to tell apart human from machine output, instead of building a more convincing language machine? Yay, expectations subverted! (While, in reality, both happen simultaneously.)
Disclaimer: this post was written by a certified paperclip optimizer.
It's probably a list of bullet points or disjointed sentences fed to the LLM to clean up. Might be a non-English speaker using it to become fluent. I won't criticize it, but it's clearly LLM generated content.
That was literally the same thought that crossed my mind. I agree wholeheartedly, accusing everything and everyone of being AI is getting old fast. Part of me is happy that the skepticism takes hold quickly, but I don't think it's necessary for everyone to demonstrate that they are a good skeptic.
(and I suspect that plenty of people will remain credulous anyway, AI slop is going to be rough to deal with for the foreseeable future).
Spammers use AI comments to build reputation on a fleet of accounts for upvoting purposes.
That may or may not be what's happening with this account, but it's worth flagging accounts that generate a lot of questionable comments. If you look at that account's post history there's a lot of familiar LLM patterns and repeated post fragments.
Yeah, you have a point... the comment - and their other comments, on average - seem to fit quite a specific pattern. It's hard to really draw a line between policing style and actually recognising AI-written content, though.
What makes you think that? it would need some prompt engineering if so since ChatGPT won't write like that (bad capitalization, lazy quoting) unless you ask it to
We finally have a blog that no one (yet) has accused of being ai generated, so obviously we just have to start accusing comments of being ai. Can't read for more than 2 seconds on this site without someone yelling "ai!".
For what it's worth, even if the parent comment was directly submitted by chatgpt themselves, your comment brought significantly less value to the conversation.
It's the natural response. AI fans are routinely injecting themselves into every conversation here to somehow talk about AI ("I bet an AI tool would have found the issue faster") and AI is forcing itself onto every product. Comments dissing anything that sounds even remotely like AI is the logical response of someone who is fed up.
Every other headline and conversation having ai is super annoying.
But also, its super annoying to sift through people saying "the word critical was used, this is obviously ai!". not to mention it really fucking sucks when you're the person who wrote something and people start chanting "ai slop! ai slop!". like, how am i going to prove is not AI?
I can't wait until ai gets good enough that no one can tell the difference (or ai completely busts and disappears, although that's unlikely), and we can go back to just commenting about whether something was interesting or educational or whatever instead of analyzing how many em-dashes someone used pre-2020 and extrapolating whether their latest post has 1 more em-dashes then their average post so that we can get our pitchforks out and chase them away.
LLMs will never get good enough that no one can tell the difference, because the technology is fundamentally incapable of it, nor will it ever completely disappear, because the technology has real use cases that can be run at a massive profit.
Since LLMs are here to stay, what we actually need is for humans to get better at recognising LLM slop, and stop allowing our communication spaces to be rotted by slop articles and slop comments. It's weird that people find this concept objectional. It was historically a given that if a spambot posted a copy-pasted message, the comment would be flagged and removed. Now the spambot comments are randomly generated, and we're okay with it because it appears vaguely-but-not-actually-human-like. That conversations are devolving into this is actually the failure of HN moderation for allowing spambots to proliferate unscathed, rather than the users calling out the most blatantly obvious cases.
Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?
The only spam I see in this chain is the flagged post by electric_muse.
It's actually kind of ironic you bring up copy-paste spam bots. Because people fucking love to copy-paste "ai slop" on every comment and article that uses any punctuation rarer than a period.
> Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?
Yes: the original comment is unequivocally slop that genuinely gives me a headache to read.
It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell.
Humans don't needlessly use a colon in every single sentence they write: abusing punctuation like this is actually really fucking irritating.
Of course, it goes beyond the punctuation: there is zero substance to the actual output, either.
> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.
> Least privilege, token scoping, and proper isolation are friction in the sales process, so they get bolted on later, if at all.
This stupid pattern of LLMs listing off jargon like they're buzzwords does not add to the conversation. Perhaps the usage of jargon lulls people into a false sense of believing that what is being said is deeply meaningful and intelligent. It is not. It is rot for your brain.
"it's not just x, it's y" is an ai pattern and you just said:
>"It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell."
So, I'm actually pretty sure you're just copy-pasting my comments into chatgpt to generate troll-slop replies, and I'd rather not converse with obvious ai slop.
Congratulations, you successfully picked up on a pattern when I was intentionally mimicking the tone of the original spambot content to point out how annoying it was. Why are you incapable of doing this with the original spambot comment?
Cultural acceptance of conversation with AI should've come because of actual AI that are indistinguishable from humans, being forced to swallow recognizable if not blatant LLM slop and turn a blind eye feels unfair
I think this class of problems can be protected against.
It's become clear that the first and most important and most valuable agent, or team of agents, to build is the one that responsibly and diligently lays out the opsec framework for whatever other system you're trying to automate.
A meta-security AI framework, cursor for opsec, would be the best, most valuable general purpose AI tool any company could build, imo. Everything from journalism to law to coding would immediately benefit, and it'd provide invaluable data for post training, reducing the overall problematic behaviors in the underlying models.
Move fast and break things is a lot more valuable if you have a red team mechanism that scales with the product. Who knows how many facepalm level failures like this are out there?
I'm saying that if executives get praise and bonuses for when good things happen, they should also have negative consequences when bad things happen. Litigate that further how you wish.
The legal world has plenty of ways for determining if you are legally responsible for the outcome of an event. Right now the standard is civil punishments for provable negligence.
It sounds like GP is proposing a framework where we tighten up the definition of negligence, and add criminal penalties in addition to civil ones.
So, 1) a public service, 2) with no authentication, 3) and no encryption? (http only??), 4) sent every single response with a token, 5) giving full admin access to every client's legal documents. This is like a law firm with an open back door, open back window, and all the confidential legal papers sprawled out on the floor.
Imagine the potential impact. You're a single mother, fighting for custody of your kids. Your lawyer has some documentation of something that happened to you, that wasn't your fault, but would look bad if brought up in court. Suddenly you receive a phone call - it's a mysterious voice, demanding $10,000 or they will send the documents to the opposition. Neither of them knows each other; someone just found a trove of documents in an open back door and wanted to make a quick buck.
This is exactly what a software building code would address (if we had one!). Just like you can't open a new storefront in a new building without it being inspected, you should not be able to process millions of sensitive files without having your software's building inspected. The safety and privacy of all of us shouldn't be optional.
but google told me everyone can vibe code apps now and software engineers should count their days... it's almost as if there's more stuff we do than just write code...
humans used open s3 buckets stuffed with text files of usernames, passwords, addresses, credit card numbers etc long before vibe coding was a thing.
And those humans would be looking for a new job or face other consequences. An AI model can merrily do this with zero consequences because no meaningful consequences can be visited upon it.
Just like if any human employee publicly sexually harassed his female CEO, he'd be out of a job and would find it very hard to find a new one. But Grok can do it and it's the CEO who ends up quitting.
Prediction: Vibe coding systems will be better at security in 2 years than 90% of devs.
Prediction: it won't.
You can't fit every security consideration into the context window.
90% of human devs are not aware of every security consideration.
90% of human devs can fit more than 3-5 files into their short-term memory.
They also know not to, say, temporarily disable auth to be able to look at the changes they've made on a page hidden behind auth, which is what I observed Gemini 3 Pro doing just yesterday.
Ok, and that’s your prediction for 2 years from now? It’d be quite remarkable if humans had a bigger short term memory than LLMs in 2 years. Or that the kind of dumb security mistakes LLMs make today don’t trigger major, rapid improvements.
Do you understand what the term "context window" means? Have you ever tried using an LLM to program anything even remotely complex? Have you observed how the quality of the output drastically reduces the longer the coversation gets?
That's what makes it bad at security. It cannot comprehend more than a floppy drive worth of data before it reverts to absolute gibberish.
You may want to read about agentic AI, you can for instance call an LLM multiple times with different security consideration everytime.
There's about a dozen workarounds around context limits, agents being one of them, MCP servers being another one, AGENTS.md being the third one, but none of them actually solve the issue of a context window being so small that it's useless for anything even remotely complex.
Let's imagine a codebase that can fit onto a revolutionary piece of technology known as a floppy drive. As we all know, a floppy drive can store <2 megabytes of storage. But a 100k tokens is only about 400 kilobytes. So, to process the whole codebase that can fit onto a floppy drive, you need 5 agents plus the sixth "parent process" that those 5 agents will report to.
Those five agents can report "no security issues found" in their own little chunk of the codebase to the parent process, and that parent process will still be none the wiser about how those different chunks interact with each other.
In one instance it could not even describe why a test is bad unit test (asserting true is equal to true), which doesn’t even require context or multi file reasoning.
Its almost as if it has additional problems beyond the context limits :)
You may want to try using it, anecdotes often differ from theories, especially when they are being sold to you for profit. It takes maybe a few days to see a pattern of ignoring simple instructions even when context is clean. Or one prompt fixes one issue and causes new issues, rinse and repeat. It requires human guidance in practice.
Strongman: LLMs aren't a tool, they're fuzzy automation.
And what keeps security problems from making it into prod in the real world?
Code review, testing, static and dynamic code scanning, and fuzzing.
Why aren't these things done?
Because there isn't enough people-time and expertise.
So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and dynamic code scanning, and fuzzing.
It seems very unlikely those forms of automation won't be improved in the near future by even the dumbest form of LLMs.
And if you offered CISOs a "pay to scan" service that actually worked cross-language and -platform (in contrast to most "only supported languages" scanners), they'd jump at it.
There is an argument here that the LLM is a tool that can multiply the addition or removal of the defects depending on how it is wielded.
I think the father figure of a developer who was bitten by a radioactive spider once made a similar quip.
And that buys you what, exactly? Your point is 100% correct and why LLMs are no where near able to manage / build complete simple systems and surely not complex ones.
Why? Context. LLMs, today, go off the rails fairly easily. As I've mentioned in prior comments I've been working a lot with different models and agentic coding systems. When a code base starts to approach 5k lines (building the entire codebase with an agent) things start to get very rough. First of all, the agent cannot wrap it's context (it has no brain) around the code in a complete way. Even when everything is very well documented as part of the build and outlined so the LLM has indicators of where to pull in code - it almost always cannot keep schemas, requirements, or patterns in line. I've had instances where APIs that were being developed were to follow a specific schema, should require specific tests and should abide by specific constraints for integration. Almost always, in that relatively small codebase, the agentic system gets something wrong - but because of sycophancy - it gleefully informs me all the work is done and everything is A-OK! The kicker here is that when you show it why / where it's wrong you're continuously in a loop of burning tokens trying to put that train back on the track. LLMs can't be efficient with new(ish) code bases because they're always having to go lookup new documentation and burning through more context beyond what it's targeting to build / update / refactor / etc.
So, sure. You can "call an LLM multiple times". But this is hugely missing the point with how these systems work. Because when you actually start to use them you'll find these issues almost immediately.
To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too. It’s not more reliable than just taking a random person off the street of SF and giving them instructions on what to copy paste without any context. They might also change unrelated things or get sidetracked when they encounter friction. My point is that when you try to compensate by prompting repeatedly, you are just adding more chances for entropy to leak in — so I am agreeing with you.
> To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too.
Spot on. If we look at, historically, "AI" (pre-LLM) the data sets were much more curated, cleaned and labeled. Look at CV, for example. Computer Vision is a prime example of how AI can easily go off the rails with respect to 1) garbage input data 2) biased input data. LLMs have these two as inputs in spades and in vast quantities. Has everyone forgotten about Google's classification of African American people in images [0]? Or, more hilariously - the fix [1]? Most people I talk to who are using LLMs think that the data being strung into these models has been fine tuned, hand picked, etc. In some cases for small models that were explicitly curated, sure. But in the context (no pun) of all the popular frontier models: no way in hell.
The one thing I'm really surprised nobody is talking about is the system prompt. Not in the manner of jailbreaking it or even extracting it. But I can't imagine that these system prompts aren't collecting mass tech debt at this point. I'm sure there's band aid after band aid of simple fixes to nudge the model in ever so different directions based on things that are, ultimately, out of the control of such a large culmination of random data. I can't wait to see how these long term issues crop and and duct taped for the quick fixes these tech behemoths are becoming known for.
[0] https://www.bbc.com/news/technology-33347866 [1] https://www.theguardian.com/technology/2018/jan/12/google-ra...
This will age badly
That’s why we make concrete measurable predictions.
Agreed, but "vibe coding will be better at security" is not one of them. Better by which metric, against which threat model, with which stakes? What security even means for greenfield projects is inherently different than for hardened systems. Vibe coding is sufficient for security today because it's not used for anything that matters.
It'll play a role in both securing and security research I'm sure, but I'm not confident it'll be better.
But also, you'd need to have some metrics - how good are developers at security already? What if the bar is on the floor and LLM code generators are already better?
Only if they work in a fundamentally different manner. We can't solve that problem the way we are building LLMs now.
AFAICT Filevine doesn't use AI programming: https://www.filevine.com/jobs/d64dfff5-e36f-4db6-adac-0fc082... There's no mention of AI there other than writing code to integrate AI pipelines.
I've seen a lot of job ads (Canva) lately that mandate AI use or AI experience, and as an AI company if they wanted that I think they would have put it in the ad.
For the record I think I may be fine with the insincerity of selling AI but not using it!
@grok all software engineers do is mindlessly turn specifications into code in one shot, right?!?
See also: HN told me that regulation is bad and this is why the EU is behind!
> it's almost as if there's more stuff we do than just write code..
Yes, but adding these common sense considerations is actually something LLMs can already do reasonably well.
In 90% of the cases. And if you don't know how to spot that other 10%, you are still screwed, cause someone else will found that (and you don't even need to be an elite black hat to find it).
What’s to say a human would catch this 10% either?
The salary you pay them, typically
Salaries make humans infallible?
No, but it makes them motivated to be thorough. There is no way to motivate a chatbot (to do better or to any end).
But money is a way to motivate the people who created AI to create better AI. Because if it doesn't perform as expected, either people won't use it or they'll turn to a competitor next time they need to do something. And these companies need recurring revenue.
If we're saying the way to ensure competency is to instill fear of not getting money tomorrow as a consequence of failure, then AI companies and humans are on equal footing.
You can run multiple chatbots in parallel. Use different models and different setups.
It's like having multiple people audit your systems. Even if everyone only catches 90%, as long as they don't catch exactly the same 90%, this parallel effort helps.
Humans are pretty good at edge cases.
If you explicitly request it which means you need to know about it.
OpenAI can put that in the system prompt for their CTO-as-a-service once, and then forget about it.
Or you need to guess that it exists, or you need to scan for places it exists.
Clearly not
Basically what happened in the Vastaamo case in Finland [1]. Except of course it wasn't individual phone calls – it was mass extortion of 30,000 people at once via email.
[1] https://en.wikipedia.org/wiki/Vastaamo_data_breach
if I remember correctly the attacker got caught in such a silly way
he wanted to demonstrate that he indeed has the private data. But he fucked up the tar command and it ended up having his username in the directory names, a username he used in other places on the internet
Can't even make the basic auth of the password is password123.
http-only makes it also very easy to sniff for LE if they decide to. This allows them to get knowledge about cases. Like, they could be scanning it with their own AI tool for all we know. In a free country with proper LE, this would neither be legal nor happening. But I am not sure the USA is remaining one, given the leader is a convicted felon with very dubious moral standards.
The problem here however is that they get away with their sloppiness as long as the security researcher who found this is a whitehat, and the regular news won't pick it up. Once regular media pick this news up (and the local ones should), their name is tarnished and they may regret their sloppiness. Which is a good way to ensure they won't make the same mistake. After all, money talks.
All the big tech companies are in the news every week. Everybody knows how bad they are. Their names are tarnished and yet everyone is still using their junk and they face zero repercussions when fucking up. I dont think things in the media would do any harm.
In the news, sure. But negatively? I consider myself included in 'everyone', and I am not using junk from all the big tech companies. More than once, I've successfully quit using certain ones, and Signal has become much more popular in my country ever since Trump II took office. Meta had to change the name of their company (Facebook) since it had such a bad name, and Zuck started a charm offensive.
This is HN. We understood exactly what “exposed … confidential files” meant before reading your overly dramatic scenario. As overdone as it is, it’s not even realistic. A likely single mother is likely tiny potatoes in comparison to deep-pocketed legal firms or large corporations.
The story is an example of the market self-correcting, but out comes this “building code” hobby horse anyway. All a software “building code” will do is ossify certain current practices, not even necessarily the best ones. It will tilt the playing field in favor of large existing players and to the disadvantage of innovative startups.
The model fails to apply in multiple ways. Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software. Local city workers inspecting by the local municipality’s code at least has clear jurisdiction because of where the physical fixed location is. Who will write the “building code”? Who will be the inspectors?
This is HN. Of all places, I’d expect to see this presented as an opportunity for new startups, not calls for slovenly bureaucracy and more coercion. The private market is perfectly capable of performing this function. E&O and professional liability insurers if they don’t already will be soon motivated after seeing lawsuits to demand regular pentests.
The reported incident is a great reminder of caveat emptor.
> Building physical buildings is a much simpler, much less complex process with many fewer degrees of freedom than building software.
I don't...think this is true? Google has no problems shipping complex software projects, their London HQ is years behind schedule and vastly over budget.
Construction is really complex. These can be mega-projects with tens of thousands of people involved, where the consequences of failure are injury or even death. When software failure does have those consequences - things like aviation control software, or medical device firmware - engineers are held to a considerably higher standard.
> The private market is perfectly capable of performing this function
But it's totally not! There are so many examples in the construction space of private markets being wholly unable to perform quality control because there are financial incentives not to.
The reason building codes exist and are enforced by municipalities is because the private market is incapable of doing so.
[dead]
The bigwigs at my company want to build out a document management suite. After talking to VP of technology about requirements I ask about security as well as what the regulatory requirements are and all I get is a blank stare.
I used to think developers had to be supremely incompetent to end up with vulnerabilities like this.
But now I understand it’s not the developers who are incompetent…
There's enough incompetence at all levels to go around.
Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.
Oh, I should have been more careful in my formulation:
There are organisations that are generally competent, and there are places that are less competent. It's not all that uncommon for the whole organisation to be generally incompetent.
The saddest places (for me) are those where almost every individual you talk to seems generally competent, but judging by their output the company might as well be stuffed by idiots. Something in the way they are organised suppresses the competence. (I worked at one such company.)
> Maybe I have just been lucky, but I have not had the displeasure of working with people either tha incompetent or willfully ignorant yet.
It's very important before you start any new job to suss out how competent people and the organisation are. Ideally, you probably want to work for a competent company. But at least you want to know what you are getting into.
There's a bit of luck involved, if you go in blindly, but you can also use skill and elbow-grease to investigate.
> suss out how competent people and the organisation are.
how does one do this, without first having the job and being embedded in there? From the outside, it's near impossible to see these details imho.
Yes, it's hard, and I'm not sure there are general strategies that always work.
It's fundamentally the same problem that the company is trying to solve when they interview you, just the other way 'round.
Some ideas: observe and ask in the interviews and hiring process in general. See what you can find out about the company from friends, contacts and even strangers. Network! Do some online research, too.
Btw, lots of the cliché interview questions ("What are your greatest weaknesses?" etc) actually make decent questions you can ask about the company and team you are about to join.
Something I've found useful is just reaching out to past employees. Usually folks that don't work there anymore will be more transparent. Only challenge is getting someone to respond to you, but you'd be surprised how many folks will talk if you don't come off like you're trying to sell them something or a bot.
[dead]
> Something in the way they are organised suppresses the competence.
It's a natural outcome of authoritarian structures when the people at the top are idiots. When that happens, the whole organization rots.
I’m governed by them
Reeves orders Treasury inquiry over Budget leaks
Chancellor’s policies found their way to the press before she announced them to MPs
https://www.telegraph.co.uk/news/2025/12/03/reeves-orders-tr...
You might want to consider voting with your feet?
"Are We the Baddies?"
Not only does the Peter principle generally show more incompetence the higher up a structure you move, but the outsized influence those positions have make for a very noticeably higher level of “things fucked up by incompetence“ coming from the C suite compared to the rest of the structure.
There’s definitely plenty of incompetence regardless. But I’ve never seen a company where the incompetence was more noteworthy in the cog positions than “leadership”.
Incompetence compounds at an astonishing rate.
I've had the same. Ask them to come up with a ToS and they're like "we'll talk about that in an upcoming meeting" it's been a few years now with nothing.
I'm always a bit surprised how long it can take to triage and fix these pretty glaring security vulnerabilities. October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed. Sure the actual bug ended up being (what I imagine to be) a <1hr fix plus the time for QA testing to make sure it didn't break anything.
Is the issue that people aren't checking their security@ email addresses? People are on holiday? These emails get so much spam it's really hard to separate the noise from the legit signal? I'm genuinely curious.
In my experience, it comes down to project management and organizational structure problems.
Companies hire a "security team" and put them behind the security@ email, then decide they'll figure out how to handle issues later.
When an issue comes in, the security team tries to forward the security issue to the team that owns the project so it can be fixed. This is where complicated org charts and difficult incentive structures can get in the way.
Determining which team actually owns the code containing the bug can be very hard, depending on the company. Many security team people I've worked with were smart, but not software developers by trade. So they start trying to navigate the org chart to figure out who can even fix the issue. This can take weeks of dead-ends and "I'm busy until Tuesday next week at 3:30PM, let's schedule a meeting then" delays.
Even when you find the right team, it can be difficult to get them to schedule the fix. In companies where roadmaps are planned 3 quarters in advance, everyone is focused on their KPIs and other acronyms, and bonuses are paid out according to your ticket velocity and on-time delivery stats (despite PMs telling you they're not), getting a team to pick up the bug and work on it is hard. Again, it can become a wall of "Our next 3 sprints are already full with urgent work from VP so-and-so, but we'll see if we can fit it in after that"
Then legal wants to be involved, too. So before you even respond to reports you have to flag the corporate counsel, who is already busy and doesn't want to hear it right now.
So half or more of the job of the security team becomes navigating corporate bureaucracy and slicing through all of the incentive structures to inject this urgent priority somewhere.
Smart companies recognize this problem and will empower security teams to prioritize urgent things. This can cause another problem where less-than-great security teams start wielding their power to force everyone to work on not-urgent issues that get spammed to the security@ email all day long demanding bug bounties, which burns everyone out. Good security teams will use good judgment, though.
Oh man this is so true. In this sort of org, getting something fixed out-of-band takes a huge political effort (even a critical issue like having your client database exposed to the world).
While there were numerous problems with the big corporate structures I worked in decades ago where everything was done by silos of specialists, there were huge advantages. No matter where there was a security, performance, network, hardware, etc. issue, the internal support infrastructure had the specialist’s pagers and for a problem like this, the people fixing it would have been on a conference call until it was fixed. There was always a team of specialists to diagnose and test fixes, always available developers with the expertise to write fixes if necessary, always ops to monitor and execute things, always a person in charge to make sure it all got done, and everybody knew which department it was and how to reach them 24/7.
Now if you needed to develop something not-urgent that involved, say, the performance department, database department, and your own, hope you’ve got a few months to blow on conference calls and procedure documents.
For that industry it made sense though.
Interesting. Wouldn't the performance department have their fingers in all the pies anyway, too, or how was that handled?
Their job was specifically managing server resource allocation— as an IT role and not a dev role— in a completely standardized environment. Most applications were given a standard allotment of resources, and they only got involved if something was running out of ram, disk access was too slow, or something just seemed to be taking a lot longer than usual. If it seemed to be a network problem, or just a program crash, for example, they were never involved unless troubleshooting indicated it involved them. More often than not, I’d get a phone call telling me the system I was working on seemed to be heavy on the disk access or something, and they had already allotted it more to keep it stable, but I should check to make sure we weren’t doing something stupid.
Now that I think of it, I’ll bet a lot of companies have a system similar to this for their infrastructure… they just outsource it to AWS, Azure, Google, etc. and comparatively fly by the seat of their pants on the dev side. You could only scale that system down so much, I imagine.
Great comment. Very true.
> Many security team people I've worked with were smart, but not software developers by trade.
A lot are people who cannot code at all, cannot administer - they just fill tables and check boxes, maybe from some automated suite. They dont know what http and https is, because they are just paper pushers what is far from real security, but more like security in name only.
And they joined the work since it pays well
A lot of the time it’s less “nobody checked the security inbox” and more “the one person who understands that part of the system is juggling twelve other fires.” Security fixes are often a one-hour patch wrapped in two weeks of internal routing, approvals, and “who even owns this code?” archaeology. Holiday schedules and spam filters don’t help, but organizational entropy is usually the real culprit.
> A lot of the time it’s less “nobody checked the security inbox” and more “the one person who understands that part of the system is juggling twelve other fires.”
At my past employers it was "The VP of such-and-such said we need to ship this feature as our top priority, no exceptions"
I've once had a whole sector of a fintech go down because one DevOps person ignored daily warning emails for three months that an API key was about to expire and needed reset.
And of course nobody remembered the setup, and logging was only accessible by the same person, so figuring out also took weeks.
I'm currently on the other side of this trying to convince management that the maintenance that should have been done 3 years ago needs to get done. They need "justification".
Write a short memo that saying you are very concerned, and describe a range of things that may happen (from "not much" over medium to maximum scare - lawsuits, brand/customer trust destroyed etc.).
Email the memo to a decision maker with the important flag on and CC: another person as a witness.
If you have been saying it for a long time and nobody has taken any action, you may use the word "escalation" as part of the subject line.
If things hit the fan, it will also make sure that what drops from the fan falls on the right people, and not on you.
It could also be someone "practicing good time management."
They have a specific time of day, when they check their email, and they only give 30 minutes to that time, and they check emails from most recent, down.
The email comes in, two hours earlier, and, by the time they check their email, it's been buried under 50 spams, and near-spams; each of which needs to be checked, so they run out of 30 minutes, before they get to it. The next day, by email check time, another 400 spams have been thrown on top.
Think I'm kidding?
Many folks that have worked for large companies (or bureaucracies) have seen exactly this.
The system would be mostly sane, if you could sort by some measure of importance, not just recency.
It's not about fixing it, it's about acknowledging it exists
security@ emails do get a lot of spam. It doesn't get talked about very much unless you're monitoring one yourself, but there's a fairly constant stream of people begging for bug bounty money for things like the Secure flag not being set on a cookie.
That said, in my experience this spam is still a few emails a day at the most, I don't think there's any excuse for not immediately patching something like that. I guess maybe someone's on holiday like you said.
This.
There is so much spam from random people about meaningless issues in our docs. AI has made the problem worse. Determining the meaningful from the meaningless is a full time job.
This is where “managed” bug bounty programs like BugCrowd or HackerOne deliver value: only telling you when there is something real. It can be a full time job to separate the wheat from the chaff. It’s made worse by the incentive of the reporters to make everything sound like a P1 hair-on-fire issue.
[dead]
Half of the emails I used to get in a previous company were pointless issues, some coming from a honey pot.
The other half was people demanding payment.
Training a tech support team of interns to solve all of them would be an enviable hacker or software dev training program.
Use AI for that :)
Not kidding, I bet llm’s are excellent at triaging these reports. Humans, in a corporate setting, are apparently not.
My favorite one is the "We've identified a security hole in your website"... and I always respond quickly that my website is statically generated, nothing dynamic and immutable on cloudflare pages. For some odd reason, I never hear back from them.
Well we have 600 people in the global response center I work at. And the priority issue count is currently 26000. That means its serious enough that its been assigned to some one. There are tens of thousands of unassigned issues cuz the traige teams are swamped. People dont realize as systems get more complex issues increase. They never decrease. And the chimp troupes response has always been a Story - we can handle it.
The security@ inbox has so much junk these days with someone reporting that if you paste alert('hacked') into devtools then it makes the website hacked!
I reckon only 1% of reports are valid.
LLM's can now make a plausible looking exploit report ('there is a use after free bug in your server side implementation of X library which allows shell access to your server if you time these two API calls correctly'), but the LLM has made the whole thing up. That can easily waste hours of an experts time for a total falsehood.
I can completely see why some companies decide it'll be an office-hours-only task to go through all the reports every day.
My favorite was "we can trigger your website to initiate a connection to the server we control". They were running their own mail servers and were creating a new accounts on our website. Of course someone needs to initiate a TCP connection to deliver an email message!
Of course this could be a real vulnerability if it would disclose the real server IP behind cloudflare. This was not the case, we were sending via AWS email gateway
Not every organization prioritizes being able to ship a code change at the drop of a hat. This often requires organizational dedication to heavy automated testing a CI, which small companies often aren't set up to do.
I can't believe that any company takes a month to ship something. Even if they don't have CI, surely they'd prefer to break the app (maybe even completely) than risk all their legal documents exfiltrated.
> I can't believe that any company takes a month to ship something.
Outside of startups and big tech, it's not uncommon to have release cycles that are months long. Especially common if there is any legal or regulatory involvement.
I can only say you havent worked anywhere i have.
I remember heartbleed dropping shortly after a deployment and not being allowed to patch for like ten months because the fix wasn't "validated". This was despite insurers stating this issue could cost coverage and legal getting involved.
What? That's crazy, wow!
It’d be pretty reasonable to take the whole API down in this scenario, and put it back up once it’s patched. They’d lose tons of cash but avoid being liable for extreme amounts of damages.
> October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed
I have unfortunately seen way worse. If it will take more than an hour and the wrong people are in charge of the money, you can go a pretty long time with glaring vulnerabilities.
I call that one of the worrisome outcomes from "Marketing Driven Development" where the business people don't let you do technical debt "Stories" because you REALLY need to do work that justifies their existence in the project.
Another aspect to consider: when you reduce the amount of permission anything has (like here the returned token), you risk breaking something.
In a complex system it can be very hard to understand what will break, if anything. In a less complex system, it can still be hard to understand if the person who knows the security model very well isn't available.
> October 27, 2025 disclosure and November 4, 2025 email confirmation seems like a long time to have their entire client file system exposed
There is always the simple answer, these are lawyers so they are probably scrambling internally to write a response that covers themselves legaly also trying to figure out how fucked they are.
1 week is surprisingly not that slow.
I'm a bit conflicted about what responsible disclosure should be, but in many cases it seems like these conditions hold:
1) the hack is straightforward to do;
2) it can do a lot of damage (get PII or other confidential info in most cases);
3) downtime of the service wouldn't hurt anyone, especially if we compare it to the risk of the damage.
But, instead of insisting on the immediate shutting down of the affected service, we give companies weeks or months to fix the issue while notifying no one in the process and continuing with business as usual.
I've submitted 3 very easy exploits to 3 different companies the past year and, thankfully, they fixed them in about a week every time. Yet, the exploits were trivial (as I'm not good enough to find the hard ones, I admit). Mostly IDORs, like changing id=123456 to id=1 all the way up to id=123455 and seeing a lot medical data that doesn't belong to me. All 3 cases were medical labs because I had to have some tests done and wanted to see how secure my data was.
Sadly, in all 3 cases I had to send a follow-up e-mail after ~1 week, saying that I'll make the exploit public if they don't fix it ASAP. What happened was, again, in all 3 cases, the exploit was fixed within 1-2 days.
If I'd given them a month, I feel they would've fixed the issue after a month. If I'd given then a year - after a year.
And it's not like there aren't 10 different labs in my city. It's not like online access to results is critical, either. You can get a printed result or call them to write them down. Yes, it would be tedious, but more secure.
So I should've said from the beginning something like:
> I found this trivial exploit that gives me access to medical data of thousands of people. If you don't want it public, shut down your online service until you fix it, because it's highly likely someone else figured it out before me. If you don't, I'll make it public and ruin your reputation.
Now, would I make it public if they don't fix it within a few days? Probably not, but I'm not sure. But shutting down their service until the fix is in seems important. If it was some hard-to-do hack chaining several exploits, including a 0-day, it would be likely that I'd be the first one to find it and it wouldn't be found for a while by someone else afterwards. But ID enumerations? Come on.
So does the standard "responsible disclosure", at least in the scenario I've given (easy to do; not critical if the service is shut down), help the affected parties (the customers) or the businesses? Why should I care about a company worth $X losing $Y if it's their fault?
I think in the future I'll anonymously contact companies with way more strict deadlines if their customers (or others) are in serious risk. I'll lose the ability to brag with my real name, but I can live with it.
As to the other comments talking about how spammed their security@ mail is - that's the cost of doing business. It doesn't seem like a valid excuse to me. Security isn't one of hundreds random things a business should care about. It's one of the most important ones. So just assign more people to review your mail. If you can't, why are you handling people's PII?
Don't do this.
I understand you think you are doing the right thing but be aware that by shutting down a medical communication services there's a non-trivial chance someone will die because of slower test results.
Your responsibility is responsible disclosure.
Their responsibility is how to handle it. Don't try to decide that for them.
> I think in the future I'll anonymously contact companies with way more strict deadlines if their customers (or others) are in serious risk. I'll lose the ability to brag with my real name, but I can live with it.
What you're describing is likely a crime. The sad reality is most businesses don't view protection of customers' data as a sacred duty, but simply another of the innumerable risks to be managed in the course of doing business. If they can say "we were working on fixing it!" their asses are likely covered even if someone does leverage the exploit first—and worst-case, they'll just pay a fine and move on.
Precisely - they view security as just one part of many of their business, instead of viewing it as one of the most important parts. They've insured themselves against a breach, so it's not a big deal for them. But it should be.
The more casualties, the more media attention -> the more likely they, and others in their field, will take security more seriously in the future.
If we let them do nothing for a month, they'll eventually fix it, but in the mean time malicious hackers may gain access to the PII. They might not make it public, but sell that PII via black markets. The company may not get the negative publicity it deserves and likely won't learn to fix their systems in time and to adopt adequate security measures. The sale of the PII and the breach itself might become public knowledge months after the fact, while the company has had a chance to grow in the meantime, and make more security mistakes that may be exploited later on.
And yes, I know it may be a crime - that's why I said I'd report it anonymously from now on. But if the company sits on their asses for a month, shouldn't that count as a crime, as well? The current definition of responsible disclosure gives companies too much leeway, in my opinion.
If I knew I operated a service that was trivial to exploit and was hosting people's PII, I'd shut it down until I fixed it. People won't die if I make everything in my power to provide the test results (in my example of medical labs) to doctors and patients via other means, such as via paper or phone. And if people do die, it would be devastating, of course, but it would mean society has put too much trust into a single system without making sure it's not vulnerable to the most basic of attacks. So it would happen sooner or later, anyway. Although I can't imagine someone dying because their doctor had to make a phone call to the lab instead of typing in a URL.
The same argument about people dying due to the disruption of the medical communications system could be made about too-big-to-fail companies that are entrenched into society because a lot of pension funds have invested in them. If the company goes under, the innocent people dependent on the pension fund's finances would suffer. While they would suffer, which would be awful, of course, would the alternative be to not let such companies go bankrupt? Or would it be better for such funds to not rely so much on one specific company in the first place? That is to say, in both cases (security or stocks in general) the reality is that currently people are too dependent on a few singular entities, while they shouldn't be. That has to change, and the change has to begin somewhere.
They took a month to fix this? That’s beyond inexcusable. I can’t imagine how any customer could justify working with them going forward.
Also … shows you what a SOC 2 audit is worth: https://www.filevine.com/news/filevine-proves-industry-leade...
Even the most basic pentest would have caught this.
SOC2 is mainly to check boxes, and forces you to think about a few things. There’s no real / actual audit, and in my experience the pen tests are very much a money grab. You’re paying way too much money for some “pentesting” automated suite to run.
The auditors themselves pretty much only care that you answered all questions, they don’t really care what the answers are and absolutely aren’t going to dig any deeper.
(I’m responsible for the SOC2 audits at our firm)
When I worked for a consulting firm some years back I randomly got put on a project that dealt with payment information. I had never had to deal with payment information before so I was a bit nervous about being compliant. I was pointed to SOC2 compliance which sounded scary. Much to my relief (and surprise), the SOC2 questionnaire was literally just what amounted to a survey monkey form. I answered as truthfully as I could and at the end it just said "congrats you're compliant!" or something to that effect.
I asked my my manager if that's all that was required and he said yes, just make sure you do it again next year. I spent the rest of my time worrying that we missed something. I genuinely didn't believe him until your comment.
Edit: missing sentence.
Once this type of issue gets publicized, does that in anyway affect the certification?
Sometimes scandals affect these things. But it's hard to predict.
Soc2 and most other certifications are akin to the tsa, security theater. After seeing the info sec security space from the inside i can only say that it blows my mind how abhorrent the security space is. Prod db creds in code? A ok. Not using some stupid vendors “pen testing” software on each mr, blasphemy?
Unless im missing something, they replied stating they would look into it and then its totally vague when they patched, with Alex apparently randomly testing later and telling them in a "follow up" that it was fixed.
I dont at all get why there is a paragraph thanking their communication if that is the case.
Probably given the alternative, being ghosted followed by a no-knock FBI raid
It looks like SOC 2 (and the other SOCs) where developed by accountants?
I wouldn't expect them to find any computer problems either to be honest.
The time to fix isn't really important, assuming that they took the system offline in the mean time... but we all know they didn't, because that would cost to much.
Where did it say that they took a month to fix? The hacker just checked in 2 weeks later and it was fixed by that point.
According to the timeline it took more than a week just for Filevine to respond saying they would review and fix the vulnerability. It was 24 days after initial disclosure when he confirmed the fix was in place.
Given that the author describes the company as prompt, communicative and professional, I think it’s fair to assume there was more contact than the four events in the top of the article.
Is there any stricter standard? Should one strive for PCI-DSS even if they are a regular SaaS?
Whatever Google does internally would be a much stricter standard, but I'm not sure they've written it up for outsiders to use, alas.
If they have a billion dollar valuation, this fairly basic (and irresponsible) vulnerability could have cost them a billion dollars. If someone with malice had been in your shoes, in that industry, this probably wouldn't have been recoverable. Imagine a firm's entire client communications and discovery posted online.
They should have given you some money.
Exactly.
They could have sold this to a ransomare group or affiliate for 5-6 figures and then the ransomware group could have exfil'd the data and attempted to extort the company for millions.
Then if they didnt pay and the ransomware group leaked the info to the public, they'd likely have to spend millions on lawsuits and fines anyways.
They should have paid this dude 5-6 figures for this find. It's scenarios like this that lead people to sell these vulns on the gray/black market instead of traditional bug bounty whitehat routes.
They should have given him a LOT of money.
Would you settle for a LOT of free AI generated legal advice? ;)
Who says they didn't give him money?
I reckon he would've mentioned it if he got a bounty, 100% deserves the bag
I work for a finance firm and everyone is wondering why we can store reams of client data with SaaS Company X, but not upload a trust document or tax return to AI SaaS Company Y.
My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it.
This article demonstrates that, but it does sort of beg the question as to why not trust one vs the other when they both promise the same safeguards.
FWIW this company was founded in 2014 and appears to have added LLM-powered features relatively recently: https://www.reuters.com/legal/transactional/legal-tech-compa...
While the FileVine service is indeed a Legal AI tool, I don't see the connection between this particular blunder and AI itself. It sure seems like any company with an inexperienced development team and thoughtless security posture could build a system with the same issues.
Specifically, it does not appear that AI is invoked in any way at the search endpoint - it is clearly piping results from some Box API.
There is none. Filevine is not even an "AI" company. They are a pretty standard SaaS that has some AI features nowadays. But the hive mind needs its food, and AI bad as we all know.
> any company with an inexperienced development team and thoughtless security posture
Point out one (1) "AI product" company that isn't described accurately by that sentence
The question is what reason did you have to trust SaaS Company X in the first place?
Because it's the Cloud and we're told the cloud is better and more secure.
In truth the company forced our hand by pricing us out of the on-premise solution and will do that again with the other on-premise we use, which is set to sunset in five years or so.
Probably has more to do with responsibility outsourcing: if SaaS has security breach AND they tell in the contract that they’re secure, then you’re not responsible. Sure, there may be reputational damage for you, but it’s a gamble with good odds in most cases.
Storing lots of legal data doesn’t seem to be one of these cases though.
I see profits and outsourcing.
Selling an on-premise service requires customer support, engineering, and duplication of effort if you’re pushing to the cloud as well. Then you get the temptations and lock in of cloud-only tooling and an army of certified consultant drones whose resumes really really need time on AWS-doc-solution-2035, so the on premise becomes a constant weight on management.
SaaS and the cloud is great for some things some of the time, but often you’re just staring at the marketing playbook of MS or Amazon come to life like a golem.
SaaS is now a "solved problem"; almost all vendors will try to get SOX/SOC2 compliance (and more for sensitive workloads). Although... its hard to see how these certifications would have prevented something like this :melting_face:.
Does SaaS X/Cloud offer IAM capabilities? Or going further, do they dogfood their own access via the identity and access policies? If so, and you construct your own access policy, you have relative peace of mind.
If SaaS Y just says "Give me your data and it will be secure", that's where it gets suspect.
> My argument is we're in the Wild West with AI and this stuff is being built so fast with so many evolving tools that corners are being cut even when they don't realize it.
The funny thing is that this exploit (from the OP) has nothing to do with AI and could be <insert any SaaS company> that integrates into another service.
It doesn't sound like your firm does any diligence that would actually prevent you from buying a vendor that has security flaws.
using ai vs not-ai as your litmus test is giving you a false sense of security. it's ALL wild west
And nobody seems to pay attention to the fact that modern copiers cache copies on a local disk and if the machines are leased and swapped out the next party that takes possession has access to those copies if nobody bothered to address it.
This was the plot of Grisham's book The Firm in 1991
The first thing that comes to my mind is SOC2 HIPAA and the whole security theater.
I am one of the engineers that had to suffer through countless screenshots and forms to get these because they show that you are compliant and safe. While the real impactful things are ignored
SemiAnalysis made this a base requirement for being appropriately ranked on their ClusterMAX report, telling me it is akin to FAA certifications, and then getting hacked themselves for not enforcing simple security controls.
https://jon4hotaisle.substack.com/i/180360455/anatomy-of-the...
It is crazy how this gets perpetuated in the industry as actually having security value, when in reality, it is just a pay-to-play checkbox.
You have to start somewhere though. Security theater sucks, and it's not like compliance is a silver bullet, but at least it's something. Having been through implementing standards compliance, it did help the company in some areas. Was it perfect? Definitely not. Was it driven by financial goals? Absolutely. It did tighten up some weak spots though.
If the options mainly consist of "trust me bro" vs "we can demonstrate that we put in some effort", the latter seems more preferable, even if it's not perfect.
This is the collision between two cultures that were never meant to share the same data: "move fast and duct-tape APIs together" startup engineering, and "if this leaks we ruin people's lives" legal/medical confidentiality.
What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals. This is a 2010-level bug pattern wrapped in 2025 AI hype. The only truly "AI" part is that centralizing all documents for model training drastically raises the blast radius when you screw up.
The economic incentive is obvious: if your pitch deck is "we'll ingest everything your firm has ever touched and make it searchable/AI-ready", you win deals by saying yes to data access and integrations, not by saying no. Least privilege, token scoping, and proper isolation are friction in the sales process, so they get bolted on later, if at all.
The scary bit is that lawyers are being sold "AI assistant" but what they're actually buying is "unvetted third party root access to your institutional memory". At that point, the interesting question isn't whether there are more bugs like this, it's how many of these systems would survive a serious red-team exercise by anyone more motivated than a curious blogger.
It's a little hilarious.
First, as an organization, do all this cybersecurity theatre, and then create an MCP/LLM wormhole that bypasses it all.
All because non-technical folks wave their hands about AI and not understanding the most fundamental reality about LLM software being fundamentally so different than all the software before it that it becomes an unavoidable black hole.
I'm also a little pleased I used two space analogies, something I can't expect LLMs to do because they have to go large with their language or go home.
My first reaction to the announcement of MCP was that I must be missing something. Surely giving an LLM unlimited access to protected data is going to introduce security holes?
Agree.
It’s assuming and estimating it will behave like other software before it when it’s nothing like the software that came before it.
LLMs today won’t behave like the software we’re used to where 1+1 will equal 2 every time.
It’s like handing access to your sensitive data to a complete stranger and hoping for the best.
Assuming a 101 security program past the quality bar, there are a number of reason why this can still happen at companies.
Summarized as - security is about risk acceptance, not removal. There’s massive business pressure to risk accept AI. Risk acceptance usually means some sort of supplemental control that’s not the ideal but manages. There are very little of these with AI tools however - small vendors, they’re not really service accounts but IMO best way to monitor them probably is that, integrations are easy, eng companies hate devs losing admin of some kind but if you have that random AI on endpoints becomes very likely.
I’m ignoring a lot of nuance but solid sec program blown open by LLM vendors is going to be common, let alone bad sec programs. Many sec teams I think are just waiting for the other shoe to drop for some evidentiary support while managing heavy pressure to go full bore AI integration until then.
You missed risk creation vs reward creation.
And then folks can gasp and faint like goats and pretend they didn’t know.
It reminds me of the time I met an IT manager who dint have an IT background. Outsourced hilarity ensued through sales people who were also non-technical.
What am I missing? Risk acceptance is what you’re referring to - risk creation and reward creation.
Sec lead might have a pretty darn clear idea of an out of whack creation of risk v reward. CEO disagrees. Risk accept and move on.
When you’re technical and eventually realize there’s a business to survive behind the tech skills, this is the stuff you learn how to do.
People “will know” as you say because it’s all documented and professionally escalated.
Nitpick, but wormholes and black holes aren't limited to space! (unless you go with the Rick & Morty definition where "there's literally everything in space")
Not a nit pick at all friend, it is even more rabbit holes to explore.
Maybe this is the key takeaway of GenAI: that some access to data, even partially hallucinated data, is better than the hoops that the security theatre puts in place that prevents average Joe doing their job.
This might just be a golden age for getting access to the data you need for getting the job done.
Next security will catch up and there'll be a good balance between access and control.
Then, as always security goes to far and nobody can get anything done.
It's a tale as old as computer security.
This is not at all what I am saying.
"GenAI" is nothing new. "AI" is just software. It's not intelligent, or alive, or sentient, or aware. People can scifi sentimentalize it if they want.
It might simulate parts of things, hopefully more reliably.
It's however a different category of software which requires management that doesn't exist yet how it should.
Cybersecurity security theatre for me is using a web browser to secure and administer what was previously already done and creating new security holes from a web interface.
Then, bypassing it to allow unmanaged MCP access to internal data moats creating it's own universe of security vulnerabilities, full stop. In a secured and contained environment, using an MCP to access data to unlock insight is one thing.
It doesn't mean dont' use MCPs. It means the AI won't figure out what the user doesn't know about security around securing MCPs which is a far more massive vulnerability because users of AI have delegated their thinking to a statistics formula ("GenAI"), because it is so impressive on the surface, but no one is checking the work to make sure it stays that way. Managing quality however, is improving.
My comment is calling out effectively letting external paths have unadulterated access to your private and corporate data.
Data is the new moat. Not UI/UX/Software.
A wormhole that exposes your data makes it available for someone to put it into their data moat far too commonly, and also for it to be mis-interpretted.
I'm less and less sure that when a billion-dollar company screws up this bad, the right thing to do is privately disclose it and let them fix it. This kind of thing just allows companies to go on taking people's money without facing the consequences of their mistakes.
Does a disclosure like this absolve them of any responsibility? They still violated whatever user privacy act.
It does. Most privacy laws are based on time-from-discovery. If they immediately sprung into action at the moment they were informed and remediated the issue, they're in compliance.
Right, that's the problem. There need to be standards that govern what can ever be released to customers/the public in the first place. When violations of those are discovered, the penalties should be based on time from release, so the longer it was out in the wild, the greater the penalty.
But you can't remove something from the internet once it's there, so once it's released, it's expected that it always will be.
It's also impossible to guarantee a 100% secure infrastructure, no matter how good your product team is.
In the grey is a term of art: "best efforts."
If data is leaking, and it wasn't because hackers bypassed a bunch of safeguards, if it can be shown that you didn't use Best Efforts to secure said data, there is liability.
A charitable way of interpreting "best effort" is that it's similar to what I said: we need standards. But the problems with our notion of "best effort" are:
1. The standards aren't clearly defined (i.e., you must specifically do this).
2. They are defined in terms of efforts rather than effects. It is like saying "every car sold must be made of steel" rather than "every car sold must be capable of withstanding an impact against a concrete wall at 60mph with X amount of deformation, etc." We want the rules to determine what level of threat is protected against, not just what motions the company went through. In the case in the article, it wasn't because hackers bypassed a bunch of safeguards; the company didn't protect against even basic threats.
3. It's not enough to have "liability". That puts the onus on individuals to sue the company for their specific damages. We need criminal penalties that are designed to punish companies (and the individuals who direct them) for the harm they do to society by the overall process of rushing ahead selling things instead of slowing down and being careful. We need large-scale enforcement so that companies actually stop doing these things because the cost of doing them becomes too enormous.
4. Our laws do not adequately take account of the differential power of those who cut corners, and the differential gains reaped. We frequently find small operators on the wrong end of painful lawsuits and onerous criminal penalties, while the biggest companies and wealthiest individuals use their position to avoid consequences. Laws need to explicitly take this into account, lowering the standard of proof for penalties against larger, wealthier, and more powerful companies and individuals, and also making those penalties exponentially higher.
So is that true if they find out when the public does too? It seems that disclosing it privately has some upside (protecting the users) and no downside.
That depends more on what the Privacy Policy is of the service, which you agree to when you sign up and use it
What are you going to do, sue them? The place is literally teeming with lawyers.
What would you suggest the right thing to do would be?
Edit: I agree with you that we shouldnt let companies like this get away with what amounts to a slap on the wrist. But everything else seems irresponsible as well.
I guess if I imagine the ideal world, it would be that you report it to the authorities and they impose penalties on the offender that are large enough that the company winds up significantly worse off than if they had just grown more slowly. In other words the punishment for moving fast and breaking things needs to be bad enough to outweigh the gains of doing so.
In the current world, I dunno. I guess it depends on what the company is. If it's something like a hedge fund or a fossil fuel company I think I'd be fine with some kind of wikileaks-like avenue for exposing it in such a way that it results in the company being totally destroyed.
I am at a loss for words. This wasn't a sophisticated attack.
I'd love to know who filevine uses for penetration testing (which they do, according to their website) because holy shit, how do you miss this? I mean, they list their bug bounty program under a pentesting heading, so I guess it's just nice internet people.
It's inexcusable.
This was my impression after reading the article too. I have no doubt that the team at Filevine attempted to secure their systems and have probably thwarted other attackers, but got their foot stuck in what is an unsophisticated attack. It only takes one chain vulnerability to bring down the site.
Security reminds me of the Anna Karenina principle: All happy families are alike; each unhappy family is unhappy in its own way.
> I am at a loss for words. This wasn't a sophisticated attack.
To be fair, data security breaches seldom are.
Man. Hopefully their remediation steps included a full audit of their Box's account.
One could only imagine that if OP wasn't the first to discover it, people could've generated tons of shared links for all kinds of folders, for instance, which would remain active even if they invalidated the API token.
I've worked in several "agentic" roles this year alone (I'm very poachable lol)
and otherwise well structured engineering orgs have lost their goddamn minds with move fast and break things
because they're worried that OpenAI/Google/Meta/Amazon/Anthropic will release the tool they're working on tomorrow
literally all of them are like this
Old school blue chip type of companies are like this too. They’ve thrown all the process and caution they used to have to the wind so that they can… apply AI to their IT org which isn’t even their core business?
It's so great that they allowed him to publish a technical blog post. I once discovered a big vulnerability in a listed consumer tech company -- exposing users' private messages and also allowing to impersonate any user. The company didn't allow me to write a public blogpost.
"Allow"?
Go on write your blog post. Don't let your dreams be dreams.
Presumably they were paid for finding the bug and inn accepting relinquished their right to blog about it.
Being a sellout is weak and sad.
No, you relinquish the right when you agree to their TOS irrespective of if they pay you.
TOS != law
They will stop letting you use the service. That's the recourse for breaking the TOS.
I don’t want to pay for a lawyer to argue that for me. != law does not equate to ‘won’t come with a cost’.
I say this as someone threatened by a billion dollar company for this very thing.
Up until Van Buren v. United States in 2020, ToS violations were sometimes prosecuted as unauthorized access under the CFAA. I suspect there are other jurisdictions that still do the equivalent to that.
Why is the control of publication in their hands and not in yours? Shouldn’t you be able to do whatever after disclosing it responsibly?
Presumably they'll threaten to sue you and/or file a criminal complaint, which can be pretty hard to deal with depending on the jurisdiction. At that point you'll probably start asking yourself if it's worth publishing a blog post for some internet points.
Yet another reason these disclosures should be anonymous (from the reporting side).
Through that API, the frontend is handed a high-privilege API key for Box. Not only was that a huge blunder on the backend, but it reveals what passes for architecture these days. Should our application's backend speak to the super sensitive file store? No, we should hand over the keys to that to the React app, because it literally did not occur to them that there's anything that physically could be driven by the frontend that shouldn't be.
My apologies to the frontend engineers out there who know what they're doing.
I don't disagree with the sentiment. But let's also be honest. There is a lot of improvement to be made in security software, in terms of ease of use and overcomplicating things.
I worked at Google and then at Meta. Man, the amount of "nonsense" of the ACL system was insane. I write nonsense in quotes because for sure from a security point of view it all made a lot of sense. But there is exactly zero chance that such a system can be used in a less technical company. It took me 4 years to understand how it worked...
So I'll take this as another data point to create a startup that simplifies security... Seems a lot more complicated than AI
> November 20, 2025: I followed up to confirm the patch was in place from my end, and informed them of my intention to write a technical blog post.
Can that company tell you to cease and desist? How does the law work?
Lawyers can and will send cease and desist letters to people whether or not there is any legal basis for it. Often the threat of a lawsuit, even a meritless one, is enough to keep people quiet.
FYI, a "cease and desist" carries the same legal weight as me sending a one-liner saying "Knock it off".
They are strongly worded requests from a legal point of view. The only real message they send is that the sender is serious enough about the issue to have involved a lawyer, unless of course you write it yourself, which is something that literally anyone can do.
If you want to actually force an action, you need a court order of some type.
NB for the actual lawyers: I'm oversimplifying, since they can be used in court to prove that you tried to get the other party to stop, and tried to resolve the issue outside of court.
You'd think with a $1B valuation they could afford a pentest
Given the absurd amount startups I see lately that have the words "healthcare" and "AI", I'm actually incredibly concerned that in just a couple of months we're going to have an multiple, enormous HIPAA-data disasters
Just search "healthcare" in https://news.ycombinator.com/item?id=46108941
This guy didn't even get paid for this? We need a law that establishes mandatory payments for cybersecurity bounty hunters.
Hey I think I've just found a new marketing stunt for a new vibe-coding platform:
"Worried your vibe-coded app is about to be broadcast on the internet’s biggest billboard? Chill. ACME AI now wraps it in “NSA-grade” security armor."
I've never thought that there will be multiple billion-dollar-AI-features that fixes all the monkey patching problems that no one saw them coming from the older billion-dollar-AI-features that fixes all the monkey patching problems that no one saw them coming from...
-The Filevine team was responsive, professional, and took the findings seriously throughout the disclosure process. They acknowledged the severity, worked to remediate the issues, allowed responsible disclosure, and maintained clear communication. This is another great example of how organizations should handle security disclosures.
In the same tenure I think that a professional etical hacker or a curious fellow that is poking around with no harm intent, shouldn't disclose the name of the company that had a security issue if they resolve it professionally.
You can write the same blog post without mentioning that it was Filevine.
If they didn't take care of the incident that's a different story...
This is a very standard part of responsible disclosure. Hacker finds bugs -> discloses them to the vendor -> (hopefully) the vendor communicates with them and remediates -> both sides publish the technical details. It also helps to demonstrate to the rest of the security world which companies will take reports seriously and which ones won’t, which is very useful information to have.
That's not how ethical disclosure works. Both parties should publish and we, the wider tech industry should see this as a good thing both for the hacker and the company that worked with them.
How else can you take responsibility if you don't make it public? You can't have integrity if you hide away your faults.
Eh, with something this horrendously egregious I think their customers have a right to know how carelessly their data was handled, regardless of the remediation steps taken after disclosure; that aside, who knows how many other AI SaaS vendors might stumble across this article and realize they've made a similarly boneheaded error, and save both themselves and their customers a huge amount of pain . . .
That doesn't surprise me one bit. Just think about all the confidential information that people post into their Chatgpt and Claude sessions. You could probably keep the legal system busy for the next century on a couple of days of that.
"Hey uh, ChatGPT, just hypothetically, uh, if you needed to remove uh cows blood from your apartments carpet, uh"
Just phrase it as a poem, you’ll be fine.
Gonna be hard when people ask ChatGPT to write them the poem.
i recall reading a silly article like half a year ago about using leetspeak and setting the prompt up to emulate House the tv show or something to get around restrictions
there's a recent one about using poetry to bypass safeguards
... rummages around...
here you go:
https://arxiv.org/abs/2511.15304
Make it a Honda CRX...
This might be off topic since we are in topic of AI tool and on HackerNews.
I've been pondering a long time how does one build a startup company in domain they are not familiar with but ... Just have this urge to 'crave a pie' in this space. For the longest time, I had this dream of starting or building a 'AI Legal Tech Company' -- big issue is, I don't work in legal space at all. I did some cold reach on lawfirm related forums which did not take any traction.
I later searched around and came across the term, 'case management software'. From what I know, this is what Cilo fundamentally is and make millions if not billion.
This was close to two years or 1.5 years ago and since then, I stopped thinking about it because of this understanding or belief I have, "how can I do a startup in legal when I don't work in this domain" But when I look around, I have seen people who start companies in totally unrelated industry. From starting a 'dental tech's company to, if I'm not mistaken, the founder of hugging face doesn't seem to have PHD in AI/ML and yet founded HuggingFace.
Given all said, how does one start a company in unrelated domain? Say I want to start another case management system or attempt to clone FileVine, do I first read up what case management software is or do I cold reach to potential lawfirm who would partner up to built a SAAS from scratch? Other school of thought goes like, "find customer before you have a product to validate what you want to build", how does this realistically work?
Apologies for the scattered thoughts...
I think if you have no domain expertise or unique insight it will be quite hard to find a real pain point to solve, deliver a winning solution, and have the ability to sell it.
Not impossible, but very hard. And starting a company is hard enough as it is.
So 9/10 times the answer will be to partner with someone who understands the space and pain point, preferably one who has lived it, or find an easier problem to solve.
I would also split the concerns:
1. Compliancy with relevant standards. HIPAA, GDPR, ISO, military, legal, etc. Realistically you're going to outsource this or hire someone who knows how to build it, and then you're going to pay an agency to confirm that you're compliant. You also need to consider whether the incumbent solution is a trust-based solution, like the old "nobody gets fired for buying Intel".
2. Domain expertise is always easier if you have a domain expert. Big companies also outsource market research. They'll go to a firm like GLG, pay for some expert's time or commission a survey.
It seems like table stakes to do some basic research on your own to see what software (or solutions) exist and why everyone uses them, and why competitors failed. That should cost you nothing but time, and maybe expense if you buy some software. In a lot of fields even browsing some forums or Reddit is enough. The difference is if you have a working product that's generic enough to be useful to other domains, but you're not sure. Then you might be able to arrange some sort of quid pro quo like a trial where the partner gets to keep some output/analysis, and you get some real-world testing and feedback.
I think it comes down to, having some insight about the customer need and how you would solve it. Having prior experience in the same domain is helpful but is neither a guarantee nor a blocker, towards having a customer insight (lots of people might work in a domain but have no idea how to improve it; alternatively an outsider might see something that the "domain experts" have been overlooking).
I just randomly happened to read about the story of, some surgeons asking a Formula 1 team to help improve its surgical processes, with spectacular results in the long term... The F1 team had zero medical background, but they assessed the surgical processes and found huge issues with communication and lack of clarity, people reaching over each other to get to tools, or too many people jumping to fix something like a hose coming loose (when you just need 1 person to do that 1 thing). F1 teams were very good at designing hyper efficient and reliable processes to get complex pit stops done extremely quickly, and the surgeons benefitted a lot from those process engineering insights, even though it had nothing specifically to do with medical/surgical domain knowledge.
Reference: https://www.thetimes.com/sport/formula-one/article/professor...
Anyways, back to your main question -- I find that it helps to start small... Are you someone who is good at using analogies to explain concepts in one domain, to a layperson outside that domain? Or even better, to use analogies that would help a domain expert from domain A, to instantly recognize an analogous situation or opportunity in domain B (of which they are not an expert)? I personally have found a lot of benefit, from both being naturally curious about learning/teaching through analogies, finding the act of making analogies to be a fun hobby just because, and also honing it professionally to help me be useful in cross-domain contexts. I think you don't need to blow this up in your head as some big grand mystery with some big secret cheat code to unlock how to be a founder in a domain you're not familiar with -- I think you can start very small, and just practice making analogies with your friends or peers, see if you can find fun ways of explaining things across domains with them (either you explain to them with an analogy, or they explain something to you and you try to analogize it from your POV).
One approach is to partner with someone who is an expert in that space.
"Companies often have a demo environment that is open" - huh?
And... Margolis allowed this open demo environment to connect to their ENTIRE Box drive of millions of super sensitive documents?
HUH???!
Before you get to the terrible security practices of the vendor, you have to place a massive amount of blame on the IT team of Margolis for allowing the above.
No amount of AI hype excuses that kind of professional misjudgement.
I don't think we have enough information to conclude exactly what happened. But my read is the researcher was looking for demo.filevine.com and found margolis.filevine.com instead. The implication is that many other customers may have been vulnerable in the same way.
Ah, I see now that I read too quickly - the "open demo environment" was clearly referencing the idea that the vendor (Filevine) would have a live demo, NOT that each client wanted an open playground demo account that is linked to a subset of their data (which would be utterly insane).
When liability of a corporation and its owners is limited, does it benefit their business to be dilligent in every step or have a mentality of move fast and break things?
"(after looking through minified code, which SUCKS to do)"
Would there be a "pretty printer" or some other "unminifier" for this task
If not, then is minification effectively a form of obfuscation
https://webcrack.netlify.app/ was made exactly for this, it's a really useful tool
I mean... in what world would you send a customers private root key to a web browsing client. Like even if the user was authenticated why would they need this? This sort of secret shouldn't even be in an environment variable or database but stored with encryption at rest. There could easily have been a proxy service between client and box if the purpose is to search or download files. It's very bad, even for a prototype... this researcher deserves a bounty!
Who is Margolis, and are they happy that OP publicly announced accessing all their confidential files?
Clever work by OP. Surely there is automatic prober tool that already hacked this product?
> Who is Margolis, and are they happy that OP publicly announced accessing all their confidential files?
Google tells me they are a NY law firm specializing in Real Estate and Immigration law. There are other firms with Margolis in the name too. Kinda doesn't matter; see below.
I doubt that they are thrilled to have their name involved in this, but that is covered by the US constitution's protections on free press.
I had a look at some of FileVine's output, and I can say I'm not surprised. This is not an organization that prizes engineering at all.
How is following a http request and guessing some variables a "reverse" engineering now?
If I had a dollar for every time I saw something like this or for every time someone exposed a .git with a .env full of secrets
Of course there will be no accountability or punishment.
The post is a disclosure about a vuln which seem to have anything to do with AI.
My thing is, even ingesting the BOK should have been done in phases, to avoid having all your virtual eggs in one basket or nest at any ONE time. Staggering tokens to these compartments would not have cost them anything at all . I always say, whatever convenience you enjoy yourself, will be highly appreciated by bad actors... WHEN, not if.. they get thru.
For all the talk in the blog of how "super-professional" their team was (probably just a courtesy on the side of the author, I don´t think he believes his own words either)... I have noticed using AI to produce some kind of API -OR- use a 3rd party point with integration into frontend, is almost guaranteed to produce code in which the frontend either exposes the API secrets directly in the frontend code (literally injecting it into a variable as string), or if you ask it for authentication, it will build some half-built lazy solution which makes no sense. So I imagine their "super-professional" team built this with AI, blindly trusting, probably even allowing it to commit and merge changes itself because if you are not merging 10K LoC a day with all this "great" technology, what are you even doing, right? It is not super-professional to work effectively with a blindfold on, I´d argue.
Attorneys are ethically obligated to follow very stringent rules to protect their client's confidential information. Having been a practicing litigator for 40+ years, I can confidently state I came across very few attorneys who truly understood their obligations.
Things were easier when I first began practicing in the 1970s. There weren't too many ways confidential materials in our files could be compromised. Leaving my open file spread out on the conference room table while I went to lunch while attorneys arriving for a deposition on my partner's case were one by one seated into the conference room. That's the kind of thing we had to keep an eye on.
But things soon got complicated. Computers. Digital copies of files that didn't disappear into an external site for storage like physical files. Then email. What were our obligations to know what could - and could not - be intercepted while email traveled the internet.
Then most dangerous of all. Digital storage that was outside our physical domain. How could we now know if the cloud vendor had access to our confidential data? Where were the backups stored? How exactly was the data securely compartmentalized by a cloud vendor? Did we need our own IT experts to control the data located on the external cloud? What did the contracts with the cloud vendor say about the fact we were a law firm and that we, as the lawyers responsible for our clients confidential information, needed to know that they - the cloud vendor - understood the legal obligations and that they - the cloud vendor - would hire lawyers to oversee the manner in which the cloud vendor blocked all access to the legal data located on their own servers. And so on and so forth.
I'm no longer in active practice but these issues were a big part of my practice my last few years at a Fortune 500 insurance company that used in-house attorneys nationwide to represent insureds in litigation - and the corporation was in engaged in signing onto a cloud service to hold all of the corporate data - including the legal departments across all 50 states. It was a nightmare. I'm confident it still is.
I'll be honest... I'm not at all surprised that this happened. Purely because it seems like everyone who wants to implement AI just forgot all of the institutional knowledge that cybersecurity has acquired over the last 30-40 years. When you "forget" all of that because you want to rush out something really fast, well, you know what they say: play stupid games, win stupid prizes and all that.
> ... after looking through minified code, which SUCKS to do ...
AI tends to be good at un-minifying code.
Legit question: when working on finding security issues, are there any guidelines on what you can send to LLMs/AI?
Personally, I'd just use common sense and good judgment. At the end of the day, would you want someone to hand your address, and other private data to OpenAI just like that? Probably not. So don't paste customer data into it if you can avoid it.
On the other hand, minified code is literally published by the company. Everyone can see it and do with it as they please. So handing that over to an AI to un-minify is not really your problem, since you're not the developer working on the tool internally.
I got downvoted, so maybe that means someone thinks un-minifying code is not advised for dealing with security issues? But on reflection surely you can just use the 'format code' command in the ide? I am no expert but surely it's ok to use AI to help track down and identify security issues with the usual caveats of 'don't believe it blindly, do your double checking and risk assessing.'
Doesn't Chrome Developer tools automatically un-minify?
"Coding is done", Adam Wolff, Anthropic
> countless data protected by HIPAA
People should really look this law up before they reference it
so the future is software engineer cosplaying an cybersecurity so we can implement BASIC security????
now that's just great hacking
[dead]
[flagged]
Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.
We detached this subthread from https://news.ycombinator.com/item?id=46137863 and marked it off topic.
That comment didn't read like AI generated content to me. It made useful points and explained them well. I would not expect even the best of the current batch of LLMs to produce an argument that coherent.
This sentence in particular seems outside of what an LLM that was fed the linked article might produce:
> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.
The users' comment history does read like generic LLM output. Look at the first lines of different comments:
> Interesting point about Cranelift! I've been following its development for a while, and it seems like there's always something new popping up.
> Interesting point about the color analysis! It kinda reminds me of how album art used to be such a significant part of music culture.
> Interesting point about the ESP32 and music playback! I've been tinkering with similar projects, and it’s wild how much potential these little devices have.
> We used to own tools that made us productive. Now we rent tools that make someone else profitable. Subscriptions are not about recurring value but recurring billing
> Meshtastic is interesting because it's basically "LoRa-first networking" instead of "internet with some radios attached." Most consumer radios are still stuck in the mental model of walkie-talkies, while Meshtastic treats RF as an IP-like transport layer you can script, automate, and extend. That flips the stack:
> This is the collision between two cultures that were never meant to share the same data: "move fast and duct-tape APIs together" startup engineering, and "if this leaks we ruin people's lives" legal/medical confidentiality.
The repeated prefixes (Interesting point about!) and the classic it's-this-not-that LLM pattern are definitely triggering my LLM suspicions.
I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.
Or maybe these are people who learned from a LLM that English is supposed to sound like this if you want to be permitted to communicate a.k.a. "to be taken into consideration"! Which is wrong and also kinda sucks, but also it sucks and is wrong for a kinda non-obvious reason.
Or, bear with me there, maybe things aren't so far downhill yet, these users just learned how English is supposed to sound, from the same place where the LLMs learned how English is supposed to sound! Which is just the Internet.
AI hype is already ridiculous; the whole "are you using an AI to write your posts for you" paranoia is even more absurd. So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere. Just like most non-AI-generated thoughts, except perhaps the one which leads to the fridge.
Or maybe the 2 month old account posting repetitive comments and using the exact patterns common to AI generated comment is, actually, posting LLM generated content.
> So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere.
FYI, spammers love LLM generated posting because it allows them to "season" accounts on sites like Hacker News and Reddit without much effort. Post enough plausible-sounding comments without getting caught and you have another account to use for your upvote army, which is a service you can now sell to desperate marketing people who promised their boss they'd get on the front page of HN. This was already a problem with manual accounts but it took a lot of work to generate the comments and content.
That's the "so what"
Wasn't there some sort of escape hatch for situations like that - for when it becomes impossible to trust the agora?
It would be massively funny if that escape hatch just sort of disappeared while we were looking at something else.
Your point stands, though.
>exact patterns common to AI generated comment
How can there be exact patterns to it?
> I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.
Yes, if this is LLM then it definitely wouldn't be zero-shot. I'm still on the fence myself as I've seen similar writing patterns with Asperger's (specifically what used to be called Asperger's; not general autism spectrum) but those comments don't appear to show any of the other tells to me, so I'm not particularly confident one way or the other.
That's ye olde memetic "immune system" of the "onlygroup" (encapsulated ingroup kept unaware it's just an ingroup). "It don't sound like how we're taught, so we have no idea what it mean or why it there! Go back to Uncanny Valley!"
It's always enlightening to remember where Hans Asperger worked, and under what sociocultural circumstances that absolutely proverbial syndrome was first conceived.
GP evidently has some very subtle sort of expectations as to what authentic human expression must look like, which however seem to extend only as far as things like word choice and word order. (If that's all you ever notice about words, congrats, you're either a replicant or have a bad case of "learned literacy in USA" syndrome.)
This makes me want to point out that neither the means nor the purpose of the kind of communication which GP seems to implicitly expect (from random strangers) are even considered to be a real thing in many places and by many people.
I do happen to find that sort of thing way more coughinterestingcough than the whole "howdy stranger, are you AI or just a pseud" routine that HN posters seem to get such a huge kick out of.
Sure looks like one of the most basic moves of ideological manipulation: how about we solved the Turing Test "the wrong way around" by reducing the tester's ability to tell apart human from machine output, instead of building a more convincing language machine? Yay, expectations subverted! (While, in reality, both happen simultaneously.)
Disclaimer: this post was written by a certified paperclip optimizer.
It's probably a list of bullet points or disjointed sentences fed to the LLM to clean up. Might be a non-English speaker using it to become fluent. I won't criticize it, but it's clearly LLM generated content.
“This comment is AI” is the new “First Post” from /. days. Please stop unless you have evidence or a good explanation.
That was literally the same thought that crossed my mind. I agree wholeheartedly, accusing everything and everyone of being AI is getting old fast. Part of me is happy that the skepticism takes hold quickly, but I don't think it's necessary for everyone to demonstrate that they are a good skeptic.
(and I suspect that plenty of people will remain credulous anyway, AI slop is going to be rough to deal with for the foreseeable future).
Also, an AI comment might have a worthwhile point to be addressed. Pointing out something was written in a new way is not addressing the point.
Spammers use AI comments to build reputation on a fleet of accounts for upvoting purposes.
That may or may not be what's happening with this account, but it's worth flagging accounts that generate a lot of questionable comments. If you look at that account's post history there's a lot of familiar LLM patterns and repeated post fragments.
Yeah, you have a point... the comment - and their other comments, on average - seem to fit quite a specific pattern. It's hard to really draw a line between policing style and actually recognising AI-written content, though.
What makes you think that? it would need some prompt engineering if so since ChatGPT won't write like that (bad capitalization, lazy quoting) unless you ask it to
“Chat, write me a blog article that seems like a lazy human who failed English wrote it”?
What’s worse being accused of an AI post or being defended because your post is so bad that AI wouldn’t have written it?
Well then that's everything.
Ya ur right, it's either LLM generated, LLM enhanced, or the author has been reading so much LLM output that its writing style has rubbed off.
or, they wrote it and asked an LLM to improve the flow
You are right, it's 100% AI written
What? It doesn't read that way to me. It reads like any other comment from the past ~15 years.
The point you raised is both a distraction... And does not engage with the ones it did.
We finally have a blog that no one (yet) has accused of being ai generated, so obviously we just have to start accusing comments of being ai. Can't read for more than 2 seconds on this site without someone yelling "ai!".
For what it's worth, even if the parent comment was directly submitted by chatgpt themselves, your comment brought significantly less value to the conversation.
It's the natural response. AI fans are routinely injecting themselves into every conversation here to somehow talk about AI ("I bet an AI tool would have found the issue faster") and AI is forcing itself onto every product. Comments dissing anything that sounds even remotely like AI is the logical response of someone who is fed up.
Every other headline and conversation having ai is super annoying.
But also, its super annoying to sift through people saying "the word critical was used, this is obviously ai!". not to mention it really fucking sucks when you're the person who wrote something and people start chanting "ai slop! ai slop!". like, how am i going to prove is not AI?
I can't wait until ai gets good enough that no one can tell the difference (or ai completely busts and disappears, although that's unlikely), and we can go back to just commenting about whether something was interesting or educational or whatever instead of analyzing how many em-dashes someone used pre-2020 and extrapolating whether their latest post has 1 more em-dashes then their average post so that we can get our pitchforks out and chase them away.
LLMs will never get good enough that no one can tell the difference, because the technology is fundamentally incapable of it, nor will it ever completely disappear, because the technology has real use cases that can be run at a massive profit.
Since LLMs are here to stay, what we actually need is for humans to get better at recognising LLM slop, and stop allowing our communication spaces to be rotted by slop articles and slop comments. It's weird that people find this concept objectional. It was historically a given that if a spambot posted a copy-pasted message, the comment would be flagged and removed. Now the spambot comments are randomly generated, and we're okay with it because it appears vaguely-but-not-actually-human-like. That conversations are devolving into this is actually the failure of HN moderation for allowing spambots to proliferate unscathed, rather than the users calling out the most blatantly obvious cases.
Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?
The only spam I see in this chain is the flagged post by electric_muse.
It's actually kind of ironic you bring up copy-paste spam bots. Because people fucking love to copy-paste "ai slop" on every comment and article that uses any punctuation rarer than a period.
> Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?
Yes: the original comment is unequivocally slop that genuinely gives me a headache to read.
It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell.
Humans don't needlessly use a colon in every single sentence they write: abusing punctuation like this is actually really fucking irritating.
Of course, it goes beyond the punctuation: there is zero substance to the actual output, either.
> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.
> Least privilege, token scoping, and proper isolation are friction in the sales process, so they get bolted on later, if at all.
This stupid pattern of LLMs listing off jargon like they're buzzwords does not add to the conversation. Perhaps the usage of jargon lulls people into a false sense of believing that what is being said is deeply meaningful and intelligent. It is not. It is rot for your brain.
"it's not just x, it's y" is an ai pattern and you just said:
>"It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell."
So, I'm actually pretty sure you're just copy-pasting my comments into chatgpt to generate troll-slop replies, and I'd rather not converse with obvious ai slop.
Congratulations, you successfully picked up on a pattern when I was intentionally mimicking the tone of the original spambot content to point out how annoying it was. Why are you incapable of doing this with the original spambot comment?
I'm not replying to your slop (well, you know, after this one).
Anyways, if you think something is ai, just flag it instead so I don't need to read the word "slop" for the 114th fucking time today.
Thankfully, this time, it was flagged. But I got sucked in to this absolutely meaningless argument because I lack self control.
Ironically, you were the first person in this thread to use the word "slop". You have become what you hate.
jokes on you, I already hate me, that’s why I spend so much time on HN arguing about nothing
oh shit I’m supposed to be done replying
[dead]
Cultural acceptance of conversation with AI should've come because of actual AI that are indistinguishable from humans, being forced to swallow recognizable if not blatant LLM slop and turn a blind eye feels unfair
the original comment in this chain is not blatant llm slop.
Thank you bearsyankees for keeping us informed.
I think this class of problems can be protected against.
It's become clear that the first and most important and most valuable agent, or team of agents, to build is the one that responsibly and diligently lays out the opsec framework for whatever other system you're trying to automate.
A meta-security AI framework, cursor for opsec, would be the best, most valuable general purpose AI tool any company could build, imo. Everything from journalism to law to coding would immediately benefit, and it'd provide invaluable data for post training, reducing the overall problematic behaviors in the underlying models.
Move fast and break things is a lot more valuable if you have a red team mechanism that scales with the product. Who knows how many facepalm level failures like this are out there?
> I think this class of problems can be protected against.
Of course, it’s called proper software development
The techniques for non-disclosure of confidential materials processed by multi-tenant services are obvious, well-known, and practiced by very few.
And jail time for executives who are responsible for data leaks.
Are you saying executives cannot make mistakes ever (ask because you didn't qualify your statement)?
I'm saying that if executives get praise and bonuses for when good things happen, they should also have negative consequences when bad things happen. Litigate that further how you wish.
The key word in that is "responsible".
The legal world has plenty of ways for determining if you are legally responsible for the outcome of an event. Right now the standard is civil punishments for provable negligence.
It sounds like GP is proposing a framework where we tighten up the definition of negligence, and add criminal penalties in addition to civil ones.
Are you saying the OP was just a single error, effectively an executives typo.
In this case, AI was a red herring.
This was just plain terrible web security.
Legal attacks engineering - font type license fee on japan consumers. Engineering attacks legal - AI info dump in above post.
How does above sound like and what kind of professional write like that?