Sounds like AI is just greasing the wheels of a long established 'grandparent scam'... goes something like this:
1) voice one: young adult calls, sobbing 2) grandparent inquires with a name... "Ben, is that you?" 3) voice one: "Yes grandma, it's me, Ben... I'm in trouble, please don't tell mom 4) voice two: "Hello, I'm attorney..."
My grandmother fell victim to this almost 20 years ago, which only stopped when Western Union refused to let her continue sending wires... she was forced to call her daughter (at which point they just called my brother.)
Our takeaway (at the time)... the voice doesn't even need to be terribly accurate, since the original interaction is brief / somewhat inaudible over the tears. Typically just requires an older vulnerable adult, a lucky strike with the initial setup (e.g. grandparent actually has a grandkid), and a lot of high pressure / duress salesmanship.
I told my parents that I will never ask for money, doesn't matter the situation, even with live video, it's trivial to generate live audio and video nowadays.
I remember my JROTC instructor also running into that and how she said afterwards they have a secret phrase between them two as a way of verifying it's truly them.
A terse, altered "Hello" is all I say. Sometimes I don't say anything. Most humans would wait a few seconds then prompt with "...Hello?", whereas bots tend to hang up after ~2s silence
> Every article published on SmarterArticles is authored and editorially controlled by Tim Green. Artificial intelligence tools are used within a structured and supervised workflow as research and drafting instruments. All arguments, framing decisions, source selections, and final publication choices remain human-directed and under my full responsibility.
There are references at the bottom, but I would have preferred direct links or footnotes within the article. Also, direct quotes are nice. I didn’t notice any glaring AI cliches.
I suspect this is much more common than you're imagining. I think it'd be silly for publishers of any kind, really, to not use AI tools for things like fact checking and so on.
What are legitimate uses for copying someone's voice without permission? I see none. Those scientists are just helping criminals to fully automate scamming and governments to create fake videos.
The problem described in the article is unsolvable, given that a mid-range desktop from a few years ago can easily clone a voice that's convincing enough and there are no guardrails to those. Some silly KYC laws might limit a highschool kid making deepfakes of his crush, but once a model exists it's trivial to spread it around, and for organized groups to get ahold of those. Similar will happen with images, it's just that nobody with any serious money bothered releasing image gen models that compete with gemini or chatgpt -- but it's just a question of time. A year or three, what difference does it really make?
As the cost goes down to near-zero you can scale it up almost infinitely, especially if the profits are high enough to get some smart people working on the problem, which going by the article is already the case ("INTERPOL's finding that AI-enhanced fraud is four and a half times more profitable than the traditional kind"; incidents rose by 26% last year). If AI does succeed on mutilating white collar work enough there will be a large supply of knowledge workers that might just join International Scam Co. rather than have their families go homeless. Drowning man clutching at straw and all.
So if technologically it's impossible to prevent and societally it's impossible to prevent (like the attorney that got pwned same as the grandma), I'm not sure if there exists an answer that isn't worse than the thing it's supposed to prevent. I suppose we'll soon be in a situation where nothing we don't directly perceive in real life is provably true. That journalism and media in general seem to be in a deep crisis of trustworthiness means that you won't even get the benefit of the chain-of-trust as a proxy for whether something is or isn't real.
Ignoring everything happening outside of your immediate surroundings is a choice, and probably even good for people's mental health, but my gut feeling is that it does make humanity as a whole dumber and disempowered. What does corruption matter if nobody cares, or even hears about it? It was AI generated by $current_enemy anyway; nothing to see here, citizen.
> The problem described in the article is unsolvable
Well, not completely unsolvable. But nobody would like the solution.
What all these scams rely on is a way to transfer money in an irrevocable fashion. Restrict that in meaningful ways and you end a lot of the abilities for these scams to operate.
You could, for example, outlaw gift cards as a start. You could force the likes of Western Union to have a holding period before releasing money. Crypto would be hard as any regulation against it is pretty easily circumvented, but you could outlaw crypto currency exchanges (I'd worry less about crypto though as it's pretty hard for grandma to reliably setup).
Could be prevented by more advanced "AI detection", especially on calls from unknown numbers.
It doesn't even have to be based on watermarking. It could be as simple as, "hold on a sec your AI countermeasure was listening and noticed you got this suspicious call, please be aware this may be a scam. Here is what you should do next..."
Yet all of this can be easily defeated with soft language. The basic check "what's the password/verification word" will defeat this every time. This is basically opsec that we taught my grandparents, who were in their 90s. Its doable.
Yes but that's more of a mitigation than prevention. It's an additional step, you have to remember to do it, and under the pressure of the situation you might easily forget to do it.
I don't know about that but finding excuses for the scum of this earth is certainly not a solution.
Take Europe for example: nobody dies of hunger in Europe. And yet there are plenty of thieves. People stealing tens of thousands, hundreds of thousands, millions of EUR aren't doing it to "feed their families".
Think of the situation today. Think of the victims today. Instead of thinking of tomorrow's hypothetical situation where supposedly all the honest fathers out of work would join the crime syndicate, think of today's victims.
Projecting your own insecurity about the future to excuse scummy behavior by the scum of this earth is of no help.
There are people, right now, who have a roof. Who have a family. And who are fucking scums stealing the hard earned money of others because they choosed the easy life of crime.
Zero tolerance for such motherfuckers. I care about the victims and you should too.
Obviously there are people who help themselves to others' money if given the chance no matter the circumstances. But if the circumstances change so that people DO start going hungry or homeless, which is a rather obvious side effect of AI-but-not-AGI maximalism brightly espoused by our overlords sama and amodei of the "I can’t wait to make half the knowledge workers worldwide obsolete" variety, the scale of the problem will obviously get worse, as well as the type of people you can get involved if you’re in the international scam market.
One reasonably effective defense: "Okay, let me call you right back." Yes, there's always the whole "my phone is dead, I borrowed someone else's" or "I'm calling from a jail payphone", so I think it might become common practice to start making authentication phrases or "tell me something only we know".
Another pillar of basic trust that's being eroded on an industrial scale. Sigh.
Article said the imposter in this case claimed her phone had been confiscated.
Fraudsters tend to also plan things such that the impersonated person can't be reached by phone at that time, either by choosing a time when they somehow know they're unavailable (e.g. impersonated person posted on social media they're boarding a plane) or in one case (12 years ago though) my SIL's parent's landline was bombarded with spam calls until they decided to leave the phone off the hook at which point the scammers phoned bank who couldn't reach the parents on their main line, of course this was the bank's problem (and there was probably an inside person facilitating) so they got their money back, but still a major inconvenience for the victim.
Probably the only sure advice is to be exceptionally wary of phone calls with supposed extreme time pressures to send the money now.
It's a very common error that happens in both written and spoken language. I've wondered if it's because weary is kind of "in between" wary and leery, like an incorrect mashup, or something.
> Another pillar of basic trust that's being eroded on an industrial scale.
Remember, trust is like a rainforest: takes a long time to grow, provides a valuable ecosystem essential to human life, but can also be burned down for a quick profit.
For extra security against these text-to-speech model zero-shot clones, you might also want to use made-up gibberish words for which the pronunciation can't be reliably inferred from the spelling
me and my wife made up a word in 2024 for this. the word doesn't exist in any language. we say it to each other all the time. even if i give you the spelling for it, you will say it wrong. i recommend everyone to do something similar. i should do it with my parents too.
I think it's a somewhat South African cultural thing, but when I get calls from businesses or spammers, the first thing the caller tends to say is "Hello, how are you?", which is completely stupid when you're calling someone who wouldn't know who you are, so it tends to immediately make me annoyed that they don't know that they should have introduced themselves first.
As 99% of the time these are spam calls, I used to respond with something like "I'm fine, but who are you / do I know you?", but that was pretty much always inefficient as that might say their name (which from a spammer is useless information), maybe a sales pitch "how much do you spend on x?" or maybe something deliberately misleading about their company and saying something like <major brand name> even though they're some independent sales crowd getting commission selling contracts for them.
Eventually I found that the most effective response is "Sorry. Where are you calling from and what is this in regard to?" which I've found without fail seems to surprise, disarm them and immediately elicit whether the call is a waste of my time. At which point I either become very friendly (because it's a call I'm expecting) or I simply respond with "Sorry, not interested, goodbye." and immediately put down the phone.
I just want the disruption to be as minimal as possible and to not let myself even get an emotional reaction from it, so I don't want to get annoyed at them, never mind wasting time telling them off, besides, I suspect that my ruthlessly efficient getting rid of them without them even having a chance to try their pitch is received as a super cold shoulder, akin to being told to f-off.
our family has had a special 'code word' we have had since the kids were in elementary school. If someone ever needed to pick up our kids from school (they never did) our kids were taught to ask for that word.
This is a good reminder that we should review that, since its been 10 years or so.
This. All of this is a solved problem. It's just not a thing that most families do and do regularly. Code word, insider info, etc. "Oh I am so sorry you got arrested Tommy. Before I wire the $, where did we go on vacation last year?'
The opening example is of a person listening to their daughter’s voice on an unknown number, how would calling them back help? Or am I missing something obvious?
What’s terrible is each time I am forced to call the bank, the more they try to tell me voice ID is secure and want me to provide my voice to authenticate. Never. Did ya’ll never play Uplink? With voice cloning as good as it is now, there’s no way a voice ID is secure enough for authentication.
We (the people) have pushed body cams on almost all law enforcement at this point, which had a noble enough motivation behind it -- but we also have pushed for and have various public disclosure laws (also well intended!) that mean those body cams are torrents of data entering the public sphere if anyone simply asks for it.
Now in the era of AI, this means anyone in the vicinity of an officer has a voice sample in the public domain, plus potentially their image.
Complex issue. I like body cams, I like freedom of information laws, but don't love this particular outcome.
Sounds like AI is just greasing the wheels of a long established 'grandparent scam'... goes something like this:
1) voice one: young adult calls, sobbing 2) grandparent inquires with a name... "Ben, is that you?" 3) voice one: "Yes grandma, it's me, Ben... I'm in trouble, please don't tell mom 4) voice two: "Hello, I'm attorney..."
My grandmother fell victim to this almost 20 years ago, which only stopped when Western Union refused to let her continue sending wires... she was forced to call her daughter (at which point they just called my brother.)
Our takeaway (at the time)... the voice doesn't even need to be terribly accurate, since the original interaction is brief / somewhat inaudible over the tears. Typically just requires an older vulnerable adult, a lucky strike with the initial setup (e.g. grandparent actually has a grandkid), and a lot of high pressure / duress salesmanship.
I told my parents that I will never ask for money, doesn't matter the situation, even with live video, it's trivial to generate live audio and video nowadays.
I hope they got the message.
Fortunately for me, my parents wouldn't be able to get the live video working.
I remember my JROTC instructor also running into that and how she said afterwards they have a secret phrase between them two as a way of verifying it's truly them.
So, you answer your phone to the scam and… now they have your voice too.
Talking on the phone is now an unmitigated liability.
A few years back, I would talk with scammers for a while to waste their time. Now I don't.
LPT: Please have a codeword or phrase that you use with your loved ones so even if the scammers use your voice, they won't know the phrase.
The only solution? Answer the phone in an over the top comedy accent, such as Simpsons characters, or just whatever comes to mind.
A terse, altered "Hello" is all I say. Sometimes I don't say anything. Most humans would wait a few seconds then prompt with "...Hello?", whereas bots tend to hang up after ~2s silence
I actually fucking do this, and have done for a year or two. Sad state of affairs.
This blog is kind of an interesting hybrid:
> Every article published on SmarterArticles is authored and editorially controlled by Tim Green. Artificial intelligence tools are used within a structured and supervised workflow as research and drafting instruments. All arguments, framing decisions, source selections, and final publication choices remain human-directed and under my full responsibility.
There are references at the bottom, but I would have preferred direct links or footnotes within the article. Also, direct quotes are nice. I didn’t notice any glaring AI cliches.
I suspect this is much more common than you're imagining. I think it'd be silly for publishers of any kind, really, to not use AI tools for things like fact checking and so on.
What are legitimate uses for copying someone's voice without permission? I see none. Those scientists are just helping criminals to fully automate scamming and governments to create fake videos.
We all have a safe word in the family just for this issue to identify if it´s the real person or not.
The problem described in the article is unsolvable, given that a mid-range desktop from a few years ago can easily clone a voice that's convincing enough and there are no guardrails to those. Some silly KYC laws might limit a highschool kid making deepfakes of his crush, but once a model exists it's trivial to spread it around, and for organized groups to get ahold of those. Similar will happen with images, it's just that nobody with any serious money bothered releasing image gen models that compete with gemini or chatgpt -- but it's just a question of time. A year or three, what difference does it really make?
As the cost goes down to near-zero you can scale it up almost infinitely, especially if the profits are high enough to get some smart people working on the problem, which going by the article is already the case ("INTERPOL's finding that AI-enhanced fraud is four and a half times more profitable than the traditional kind"; incidents rose by 26% last year). If AI does succeed on mutilating white collar work enough there will be a large supply of knowledge workers that might just join International Scam Co. rather than have their families go homeless. Drowning man clutching at straw and all.
So if technologically it's impossible to prevent and societally it's impossible to prevent (like the attorney that got pwned same as the grandma), I'm not sure if there exists an answer that isn't worse than the thing it's supposed to prevent. I suppose we'll soon be in a situation where nothing we don't directly perceive in real life is provably true. That journalism and media in general seem to be in a deep crisis of trustworthiness means that you won't even get the benefit of the chain-of-trust as a proxy for whether something is or isn't real.
Ignoring everything happening outside of your immediate surroundings is a choice, and probably even good for people's mental health, but my gut feeling is that it does make humanity as a whole dumber and disempowered. What does corruption matter if nobody cares, or even hears about it? It was AI generated by $current_enemy anyway; nothing to see here, citizen.
> The problem described in the article is unsolvable
Well, not completely unsolvable. But nobody would like the solution.
What all these scams rely on is a way to transfer money in an irrevocable fashion. Restrict that in meaningful ways and you end a lot of the abilities for these scams to operate.
You could, for example, outlaw gift cards as a start. You could force the likes of Western Union to have a holding period before releasing money. Crypto would be hard as any regulation against it is pretty easily circumvented, but you could outlaw crypto currency exchanges (I'd worry less about crypto though as it's pretty hard for grandma to reliably setup).
Could be prevented by more advanced "AI detection", especially on calls from unknown numbers.
It doesn't even have to be based on watermarking. It could be as simple as, "hold on a sec your AI countermeasure was listening and noticed you got this suspicious call, please be aware this may be a scam. Here is what you should do next..."
Yet all of this can be easily defeated with soft language. The basic check "what's the password/verification word" will defeat this every time. This is basically opsec that we taught my grandparents, who were in their 90s. Its doable.
So they trick the kid out of the password first by calling them and pretending to be the parent.
Yes but that's more of a mitigation than prevention. It's an additional step, you have to remember to do it, and under the pressure of the situation you might easily forget to do it.
I don't know about that but finding excuses for the scum of this earth is certainly not a solution.
Take Europe for example: nobody dies of hunger in Europe. And yet there are plenty of thieves. People stealing tens of thousands, hundreds of thousands, millions of EUR aren't doing it to "feed their families".
Think of the situation today. Think of the victims today. Instead of thinking of tomorrow's hypothetical situation where supposedly all the honest fathers out of work would join the crime syndicate, think of today's victims.
Projecting your own insecurity about the future to excuse scummy behavior by the scum of this earth is of no help.
There are people, right now, who have a roof. Who have a family. And who are fucking scums stealing the hard earned money of others because they choosed the easy life of crime.
Zero tolerance for such motherfuckers. I care about the victims and you should too.
Not sure I understand the argument.
Obviously there are people who help themselves to others' money if given the chance no matter the circumstances. But if the circumstances change so that people DO start going hungry or homeless, which is a rather obvious side effect of AI-but-not-AGI maximalism brightly espoused by our overlords sama and amodei of the "I can’t wait to make half the knowledge workers worldwide obsolete" variety, the scale of the problem will obviously get worse, as well as the type of people you can get involved if you’re in the international scam market.
> scum of the earth
> fucking scums
> Zero tolerance for such motherfuckers
Who watches the watchers etc.
We will just end up with some jingoist dude that will go after us instead.
Slow reforms to regulate the banking industry with this "identity theft" nonsense...
One reasonably effective defense: "Okay, let me call you right back." Yes, there's always the whole "my phone is dead, I borrowed someone else's" or "I'm calling from a jail payphone", so I think it might become common practice to start making authentication phrases or "tell me something only we know".
Another pillar of basic trust that's being eroded on an industrial scale. Sigh.
In practice this often doesn't work.
Article said the imposter in this case claimed her phone had been confiscated.
Fraudsters tend to also plan things such that the impersonated person can't be reached by phone at that time, either by choosing a time when they somehow know they're unavailable (e.g. impersonated person posted on social media they're boarding a plane) or in one case (12 years ago though) my SIL's parent's landline was bombarded with spam calls until they decided to leave the phone off the hook at which point the scammers phoned bank who couldn't reach the parents on their main line, of course this was the bank's problem (and there was probably an inside person facilitating) so they got their money back, but still a major inconvenience for the victim.
Probably the only sure advice is to be exceptionally wary of phone calls with supposed extreme time pressures to send the money now.
> Probably the only sure advice is to be exceptionally weary of phone calls with supposed extreme time pressures to send the money now.
Quick note: you mean “wary” instead of “weary” there.
It's a very common error that happens in both written and spoken language. I've wondered if it's because weary is kind of "in between" wary and leery, like an incorrect mashup, or something.
Appreciated and fixed. I'm a native English speaker, but I think not a word I often write.
> Another pillar of basic trust that's being eroded on an industrial scale.
Remember, trust is like a rainforest: takes a long time to grow, provides a valuable ecosystem essential to human life, but can also be burned down for a quick profit.
The example in the article says the police took her phone. Then her "attorney" gets on to talk instead.
Yes, having a secret code is probably the right answer. My wife's family always has, but mine doesn't. I suppose we should probably fix that.
For extra security against these text-to-speech model zero-shot clones, you might also want to use made-up gibberish words for which the pronunciation can't be reliably inferred from the spelling
me and my wife made up a word in 2024 for this. the word doesn't exist in any language. we say it to each other all the time. even if i give you the spelling for it, you will say it wrong. i recommend everyone to do something similar. i should do it with my parents too.
I mostly answer unknown calls with monotone "hello" and then wait for their introduction before talking normally.
I think it's a somewhat South African cultural thing, but when I get calls from businesses or spammers, the first thing the caller tends to say is "Hello, how are you?", which is completely stupid when you're calling someone who wouldn't know who you are, so it tends to immediately make me annoyed that they don't know that they should have introduced themselves first.
As 99% of the time these are spam calls, I used to respond with something like "I'm fine, but who are you / do I know you?", but that was pretty much always inefficient as that might say their name (which from a spammer is useless information), maybe a sales pitch "how much do you spend on x?" or maybe something deliberately misleading about their company and saying something like <major brand name> even though they're some independent sales crowd getting commission selling contracts for them.
Eventually I found that the most effective response is "Sorry. Where are you calling from and what is this in regard to?" which I've found without fail seems to surprise, disarm them and immediately elicit whether the call is a waste of my time. At which point I either become very friendly (because it's a call I'm expecting) or I simply respond with "Sorry, not interested, goodbye." and immediately put down the phone.
I just want the disruption to be as minimal as possible and to not let myself even get an emotional reaction from it, so I don't want to get annoyed at them, never mind wasting time telling them off, besides, I suspect that my ruthlessly efficient getting rid of them without them even having a chance to try their pitch is received as a super cold shoulder, akin to being told to f-off.
I mostly just don't answer them unless it seems like something that may be legit.
This is the only way to avoid validating your number for spam lists,
and receiving more.
our family has had a special 'code word' we have had since the kids were in elementary school. If someone ever needed to pick up our kids from school (they never did) our kids were taught to ask for that word.
This is a good reminder that we should review that, since its been 10 years or so.
This. All of this is a solved problem. It's just not a thing that most families do and do regularly. Code word, insider info, etc. "Oh I am so sorry you got arrested Tommy. Before I wire the $, where did we go on vacation last year?'
Family OTP helps against passphrase leakage
The opening example is of a person listening to their daughter’s voice on an unknown number, how would calling them back help? Or am I missing something obvious?
We're gonna need two-way passwords for conversations.
Fun.
What’s terrible is each time I am forced to call the bank, the more they try to tell me voice ID is secure and want me to provide my voice to authenticate. Never. Did ya’ll never play Uplink? With voice cloning as good as it is now, there’s no way a voice ID is secure enough for authentication.
My voice is my passport.
(never heard of anyone actually using that in real life, sounds uttery insane)
Yeah my bank does the same except don’t think there is an opt out.
Kinda crazy
Spam 0, you’ll eventually get some customer service agent
name and shame the bank
rofl, IYKYK
They make you give a voice sample now when you're arrested. You need to do so in order to use the phone.
who is they?
The US government
We (the people) have pushed body cams on almost all law enforcement at this point, which had a noble enough motivation behind it -- but we also have pushed for and have various public disclosure laws (also well intended!) that mean those body cams are torrents of data entering the public sphere if anyone simply asks for it.
Now in the era of AI, this means anyone in the vicinity of an officer has a voice sample in the public domain, plus potentially their image.
Complex issue. I like body cams, I like freedom of information laws, but don't love this particular outcome.
But which US governments? There are thousands of them, and they all have different policies.
Montgomery County in Indiana, specific enough for you?