Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance
-
Wait . . . What?
This new protection system works similar to the child-porn detection systems in use at Facebook, and other social media companies. It works on a database of file hashes and when someone tries to upload an image marked as "revenge porn" in its database, the system will block the upload process.
In order to build a database of "revenge porn" file hashes, Facebook will rely on people uploading nude photos in advance. This process involves sending a nude photo to your own account, via Facebook Messenger, then reporting the photo to Facebook, which will create a hash of the image that will be used to block further uploads of the same photo.
What could possibly go wrong?
https://i.imgur.com/sHj1RR7.pngFiled Under:
-
@el_heffe I suppose the sort of people that would upload revenge porn aren't the sort that are smart enough to evade the block by changing a single pixel to get a different hash.
-
@hardwaregeek said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@el_heffe I suppose the sort of people that would upload revenge porn aren't the sort that are smart enough to evade the block by changing a single pixel to get a different hash.
Depending on the hashing algorithm, this might not work.
-
Obviously, this system would also prevent the photos being uploaded again after they've been reported and deleted once. Self-uploading to block may be a side effect of that.
-
Quick! Someone upload company logos and get them blocked!
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
Microsoft's PhotoDNA
If I read that correctly, the "hash" is basically a low-res gray-scale copy of the image. That would be more resistant to tampering with the image, but far from immune. It seems like it could also be somewhat vulnerable to false positives, although as long as there is an avenue for appeal of blocked photos, a quick glance by a human should be able to resolve that. OTOH, if you are uploading images that are similar enough to revenge porn to be blocked, you probably shouldn't be uploading them in the first place.
-
@hardwaregeek Just for fun, I tried testing out Google's reverse image search a few times to see how I could break it. It was pretty good at finding a picture if I cropped out part of it, but if I mirrored the image (ie flip over X axis) that would defeat it every time.
As long as there's nothing obvious such as text in the image, you can't even tell it's been manipulated. I wonder how well PhotoDNA would deal with that?
-
Also, will people really think, "I just broke up with my boy/girlfriend; I should upload porn of myself so that he/she can't get revenge?" And if your first thought is your ex is likely to post revenge porn, maybe you should be more selective about your lovers.
Edit: Also, it will only work if you have copies of all the pictures he/she has. You can't upload pictures you don't have.
Edit 2 — 4: I grammar good today.
-
@masonwheeler ever heard of TinEye?
-
@hardwaregeek I could see the usefulness if, say, SnapChat had an option to turn on that automatically uploaded the image's hash to Facebook.
It seems dumb that you have to actually upload the image, though, in order to block it. Only the hash should need to be uploaded.
-
@masonwheeler said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
It was pretty good at finding a picture if I cropped out part of it
Yeah, from my read of the PhotoDNA article, it sounds like their "hash" is basically a sequence of numbers corresponding to the pixels in a fuzzy, low-res version of the image. Partial matching should be feasible.
@masonwheeler said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
but if I mirrored the image (ie flip over X axis) that would defeat it every time.
Maybe allow for forward and reverse matching of the "hash" sequence. Seems like an obvious enhancement of the algorithm, if it isn't already a feature.
-
@hardwaregeek said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@masonwheeler said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
It was pretty good at finding a picture if I cropped out part of it
Yeah, from my read of the PhotoDNA article, it sounds like their "hash" is basically a sequence of numbers corresponding to the pixels in a fuzzy, low-res version of the image. Partial matching should be feasible.
It's a bit more complicated:
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
It's a bit more complicated
Ok, so an "intensity derivative," whatever that is — could be just the average, or something more complicated — of each "patch" — basically a low-res pixel — and then a delta of the "intensity derivative" from the previous one, rather than just the "intensity derivative" itself. Probably more resistant to some kinds of image changes, and seems like it could still be used to match flipped images.
-
@hardwaregeek said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
It's a bit more complicated
Ok, so an "intensity derivative," whatever that is — could be just the average, or something more complicated — of each "patch" — basically a low-res pixel — and then a delta of the "intensity derivative" from the previous one, rather than just the "intensity derivative" itself. Probably more resistant to some kinds of image changes, and seems like it could still be used to match flipped images.
An intensity derivative is basically the numerical change in intensity. They're calculating the difference in intensity for each pixel in relation to it's four neighbours.
-
@hardwaregeek said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
Edit: Also, it will only work if you have copies of all the pictures he/she has. You can't upload pictures you don't have.
That was my first thought...
-
@masonwheeler said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@hardwaregeek Just for fun, I tried testing out Google's reverse image search a few times to see how I could break it. It was pretty good at finding a picture if I cropped out part of it, but if I mirrored the image (ie flip over X axis) that would defeat it every time.
The obvious solution to that in this case would be to mirror the image once during the original hashing and record that as well. Or just mirror the image as part of its check.
As long as there's nothing obvious such as text in the image, you can't even tell it's been manipulated. I wonder how well PhotoDNA would deal with that?
This is more likely to be a problem. Think of the stuff people do to get past YouTube's filters for copyrighted videos. There's tons of ways to manipulate photos to fool computers while still having an image that more or less provides what is intended to show.
Doesn't Facebook already have a strict ban on explicit nudity in posts? It has some grey areas where some nipslips or breastfeeding photos are allowed, but I thought in general any nudity on Facebook is banned. Shouldn't they just use machine learning algorithms to simply identify nude photos?
-
I thought Facebook had a voraciously enforced "NO PORN OF ANY SORT" rule and algorithm.
-
@weng said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
I thought Facebook had a voraciously enforced "NO PORN OF ANY SORT" rule and algorithm.
(in the very near future) : Today we are pleased to announce the opening of Facebook for Adults! For the lowlow fee of 25$/month, you can have all the fun of Facebook with none of the censoring!
-
@hardwaregeek said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
Edit 2 — 4: I grammar good today.
X-Thread: Are you working at Yahoo! these days?
-
@dcon Why the hell would anyone want that? The only people using Facebook are old.
-
What I'm wondering: Why is the "upload and then Facebook does the hash" even necessary?
The algorithm is something which can run on any computer thus the upload is not even necessary. You could just as easily just upload the hash. Unless, of course, they don't want the algorithm to become public knowledge. But security through obscurity is not such a great idea.
And to prevent abuse (i.e. someone uploading hashes of pictures not of him-/herself) you could instigate some kind of verification that the account does indeed belong to you (which is something Facebook mandates anyway) and that you make yourself liable if you are found guilty of abuse.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
What I'm wondering: Why is the "upload and then Facebook does the hash" even necessary?
The algorithm is something which can run on any computer thus the upload is not even necessary. You could just as easily just upload the hash. Unless, of course, they don't want the algorithm to become public knowledge. But security through obscurity is not such a great idea.
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
What I'm wondering: Why is the "upload and then Facebook does the hash" even necessary?
The algorithm is something which can run on any computer thus the upload is not even necessary. You could just as easily just upload the hash. Unless, of course, they don't want the algorithm to become public knowledge. But security through obscurity is not such a great idea.
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
That's why I added this part:
And to prevent abuse (i.e. someone uploading hashes of pictures not of him-/herself) you could instigate some kind of verification that the account does indeed belong to you (which is something Facebook mandates anyway) and that you make yourself liable if you are found guilty of abuse.
Also note that they already have the infrastructure to verify accounts
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
So who gets the job of checking the original images to make sure they're legitimate images to ban?
I want that job.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
What I'm wondering: Why is the "upload and then Facebook does the hash" even necessary?
The algorithm is something which can run on any computer thus the upload is not even necessary. You could just as easily just upload the hash. Unless, of course, they don't want the algorithm to become public knowledge. But security through obscurity is not such a great idea.
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
That's why I added this part:
And to prevent abuse (i.e. someone uploading hashes of pictures not of him-/herself) you could instigate some kind of verification that the account does indeed belong to you (which is something Facebook mandates anyway) and that you make yourself liable if you are found guilty of abuse.
Also note that they already have the infrastructure to verify accounts
I don't see how that would actually prevent abuse, though (aside from the deterrent effect, which probably wouldn't matter to the sort of person intent on abusing the feature in the first place). How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
OK, I didn't RTFAs, but I'd assume from the name that the "hash" isn't reversible in any meaningful way.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
They'll have an image to check for legitimacy when and if someone else ever tries to upload a image that matches the hash. At that point, some real person at Facebook is going to have to check the image anyway, so if they decide that the user who blocked the picture is deliberately abusing the feature, they can take appropriate measures to curtail that user's ability to do so.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
What I'm wondering: Why is the "upload and then Facebook does the hash" even necessary?
The algorithm is something which can run on any computer thus the upload is not even necessary. You could just as easily just upload the hash. Unless, of course, they don't want the algorithm to become public knowledge. But security through obscurity is not such a great idea.
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
That's why I added this part:
And to prevent abuse (i.e. someone uploading hashes of pictures not of him-/herself) you could instigate some kind of verification that the account does indeed belong to you (which is something Facebook mandates anyway) and that you make yourself liable if you are found guilty of abuse.
Also note that they already have the infrastructure to verify accounts
I don't see how that would actually prevent abuse, though (aside from the deterrent effect, which probably wouldn't matter to the sort of person intent on abusing the feature in the first place). How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
OK, I didn't RTFAs, but I'd assume from the name that the "hash" isn't reversible in any meaningful way.
Simple: You will have a system for appeals. I mean, collisions can happen anyway and actually storing the images is a bad idea to boot - so Facebook will only have the hash in any case. Because no one on Earth will upload images if they're telling people: "Yeah, we need to have someone look at those pictures first!"
-
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
You need the image to be able to make sure it's actually a legitimate image to ban. Never trust the client and all that.
So who gets the job of checking the original images to make sure they're legitimate images to ban?
I want that job.
I assume it'd be the same people being abused by being employed to look at the rest of the flagged images. There was a thread somewhere where @Lorne-Kates and @fox got into it over this, IIRC, but I can't find it just now.
-
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
They'll have an image to check for legitimacy when and if someone else ever tries to upload a image that matches the hash. At that point, some real person at Facebook is going to have to check the image anyway, so if they decide that the user who blocked the picture is deliberately abusing the feature, they can take appropriate measures to curtail that user's ability to do so.
Right, that's what I'd expect. But if all you have is a hash, how do you know it's not a dumb false positive?
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
They'll have an image to check for legitimacy when and if someone else ever tries to upload a image that matches the hash. At that point, some real person at Facebook is going to have to check the image anyway, so if they decide that the user who blocked the picture is deliberately abusing the feature, they can take appropriate measures to curtail that user's ability to do so.
Right, that's what I'd expect. But if all you have is a hash, how do you know it's not a dumb false positive?
Once is coincidence, twice is enemy action.
Also, you do have the picture of the person who uploaded the picture later.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
They'll have an image to check for legitimacy when and if someone else ever tries to upload a image that matches the hash. At that point, some real person at Facebook is going to have to check the image anyway, so if they decide that the user who blocked the picture is deliberately abusing the feature, they can take appropriate measures to curtail that user's ability to do so.
Right, that's what I'd expect. But if all you have is a hash, how do you know it's not a dumb false positive?
Well, I mean, for starters, you look at the image that the hash matched, and see if it's actually porn. If it is, it should probably be blocked, and the hash kept in the blocked image database.
If it's not, then either the system generated a false positive or it was a true positive to an image that was blocked abusively. There, you play a statistics game: if the system is generating a lot of false positives, those false positives will probably be widespread; true positives, on the other hand, will likely be concentrated to the users who are trying to abuse and/or break the system. So it shouldn't be too difficult for them to tell if certain users are trying to do that.
-
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
So who gets the job of checking the original images to make sure they're legitimate images to ban?
I want that job.I was given moderation rights on Okcupid a few years ago, and based on my experience, you don't want that job. If you want to see hot girls naked, it's much more efficient to go to any number of websites that specialize in that sort of thing. Oh, you meant getting paid for it? Never mind, carry on.
-
@anotherusername Yeah...I dunno...seems like a determined attacker (especially a nation state but even someone with control of a botnet) could find ways to get around that.
But given that we're already past the "Bad Idea" gate IMO, what's a few more?
Another reason to have the image would be an improved hash. Then you can rehash what you have and forget about the older (and crappier) hashes and hashing. Otherwise the options seem to be:
- Locked into the same algorithm forever
- Throw out all of the previously submitted hashes when you improve on what you have
- Do multiple hashes of every image upload to check against each hash version
-
@boomzilla That kind of reasoning would also necessitate the storing of passwords as plaintext.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla That kind of reasoning would also necessitate the storing of passwords as plaintext.
They need those CPU cycles for hashing all the porn now. P1 - make it happen! (and the NSA has better things to do with their CPU cycles)
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
seems like a determined attacker (especially a nation state but even someone with control of a botnet) could find ways to get around that
I doubt that it would be any more prone to abuse than their "report" feature already is.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla That kind of reasoning would also necessitate the storing of passwords as plaintext.
That would make sense if they wanted to prevent anyone from ever typing anyone else's password in a status update or a comment, but the way they're using these hashes and the way they use password hashes are so totally different that I can't see how it could possibly be true.
-
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
seems like a determined attacker (especially a nation state but even someone with control of a botnet) could find ways to get around that
I doubt that it would be any more prone to abuse than their "report" feature already is.
Well, in theory uploading a single hash could hit multiple image uploads in a way that reporting a particular post presumably wouldn't.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@anotherusername said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
How do you prove that the hash wasn't a picture of xirself if all you have is the hash?
They'll have an image to check for legitimacy when and if someone else ever tries to upload a image that matches the hash. At that point, some real person at Facebook is going to have to check the image anyway, so if they decide that the user who blocked the picture is deliberately abusing the feature, they can take appropriate measures to curtail that user's ability to do so.
Right, that's what I'd expect. But if all you have is a hash, how do you know it's not a dumb false positive?
Have someone with the same gender and sexual orientation to look.
Wait... isn't it child porn if it's child porn.
Hmm... someone didn't think things through.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla That kind of reasoning would also necessitate the storing of passwords as plaintext.
That would make sense if they wanted to prevent anyone from ever typing anyone else's password in a status update or a comment, but the way they're using these hashes and the way they use password hashes are so totally different that I can't see how it could possibly be true.
Not exactly. I mean, there's a reason why we phased out certain hash generators for passwords over the years.
And before you say: "Yeah, but to make use of password hashes created by weak algorithms they'd need to get a copy of the password table."
That would be exactly my point: Because a leak of a database of (hashed) passwords is one thing. A leaked database of nude pictures the persons depicted wanted to actually prevent from being leaked? That would be a completely new dimension.
-
@boomzilla It should not, unless the images are all the same. If the images "hit" by the hash are all different, then it would indicate that the hash is either a deliberate attempt to disrupt the system or that their system is not as robust as they thought (or both). If they are all the same, then it indicates that the hash is working as designed, and all they have to do is determine whether the image should be blocked.
Given that they've been using it for a while, they probably think it's pretty robust.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
A leaked database of nude pictures the persons depicted wanted to actually prevent from being leaked? That would be a completely new dimension.
hmm... would you keep the original images, or just look at the image that matched the retained hash and delete if it's nude?
Is this not a case where some degree of false positives could be acceptable?
The bonus is that the employees aren't looking at retained images of people who are trying to block images. They're only looking at uploads. So that kinda removes the controversy a little.
-
@xaade said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
A leaked database of nude pictures the persons depicted wanted to actually prevent from being leaked? That would be a completely new dimension.
hmm... would you keep the original images, or just look at the image that matched the retained hash and delete if it's nude?
Is this not a case where some degree of false positives could be acceptable?
That's why I mentioned an appeals system: If you are of the opinion that your picture was blocked wrongly then a human should have a look at it.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
Not exactly. I mean, there's a reason why we phased out certain hash generators for passwords over the years.
Um...yes, exactly. They aren't using password hashes at all like they're planning to use these image hashes.
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
A leaked database of nude pictures the persons depicted wanted to actually prevent from being leaked? That would be a completely new dimension.
Yes, totally. This is a really bad idea in a lot ways.
-
@boomzilla And that's why they are not storing the pictures.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla And that's why they are not storing the pictures.
Yes, which impacts how it all works (now and in the future) in the ways I described.
-
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla And that's why they are not storing the pictures.
Yes, which impacts how it all works (now and in the future) in the ways I described.
And? If the algorithm is robust then there's no problem.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla And that's why they are not storing the pictures.
Yes, which impacts how it all works (now and in the future) in the ways I described.
And? If the algorithm is robust then there's no problem.
Yes.
-
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla And that's why they are not storing the pictures.
But aren't they storing the pictures that people upload?
So, couldn't one get ahold of the database. Leak it. Find the hash that matches and connect the account with the hash to the nude uploaded by someone else.
I mean, they could always have downloaded leaked nudes before, but now they have a way to make matches.
-
@xaade said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@rhywden said in Facebook will Fight Revenge Porn by Letting Potential Victims Upload Nudes in Advance:
@boomzilla And that's why they are not storing the pictures.
But aren't they storing the pictures that people upload?
So, couldn't one get ahold of the database. Leak it. Find the hash that matches and connect the account with the hash to the nude uploaded by someone else.
I mean, they could always have downloaded leaked nudes before, but now they have a way to make matches.
It will be "amusing" to see how many people try to do the upload / report dance but get it wrong and end up getting revenge on themselves.