For several years, I’ve been covering the bizarre phenomenon of “adversarial examples (AKA “adversarial preturbations”), these being often tiny changes to data than can cause machine-learning classifiers to totally misfire: imperceptible squeaks that make speech-to-text systems hallucinate phantom voices; or tiny shifts to a 3D image of a helicopter that makes image-classifiers hallucinate a rifle
A friend of mine who is a very senior cryptographer of longstanding esteem in the field recently changed roles to managing information security for one of the leading machine learning companies: he told me that he thought that it may be that all machine-learning models have lurking adversarial examples and it might be impossible to eliminate these, meaning that any use of machine learning where the owners of the system are trying to do something that someone else wants to prevent might never be secure enough for use in the field — that is, we may never be able to make a self-driving car that can’t be fooled into mistaking a STOP sign for a go-faster sign.
What’s more there are tons of use-cases that seem non-adversarial at first blush, but which have potential adversarial implications further down the line: think of how the machine-learning classifier that reliably diagnoses skin cancer might be fooled by an unethical doctor who wants to generate more billings; or nerfed down by an insurer that wants to avoid paying claims.
My MIT Media Lab colleague Joi Ito (previously) has teamed up with Harvard’s Jonathan Zittrain (previously to teach a course on Applied Ethical and Governance Challenges in AI, and in reading the syllabus, I came across Motivating the Rules of the Game for Adversarial Example Research, a 2018 paper by a team of Princeton and Google researchers, which attempts to formulae a kind of unified framework for talking about and evaluating adversarial examples.
The authors propose a taxonomy of attacks, based on whether the attackers are using “white box” or “black box” approaches to the model (that is, whether they are allowed to know how the model works), whether their tampering has to be imperceptible to humans (think of the stop-sign attack — it works best if a human can’t see that the stop sign has been altered), and other factors.
It’s a fascinating paper that tries to make sense of the to-date scattershot adversarial example research. It may be that my cryptographer friend is right about the inevitability of adversarial examples, but this analytical framework goes a long way to helping us understand where the risks are and which defenses can or can’t work.
If this kind of thing interests you, you can check out the work that MIT Media Lab students are doing with Labsix, a student-only, no-faculty research group that studies adversarial examples.
We should do our best to be clear about the motivation for our work, our definitions, and our game rules. Defenses against restricted perturbation adversarial examples are oftenmotivated by security concerns, but the security motivation of the standard set of gamerules seems much weaker than other possible rule sets. If we make claims that our work improves security, we have a responsibility to understand and attempt to model the threat we are trying to defend against and to study the most realistic rules we are able to study.Studying abstractions of a security problem without corresponding applied work securing real systems makes careful threat modeling more difficult but no less essential.
An appealing alternative for the machine learning community would be to recenter defenses against restricted adversarial perturbations as machine learning contributions and not security contributions. Better articulating non-security motivations for studying the phenomenon of errors caused by small perturbations could highlight why this type of work has recently become so popular among researchers outside the security community. At the end of the day, errors found by adversaries solving an optimization problem are still just errors and are worth removing if possible. To have the largest impact, we should both recast future adversarial example research as a contribution to core machine learning functionality and develop new abstractions that capture realistic threat models.
Motivating the Rules of the Game for Adversarial Example Research [Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen and George E. Dahl/Arxiv]
Security researchers announced at RSAC today announced they have discovered a trove of 809 million personal records exposed on the internet. This time more than just emails and passwords were exposed — data also includes physical addresses, personal mortgage details, social media accounts, and credit score analysis.
Last September, Facebook drew fire for abusing the phone numbers users provided for two-factor authentication messages, sending spam advertising messages over the same channel — now, rather than reforming its ways, Facebook has doubled down on poisoning the security well, by adding a no-opt-out policy of allowing anyone in the world to search for you […]
Hannah Robbins and Scott Brink, two student interns at IBM division X-Force Red set out to study potential vulnerabilities in sign-in reception kiosks, found at many offices and retailers, and discovered 19 bugs in kiosks from industry leaders Jolly Technologies, HID Global, Threshold Security, Envoy, and The Receptionist (the vendors say they have now patched […]
Puzzle fanatics, take note. If you’re bored with 3D gimmick constructions or complex panorama puzzles, take a gander at this new, minimalist offering that might change your game. A simple concept executed with fiendish complexity, Clemens Habicht’s Colour Puzzles reduce the challenge of puzzle construction to its most abstract. Do not adjust your monitor – […]
The moving image is a powerful thing, whether it’s telling a story or selling a brand. And with the advent of animation software, what used to be a painstaking process is now easy and available to anyone. All that’s needed is creativity. If you’ve got that end covered, may we suggest CrazyTalk Animator 3 Pro […]
There’s an audience for everything on the web, so if your brand or business hasn’t found it yet, it’s time to get smarter with your marketing. There are nearly as many ways to reach people as there are websites, but the Marketing, SEO & Affiliate Marketing Super Bundle teaches the ones that get successful companies […]