Imperceptible NLP Attacks

Nicholas Boucher on adversarial examples that can be used to attack text-based models, and the state of homomorphic encryption for machine learning.

Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

Nicholas Boucher is a PhD at Cambridge University where his focus is on security including on topics like homomorphic encryption, voting systems, and adversarial machine learning. He is the lead author of a fascinating new paper – “Bad Characters: Imperceptible NLP Attacks” – which provides a taxonomy of attacks against text-based NLP models, that are  based on Unicode and other encoding systems.  We discussed the key findings in their paper, and we also briefly talked about the state of homomorphic encryption for machine learning and analytics.

Nicholas Boucher:

It started with a conversation I was having with another of the future authors on the paper, and we were talking about how challenging it can be for multilingual speakers to type their language. If those languages for example, don’t use the same set of characters. These different characters, they’re encoded differently. Sometimes you may just choose to type the thing that looks the most similar in English to the language that you’re ultimately writing in, because you don’t want to switch modes on your keyboard or switch over to your other keyboard. We started thinking: “Wait a minute, natural language processing and other areas  of machine learning models that take text as input, they really assume that text is going to look a certain way.” Moments like this are when security people start to get really excited.

Highlights in the video version:

Introduction to Nicolas Boucher
General types of attacks and spam detection
ML Security and attacks on models 
Responsible AI and growth of ML as the discipline grows
What led you to NLP models in particular?
Real world examples and specific tasks
What class of application are we talking about? 
4 types of attacks: Invisible characters, homoglyphs, reorderings, and deletions
Invisible Character Attacks
Invisible characters: Do you find characters not seen in training?
Did you create a testing tools that pointed to an NLP application?
It is conceivable that we will have tools like this in 2-3 years from now?
What are some of the other NLP attacks in your paper that scare you?
Homoglyph Attacks
Reordering Attacks
Deletion Attacks
To what extent should NLP teams be worried about these attacks?
NLP and verticals where text is important
Security practitioners and how they think about breaking into a system
Supply chain problem in all of software
Homomorphic Encryption and Fully Homomorphic Encryption (FHE)
Security domain, security of a system, and social engineering

Related content:

• A video version of this conversation is available on our YouTube channel.
• “Resurgence of Conversational AI”
• Ram Shankar: “Securing machine learning applications”
• Jack Morris: “Improving the robustness of natural language applications”
• Marco Ribeiro: “Testing Natural Language Models”
• Yoav Shoham: “Making Large Language Models Smarter”
• Alan Nichol: “Best practices for building conversational AI applications”
• Lauren Kunze: “How to build state-of-the-art chatbots”

FREE Report:

[Image: Wubi keyboard layout from Wikimedia.]