Nick Horton (00:06):
Welcome to everyone to Amherst Conversations. Thank you very much for joining us for today's session. My name is Nick Horton, Beitzel Professor in Technology and Society. I'm pleased to be your moderator for what promises to be a stimulating hour. Let me start with some housekeeping issues. The event will be recorded. So it could be shared with those who are unable to attend. Once the discussion begins, the Q&A feature will be opened up for live submissions. You can locate the Q&A box next to this live stream feed. We live in a world full of data with systems that automatically extract meaning from it, and often make decisions on our behalf. These systems are increasingly autonomous and are making higher stakes decisions every day, but there are many important questions that we need to consider. How do we ensure that these systems are robust and reflect best judgements?
Nick Horton (01:04):
How can we prevent bad actors from influencing these decisions in malicious or unexpected ways? The type of machine learning models that Scott will talk about today are familiar to all of us. Imagine, for example, a spam filter for your email that classifies messages as spam, unwanted unsolicited communications or valid messages. When spam first became an issue, simple filters were developed that worked fairly effectively to classify messages in some cases, by using certain words as keys to let the message pass through. But then the spammers started to retaliate. They started to inject those same keywords into subject lines to trick the classifier. More spam started to get through. Here the consequences eally aren't so severe. If the scammer can game the system. You know, it's relatively modest if there's another unwanted email message in one's inbox, but other examples have far worse implications. In 2019, security researchers at McAfee added two strips of tape to a speed limit sign.
Nick Horton (02:16):
And were able to fool the autonomous driving system in a 2016 Tesla into driving 50 miles per hour over the speed limit. What other weaknesses do these self-driving systems have that could be exploited given the increasingly sophisticated and powerful systems that are now being created and deployed, there's a really important need to rigorously test and ensure that these systems are reliable. I'm pleased to report that these are questions really at the core of a liberal arts education that are being studied by students and faculty at Amherst College. Today's talk and discussion will address the topic of adversarial machine learning, and how faculty students at the college are contributing to both the science and the practice of automated decision-making. Our speaker today is Scott Alfeld, assistant professor of computer science, who teaches topics in AI, machine learning and security ranging from the highly practical to the purely theoretical.
Nick Horton (03:21):
He graduated from the University of Utah and earned both an MS and PhD from the department of computer sciences at the University of Wisconsin, Madison. Alfeld's primary research is at the intersection of machine learning and security. They study settings where an intelligent adversary has limited access to perturbed data that's fed into a learned or learning system. The goal of the research is twofold: to both detect attacks and to build an augment learning systems, to be more robust, to undetected attacks. In addition, they develop methods for inferring properties of the underlying sensors, whether trustworthy or not, and incorporating that knowledge into the data analysis pipeline. Before coming to Amherst in 2017, Scott taught computer science and public speaking and debate professionally in Salt Lake City, Utah, and Madison, Wisconsin. As a volunteer, they gave guest lectures for courses from the Wisconsin Center for Academically Talented Youth, and taught lock sport, recreational lock-picking and related physical security topics through sector 67 in Madison. Just a quick reminder that once the discussion begins, the Q&A feature will be opened up for live submissions. Attendees can locate the Q&A box next to the live stream feed. Scott, thanks again for agreeing to share your wisdom with the community. I'm really looking forward to hearing from you and later moderating the discussion.
Scott Alfeld (04:53):
Thank you, Nick. Thank you all for coming. I want to talk first about what we do as adversarial learning researchers, and then I'll go into some specifics of what my students have done. So just as Nick said, machine learning algorithms are out there and they make decisions and those decisions affect people.
Scott Alfeld (05:12):
So they affect people like Alice here. Now, as Nick mentioned, a lot of these, they vaguely change Alice's life, right.
Scott Alfeld (05:20):
What gets recommended on Amazon or, you know, credit card transactions being fraudulent or not many decisions are made without any human being involved, but more and more decisions that really affect a person are being made. So things like should this person be denied alone, or what should we set this person's bail to be come from these automated decision making systems with no human involved. And as Nick alluded to with the self driving cars, this is becoming more and more of a problem, right? We are putting engines behind AI that can then, you know, go 50 miles an hour over the limit or whatnot.
Scott Alfeld (05:58):
Now people try to trick these systems. So innocent little Alice here is a hacker. And what she might do is manipulate the data, fed into these systems, so as to trick them into doing what benefits her. We'll see a variety of examples of this. I'll mention this is separate from the sort of traditional hacking, which is more on the system security side of things. So what we do in AML is look at how do we understand these forms of attack and how do we make defenses against them? So if you get nothing else out of this talk, just know that my students and I are making life harder for the bad guys. In addition to that, there are some research aspects to this that by understanding the vulnerability of these learning systems, we're able to understand the learning systems themselves. So I'm going to talk a bit about just terminology so that we're all on the same page where how this machine learning will work is you give me what we're going to call a training set.
Scott Alfeld (06:54):
So you might say, okay, here are a whole bunch of medical records where for each one I've measured a bunch of things about the patients, right? That what's their blood pressure, what's their whatnot. And it comes with a label saying this patient either responded well or responded poorly to say some medication. So I, the machine learner, I take this training such and I produce what's called a model. For our purposes, a model is just a computer program, right? So I'm a machine learner. You give me a bunch of data. I give you a program that you can run. And then what you do, is you use that model to make a prediction. So you say, hey, this new person has come in. I'm going to run all these tests on them. Now I have these features. I don't know if they're going to respond well or poorly to this medication.
Scott Alfeld (07:42):
I haven't given it to them yet. But when I plug all that data into this model, it tells me its best guess as to whether or not it performs, they'll respond well. Now this is a very general setup. It doesn't need to be medical records. It could be something like you give me historical records of say, stock prices. So you say, this is the price on Monday, this is the price on Tuesday, Wednesday, Thursday, and so on. Again, this machine learning produces a model, just a computer program. And then you can go to that program and say, okay, I'm typing in the last 10 days of stock prices. Give me your best guess as to what tomorrow's stock price will be. So I'm going to stick with these two examples, but of course there are many others where machine learning gets involved. So what I'm going to talk about today all is based on this particular pipeline.
Scott Alfeld (08:34):
So I've got some same medical record and some label for that medical record. And then I have a bunch of these and together they make the training set. This has what were called feature vectors of all the things you measured. And then the labels saying, this is what I'd like to predict in the future responds well or responds poorly. This gets fed into a machine learner. Mathematical magic happens and out pops a model. Then we say, okay, great. I've got this model. I've got this new say medical record where I don't know the label, but I have all the measurements, all the features, we'll call this a test point. We pass that into the model, and what it passes out is a prediction. So this prediction says either, I think this person will respond well to the medication, or I think they will respond poorly.
Scott Alfeld (09:30):
This top bit, we're going to call training time. And then this bottom bit we'll call either test time or deployment time. Just keep in mind, there's sort of the training part where it says, here's a whole bunch of data where we know the answers, go do your magic to produce a model. And then there's the deployment part where it says, okay, you're done training, now you've deployed a model. It's fixed, just make predictions. And importantly, remember that this machine learner is a machine, right? This is a computer doing things. It's not a human. So it's, you know, some piece of code is doing this. So in today's talk, I'm going to talk about three projects. The first looks at defense. So we're going to say, suppose I have some model I'm worried about it being attacked. I'm worried about someone manipulating data. So how can I make it more robust. Then I'm going to look at attack.
Scott Alfeld (10:19):
So here it says, if I have some limited access to the training data, if I can make perturbations to it, how can I bend the machine learner to my will? And then we'll look at privacy issues, which says, okay, there's some training set out there. I want to know things about it. You've tried to keep it hidden, but through giving me limited access to your model, maybe I can reverse engineer aspects of that. So, I know that the name of the talk is Attack, Defense, Steal, that had a better ring to it. This makes more sense, pedagogically. So it's the order we're going. So first, this is work I did with Mackenzie. So she did her senior thesis with me. And then shortly afterwards, we turned that into a paper. This is currently under review. If you remember our pipeline here, what Mackenzie's work does is it says, we're going to assume that our attacker is affecting the test point.
Scott Alfeld (11:17):
So imagine that our attacker here is say a doctor who gets paid by some pharmaceutical company. And they say, Oh, I really want my patients to be flagged as responds well to this medication, right? The more people that I'm able to prescribe this medication to, the more money I get. So I'm going to perturb these test points, right? I'm going to perturb the records. Now presumably I can't, you know, say, Oh, this two year old is pregnant, right? I don't have complete control over how I change these medical records. But maybe I can lie about the weight of the patient, a few pounds or something just to change it in my favor. So Mackenzie's work does, is she says, we're going to take the machine learning algorithm and we're going to augment it so that it knows that attacks are coming. It says, when I deploy a model, not all of my data will be clean, some of it will be attacked.
Scott Alfeld (12:11):
So instead, I'm going to learn a more robust model. And this more robust model will say still perform reasonably well. We still want it to make correct predictions as much as possible, but it will be resistant to these fluctuations and data caused by an adversary. So, I'm just going to go through some of the highlights of this project. Back three years ago, some schmuck figured out how to do this optimal defense when you have a very small set of actions. So you can imagine, you have your machine learner and then you get to pick one of like five things. So these things could be, you know, I have five different models to pick from, or I have five different ways of making it harder to attack or whatnot. This work said, okay, here's how you figure out the best one to do. What Mackenzie did is she took that and extended it to a setting where now I have a continuum.
Scott Alfeld (13:10):
So I have an infinite set of possible actions, and that set can be continuous. What this did is it opened the door for saying, well, Hey, my actions can be to change my model to any other model. And so now it says, let me find a model that is robust to attacks. Now it's pretty easy to make a model robust to a text. All you do is say, ignore the data entirely and just return responds well. That's not a great model, right? And there's this sacrifice and accuracy for a gain in robustness. So through a whole bunch of experiments, we found it's worth it. You can get a huge reduction in the vulnerability for a very small reduction in the accuracy. And what's more with Mackenzie's framework, you can tune it depending on what you want. So if you say, well, look, I mean this data, you know, I just asked a thousand people for it.
Speaker 2 (14:05):
And every single one of them has an incentive to lie. I'm really worried about it being attacked. You just turn one knob and Mackenzie's algorithm, and it says, okay, you're, you're more robust and you'll sacrifice some accuracy for it. If you're doing weather prediction and you're collecting your own data and you say, I don't think there's any adversary out there, then fine. You turn that knob the other way. You're now balancing more for accuracy, less for robustness. Now, in order to understand how to defend, we need to understand the attacks and how it works. So as a separate project, this is work I did with Lucas. He also did his senior thesis with me at the same time. And then this led to a publication in AAAI 2019. AAAI is one of the big AI conferences out there. So remember I said, we've got this pipeline.
Scott Alfeld (14:58):
I kind of lied to you. The pipeline is typically more complicated than this. So often what'll happen is there'll be some set of preprocessing steps. So when you get your data, maybe you synthesize some features from it. Maybe you clean it up, maybe you remove outliers. Maybe you normalize the features. You do some form of preprocessing. So you have this original data, it gets preprocessed, and then that turns into your training set, and then come test time. You have some unseen point. You do that same preprocessing and then it becomes your test point. Now, when Lucas started, the bulk of the work out there said the attacker, if they're acting at training time, affects the training set directly, right? So they say, okay, you've preprocessed your data, now I'm going to go in with my hand and muck about what that data. Now what Lucas, his work said let's go before that step.
Speaker 2 (15:54):
So you, as the attacker, you affected the original data, then it gets fed into the preprocessing, and from that point on you don't get to touch it. So in this work, Lucas here is behaving as the attacker. So I'm also gonna show some highlights of his work. One is that his work had this plug and play aspect to it. And what I mean by that is if we look at our pipeline, okay, I'm sorry. I lied again. Typically this preprocessing step is actually several different preprocessing steps. And what he did is say, okay, if you figured out how to attack A, and you figured out how to attack B, and you figured out how to attack C, and you figured out how to attack the learner in the old form of attack, right, where you can affect the training set directly. Well, then you figured out how to attack something that does A then B then C and then that learner, right?
Scott Alfeld (16:45):
You've also figured out how to attack something that just does B and then A and then C or whatever, right? By learning the little pieces of this pipeline and how to attack those, you can just click them together like Legos to figure out the best attack against the entire pipeline. In addition to this plug in play framework, this idea of preprocessing is actually very broad. So you might think of this as, Oh, okay. I'm going to remove some outliers. Maybe if I have my features, I'm going to scale them to be between zero and one, or maybe if I've measured a whole bunch of things, I'm going to convert them all to be, you know, metric units are all to be freedom units or whatever. It can be much more than that. So I'm going to show what's called the Hankel Transform, is an illustration of how these preprocessing steps can actually capture a lot.
Scott Alfeld (17:36):
So suppose we have stock price data. So here, I'm going to say we have some values. What was the price on Monday, Tuesday, Wednesday, Thursday, Friday, and Saturday? And now we want to build a system that says, given the few days, predict the next day. Well, we can phrase this as a data transformation step. So we'll say, we'll construct this matrix, it's called a Hankel Matrix, where we put the Monday here, the Tuesday gets this diagonal line, Wednesday gets this diagonal line, and so on. Then we take that matrix. We're going to chop off the first two columns.
Scott Alfeld (18:15):
So we've got those, and then chop off the last one. And now we say, okay, these are our features, and these are our targets. So if you look at each row across this, it says, well, first use Monday and Tuesday, and try to predict Wednesday. Then use Tuesday and Wednesday and try to predict Thursday. Then take Wednesday, Thursday, and try to predict Friday and so on. So this process of going from the sort of simple, I have a bunch of data points let me learn some predictor, to I have a time series and I want to learn a predictor. He was able to put into his framework as a simple preprocessing step. And so this allowed us to find attacks against these forecasters, right, these time series forecasters. In addition to various other preprocessing steps, what he also did and this made it into his thesis, but it was worked on after the paper is take limitations that the attacker would have.
Scott Alfeld (19:18):
So if you're familiar with channel stuffing attacks, it's this trick where basically I can say, okay, the quarter is almost over and I'm going to get charged a bunch of money based on how much inventory I have. So I'm not going to have any inventory. I'm going to put it all on trucks and send them on a big loop, you know, a hundred miles. So that at the moment that we switched to the new quarter, I have very low inventory, so I pay very low. So various forms of constraints that exist in the real world for what types of attacks you can do, he was able to put that into this same preprocessing framework to say, okay, now we can figure out what real attackers out there are actually capable of doing, which is the first step to figuring out how to defend against that.
Scott Alfeld (20:04):
And lastly, I'm going to talk about some work with Billy. So, Billy also did a senior thesis with me. This was the following year, and he presented some of this work at the AAAI 2020 Undergraduate Consortium. So we've got this pipeline here. I'm going to ignore a couple bits of it. So for this work, ignore the preprocessing step and we can fit it in later, if we want to. And for now ignore actually using your model. So, you know, okay, you've got these test points in these predictions, forget about that. Suppose we just have this framework. So you can imagine this as a setting where Amazon does this, for example, they have a huge amount of data. They pump it through some machine learning, they have a model, and then they give you access to that model, right? They say, Hey, if you give us points, we'll predict them for you.
Speaker 2 (20:53):
Your phone probably does this, too. You give it an image and it highlights where the face is, right? It's simply saying we did the training some time ago. Here's access to the model. Now, of course, I kind of lied to you. This isn't really how the pipeline works. In practice, what would happen is, time would pass and you would get new data. You would say, Oh, Hey look, I've collected new medical records. And I gave them the drug and I've seen whether or not they performed well or not. Right? And then this new data you say, well, look, I have this new data. I might as well add it to my training set. Now I've got this bigger training set. Let me pump that into the machine learner. That'll crank out some new model, and now I have this new model. Then, maybe I'll go ahead and deploy that model.
Scott Alfeld (21:36):
I'll say, Oh, okay, I have this new patient here. Let me, you know, get a prediction. Well, of course time goes on and this continues. So I'd get some additional data. I now have this newer data. So I'm going to take that, put it into my training, set, pump that into the machine learner and get an even newer model. And this process continues. So in Billy's work, he says, okay, if I get to see the newer model, so say I, you know, query these Amazon models and I learn what they're doing every so often over time, I see the updated models. And suppose I see something about the new data, but importantly, I don't see anything about the original training set, but I want to. What Billy's work does is it says, okay, by observing how this learner changes over time or how the learned model changes over time, with this updated training set, what can I infer about the original training set?
Scott Alfeld (22:33):
So this, and I'll mention here, Billy was being the attacker in this. This led to some really cool math. I'm not going to go into the details of it, but what it captured, there were some impossibility results saying for this particular learner, under a bunch of assumptions, there are things you fundamentally cannot learn about the training set. And you can think of this is a whole bunch of machine learning says, Oh, what does it learn? Right when we give it a bunch of data and it does all this math and it produces this model, what did it actually learn? Billy's work answers the question: what did it forget? What aspects of this training set were lost in the process? In addition to that, privacy is really important. So there are security and modifications of privacy. You can imagine if, let's call it the President of Enemyland shoots down an American drone.
Scott Alfeld (23:28):
While on this drone, there might be a learned model that says, you know, given an image classified as military compound or not military compound. But the training data, of course, is not kept on the drone. But if I'm the President of Enemyland, and I have access to this model, I might say, my number two has this aerial photograph. They're the only person in the world who has this aerial photograph. And I really, really want to know if this photograph was used in the training set for this model. If it was, my number two is an unlucky guy, no longer my number two. Right? So insecurity domains, you want to know by releasing access to the model, what are you actually revealing about the training site. Similarly in the medical domain? So you could imagine a setting where I say, Hey, potential employee, thanks for coming and interviewing here.
Scott Alfeld (24:22):
Now I know that you participated in this medical study and I know certain things about you, right? I know your height, your age and whatnot. I really want to know whether or not you have diabetes, say it's a hobby I have. Can I look at some model that was trained on your data as well as other people's data and reverse engineer certain aspects of your medical record in that training set. Right? Typically we want to design our machine learning setup such that if I give you access to the model, I'm not revealing any private information about the elements of my training set. And of course there are many other reasons that people care about privacy. And hopefully I don't need to convince you that privacy is important. I'm going to quickly cover a few things I've done recently, not related to students.
Scott Alfeld (25:15):
Some of the stuff I've looked at is what if there are multiple attackers at play, they might be aligned. They might be competing with one another. And then similarly, what if there are multiple defenders? So you can imagine a setting where I have some message that I would like to send, and I would like to beat your spam filter. I don't really care about your spam filter. I just want to send it to a million people and bypass as many spam filters as I can. So, you know, I'm sending this message out and there are a thousand defenders. I don't care which ones I beat, I I just want to construct the attack to beat as many as possible. And then similarly, these thousand defenders, they know that and they want to tune their systems such that they can best defend in this competitive environment. Then I've also done a bunch of stuff looking at graphs. So I say, if, you know, the attackers are attacking a network, or if agents are interacting on a network where what happens to me affects you in some way, how can we best understand how to defend these, these more complicated systems where the machine learner is just an integrated part of it all.
Scott Alfeld (26:26):
I do other things as well. I'm going to skip most of them. So I do want to thank some people first and foremost, of course my students, this talk would have been incredibly boring without their contributions. I'll mention Mackenzie is now a software design engineer over at Amazon. Lucas is now in Indiana pursuing a master of music and vocal performance. And then Billy is also at Amazon. Especially on the work with Lucas and Billy, I want to thank some collaborators, namely Ara and Ben. And then as far as funding goes, so the summer after graduating, Mackenzie was awarded this fellowship that allowed her to come back for a few weeks over the summer, and that's when we wrote the paper. So thank you to all of you donors present who made that possible. And I want to give a special shout out to Alex and Hillary. I don't know if they're here or not, but if so, hi. They all of these students benefited from the McGeoch Fellowships over the years. And with that, I will open it up to questions.
Nick Horton (27:36):
Great, Scott. Well, a really interesting, fascinating, and at some level, quite disturbing talk in terms of the need to be thinking about these questions of attack, defend and steal, and the many applications of these methods are coming in. I was struck during your presentation about how it really brings together a number of areas, you referred a couple of times to the underlying math and kind of probability that's part of differential privacy, kind of big questions that are kind of now out there in terms of data science. The kind of game theoretic aspect of this also seems, seems really, really relevant in so many ways. And I guess my question would be coming back to Mackenzie's project, how do you balance the different parts and pieces here? You gave us one example where, you know, you were assuming that it was just all bad actors providing data to you, and another example where, you know, you were saying there was not, but, but in the middle, how do you, how does one make decisions and some of those areas?
Scott Alfeld (28:41):
Yeah. So there's this issue of I've been given training data, and then I want to learn a model that predicts future training data. So in essence, I'm learning how the world works, right?
Scott Alfeld (28:59):
The data arose from the world in some way, some phenomenon caused it to be the way it is. Now, if you're the attacker and suppose you have access to 1% of my training data, right? So you can perturb 1% of it. Well, then what I want to do is be robust to that. I want to say, no, no, no, your attacks, they're not going to change what I learned that much. Right? I'm learning the core of nature. Now, if you have access to a hundred percent of my data, well, then you are the world. And the only thing I can do is learn how you create this data, right? And so there's an underlying assumption often made whether implicit or explicit that is big world, small attacker, right? If the attacker is really, you know, if they're controlling sixty percent of the data, well, then who's to say that they're not nature, and that what we call nature before is actually, you know, some separate, random process.
Scott Alfeld (29:51):
What Mackenzie's work did in particular is that 1% becomes tuneable, right? So do we, it's not exactly affecting 1% of the data, but, you know, put a bound on how much you can perturb points, say, on medical records, if you lie about a patient's weight by plus or minus five pounds, that's fine, but you lie by 150 pounds, that's not OK. She's able to tune her system based on how afraid you are of the attacker's strength, but it's always under the assumption that that attacker strength is fairly small.
Nick Horton (30:28):
Great. So some questions are coming in now. Here's one, do two hackers actually perform data manipulation attacks?
Scott Alfeld (30:37):
That's a great question. Yes, we don't know, or at least, I don't know the extent of it. So a simple example of this is suppose I write a book and I sell that book on Amazon and I wanted to sell more.
Scott Alfeld (30:55):
Well, what I can do is I can create a fake Amazon account and then go review that book and give it a very positive review. Right now, if I'm clever, I can go write a script that creates 10,000 Amazon accounts, and then strategically browses Amazon, ending up landing on that book to change what it learns. Right? So this sort of you know, astroturfing, as an example of this attack. Now, how sophisticated these attacks are, there are certainly cases where they're pretty sophisticated. There was a company I'm not gonna say their name, that would do this as a service. So if you had an app that you wanted to sell on the iOS market or the Android market, you could pay this company. And what they would do is they would take your app and attack the market for you so that it would learn to recommend your app more. So the attacks certainly do happen in any situation where there's money involved. So in, you know, stock trading or Amazon sales or whatnot, I'm sure a lot of these attacks are happening. The level of sophistication behind them varies a lot. And it's not really known where the, how sophisticated most of the attacks are.
Nick Horton (32:16):
Great. There's a kind of a question about, about these models and kind of constraints on them. Cause you talked about kind of like you could see the model or you could query it in some way. You know, what kind of results would hold if the learner really had unfettered access to kind of getting a computable function from that model, is that, is that a kind of an interesting scenario and is that kind of realistic in some of these settings?
Scott Alfeld (32:47):
Yeah. So the, how a learner works is it says, this is the space of all possible models I might end up learning. And maybe this is really a simple collection, right, it just says, Oh, okay, whatever, you know, my prediction for tomorrow's stock price is going to be some linear combination of the previous five days, or maybe it's something far more complicated.
Scott Alfeld (33:11):
And then what it does is it selects some model from this space. And there's a question of, if I'm learning a super simple model, it's conceivable that I'm more robust, right? Because you can't make me learn something that's all that complicated. And so if your goal is that, you know, I predict everything perfectly, except for when I see this particular pattern, at which point I predict that it will be a trillion dollars. I simply as the learner don't have the capability of doing that, these extremely complicated models on the other hand, like what's behind self driving cars. They can learn that they can learn all sorts of weird things. And what people have seen in the community is those attacks are super duper susceptible to these forms of attacks. So as you mentioned, Nick, you know, you can put a couple pieces of tape on a stop sign.
Scott Alfeld (34:04):
And I mentioned, you place the tape very strategically on this stop sign. And it tricks a self driving car to see the stop sign misinterpret, the, you know, it takes an image of it, and then it passes that image through a learned model, misinterpreting it to a speed limit 2,000 miles per hour or whatnot sign. So as you get these more and more complicated learners making these more and more complicated models, the susceptibility to attack potentially goes up and it becomes...the attacker in some sense has more power in what they might be able to achieve.
Nick Horton (34:48):
Great. So fairness, transparency and accountability have become more prominent conversations in the machine learning world. We could talk a little bit more about how this plays out in the curriculum at the college later on. But the question from the audience: can machine learning take human bias out of gateway pipeline processes in our society?
Nick Horton (35:12):
And they gave examples, admissions hiring promotion. And then how do you see those aspects playing out in terms of the adversarial modeling work that you're doing?
Scott Alfeld (35:24):
That's a great question. So the, let me make sure we're all on the same page and giving an example of fairness. When people talk about algorithmic fairness or fairness in the context of ML, suppose I have a model that predicts, you know, responds well or responds poorly to this medication and it's 95% accurate. So 95% of the time I correctly guess, and then you have a model which is 94% accurate. Well, you might just say, Oh, Hey, let's go with the 95. That's better. Well, if we drill down into it and we find, say, there are two groups of people, we'll just call them majority and minority, and I get 99% accuracy on the majority and 50 on the minority and you get 94 on the majority and 94 on the minority, in some sense you're more fair, right?
Scott Alfeld (36:18):
Yes. Technically across the entire population I perform better on, you know, on my model is more accurate, but you're more fair. So this is the issue of fairness. And I'll mention it's complicated. We can't just have a system where, Oh, if you're in the majority, you get plugged into my machine learning model, and if you're in the minority, you get plugged into this other one and then everybody's happy. As you can imagine, this is problematic and that you go to a doctor and they say, Oh, yes, here it will test you step through this door. Your friend goes through that door. Trust me, it's just as good, we totally put just as much funding behind that door. So this is the issue of fairness. What adversarial learning does is let us better understand how learners respond to changes in a training set.
Scott Alfeld (37:08):
Right? We do this through the lens of an attacker who says, Oh, I'm trying to change it in this particular way. That benefits me. You can directly apply the methods by saying I'm an attacker. I want your learned model to be more fair. How can I perturb the data in such a way that it becomes, what you learn is more fair? You can also say, all right, well, through our understanding of adversarial learning, how attacks work how we're able to defend against them, can we craft a model or a learner that is more robust to bias being put in the training data. If we can, then that's great, right. And we'll say, okay, this is now more robust in a different sense, right, it's more fair, separate from that on the interoperability side, the adversarial machine learning for mathematical reasons connects fairly directly to interpretability of models.
Scott Alfeld (38:10):
So right now, if I have my self driving car and it sees a stop sign and it says, Hey, that's a 3,000 mile an hour sign. I can't say to the car. Why, why did you, this is clearly a stop sign, what, what, what were you doing here? Or at least if I do say that I get a very hard to interpret answer back. And some of the methods from adversarial learning have helped us ask those and say, can we make this model capable of justifying its decision or at least explaining its decisions? And if so, that'll help us, you know, audit these systems when it comes to issues of, of bias or fairness.
Nick Horton (38:50):
Yeah. There's kind of a follow up question to that. That analogy with the the tape on the traffic signal, and it's kind of thinking about medical databases and you talked a little bit about one of those, those examples. You know, is it usual, or can the training set have hard stops on determining kind of like these features or artifacts, like that your heart rate has to be below something, or as you said that the weights are within some kind of usual ranges, is there any ways to automate that process of ensuring that the training data has those, is that what the defense entails?
Scott Alfeld (39:29):
That's a great question. So the broadly speaking, say, I want to write a paper on some attack, right? I have an idea for, oh, this, this learning method is susceptible to this attack. So I'm going to formulate what the attacker can do, what the defender does, and then I go off and I write a paper for it. Well, what I need to do is find a description of the attacker, which has two aspects to it. One is that it needs to relate at least vaguely—I mean, I am an academic—to reality, and second, it needs to be mathematically friendly enough to where I can actually say, okay, now we can solve the optimization problem. Right? We can figure out what the the best attack is. The hard limits on saying a heart rate, can't be above blah, you know, the various aspects have these sort of are hard cutoffs and strange aspects to them.
Scott Alfeld (40:24):
You know, they might relate to each other in certain ways that makes it very hard to end up with a mathematically friendly problem. So there are heuristic ways of taking a guess and saying, yeah, this attack may not be the best, but it's probably pretty good. And then empirically, we verify that it works well. And on the defense side, there are certainly times where you can say, so I did some work for a company years ago doing inhuman traffic detection. So this was, if you have a webpage and you get traffic to it, you want to know how much of that was from humans and how much of that was from the bots that roam the internet. As a first pass, you can do a lot of things like, Oh, Hey, look, this user clicked on this link a hundred thousand times in a 10th of a second—probably a bot. This, you know, various what you see in the impression of the traffic, you say, Oh, this value is negative.
Scott Alfeld (41:21):
That's clearly about or whatnot. That's a good first pass. But then there are the, you know, clever attacks that are constraining themselves to the valid values, right? The valid inputs to the system. And so that definitely works as a first pass of a defense system, but then you typically need something more beyond that to catch the trickier attackers.
Nick Horton (41:47):
Great. So a more technical question for you: can one aggregate the attack transformations at different steps of the preprocessing pipeline with the same results? Is it, you talked a little bit about separating those are kind of mixing matching, I believe there was a Lego metaphor involved.
Scott Alfeld (42:08):
Yeah. So so, so what was the question about this, is can I aggregate the...
Nick Horton (42:13):
Can you aggregate the, you know, the attack transformations at those different steps?
Scott Alfeld (42:23):
Okay. So how this works is, I have my data is some function of some function of some function of some function of the original data. I take my original data, I pass it through some data transformation preprocessing step. Then I take that, pass it through some other, and so I get this composition of functions. And what Lucas' framework leverages is the chain rule from calculus. So it says if I need to figure out how I affect the...you can kind of, here's a potentially reasonable way of thinking about it. I'm an attacker. I say, this is the attack I'm currently considering. Now I want to figure out how can I change it slightly. If I say, Oh, should I change it in this direction?
Scott Alfeld (43:11):
Well, I want to know how I, how that change affects the attack once it's passed through the entire pipeline. I can figure out how it affects it at each individual step, and then through the chain rule from calculus, that lets me determine how it is, how it affects the entire pipeline.
Nick Horton (43:34):
So just on behalf of my math and stat colleagues, I do appreciate the shout out to the chain rule. So that's the fact that you said it twice was more than we'd agreed to. Question about sample size does your work assume very large quantities of data in those, and are there times when the data set is small but critical, and what impact does that have on, on these three strategies?
Scott Alfeld (43:58):
That's a great question. So my work typically focuses on machine learning methods that work on relatively small amounts of data.
Speaker 2 (44:13):
So there are models out there, if you look at like deep learning models that are methods of, if you look at deep learning, it often takes a huge amount of data to produce a model, and they're able to benefit from that huge amount data and benefit from that huge amount of computational power to produce, you know these really amazing models. These are what's behind the self driving cars that we've been talking about. Much of what I do is closer to the the simpler models that behave on much smaller data sets and use less computation. One of the advantages of this is you're able to analyze them and get stronger guarantees often. It says, this is indeed the optimal attack, there is no attack better than this. And then some of the work I do also says the learner is just a component in this system.
Scott Alfeld (45:16):
It could be one of these simple ones that, you know, we have all these results for. It could be one of these extremely complicated ones and extremely computationally, expensive ones. Either way, what it can just be plugged into my framework and then go from there. But typically I work with the models that are utilizing less data.
Nick Horton (45:41):
Are certain machine learning algorithms, more robust against the kind of reverse engineering and adversarial issues that you've described and others?
Scott Alfeld (45:57):
Yeah, definitely. So the, you know, I mentioned this idea of my machine learner, you know, that's gonna predict tomorrow's stock price always says $15. This is an incredibly robust model. You can perturb the training set, however you want. You can perturb the test set however you want. It does not change, it's fully robust. On an extreme you have, you know, I'm classifying images for my self driving car.
Scott Alfeld (46:26):
If you perturb a few pixels by a little bit, I will dramatically change what I, my prediction is. There is a spectrum between these, right. You can have, Oh, okay, I'm, you know, learning a linear function. I'm in a low dimensional space. I am more, I am harder to trick. There's also a question of the attacker's intentions and the underlying data distribution. So if I'm trying to classify images between cat and dog, and you as the attacker, you know, you have a picture of, I dunno, let's say that I'm doing more fine grain detail of that. You have a picture of a cat and you want me to predict Chihuahua, maybe that's an easy task. You have a picture of a cat and you want me to predict great dane, maybe that's harder. So the robustness depends not just on the learner, but on the learned model and on the attacker's goals and capabilities.
Scott Alfeld (47:35):
So like in Mackenzie's work, what she did is said here is the space of all the models I might learn. How can I take what I would learn, assuming there's no attacker, and shift it to be a different model that has similar accuracy, but much higher robustness, much higher resistance to these attacks.
Nick Horton (47:55):
So you're starting your fourth year at the college. And the question comes up, what will you work on next? What's anything in future we should start to be looking for?
Scott Alfeld (48:06):
So I have changed my plans for this year a bit. I'm on sabbatical this coming year, and some trips got canceled. I think I'm going to do stuff looking at the— One way to think of this as adversarial AI. So there's this whole question of where machine learning and AI land, but basically there are methods of AI that look at less data-driven approaches to learning, perhaps more experience approaches to learning.
Scott Alfeld (48:42):
So like how can I learn to play chess, these sorts of things. And I'm very interested in how those can be attacked and defended and perturbed. So that aspect of it, I find very interesting and I hope to explore that more. Just as a hobby, I'm looking into quantum computing. I have some theories about how some aspects of quantum computing relate to the optimization problems we often see in adversarial learning, which I'm pursuing. And I have a thesis student this year, Sam, who I'm very excited to work with, and I'm sure she will do great work. And I will see where she, what path she leads me down in research.
Nick Horton (49:27):
So that's kind of a great transition to one of mine, which is what's it like teaching machine learning to Amherst students?
Scott Alfeld (49:35):
It's great. Like I like the students a lot.
Scott Alfeld (49:38):
And one of the so I've taught at other institutions too. And one of the things I've noticed here is the in part due to the open curriculum and the structure of prereq requirements and whatnot, in part due to other reasons, there's a huge breadth of background in any single class. So I'll have students who, you know, they're in their last semester of a math major. And they're interested in the machine learning side and they've never written a line of code. I have other students who, you know, have never seen any calculus before and are early on in their computer science career. And I need to design the course in such a way that they both get something out of it, which is terrifying, but also a lot of fun. And then in addition, whenever we bring in something outside the core of ML, which happens all the time in adversarial learning, students have these different perspectives.
Scott Alfeld (50:40):
So like I love discussing fairness because these students, you know, they say, Oh, in this thing that I'm interested in, this came up and here was the issue, and you know, what you talked about before and machine learning land can apply directly to that. And these making connections with very different fields is great. And it's also super helpful in that, you know, if you're taking the role of the attacker, well, you get to do anything you're capable of doing. And the students have been great about coming up with motivations behind attackers, figuring out what is realistic, what's possible, and at the same time, they've got the mathematical chops to, you know, run through it and, and do really, really great work. So in teaching the courses and advising students for thesis and for summer research and whatnot, it's been fantastic.
Nick Horton (51:35):
Great. A hypothetical question. Various things that have been coming in on the, on the Q&A. Imagine that you've been asked during your sabbatical, which you've earned, and we're going to excited to see what you do during that, but say that, you know, the president's science advisor pulls you in for a consult. And they're interested in trying to understand, you know, whether machine learning models are being in reinforcing red line red lining, whether there's the dark trace algorithm, they don't know what's going on with that algorithm. They've heard that, you know, that there's, there's credible evidence that that a foreign nation has been kind of stealing data from Anthem health systems. And they're trying to get their hands around this one. What advice would you give them advice?
Scott Alfeld (52:26):
That's interesting. So my go to advice whenever I get any security question is typically use two factor authentication that doesn't seem to apply in this setting. I think the, in my work at least, and I think generally more globally, there's a fairly solid line between detecting an attack and then handling it outside, as in Hey attacker, how about you and me go take this outside, right? Where you say, okay, I've detected that you've done this, so I will shut down your account. I will delete the data whatnot. And then there's a wall between that and I am building a system robust to attacks that I did not detect. Most of what I've done is in that box. I've done a little bit, so in this inhuman traffic detection stuff I talked about, I did the how to detect attacks. And, you know, a lot of it is to recognize everybody gets lucky.
Scott Alfeld (53:32):
So if you imagine I'm looking at the data and Oh, this person predicted stocks would go up and they did, and whatnot, everybody gets lucky. You want to look for the times that people are getting lucky more often than skill can explain how you do that varies dramatically based on the setup and, you know, depending on the details of what the president's science advisor has asked me about, I might have more concrete questions of or contributions to, but a big part of it is, you know, you're performing an investigation and there are many, many different angles you need to look at and get the sort of holistic view.
Nick Horton (54:14):
Great. Well, unfortunately, we're coming near the end of the time. Just one question left for you. What's this business in your bio about recreational lock picking? Have you been practicing during quarantine?
Scott Alfeld (54:34):
A bit. So for those of you don't know, lock sport is you take a lock, you try to pick it open usually quickly. And so, I've done a lot of lock sport in the past. I've done a lot of miscellaneous, you know, physical security stuff in the past, and I've connected that to computer security in my teaching. So I teach our computer security class, which is much more on the system security side of things, not this adversarial learning stuff so much, and many of the problems we face in computer security of, you know, how do I make this resistant to attack while still convenient? And how do I make it so I can handle multiple users and stuff? Many of these problems were solved decades ago for other systems where there was no electricity involved. So, you know, if you look at bank vault, safes, and various, you know, master keying methods and whatnot, there's a lot of there's a lot of value in saying, we're going to discuss this security topic, and we're going to begin with you seeing it in a piece of brass that you can hold in your hands and play with.
Scott Alfeld (55:50):
And so I've drifted from sort of the competition side of, you know, I want to be the best lock picker in the world to more of the, I want to teach people how to pick locks and how locks work and how physical security works in general. Near the start of the pandemic, I picked up safe cracking, which is a slow process to learn. But you know, now's a good time to do it.
Nick Horton (56:16):
Well, Scott, again, I'm glad that you're keeping yourself busy, that you're keeping yourself safe.
Nick Horton (56:24):
It's really remarkable the work that you're doing on behalf of our students and the college on a really important area that I do think really spans the curriculum. So I want to just thank you again for taking the time tonight to share those thoughts with with the alumni and our students and other members of the community. Thank you.
Scott Alfred (56:43):
Yeah. Thank you.