What stands behind winning the Trojan Detection Challenge featured on the NeurIPS
Neural Trojans are a growing concern for the security of ML systems, but little is known about the fundamental offense-defense balance of Trojan detection. So Academia decided to throw a challenge to address it.
The Trojan Detection Challenge interested three of our colleagues: Pavel Pleskov, Senior Quant in Trading Antifraud, Nikita Benkovich, Business Data Scientist, and Roman Smirnov, ML Engineer at NLP. They not only accepted the challenge, but achieved excellent results and took first place in the Trojan Detection Track as well as third place in two more tracks and the Final Round.
We talked to Pavel to find out how the team achieved these results and how difficult it was.
As a Senior Quant in Trading Antifraud, I am responsible for developing quantitative models to tackle anti-fraud initiatives that are crucial to the growth of our company.
Every day this work is becoming more important and useful for the company. As the volume of our operations expands, the mathematical models we create help ease the workload of our operational team. With five years of experience in machine learning and data analysis, I find great satisfaction in tackling these complex and challenging tasks.
What is Trojan Detection Challenge
In this competition, participants were challenged to detect and analyze Trojan attacks on deep neural networks that are designed to be difficult to detect. Participants competed to obtain the best score on a machine learning task of interest to the community based on a well-defined problem and corresponding data defined and released by the competition organizers.
The competition was based on the problem of image classification in computer vision – where the algorithm must recognize an image and understand which group it belongs to, e.g. identify a handwritten digit. Various datasets are used to train and operate such systems. They are also commonly referred to as "toy datasets", such as MNIST (a set of 10 handwritten digits), GTSRB (a set of German road signs) and many others.
However, if a neural network that is capable of solving such a task is attacked and poisoned, it may incorrectly recognize images. The attack is often performed by slightly altering the image, for example, by applying a patch of noise or a black-and-white mask. While the picture may appear unchanged to the human eye, it looks different to the computer and can lead to severe consequences – we will talk about them a bit further.
If someone applies a specific type of “noise” to an image of a panda, a computer can easily mistake it for a gibbon
The competition had 4 main tracks:
Identify the class to which a poisoned neural network would incorrectly assign images.
Determine the patch that was used for the attack.
Recognize whether a neural network has been poisoned or not.
Organize an attack on a neural network, although our team did not participate in this part.
What is the challenge in Trojan detection?
The main challenge was that the task required working with a neural network. Typically, competitions involve working with classifiers, images, text, or tables, such as teaching an algorithm to distinguish a cat from a dog. So this task was more complex, unique, and highly fascinating.
However, currently, this topic is not widely popular yet. Attacks on neural networks are not that frequent, and such cases are mostly theoretical. So the struggle is confined to the academic realm rather than practical applications. But the more models are implemented in the workplace, the more relevant this problem will become. After all, many models are trained on open datasets, and if malicious elements are embedded in them, all neural networks trained on them can become vulnerable.
For example, the autonomous vehicle industry relies entirely on computer vision, where cameras recognize road signs in particular. If an attack affects the recognition of the "Stop" sign, the car can make the wrong decision and may not stop where it needs to, causing an accident.
Why we wanted to accept the challenge
As a team, we were particularly drawn to this challenge because it is related to security, which is a paramountly important issue in our daily work and for our company as a whole. We also considered it an exciting opportunity to contribute to the field of data science by applying our knowledge and experience to practical problems. Our ideas and results impressed the academic community, and we were even offered to co-author a scientific paper based on our work: 👉 How Hard is Trojan Detection in DNNs? Fooling Detectors With Evasive Trojans Besides, the competition was featured on NeurIPS, which is a conference and workshop on Neural Information Processing Systems. It is the largest machine learning and computational neuroscience academic conference held every December starting from 1987, where the best minds from around the world gather to share their latest research and findings. I have participated in professional machine-learning competitions on the Kaggle platform for 3 years. And for several years, my teammate Nikita and I have been striving to reach NeurIPS competitions to win and get a chance to present our work at the conference, which is a great honor.
We also wanted to showcase that our company has a very skilled team when it comes to Machine Learning. And that we can compete on a global level. It is exactly what happened. The other winning teams consisted mostly of scientists from China and the Middle East. Our company and, particularly, Denis Kravchenko, our Chief Data Officer, who is responsible for developing ML research, strongly supported us in this endeavor. We officially had 1 day a week to work on competition tasks. And we had lots of opportunities to share our results. In my experience, not all companies treat it that way. That helped and kept us motivated.
The challenge as a project
The Trojan Detection Challenge took place from July to November. We knew all tasks in advance and worked on them simultaneously: solving one problem helped with another. As we went along, new details emerged, which added spice to the competition.
The timeline was typical for long competitions. At first, we slowly got into the task: brainstorming ideas, discussing hypotheses, and reading articles. Then, we had our first breakthrough idea, which got us ahead on the leaderboard and inspired new ideas. Then, our competitors began to catch up with us. And we came up with another approach, which proved effective and pulled us ahead again. The situation repeated at the very end, and we devised a third approach. The intensity grew as we got closer to the final, and the last 2 weeks were very hectic.
The process of working on the challenge went very naturally for our team. It resembled working on a small project in a small team. The only difference is that we did not have a project manager or formal leader - but I've seen that a lot in other competitions.
We regularly held syncs to discuss hypotheses, directions, and upcoming plans. Otherwise, everything was quite democratic and independent. Each team member worked on his own track—on the hypotheses that seemed most important to him. Roman was also working on another competition. Then, we met again to check and combine our solutions, and if a hypothesis proved useful, the team member who worked on it would move on while the others shifted to the next plan.
Working on the task took up all our free time. But none of us put off the rest of our lives.
1. We had a clear understanding of what we were going for. Sure, it is impossible to predict how much time it may take because everything depends on many nuances: how many hypotheses we may have, how many we may want to test, and how quickly it may happen. But we were aware that it was a finite process. And it made it a lot easier since we understood the purpose of our efforts: the prize served as a strong motivator for us.
We also carefully evaluated the competition before we applied. It was like due diligence: the task's interest level, the complexity of the assignments, the level of competition, and the computing power required.
2. We made arrangements both at home and at work. Our families understood that this was a temporary but potentially rewarding endeavor. For example, my wife has watched me participate in dozens of competitions for 3 years and is sympathetic to the fact that I can sit down at 5 on a Saturday morning to solve this or that problem.
We also had no conflicts of interest with our work; on the contrary, our company supported us. We also actively participated in our corporate projects, dedicating a specific day and free time to the Trojan Detection Challenge.
3. When working on a competition, you aren’t sitting in front of a computer the whole time. It's an uneven process. Hypotheses don't appear 24/7. But as soon as they do, you better run, and test it to see if it leads to any results. Also, training models during experiments takes time. And during those moments, you're essentially doing nothing: the computer can do it for hours while you engage in something else.
Yes, of course, there was a moment when we were testing a new hypothesis on a Sunday evening, just a few hours before the deadline, already sleepy and tired. But this is also a classic trait of such competitions. And you are usually spurred on by a lot of excitement and you can't help but do it.
Powerful hardware is a big plus for competitions. For example, it is impossible to solve the current problem on a laptop. We ran all experiments on a computer that was on my balcony—I built it specifically for competitions and have already won many with its help. The weather was dry all 4 months of the challenge. But on the penultimate night, it suddenly started raining heavily. I heard thunder, woke up, and rushed to the balcony to take care of the computer. Fortunately, everything turned out okay, and our data and models were unharmed. By the way, there were a lot of them—my electricity bill increased by 2 times.
How to do something like this if you've never done it before
1. Find someone who's had a similar experience and work with them on the same team. It helps a lot, especially if it's your first competition.
2. Remember the goal and motivate yourself. At the end of each competition, the winners get a prize. Remember it as well as the fact that you have the opportunity to solve a non-trivial problem. It can be very motivating. Everyone is excited in the beginning, but then the flame fades. But if there is an ultimate goal, and everyone wants to achieve it, then everything will work out.
3. Don't forget that the process is also a pleasure and fun. Sure, participating in a competition is a real emotional roller coaster. But the feeling that teamwork can give you—when progress and results are visible when you work together in a friendly and fun way—is quite inspiring. So do not force yourself or treat the process as something unnatural. It's just a choice that you add to your life for a while.
It's just like amateur sports. When somebody is going to play soccer in the yard, they understand why they're doing it. They come to enjoy the game instead of treating it as a chore. It's the same here.
The most difficult part
To be honest, everything turned out to be more challenging than we expected. But this is also typical in competitions. In the beginning, almost all tasks are simple and there are few competitors. But the further you move, the higher the level of difficulty goes. You have to accept that not all hypotheses can be tested. Competitors also become more active and breathe down your neck. For example, there were about 160 teams in the Trojan Detection Challenge.
Everything didn't go according to plan, but not to the point where it could demotivate or disrupt us and prevent us from giving our all. And that's OK.
The most pleasant part
The organizers were very responsive: they communicated well with all participants and adequately responded to feedback. And it is not always the case: in some competitions, the organizing committee is detached from the process and does not make decisions transparently.
Of course, it was great that we were able to propose 3 different solutions to the main task, which turned out to be innovative and effective and impressed the organizers. It was a pleasant surprise even for us and hard to predict at the beginning. Therefore, along with the victory, the prize, and the participation in the article, this added plenty of joy.
Back to all posts