New anti-APT tools are no silver bullets:
An independent test of APT attack detection appliances
CrySyS Lab, BME http://www.crysys.hu/
MRG-Effitas https://www.mrg-effitas.com/
November 26, 2014.
The term Advanced Persistent Threat (APT) refers to a potential attacker that has the capability and the intent to carry out advanced attacks against specific high profile targets in order to compromise their systems and maintain permanent control over them in a stealthy manner. APT attacks often rely on new malware, which is not yet known to and recognized by traditional anti-virus products. Therefore, a range of new solutions, specifically designed to detect APT attacks, have appeared on the market in the recent past, including Cisco’s SourceFire, Checkpoint, Damballa, Fidelis XPS, FireEye, Fortinet, LastLine, Palo Alto’s WildFire, Trend Micro’s Deep Discovery and Websense.
While these tools are useful, determining their real effectiveness is challenging, because measuring their detection rate would require testing them with new, previously unseen malware samples with characteristics similar to those of advanced malware used by APT attackers. Developing such test samples require special expertise and experience obtained either through the development of advanced targeted malware or at least through extensive analysis of known samples.
We in the CrySyS Lab, together with our colleagues at MRG Effitas, decided to join our forces and perform a test of leading APT attack detection tools using custom developed samples. MRG Effitas has a lot of experience in testing anti-virus products, while the CrySyS Lab has a very good understanding of APT attacks gained through the analysis of many targeted malware campaigns. Therefore, collaborating and bringing together our complementary sets of expertise looked like a promising idea. Our goal was not to determine the detection rates of different APT attack detection products, because that would have required testing with a large set of custom developed malware samples, which was not feasible to obtain within the limited time frame and with the limited resources we had for the test. Instead, our goal was simply to implement some ideas we had for bypassing cutting-edge APT attack detection tools without actually being detected, and to test if our ideas really work in practice.
We developed 4 custom samples in 2 weeks and without access to any APT attack detection tools during the development, and then later tested with these samples 5 APT attack detection solutions in Q3 2014. All 5 tested products are well-established in the market; however, we cannot mention vendor names publicly. The result of the test was alarming:
– one of our 4 custom samples bypassed all 5 products,
– another sample of the remaining 3 samples bypassed 3 products,
– only the two simplest samples have been detected by the tested products, and even those triggered alarms with low severity in some cases.
We made the full report (https://blog.mrg-effitas.com/wp-content/uploads/2014/11/Crysys_MRG_APT_detection_test_2014.pdf) on our test available online. It contains our test methodology, including a brief description of each sample we developed for the purpose of the test, and we also present in it the test results in more details. We decided to publish this report for multiple reasons:
– First of all, we believe that our test was more appropriate for evaluating the detection capabilities of APT attack detection tools than some earlier, heavily criticized tests were, because unlike earlier tests, we used custom developed samples that resemble the malware used in APT attacks.
– Second, some of the products that we tested seem to be overestimated by the users who believe that those products are silver bullets. On the other hand, we have already emphasized at multiple occasions that these products can and will be bypassed by determined attackers. Our test is a clear proof of this, and if we could do that, then APT attackers will also be able to do that, if they have not done so yet.
– Third, we observed that some vendors of APT attack detection tools are often reluctant to participate in tests that try to evaluate the effectiveness of their products. On the one hand, we understand their caution, but on the other hand, we all know that the approach of security by obscurity has its own pitfalls. By publishing this report, we would like to encourage anti-APT tool vendors to participate in independent tests more readily and cooperatively, in order to have sufficient amount of convincing results about their products, based on which well-informed decisions can be made by the users.
– And last but not least, we believe that there are significant differences in the APT detection capabilities of the tested products, and users should be aware that not all vendors provide the same detection rate.
The test sample that bypassed all 5 tested products was developed by the CrySyS Lab. It is a custom designed sample written in C++ with a server side written in PHP. It was designed to be as stealthy as possible. It is downloaded by the victim as part of an HTML page, where it is actually hidden in an image with steganography. The downloaded page also contains scripts that extract an executable from the image when the user clicks on something that appears to be a download button. Once the sample is running, it can communicate with a remote C&C server. To hide the C&C network traffic, the sample simulates a user clicking on links in a web forum, and downloads full HTML pages with CSS style sheets and images. The real C&C traffic is hidden inside these HTTP requests. The sample allows for file download from and upload to the C&C server, as well as remote execution of commands on the victim computer.
We named this test sample BAB0, which (babo) means hobbit in Hungarian, as its objective was to stealthily bypass all state-of-the-art defenses, while actually being very simple, and this situation shows a parallel to the story of the Lord of the Rings, where Frodo, the small hobbit managed to bypass all defenses of the fearsome Sauron, the Lord of Mordor, and reached Amon Amarth, where the One Ring was finally destroyed.
We have a strong intention to publish BAB0 in the near future. This may seem to be controversial, as making the details of BAB0 publicly available can help attackers. We have a different opinion: Powerful attackers have probably been using already similar tricks, but apparently detection tools are not yet prepared to cope with them. By publishing BAB0, we push anti-APT vendors to strengthen their products, which will ultimately make the attackers’ job harder.
For further information, please contact either Zoltan Balázs (Zoltan.Balazs@mrg-effitas.com) or Levente Buttyán (buttyan@crysys.hu). Please note that we cannot provide any vendor specific information about the tests, but we can help organizations to test the products integrated in their environment.