Hello there! My name is Dávid Maliga, a first-year PhD student – under the supervision of Prof. Levente Buttyán – in the CrySyS lab, and I would like to share with you our latest work, which we just recently presented at this year’s CITDS conference (2026 IEEE 4th Conference on Information Technology and Data Science) in the city of Debrecen, at the beautiful campus of the University of Debrecen.
Repeatability of expreiments is an important requirement for any scientific approach. Repeatability means that if someone wants to apply an existing approach to their own problem, they should be able to do so reliably, by following the documented methodology. The same methodology under which the original approach was developed and validated. This means, to verify an approach, repeating the same measurements on the same dataset should produce exactly the same results.
Unfortunately, in the field of IoT malware detection, the current tendency is that individual research groups use their own, proprietary datasets. This renders reproducibility difficult, because even if we succesfully reproduce their approach, we don’t have their data to validate it. If we consider machine-learning based approaches, even if two approaches rely on the same dataset, the specific train–test split could be unavailable. This way we couldn’t tell whether the difference in results come from the approaches or the split. This raises the question of whether results obtained in different settings are genuinely comparable. Without a shared reference point, it becomes hard to say whether one approach outperforms the other.
To address this issue, we have developed a methodology for creating EMBeD (Embedded Malware Benchmark Dataset), a publicly available benchmark dataset specifically for IoT malware binaries. This method processes existing malware datasets to extract metadata and filter out unreliable or underrepresented samples. It also verifies malware family labels and balances the dataset in terms of number of samples per malware family.
Data preparation
The first step is to build a malware metadata database for the next stages of the workflow. Malware samples from existing public datasets are processed through a pipeline that extracts basic information about each file, such as its hash, architecture, bitness and entropy, as well as various statistics from VirusTotal scan results. Samples that appear to be benign or have underrepresented properties are filtered out. The result is a clean, detailed database of potential malware samples that can be added to the benchmark dataset.
Enhancing family labeling
Next, a reliable family label is required for each sample. This information can be extracted from the VirusTotal scan results by aggregating the votes and labels provided by various antivirus engines. As the antivirus engines often disagree about the family label, the votes are weighted according to the historical accuracy of the engines to provide an initial family label for each sample.
These initial labels are then validated using a similarity graph. In this structure, each node represents a sample, and two nodes are connected if, and only if their corresponding samples are similar. We consider two samples similar if their TLSH difference score is below a certain threshold, where TLSH is well-known fuzzy hashing scheme that can quantify the dissimilarity of two samples. For each sample, the weighted initial family labels of its neighbours in the similarity graph are aggregated, with those that have lower dissimilarity score getting larger weight. If the new aggregated label differs from the sample’s initially assigned label, then we consider the initial label to be inconsistent, and the sample is removed. This process is repeated until all remaining labels are consistent.
Balancing the family sizes
Finally, the dataset needs to be balanced, as some malware families have far more samples than others. For large families, we select a representative subset of the samples conatining a target number of samples. For small families below the target size, we search for additional samples on VirusTotal using the LiveHunt and RetroHunt features. We also perform similarity search in Ukatemi’s Kaibou Repo, a large repository of around 800 million malware binaries with a fast similiraty search engine. This process is repeated until family sizes become balanced.
Preliminary results
To verify that our method was working, we created a proof-of-concept (PoC) benchmark dataset, aiming to collect 100 samples per family. To achieve this, we applied the process to 67,800 MIPS-based IoT malware samples. After labelling and balancing, we achieved an evenly sized benchmark dataset covering seven malware families: mirai, gafgyt, hajime, kaiji, tsunami, ddostf and dofloo. With this small-scale PoC dataset, we demonstrated that the entire workflow is effective and can produce a benchmark dataset.
Our goal for the future is to expand this project and publish a much larger version of EMBeD as soon as possible. This will involve extending the scope beyond MIPS to include other architectures commonly used in IoT devices, such as ARM. This would make the benchmark useful for a much wider range of malware detection research across the IoT domain.
If you are interested in the methodology and would like to know more details, the paper presented at the conference is available here. It should be cited as follows: Dávid Maliga, Dorottya Papp and Levente Buttyán, Creating EMBeD, the Embedded Malware Benchmark Dataset, IEEE Conference on Information Technology and Data Science, 2026.
We would like to thank VirusTotal for providing an academic license, and Ukatemi Technologies for granting access to their Kaibou malware repository.