The rogue cryptography behind ransomware

14 March 2023
claves para conocer qué es GitHub
Marta López
Marta López

Head of Marketing and Communication

Dr. Gonzalo Álvarez MarañónDirector of the Master in Cybersecurity Online and R&D Director of Funditec..

Who would have thought that cryptography would be maliciously exploited to create one of the most devastating categories of malware. And so it is: the ransomware blocks access to the victim's files and demands a ransom in exchange for the encryption key.

The cryptography behind the first samples of ransomware was simplistic and naive. Typically, the ransomware encrypted each file with a different AES key (or weak algorithms, anyway), but all AES keys were stored in a file on the hijacked system. The ransomware's decryptor module could easily find the keys and decrypt each file, but so could the security researchers by means of reversing. If you don't want someone to decrypt a file, it's a bad idea to leave the keys lying around... under the mat, so to speak.

Ransomware authors logically turned to asymmetric encryption for a solution. The most obvious advantage of asymmetric cryptography is that a public key may be on the victim's system and its corresponding private key may be elsewhere. However, for reasons of efficiency, files cannot be encrypt with RSA directly. It is necessary to resort to faster algorithms to encrypt the contents of the files, while the corresponding symmetric encryption keys can be encrypted individually with RSA. After all, an AES key is only 16 bytes for AES-128 and 32 bytes for AES-256. Each key can easily be encrypted with RSA before being stored on the victim's system. RSA encrypts them with the public key, so as long as the private key is not available to the victim, he will not be able to decrypt the AES keys.

There are two ingenious variants of this approach, both of which are problematic:

  • The first approach is to generate the key pair in advance and encrypt the public key in the malware itself. Once the malware has encrypted all the AES keys with the public key, the victim has to pay the ransom to have the private key sent for decryption. The obvious flaw in this design is that the same private key would unlock all systems attacked by the ransomware, as each copy of the malware incorporates the same public key.
  • The second approach is for the ransomware to generate the RSA key pair on the victim's system and transmit the private key to the command and control (C&C) server. Now a unique public key per malware sample encrypts the AES keys. When the attacker releases the private key for decryption, he only unlocks the files of the specific victim. The problem here is that the system has to be online to get rid of the private key and many network monitoring systems will detect transmissions to risky IPs, from which the C&C servers operate. The mere transmission of the private key could reveal the presence of the ransomware before it has even started encrypting system files. It is more stealthy to do everything locally, until the system is completely locked down.

Modern ransomware solves all these problems with a rather clever approach:

  • The attacker generates in his own system an asymmetric key pair for long-term use, which we will call permanent asymmetric keys.
  • The attacker creates a malware and encrypts the permanent public key in the malware. When the malware is activated on the victim's machine, the first thing it does is locally generate a new asymmetric key pair, which we will call the local key pair. It immediately encrypts the newly generated local private key using the attacker's permanent public key, embedded in the malware. Before proceeding, it thoroughly removes the unencrypted local private key.
  • The malware starts encrypting the files on the disk using AES-CTR or AES-CBC. Each file is encrypted with a different, randomly generated symmetric key, and then each key is encrypted with the local public key. The unencrypted version of each symmetric key is consciously destroyed as soon as the file is finished being encrypted.
  • Once the whole process has been completed, the victim's files have ended up encrypted with random AES keys, which in turn are encrypted with the local public key. These AES keys can be decrypted with the local RSA private key, but that key is stored encrypted with the attacker's permanent public key and the corresponding private key is not on the victim's machine, but on the attacker's machine.
  • The attacker displays a ransom demand message to the victim. If the victim accepts and pays the ransom (usually via Bitcoin), the attacker provides some kind of authentication code to the malware. The malware transmits the encrypted local private key to the attacker. Using its permanent private key, it decrypts the local private key and sends it back to the victim. All AES keys can now be decrypted, and with them all encrypted files on that machine, but not on any other machine.

The subtle thing about this protocol is that the attacker never reveals his permanent private key. It remains private. The attacker merely decrypts a secondary private key for the victim to use to unlock the rest of the system. As this local private key is different for each compromised system, its disclosure only affects the system in question.

Properly implemented, ransomware cryptography is unassailable. Eventually, the reversing engineers at the big antivirus firms will find no way to create "decryptors". All we will be left with is prevention and iron discipline with backups.

Subscribe to our newsletter
menuchevron-down