Today in this tutorial, I'm going to demonstrate with pseudocode on how to decrypt Windows executable files that use simple XOR encryption. The decryption process relies less on logic, but more on bruteforcing.
XOR is used in various types of encryption, and is sometimes used as it's own encryption in it's raw form without any supporting algorithms. When binary values at the same index are equal, it's 0, when they're opposites it's 1 (For XOR).
Let's take two binary values called data and key:
The result is: 01001110
Need to know:
Windows executable files will usually start with two bytes, 0x4D and 0x5A which in ASCII are "MZ". This is a file format to differentiate between COM files and executables.
How It's Used in Programs
Some programs will use XOR encryption on their bytes, XOR can be used to encrypt whole files, a character, a string, a portion of a file, etc...
In pseudocode, this is an algorithm that encrypts a whole file
Bruteforcing The Encryption
Now that EncryptedFile has been encrypted with a key between 0 and 255, the file format at the beginning of a file 0x00000000 and 0x00000001 won't be "MZ" but something random like "%:". This would be an algorithm to get the key from just the first two bytes and then decrypt the entire file. *pseudocode*
Now you have the encryption key for the encrypted file. All you have to do now is call the xorFile function with the encryption key now and decrypt the file to it's original data.
You've successfully decrypted a file that uses a single character XOR key cipher.
Edited by jasonfish4, 16 April 2016 - 01:28 PM.