The Challenge

The challenge hands you three things: an encryptor binary, a 12-byte key file, and flag.enc. The binary takes a key file, an input file, and an output file, and writes the encrypted result. There is no decryptor. The whole task is to reverse the cipher out of the binary and invert it.

How to use: ./encryptor <key_file> <input_file> <output_file>

So this is a pure reversing problem: understand the algorithm, then write the inverse.

Reading the Binary

main reads the 12-byte key, then encrypts the input in blocks. The key reader just pulls six 16-bit words out of the file:

// sub_1229 - read 6 * 2 = 12 bytes into the key buffer
fread(key_buffer, 2, 6, stream);

The encryption driver reads the input 4 bytes at a time and encrypts each block in place:

// sub_1381
for (s = 0; ; fwrite(&s, 2, 2, out)) {
    if (!fread(&s, 1, 4, in)) break;   // read one 4-byte block
    sub_129F(key, &s);                 // encrypt it
}

Two facts fall out of this immediately:

• The block size is 4 bytes, split into two 16-bit words.
• The key is six 16-bit words, enough for two words per round across three rounds.

The ARX Round

sub_129F is the heart of the cipher. It loads the block as two 16-bit halves v3, v4, then runs three rounds of Add-Rotate-XOR:

v3 = *a2;        // first half
v4 = a2[1];      // second half

for (i = 0; i <= 2; ++i) {         // 3 rounds
    v5 = key[2*i]     ^ (v4 + ROL(v3, 7));
    v3 = key[2*i + 1] ^  ROL(v4, 7);
    v4 = v5;
}

*a2   = v3;
a2[1] = v4;

Each round mixes the two halves with three reversible primitives:

• Addition modulo 2^16
• XOR with a round key word
• ROL 7: a 7-bit left rotation on a 16-bit word

This is a Feistel-flavored ARX construction. The important property of ARX is that every operation is invertible, so as long as we know the key, we can walk the whole thing backwards.

Inverting Each Round

Take one encryption round with input (v3, v4) and round keys k0 = key[2*i], k1 = key[2*i+1]:

v5   = k0 ^ (v4 + ROL(v3, 7))
v3'  = k1 ^ ROL(v4, 7)
v4'  = v5

Given the output (v3', v4'), undo it step by step. Each ARX primitive has a clean inverse: addition becomes subtraction, and ROL 7 becomes ROR 7; XOR is its own inverse:

v4  = ROR(v3' ^ k1, 7)                 # recover the original v4
v3  = ROR((v4' ^ k0) - v4, 7)          # recover the original v3  (mod 2^16)

Run this for i = 2, 1, 0, the rounds in reverse order, and each 4-byte block decrypts cleanly.

The Decryptor

import struct

def ROR(val, r, bits=16):
    r %= bits
    return ((val & (2**bits - 1)) >> r) | (val << (bits - r) & (2**bits - 1))

def decrypt_block(key, block):
    v3, v4 = struct.unpack('<HH', block)
    for i in range(2, -1, -1):           # rounds in reverse
        k0, k1 = key[2*i], key[2*i + 1]
        v5 = v4
        v4 = ROR(v3 ^ k1, 7)             # ROL -> ROR
        v3 = ROR(((v5 ^ k0) - v4) & 0xFFFF, 7)   # add -> sub, then ROL -> ROR
    return struct.pack('<HH', v3, v4)

key = struct.unpack('<HHHHHH', open('key', 'rb').read())
enc = open('flag.enc', 'rb').read()

flag = b''.join(decrypt_block(key, enc[i:i+4]) for i in range(0, len(enc), 4))
print(flag.rstrip(b'\x00').decode(errors='ignore'))

DH{REDACTED}

Takeaways

The reversing here is not about a clever trick. It is about being careful. Everything comes down to reading the round exactly, respecting the 16-bit width (so subtraction wraps modulo 2^16), and remembering the little-endian word layout on both the key and the data.

ARX ciphers show up constantly because they are cheap and fast, and this challenge is a clean reminder of why they are also reversible by design: an attacker who recovers the key can run the addition, rotation, and XOR steps backward with nothing more than arithmetic.