HOLLOWDEX (DefCamp CTF 2026) — Self-Decrypting DEX Bytecode

Challenge setup

You get one file — hollowdex.apk, 16,984 bytes, eight entries, three of them under META-INF/. No server, no network, no permissions. Everything needed is inside the archive.

AndroidManifest.xml          1,776 B
res/layout/activity_main.xml 1,480 B
resources.arsc                 812 B   (stored)
classes.dex                  3,828 B
lib/arm64-v8a/libhollowdex.so 5,632 B
META-INF/                     2,320 B

The layout describes one activity — a title, a monospace field hinted Enter 32-char hex key, and a Verify button. The verdict label is empty in the layout and filled from Java, with ACCESS GRANTED or ACCESS DENIED. Four lines of that Java matter:

private final Verifier verifier = new Verifier();
...
if (verifier.verifyFlag(s)) {
    result.setText("ACCESS GRANTED");
}

verifyFlag is not a native method — it is ordinary Java, returning boolean, and it is the whole challenge.

Step 1 — the method that will not disassemble

Unzip the APK and run baksmali on classes.dex. Every class comes out except one:

Error occurred while disassembling class Lcom.hollowdex.Verifier; - skipping class
org.jf.util.ExceptionWithContext: The last instruction in method Lcom/hollowdex/Verifier;->verifyFlag(Ljava/lang/String;)Z is truncated

It emits MainActivity.smali, the synthetic lambda class, the R classes — and no Verifier.smali at all. The whole class is dropped, not partially rendered, so the first signal is an absence.

The only other thing Verifier does is load a library:

static {
    System.loadLibrary("hollowdex");
}

That string is in the DEX string pool — the one thread connecting the broken method to the native code.

libhollowdex.so exports exactly one symbol:

Symbol table '.dynsym' contains 16 entries:
    15: 0000000000001900   732 FUNC    GLOBAL DEFAULT   12 JNI_OnLoad

The other entries are fourteen UND imports and the reserved null symbol. There is no RegisterNatives and no Java_com_hollowdex_* stub — the library exists for a side effect, and nothing in Java ever calls into it.

Step 2 — the build-time XOR

The build runs aapt2, javac and d8 by hand, then patches the DEX before packaging:

echo "[5] Patching classes.dex (XOR-encrypting verifyFlag)..."
python3 "$SCRIPT_DIR/patch_dex.py" "$BUILD/dex/classes.dex"

The patcher walks the DEX the long way — it reads type_ids to find Lcom/hollowdex/Verifier;, scans method_ids for a verifyFlag owned by that type, locates the matching class_def, and decodes the uleb128 class_data_item to reach the encoded_method entries.

One line in that walk is easy to get wrong:

if list_idx == direct_methods_size:
    method_idx_acc = 0  # accumulator resets at virtual methods boundary

The DEX spec defines method_idx_diff relative to the previous element of the same list — so the first virtual method's diff is an absolute index. Without the reset the accumulator reads 16 + 1 + 18 = 35, matches nothing, and the lookup fails.

For the shipped APK the walk lands here:

code_item @ 0x0000062C
  registers_size = 15
  ins_size       = 2
  outs_size      = 2
  tries_size     = 0
  debug_info_off = 0x7D9
  insns_size     = 156 code units
insns @ 0x0000063C,  len = 312 bytes

Only insns[] is encrypted — the 16-byte code_item header is untouched, which is what lets a solver read insns_size out of the file and know how much to decrypt.

tries_size is zero and insns_size is even, so there is no padding half-word. The 312 bytes are exactly the payload.

Two header fields have to be repaired afterwards:

sha1 = hashlib.sha1(bytes(raw[0x20:])).digest()
raw[0x0C:0x0C+20] = sha1

cksum = zlib.adler32(bytes(raw[0x0C:])) & 0xFFFFFFFF
struct.pack_into('<I', raw, 0x08, cksum)

The DEX header defines checksum at 0x08 as Adler-32 over everything after itself, and signature at 0x0C as SHA-1 over everything after itself. The insns region sits inside both ranges, so flipping 312 bytes invalidates both.

The order matters — SHA-1 first, because it does not cover the checksum field, then Adler-32, because it does cover the signature.

Step 3 — the runtime decryptor

ART calls JNI_OnLoad once dlopen returns, inside Verifier's static initializer, which runs when MainActivity constructs its field. The patch is meant to land long before the first verifyFlag call — whether it does is the last section.

Finding the method

It starts by asking JNI for the method it is about to rewrite:

jclass cls = env->FindClass("com/hollowdex/Verifier");
jmethodID mid = env->GetMethodID(cls, "verifyFlag", "(Ljava/lang/String;)Z");

ART does not box jmethodID under the default kPointer id type — EncodeArtMethod() casts an ArtMethod* straight to the opaque handle, so what JNI hands back is a raw pointer into the runtime's method array. The decryptor reads a field out of it:

// jmethodID is ArtMethod* on ART; dex_code_item_offset_ lives at +0x08 (arm64, API 34)
const uint8_t *art_method = reinterpret_cast<const uint8_t *>(mid);
uint32_t code_item_off;
memcpy(&code_item_off, art_method + 0x08, sizeof(code_item_off));

One instruction after optimization — ldr w25, [x0, #8] at 0x19b4. The arithmetic assumes the ArtMethod prologue of a 64-bit build, where declaring_class_ is a GcRoot and therefore a 32-bit compressed reference even on ARM64:

+0x00  GcRoot<mirror::Class> declaring_class_   4 bytes
+0x04  atomic<uint32_t>     access_flags_      4 bytes
+0x08  uint32_t             dex_code_item_offset_
+0x0C  uint32_t             dex_method_index_

Finding the mapped DEX

Next it has to find where the DEX is mapped:

while (fgets(line, sizeof(line), f)) {
    if (!strstr(line, ".apk"))   continue;
    if (!strstr(line, "r--p") && !strstr(line, "r-xp")) continue;
    if (sscanf(line, "%lx-%lx", &start, &end) == 2) {
        const uint8_t *p = reinterpret_cast<const uint8_t *>(start);
        if (p[0]=='d' && p[1]=='e' && p[2]=='x' && p[3]=='\n') {
            result.base = start; ...

Three filters — the pathname contains .apk, the mapping is readable, private and not writable (r--p or r-xp), and the first four bytes at its base are dex\n. In the disassembly this is three strstr call sites and a byte-wise magic check against 0x64, 0x65, 0x78, 0x0a.

Then the target address, mirroring the patcher exactly:

const uint8_t *ci = reinterpret_cast<const uint8_t *>(dex.base + code_item_off);
uint32_t insns_size;
memcpy(&insns_size, ci + 12, sizeof(insns_size));
size_t insns_len = insns_size * 2;
uintptr_t insns_abs = dex.base + code_item_off + 16;

Writing through /proc/self/mem

The write is the interesting part:

int fd = open("/proc/self/mem", O_RDWR);
if (fd >= 0) {
    lseek(fd, static_cast<off_t>(insns_abs), SEEK_SET);
    write(fd, decrypted, insns_len);
    close(fd);
}

A direct store would fault — the mapping was filtered to r--p or r-xp, so it has no PROT_WRITE. mprotect is the obvious alternative.

It is also the loud one. It rewrites the permission bits and, on a sub-range, splits the VMA, so anything sampling /proc/self/maps while the window is open sees a writable dex mapping.

The write here goes through access_remote_vm() with FOLL_FORCE, which bypasses the PTE write protection on a private mapping by breaking COW. No VMA permission changes, so the maps line reads the same before and after.

The region is MAP_PRIVATE, so the forced write is copy-on-write — only the writing process would see the plaintext. The file on disk is never touched, so pulling the APK still yields the encrypted DEX.

Step 4 — the cipher

The key in source is two halves combined at runtime:

static const uint8_t KEY_HALF_1[16] = {
    0x4B,0x3A,0x2C,0x1D,0x8E,0x7F,0x60,0x51,
    0xF2,0xE3,0xD4,0xC5,0xB6,0xA7,0x98,0x89
};
static const uint8_t KEY_HALF_2[16] = {
    0x12,0x34,0x56,0x78,0x9A,0xBC,0xDE,0xF0,
    0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88
};
...
for (int i = 0; i < 16; i++) xor_key[i] = KEY_HALF_1[i] ^ KEY_HALF_2[i];

None of that survives -O2. Both arrays are static const, never address-taken, and consumed only by that loop — so LLVM unrolls it, folds all sixteen XORs at compile time, and dead-strips the sources. What remains is a single 128-bit load:

    1ac4: 3dc1c140     ldr     q0, [x10, #1792]

Grepping the 5,632-byte library for either half, or even their 4-byte prefixes, finds nothing. Only the folded result exists — file offset 0x700, the first object in .rodata:

00000700: 590e 7a65 14c3 bea1 e3c1 e781 e3c1 ef01  Y.ze............
00000710: 2f70 726f 632f 7365 6c66 2f6d 6170 7300  /proc/self/maps.
00000720: 636f 6d2f 686f 6c6c 6f77 6465 782f 5665  com/hollowdex/Ve
00000730: 7269 6669 6572 0072 2d78 7000 256c 782d  rifier.r-xp.%lx-

The two halves are a decoy that costs a reverser nothing, because the optimizer already removed them. Source-level obfuscation below the compiler's abstraction level is free to delete.

XOR the 312 bytes with key[i % 16], write them back into a copy of classes.dex, and baksmali renders the method — it checks the magic and the endian tag, not the two header fields. Repairing those is what a dex needs to run, not to disassemble. The core:

const-wide v1, 0x243f6a8885a308d3L
add-long  v9, v5, v1
const/16  p1, 0x11          # 17
shl-long  v11, v9, p1
const/16  v3, 0x2f          # 47
ushr-long/2addr v9, v3
or-long/2addr  v9, v11      # ROTL64(R+K0, 17)
xor-long/2addr v7, v9
xor-long/2addr v1, v7       # ^ K0

Three round constants:

K0 = 0x243F6A8885A308D3
K1 = 0x13198A2E03707344
K2 = 0xA4093822299F31D0

Those are the first three 64-bit words of the fractional part of pi in hex — the Blowfish P-array initializers, taken in consecutive 64-bit chunks. Recognizing them is the point: the round keys are public constants, so there is no key schedule and nothing secret to recover.

The round function is Add, Rotate, Xor:

def rotl64(v, n): return ((v << n) | (v >> (64 - n))) & M
def F(R, Ki):     return rotl64((R + Ki) & M, 17) ^ Ki

and the structure is a balanced Feistel on a 128-bit block — three rounds, new_R = L ^ F(R, Ki) and new_L = R. Input is 32 hex characters decoded to 16 bytes, packed big-endian into two 64-bit halves.

The final test compares against two constants with a non-short-circuit &. That is why the smali ends in and-int/2addr on two materialized booleans instead of a branch.

The two constants are where the challenge closes:

EXP_L = 0x554D47C35CF8EA95
EXP_R = 0xB4BC6CAE39400B7A

SHA-256("hollowdex") = 554d47c35cf8ea95 b4bc6cae39400b7a dbe08223b8cea01d...
                       |___ EXP_L ____| |___ EXP_R ____|

The string "hollowdex" is already in the DEX pool as the loadLibrary argument, which is the only hint tying the constants to a preimage.

A Feistel round is a bijection regardless of what F does, because the old L is recovered as new_R ^ F(new_L, K). With public round keys and three rounds there is nothing to attack. Run it backwards:

def inverse_arx(L, R):
    for Ki in reversed(K):
        R, L = L, (R ^ F(L, Ki)) & M
    return L, R

That gives the 16-byte preimage verifyFlag accepts. Swapping the two constants yields a different preimage and no complaint from anything, so it is worth running the cipher forward on the result and checking that it lands back on EXP_L and EXP_R. The flag is SHA-256 of those bytes, so forcing the comparison to return true would never produce it — the preimage is the answer, not the verdict.

Step 5 — solve.py

The static path is entirely offline — unzip, walk the DEX, decrypt, invert, hash.

Two values are literals, because each was recovered once by hand: the folded key out of .rodata, and the round constants out of the decrypted smali. The target ciphertext is not one of them — the script recomputes it as SHA-256("hollowdex"), which is what those two const-wide values spell out. The decryption stage is there to show its work, printing the first 32 bytes of plaintext, but nothing downstream consumes it.

#!/usr/bin/env python3
# solve.py <hollowdex.apk>
import sys, struct, hashlib, zipfile

# XOR_KEY = first 16 bytes of .rodata in libhollowdex.so (pre-computed KEY_HALF_1 ^ KEY_HALF_2)
XOR_KEY = bytes.fromhex("590e7a6514c3bea1e3c1e781e3c1ef01")

TARGET_CLASS  = b'Lcom/hollowdex/Verifier;'
TARGET_METHOD = b'verifyFlag'

# round keys: consecutive 64-bit chunks of pi hex fractional part
K = [0x243F6A8885A308D3, 0x13198A2E03707344, 0xA4093822299F31D0]
M = 0xFFFFFFFFFFFFFFFF


def rotl64(v, n):
    return ((v << n) | (v >> (64 - n))) & M

def F(R, Ki):
    return rotl64((R + Ki) & M, 17) ^ Ki

def inverse_arx(L, R):
    for Ki in reversed(K):
        R, L = L, (R ^ F(L, Ki)) & M
    return L, R


def u32(data, off): return struct.unpack_from('<I', data, off)[0]
def u16(data, off): return struct.unpack_from('<H', data, off)[0]

def read_uleb128(data, off):
    result, shift = 0, 0
    while True:
        b = data[off]; off += 1
        result |= (b & 0x7F) << shift
        if not (b & 0x80):
            return result, off
        shift += 7

def parse_string(data, string_ids_off, idx):
    ptr = u32(data, string_ids_off + idx * 4)
    _, off = read_uleb128(data, ptr)
    end = data.index(b'\x00', off)
    return data[off:end]


def find_code_item(raw):
    string_ids_off  = u32(raw, 0x3C)
    type_ids_size   = u32(raw, 0x40)
    type_ids_off    = u32(raw, 0x44)
    method_ids_size = u32(raw, 0x58)
    method_ids_off  = u32(raw, 0x5C)
    class_defs_size = u32(raw, 0x60)
    class_defs_off  = u32(raw, 0x64)

    target_type_idx = None
    for ti in range(type_ids_size):
        si = u32(raw, type_ids_off + ti * 4)
        if parse_string(raw, string_ids_off, si) == TARGET_CLASS:
            target_type_idx = ti
            break
    assert target_type_idx is not None, "Verifier class not found"

    target_method_idx = None
    for mi in range(method_ids_size):
        base = method_ids_off + mi * 8
        if u16(raw, base) != target_type_idx:
            continue
        if parse_string(raw, string_ids_off, u32(raw, base + 4)) == TARGET_METHOD:
            target_method_idx = mi
            break
    assert target_method_idx is not None, "verifyFlag not found"

    class_data_off = None
    for cd in range(class_defs_size):
        base = class_defs_off + cd * 32
        if u32(raw, base) == target_type_idx:
            class_data_off = u32(raw, base + 24)
            break
    assert class_data_off

    off = class_data_off
    sf, off = read_uleb128(raw, off)
    if_, off = read_uleb128(raw, off)
    dm, off = read_uleb128(raw, off)
    vm, off = read_uleb128(raw, off)
    for _ in range(sf + if_):
        _, off = read_uleb128(raw, off)
        _, off = read_uleb128(raw, off)

    idx_acc = 0
    for list_idx in range(dm + vm):
        if list_idx == dm:
            idx_acc = 0
        diff, off = read_uleb128(raw, off)
        idx_acc += diff
        _, off = read_uleb128(raw, off)
        ci, off = read_uleb128(raw, off)
        if idx_acc == target_method_idx:
            return ci
    raise RuntimeError("code_item not found")


def solve(apk_path):
    with zipfile.ZipFile(apk_path) as z:
        raw = bytearray(z.read('classes.dex'))

    ci_off = find_code_item(raw)
    insns_size = u32(raw, ci_off + 12)
    insns_off  = ci_off + 16
    insns_len  = insns_size * 2

    print(f"[*] code_item @ 0x{ci_off:08X}, insns_len={insns_len}")

    insns = bytearray(raw[insns_off:insns_off + insns_len])
    for i in range(len(insns)):
        insns[i] ^= XOR_KEY[i % 16]

    print(f"[*] decrypted {len(insns)} insn bytes")
    print(f"[*] first 32 bytes: {insns[:32].hex()}")

    # EXP_L || EXP_R = SHA-256("hollowdex")[0:16] -- visible in const-wide opcodes after decryption
    expected = hashlib.sha256(b"hollowdex").digest()[:16]
    L_exp = int.from_bytes(expected[:8], 'big')
    R_exp = int.from_bytes(expected[8:], 'big')

    L_flag, R_flag = inverse_arx(L_exp, R_exp)
    flag_bytes = L_flag.to_bytes(8, 'big') + R_flag.to_bytes(8, 'big')
    flag = hashlib.sha256(flag_bytes).hexdigest()
    print(f"[+] preimage : {flag_bytes.hex()}")
    print(f"[+] CTF{{{flag}}}")


if __name__ == '__main__':
    if len(sys.argv) != 2:
        sys.exit(f"usage: python3 {sys.argv[0]} <hollowdex.apk>")
    solve(sys.argv[1])

Expected output:

[*] code_item @ 0x0000062C, insns_len=312
[*] decrypted 312 insn bytes
[*] first 32 bytes: 1200380e9a006e1017000e000a01130220003221040029009000130110002312
[+] preimage : <32 hex chars>
[+] CTF{<64 hex chars>}

There is a shorter route, for a challenge packaged correctly. Let JNI_OnLoad do the work, then read the patched bytecode out of the mapped DEX:

// JNI_OnLoad has already run by the time the activity is up
Process.enumerateRanges("r--").forEach(function (r) {
    try {
        if (r.base.readU32() !== 0x0A786564) return;   // "dex\n" LE
        console.log(hexdump(r.base.add(0x062C + 16).readByteArray(312)));
    } catch (_) {}
});

Same 312 bytes, no key extraction, no reversing of the library at all — on a build where the decryptor lands.

This is not that build, and the last section explains why. The scan has its own problem too: it needs the DEX to start at a mapping base, which stops being true once the dex is served out of base.vdex.

A build that ignored build.gradle

The first APK shipped to the platform would not install:

Unable to install app com.hollowdex: INSTALL_FAILED_DEPRECATED_SDK_VERSION

Android 14 refuses any app whose targetSdkVersion is below 23. The confusing part is that app/build.gradle plainly says otherwise:

minSdk        34
targetSdk     34

Those values are injected into the manifest by the Android Gradle Plugin — and the plugin never runs. The build drives aapt2 link directly against a hand-written AndroidManifest.xml, with no SDK flags on the command line.

The manifest had no <uses-sdk> element, so the platform fell back to the framework default of 1.

aapt dump badging names the problem, in a roundabout way:

uses-permission: name='android.permission.WRITE_EXTERNAL_STORAGE'
uses-implied-permission: name='android.permission.WRITE_EXTERNAL_STORAGE' reason='targetSdkVersion < 4'
uses-permission: name='android.permission.READ_PHONE_STATE'
uses-implied-permission: name='android.permission.READ_PHONE_STATE' reason='targetSdkVersion < 4'
uses-permission: name='android.permission.READ_EXTERNAL_STORAGE'
uses-implied-permission: name='android.permission.READ_EXTERNAL_STORAGE' reason='requested WRITE_EXTERNAL_STORAGE'
supports-screens: 'normal'
supports-any-density: 'false'

An app that declares zero permissions is suddenly asking for three, and the reason string says why. The conclusive tell is quieter — the sdkVersion: and targetSdkVersion: lines are missing from the dump entirely. The fixed build has them:

sdkVersion:'34'
targetSdkVersion:'34'
supports-screens: 'small' 'normal' 'large' 'xlarge'
supports-any-density: 'true'
native-code: 'arm64-v8a'

The fix is three lines in the manifest:

<uses-sdk
    android:minSdkVersion="34"
    android:targetSdkVersion="34" />

Anything at or above 23 clears the install error — but 34 is the right number here, for a reason specific to this challenge.

The native library is compiled with aarch64-linux-android34-clang++, which sets __ANDROID_API__ to 34 and links against the API 34 sysroot, so a lower floor would be a lie about what the code links against.

It also settles the +0x08 question the wrong way. Android 14 is exactly where that offset is dex_method_index_, which for verifyFlag is 18.

The decryptor would then read insns_size from dex_base + 30, landing on the tail of the signature and the low half of file_size. That decodes to 250,874,976 code units — an allocation of 501,749,952 bytes.

It never gets that far. libhollowdex.so declares DT_NEEDED libc++_shared.so, which the APK does not bundle and which is not a public system library — the NDK driver was invoked directly, so nothing linked it statically. System.loadLibrary throws from Verifier's static initializer and the app dies at launch, on every version it can be installed on.

Past that there is a second blocker. classes.dex is stored deflated, so once the app is dexopt'd the mapping comes from base.vdex, whose maps line contains no .apk and never matches the filter.

None of this touches the challenge. The flag tip says static, the intended solve never loads the library, and the path above is offline from start to finish. But the runtime decryptor is not merely unused — it is the thing that would crash the app, and it was never run on a device.

Rebuilding with the manifest fixed left classes.dex and libhollowdex.so byte-identical to the broken build — the code_item offset, the XOR key and every number here survived unchanged. The resources did not: without <uses-sdk> aapt2 emits four layout variants and a 1,240-byte resources.arsc, so the broken APK had eleven entries instead of eight.