Vladyslav Pelykh
Back to home

~ $ ./hard_mode --verbose▌

HARD MODE

The hardest things I do on my stack: rebuilding APIs that were never published, reading code that was written not to be read, and keeping money and real-time traffic correct under pressure.

27
challenges
8
categories
10
security
521
lines of code

All reverse engineering is done on my own accounts, authorized integrations or with the owner’s consent.

guest@pelykh.lab — zshInteractive shell — type help

Threat matrix

Click a cell to open the challenge

security
reverse
fintech
distributed
data
realtime
performance
media

Security audit

OWASP Top 10 · CWE · NIST SP 800-63B

Findings from a full-stack security audit I ran on a production platform I lead — how each one was found, why it mattered and how it was closed. Details are anonymized; fixes are rewritten from scratch.

47

findings

  • critical8
  • high14
  • medium17
  • low8
surface
Account onboarding
impact
Lateral privilege escalation to full access over creatives and team data.
root cause
The approval flow blocked only one top role. Any approver could grant other roles with near-admin reach, so a single compromised approver account could mint privileged users.
fix
Elevated roles became a declared set; granting any of them requires a higher-tier approver and lands in the audit log.
approve-application.ts
1
2
3
4
5
6
7
8
const ELEVATED = new Set<Role>(['admin', 'head', 'team_lead', 'team_lead_design'])

export function assertCanGrant(approver: Actor, requested: Role) {
  if (!ELEVATED.has(requested)) return
  if (!can(approver, P.ROLES_GRANT_ELEVATED)) {
    throw new ForbiddenException(`${approver.role} cannot grant ${requested}`)
  }
}
surface
S2S revenue postbacks
impact
Lost revenue events: FTD and redeposits missing from the tracker and from payout reconciliation.
root cause
Parallel deposit postbacks for the same click competed for a lock. After a fixed number of retries the losing request returned success and was never processed or logged.
fix
Every attempt is logged before locking; lock misses go to a retry queue with exponential backoff instead of being discarded.
deposit.handler.ts
1
2
3
4
5
6
7
8
9
10
11
await this.ch.insert('deposit_attempts', [toAttemptRow(pb)])

const result = await this.locks.tryWithLock(`s2s:${pb.clickId}`, () => this.apply(pb))
if (result.acquired) return result.value

await this.retryQueue.add('deposit', pb, {
  jobId: `dep:${pb.clickId}:${pb.externalId}`,
  attempts: 8,
  backoff: { type: 'exponential', delay: 1_000 },
})
return { queued: true }
surface
Authentication
impact
Persistent account takeover that survives the victim refreshing or logging in again.
root cause
Refresh did not rotate the token and kept no lineage, so a captured token and the legitimate one lived side by side.
fix
Rotation on every refresh, token families with reuse detection, and a short grace window for parallel tabs.
refresh.ts
1
2
3
4
5
6
7
8
9
10
const record = await sessions.find(hash(token))
if (!record) throw new Unauthorized()

if (record.rotatedTo) {
  if (Date.now() - record.rotatedAt < 30_000) return sessions.issueFor(record.rotatedTo)
  await sessions.revokeFamily(record.familyId)
  throw new Unauthorized('refresh token reuse')
}

return sessions.rotate(record)
surface
Login
impact
Targeted credential stuffing and phishing against confirmed accounts.
root cause
Unknown emails returned in milliseconds, known ones only after a password hash comparison. Response time revealed which accounts exist.
fix
The same hash comparison runs for unknown users against a dummy hash; responses are identical in body and timing.
login.ts
1
2
3
4
5
6
7
const user = await users.findByEmail(input.email)
const ok = await bcrypt.compare(input.password, user?.passwordHash ?? DUMMY_HASH)

if (!user || !ok) {
  await failedLogins.record(input.email, req.ip)
  throw new Unauthorized('invalid credentials')
}
surface
Knowledge base renderer
impact
Script or CSS injection in pages viewed by administrators.
root cause
Rich content was rendered with v-html after a hand-rolled markdown pass and a sanitizer configured to allow inline styles.
fix
A real markdown parser with HTML disabled, a strict sanitizer allow-list, and CSP as the second line of defense.
render-content.ts
1
2
3
4
5
6
7
8
9
const md = new MarkdownIt({ html: false, linkify: true })

export function renderSafe(source: string) {
  return DOMPurify.sanitize(md.render(source), {
    ALLOWED_TAGS: ['p', 'a', 'strong', 'em', 'ul', 'ol', 'li', 'code', 'pre', 'h2', 'h3', 'blockquote', 'img'],
    ALLOWED_ATTR: ['href', 'src', 'alt', 'title'],
    ALLOWED_URI_REGEXP: /^(?:https:|\/)/,
  })
}
surface
Support tickets
impact
Any logged-in user could read private conversations by changing an id.
root cause
The messages endpoint checked authentication but not ownership of the ticket.
fix
Ownership or support-role check before every read, plus row-level security as a backstop.
ticket-messages.ts
1
2
3
4
5
6
7
8
export async function listMessages(actor: Actor, ticketId: string) {
  const ticket = await tickets.findById(ticketId)
  if (!ticket) throw new NotFoundException()
  if (ticket.authorId !== actor.id && !can(actor, P.SUPPORT_READ_ALL)) {
    throw new NotFoundException()
  }
  return messages.byTicket(ticketId)
}
surface
Permission management
impact
Leaking another team’s access profile and granting rights outside the actor’s scope.
root cause
A bulk "copy permissions from user" action validated the target but never checked that the source belonged to the same team.
fix
Both users must share the actor’s team, and copied permissions are capped at what the actor can grant.
copy-permissions.ts
1
2
3
4
5
6
7
8
const [target, source] = await users.byIds([targetId, sourceId])
if (!target || !source) throw new NotFoundException()
if (target.teamId !== actor.teamId || source.teamId !== actor.teamId) {
  throw new ForbiddenException('cross-team copy')
}

const grantable = source.permissions.filter(p => canGrant(actor, p))
await permissions.replace(target.id, grantable, { auditBy: actor.id })
surface
File uploads
impact
Writing outside the upload directory with a crafted file name.
root cause
File names were sanitized first and normalized afterwards. Decomposed characters passed the filter and recombined into forbidden sequences.
fix
Normalize first, then sanitize; generate storage names server-side and resolve the final path against the root.
upload-path.ts
1
2
3
4
5
6
7
8
9
export function storagePath(root: string, original: string) {
  const clean = sanitizeFilename(original.normalize('NFKC'))
  const ext = extname(clean).toLowerCase()
  if (!ALLOWED_EXT.has(ext)) throw new BadRequestException('file type')

  const full = resolve(root, `${randomUUID()}${ext}`)
  if (!full.startsWith(resolve(root) + sep)) throw new BadRequestException('path')
  return full
}
surface
Creative workflow
impact
Paid limits became unenforceable; unlimited tasks through a side door.
root cause
A secondary approval path created derived tasks through a lower-level repository call that skipped the limit check enforced on the main path.
fix
Limit enforcement moved into the single creation service every path must call, with a test asserting no other writer exists.
create-task.ts
1
2
3
4
5
6
export async function createTask(tx: Tx, actor: Actor, input: NewTask) {
  await limits.assertWithinQuota(tx, actor.teamId, input.kind)
  const task = await tx.insert(tasks).values({ ...input, teamId: actor.teamId }).returning()
  await history.record(tx, task[0]!, actor)
  return task[0]!
}

Arsenal

traffic

  • Chrome DevTools / CDP
  • mitmproxy
  • Burp Suite
  • Charles
  • Wireshark

binaries

  • Frida
  • JADX
  • apktool
  • Ghidra

javascript

  • Babel AST
  • source maps
  • webpack chunk graphs
  • Puppeteer
  • Playwright

protocols

  • Protobuf wire format
  • MTProto
  • WebSocket frames
  • HMAC request signing

Challenges · 27

why it is hard

No docs, no SDK, request shapes change between releases, and the only source of truth is what the web app sends from a signed-in session.

how I solve it

  1. 01Attach to the browser over CDP and record every XHR / fetch the UI fires for one user action.
  2. 02Group calls by route template, diff payloads across runs and infer a schema per endpoint.
  3. 03Generate Zod schemas from the samples so any upstream change fails loudly instead of corrupting data.
  4. 04Replay through a thin typed client with backoff, coverage checkpoints and a kill switch.
capture.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const cdp = await page.context().newCDPSession(page)
await cdp.send('Network.enable')

const calls = new Map<string, unknown[]>()

cdp.on('Network.responseReceived', async ({ requestId, response }) => {
  if (!response.mimeType.includes('json')) return
  const { body } = await cdp.send('Network.getResponseBody', { requestId })
  const route = new URL(response.url).pathname.replace(/\/\d+/g, '/:id')
  calls.set(route, [...(calls.get(route) ?? []), JSON.parse(body)])
})

await runUserFlow(page)

for (const [route, samples] of calls) {
  await writeFile(`schemas/${slug(route)}.ts`, inferZod(route, samples))
}

why it is hard

String-array rotation, proxy functions and control-flow flattening hide the one function you actually need to understand — the request signer.

how I solve it

  1. 01Parse the bundle into an AST instead of reading it as text.
  2. 02Locate the string-array decoder, evaluate it once in a sandbox and inline every call site.
  3. 03Constant-fold, remove dead branches and rename by usage until the signer reads like normal code.
  4. 04Port the signer to a typed module with test vectors captured from the real app.
deobfuscate.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const ast = parse(source, { sourceType: 'script' })
const decode = evalInSandbox(findDecoder(ast))

traverse(ast, {
  CallExpression(path) {
    const { callee, arguments: args } = path.node
    if (!isIdentifier(callee, { name: decode.name })) return
    if (!args.every(a => isNumericLiteral(a) || isStringLiteral(a))) return
    path.replaceWith(stringLiteral(decode.fn(...args.map(a => a.value))))
  },
  BinaryExpression: { exit: path => foldConstant(path) },
  IfStatement: { exit: path => pruneDeadBranch(path) },
})

return generate(ast, { comments: false }).code

why it is hard

Binary payloads over gRPC-web or WebSockets carry no field names. You only get tags, wire types and bytes.

how I solve it

  1. 01Walk the wire format by hand: varint key → field number and wire type.
  2. 02Recurse into length-delimited fields that themselves parse as valid messages.
  3. 03Correlate field numbers with what the UI renders to name them.
  4. 04Freeze the result into a .proto and generate a typed decoder.
wire.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
export function decode(buf: Uint8Array): Field[] {
  const out: Field[] = []
  let pos = 0
  const varint = () => {
    let shift = 0n, value = 0n, byte: number
    do {
      byte = buf[pos++]!
      value |= BigInt(byte & 0x7f) << shift
      shift += 7n
    } while (byte & 0x80)
    return value
  }

  while (pos < buf.length) {
    const key = Number(varint())
    const field = key >> 3
    const wire = key & 0x7
    if (wire === 0) out.push({ field, type: 'varint', value: varint() })
    else if (wire === 2) {
      const len = Number(varint())
      const bytes = buf.subarray(pos, (pos += len))
      out.push({ field, type: 'bytes', value: tryDecode(bytes) ?? bytes })
    }
    else if (wire === 1) out.push({ field, type: 'fixed64', value: buf.subarray(pos, (pos += 8)) })
    else if (wire === 5) out.push({ field, type: 'fixed32', value: buf.subarray(pos, (pos += 4)) })
    else throw new Error(`unsupported wire type ${wire}`)
  }
  return out
}

why it is hard

Application-level checks fail under concurrency, partial deploys and manual SQL. Money has to be correct even when the code is wrong.

how I solve it

  1. 01Append-only journal: every movement is a set of postings that sum to zero.
  2. 02A deferred constraint trigger validates balance at COMMIT, not per row.
  3. 03UPDATE, DELETE and TRUNCATE are revoked and blocked by triggers.
  4. 04Idempotency key per transaction, amounts as bigint with fixed 18-decimal scale.
ledger.sql
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
CREATE FUNCTION assert_balanced() RETURNS trigger AS $$
BEGIN
  IF (SELECT SUM(amount) FROM postings WHERE tx_id = NEW.tx_id) <> 0 THEN
    RAISE EXCEPTION 'transaction % is not balanced', NEW.tx_id;
  END IF;
  RETURN NULL;
END $$ LANGUAGE plpgsql;

CREATE CONSTRAINT TRIGGER postings_balanced
  AFTER INSERT ON postings
  DEFERRABLE INITIALLY DEFERRED
  FOR EACH ROW EXECUTE FUNCTION assert_balanced();

CREATE TRIGGER postings_immutable
  BEFORE UPDATE OR DELETE ON postings
  FOR EACH ROW EXECUTE FUNCTION reject_mutation();

why it is hard

Media buyers slice ROI by any dimension in real time. Naive GROUP BY over raw events either times out or takes the cluster down.

how I solve it

  1. 01Raw clicks land in a ReplacingMergeTree partitioned by month with a dedup-safe sort key.
  2. 02Materialized views pre-aggregate into AggregatingMergeTree state per hour and dimension.
  3. 03Reports merge states, never scan raw rows.
  4. 04Every query runs with max_memory_usage and max_execution_time caps.
rollup.sql
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
CREATE TABLE clicks_hourly
(
  hour DateTime,
  campaign_id UInt32,
  geo LowCardinality(String),
  clicks AggregateFunction(uniqExact, UInt64),
  revenue AggregateFunction(sum, Decimal(18, 6))
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(hour)
ORDER BY (campaign_id, geo, hour);

CREATE MATERIALIZED VIEW clicks_hourly_mv TO clicks_hourly AS
SELECT toStartOfHour(ts) AS hour, campaign_id, geo,
       uniqExactState(click_id) AS clicks,
       sumState(revenue) AS revenue
FROM clicks
GROUP BY hour, campaign_id, geo;

why it is hard

A tracker redirect sits in the user’s critical path. Every millisecond and every allocation shows up as lost conversions at peak.

how I solve it

  1. 01Redirect first, record later: the handler never waits on storage.
  2. 02Pooled buffers and a bounded channel absorb bursts without unbounded memory.
  3. 03A single writer batches inserts into ClickHouse by size or time.
  4. 04Back-pressure drops to a local spool instead of blocking the redirect.
click.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
var events = make(chan Click, 1<<16)

func handleClick(w http.ResponseWriter, r *http.Request) {
	c := clickPool.Get().(*Click)
	parseInto(c, r)
	http.Redirect(w, r, route(c), http.StatusFound)

	select {
	case events <- *c:
	default:
		spool.Write(c)
	}
	clickPool.Put(c)
}

func writer(ctx context.Context, ch *clickhouse.Conn) {
	batch := make([]Click, 0, 5000)
	tick := time.NewTicker(time.Second)
	for {
		select {
		case c := <-events:
			if batch = append(batch, c); len(batch) == cap(batch) {
				flush(ctx, ch, &batch)
			}
		case <-tick.C:
			flush(ctx, ch, &batch)
		case <-ctx.Done():
			flush(context.Background(), ch, &batch)
			return
		}
	}
}

why it is hard

Live odds tick many times a second. Emitting every update to every socket across nodes saturates Redis pub/sub and the event loop.

how I solve it

  1. 01Shard rooms by market, subscribe nodes only to rooms they hold.
  2. 02Coalesce updates per room into one frame every 100 ms.
  3. 03Send deltas with a sequence number; clients resync on a gap.
  4. 04Track per-socket buffered bytes and drop slow consumers instead of the cluster.
fanout.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
const pending = new Map<string, Delta[]>()

bus.on('odds', (d: Delta) => {
  const list = pending.get(d.room)
  list ? list.push(d) : pending.set(d.room, [d])
})

setInterval(() => {
  for (const [room, deltas] of pending) {
    const frame = { seq: nextSeq(room), d: compact(deltas) }
    for (const socket of io.sockets.adapter.rooms.get(room) ?? []) {
      const s = io.sockets.sockets.get(socket)
      if (!s || s.conn.transport.writable === false) continue
      if (bufferedBytes(s) > MAX_BUFFER) s.disconnect(true)
      else s.emit('odds', frame)
    }
  }
  pending.clear()
}, 100)

why it is hard

The signature is computed in native code behind certificate pinning. Static analysis alone gives thousands of functions and no names.

how I solve it

  1. 01Bypass pinning on a test device and confirm traffic flows through the proxy.
  2. 02Hook javax.crypto.Mac and the JNI boundary to see exactly what bytes get signed.
  3. 03Correlate inputs with request fields until the canonical string is fully known.
  4. 04Reimplement the signer, then verify it bit-for-bit against captured traffic.
trace-mac.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Java.perform(() => {
  const Mac = Java.use('javax.crypto.Mac')
  const hex = (b) => Array.from(b, x => ('0' + (x & 0xff).toString(16)).slice(-2)).join('')
  const utf8 = (b) => Java.use('java.lang.String').$new(b, 'UTF-8')

  Mac.init.overload('java.security.Key').implementation = function (key) {
    send({ op: 'init', alg: this.getAlgorithm(), key: hex(key.getEncoded()) })
    return this.init(key)
  }

  Mac.doFinal.overload('[B').implementation = function (input) {
    const out = this.doFinal(input)
    send({ op: 'sign', input: utf8(input), mac: hex(out), stack: Thread.backtrace(this.context, Backtracer.FUZZY).map(DebugSymbol.fromAddress).slice(0, 6) })
    return out
  }
})

why it is hard

Fixed windows let clients send 2× the limit across a boundary, and read-modify-write in app code races across nodes.

how I solve it

  1. 01Use GCRA: store one timestamp per key — the theoretical arrival time.
  2. 02Run the whole check atomically inside Redis with a Lua script.
  3. 03Return retry-after so clients back off precisely instead of hammering.
  4. 04Key by tenant + route, so one noisy integration cannot starve the others.
gcra.lua
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
-- KEYS[1] limiter key · ARGV: now_ms, period_ms, limit, cost
local now = tonumber(ARGV[1])
local interval = tonumber(ARGV[2]) / tonumber(ARGV[3])
local burst = tonumber(ARGV[2])
local cost = tonumber(ARGV[4]) * interval

local tat = tonumber(redis.call('GET', KEYS[1]) or now)
local new_tat = math.max(tat, now) + cost
local allow_at = new_tat - burst

if allow_at > now then
  return { 0, math.ceil(allow_at - now) }
end

redis.call('SET', KEYS[1], new_tat, 'PX', math.ceil(new_tat - now))
return { 1, 0 }

why it is hard

A worker holding a Redis lock can freeze, lose the lease, wake up and still write — corrupting data another worker already owns.

how I solve it

  1. 01Every lock acquisition returns a monotonically increasing fencing token.
  2. 02The storage layer rejects writes carrying a token older than the last seen one.
  3. 03Lease TTL plus heartbeat extension, with release only by the owner.
  4. 04Treat the lock as an optimization; the fence is what guarantees safety.
fencing.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
async function withLease<T>(resource: string, fn: (token: bigint) => Promise<T>) {
  const token = BigInt(await redis.incr(`fence:${resource}`))
  const owner = randomUUID()
  const ok = await redis.set(`lock:${resource}`, owner, 'PX', 15_000, 'NX')
  if (ok !== 'OK') throw new LockBusyError(resource)

  const beat = setInterval(() => redis.eval(EXTEND_IF_OWNER, 1, `lock:${resource}`, owner, 15_000), 5_000)
  try {
    return await fn(token)
  }
  finally {
    clearInterval(beat)
    await redis.eval(RELEASE_IF_OWNER, 1, `lock:${resource}`, owner)
  }
}

await withLease(`click:${clickId}`, token => db.execute(sql`
  UPDATE click_state SET data = ${next}, fence = ${token}
  WHERE click_id = ${clickId} AND fence < ${token}`))

why it is hard

Writing to the database and then publishing a job is two systems. A crash between them silently drops or duplicates revenue events.

how I solve it

  1. 01Write the business row and the outbox row in one Postgres transaction.
  2. 02A relay claims outbox rows with FOR UPDATE SKIP LOCKED and enqueues them.
  3. 03Jobs carry the outbox id as their BullMQ jobId, so re-delivery is a no-op.
  4. 04LISTEN / NOTIFY wakes the relay instantly; polling is only the safety net.
outbox.relay.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
async function relay() {
  await db.transaction(async (tx) => {
    const rows = await tx.execute<OutboxRow>(sql`
      SELECT id, topic, payload FROM outbox
      WHERE published_at IS NULL
      ORDER BY id
      LIMIT 500
      FOR UPDATE SKIP LOCKED`)

    if (!rows.length) return

    await Promise.all(rows.map(r =>
      queues[r.topic].add(r.topic, r.payload, { jobId: `outbox:${r.id}`, removeOnComplete: 10_000 }),
    ))

    await tx.execute(sql`
      UPDATE outbox SET published_at = now()
      WHERE id = ANY(${rows.map(r => r.id)})`)
  })
}

pg.on('notification', relay)
await pg.query('LISTEN outbox')
setInterval(relay, 5_000)

why it is hard

ALTER on a hot table takes locks that stall every write. Rolling back a half-done migration on live money data is not an option.

how I solve it

  1. 01Expand: create the new table and dual-write via trigger.
  2. 02Backfill in primary-key batches with throttling against replication lag.
  3. 03Verify with chunked checksums, then switch reads behind a flag.
  4. 04Contract: stop dual-writes and drop the old table only after a soak period.
backfill.sql
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE TRIGGER clicks_dual_write
  AFTER INSERT OR UPDATE ON clicks
  FOR EACH ROW EXECUTE FUNCTION copy_to_clicks_v2();

WITH batch AS (
  SELECT id FROM clicks
  WHERE id > :last_id
  ORDER BY id
  LIMIT 10000
)
INSERT INTO clicks_v2 (id, click_id, campaign_id, geo, created_at)
SELECT c.id, c.click_id, c.campaign_id, c.geo, c.created_at
FROM clicks c JOIN batch USING (id)
ON CONFLICT (id) DO NOTHING
RETURNING id;

why it is hard

Mobile clients drop SSE connections constantly. Reconnecting must replay exactly what was missed — no gaps, no duplicates.

how I solve it

  1. 01Every event goes to a capped Redis Stream; its id is the SSE event id.
  2. 02On reconnect the browser sends Last-Event-ID automatically.
  3. 03The server replays with XRANGE from that id, then switches to blocking XREAD.
  4. 04If the id was trimmed, send a full snapshot first, then continue live.
stream.get.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
export default defineEventHandler(async (event) => {
  const key = `job:${getRouterParam(event, 'id')}:events`
  const stream = createEventStream(event)
  let cursor = getHeader(event, 'last-event-id') ?? '0'

  const [oldest] = await redis.xrange(key, '-', '+', 'COUNT', 1)
  if (cursor !== '0' && oldest && cursor < oldest[0]) {
    await stream.push({ event: 'snapshot', data: JSON.stringify(await loadSnapshot(key)) })
    cursor = '$'
  }

  const reader = redis.duplicate()
  stream.onClosed(() => reader.disconnect())

  ;(async () => {
    while (true) {
      const res = await reader.xread('BLOCK', 15_000, 'STREAMS', key, cursor)
      for (const [id, fields] of res?.[0]?.[1] ?? []) {
        await stream.push({ id, event: 'progress', data: fields[1]! })
        cursor = id
      }
    }
  })()

  return stream.send()
})

why it is hard

Server-side FFmpeg for every preview and export burns money. The browser can encode — if you keep the main thread free and memory bounded.

how I solve it

  1. 01Composite each frame on an OffscreenCanvas inside a Web Worker.
  2. 02Encode with WebCodecs VideoEncoder using hardware acceleration when available.
  3. 03Apply back-pressure on encodeQueueSize so memory never grows unbounded.
  4. 04Mux chunks into MP4 in the worker and transfer the final buffer once.
render.worker.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
const canvas = new OffscreenCanvas(width, height)
const ctx = canvas.getContext('2d')!
const muxer = new Muxer({ target: new ArrayBufferTarget(), video: { codec: 'avc', width, height }, fastStart: 'in-memory' })

const encoder = new VideoEncoder({
  output: (chunk, meta) => muxer.addVideoChunk(chunk, meta),
  error: e => postMessage({ type: 'error', message: e.message }),
})
encoder.configure({ codec: 'avc1.640028', width, height, bitrate: 6_000_000, framerate: fps, hardwareAcceleration: 'prefer-hardware' })

for (let i = 0; i < totalFrames; i++) {
  drawTimeline(ctx, timeline, i / fps)
  const frame = new VideoFrame(canvas, { timestamp: Math.round((i * 1e6) / fps) })
  encoder.encode(frame, { keyFrame: i % (fps * 2) === 0 })
  frame.close()

  while (encoder.encodeQueueSize > 8) await new Promise(r => setTimeout(r, 0))
  if (i % fps === 0) postMessage({ type: 'progress', value: i / totalFrames })
}

await encoder.flush()
muxer.finalize()
const { buffer } = muxer.target
postMessage({ type: 'done', buffer }, [buffer])

why it is hard

One user queuing 200 videos must not block everyone else, GPUs are rented by the hour, and a job can die halfway through a 3-minute render.

how I solve it

  1. 01Per-tenant queues served round-robin, weighted by plan.
  2. 02Workers lease jobs with a heartbeat; expired leases go back to the queue.
  3. 03Keep model weights warm per GPU and route jobs to a GPU that already has them loaded.
  4. 04Scale rented GPUs on queue depth × average job time, not on CPU.
scheduler.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
func (s *Scheduler) Next(gpu *GPU) (*Job, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()

	for range len(s.ring) {
		tenant := s.ring[s.cursor]
		s.cursor = (s.cursor + 1) % len(s.ring)

		q := s.queues[tenant]
		if q.Len() == 0 || s.running[tenant] >= tenant.Concurrency {
			continue
		}
		job := q.PopPreferring(func(j *Job) bool { return gpu.Warm(j.Model) })
		s.running[tenant]++
		job.Lease = Lease{GPU: gpu.ID, Until: time.Now().Add(30 * time.Second)}
		return job, true
	}
	return nil, false
}

func (s *Scheduler) reap(now time.Time) {
	for _, job := range s.leased() {
		if now.After(job.Lease.Until) {
			s.requeue(job)
		}
	}
}

why it is hard

Players assume the house cheats. The operator must commit to randomness before the bet and prove afterwards that it did not change it.

how I solve it

  1. 01Publish SHA-256 of the server seed before any bet is placed.
  2. 02Each outcome is HMAC-SHA512(serverSeed, clientSeed:nonce).
  3. 03Derive the number with rejection sampling so there is no modulo bias.
  4. 04Reveal the seed on rotation; anyone can recompute every past round.
fair.ts
1
2
3
4
5
6
7
8
9
10
11
12
export function roll(serverSeed: string, clientSeed: string, nonce: number, max: number) {
  const digest = createHmac('sha512', serverSeed).update(`${clientSeed}:${nonce}`).digest()
  const limit = Math.floor(2 ** 32 / max) * max

  for (let offset = 0; offset + 4 <= digest.length; offset += 4) {
    const n = digest.readUInt32BE(offset)
    if (n < limit) return n % max
  }
  return roll(serverSeed, clientSeed, nonce + 1_000_000, max)
}

export const commitment = (serverSeed: string) => createHash('sha256').update(serverSeed).digest('hex')

why it is hard

Affiliate partners configure their own postback URLs. One URL pointing at 169.254.169.254 or an internal admin panel turns your tracker into a proxy into your own network.

threat model

Attacker-controlled URL → server-side request to cloud metadata or internal services; DNS rebinding flips a public hostname to 127.0.0.1 between check and use.

how I solve it

  1. 01Allow only http/https and resolve every address the hostname maps to.
  2. 02Reject private, loopback, link-local, CGNAT and reserved ranges — v4 and v6.
  3. 03Pin the connection to the vetted IP so a second DNS answer cannot rebind it.
  4. 04Follow redirects manually and re-validate every hop; hard timeout and size limit.
safe-fetch.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const BLOCKED = new Set(['private', 'loopback', 'linkLocal', 'uniqueLocal', 'unspecified', 'reserved', 'carrierGradeNat', 'broadcast'])

export async function safeFetch(raw: string, init: RequestInit = {}, hops = 0): Promise<Response> {
  const url = new URL(raw)
  if (url.protocol !== 'https:' && url.protocol !== 'http:') throw new SsrfError('scheme')

  const addrs = await lookup(url.hostname, { all: true, verbatim: true })
  for (const { address } of addrs) {
    const range = ipaddr.process(address).range()
    if (BLOCKED.has(range)) throw new SsrfError(`${address} is ${range}`)
  }

  const res = await fetch(url, {
    ...init,
    redirect: 'manual',
    dispatcher: pinnedAgent(url.hostname, addrs[0]!.address),
    signal: AbortSignal.timeout(5_000),
  })

  const location = res.headers.get('location')
  if (res.status >= 300 && res.status < 400 && location) {
    if (hops >= 3) throw new SsrfError('redirect limit')
    return safeFetch(new URL(location, url).toString(), init, hops + 1)
  }
  return res
}

why it is hard

Payment and conversion webhooks move money. An unsigned endpoint can be spoofed; a signed one can still be replayed a thousand times.

threat model

Forged FTD events inflate payouts; captured legitimate requests are replayed to double-credit.

how I solve it

  1. 01HMAC-SHA256 over timestamp + raw body, never over parsed JSON.
  2. 02Constant-time comparison to avoid timing oracles.
  3. 03Reject timestamps outside a 5-minute window.
  4. 04Store the event id with SET NX for the tolerance window — duplicates are acknowledged, not processed.
verify-webhook.ts
1
2
3
4
5
6
7
8
9
10
11
12
export async function verifyWebhook(req: FastifyRequest, secret: string) {
  const ts = Number(req.headers['x-signature-ts'])
  const sig = Buffer.from(String(req.headers['x-signature'] ?? ''), 'hex')
  if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > 300) throw new Unauthorized('stale')

  const expected = createHmac('sha256', secret).update(`${ts}.`).update(req.rawBody!).digest()
  if (sig.length !== expected.length || !timingSafeEqual(sig, expected)) throw new Unauthorized('signature')

  const { id } = WebhookEvent.parse(req.body)
  const fresh = await redis.set(`wh:${id}`, '1', 'EX', 600, 'NX')
  return { duplicate: fresh !== 'OK' }
}

why it is hard

Media buyers get phished constantly, and TOTP codes can be relayed in real time. Passkeys bind the credential to the origin, so a lookalike domain gets nothing.

threat model

Real-time phishing proxy captures password + OTP and hijacks the session.

how I solve it

  1. 01Issue a single-use challenge per ceremony, stored server-side with a short TTL.
  2. 02Verify origin, RP ID, user verification flag and signature against the stored public key.
  3. 03Track the signature counter to detect cloned authenticators.
  4. 04Keep TOTP only as a fallback behind step-up and alerting.
passkey.login.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
export async function finishLogin(sessionId: string, response: AuthenticationResponseJSON) {
  const challenge = await redis.getdel(`webauthn:challenge:${sessionId}`)
  if (!challenge) throw new Unauthorized('challenge expired')

  const cred = await db.query.passkeys.findFirst({ where: eq(passkeys.id, response.id) })
  if (!cred) throw new Unauthorized('unknown credential')

  const { verified, authenticationInfo } = await verifyAuthenticationResponse({
    response,
    expectedChallenge: challenge,
    expectedOrigin: env.ORIGIN,
    expectedRPID: env.RP_ID,
    requireUserVerification: true,
    credential: { id: cred.id, publicKey: cred.publicKey, counter: cred.counter },
  })
  if (!verified) throw new Unauthorized('signature')

  if (authenticationInfo.newCounter <= cred.counter && cred.counter !== 0) await flagClonedAuthenticator(cred)
  await db.update(passkeys).set({ counter: authenticationInfo.newCounter }).where(eq(passkeys.id, cred.id))
  return issueSession(cred.userId, { amr: ['hwk', 'user'] })
}

why it is hard

One forgotten WHERE team_id = ? in 60 modules leaks another team’s spend, creatives and payouts. Code review alone does not scale.

threat model

Authenticated user changes an id in the URL and reads or edits another tenant’s data.

how I solve it

  1. 01Enable Postgres row-level security on every tenant table.
  2. 02Each request runs in a transaction that sets app.team_id via set_config.
  3. 03Policies compare rows to current_setting — a missing filter returns nothing instead of everything.
  4. 04The app role has no BYPASSRLS; migrations run under a separate owner.
rls.sql
1
2
3
4
5
6
7
8
9
10
11
ALTER TABLE campaigns ENABLE ROW LEVEL SECURITY;
ALTER TABLE campaigns FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON campaigns
  USING (team_id = current_setting('app.team_id')::uuid)
  WITH CHECK (team_id = current_setting('app.team_id')::uuid);

BEGIN;
SELECT set_config('app.team_id', $1, true);
SELECT * FROM campaigns WHERE id = $2;
COMMIT;

why it is hard

Bots click ads to drain budgets or fake conversions. Each signal alone is noisy; together they separate humans from farms.

threat model

Datacenter traffic, emulators and click injection burn ad spend and poison ROI analytics.

how I solve it

  1. 01Enrich every click with ASN type, device fingerprint and time-to-click.
  2. 02Score velocity per IP, /24, ASN and fingerprint in sliding windows inside ClickHouse.
  3. 03Flag impossible patterns: conversion before install, sub-second click-to-install, identical fingerprints across GEOs.
  4. 04Feed scores back to the tracker to exclude sources before they cost more.
fraud-score.sql
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
SELECT *
FROM
(
  SELECT
    click_id,
    source_id,
    (asn_type = 'hosting') * 40
    + (count() OVER w_ip > 30) * 25
    + (uniqExact(geo) OVER w_fp > 3) * 20
    + (time_to_click_ms < 400) * 15
    + (headless_signals > 1) * 30 AS score
  FROM clicks
  WHERE ts >= now() - INTERVAL 10 MINUTE
  WINDOW
    w_ip AS (PARTITION BY ip_prefix ORDER BY ts RANGE BETWEEN 60 PRECEDING AND CURRENT ROW),
    w_fp AS (PARTITION BY fingerprint)
)
WHERE score >= 60

why it is hard

Leaked password lists get replayed from residential proxies — one attempt per IP, thousands of IPs. Per-IP limits never trigger.

threat model

Account takeover at scale using breached credentials and rotating proxies.

how I solve it

  1. 01Rate-limit on several keys at once: account, IP, /24, ASN and device fingerprint.
  2. 02Check passwords against breach corpora with k-anonymity — only a 5-char hash prefix leaves the server.
  3. 03Return identical responses and timing for unknown users and wrong passwords.
  4. 04Step-up to a passkey or email challenge on risk, not on every login.
breached.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
export async function isBreached(password: string) {
  const hash = createHash('sha1').update(password).digest('hex').toUpperCase()
  const prefix = hash.slice(0, 5)
  const suffix = hash.slice(5)

  const res = await fetch(`https://api.pwnedpasswords.com/range/${prefix}`, {
    headers: { 'Add-Padding': 'true' },
    signal: AbortSignal.timeout(2_000),
  })
  const body = await res.text()
  return body.split('\n').some(line => line.startsWith(suffix) && !line.endsWith(':0'))
}

const keys = [`acct:${email}`, `ip:${ip}`, `net:${prefix24(ip)}`, `asn:${asn}`, `fp:${fingerprint}`]
const verdicts = await Promise.all(keys.map(k => limiter.check(k)))
if (verdicts.some(v => !v.allowed)) return challenge(req)

why it is hard

API keys of ad accounts and payment credentials sit in the database. One leaked backup must not mean leaked secrets, and rotating the master key must not mean re-encrypting everything at once.

threat model

Database dump or backup exfiltration exposes third-party credentials.

how I solve it

  1. 01A fresh data key per record, wrapped by a master key in KMS.
  2. 02AES-256-GCM with the record id as additional authenticated data, so ciphertexts cannot be swapped between rows.
  3. 03Key id is stored in the ciphertext header — old keys decrypt, new writes use the current key.
  4. 04Background job lazily re-wraps data keys; plaintext never touches disk.
envelope.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
export async function seal(plaintext: string, aad: string) {
  const { keyId, plaintextKey, wrappedKey } = await kms.generateDataKey(CURRENT_KEY)
  const iv = randomBytes(12)
  const cipher = createCipheriv('aes-256-gcm', plaintextKey, iv).setAAD(Buffer.from(aad))
  const body = Buffer.concat([cipher.update(plaintext, 'utf8'), cipher.final()])
  plaintextKey.fill(0)
  return pack({ v: 1, keyId, wrappedKey, iv, tag: cipher.getAuthTag(), body })
}

export async function open(sealed: Buffer, aad: string) {
  const { keyId, wrappedKey, iv, tag, body } = unpack(sealed)
  const key = await kms.decrypt(keyId, wrappedKey)
  const decipher = createDecipheriv('aes-256-gcm', key, iv).setAAD(Buffer.from(aad)).setAuthTag(tag)
  try {
    return Buffer.concat([decipher.update(body), decipher.final()]).toString('utf8')
  }
  finally {
    key.fill(0)
  }
}

why it is hard

A compromised npm package or a hijacked GitHub Action runs with your deploy credentials. Most teams trust every dependency by default.

threat model

Malicious postinstall script or retagged Action exfiltrates secrets from CI.

how I solve it

  1. 01Pin every GitHub Action to a full commit SHA, not a tag.
  2. 02Install with a frozen lockfile and lifecycle scripts disabled by default.
  3. 03Verify registry signatures and provenance attestations before build.
  4. 04Short-lived OIDC credentials instead of long-lived cloud secrets in CI.
.github/workflows/build.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
permissions:
  contents: read
  id-token: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - run: corepack enable
      - run: pnpm install --frozen-lockfile --ignore-scripts
      - run: npm audit signatures
      - run: pnpm rebuild esbuild sharp
      - run: pnpm build
      - run: pnpm test --run

why it is hard

Libraries that trust the alg header accept "none" or verify an RS256 token with the public key as an HMAC secret. Both mean anyone can mint admin tokens.

threat model

Forged tokens via alg=none or RS256→HS256 confusion; tokens from another service accepted because aud was never checked.

how I solve it

  1. 01Pin the accepted algorithm list per issuer — never read it from the token.
  2. 02Fetch keys from JWKS by kid with caching and rotation.
  3. 03Validate iss, aud, exp, nbf with small clock tolerance.
  4. 04Short access-token TTL; revocation lives in refresh rotation, not in the JWT.
verify-jwt.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const jwks = createRemoteJWKSet(new URL(`${env.ISSUER}/.well-known/jwks.json`), {
  cooldownDuration: 30_000,
  cacheMaxAge: 600_000,
})

export async function verifyAccessToken(token: string) {
  const { payload } = await jwtVerify(token, jwks, {
    algorithms: ['EdDSA'],
    issuer: env.ISSUER,
    audience: 'api.admatrix',
    clockTolerance: 5,
    maxTokenAge: '15m',
  })
  return AccessClaims.parse(payload)
}

why it is hard

The signer moved from JS into a stripped .wasm module. No source, no names — just linear memory and a handful of imports.

how I solve it

  1. 01Disassemble to WAT and map exported functions and the import table.
  2. 02Instantiate the module with proxied imports that log every call and argument.
  3. 03Snapshot linear memory before and after the signing call to locate inputs and outputs.
  4. 04Reimplement in TypeScript and diff against the module on thousands of random inputs.
trace-wasm.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
const bytes = await readFile('signer.wasm')
const module = await WebAssembly.compile(bytes)

const imports = Object.fromEntries(
  WebAssembly.Module.imports(module).map(({ module: ns }) => [ns, {}]),
) as WebAssembly.Imports

for (const { module: ns, name, kind } of WebAssembly.Module.imports(module)) {
  if (kind !== 'function') continue
  imports[ns]![name] = (...args: number[]) => {
    trace.push({ fn: `${ns}.${name}`, args })
    return realImpl(ns, name)(...args)
  }
}

const { exports } = await WebAssembly.instantiate(module, imports)
const memory = exports.memory as WebAssembly.Memory
const before = new Uint8Array(memory.buffer).slice()
const ptr = (exports.sign as Function)(writeInput(memory, payload), payload.length)
const changed = diffRegions(before, new Uint8Array(memory.buffer))

why it is hard

Landing builders render user-supplied HTML and tracking pixels. Sanitizers get bypassed; the browser itself has to refuse to run injected script.

threat model

Stored XSS in a landing or creative steals admin sessions and rewrites payout settings.

how I solve it

  1. 01Per-response nonce with strict-dynamic — no allow-listed hosts, no unsafe-inline.
  2. 02Require Trusted Types so DOM sinks only accept values from a vetted policy.
  3. 03Sanitize once in the policy with DOMPurify, nowhere else.
  4. 04Report-only rollout first, then enforce with violation reports wired to alerts.
security-headers.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
export default defineEventHandler((event) => {
  const nonce = randomBytes(16).toString('base64')
  event.context.cspNonce = nonce

  setHeaders(event, {
    'Content-Security-Policy': [
      `script-src 'nonce-${nonce}' 'strict-dynamic'`,
      "object-src 'none'",
      "base-uri 'none'",
      "frame-ancestors 'none'",
      "require-trusted-types-for 'script'",
      'trusted-types app-html dompurify',
      'report-to csp',
    ].join('; '),
    'Cross-Origin-Opener-Policy': 'same-origin',
    'X-Content-Type-Options': 'nosniff',
  })
})

trustedTypes.createPolicy('app-html', {
  createHTML: input => DOMPurify.sanitize(input, { RETURN_TRUSTED_TYPE: false }),
})