A NOP Did Not Break My YARA Rule. One Encoding Change Did.
Security Engineering

A NOP Did Not Break My YARA Rule. One Encoding Change Did.

A byte-level investigation for detection engineers: why inserting a NOP leaves a YARA hex rule intact, while a single equivalent x86-64 encoding change defeats it — reproduced with Aether 1.0.0 as the reference implementation.

C
Christian Schnapka
Macstab Engineering
August 9, 2026
6 min read
Also on Medium Read on Medium

A signature pins one point in encoding space

A YARA hex rule is a syntactic predicate over a byte buffer. What it tries to catch is a semantic object, for example a function that returns 7. Between those two planes sits the defining property of x86-64: a variable-length encoding in which the map from byte strings to architectural effects is massively many-to-one. Setting RAX to 7 and returning has, conservatively, more than a dozen legal encodings. A signature pins one point in that space. A mutator's entire job is to walk to a different one. So the intuitive assumption — insert a NOP, change the hash and size, and the identity rule is defeated — measures the wrong plane. The question is never whether some bytes moved. It is whether the exact point the rule pins still occurs anywhere in the buffer, and whether the rule pins a point or a family. This is an x86-64 laboratory analysis for authorized detection evaluation, using Aether 1.0.0 with Zydis v4.1.1 as a reproducible reference implementation. It does not claim arbitrary .text semantics, general-purpose binary rewriting, or a bypass of any security product.
Diagram contrasting the identity plane (SHA-256, size, hex dump, YARA string match) with the semantic plane (multi-input oracle, simulator, native execution)
Identity plane versus semantic plane: a byte signature and a behavioural oracle observe different things.

Six bytes, one NOP, still a hit

The test function is six bytes: B8 07 00 00 00 C3, that is mov eax, 7; ret. The teaching rule matches exactly those bytes, contiguously. It is a lab-only rule, not a production signature; its value is that it makes the identity plane visible. The first SAFE capture with seed 42 produced 90 B8 07 00 00 00 C3: a single-byte NOP in front of the unchanged original. The hash changed, the size changed, the return value still holds, and the rule still matches, because the six-byte needle is intact one byte further on. The reason is mechanical. YARA reduces the pattern to short atoms, typically substrings of up to four bytes such as 07 00 00 00, places them in an Aho-Corasick automaton and runs the full verifier only at atom hits. The automaton is position-independent, so a one-byte shift changes nothing. The corollary: an identity rule's real strength is the quality of its rarest atom, not its length. Wildcarding the immediate as B8 ?? ?? ?? ?? C3 forces YARA to anchor on the 1-byte atoms B8 and C3, which saturate real .text and turn the rule into a slow, low-precision verify pass. So the conclusion that junk insertion defeated the signature is not acceptable without testing the rule itself. This experiment measured padding, not rule destruction.

                      { B8 07 00 00 00 C3 }   // contiguous identity
                    

One encoding change, and the rule misses

Seed 5 takes a different path through the same SAFE policy and emits 48 C7 C0 07 00 00 00 C3. The original string no longer occurs because the instruction changed encoding class: a REX prefix whose W bit selects 64-bit operand size, the C7 /0 MOV opcode group, a register-direct ModR/M selecting RAX, the same immediate 7, and ret. The result is effectively mov rax, 7; ret. The claim has a precise boundary. B8 zero-extends its 32-bit immediate into RAX, while REX.W + C7 /0 sign-extends it. For the positive immediate 7 both yield the same 64-bit value. Aether's pure_rax field, however, is a 32-bit return-value oracle. With an immediate such as 0x80000000 the two forms would differ in the upper 32 bits, and the oracle could not see it. The semantic claim is deliberately restricted to this positive-immediate shell and the declared 32-bit output domain. The seed alone does not determine the output either; a published capture must state a fresh process, the first seeded morph and the exact build. With that discipline, the comparison is clean:
  • Original B8 … C3: identity rule HIT, pure_rax 7
  • Seed 42 (90 + original): identity rule HIT, pure_rax 7 → 7 OK
  • Seed 5 (48 C7 C0 … C3): identity rule MISS, pure_rax 7 → 7 OK
Diagram of the original B8 encoding branching into the seed 42 NOP variant (rule hit) and the seed 5 REX variant (rule miss), both with pure RAX equal to 7
Same pure result, different encoding class: the NOP variant keeps the needle intact, the REX form removes it.

What makes the result credible

This is not an argument that detection is impossible. A rule that recognizes both encoding families can still match both. It is an argument that a single identity fragment is incomplete. Aether lifts each instruction with Zydis and classifies it as Control, Memory, Identity, PureRax or another class; only classified instructions enter a semantics-bounded substitution catalogue, and ret is never diversified as pure. Bytes Zydis cannot decode are kept as opaque, so a bad decode shrinks the transform surface instead of inventing a rewrite. Every output is re-lifted and checked by a 32-bit pure return comparison, a host-independent simulator with fixed multi-input pairs and, on x86-64 or in the amd64 Docker lab, native execution. In Lab or Industry mode a pure mismatch is fail-closed. The author-lab benchmark on Apple Silicon used 5,096 corpus entries (1,000 synthetic, 4,096 from ELF). It reports 10,192 successful morphs, 5,020 pure checks and 4,862 native checks with zero breaks, and 94.89% unique outputs by SHA-256. On the focused 128-seed matrix, the lab identity rule hit 30 times (23.44%) and missed 98 times (76.56%), with zero pure breaks. These are capture facts for that build and host, not permanent product numbers.
Flowchart of the Aether MorphEngine pipeline: lift, analyze, product-mode branch, diversify, expand, shuffle, split, permute, assemble and verify
The MorphEngine pipeline: every emitted variant passes through re-lifting and verification before it counts.

The layers that have to agree

The lesson is not to use more signatures, and it is not that morphing wins. It is to test rules against the plane they actually observe. The six-byte example crosses five layers, and treating any one of them as the whole system is how a convincing byte dump becomes a false claim. A structural re-lift cannot prove full machine semantics, a passing pure oracle cannot prove an EDR did not alert, and a YARA MISS cannot prove an encoding is absent from every detection family.
  • Decode: what bytes and control-flow graph did we actually lift?
  • Analysis: is this transformation permitted by the current model?
  • Lowering: do branches still target the intended logical blocks?
  • Oracle: did the stated output contract hold in this domain?
  • Detection: which observable changed, and which sensor still sees it?

Turning the experiment into detection CI

The useful pattern: generate legal semantic twins within a narrowly declared domain, run the rule and the oracle, retain the evidence, then repair the sensor that failed. Run the job on at least three planes: the semantic oracle rejects mutations that violate the declared contract, identity is recorded as exact bytes, size and hash, and the named detector runs with its exact rule. A ClamAV or EDR observation is a separate fourth measurement, not a replacement for the YARA result. Keep the exact lab YARA source, the JSON bench report, decoded before and after listings, and the detector version and date. Name the oracle rather than writing validated: Apple Silicon can run the simulation but not execute x86-64 natively, so use the amd64 Docker lab when hardware execution is required. When a rule fails, the repair should follow the plane. A contiguous byte rule may need encoding-family alternatives, context or a second signal; a rule covering both B8 ?? … C3 and 48 C7 C0 ?? … C3 may recognize both forms. A size or entropy detector may already see the roughly 2.19× mean expansion and only need calibration. A behavioural sensor should be tested against execution, allocation, timing and memory surfaces rather than inferred from a static hash.
Lab topology: author host building Aether with Zydis v4.1.1, a Linux amd64 container for native proof, and an optional scan container with ClamAV and YARA
Lab topology: native proof in an amd64 container, detector measurements in a separate scan container.

tags

#detection-engineering #yara #x86-64 #reverse-engineering #malware-analysis

conclusion

A NOP did not break the rule; one equivalent encoding did. The experiment demonstrates a weakness in one contiguous identity rule, not a universal detection failure, and it holds only inside its declared domain: a positive-immediate shell, a 32-bit return oracle, a named build and a named rule. The right response is not to promise invulnerability; it is to make a rule's assumptions executable and regression-tested. Aether is intended for authorized lab use under the Aether Research License. The full Medium article contains every measurement and caveat, and the SCOPE, SCORECARD, LIBRARY and SECURITY documents in the repository should be read before treating any result as a deployment claim.
C

Christian Schnapka

Macstab Engineering

Christian Schnapka builds the open engineering tools behind Macstab. He publishes the long-form versions of these investigations on Medium — every measurement, every caveat and every loss included.