Asn1 oer uper contrib - #5050
Conversation
e7bc1d3 to
4cdc2de
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5050 +/- ##
==========================================
+ Coverage 80.81% 81.10% +0.29%
==========================================
Files 390 401 +11
Lines 96947 98621 +1674
==========================================
+ Hits 78344 79991 +1647
- Misses 18603 18630 +27
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds ASN.1 OER and UPER (registered on ASN1_Codecs.OER and ASN1_Codecs.PER) plus a new “field hooks” mechanism to let codecs override compound-field behavior (tagging, SEQUENCE/CHOICE/SEQUENCE OF) while keeping BER as the default behavior.
Changes:
- Introduces
ASN1Codec.register_field_hooks()/field_hook()and updates ASN.1 fields to consult codec-specific hooks. - Adds new contrib codecs:
scapy.contrib.oer(OER) andscapy.contrib.uper(UPER/PER). - Expands/creates UTS coverage for cross-codec build/dissect and OER vectors/fuzzing.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/scapy/layers/ber.uts | Updates tests for the new BER tagging hooks + adds BER build/dissect test coverage. |
| test/scapy/layers/asn1.uts | Adds cross-codec (BER/OER/PER) build/dissect tests and codec-opts/default-component checks. |
| test/contrib/oer.uts | New OER-focused test suite (vectors, fuzzing, interop, conformance checks). |
| scapy/contrib/uper.py | New UPER implementation and PER field hooks for bitstream-oriented encoding/decoding. |
| scapy/contrib/oer.py | New OER implementation and OER field hooks for preamble/CHOICE-tag behavior. |
| scapy/asn1fields.py | Adds codec_opts plumbing and consults codec field hooks for tagging/compound-field operations. |
| scapy/asn1/ber.py | Registers BER field hooks (tagging) via the new hook mechanism. |
| scapy/asn1/asn1.py | Adds codec-level field hook registration + safe default for _field_hooks. |
| .config/codespell_ignore.txt | Adds OER/UPER-related ignore words. |
Suppressed comments (1)
test/scapy/layers/asn1.uts:402
- Duplicate helper function:
_roundtripis defined twice back-to-back here. One of them should be removed to avoid confusion and reduce noise in the test file.
def _roundtrip(cls, pkt):
# type: (type, ASN1_Packet) -> ASN1_Packet
return cls(raw(pkt))
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
gpotter2
left a comment
There was a problem hiding this comment.
This is very hard to review. I think it needs some cleanup phase where it reduces the number of functions that are used only once.
I also think that eventually it makes more sense to include all of this in scapy/asn1 directly
| def _OER_check_len(name, s, number_of_bytes, offset=0): | ||
| # type: (str, bytes, int, int) -> None | ||
| """Raise unless s carries number_of_bytes octets past its first offset.""" | ||
| available = len(s) - offset | ||
| if available < number_of_bytes: | ||
| raise OER_Decoding_Error( | ||
| "%s: Got %i bytes while expecting %i" % | ||
| (name, available, number_of_bytes), | ||
| remaining=s | ||
| ) |
There was a problem hiding this comment.
Useful as a standalone function? I don't like this pattern of calling a function that might throw an error, it's not clear from the parent code
| def OER_signed_integer_enc(i): | ||
| # type: (int) -> bytes | ||
| # X.696 10.4: the shortest two's complement encoding. A negative value | ||
| # needs one bit less than its magnitude suggests, as -2**(8n-1) still | ||
| # fits in n octets, hence the increment before measuring. | ||
| magnitude = i + 1 if i < 0 else i | ||
| number_of_bytes = (magnitude.bit_length() + 8) // 8 | ||
| value = i & ((1 << (8 * number_of_bytes)) - 1) | ||
| return OER_len_enc(number_of_bytes) + value.to_bytes(number_of_bytes, "big") | ||
|
|
||
|
|
||
| def OER_signed_integer_dec(s): | ||
| # type: (bytes) -> Tuple[int, bytes] | ||
| number_of_bytes, s = OER_len_dec(s) | ||
| _OER_check_len("OER_signed_integer_dec", s, number_of_bytes) | ||
| if number_of_bytes == 0: | ||
| raise OER_Decoding_Error( | ||
| "OER_signed_integer_dec: got an empty length determinant", | ||
| remaining=s | ||
| ) | ||
| value = int.from_bytes(s[:number_of_bytes], "big") | ||
| number_of_bits = 8 * number_of_bytes | ||
| if value & (1 << (number_of_bits - 1)): | ||
| value -= (1 << number_of_bits) - 1 | ||
| value -= 1 | ||
| return value, s[number_of_bytes:] | ||
|
|
||
|
|
||
| def OER_unsigned_integer_enc(i): | ||
| # type: (int) -> bytes | ||
| if i < 0: | ||
| raise OER_Encoding_Error( | ||
| "OER_unsigned_integer_enc: %i is negative" % i | ||
| ) | ||
| number_of_bits = max(i.bit_length(), 1) | ||
| number_of_bytes = (number_of_bits + 7) // 8 | ||
| return OER_len_enc(number_of_bytes) + i.to_bytes(number_of_bytes, "big") | ||
|
|
||
|
|
||
| def OER_unsigned_integer_dec(s): | ||
| # type: (bytes) -> Tuple[int, bytes] | ||
| number_of_bytes, s = OER_len_dec(s) | ||
| _OER_check_len("OER_unsigned_integer_dec", s, number_of_bytes) | ||
| value = int.from_bytes(s[:number_of_bytes], "big") | ||
| return value, s[number_of_bytes:] |
There was a problem hiding this comment.
We don't have all of those for BER? They're part of the classes directly, I think it makes more sense
|
Very sorry @polybassa but this still looks like slop for the most part. This probably means we should try to add more coding guidances to the AI |
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
3aba643 to
07160bb
Compare
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fix UPER trailing-octet handling, OER integer constraints, X.509 underlayer extraction, known-multiplier string typing, and restore codec tagging/_codec_kwargs contracts with regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
071ffa8 to
2f31055
Compare
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size constraints and named enum encode values, and drop leftover dead helpers. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
584768f to
faaa99b
Compare
Move compound wire rules onto encoder/decoder methods, reject empty OER SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER constraint kwargs. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not false-fail on master tips until the standalone CI PR lands. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Move OER/UPER codec implementations to scapy.contrib and wire asn1fields for OER/PER using the pluggable tagging/kwargs hooks. AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
Finish the OER/UPER architectural cleanup by reading constraints from fields directly, routing build/dissect through encode_to/decode_from, and sharing SEQUENCE/CHOICE/SEQUENCE OF logic in compound.py. Contrib modules become thin re-exports. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Unify compound child dispatch on encode_to/decode_from, add OER encoder contexts and PER bit-stream helpers, and read field constraints directly in primitive codecs instead of round-tripping through codec_kwargs. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Remove EncodingParams, codec_kwargs, and field codec_opts/_codec_kwargs; codecs resolve size_len and constraints via field= directly. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Remove unused imports, tighten typing for observed tags and codec returns, and satisfy line-length checks on BER override annotations. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fix UPER trailing-octet handling, OER integer constraints, X.509 underlayer extraction, known-multiplier string typing, and restore codec tagging/_codec_kwargs contracts with regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Rebase onto master picked up secdev#5062; replace remaining orb() uses with direct byte indexing so the contrib codecs import cleanly. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Rewrite the intentional typo kwargs in oer.uts so codespell stays quiet, and flatten the oer_int_wire_params docstring for Sphinx -W. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
ASN1F_PACKET.any2i and UPER nestings set parent only, which broke EncryptedData.encrypt in KerberosSSP; set underlayer and parent together. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Import codecs from scapy.asn1 and use minimum/maximum/extensible/unsigned in tests. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Drop unused compound helper re-exports, constraint aliases, and stale smoke tests. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Write OER SEQUENCE children into one encoder, resolve UPER bounds once, and centralize two's-complement octet math. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size constraints and named enum encode values, and drop leftover dead helpers. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
actions/checkout builds a merge of the PR into the base branch, so rev-list from HEAD was also validating base-branch tips. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Move compound wire rules onto encoder/decoder methods, reject empty OER SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER constraint kwargs. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not false-fail on master tips until the standalone CI PR lands. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Cast codec.enc results after neutral get_codec typing, and drop obsolete attr-defined ignores. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Bind compound encode/decode hooks directly on BER/OER/UPER contexts, separate field-layer contexts from raw UPER bit streams, and inline trivial constraint getters and single-use compound forwarders. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop _codec_kwargs and ASN1Codec.new_encoder/new_decoder wrappers, stream BER SEQUENCE children through a nested context, and avoid SEQUENCE OF fragment list slices. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep OPTIONAL child walking in compound.py and move BER/OER/UPER
SEQUENCE, CHOICE, SEQUENCE OF, and PACKET implementations into
compound_{ber,oer,uper}.py.
AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop dead decoder offset/chunk counters, use int.from_bytes/to_bytes for OER length determinants, and make UPER read_bit avoid redundant arithmetic. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fold CHOICE/PACKET second-layer helpers into the bound codec hooks, drop sequence_encode_children and UPER set_remainder, and keep only the shared OPTIONAL decode walk. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Store UPER input as a byte buffer with a bit cursor instead of one giant integer, drop per-field kwargs copies, look up CHOICE by tag, and join OER SEQUENCE OF payloads once. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Large OCTET STRING reads at a non-byte offset use one bulk int.from_bytes/shift instead of growing an integer per source byte; delegate whole-octet read_bits to that path and drop duplicate bounds checks on the small-field integer reader. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Reject SEQUENCE/CHOICE/SEQUENCE OF as SEQUENCE OF field elements with an explicit error, keep ASN1F_PACKET (used by Kerberos) via UPER context hooks, and drop misleading compound encode_into/dissect_from_decoder wrappers so only primitives use the raw-bit API. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Replace chunked giant-int finalization with a bytearray plus pending-bit accumulator, speed up Decoder.remaining(), and reject compound SEQUENCE OF elements only on the UPER path so BER/OER keep master construction behavior. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the per-byte _peek_bits_int loop, keep append_bits on the byte path except for a trailing partial octet, bulk-shift unaligned append_bytes, and trim redundant SEQUENCE OF PACKET checks. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Cache ENUMERATED PER value order, avoid fragment bytes copies with memoryview, replace OER_Exception with OER_Encoding_Error, collapse redundant ASN1_Error handlers, and restore check_commits.sh so the CI fix stays out of the ASN.1 PR. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the broken _uper_enum_values cache (sorted the swapped local map), create memoryviews only for >=16K fragmented payloads, slim the UPER enum helper, and avoid copying textual BIT STRING padding. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
c3368cb to
37fc5bf
Compare
No description provided.