Changes between Version 18 and Version 19 of StructuredBinaryData

Timestamp:

2012-08-06T21:42:18Z (12 years ago)

Author:

Sean Bartell

Comment:

Document bit blobs and add future features

StructuredBinaryData

@@ -68 +68 @@
 integer, and a string holds a Unicode string.
 
-A blob node represents an arbitrary sequence of raw bytes. Blob nodes are
-polymorphic, allowing any source of raw binary data to be used. Bithenge
+A blob node represents an arbitrary sequence of raw bytes or bits. Blob nodes
+are polymorphic, allowing any source of raw binary data to be used. Bithenge
 includes blob node implementations for in‐memory buffers, files, and block
 devices. An implementation has also been written that reads another task’s
@@ -115 +115 @@
 which takes a 4‐byte blob node as the input tree and provides an integer node
 as the output tree. Another example would be `FAT16_filesystem`, a transform
-that takes a blob node as the input tree and provides a complex output tree
-with various decoded information about the filesystem. Some transforms, like
-`uint32le`, are built in to Bithenge; more complicated transforms can be loaded
-from a script file.
+that takes a byte blob node as the input tree and provides a complex output
+tree with various decoded information about the filesystem. Some transforms,
+like `uint32le`, are built in to Bithenge; more complicated transforms can be
+loaded from a script file.
 
 Transforms are represented in Bithenge with a polymorphic object. The primary
@@ -132 +132 @@
 
 ||= name =||= input =||= output =||= description =||= example =||
-||ascii             ||blob node        ||string       ||decodes some bytes as ASCII characters ||  `hex:6869` becomes `"hi"` ||
-||known_length(len) ||blob node        ||blob node    ||requires the input to have a known length || ||
-||nonzero_boolean   ||integer          ||boolean      ||decodes a boolean where nonzero values are true || `0` becomes `false` ||
-||uint8             ||1‐byte blob node ||integer node ||decodes a 1‐byte unsigned integer ||  `hex:11` becomes `17` ||
-||uint16be          ||2‐byte blob node ||integer node ||decodes a 2‐byte big‐endian unsigned integer ||  `hex:0201` becomes `513` ||
-||uint16le          ||2‐byte blob node ||integer node ||decodes a 2‐byte little‐endian unsigned integer ||  `hex:0101` becomes `257` ||
-||uint32be          ||4‐byte blob node ||integer node ||decodes a 4‐byte big‐endian unsigned integer ||  `hex:00000201` becomes `513` ||
-||uint32le          ||4‐byte blob node ||integer node ||decodes a 4‐byte little‐endian unsigned integer ||  `hex:01010000` becomes `257` ||
-||uint64be          ||8‐byte blob node ||integer node ||decodes a 8‐byte big‐endian unsigned integer ||  `hex:0000000000000201` becomes `513` ||
-||uint64le          ||8‐byte blob node ||integer node ||decodes a 8‐byte little‐endian unsigned integer ||  `hex:0101000000000000` becomes `257` ||
-||zero_terminated   ||blob node        ||blob node    ||takes bytes up until the first `00` ||  `hex:7f0400` becomes `hex:7f04` ||
+||ascii             ||byte blob node   ||string        ||decodes some bytes as ASCII characters ||  `hex:6869` becomes `"hi"` ||
+||bit               ||1‐bit blob node  ||boolean       ||decodes a single bit || `1` becomes `true` ||
+||bits_be           ||byte blob node   ||bit blob node ||decodes bytes as bits, starting with the most‐significant bit || `hex:0f` becomes `bit:00001111` ||
+||bits_le           ||byte blob node   ||bit blob node ||decodes bytes as bits, starting with the least‐significant bit || `hex:0f` becomes `bit:11110000` ||
+||known_length(len) ||blob node        ||blob node     ||requires the input to have a known length || ||
+||nonzero_boolean   ||integer          ||boolean       ||decodes a boolean where nonzero values are true || `0` becomes `false` ||
+||uint8             ||1‐byte blob node ||integer node  ||decodes a 1‐byte unsigned integer ||  `hex:11` becomes `17` ||
+||uint16be          ||2‐byte blob node ||integer node  ||decodes a 2‐byte big‐endian unsigned integer ||  `hex:0201` becomes `513` ||
+||uint16le          ||2‐byte blob node ||integer node  ||decodes a 2‐byte little‐endian unsigned integer ||  `hex:0101` becomes `257` ||
+||uint32be          ||4‐byte blob node ||integer node  ||decodes a 4‐byte big‐endian unsigned integer ||  `hex:00000201` becomes `513` ||
+||uint32le          ||4‐byte blob node ||integer node  ||decodes a 4‐byte little‐endian unsigned integer ||  `hex:01010000` becomes `257` ||
+||uint64be          ||8‐byte blob node ||integer node  ||decodes a 8‐byte big‐endian unsigned integer ||  `hex:0000000000000201` becomes `513` ||
+||uint64le          ||8‐byte blob node ||integer node  ||decodes a 8‐byte little‐endian unsigned integer ||  `hex:0101000000000000` becomes `257` ||
+||uint_be(len)      ||bit blob node    ||integer node  ||decodes bits as an unsigned integer, starting with the most‐significant bit || ||
+||uint_le(len)      ||bit blob node    ||integer node  ||decodes bits as an unsigned integer, starting with the least‐significant bit || ||
+||zero_terminated   ||byte blob node   ||byte blob node||takes bytes up until the first `00` ||  `hex:7f0400` becomes `hex:7f04` ||
 
 == Basic syntax ==
@@ -267 +272 @@
    could pass the whole blob as a parameter and apply transforms to subblobs.
    This is essential for non‐sequential blobs like filesystems.
- Bitfields:: `struct` will be extended to work with bits instead of just bytes.
  Complex expressions:: Expressions that use operators or call transforms.
+ Accessing outer fields:: Expressions can use previously decoded fields of the
+   current `struct`, but they need a way to access the previously decoded
+   fields of an outer `struct`.
  Assertions:: These could be implemented as transforms that don't actually
    change the input. There could be multiple levels, ranging from “warning” to
@@ -274 +281 @@
  Enumerations:: An easier way to handle many constant values, like
    `enum { 0: "none", 1: "file", 2: "directory", 3: "symlink" }`.
+ Merge blobs and internal nodes:: Currently, `struct`, `repeat`, and so on only
+   work with blobs, which must be either byte sequences or bit sequences.
+   Numbered internal nodes (such as those made by `repeat`) should be supported
+   as well.
  Transforming internal nodes:: After binary data is decoded into a tree, it
    should be possible to apply further transforms to interpret the data
@@ -279 +290 @@
    filesystem have been decoded, a further transform could determine the data
    for each file.
+ More information in repeat subtransforms:: Repeat subtransforms should have
+   access to the current index and previously decoded items.
  Hidden fields:: Some fields, such as length fields, are no longer interesting
    after the data is decoded, so they should be hidden by default.
@@ -290 +303 @@
    decoded within a blob. There would need to be some sort of scoping to
    determine which transforms have the automatic parameters.
+ Smarter length calculation:: Bithenge should automatically detect the length
+   of certain composed transforms, such as `repeat(8) {bit} <- bits_le`. This
+   would also be addressed by the constraint‐based version.
 
 === Constraint‐based version ===