Hachoir is the French name for a mincer: a tool used by butchers to cut meat.
Hachoir is also a tool written for hackers to cut a file or any binary stream.
A file is split in a tree of fields where the smallest field can be a
bit. There are various field types: integer, string, bits, padding, sub file,
etc.
This document is a presentation of Hachoir API. It tries to show most
the interesting part of this tool, but is not exhaustive. Ok, let's start!
hachoir.stream: Stream manipulation
===================================
To split data we first need is to get data :-) So this section presents the
"hachoir.stream" API.
In most cases we work on files using the FileInputStream function. This function
takes one argument: a Unicode filename. But for practical reasons
we will use StringInputStream function in this documentation.
>>> data = "point\0\3\0\2\0"
>>> from hachoir_core.stream import StringInputStream, LITTLE_ENDIAN
>>> stream = StringInputStream(data)
>>> stream.source
'<string>'
>>> len(data), stream.size
(10, 80)
>>> data[1:6], stream.readBytes(8, 5)
('oint\x00', 'oint\x00')
>>> data[6:8], stream.readBits(6*8, 16, LITTLE_ENDIAN)
('\x03\x00', 3)
>>> data[8:10], stream.readBits(8*8, 16, LITTLE_ENDIAN)
('\x02\x00', 2)
First big difference between a string and a Hachoir stream is that sizes
and addresses are written in bits and not bytes. The difference is a factor
of eight, that's why we write "6*8" to get the sixth byte for example. You
don't need to know anything else to use Hachoir, so let's play with fields!
hachoir.field: Field manipulation
=================================
Basic parser
------------
We will parse the data used in the last section.
>>> from hachoir_core.field import Parser, CString, UInt16
>>> class Point(Parser):
... endian = LITTLE_ENDIAN
... def createFields(self):
... yield CString(self, "name", "Point name")
... yield UInt16(self, "x", "X coordinate")
... yield UInt16(self, "y", "Y coordinate")
...
>>> point = Point(stream)
>>> for field in point:
... print "%s) %s=%s" % (field.address, field.name, field.display)
...
0) name="point"
48) x=3
64) y=2
`point` is a the root of our field tree. This tree is really simple, it just
has one level and three fields: name, x and y. Hachoir stores a lot of
information in each field. In this example we just show the address, name and
display attributes. But a field has more attributes:
>>> x = point["x"]
>>> "%s = %s" % (x.path, x.value)
'/x = 3'
>>> x.parent == point
True
>>> x.description
'X coordinate'
>>> x.index
1
>>> x.address, x.absolute_address
(48, 48)
The index is not the index of a field in a parent field list, '1' means that it's
the second since the index starts at zero.
Parser with sub-field sets
--------------------------
After learning basic API, let's see a more complex parser: parser with
sub-field sets.
>>> from hachoir_core.field import FieldSet, UInt8, Character, String
>>> class Entry(FieldSet):
... def createFields(self):
... yield Character(self, "letter")
... yield UInt8(self, "code")
...
>>> class MyFormat(Parser):
... endian = LITTLE_ENDIAN
... def createFields(self):
... yield String(self, "signature", 3, charset="ASCII")
... yield UInt8(self, "count")
... for index in xrange(self["count"].value):
... yield Entry(self, "point[]")
...
>>> data = "MYF\3a\0b\2c\0"
>>> stream = StringInputStream(data)
>>> root = MyFormat(stream)
This example presents many interesting features of Hachoir. First of all, you
can see that you can have two or more levels of fields. Here we have a tree
with two levels:
>>> def displayTree(parent):
... for field in parent:
... print field.path
... if field.is_field_set: displayTree(field)
...
>>> displayTree(root)
/signature
/count
/point[0]
/point[0]/letter
/point[0]/code
/point[1]
/point[1]/letter
/point[1]/code
/point[2]
/point[2]/letter
/point[2]/code
A field set is also a field, so it has the same attributes than another field
(name, address, size, path, etc.) but has some new attributes like stream or
root.
Lazy feature
------------
Hachoir is written in Python so it should be slow and eat a lot of CPU and
memory, and it does. But in most cases, you don't need to explore an entire
field set and read all values; you just need to read some values of some
specific fields. Hachoir is really lazy: no field is parsed before you ask for
it, no value is read from stream before you read a value, etc. To inspect this
behaviour, you can watch "current_length" (number of read fields) and
"current_size" (current size in bits of a field set):
>>> root = MyFormat(stream) # Rebuild our parser
>>> print (root.current_length, root.current_size)
(0, 0)
>>> print root["signature"].display
"MYF"
>>> print (root.current_length, root.current_size, root["signature"].size)
(1, 24, 24)
Just after its creation, a parser is empty (0 fields). When we read the first
field, its size becomes the size of the first field. Some operations requires
to read more fields:
>>> print root["point[0]/letter"].display
'a'
>>> print (root.current_length, root.current_size)
(3, 48)
Reading point[0] needs to read field "count". So root now contains three
fields.
List of field types
===================
Number:
* Bit: one bit (True/False) ;
* Bits: unsigned number with a size in bits ;
* Bytes: vector of know bytes (e.g. file signature) ;
* UInt8, UInt16, UInt24, UInt32, UInt64: unsigned number (size: 8, 16, ... bits) ;
* Int8, Int16, Int24, Int32, Int64: signed number (size: 8, 16, ... bits) ;
* Float32, Float64, Float80: IEEE 754 floating point number (32, 64, 80 bits) ;
Text:
* Character: 8 bits ASCII character ;
* String: fixed length string ;
* CString: string ending with nul byte ("\\0") ;
* UnixLine: string ending with new line character ("\\n") ;
* PascalString8, PascalString16 and PascalString32: string prefixed with
length in a unsigned 8 / 16 / 32 bits integer (use parent endian) ;
Timestamp (date and time):
* TimestampUnix32, TimestampUnix64: 32/64 bits UNIX, number of seconds since
the January 1st 1970 ;
* TimestampMac32: 32-bit Mac, number of seconds since the January 1st 1904 ;
* TimestampWin64: 64-bit Windows, number of 1/10 microseconds since
the January 1st 1600 ;
* DateTimeMSDOS32 and TimeDateMSDOS32: 32-bit MS-DOS structure,
since the January 1st 1980.
Timedelta (duration):
* TimedeltaWin64: 64-bit Windows, number of 1/10 microseconds
Padding and raw bytes:
* PaddingBits/PaddingBytes: padding with a size in bits/bytes ;
* NullBits/NullBytes: null padding with a size in bits/bytes ;
* RawBits/RawBytes: unknown content with a size in bits/bytes.
* SubFile: a file contained in the stream ;
To create your own type, you can use:
* GenericInteger: integer ;
* GenericString: string ;
* FieldSet: Set of other fields ;
* Parser: The main class to parse a stream.
Field class
===========
Read only attributes:
* name (str): Name of the field, is unique in parent field set
* address (long): address in bits relative to parent address
* absolute_address (long): address in bits relative to input stream
* parent (GenericFieldSet): parent field (is a field set)
* is_field_set (bool) <~~~ can be replaced: the field contains other fields?
* index (int): index of the field in parent field set (first index is 0)
Read only and lazy attributes:
* size (long), cached: size of the field in bits
* description (str|unicode), cached: informal description
* display (unicode): value with human representation as unicode string
* raw_display (unicode): value with raw representation as unicode string
* path (str): concatenation with slash separator of all field name from
the root field
Method that can be replaced:
* createDescription(): create value of 'description' attribute
* createValue(): create value of 'value' attribute
* createDisplay(): create value of 'display' attribute
* _createInputStream(): create an InputStream containing the field content
Aliases (method):
* __str__() <=> read display attribute
* __unicode__() <=> read display attribute
* __getitem__(key): alias to getField(key, False)
Other methods:
* static_size: helper to compute field size. If the value is an integer, the
type has constant size. If it's a function, the size depends of the arguments.
* hasValue(): check if the field has a value or not (default: self.value is not None)
* getField(key, const=True): get the field with specified key,
if const is True the field set will not be changed
* __contains__(key)
* getSubIStream(): return a tagged InputStream containing the field content
* setSubIStream(): helper to replace _createInputStream (the old one is passed
to the new one to allow chaining)
Field set class
===============
Read only attributes:
* endian: value is BIG_ENDIAN or LITTLE_ENDIAN, the way the bits are written
in input stream <~~ can be replaced
* stream (InputStream): input stream
* root (FieldSet): root of all fields
* eof (bool): End Of File: are we at the end of the input stream?
* done (bool): The parser is done or not?
Read only and lazy attributes:
* current_size (long): Current size in bits
* current_length (long): Current number of children
Methods:
* connectEvent(event, handler, local=True): connect an handler to an event
* raiseEvent(event, \*args): raise an event
* reset(): clear all caches but keep its size if we know it
* setUniqueFieldName(): for field with name ending with "[]",
replaces "[]" with an unique identifier like, "item[]" => "item[0]".
* seekBit(address, ...): create a field to seek to specified address or
returns None if we are already there
* seekByte(address, ...): create a field to seek to specified address or
returns None if we are already there
* replaceField(name, fields): replace a field with
one or more fields <~~~ I don't like this method :-(
* getFieldByAddress(address, feed=True): get the field at the
specified address
* writeFieldsIn(old, address, new): helper for replaceField() <~~~ can be an helper?
* getFieldType(): get the field type as a short string. The type may contains
extra informations like the string charset.
Lazy methods:
* array(): create a FakeArray to easily get a field by its index
(see FakeArray API to get more details)
* __len__(): number of children in the field set
* readFirstFields(number): read first 'number' fields,
returns number of new fields
* readMoreFields(number): read more 'number' fields,
returns number of new fields
* __iter__(): iterate over children
* createFields(): main function of the parser, create the fields. Don't call
this function directly.
