<html> <head> <link rel="stylesheet" href="page.css" type="text/css"> <title>Documentation: Serialization of Data and Objects</title> </head> <body bgcolor=#ffffff link=#990033 vlink=#990033 alink=#990033 text=#000000> <!---- TOPIC TITLE WITH LOGO---> <table border=0 cellpadding= cellspacing=2 width=100% ><tr><td><a href='http://www.fox-toolkit.org' target=_top><img src='art/foxlogo_small.jpg' border=0></a></td><td width=100% valign=bottom id="HEADLINE"><b> Documentation: Serialization of Data and Objects <A href='serialization.html' target="_top" align=left><font size=-2>[Remove Frame]</font></a> <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE WITH LOGO ---> <ul> <p>Often, your application needs to save and load data in a machine-independent, binary format. This data may be very simple, such as an array of numbers, or it may be a complex networks of objects arranged in some application-defined data structure.</p> <p>FOX offers some tools to make implementation of such basic save and load facilities in an application fairly straighforward: <b>Serialization</b> and <b>Deserialization</b>. Serialization refers to the process of taking a network of objects and their member data, and turning it into a linear byte stream; deserialization of course refers to the opposite. This process is also sometimes referred to as <i>streaming</i>, <i>flattening</i>, or more prosaically, <i>pickling</i>.</p> <p>The <b>FXStream</b> classes support streaming of objects and data in a type-safe and architecture-neutral manner; this means that a) your data will be read in the way you wrote it out, and b) streaming works as efficient on little-endian machines as it does on big-endian ones:- there is no byte-order preference.</p> <p>The FXStream class are extremely flexible, in that you may subclass them ad libitum to implement esoteric applications ranging from compression to encryption, BSD sockets, UNIX pipes, clipboard, drag & drop, and what have you. Code you write to serialize your data may be reused to perform any of these functions simply by substituting the FXStream class upon which they operate.</p> <p>Once code for an object's serialization has been written, this streaming capability can be used for a variety of purposes: <br> <ul> <li>Saving or loading from <b><i>files</i></b>, in a machine-independent manner.</li> <li>Saving into <b><i>memory buffers</i></b>, or loading back from memory buffers.</li> <li>Loading of <b><i>resources</i></b> compiled into the application using <b><i><a href="icons.html#RESWRAPDOC">reswrap</a></i></b>.</li> <li>Exchanging objects and data between applications using <b><i><a href="draganddrop.html#DRAGNDROP">Drag and Drop</a></i></b> techniques.</li> <li>Just <b><i>counting the bytes</i></b>, e.g. to determine buffer sizes.</li> <li>Transfer objects and data over the network, e.g. via <b><i>sockets</i></b>, <b><i>pipes</i></b>, <b><i>PVM</i></b>, <b><i>MPI</i></b>, etc.</li> </ul> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> Philosophy in FOX Serialization <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> <p>The FOX Stream classes have been designed with a number of goals in mind:</p> <ul> <li><b>Speed</b>. The serialization and deserialization should be very fast. Thus, a minimal amount of computing overhead is required; also, I/O should be minimized.</li> <br> <li><b>Flexibility</b>. At some small expense in speed, all I/O eventuallyboils down to a few basic virtual I/O functions; thus, it is possible to derive subclasses and serialize data into byte streams with different destinations or sources:- not just files, but also memory buffers, sockets, or perhaps shared memory segments or mapped files.</li> <br> <li> <b>Type Safety</b>. In order to make sure that the number of bytes saved exactly matches the number of bytes loaded, all stream insertion/extraction operators are defined for all basic machine types, and these <i>types </i>are<i> guaranteed </i>to be the<i> same size </i>on all FOX implementations.</li> <br> <li> <b>Byte Swapping</b>. Since the types are known, the FOX Stream class is able to swap bytes upon stream deserialization. The FOX Stream does NOT swap bytes on <i>saving</i>, but only on <i>loading</i>. This is for the following reasons:</li> <br> <ul> <li> It is faster to serialize in a machine-natural order, so that as long as one works on machines of the same architecture, no cost is incurred with swapping bytes at all. Loading and saving on the same type of machine is expected to be a very, very common case.</li> <br> <li> By byte swapping on the receiving end, an <i>in-situ</i> swap can be performed, which will lead to much better caching, and eliminates the need to temporary buffers etc.</li> <br> </ul> <li> <b>Predictability</b>. With the exception of serialization of FOX Objects, the FOX Stream class serializes exactly as many bytes as it is given by the application. This has a number of interesting benefits:- for example, the FOX GIF Image loading routine works based on a FOX Stream, permitting it to read both from files as well as from memory data arrays; this makes handling of compiled-in or embedded resources (e.g. by using <a href="icons.html#RESWRAPDOC">reswrap</a>) very simple indeed.</li> <br> <li> <b>Future expansion</b>. An escape tag is prepended for serialized FOX Objects. This will in the [near] future allow deserialization of FOX Objects that are available in <i>dynamic link libraries (DLL's). </i>Currently, FOX can only deserialize objects that have been compiled into the application code.</li> </ul> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> So How Does It Work? <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> From the application programmer's point of view, it works very simply: <pre> FXuint data[100],numdata; // Save my stuff to a stream void savemystuff(FXStream& stream){ stream << numdata; // Save the number of data values stream.save(data,numdata); // Save the data } // Save stuff to a FILE stream FXFileStream stream; stream.open("datafile.dat",FXStreamSave); savemystuff(stream); stream.close(); </pre> <p>As you see, this is pretty simple. Note that the code fragment doing the actual serialization does not depend on the type of FXStream being used; I recommend simply passing in an FXStream&, so that the same code may be used to serialize to FXFileStreams, FXMemoryStreams or other stream classes as yet to be invented.</p> <p>From the stream's point of view, things are a bit more complicated. Saving basic types (FXchar, FXshort, etc) into an FXStream is done by C++-tradional insertion and extraction operators <b><<</b> and <b>>></b>.<br> Note that all operators take a <i>reference</i>, rather than a <i>value</i>. If we would save a value, regular C++ type promotions might be silenty invoked, and more bytes might be saved than expected; by taking reference arguments, one has to first store a value into a variable of <i>known type</i>, then call the insertion operator.</p> <p>For <i>arrays</i> of basic types, the FXStream class supplies a few regular member functions called save() and load(), one for each basic type. Note that FOX may support a type FXlong on certain machines; FXlong is always 64 bits, or 8 bytes, if supported by the system. If 64 bit numbers can not be supported, FXlong is NOT defined.</p> <p>For objects, things are a more complex. A network of objects can be saved into a stream, and should be restored upon a load. Of course, upon load not all objects will occupy the same address as where they were initially stored from, so pointer-values can not be simpy stored in the stream:- a translation is necessary. <br> Also, objects may refer to each other, that is to say, the program's data structures may have circular references.<br> Thus, care must be taken to ensure that each object will be saved only once.</p> <p>FOX currently implements the object save by means of a hash table to translate object pointers into reference numbers and vice versa. In a nutshell, here's how it works:</p> <u>To save an object-pointer to the stream:</u> <ol> <li>If the pointer is NULL, save the speciall <b>null</b> tag.</li><br><br> <li>Consult the hash table to see if the object pointer has been saved before. If the object has been encountered previously, its data must already have been saved, and the <b>reference</b> tag found in the hash table is saved to the stream.</li><br><br> <li>If the object has never been encountered before, generate a new reference tag, and add the object pointer and the reference tag to the hash table. Subsequently, a <b>class</b> tag, an <b>escape</b> code [0 for now], and the object's <b>class name</b> is saved to the stream. Then the object's <b>member data</b> are saved by calling the object's overloaded <b>save()</b> member function.</li> </ol> <p> <u>To load an object-pointer from the stream:</u> <br> <ol> <li>Read the tag. If the tag was the <b>null</b> tag, the pointer was NULL, and a NULL is returned.</li><br><br> <li>If the tag was the <b>reference</b> tag, the object has already been loaded, and the hash table is consulted to return the object-pointer.</li><br><br> <li>If the tag was the <b>class</b> tag, the <b>escape</b> tag is read and [for now] discarded, and subsequently the classname is read. The <b>FXMetaClass</b> is localized from the class name, and a new object is constructed by means of its <b>makeInstance() </b>function. The a new reference number is generated and the reference number and the object-pointer are stored into the hash table. Then the object member data are loaded by calling the object's overloaded <b>load()</b> member function.</li> </ol> <p>In the current implementation, only those objects whose implementation has been compiled into the application can be [de-] serialized. <p>Future versions of FOX will use the <b>escape</b> code information for additional methods to localize the <b>FXMetaClass</b> objects. In particular, the thinking is that certain object-implementations may live in DLL's (Dynamic Link Libraries) and the escape code will help localize the DLL and pull it in to provide the object implementation. It is clear that this will be a very powerful mechanism, enabling for example drag and drop of objects whose implementations are not a-priori known at the time the application is compiled.</p> <p>I added the escape code so as to <b>not</b> break people's streamed object files when this capability will be introduced.</p> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> Future FOX uses of Serialization <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> <p>Serialization is not only intended for features such as saving/restoring from files, and drag-and-drop of objects. Future versions of FOX will also allow FOX GUI Widgets to be serialized or deserialized; in fact, it is with this in mind that the two-step [Construct/Create] sequence is so religiously carried out throughout the Library. Once FOX Widgets have been deserialized from either an external file or perhaps from a compiled-in [reswrapped] resource, a GUI can be created in one fell swoop with a single call to FXApp::create().</p> <p>A FOX GUI Builder will be a program that builds a nice-looking GUI, and then serializes it for incorporation into an application [using reswrap]. Using the escape-code mechanism, the FOX GUI builder will be able to build GUI's that contain Custom Controls or Widgets written by third parties.</p> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> Tips and Hints for Serialization: Byte Swapping <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> <p>Proper use of the serialization mechanism will allow serialized data to be read across different machines, with different byte orders. In the scope of ``predictability,'' FOX's stream mechanism does NOT contain any tags or markers, nor does it contain things like byte order and such, with the exception of course being the saving of object-pointers. <p>It <i>does</i> however try to help:</p> <pre> FXbool FXStream::<b>isLittleEndian</b>(); </pre> <p>returns <i>TRUE</i> for Little Endian machines [e.g. i386 and Alpha CPU's] and <i>FALSE</i> for Big Endian machines [e.g. 68k, SPARC CPU's]. <br>Note that <b>FXbool</b> is defined as <b>FXuchar</b>, NOT as C++ <b>bool</b>. [I've never been able to find a statement that says how big the standard type <b>bool</b> is, but I'm pretty sure a char is 1 byte!].</p> <p>Thus, the following chunk of code may be executed before saving any actual application data:</p> <pre> FXbool endianness=FXStream::isLittleEndian(); stream << endianness; .... save the data .... </pre> <p>Then upon loading:</p> <pre> FXbool endianness; stream >> endianness; stream.swapBytes(endianness!=FXStream::isLittleEndian()); .... load the data .... </pre> <p>In other words, the bytes are swapped <b><i>on input</i></b>, if and only if<i> </i>the byte order of the saving application <i>differs</i> from the loading one.</p> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> Tips and Hints for Serialization: Container Object <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> <p>Many applications have one so-called <b>container</b> object, which may not itself participate in serialization for one reason or another. For example, the FOX FXApp object is normally created by the main startup routine of an application, and will probably never be serialized [although its member data may be].</p> <p>In order to accomodate references to such an object without saving it, the FXStream class allows you to specify a <b>container</b> object. During serialization, when a pointer to the container object is encountered, only a reference tag is saved to the stream; likewise, on deserialization a reference to the container object is translated into the pointer passed in with the FXStream constructor.</p> </ul> <!--- TOPIC TITLE --> <p> <table width=100% cellpadding=0 cellspacing=2><tr><td width=100% valign=bottom id=HEADLINE><b> Tips and Hints for Serialization: Use FX Types <br><img src='art/line.gif' width=100% height=1></b></td></tr></table> </p> <!--- TOPIC TITLE --> <ul> <p>FOX defines a number of typedefs for the basic types, such as FXchar, FXshort, and so on. The idea is that the size of these types is <i>fixed</i>, and the <i>same</i> on all implementations; there is an FXASSERT somewhere that will trip if this is not true.</p> <p>Writing applications that should work on heterogeneous mixes of hardware becomes simpler if variables you intend to serialize are defined in terms of these basic types; for loop variables and such ephemeral things, you may want to use the ``suggested'' system-specific types, as these may be faster.</p> <p>The type <b>FXlong</b> may NOT be available on all platforms. It represents a 64 bit integer type. You use this at your own risk of potential portability loss.</p> </ul> <!--- COPYRIGHT --> <p> <table width=100% cellpadding=0 cellspacing=0><tr><td width=100% valign=top id=HEADLINE align=right> <img src='art/line.gif' width=100% height=1> <font size=-1>Copyright © 1997-2004 Jeroen van der Zijp</font> </td></tr></table> </p> <!--- COPYRIGHT --> </body> </html>