-*- coding: utf-8; fill-column: 72; -*-
The Embedded Resources System
=============================
This document gives an overview of FlightGear's embedded resources
system and related classes. For specific information on the C++
functions, the reference documentation is in the corresponding header
files.
Contents
--------
1. The CharArrayStream and ZlibStream classes
2. The âembedded resourcesâ system
3. About the XML resource declaration files
4. The EmbeddedResourceProxy class
Introduction
------------
The embedded resources system allows FlightGear to use data from files
without relying on FG_ROOT to be set. This can be used, for instance, to
grab the contents of XML files at FG build time, from any repository[1],
and use said contents in the C++ code. The term âembeddedâ is used to
avoid confusion with the ResourceProvider and ResourceManager classes
provided by SimGear, which have nothing to do with the system described
here.
The embedded resources system relies on classes present in
simgear/io/iostreams/{zlibstream.cxx,CharArrayStream.cxx}, which were
implemented as a way to address a concern that embedding a few XML files
in the fgfs binary could use precious memory. The resource compiler
(fgrcc) compresses resources before writing them in C++ form---except
for some extensions, and it's configurable on a per-resource basis
anyway. Then, the EmbeddedResourceManager instance, which lives in the
fgfs process, can decompress them on-the-fly, incrementally,
transparently. So, there is really no reason to worry about memory
consumption, even for several dozens of XML files.
fgrcc is the resource compiler: it turns arbitrary files into C++ code
the EmbeddedResourceManager can make use of, in order to âserveâ the
files' contents at runtime. It is named this way, because it fulfills
the same role as Qt's rcc tool. It supports a thin superset of the
XML-based format used by rcc for declaring resources[2][3].
'fgrcc --help' gives a lot of info.
1) The CharArrayStream and ZlibStream classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The CharArrayStream* files in simgear/io/iostreams/ implement
CharArrayStreambuf and related IOStreams classes for working with char
arrays, namely:
- CharArrayStreambuf subclass of std::streambuf stream buffer
- ROCharArrayStreambuf subclass of CharArrayStreambuf stream buffer
- CharArrayIStream subclass of std::istream input stream
- CharArrayOStream subclass of std::ostream output stream
- CharArrayIOStream subclass of std::iostream input/output stream
(in the 'simgear' namespace, of course)
CharArrayStreambuf is a stream buffer class allowing to read from, and
write to char arrays (std::strstream has been deprecated since C++98).
Contrary to std::strstream, this class does no dynamic allocation: it is
very simple, strictly staying for both reads and writes within the
bounds of the buffer specified in its constructor. Contrary to
std::stringstream, CharArrayStreambuf allows one to work on an array of
char (that could be static data, on the stack, whatever) without having
to make a whole copy of it.
ROCharArrayStreambuf is a read-only subclass of CharArrayStreambuf
(useful for const-correctness). CharArrayIStream, CharArrayOStream and
CharArrayIOStream are very simple convenience stream classes using
either CharArrayStreambuf or ROCharArrayStreambuf as their associated
stream buffer class.
While these classes can be of general-purpose usefulness, the particular
reason they have been written for is to make the embedded resources
system clean and memory-friendly. Concretely, this system supports both
compressed and uncompressed resources, all of which can be read from
their respective static arrays like this (think pipelines):
static char array
(uncompressed ---------------> data available via an std::istream
resource) CharArrayIStream or std::streambuf interface
or ROCharArrayStreambuf
static char array
(compressed ---------------> compressed data -------------------> ditto
resource) CharArrayIStream ZlibDecompressorIStream
or ZlibDecompressorIStreambuf
where ditto = uncompressed data available via an std::istream or
std::streambuf interface
So, whether the resource data stored in static arrays by fgrcc is
compressed or not, end-user code can read it in uncompressed form using
an std::istream or std::streambuf interface, which means the resource
never needs to be copied in memory a second time. This is particularly
interesting with compressed resources, because:
1) The in-memory static data is much smaller in general than the
uncompressed contents, and it's the only one we really have to
âpayâ for if one uses these stream-based interfaces.
2) The data is transparently decompressed on-demand as the end-user
code reads from the ZlibDecompressorIStream or
ZlibDecompressorIStreambuf instance.
In other words, these CharArrayStream classes complement the ones in
zlibstream.cxx and make it easy to implement all kinds of pipelines to
incrementally read or write, and possibly on-the-fly compress or
decompress data from or to in-memory buffers (cf.
writeCompressedDataToBuffer() in
simgear/simgear/embedded_resources/embedded_resources_test.cxx, or
ResourceCodeGenerator::writeEncodedResourceContents() in
flightgear/src/EmbeddedResources/fgrcc.cxx for examples).
Since all of these provide standard IOStreams interfaces, they can be
easily plugged into existing code. For instance, readXML() in
simgear/simgear/xml/easyxml.cxx and readProperties() in
simgear/props/props_io.cxx can incrementally read and parse data from an
std::istream instance, and thus are able to directly read from a
resource containing the compressed version of an XML file.
This incremental stuff is of course really interesting with large
resources... which probably won't be used in FlightGear, in order not to
waste RAM[4][5]. The EmbeddedResourceManager also has a getString()
method to simply get an std::string when you don't care about the fact
that this operation, by std::string design, will necessarily make a copy
of the whole resource contents (in uncompressed form in the case of a
compressed resource). This getString() method should be convenient and
quite acceptable for reasonably-sized resources.
Finally, all of these classes---CharArray*Stream*, the classes in
zlibstream.cxx, the EmbeddedResourceManager and related classes---can
handle text and binary data in exactly the same way (std::string doesn't
care, and neither do the other classes).
2) The âembedded resourcesâ system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The embedded resources system works this way:
(1) The fgrcc resource compiler reads an XML file which has almost the
same syntax[2] as Qt's .qrc files[3] and writes a .cxx file
containing:
- static char arrays initialized with resource contents
(possibly compressed, this is automatic unless explicitly
specified in the XML file);
- a function definition containing calls to
EmbeddedResourceManager::addResource() that register each of
these resources with the EmbeddedResourceManager instance.
If you pass the --output-header-file option to fgrcc, it also
writes a header file that goes with the generated .cxx file. For
other options, see the output of 'fgrcc --help'.
It is quite possible to call fgrcc several times, each time with a
different (XML input file, .cxx/.hxx output files) tuple: for
instance, one call for resources present in the FlightGear repo,
and possibly another call for resources in FGData. The point of
this is that paths in the XML input file should be relative to
avoid being system-dependent, and fgrcc accepts a --root option to
indicate what you want them to be relative to, in order to let it
find the real files. Thus, on a first invocation of fgrcc, one can
make --root point to a path to the FlightGear repository when
building, and on the second call use it to indicate a path to the
FGData repository. Other variations are possible, of course.
Notes:
1) The example given here with FGData would *not* freeze the
FGData location at FG compile time; this is only to allow
files from FGData to be turned into generated .cxx files
inside the FG source tree, that will make their contents
available as embedded resources at runtime.
2) At the time of this writing, resources from the FlightGear
repository are compiled at build time, and resources from the
FGData repository are compiled offline using the
'rebuild-fgdata-embedded-resources' script[6] (a
convenience wrapper for fgrcc), before being committed to the
FlightGear repository.
(2) SimGear contains an EmbeddedResourceManager class with, among
others, createInstance() and instance() methods similar to the
ones of NavDataCache. See [7] for the corresponding code.
FlightGear creates an EmbeddedResourceManager instance at startup
and calls the various init functions generated by fgrcc, each of
which registers the resources present in its containing .cxx file
(using EmbeddedResourceManager::addResource()).
End-user FG code can then use EmbeddedResourceManager methods such
as getResource(), getString(), getStreambuf() and getIStream()
to access resource contents:
- getResource() returns an
std::shared_ptr<const AbstractEmbeddedResource>
- getString() returns an std::string
- getStreambuf() returns an std::unique_ptr<std::streambuf>
- getIStream() returns an std::unique_ptr<std::istream>
AbstractEmbeddedResource is an abstract base class that you can
think of as a resource descriptor: it points to (not contains!)
the resource data (which is normally of static storage class), and
contains + gives access to metadata such as the compression type
and resource size (compressed and uncompressed).
AbstractEmbeddedResource currently has two derived concrete
classes: RawEmbeddedResource for resources stored as-is
(uncompressed) and ZlibEmbeddedResource for resources compressed by
fgrcc. It's quite easy to add new subclasses if wanted, e.g. for
LZMA compression or other things.
Resource fetching requires two things:
- an std::string key (fgrcc manipulates them with SGPath, but the
EmbeddedResourceManager code in SimGear is so far completely
agnostic of the kind of data stored in keys; this could be
changed, though, if we wanted for example to be able to query
at runtime all available resources in a given âvirtual
directoryâ);
- a âlocaleâ name, similar to what FlightGear's XML translation
files and FGLocale use. We used double quotes here, because
fgrcc and the EmbeddedResourceManager expect âlocaleâ names to
be of one of these forms:
* empty string: default locale, typically but not necessarily
English (it is âengineering Englishâ in FlightGear, i.e.,
English written by programmers in the code, before
translators possibly fix it up :)
* en, fr, de, es, it...
* en_GB, en_US, fr_FR, fr_CA, de_DE, de_CH, it_IT...
There is no encoding part, contrary to POSIX locales, hence the
use of double quotes around the term âlocaleâ in this context.
The FGLocale::getPreferredLanguage() method returns the preferred
âlocaleâ in the form described above, according to user choice
(from fgfs' --language option) and/or settings (system locale).
This allows FG to tell the EmbeddedResourceManager the preferred
âlocaleâ for resource fetching (same syntax as in Qt's rcc tool for
declaration in the XML file, using the 'lang' attribute on
'qresource' elements).
[ Regarding the default locale, the way things are currently set
up, I would use no 'lang' attribute for resources suitable for
English in the XML input file for fgrcc, except when a
country-specific variant is desired (en_GB, en_US, en_AU...). In
such a case, there should also be a generic variant with no
'lang' attribute declared for the same resource virtual path.
This matches what I did for FGLocale::getPreferredLanguage(),
that maps unset locales and locales such as C and C.UTF-8 to the
default locale for the EmbeddedResourceManager, which is the
empty string. This is a matter of policy, of course, and could be
changed if desired. ]
The EmbeddedResourceManager class has getLocale() and
selectLocale() methods to manage the _selected locale_. Each
resource-fetching method of this class (getResourceOrNullPtr(),
getResource(), getString(), getStreambuf() and getIStream()) has
two overloads:
- one taking only a virtual path (the key mentioned above);
- one taking a virtual path and a âlocaleâ name.
(we'll write âlocaleâ without enclosing double-quotes from now on,
otherwise it gets too painful to read; but we're *not* talking
about POSIX-style locales ending with an encoding part)
The first kind of overload uses the selected locale to look up the
resource, whereas the second kind uses the explicitly specified
locale. Then resource lookup behaves as one could expect. For
instance, assuming a resource is looked up for in the "fr_FR"
locale, then the EmbeddedResourceManager tries in this order:
- "fr_FR";
- if no resource has been registered for "fr_FR" with the provided
virtual path, it then tries with the "fr" locale;
- if this is also unsuccessful, it finally tries with the default
locale: "";
- if this third attempt fails, the resource-fetching method
throws an sg_exception, except for getResourceOrNullPtr(),
which returns a null
std::shared_ptr<const AbstractEmbeddedResource> instead.
To see how this is used, you can look at
simgear/simgear/embedded_resources/embedded_resources_test.cxx. The
only difference with real use is that in this file, resource
contents and registering calls with the EmbeddedResourceManager
have been written manually instead of by fgrcc. Apart from
embedded_resources_test.cxx, here are two examples of client usage
of the EmbeddedResourceManager:
(a) With EmbeddedResourceManager::getString():
#include <simgear/embedded_resources/EmbeddedResourceManager.hxx>
#include <simgear/debug/logstream.hxx>
[...]
const auto& resMgr = simgear::EmbeddedResourceManager::instance();
SG_LOG(SG_GENERAL, SG_INFO,
"Resource contents: '" <<
resMgr->getString("/virtual/path/to/resource") << "'");
(b) With EmbeddedResourceManager::getIStream():
#include <cstddef> // std::size_t
#include <simgear/io/iostreams/sgstream.hxx>
#include <simgear/embedded_resources/EmbeddedResourceManager.hxx>
[...]
sg_ofstream outFile(SGPath("/tmp/whatever"));
if (!outFile) {
<handle open error>
}
const auto& resMgr = simgear::EmbeddedResourceManager::instance();
auto resStream = resMgr->getIStream("/virtual/path/to/resource");
// One possible way of handling errors from resStream[8]:
// resStream->exceptions(std::ios_base::badbit);
constexpr std::size_t bufSize = 4096;
std::unique_ptr<char[]> buf(new char[bufSize]); // intermediate buffer
do {
resStream->read(buf.get(), bufSize);
outFile.write(buf.get(), resStream->gcount());
} while (*resStream && outFile); // resStream *points* to an std::istream
<handle possible errors that might have caused to loop to stop
prematurely>
3) About the XML resource declaration files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You may want to read the output of 'fgrcc --help', which explains a few
things, in particular how to write an XML resource declaration file that
fgrcc can use. At the time of this writing, such files are already
present as flightgear/src/EmbeddedResources/FlightGear-resources.xml and
flightgear/src/EmbeddedResources/FGData-resources.xml in the FlightGear
repository. In case you need resources from elsewhere, it's easy to add
other XML resource declaration files:
1) If you want the .cxx/.hxx resource files to be automatically
generated as part of the FlightGear build:
Copy and adapt the add_custom_command() call in
flightgear/src/Main/CMakeLists.txt[9] that invokes fgrcc on
flightgear/src/EmbeddedResources/FlightGear-resources.xml.
2) In flightgear/src/Main/CMakeLists.txt, add paths for your new
fgrcc-generated .cxx and .hxx files to the SOURCES and HEADERS
CMake variables for the 'fgfs' target.
3) Assuming you passed for instance
--init-func-name=initFoobarEmbeddedResources in step 1, add a call
to initFoobarEmbeddedResources() after this code in fgMainInit()
(flightgear/src/Main/main.cxx):
simgear::EmbeddedResourceManager::createInstance();
initFlightGearEmbeddedResources();
4) The EmbeddedResourceProxy class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SimGear contains an EmbeddedResourceProxy class that allows one to
access real files or embedded resources in a unified way. When using it,
one can switch from one data source to the other with minimal code
changes, possibly even at runtime (in which case there is obviously no
code change at all).
Sample usage (from FlightGear):
simgear::EmbeddedResourceProxy proxy(globals->get_fg_root(), "/FGData");
proxy.setUseEmbeddedResources(false); // can also be set via the constructor
std::string s = proxy.getString("/some/path");
std::unique_ptr<std::istream> streamp = proxy.getIStream("/some/path");
This example would retrieve contents from the real file
$FG_ROOT/some/path. If true had been passed in the
proxy.setUseEmbeddedResources() call, it would instead have used the
default-locale version of the embedded resource whose virtual path is
/FGData/some/path.
For more information about this class, see [10] and [11].
Footnotes
=========
[1] E.g., FlightGear or FGData, as long as the path to the latter is
provided to the FG build system, which is currently possible but not
required (passing '-D FG_DATA_DIR:PATH=...' to CMake when
configuring the FlightGear build).
[2] The differences with the QRC format[3] are explained in the output
of 'fgrcc --help'. Here is the relevant excerpt:
,----
| 1. The <!DOCTYPE RCC> declaration at the beginning should be omitted (or
| replaced with <!DOCTYPE FGRCC>, however such a DTD currently doesn't
| exist). I suggest to add an XML declaration instead, for instance:
|
| <?xml version="1.0" encoding="UTF-8"?>
|
| 2. <RCC> and </RCC> must be replaced with <FGRCC> and </FGRCC>,
| respectively.
|
| 3. The FGRCC format supports a 'compression' attribute for each 'file'
| element. At the time of this writing, the allowed values for this
| attribute are 'none', 'zlib' and 'auto'. When set to a value that is
| not 'auto', this attribute of course bypasses the algorithm for
| determining whether and how to compress a given resource (algorithm
| which relies on the file extension).
|
| 4. Resource paths (paths to the real files, not virtual paths) are
| interpreted relatively to the directory specified with the --root
| option. If this option is not passed to 'fgrcc', then the default root
| directory is the one containing INFILE, which matches the behavior of
| Qt's 'rcc' tool.
`----
[3] http://doc.qt.io/qt-5/resources.html
[4] The main reason why I wrote the classes in
simgear/simgear/io/iostreams/{CharArrayStream,zlibstream}.cxx is
thus not to maximize memory-efficiency with very large resources;
rather, it is to make the implementation of the following parts
simple, clean and modular:
- the resource compiler (fgrcc);
- the EmbeddedResourceManager.
[5] The EmbeddedResourceManager architecture would make it quite easy to
also support runtime loading of resources from files (a thing the Qt
resource system supports), but it is not very clear how interesting
this would be, compared to having the files loaded from $FG_ROOT.
Well, maybe for large files [apt.dat.gz & Co] that we would want to
load but not see in the FGData repository at all. But then there
would be the requirement, of course, that âsomethingâ puts the files
in a clearly-defined, platform-dependent location known to the
EmbeddedResourceManager.
[6] https://sourceforge.net/p/flightgear/fgmeta/ci/next/tree/python3-flightgear/rebuild-fgdata-embedded-resources
[7] https://sourceforge.net/p/flightgear/simgear/ci/next/tree/simgear/embedded_resources/
[8] We know that in some buggy C++ implementations, the
std::ios_base::failure exception can't be caught, at least not under
its name, due to some ABI compatibility mess:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66145
However, it stills causes the program to abort, and since this
error handling technique makes for much more readable and less
error-prone code, I think it's still a good way to handle IOStreams
errors even now, unless you really need to *catch* the
std::ios_base::failure exception.
[9] flightgear/CMakeModules/GenerateFlightgearResources.cmake in my
'i18n-and-init-work-v2' branch (not merged into 'next' at the time
of this writing).
[10] https://sourceforge.net/p/flightgear/simgear/ci/next/tree/simgear/embedded_resources/EmbeddedResourceProxy.hxx
[11] https://sourceforge.net/p/flightgear/simgear/ci/next/tree/simgear/embedded_resources/embedded_resources_test.cxx
