<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Base classes — Translate Toolkit 1.9.0 documentation</title>
<link rel="stylesheet" href="../_static/basic.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/bootstrap.css" type="text/css" />
<link rel="stylesheet" href="../_static/bootstrap-sphinx.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '1.9.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/bootstrap.js"></script>
<script type="text/javascript" src="../_static/bootstrap-sphinx.js"></script>
<link rel="top" title="Translate Toolkit 1.9.0 documentation" href="../index.html" />
<link rel="up" title="Translation Related File Formats" href="index.html" />
<link rel="next" title="Quoting and Escaping" href="quoting_and_escaping.html" />
<link rel="prev" title="Standards conformance" href="conformance.html" />
</head>
<body>
<div id="navbar" class="navbar navbar-fixed-top">
<div class="navbar-inner">
<div class="container-fluid">
<a class="brand" href="../index.html">Translate Toolkit</a>
<span class="navbar-text pull-left"><b>1.9.0</b></span>
<ul class="nav">
<li class="divider-vertical"></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Site <b class="caret"></b></a>
<ul class="dropdown-menu globaltoc"><ul class="simple">
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../features.html">Features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../commands/index.html">Converters</a></li>
<li class="toctree-l1"><a class="reference internal" href="../commands/index.html#tools">Tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../commands/index.html#scripts">Scripts</a></li>
<li class="toctree-l1"><a class="reference internal" href="../guides/index.html">Use Cases</a></li>
<li class="toctree-l1"><a class="reference internal" href="index.html">Supported formats</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../styleguide.html">Translate Styleguide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../styleguide.html#documentation">Documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development/building.html">Building</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development/contributing.html">Contributing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development/developers.html">Translate Toolkit Developers Guide</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../api/index.html">API</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../changelog.html">Important Changes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../history.html">History of the Translate Toolkit</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
</ul>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Page <b class="caret"></b></a>
<ul class="dropdown-menu localtoc"><ul>
<li><a class="reference internal" href="#">Base classes</a><ul>
<li><a class="reference internal" href="#requirements">Requirements</a></li>
<li><a class="reference internal" href="#the-classes">The classes</a><ul>
<li><a class="reference internal" href="#message-string-multistring">Message/string (multistring)</a></li>
<li><a class="reference internal" href="#translation-unit">Translation unit</a></li>
<li><a class="reference internal" href="#store">Store</a></li>
<li><a class="reference internal" href="#multifile">Multifile</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#additional-notes">Additional Notes</a></li>
</ul>
</ul>
</li>
<li><a href="conformance.html"
title="previous chapter">« Standards conformance</a></li>
<li><a href="quoting_and_escaping.html"
title="next chapter">Quoting and Escaping »</a></li>
</ul>
<form class="navbar-search pull-right" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</ul>
</div>
</div>
</div>
</div>
<div class="container content">
<div class="section" id="base-classes">
<span id="id1"></span><h1>Base classes<a class="headerlink" href="#base-classes" title="Permalink to this headline">¶</a></h1>
<p>NOTE: This page is mostly useful for <tt class="xref doc docutils literal"><span class="pre">developers</span></tt> as it describes some
programming detail of the <a class="reference internal" href="index.html"><em>toolkit</em></a>.</p>
<p>For the implementation of the different storage classes that the toolkit
supports, we want to define a set of base classes to form a common API for all
formats. This will simplify implementation of new storage formats, and enable
easy integration into external tools, such as Pootle. It will also mean less
duplication of code in similar storage formats.</p>
<p>These ideas explained here should be seen as drafts only.</p>
<div class="section" id="requirements">
<span id="base-classes-requirements"></span><h2>Requirements<a class="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>
<p>The base classes should be rich enough in functionality to enable users of the
base classes to have access to all or most of the features that are available
in the formats. In particular, the following are considered requirements:</p>
<ul class="simple">
<li>Seamless and hidden handling of escaping, quoting and character sets</li>
<li>Parsing a file when given a file name or file contents (whole file in a
string)</li>
<li>Writing a file to disk</li>
<li>Getting and setting source and target languages</li>
<li>Accessing units, and determining if they are translatable, translated, a
unique identifier for the unit in the file, etc.</li>
<li>Support for plural units that can vary between different languages (as the PO
format allows with msgid_plural, etc.)</li>
</ul>
<p>Other possibilities:</p>
<ul class="simple">
<li>Support for variable number of languages in the format. Examples: .txt and
.properties support one language, PO supports two, <a class="reference internal" href="tmx.html"><em>TMX</em></a> supports many.</li>
<li>Support for “multifiles”, in other words a file that contain other entities
that corresponds to files in other formats. Examples: ZIP, XPI and
<a class="reference internal" href="xliff.html"><em>XLIFF</em></a>. In reality this is only used by some of the converters. This
isn’t present in the base class yet.</li>
</ul>
<p>All these do not mean that all formats must support al these features, but in
the formats that do support these features, it must be accessible through the
base class, and it must be possible to interrogate the storage format through
the base class to know which features it supports.</p>
</div>
<div class="section" id="the-classes">
<span id="base-classes-the-classes"></span><h2>The classes<a class="headerlink" href="#the-classes" title="Permalink to this headline">¶</a></h2>
<p>A file contains a number of translation units, and possibly a header. Each
translation unit contains one or more strings corresponding to each of the
languages represented in that unit.</p>
<div class="section" id="message-string-multistring">
<span id="base-classes-message-string-multistring"></span><h3>Message/string (multistring)<a class="headerlink" href="#message-string-multistring" title="Permalink to this headline">¶</a></h3>
<p>This class represents a single conceptual string in a single language. It must
know its own requirements for escaping and implement it internally. Escaped
versions are only used for internal representation and only exposed for file
creation and testing (unit tests, for example).</p>
<p>Note that when storing different plural forms of the same string, they should
be stored in this class. The main object is the singular string, and all of the
string forms can be accessed in a list at x.strings. Most of the time the
object can be dealt with as a single string, only when it is necessary to deal
with plural forms do the extra strings have to be taken into account.</p>
<p>Any string from a plural unit must be a multistring.</p>
</div>
<div class="section" id="translation-unit">
<span id="base-classes-translation-unit"></span><h3>Translation unit<a class="headerlink" href="#translation-unit" title="Permalink to this headline">¶</a></h3>
<p>This class represents a unit of one or several related messages/strings. In
most formats the contained strings will be translations of some original
message/string. It must associate a language value with each message/string. It
must know how to join all contained messages/strings to compile a valid
representation. For formats that support at least two languages, the first two
languages will serve as “source” and “target” languages for the common case of
translating from one language into another language.</p>
<p>Some future ideas:</p>
<p>As the number of of languages can be seen as one “dimension” of the translation
unit, plurality can be seen as a second dimension. A format can thus be
classified according to the dimensionality that it supports, as follows:</p>
<ul class="simple">
<li>.properties files supports one language and no concept of plurals. This
include most document types, such as .txt, HTML and OpenDocument formats.</li>
<li>Old style PO files supported two languages and no plurals.</li>
<li>New style PO files support two languages and any number of plurals as
required by the target language. The plural forms are stored in the original
or target strings, as extra forms of the string (See message/string class
above).</li>
<li>TMX files support any number of languages, but has no concept of plurality.</li>
</ul>
<p>Comments/notes are supported in this class. Quality or status information
(fuzzy, last-changed-by) should be stored. TODO: see if this should be on unit
level or language level.</p>
</div>
<div class="section" id="store">
<span id="base-classes-store"></span><h3>Store<a class="headerlink" href="#store" title="Permalink to this headline">¶</a></h3>
<p>This class represents a whole collection of translation units, usually stored
in a single file. It supports the concept of a header, and possibly comments at
file level. A file will not necessarily be contained alone in single file on
disc. See “multifile” below.</p>
</div>
<div class="section" id="multifile">
<span id="base-classes-multifile"></span><h3>Multifile<a class="headerlink" href="#multifile" title="Permalink to this headline">¶</a></h3>
<p>This abstraction is only used by a few converters.</p>
<p>This class represents a storage format that contains other files or file like
objects. Examples include ZIP, XPI and XLIFF, and OpenOffice SDF files. It must
give access to the contained files, and possibly give access to the translation
units contained in those files, as if they are contained natively.</p>
</div>
</div>
</div>
<div class="section" id="additional-notes">
<span id="base-classes-additional-notes"></span><h1>Additional Notes<a class="headerlink" href="#additional-notes" title="Permalink to this headline">¶</a></h1>
<p>Dwayne and I (Andreas) discussed cleaning up the storage base class. A lot of
what we discussed is related to the above. A quick summary:</p>
<ul>
<li><p class="first">Implement a new base class.</p>
<ul class="simple">
<li>Flesh out the API, clean and clear definitions.</li>
<li>Document the API.</li>
</ul>
</li>
<li><p class="first">We need to discuss the class hierarchy, e.g.:</p>
<div class="highlight-python"><pre>base
-- po
-- text
-- xml
-- xhtml
-- lisa
-- xliff
-- tmx
-- tbx</pre>
</div>
</li>
<li><p class="first">Clean up converters.</p>
<ul>
<li><p class="first">Parsing of file content needs to happen only in the storage implementation
of each filetype/storage type. Currently parsing happens all over the
place.</p>
</li>
<li><p class="first">Currently there are separate conversion programs for each type and
direction to convert to, e.g. po2xliff and xliff2po (24 commands with lots
of duplicate code in them). Ideally conversion should be as simple as:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">po_store</span> <span class="o">=</span> <span class="n">POStore</span><span class="p">(</span><span class="n">filecontent</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="nb">str</span><span class="p">(</span><span class="n">po_store</span><span class="p">)</span>
<span class="go">msgid "bleep"</span>
<span class="go">msgstr "blorp"</span>
<span class="gp">>>> </span><span class="n">xliff_store</span> <span class="o">=</span> <span class="n">XliffStore</span><span class="p">(</span><span class="n">po_store</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="nb">str</span><span class="p">(</span><span class="n">xliff_store</span><span class="p">)</span>
<span class="go"><xliff></span>
<span class="go"> <file></span>
<span class="go"> <trans-unit></span>
<span class="go"> <source>bleep</source></span>
<span class="go"> <target>blorp</target></span>
<span class="go"> </trans-unit></span>
<span class="go"> </file></span>
<span class="go"></xliff></span>
</pre></div>
</div>
</li>
</ul>
</li>
</ul>
<p>Note that the xliffstore is being instantiated using the postore object. This
works because all the data in any translation store object is accessible via
the same well-defined base API. A concept class implementing the above code
snippet has already been written.</p>
<ul class="simple">
<li>Move certain options into their respective storage classes.<ul>
<li>e.g. the <tt class="docutils literal"><span class="pre">--duplicates</span></tt> option can move into po.py</li>
</ul>
</li>
<li>Store the meta data for a storage object.<ul>
<li>Can be implemented as separate sqlite file that accompanies the real file.</li>
<li>Features not directly supported by a file format can be stored in the
metadata file.</li>
</ul>
</li>
<li>A storage object should know all information pertaining to itself.<ul>
<li>e.g. “am I monolingual?”</li>
</ul>
</li>
<li>We should discuss how to make an object aware that it is monolingual,
bilingual or multilingual.<ul>
<li>Maybe through mixin-classes?</li>
<li>How will the behaviour of a monolingual store differ from a bilingual
store?</li>
</ul>
</li>
</ul>
</div>
</div>
<hr>
<footer class="footer">
<div class="container">
<p class="pull-right"><a href="#">Back to top ↑</a></p>
<ul class="unstyled muted">
<li><small>
© 2012, Translate.org.za.<br/>
</small></li>
<li><small>
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</small></li>
</ul>
</div>
</footer>
</body>
</html>
