<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>In-memory HDF5 files — PyTables 3.0.0 documentation</title>
<link rel="stylesheet" href="../_static/cloud.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '3.0.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/jquery.cookie.js"></script>
<script type="text/javascript" src="../_static/toggle_sections.js"></script>
<script type="text/javascript" src="../_static/toggle_sidebar.js"></script>
<link rel="shortcut icon" href="../_static/favicon.ico"/>
<link rel="top" title="PyTables 3.0.0 documentation" href="../index.html" />
<link rel="up" title="PyTables Cookbook" href="index.html" />
<link rel="next" title="FAQ" href="../FAQ.html" />
<link rel="prev" title="SimpleTable: simple wrapper around the Table object" href="simple_table.html" />
</head>
<body>
<div class="relbar-top">
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> </li>
<li class="right" >
<a href="../np-modindex.html" title="Python Module Index"
>modules</a> </li>
<li class="right" >
<a href="../FAQ.html" title="FAQ"
accesskey="N">next</a> </li>
<li class="right" >
<a href="simple_table.html" title="SimpleTable: simple wrapper around the Table object"
accesskey="P">previous</a> </li>
<li><a href="../index.html">PyTables 3.0.0 documentation</a> »</li>
<li><a href="index.html" accesskey="U">PyTables Cookbook</a> »</li>
</ul>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="in-memory-hdf5-files">
<h1>In-memory HDF5 files<a class="headerlink" href="#in-memory-hdf5-files" title="Permalink to this headline">¶</a></h1>
<p>The HDF5 library provides functions to allow an application to work with a
file in memory for faster reads and writes. File contents are kept in memory
until the file is closed. At closing, the memory version of the file can be
written back to disk or abandoned.</p>
<div class="section" id="open-an-existing-file-in-memory">
<h2>Open an existing file in memory<a class="headerlink" href="#open-an-existing-file-in-memory" title="Permalink to this headline">¶</a></h2>
<p>Assuming the <tt class="file docutils literal"><span class="pre">sample.h5</span></tt> exists in the current folder, it is possible to
open it in memory simply using the CORE driver at opening time.</p>
<p>The HDF5 driver that one intend to use to open/create a file can be specified
using the <em>driver</em> keyword argument of the <tt class="xref py py-func docutils literal"><span class="pre">tables.openFile()</span></tt> function:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">tables</span>
<span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"sample.h"</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s">"H5FD_CORE"</span><span class="p">)</span>
</pre></div>
</div>
<p>The content of the :file`sample.h5` is opened for reading. It is loaded into
memory and all reading operations are performed without disk I/O overhead.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">the initial loading of the entire file into memory can be time expensive
depending on the size of the opened file and on the performances of the
disk subsystem.</p>
</div>
<div class="admonition-see-also admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">general information about HDF5 drivers can be found in the <a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/UG/08_TheFile.html#Drivers">Alternate
File Storage Layouts and Low-level File Drivers</a> <a class="footnote-reference" href="#id6" id="id7">[3]</a> section of the <a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/UG/index.html">HDF5
User’s Guide</a> <a class="footnote-reference" href="#id2" id="id3">[1]</a>.</p>
</div>
</div>
<div class="section" id="creating-a-new-file-in-memory">
<h2>Creating a new file in memory<a class="headerlink" href="#creating-a-new-file-in-memory" title="Permalink to this headline">¶</a></h2>
<p>Creating a new file in memory is as simple as creating a regular file, just
one needs to specify to use the CORE driver:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">tables</span>
<span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"new_sample.h5"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s">"H5FD_CORE"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">h5file</span><span class="o">.</span><span class="n">createArray</span><span class="p">(</span><span class="n">h5file</span><span class="o">.</span><span class="n">root</span><span class="p">,</span> <span class="s">"array"</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="mi">300</span><span class="p">,</span> <span class="mi">300</span><span class="p">)))</span>
<span class="gp">>>> </span><span class="n">h5file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="section" id="backing-store">
<h2>Backing store<a class="headerlink" href="#backing-store" title="Permalink to this headline">¶</a></h2>
<p>In the previous example contents of the in-memory <cite>h5file</cite> are automatically
saved to disk when the file descriptor is closed, so a new
<tt class="file docutils literal"><span class="pre">new_sample.h5</span></tt> file is created and all data are transferred to disk.</p>
<p>Again this can be time a time expensive action depending on the amount of
data in the HDF5 file and depending on how fast is the disk I/O.</p>
<p>Saving data to disk is the default behavior for the CORE driver in PyTables.</p>
<p>This feature can be controlled using the <em>driver_core_backing_store</em>
parameter of the <tt class="xref py py-func docutils literal"><span class="pre">tables.openFile()</span></tt> function. Setting it to <cite>False</cite>
disables the backing store feature and all changes in the working <cite>h5file</cite>
are lost after closing:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"new_sample.h5"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s">"H5FD_CORE"</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">driver_core_bacling_store</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
</pre></div>
</div>
<p>Please note that the <em>driver_core_backing_store</em> disables saving of data, not
loading.
In the following example the <tt class="file docutils literal"><span class="pre">sample.h5</span></tt> file is opened in-memory in
append mode. All data in the existing <tt class="file docutils literal"><span class="pre">sample.h5</span></tt> file are loaded into
memory and contents can be actually modified by the user:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">tables</span>
<span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"sample.h5"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s">"H5FD_CORE"</span><span class="p">,</span>
<span class="go"> driver_core_backing_store=0)</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span>
<span class="gp">>>> </span><span class="n">h5file</span><span class="o">.</span><span class="n">createArray</span><span class="p">(</span><span class="n">h5file</span><span class="o">.</span><span class="n">root</span><span class="p">,</span> <span class="s">"new_array"</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">20</span><span class="p">),</span>
<span class="go"> title="New array")</span>
<span class="gp">>>> </span><span class="n">array2</span> <span class="o">=</span> <span class="n">h5file</span><span class="o">.</span><span class="n">root</span><span class="o">.</span><span class="n">array2</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">array2</span>
<span class="go">/array2 (Array(20,)) 'New array'</span>
<span class="gp">>>> </span><span class="n">h5file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>Modifications are lost when the <cite>h5file</cite> descriptor is closed.</p>
</div>
<div class="section" id="memory-images-of-hdf5-files">
<h2>Memory images of HDF5 files<a class="headerlink" href="#memory-images-of-hdf5-files" title="Permalink to this headline">¶</a></h2>
<p>It is possible to get a memory image of an HDF5 file (see
<a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf">HDF5 File Image Operations</a> <a class="footnote-reference" href="#id4" id="id5">[2]</a>). This feature is only available if PyTables
is build against version 1.8.9 or newer of the HDF5 library.</p>
<p>In particular getting a memory image of an HDF5 file is possible only if the
file has been opened with one of the following drivers: SEC2 (the default
one), STDIO or CORE.</p>
<p>An example of how to get an image:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">tables</span>
<span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"sample.h5"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">image</span> <span class="o">=</span> <span class="n">h5file</span><span class="o">.</span><span class="n">get_file_image</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">h5file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>The memory ìmage of the <tt class="file docutils literal"><span class="pre">sample.h5</span></tt> file is copied into the <cite>ìmage</cite>
string (of bytes).</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">the <cite>ìmage</cite> string contains all data stored in the HDF5 file so, of
course, it can be quite large.</p>
</div>
<p>The <cite>ìmage</cite> string can be passed around and can also be used to initialize a
new HDF55 file descriptor:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">tables</span>
<span class="gp">>>> </span><span class="n">h5file</span> <span class="o">=</span> <span class="n">tables</span><span class="o">.</span><span class="n">openFile</span><span class="p">(</span><span class="s">"in-memory-sample.h5"</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s">"H5DF_CORE"</span><span class="p">,</span>
<span class="go"> driver_core_backing_store=0)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">h5file</span><span class="o">.</span><span class="n">root</span><span class="o">.</span><span class="n">array</span>
<span class="go">/array (Array(300, 300)) 'Array'</span>
<span class="gp">>>> </span><span class="n">h5file</span><span class="o">.</span><span class="n">setNodeAttr</span><span class="p">(</span><span class="n">h5file</span><span class="o">.</span><span class="n">root</span><span class="p">,</span> <span class="s">"description"</span><span class="p">,</span> <span class="s">"In memory file example"</span><span class="p">)</span>
</pre></div>
</div>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="id2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[1]</a></td><td><a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/UG/index.html">http://www.hdfgroup.org/HDF5/doc/UG/index.html</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[2]</a></td><td><a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf">http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id7">[3]</a></td><td><a class="reference external" href="http://www.hdfgroup.org/HDF5/doc/UG/08_TheFile.html#Drivers">http://www.hdfgroup.org/HDF5/doc/UG/08_TheFile.html#Drivers</a></td></tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/logo-pytables-small.png" alt="Logo"/>
</a></p>
<h3><a href="../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">In-memory HDF5 files</a><ul>
<li><a class="reference internal" href="#open-an-existing-file-in-memory">Open an existing file in memory</a></li>
<li><a class="reference internal" href="#creating-a-new-file-in-memory">Creating a new file in memory</a></li>
<li><a class="reference internal" href="#backing-store">Backing store</a></li>
<li><a class="reference internal" href="#memory-images-of-hdf5-files">Memory images of HDF5 files</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="simple_table.html"
title="previous chapter">SimpleTable: simple wrapper around the Table object</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../FAQ.html"
title="next chapter">FAQ</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/cookbook/inmemory_hdf5_files.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="relbar-bottom">
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> </li>
<li class="right" >
<a href="../np-modindex.html" title="Python Module Index"
>modules</a> </li>
<li class="right" >
<a href="../FAQ.html" title="FAQ"
>next</a> </li>
<li class="right" >
<a href="simple_table.html" title="SimpleTable: simple wrapper around the Table object"
>previous</a> </li>
<li><a href="../index.html">PyTables 3.0.0 documentation</a> »</li>
<li><a href="index.html" >PyTables Cookbook</a> »</li>
</ul>
</div>
</div>
<div class="footer">
© Copyright 2011-2013, PyTables maintainers.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
<!-- cloud_sptheme 1.3 -->
</body>
</html>
