<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<!--
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* Copyright by the Board of Trustees of the University of Illinois. *
* All rights reserved. *
* *
* This file is part of HDF5. The full HDF5 copyright notice, including *
* terms governing use, modification, and redistribution, is contained in *
* the files COPYING and Copyright.html. COPYING can be found at the root *
* of the source code distribution tree; Copyright.html can be found at the *
* root level of an installed copy of the electronic HDF5 document set and *
* is linked from the top-level documents page. It can also be found at *
* http://hdf.ncsa.uiuc.edu/HDF5/doc/Copyright.html. If you do not have *
* access to either file, you may request a copy from hdfhelp@ncsa.uiuc.edu. *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
-->
<head>
<title>Big Datasets on Small Machines</title>
</head>
<body>
<h1>Big Datasets on Small Machines</h1>
<h2>1. Introduction</h2>
<p>The HDF5 library is able to handle files larger than the
maximum file size, and datasets larger than the maximum memory
size. For instance, a machine where <code>sizeof(off_t)</code>
and <code>sizeof(size_t)</code> are both four bytes can handle
datasets and files as large as 18x10^18 bytes. However, most
Unix systems limit the number of concurrently open files, so a
practical file size limit is closer to 512GB or 1TB.
<p>Two "tricks" must be imployed on these small systems in order
to store large datasets. The first trick circumvents the
<code>off_t</code> file size limit and the second circumvents
the <code>size_t</code> main memory limit.
<h2>2. File Size Limits</h2>
<p>Systems that have 64-bit file addresses will be able to access
those files automatically. One should see the following output
from configure:
<p><code><pre>
checking size of off_t... 8
</pre></code>
<p>Also, some 32-bit operating systems have special file systems
that can support large (>2GB) files and HDF5 will detect
these and use them automatically. If this is the case, the
output from configure will show:
<p><code><pre>
checking for lseek64... yes
checking for fseek64... yes
</pre></code>
<p>Otherwise one must use an HDF5 file family. Such a family is
created by setting file family properties in a file access
property list and then supplying a file name that includes a
<code>printf</code>-style integer format. For instance:
<p><code><pre>
hid_t plist, file;
plist = H5Pcreate (H5P_FILE_ACCESS);
H5Pset_family (plist, 1<<30, H5P_DEFAULT);
file = H5Fcreate ("big%03d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, plist);
</code></pre>
<p>The second argument (<code>1<<30</code>) to
<code>H5Pset_family()</code> indicates that the family members
are to be 2^30 bytes (1GB) each although we could have used any
reasonably large value. In general, family members cannot be
2GB because writes to byte number 2,147,483,647 will fail, so
the largest safe value for a family member is 2,147,483,647.
HDF5 will create family members on demand as the HDF5 address
space increases, but since most Unix systems limit the number of
concurrently open files the effective maximum size of the HDF5
address space will be limited (the system on which this was
developed allows 1024 open files, so if each family member is
approx 2GB then the largest HDF5 file is approx 2TB).
<p>If the effective HDF5 address space is limited then one may be
able to store datasets as external datasets each spanning
multiple files of any length since HDF5 opens external dataset
files one at a time. To arrange storage for a 5TB dataset split
among 1GB files one could say:
<p><code><pre>
hid_t plist = H5Pcreate (H5P_DATASET_CREATE);
for (i=0; i<5*1024; i++) {
sprintf (name, "velocity-%04d.raw", i);
H5Pset_external (plist, name, 0, (size_t)1<<30);
}
</code></pre>
<h2>3. Dataset Size Limits</h2>
<p>The second limit which must be overcome is that of
<code>sizeof(size_t)</code>. HDF5 defines a data type called
<code>hsize_t</code> which is used for sizes of datasets and is,
by default, defined as <code>unsigned long long</code>.
<p>To create a dataset with 8*2^30 4-byte integers for a total of
32GB one first creates the dataspace. We give two examples
here: a 4-dimensional dataset whose dimension sizes are smaller
than the maximum value of a <code>size_t</code>, and a
1-dimensional dataset whose dimension size is too large to fit
in a <code>size_t</code>.
<p><code><pre>
hsize_t size1[4] = {8, 1024, 1024, 1024};
hid_t space1 = H5Screate_simple (4, size1, size1);
hsize_t size2[1] = {8589934592LL};
hid_t space2 = H5Screate_simple (1, size2, size2};
</pre></code>
<p>However, the <code>LL</code> suffix is not portable, so it may
be better to replace the number with
<code>(hsize_t)8*1024*1024*1024</code>.
<p>For compilers that don't support <code>long long</code> large
datasets will not be possible. The library performs too much
arithmetic on <code>hsize_t</code> types to make the use of a
struct feasible.
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Apr 10 13:26:04 EDT 1998 -->
<!-- hhmts start -->
Last modified: Sun Jul 19 11:37:25 EDT 1998
<!-- hhmts end -->
</body>
</html>
