<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html lang="en"> <head> <title> WebMake: Documentation: Data Sources </title> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /> <meta name="generator" content="WebMake/2.3" /> <style type="text/css"> body { background-color: #ffffff; color: #000000; line-height: 110%; margin-left: 10px; margin-right: 10px; } p, table, td, th { font-family: verdana,lucida,helvetica,sans-serif; font-size: 11px; line-height: 110%; } pre { margin-left: 3%; white-space: pre; } code, samp, pre, p pre { font-family: "lucida console", "Courier New", courier, "fixed-width", monospace; font-weight: bold; } H1 { font-size: 150%; font-family: Garamond, "Book Antiqua",Times,serif; background: #FFCC66; text-align: center; padding: 0.5em 1em 0.5em 1em; border-width: 1px; border-color: black; border-style: solid; line-height: 120%; } H2 { font-size: 125%; font-family: Garamond, "Book Antiqua",Times,serif; background: #FFDD77; text-align: center; padding: 0.5em 1em 0.5em 1em; border-width: 1px; border-color: black; border-style: solid; line-height: 100%; } H3 { font-size: 100%; font-family: Garamond, "Book Antiqua",Times,serif; background: #FFEE88; text-align: center; padding: 0.5em 1em 0.5em 1em; border-width: 1px; border-color: black; border-style: solid; } H4 { font-size: 75%; font-family: Garamond, "Book Antiqua",Times,serif; } H5 { font-size: 50%; font-family: Garamond, "Book Antiqua",Times,serif; } H6 { font-size: 25%; font-family: Garamond, "Book Antiqua",Times,serif; } A:link { font-weight: bold; color: #004000; text-decoration: underline; } A:visited { font-weight: bold; color: #008000; text-decoration: underline; } A:active { font-weight: bold; color: #800000; text-decoration: underline; } dt { font-size: medium; font-weight: bold; padding-top: 8px; padding-bottom: 8px; } dd { padding-top: 8px; padding-bottom: 8px; } </style> </head> <body bgcolor="#ffffff" text="#000000" link="#3300cc" vlink="#660066"> <!-- font tag for compat with non-CSS browsers --> <font face="lucida,verdana,sans-serif"> <div align="center"> <img src="images/WebMakeTitle.png" alt="WebMake" width="500" height="122" /> </div> <table width="100%"> <tr> <td valign="top"> <strong><a href="http://webmake.taint.org/">WebMake</a> Documentation</strong> (version 2.3) </td> <td valign="top"> <div align="right"> [ <a href="media.html">Back</a> | <a href="for.html">Forward</a> | <a href="index.html">Index</a> | <a href="allinone.html">All In One</a> ] </div> </td> </tr> </table> <!-- yes, it's that Mozilla black-border code again ;) --> <!-- stolen from www.mozilla.org via rc3.org --> <table border="0" cellpadding="0" cellspacing="0" width="100%"> <tr> <td bgcolor="#aaaaaa"> <table border="0" cellspacing="4" cellpadding="4" width="100%"> <tr> <td bgcolor="#ffffff"> <table border="0" cellspacing="4" cellpadding="4" width="100%"> <tr> <td> <h1>Data Sources</h1><p> Contents or URLs can be searched for using the <a href="contents.html"><contents></a>, <a href="templates.html"><templates></a> or <a href="media.html"><media></a> tags, which allow you to search a data source (directory, delimiter-separated-values file, database etc.) for a pattern. </p> <p> <a href="contents.html"><contents></a> and <a href="media.html"><media></a> tags can also pick up metadata from metatable files while searching for content or media items, using the <code>metatable</code> attribute. </p> <p> Currently two data source protocols are defined, <a href="#file">file:</a> and <a href="#svfile">svfile:</a> . </p> <p> <a name="Attributes_Supported_By_Datasource_Tags" id="Attributes_Supported_By_Datasource_Tags"><h2>Attributes Supported By Datasource Tags</h2></a><dl> <dt> <em>src</em> </dt> <dd> All datasources require this attribute, which specifies a protocol and path, in a URL-style syntax: <strong>protocol</strong>:<strong>path</strong> . file: is the default protocol, if none is specified. </dd> <dt> <em>name</em> </dt> <dd> This attribute is used to specify the pattern of data, under this path, which will be converted into content or media items. The part of the data's location which matches this name pattern will become the name of the item. Typically, WebMake glob patterns, such as "*.txt" or ".../*.html" are used. </dd> <dt> <em>skip</em> </dt> <dd> A pattern which should match filenames that should be skipped. Files that match this pattern will not be included as content or media items, or as metatables. Glob patterns, again, are used here. </dd> <dt> <em>prefix</em> </dt> <dd> The items' names can be further modified by specifying a <em>prefix</em> and/or <em>suffix</em>; these strings are prepended or appended to the raw name to make the name the content is given. </dd> <dt> <em>suffix</em> </dt> <dd> See above. </dd> <dt> <em>namesubst</em> </dt> <dd> a Perl-formatted s// substitution, which is used to convert source filenames to content names. See the example under <strong>The File: Protocol</strong>, below. </dd> <dt> <em>nametr</em> </dt> <dd> a Perl tr// translation, which is used to convert source filenames to content names. </dd> <dt> <em>listname</em> </dt> <dd> a name of a content item. This content item will be created, and will contain the names of all content items picked up by the <a href="contents.html"><contents></a> or <media> search. </dd> <dt> <em><a href="metatable.html">metatable</a></em> </dt> <dd> a search pattern, similar to <em>name</em> above, which provides filenames from which <a href="metadata.html">metadata</a> will be loaded. </dd> <dt> <em>metatableformat</em> </dt> <dd> The format for the above <a href="metatable.html">metatable</a> files. </dd> </dl> <p> In addition, the attributes supported by the <a href="content.html">content tag</a> can be specified as attributes to <a href="contents.html"><contents></a>, including <em>format</em>, <em>up</em>, <em>map</em>, etc. </p> <p> Also, the attributes supported by the <a href="metatable.html"><metatable></a> tag can be used if you've specified a <em><a href="metatable.html">metatable</a></em> attribute. Note that <em>metatableformat</em> should be used instead of <em>format</em>, as <em>format</em> is already used for the content items. </p> <p> The content blocks picked up from a <a href="contents.html"><contents></a> search can also contain meta-data, such as headlines, visibilty dates, workflow approval statuses, etc. by including <a href="wmmeta.html">metadata</a>. </p> <br /> <a name="file"> <a name="The_file_Protocol" id="The_file_Protocol"><h2>The file: Protocol</h2></a></a><p> The file: protocol loads content from a directory; each file is made into one content chunk. The <em>src</em> attribute indicates the source directory, the <em>name</em> attribute indicates the <a href="globs.html">glob pattern</a> that will pick up the content items in question. </p> <blockquote> <contents src="stories" name="*.txt" /> </blockquote> <p> The filename of the file will be used as the content chunk's name -- unless you use the <em>namesubst</em> command; see below for details on this. </p> <p> Note that, for efficiency, the files in question are not actually opened until their content chunks are referenced using ${name} or get_content("name"). </p> <a name="Searching_Recursively_Through_A_Directory_Tree" id="Searching_Recursively_Through_A_Directory_Tree"><h3>Searching Recursively Through A Directory Tree</h3></a><p> Normally only the top level of files inside the <em>src</em> directory are added to the content set. However, if the <em>name</em> pattern starts with <strong>.../</strong>, the directory will be searched recursively: </p> <blockquote> <contents src="stories" name="<strong>.../</strong>*.txt" /> </blockquote> <p> The resulting content items will contain the full path from that directory down, i.e. the file <strong>stories/dir1/foo/bar.txt</strong> exists, the example above would define a content item called <strong>${dir1/foo/bar.txt}</strong>. </p> <a name="The_namesubst_Option" id="The_namesubst_Option"><h3>The namesubst Option</h3></a><p> If you use the <em>namesubst</em> command, the filename will be modified using that substitution, to give the content item's name. So, for example, this contents tag: </p> <blockquote> <contents src="stories" name="*.txt" namesubst="s/.txt//" /> </blockquote> <p> will load these example files as follows: </p> <blockquote> <table> <tr> <th> <em>Filename</em> </th> <th> <em>Content Name</em> </th> </tr> <tr> <td> stories/index.txt </td> <td> ${index} </td> </tr> <tr> <td> stories/foo.txt </td> <td> ${foo} </td> </tr> <tr> <td> stories/directory/bar.txt </td> <td> ${directory/bar} </td> </tr> <tr> <td> stories/zz/gum/baz.txt </td> <td> ${zz/gum/baz} </td> </tr> </table> </blockquote> <a name="Loading_Metadata_Using_the_Metatable_Attribute" id="Loading_Metadata_Using_the_Metatable_Attribute"><h3>Loading Metadata Using the Metatable Attribute</h3></a><p> You can now load <a href="metadata.html">metadata</a> from external files while searching a directory tree for content items or media files. This allows you to load image titles, etc. from files which match the filename pattern you specify in the <em><a href="metatable.html">metatable</a></em> attribute. </p> <p> The attributes supported by the <a href="metatable.html"><metatable></a> tag can be used in the datasource tag's attribute set, if you've specified a <em><a href="metatable.html">metatable</a></em> attribute, allowing you to define the format of the <a href="metatable.html">metatable</a> files you expect to find. </p> <p> There's one major difference between normal metatables and metatables found via a data source; the names in this kind of <a href="metatable.html">metatable</a> refer to the content or media object's <em>filename</em>, not its <em>content name</em>. </p> <p> In other words, the names of any content items referred to in the <a href="metatable.html">metatable</a> files will be modified, as follows: </p> <ul> <li> if the <em>name</em> attribute contains <code>.../</code>, then the content items could be deep in a subdirectory. The <a href="metatable.html">metatable</a> file does not have to contain the full path to the content item's name; it can just contain the item's filename <em>relative to the <a href="metatable.html">metatable</a> itself</em>. </li> <li> if a <em>namesubst</em> or <em>nametr</em> function is specified, the content names in the <a href="metatable.html">metatable</a> will be processed with this. Again, this means that the <a href="metatable.html">metatable</a> data just has to provide the <em>filename</em>, not whatever the resulting content item will be called. </li> </ul> <p> These features will hopefully make the operation a little more intuitive, as users who add files to a media or contents directory will not have to figure out what the resulting content item will be called; they can just refer to them by their filename, when tagging them with <a href="metadata.html">metadata</a>. </p> <br /> <a name="svfile"> <a name="The_svfile_Protocol" id="The_svfile_Protocol"><h2>The svfile: Protocol</h2></a></a><p> The svfile: protocol loads content from a delimiter-separated-file; the <em>src</em> attribute is the name of the file, the <em>name</em> is the <a href="globs.html">glob pattern</a> used to catch the relevant content items. The <em>namefield</em> attribute specifies the field number (counting from 1) which the <em>name</em> pattern is matched against, and the <em>valuefield</em> specifies the number of the field from which the content chunk is read. The <em>delimiter</em> attribute specifies the delimiter used to separate values in the file. </p> <blockquote> <contents src="svfile:stories.csv" name="*" <blockquote> <p> <br /> namefield=1 valuefield=2 delimiter="," /> </p> </blockquote> </blockquote> <br /> <a name="Adding_New_Protocols" id="Adding_New_Protocols"><h2>Adding New Protocols</h2></a><p> New data sources for <contents> and <media> tags are added by writing an implementation of the <em>DataSourceBase.pm</em> module, in the <em>HTML::WebMake::DataSources</em> package space (the <em>lib/HTML/WebMake/DataSources</em> directory of the distribution). </p> <p> Every data source needs a <em>protocol</em>, an alphanumeric lowercase identifier to use at the start of the <em>src</em> attribute to indicate that a data source is of that type. </p> <p> Each implementation of this module should implement these methods: </p> <dl> <dt> <code>new ($parent)</code> </dt> <dd> instantiate the object, as usual. </dd> <dt> <code>add ()</code> </dt> <dd> add all the items in that data source as content chunks. (See below!) </dd> <dt> <code>get_location_url ($location)</code> </dt> <dd> get the location (in URL format) of a content chunk loaded by <code>add()</code>. </dd> <dt> <code>get_location_contents ($location)</code> </dt> <dd> get the contents of the location. The location, again, is the string provided by <code>add()</code>. </dd> <dt> <code>get_location_mod_time ($location)</code> </dt> <dd> get the current modification date of a location for dependency checking. The location, again, is in the format of the string provided by <code>add()</code>. </dd> </dl> <p> Notes: </p> <ul> <li> If you want <code>add()</code> to read the content immediately, call <code>$self->{parent}->add_text ($name, $text, $self->{src}, $modtime)</code>. </li> <li> <code>add()</code> can defer opening and reading content chunks straight away. If it calls <code>$self->{parent}->add_location ($name, $location, $lastmod)</code>, providing a location string which starts with the data source's protocol identifier, the content will not be loaded until it is needed, at which point <code>get_location_contents()</code> is called. </li> <li> This location string should contain all the information needed to access that content chunk later, even if <code>add()</code> was not been called. Consider it as similar to a URL. This is required so that <code>get_location_mod_time()</code> (see below) can work. </li> <li> All implementations of <code>add()</code> should call <code>$fixed = $self->{parent}->fixname ($name);</code> to modify the name of each content chunk appropriately, followed by <br /> <code>$self->{parent}->add_file_to_list ($fixed);</code> to add the content chunk's name to the <em>filelist</em> content item. </li> <li> Data sources that support the <media> tag need to implement <code>get_location_url</code>, otherwise an error message will be output. </li> <li> Data sources that support the <contents> tag, and defer reading the content until it's required, need to implement <code>get_location_contents</code>, which is used to provide content from a location set using <code>$self->{parent}->add_location()</code>. </li> <li> <p> Data sources that support the <contents> tag need to implement <code>get_location_mod_time</code>. This is used to support dependency checking, and should return the modification time (in UNIX <code>time_t</code> format) of that location. Note that since this is used to compare the modification time of a content chunk from the previous time webmake was run, and the current modification time, this is called before the <strong>real</strong> data source is opened. </p> </li> </ul> </p> </td> </tr> </table> </td> </tr> </table> </td> </tr> </table> <table width="100%"> <tr> <td valign="top"> <strong><a href="http://webmake.taint.org/">WebMake</a> Documentation</strong> (version 2.3) </td> <td valign="top"> <div align="right"> [ <a href="media.html">Back</a> | <a href="for.html">Forward</a> | <a href="index.html">Index</a> | <a href="allinone.html">All In One</a> ] </div> </td> </tr> </table> <div align="right"> <a href="http://webmake.taint.org/"> <img src="images/BuiltWithWebMake.png" alt="Built With WebMake" border="0" width="88" height="31" /></a> </div> </font> </body> </html>