<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head profile="http://internetalchemy.org/2003/02/profile">
<link rel="foaf" type="application/rdf+xml" title="FOAF" href="http://www.openlinksw.com/dataspace/uda/about.rdf" />
<link rel="schema.dc" href="http://purl.org/dc/elements/1.1/" />
<meta name="dc.title" content="6. Administration" />
<meta name="dc.subject" content="6. Administration" />
<meta name="dc.creator" content="OpenLink Software Documentation Team ; " />
<meta name="dc.copyright" content="OpenLink Software, 1999 - 2009" />
<link rel="top" href="index.html" title="OpenLink Virtuoso Universal Server: Documentation" />
<link rel="search" href="/doc/adv_search.vspx" title="Search OpenLink Virtuoso Universal Server: Documentation" />
<link rel="parent" href="server.html" title="Chapter Contents" />
<link rel="prev" href=".html" title="" />
<link rel="next" href="htmlconductorbar.html" title="HTML based Administration Console (Conductor) Guide" />
<link rel="shortcut icon" href="../images/misc/favicon.ico" type="image/x-icon" />
<link rel="stylesheet" type="text/css" href="doc.css" />
<link rel="stylesheet" type="text/css" href="/doc/translation.css" />
<title>6. Administration</title>
<meta http-equiv="Content-Type" content="text/xhtml; charset=UTF-8" />
<meta name="author" content="OpenLink Software Documentation Team ; " />
<meta name="copyright" content="OpenLink Software, 1999 - 2009" />
<meta name="keywords" content="" />
<meta name="GENERATOR" content="OpenLink XSLT Team" />
</head>
<body>
<div id="header">
<a name="databaseadmsrv" />
<img src="../images/misc/logo.jpg" alt="" />
<h1>6. Administration</h1>
</div>
<div id="navbartop">
<div>
<a class="link" href="server.html">Chapter Contents</a> | <a class="link" href="server.html" title="Administration">Prev</a> | <a class="link" href="htmlconductorbar.html" title="HTML based Administration Console (Conductor) Guide">Next</a>
</div>
</div>
<div id="currenttoc">
<form method="post" action="/doc/adv_search.vspx">
<div class="search">Keyword Search: <br />
<input type="text" name="q" /> <input type="submit" name="go" value="Go" />
</div>
</form>
<div>
<a href="http://www.openlinksw.com/">www.openlinksw.com</a>
</div>
<div>
<a href="http://docs.openlinksw.com/">docs.openlinksw.com</a>
</div>
<br />
<div>
<a href="index.html">Book Home</a>
</div>
<br />
<div>
<a href="contents.html">Contents</a>
</div>
<div>
<a href="preface.html">Preface</a>
</div>
<br />
<div class="selected">
<a href="server.html">Administration</a>
</div>
<br />
<div class="selected">
<a href="databaseadmsrv.html">Database Server Administration</a>
<div>
<a href="#dbadm" title="Database">Database</a>
<a href="#vdbconcepts" title="Virtual Database">Virtual Database</a>
<a href="#usermodel" title="Virtuoso User Model">Virtuoso User Model</a>
<a href="#vaddistr" title="VAD - Virtuoso Application Distribution">VAD - Virtuoso Application Distribution</a>
<a href="#backup" title="Data Backup & Recovery">Data Backup & Recovery</a>
<a href="#perfdiag" title="Performance diagnostics">Performance diagnostics</a>
<a href="#ptune" title="Performance Tuning">Performance Tuning</a>
</div>
</div>
<div>
<a href="htmlconductorbar.html">HTML based Administration Console (Conductor) Guide</a>
</div>
<div>
<a href="fault.html">Virtuoso Cluster Fault Tolerance</a>
</div>
<br />
</div>
<div id="text">
<a name="databaseadmsrv" />
<h2>6.1. Database Server Administration</h2>
<p>
This section explains the use of the server executables, configuration files
and general administration procedures. These include backup, server tuning and profiling,
installing extension packages etc. See the Access Interfaces chapter for a discussion of the
command line SQL interface.
</p>
<a name="dbadm" />
<h3>6.1.1. Database</h3>
<p>See details for <a href="installwin32.html#srvadminstallreqt">Installation Requirements</a>,
<a href="installwin32.html#srvadmopreq">Operational Requirements</a>,
<a href="installwin32.html#srvadmossupport">Operating System Support</a> and
<a href="installwin32.html#limitsandparameters">Limits</a>.</p>
<a name="srvadmsrvinst" />
<h4>6.1.1.1. Server Instance Creation</h4>
<p>Multiple Virtuoso server instances may coexist on a single machine. Each
database will need to be assigned a unique TCP port number.
On the windows platforms (except 95,98,ME) the Virtuoso can
be configured for multiple services. For further details, see the
<a href="installwin32.html#CreatingDeletingServices">Creating and Deleting
Virtuoso Services</a> in the <a href="installation.html">
Installation chapter.</a>
</p>
<p>To run Virtuoso on a machine, only the Virtuoso server executable and a valid virtuoso.ini file are needed. An empty database file will be created automatically at first startup. None of the other files in the installation are needed for basic operation. Client interfaces may additionally need ODBC drivers or the like to be set up in system configuration files or the Windows registry but the server itself does not depend on these.
</p>
<br />
<a name="srvextinst" />
<h4>6.1.1.2. Installing Application Packages</h4>
<p>
Virtuoso comes with optionally installable SQL application packages (VAD packages) for web based admin, on-line documentation, programming tutorials and a BPEL processor.
The installer typically asks whether to install these into the demo or the default empty database.
Depending on the OS and form of installer, also depending on whether the installation is the commercial or open source one, the package files will be in different locations. To locate them, look for *.vad.
The packages are typically made in two variants, where one keeps the installed items in the DAV repository and the other in the file system. They are functionally equivalent in most cases, except for some tutorials that will only work in the file system. Keeping installed resources in DAV has advantages when moving the database or backing up, since the installed items will not have to be treated separately.
To install these on a database, login as dba using the isql utility and issue the SQL command:
</p>
<div>
<pre class="programlisting">
# at the OS command line prompt, assuming the dba password is dba and the server is at the default port 1111:
$ isql 1111 dba dba
-- At the SQL prompt:
SQL> vad_install ('conductor_dav.vad', 0);
-- The path is relative to the server's working directory.
</pre>
</div>
<p>
Since the packages are read from and sometimes the extracted contents are written to the file system, the server must have access to the directories in question. Checjk OS permissions and set the DirsAllowed parameter in the ini file to allow the needed access.
</p>
<p>
After the packages are installed, they can be used by pointing the web
browser to their start page. The start pages are listed on the root
page of the demo database and at the Getting Started section of the
Open Virtuoso web site. The start page for the Conductor web admin
interface is /conductor, the docs are at /doc/html, the tutorials at
/tutorial, the BPEL admin and demos at /BPELGUI. All these are directly
under the default Virtuoso web listener, the port is shown in the
messages log and read from the HTTP Server section of the virtuoso.ini
file.
</p>
<br />
<a name="srvadmlicensing" />
<h4>6.1.1.3. Server Licensing</h4>
<p>The Virtuoso server requires a valid license file before it will
successfully start. The license file is a file always called
<strong>virtuoso.lic</strong> and must reside in the working directory
of the database instance - where the <database-name>.db file resides
for the instance.</p>
<p>Evaluations versions of Virtuoso come without a license file, however
the registration procedure takes your email address, which will be used to
email a new license file for you evaluation.</p>
<br />
<a name="srvadmlogging" />
<h4>6.1.1.4. Server Logging - Detecting Errors</h4>
<p>Virtuoso provides extensive log information for resolving problems.
Should any problems arise Virtuoso logs should always be consulted.</p>
<p>The Virtuoso server by default will send log information to two
places, the appropriate system log for the operating system and the
virtuoso.log file. On Unix based operating systems, including Linux,
this information will appear in the system log files. On Windows the Virtuoso
log information will appear in the Application Event Log.</p>
<p>Virtuoso logs information in the system logs before the Virtuoso.log
file is open. This advantageously will log errors that cannot be placed in the
virtuoso.log file, such as when the virtuoso.ini file cannot be located during
Virtuoso startup.</p>
<p>The system log feature can be disabled using the "Syslog" parameter
in the [Database] section. This is described in more detail in the following
section.</p>
<br />
<a name="configsrvstupfiles" />
<h4>6.1.1.5. Configuring Server Startup Files</h4>
<a name="VIRTINI" />
<h5>6.1.1.5.1. Virtuoso Configuration File</h5>
<p>The virtuoso.ini file is read from the directory that is current at
server startup. This file contains an entry for all user settable options in the server.
It is divided into the following sections:
</p>
<ul>
<li>
<p>[Database] Location of database files</p>
</li>
<li>
<p>[Parameters] Server tuning, options</p>
</li>
<li>
<p>[HTTPServer] Settings in this section control the web server component of the Virtuoso Server.</p>
</li>
<li>
<p>[URIQA] Settings in this optional section control URIQA semantic web enabler.</p>
</li>
<li>
<p>[SPARQL] Settings in this optional section control SPARQL protocol endpoint default parameters and limits.</p>
</li>
<li>
<p>[I18N] encoding settings</p>
</li>
<li>
<p>[Replication] The replication section sets the transactional replication parameters for the server. </p>
</li>
<li>
<p>[Mono] The Mono section contains settings for Virtuoso with Mono Runtime CLR support. </p>
</li>
<li>
<p>[Client] Client default settings</p>
</li>
<li>
<p>[AutoRepair] Corrupted database recovery</p>
</li>
<li>
<p>[Striping] Multi-file / multi-disk databases</p>
</li>
<li>
<p>[VDB] Virtual database functionality related options</p>
</li>
<li>
<p>[Ucms] location of UCM files describing multi-byte encodings such as Far East languages</p>
</li>
<li>
<p>[Zero Config] ZeroConfig related options</p>
</li>
<li>
<p>[Plugins] VSEI plugin modules</p>
</li>
</ul>
<p>Below are the descriptions for each parameter</p>
<a name="ini_Database" />
<h6>[Database]</h6>
<ul>
<li>
<div class="formalpara">
<strong>DatabaseFile=virtuoso.db</strong>
<p>For a single file database, this is the relative path of the file
in the format appropriate to the platform. The path is resolved relative
to the directory that is current at server startup. All other paths are interpreted similarly.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TransactionFile=virtuoso.trx</strong>
<p>This is the transaction log file. If this parameter is omitted, which should never
be the case in practice, the database will run without log, meaning that it cannot
recover transactions committed since last checkpoint if it should abnormally terminate.
There is always a single log file for one server. The file grows as transactions get committed until a checkpoint is reached at which time the
transactions are applied to the database file and the trx file is reclaimed, unless CheckpointAuditTrail is enabled.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ErrorLogFile=virtuoso.log</strong>
<p>This file logs database error messages, e.g. 'out of disk'. By viewing this the
dba can trace problems and see at which times the server has started, checkpoints have been made, etc.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ErrorLogLevel=7</strong>
<p>This controls what events get logged into the database error log. This should always be 7.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LockFile=virtuoso.lck</strong>
<p>This optional parameter can be used to manually specify the location
of the Virtuoso lock (.lck) file. This can be relative or the full path to the
lock file. Virtuoso, by default, creates a file with the
same name as the DatabaseFile but with the extension of .lck. This file
exists when the Virtuoso server is running to prevent it starting multiple
times using the same parameters, and should be automatically removed by
the server upon exit. However, not all file systems support file locking,
such as NFS, therefore this parameter can be set to keep the lock file
on a more appropriate file system.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FileExtend=100</strong>
<p>This is the size that the database file automatically grows (in 8k pages) when the current file is not large enough. Default = minimum = 100.
The parameter has no effect if striping is set.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Striping=0</strong>
<p>A non-zero value will enable the settings in [Striping] to take effect. If this
is the case the DatabaseFile parameter is ignored and the files in [striping] are used.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LogSegments</strong>
<p>If this is non-zero log segmentation is enabled. This
is only used for crash dumps where several files may be needed to
accommodate the recovery logs. If non-zero, this will be followed by
entries of the form Log1=...</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Log1=/tmp/log1.trx 100M</strong>
<p>The number in Log<n> is the ordinal number of the log, starting at 1. The entry
consists of the file name and allocated size with an optional size modifier. Available modifiers are B for blocks of 4k (default if unspecified,
K for kilobytes, M for megabytes, and G for gigabytes. The log is filled up to the first
transaction that exceeds the size. The log therefore will be longer than the allocated size. If blobs are involved, this amount can be quite substantial.
It is therefore advisable to have some extra space available on the storage device used for recovery logs.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>crashdump_start_dp, crashdump_end_dp</strong>
<p>
These options make it possible to produce a crash dump of a specified range of disk pages.
In case the server runs out of disk space during a dump, the error message indicates the page at
which the next dump should be started after the user has made more space available.
In this case it is important to rename the already produced crash dump or change the Logx= -entries in virtuoso.ini to avoid overwriting said files. If none of these options are provided, the server will attempt a crash dump of the entire database storage.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TempStorage = <TempDatabase_Name></strong>
<p>The name of a section in the INI file containing temporary
database details. If this parameter is omitted or the section does not
exist the default values for temporary storage are used.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Syslog = 1/0 (default 0)</strong>
<p>Virtuoso can writes log worthy messages to the system log (Unix based
operating systems including Linux) or the Windows Event Log (Windows
operating systems). Messages are written in the system/event log
before the virtuoso.log file is opened, therefore errors due to
absence of virtuoso.ini log are loggable there. This system/event
logging can be enabled using this option, by default it is set to 0
meaning off.</p>
</div>
<p>On Unix/Linux messages are written as "Virtuoso" events.</p>
<p>On Windows messages are written in the Application event log.</p>
</li>
</ul>
<br />
<a name="ini_TempDatabase" />
<h6>[<TempDatabase_Name>]</h6>
<p>This section name must match the TempStorage parameter in the
Database section of the Virtuoso INI file to be of any use. Otherwise this
section will be ignored.</p>
<ul>
<li>
<div class="formalpara">
<strong>DatabaseFile = <database file name>.tdb </strong>
<p>Name of temporary database file.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TransactionFile = <transaction file name>.ttr</strong>
<p>Name of temporary transaction file.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FileExtend = NNN</strong>
<p>Increment amount by which the database file will dynamically grow.
This setting is identical in use and purpose to parameter in the
Database section with the same name.</p>
</div>
</li>
</ul>
<br />
<a name="ini_Parameters" />
<h6>[Parameters]</h6>
<ul>
<li>
<div class="formalpara">
<strong>SingleCPU=0</strong>
<p>This is a Win32 specific option that
forces Virtuoso to only run on one CPU in a multiprocessor environment.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerPort=[<IP Address>]:<port></strong>
<p>This is the IP Address and port number where the server will start listening.
You do not need to specify the listening IP Address but can do in a situation that you want the server to bind to a specific address only.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerThreads=10</strong>
<p>This is the maximum number of threads used in the server. This should be
close to the number of concurrent connections if heavy usage is expected.
Different systems have different limitations on threads per process but a value of
100 should work in most places.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerThreadSize=50000</strong>
<p>Stack size of thread used for reading client messages and accepting connections.(default : 50 000 bytes)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MainThreadSize=100000</strong>
<p>Stack size of the main thread (default : 100 000 bytes)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ThreadCleanupInterval</strong>
<p>The interval in minutes (default : 0) after which
threads in the thread pool should be released.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ThreadThreshold</strong>
<p>The maximum number of threads (default : 10) to leave in
the thread queue after thread clean-up interval has expired.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SchedulerInterval</strong>
<p>Defines the scheduler wake-up interval ( in minutes).
By default is 0 i.e. the scheduler is disabled. </p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ResourcesCleanupInterval</strong>
<p>The interval in minutes (default : 0) after which allocated resources will be flushed..</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FutureThreadSize=100000</strong>
<p>Stack size of worker threads. This is the stack size for serving any client SQL statements or HTTP requests. This can be increased if the application uses recursive stored procedures or links in external code needing a large stack. (default 100 000 bytes)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TempAllocationPct</strong>
<p>A Percentage that may be greater than 100%. This gives a
percentage of the main .db file to which the temp db file may grow
before starting to reclaim the oldest persistent hash index.
Basically if a particular hash index is reusable (i.e. it references
only table columns and the values in these columns have not changed)
the engine keeps the hash index defined into the temp db for reuse.
This parameter allows some control over the temp db file size.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>O_DIRECT</strong>
<p>If this is non-zero, the database file(s) will be opened with the
O_DIRECT option on platforms where this is supported. This has the effect of doing file system I/O
from application buffers directly, bypassing caching by the operating system. This may be useful
if a large fraction of RAM is configured as database buffers. If this is on, the file system cache
will not grow at the expense of the database process, for example it is less likely to swap out
memory that Virtuoso uses for its own database buffers. Mileage will vary according to operating
system and version. For large databases where most of system memory is used for database buffers it is advisable to try this.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>CheckpointInterval=60</strong>
<p>This is the interval in minutes at which Virtuoso will
automatically make a database checkpoint. The automatic checkpoint will not be made if
there is less than MinAutoCheckpointSize bytes in the current transaction log.
A checkpoint interval of 0 means that no periodic automatic checkpoints are made.</p>
<p>
Setting the value to -1 disables also the checkpoints made after a roll forward upon database startup. This setting should be used when backing up the database file(s). This guarantees that even if the server dies and restarts during the copy, no checkpoints that would change these files will take place and thus the backup is clean.
</p>
<p>the checkpoint_interval SQL function may be used to change the checkpoint interval value while the database is running.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>CheckpointSyncMode=2</strong>
<p>This controls how the file system is synchronized after a checkpoint. Once the checkpoint has issued all
write system calls it needs it can do one of the following depending on this setting:
</p>
<p>0. - Continue, leave the OS to flush buffers when it will.</p>
<p>1. - Initiate the flush but do not wait for it to complete.</p>
<p>2. - Block until the OS has flushed all writes pertaining to any of the database files.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PageMapCheck=0</strong>
<p>This controls the check of page maps:
</p>
<p>0. - disables the check of page maps.</p>
<p>1. - enables the check of page maps.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>NumberOfBuffers=2000</strong>
<p>This controls the amount of RAM used by Virtuoso to cache database files. This has a critical
performance impact and thus the value should be fairly high for large databases. Exceeding
physical memory in this setting will have a significant negative impact. For a database-only
server about 65% of available RAM could be configured for database buffers.
</p>
<p>Each buffer caches one 8K page of data and occupies approximately 8700 bytes of memory.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxCheckpointRemap=2000</strong>
<p>Specifies how many pages Virtuoso is allowed to remap.
Remapping means that pages can consume the space of two actual pages
until checkpoint. See the
<a href="CHECKPOINT.html#checkpointparams">Checkpoint & Page Remapping</a>
section in the SQL Reference chapter for more information.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PrefixResultNames=0</strong>
<p>This setting should always be 0.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>CaseMode=1</strong>
<p>This controls the case sensitivity of the Virtuoso SQL interpreter. The following values
are supported:</p>
<ul>
<li>
<p>0 - SQL is case sensitive and identifiers are stored in the case they are entered in. This is similar to the Progress or Informix default.</p>
</li>
<li>
<p>1 - SQL is case sensitive and Unquoted identifiers are converted to upper case when read.
To use non-upper case identifiers, the identifiers must be quoted with double quotes (").
This is similar to Oracle.
</p>
</li>
<li>
<p>2 - SQL is case-insensitive and identifiers are stored in the case where first seen. Unlike the
situation in other modes, two identifiers differing only in case cannot exist.
This is similar to the MS SQL Server 7 default behavior.
</p>
</li>
</ul>
<div class="note">
<div class="notetitle">Note:</div>
<p>Once a database is created in one case mode the case mode should not be
changed as that may change the interpretation of stored procedures etc.</p>
</div>
</div>
</li>
<li>
<div class="formalpara">
<strong>MinAutoCheckpointSize=4000000</strong>
<p>See CheckpointInterval.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>AutoCheckpointLogSize</strong>
<p>This is the size of transaction log in bytes after which an automatic checkpoint is initiated. If this is non-zero, whenever the transaction log size
exceeds this size an automatic checkpoint is started. This will result in approximately
like sized logs to be generated. This is useful with the CheckpointAuditTrail option
for generating a trail of equal sized consecutive logs.</p>
</div>
</li>
<li>
<a name="fp_checkpointAuditTrail" />
<div class="formalpara">
<strong>CheckpointAuditTrail=1</strong>
<p>If this is non-zero each checkpoint will start a new log and leave the
old transaction log untouched. A 0 value indicates that the transaction log may
be reused once the transactions in it have been written in a checkpoint.
</p>
<p>
If it is important to keep an audit trail of all transactions in the
database's life time then this option should be true. A new log file will
be generated in the old log file's directory with a name ending with the
date and time of the new log file's creation. Old log files can be manually
copied into backup storage or deleted. Only one log file is active at one time.
The newest log file may at any time be written to by the server, but that is
the only log file Virtuoso has open at each time. Thus reading any logs is safe.
Writing or deleting the active log file will result in loss of data, and
possibly referential integrity, in the database. See the
<a href="databaseadmsrv.html#backup">Back and Recovery</a> chapter for more information on
this and related parameters.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>AllowOSCalls=0</strong>
<p>If non-zero the system SQL function is enabled. This will allow a dba group user
to run shell commands through SQL. This poses a potential security risk and hence the setting should normally be 0.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxStaticCursorRows=5000</strong>
<p>This is the maximum number of rows returned by a static cursor. Default = 5000</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FreeTextBatchSize=10000000</strong>
<p>This is the amount of text data processed in one batch of the free-text
index when doing a batch update or non-incrementally reindexing the data.
Default : 10,000,000</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>NullUnspecifiedParams</strong>
<p>When set to 1, if an
application prepares a statement with insufficient number of input
parameters, the unspecified ones are assumed to be NULL.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Collation</strong>
<p>Defines a sorting order according to SYS_COLLATIONS. The
name supplied to this parameter must be in
<a href="fn_charsets_list.html">charsets_list(1)</a>.</p>
</div>
</li>
<li>
<a name="fp_acliniallowed" />
<div class="formalpara">
<strong>DirsAllowed=<path> [, <path>]</strong>
<p><path> := <absolute_path> or <relative_path>
comma-delimited list of OS directories allowed for file
operations such as <span class="computeroutput">file_to_string()</span>. The server
base directory (the directory containing this INI file) must appear on
this list in order to enable File DSNs to work. On Windows use in the path "\".
</p>
<p>The Virtuoso ISQL utility can be used to check the Server DirsAllowed params
as follows:</p>
<div>
<pre class="programlisting">
SQL> select server_root (), virtuoso_ini_path ();
</pre>
</div>
<p>The above should show in the result the server working directory and INI
file name.</p>
<p>Also you can check the relevant INI setting by running following
statement via ISQL command line utility:</p>
<div>
<pre class="programlisting">
SQL> select cfg_item_value (virtuoso_ini_path (), 'Parameters',
'DirsAllowed');
</pre>
</div>
</div>
</li>
<li>
<a name="fp_aclinidenied" />
<div class="formalpara">
<strong>DirsDenied=<path> [, <path>]</strong>
<p><path> := <absolute_path> or <relative_path>
OS directories denied for file operations. See <a href="databaseadmsrv.html#acl">Virtuoso
ACL's</a> for information on functions that are restricted.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLServerPort</strong>
<p>Specifies the port on which the server listens for incoming SSL CLI requests.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLCertificate</strong>
<p>The SSL certificate to use (same meaning as the SSLCertificate in HTTPServer section)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLPrivateKey</strong>
<p>The server's private key (same meaning as the SSLCertificate in HTTPServer section)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxOptimizeLayouts = 1000</strong>
<p>This parameter governs the maximum number of partial or full
join orders that the Virtuoso SQL Optimized compiler will calculate per
select statement. When MaxOptimizeLayouts has been reached,
the best execution plan reached thus far will be used. The default
value is 1000, specifying 0 will try all possible orders and guarantee that the best plan is reached.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>StopCompilerWhenXOverRunTime = 0</strong>
<p>
If non-zero, this specifies that the SQL compiler should stop considering alternative execution plans after the elapsed compilation time exceeds the best run time estimate times the parameter.
For example, if this is 2, then compilation stops after using twice the time of the best plan reached thus far.
Enabling this option increases performance when processing short running queries that are each executed once. Using this with long running queries or prepared parametrized queries is not useful and may lead to non-optimal plans being selected.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TraceOn = option1 [, option2 [, ..]]</strong>
<p>This parameter accepts a comma-delimited list of tracing options to
activate by default. Enabled trace options will list there respective
errors in the virtuoso.log file when encountered. Valid options are:</p>
<ul>
<li>user_log</li>
<li>failed_log</li>
<li>compile</li>
<li>ddl_log</li>
<li>client_sql</li>
<li>errors</li>
<li>dsn</li>
<li>sql_send</li>
<li>transact</li>
<li>remote_transact</li>
<li>exec</li>
<li>soap</li>
</ul>
<p>If an invalid option is set then this error will be listed in the
virtuoso.log file upon server startup. Virtuoso will continue to log
selected options unless the trace_off() function is called for that item.</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>The functions: <a href="fn_trace_on.html">trace_on()</a>
and <a href="fn_trace_off.html">trace_off()</a>
which use the same options and can turn logging options on/off while
the server is running.</p>
</div>
<a name="ex_traceoniniopt" />
<div class="example">
<div class="exampletitle">Using the TraceOn ini file option</div>
<div>
<pre class="programlisting">
[Parameters]
....
TraceOn = soap, errors
...
</pre>
</div>
<p>This will enable logging of additional information regarding
SOAP calls and SQL run-time errors into the virtuoso.log file.</p>
</div>
<a name="ex_threadcleanupinterval" />
<div class="example">
<div class="exampletitle">Using the ThreadCleanupInterval and ResourcesCleanupInterval ini file option</div>
<div>
<pre class="programlisting">
[Parameters]
....
ThreadCleanupInterval = 1
ResourcesCleanupInterval = 1
...
</pre>
</div>
<p>Set both to 1 in order to reduce memory leaking.</p>
</div>
</div>
</li>
<li>
<div class="formalpara">
<strong>AllowPasswordEncryption = 1/0</strong>
<p>Determines whether Virtuoso encryption should be accepted from
client connections. The default value is 1 - on. If set to 0 then
only clear text and digest authentication will be accepted.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>JavaClasspath</strong>
<p>This parameter is applied to the environment prior to the server's
startup. It is valid only for binaries hosted in the Virtuoso Java VM.
This has the same format as the Java CLASSPATH environment variable for
the platform being used.</p>
</div>
<p>Virtuoso searches for classes in the following order:</p>
<ul>
<li>The java_vm_create VSE parameter list</li>
<li>This "JavaClasspath" in [Parameters] INI section</li>
<li>The CLASSPATH environment variable</li>
<li>If none of the above the CLASSPATH is the current directory.</li>
</ul>
</li>
<li>
<div class="formalpara">
<strong>JavaVMOption1..N = <opts></strong>
<p>These can be used for setting Java options for the Java runtime
hosted in Virtuoso. These options work as if provided as command
line options to the JRE's Java command line.</p>
</div>
<p>More than one line of options can be specified by using
consecutively numbered options:</p>
<div>
<pre class="programlisting">
JavaVMOption1 = -Ddt1=val1
JavaVMOption2 = -Ddt2=val2
...
JavaVMOption5 = -Ddt5=val5
</pre>
</div>
</li>
<li>
<div class="formalpara">
<strong>PLDebug = 0</strong>
<p>The PLDebug switch controls the type of debugging enabled:</p>
</div>
<ul>
<li>
<strong>PLDebug = 0</strong> - default debugging information and
test coverage disabled.</li>
<li>
<strong>PLDebug = 1</strong> - debugging information enabled.</li>
<li>
<strong>PLDebug = 2</strong> - debugging information enabled, test
coverage data will be written to file specified in TestCoverage Virtuoso ini
file parameter.</li>
</ul>
</li>
<li>
<div class="formalpara">
<strong>TestCoverage = cov.xml</strong>
<p />
</div>
</li>
<li>
<div class="formalpara">
<strong>CallstackOnException = 0</strong>
<p>Controls whether Virtuoso will report call stack on errors.</p>
</div>
<p>This parameters takes the following values:</p>
<ul>
<li>0 (default) - Call stack reporting disabled.</li>
<li>1 - Call stack reporting enabled.</li>
</ul>
</li>
<li>
<div class="formalpara">
<strong>CompileProceduresOnStartup = 1</strong>
<p>This controls whether Virtuoso will recompile all stored
procedures listed in SYS_PROCEDURES and internal PL procedures
defined during server startup. By default Virtuoso will recompile all
procedures, with this setting set to 0 Virtuoso will defer compilation
until procedures' first call. The benefits of this are that Virtuoso
may start up up to 2-3 times faster, also the initial memory consumption
will be significantly reduced as it does not need to analyze all
the long varchar data allocating memory for execution.
This setting is a boolean, either 1 or 0. This setting does not
apply to attached VDB procedures or modules.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FDsPerFile = 1</strong>
<p>Controls the number of file descriptors per file to be
obtained from the OS. The default and minimum value is 1.
This parameter only effects databases that use striping.
Having multiple FDs per file means that as many concurrent I/O operations
may simultaneously be pending per file. This allows more flexibility for
the OS to schedule the operations, potentially improving file I/O
throughput</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>RecursiveFreeTextUsage = 1/0 default 1</strong>
<p>This option controls the behavior of free-text triggers in
super-tables. If this option is set to 1 then Virtuoso will scan the hierarchy
of tables until a free-text index is used to use when compiling
SQL statements involving contains, xcontains or xpath_contains.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>RecursiveTriggerCalls = 1/0 default 1</strong>
<p>This option controls the behavior of super-table triggers. When this
option is set to 1 then triggers in the super-table will be called before
its own (sub-table) triggers. This behavior is recursive and will
continue up the branch of recursion, hence the triggers in the top
most table in the chain will be called first.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxSortedTopRows = 10000</strong>
<p>The TOP select statement clause caches in memory the rows
pertinent to the result. The number of rows allowed to be cached
within memory is limited by this parameter.</p>
</div>
<p>
<strong>Simple example using OFFSET and LIMIT:</strong>
</p>
<p>Virtuoso uses a zero index in the OFFSET. Thus in the example below, will be taken position at
record 9000 in the result set, and will get the next 1000 rows starting from 9001 record. Note that the
MaxSortedTopRows in parameters Virtuoso ini section needs to be increased (default is 10000).</p>
<div>
<pre class="programlisting">
select ?name
ORDER BY ?name
OFFSET 9000
LIMIT 1000
</pre>
</div>
</li>
<li>
<div class="formalpara">
<strong>DisableUnixSocket = 0/1 default 0</strong>
<p>This parameter is only applicable to Unix servers. Virtuoso
clients on the <span class="computeroutput">localhost</span> of the server
can benefit from using Unix Domain sockets to improve connection
performance. By default (DisableUnixSocket = 0) Virtuoso will open a
Unix Domain listen socket in addition to the TCP listen socket. The name of
the UD socket is <span class="computeroutput">/tmp/virt-<tcp-listen-port></span>.
When a client attempts to connect to the Virtuoso server using the
specific address <span class="computeroutput">localhost</span> it will
first try connecting to the UD socket, failing that it will silently revert
to the TCP socket. See the
<a href="accintudsockets.html">Unix Domain Socket Connections</a>
section for more details.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TransactionAfterImageLimit = N bytes default 50000000</strong>
<p>When the roll-forward log entry of a transaction exceeds this size, the transaction is too large and is marked as uncommittable. This work as upper limit otherwise infinite (transactions). The default is 50Mb . Then also note that transaction roll-back data takes about 2x of roll-forward data. Hence when the transaction roll-forward data which is 50Mb the total transient consumption is closer to 150 Mb.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TempSesDir</strong>
<p> Directory for storing temporary data for large object handled in replication and HTTP server. Defaults to server home directory.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DbevEnable = 0/1 default 1</strong>
<p>Enable or disable <a href="hooks.html">Database Event Hooks</a>.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>RunAs</strong>
<p>Specifies the OS user name to which the server will switch after opening the listen ports. Has an effect only on the operating systems that support it.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxMemPoolSize</strong>
<p>Specifies the limit of the memory to be used for compiling a SQL statement. If the query compilation requires more memory an error will be signalled. If this is a zero then no limit will be applied. The default is 10Mb i.e. when no parameter is specified, also if negative number or less than 5000000 is given then it would be set to 5000000 bytes.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DefaultIsolation</strong>
<p>This specifies the default transaction isolation. This isolation is used
unless overridden by a client setting it using the SQL_TXN_ISOLATION or
equivalent connection option or a stored procedure locally setting it with
the SET statement. The values are as by the SQL_TXN_ constants in ODBC,
that is, 1 for read uncommitted, 2 for read committed, 4 for repeatable
read and 8 for serializable. If nothing is specified, the default is
repeatable read.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>UseAIO</strong>
<p>This specifies whether to use asynchronous file I/O on supporting Unix
systems. A value of 0 means not using it. A value of 1 means using
lio_listio for any background write or read ahead if available. A value of
2 means to use the regular blocking read and write but to merge adjacent
operations into a single system call when possible.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TempDBSize</strong>
<p>Controls the acceptable size of the temp database file. If on startup it's size (in MB) is
greater than TempDBSize the file gets deleted and reset. This feature can be turned off by
setting TempDBSize to 0. Note that the temp db file serves as an optimization storage only
and doesn't have any client data that are not in either the main database files or the
corresponding transaction log files.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LiteMode = 0/1 (default 0)</strong>
<p>Runs server in lite mode. When Lite mode is on:
<ul>
<li>the web services are not initialized i.e. no web server, dav, soap, pop3 etc.</li>
<li>the replication is stopped</li>
<li>the pl debugging is disabled</li>
<li>plugins are disabled</li>
<li>rendezvous is disabled</li>
<li>the relevant tables to the above are not created</li>
<li>the index tree maps is set to 8 if no other setting is given</li>
<li>memory reserve is not allocated</li>
<li>affects DisableTcpSocket. So DisableTcpSocket setting is treated as 1
when LiteMode=1, regardless of value in INI file</li>
</ul>
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>RdfFreeTextRulesSize = 10 or more </strong>
<p>The size of hash to control rdf free text index
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>IndexTreeMaps = 2 -1024 (power of 2) </strong>
<p>Size of index tree maps, larger is better for speed but consume memory, in
lite is 2 in 'normal' mode is 256 by default.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DisableTcpSocket = 1/0 </strong>
<p>Default = 0. If set to 1, disables database listener on TCP port; unix socket must be
used for data access connections (ODBC, JDBC, ADO.NET, OLE DB). When LiteMode=1,
DisableTcpSocket setting in INI file is ignored and treated as if set to 1.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ExtentReadThreshold</strong>
<p>Controls speculative read of disk pages. If pages are read in close succession
from an extent of 256 consecutive pages, the system may decide to speculatively read
the entire extent.</p>
<p>ExtentReadThreshold parameter gives how many consecutive reads are needed to
trigger this.</p>
<p>When is set to 0, this means that anytime a page is read, the whole extent is
read along with it</p>
<p>When is set to 1, this means that if the first read is at time <strong>t</strong>
and the next one at time <strong>t1</strong> and <strong>t1-t</strong> < ini_ExtentReadWindow msec,
then 2nd read triggers the speculative read.</p>
<p>Default is 2.</p>
<p>Takes effect after the buffer pool is full.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ExtentReadWindow</strong>
<p>Controls speculative read of disk pages. If pages are read in close succession
from an extent of 256 consecutive pages, the system may decide to speculatively read
the entire extent.</p>
<p>ExtentReadWindow parameter gives the time within which the reads must fall.</p>
<p>Default is 1000.</p>
<p>Takes effect after the buffer pool is full.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ExtentReadStartupThreshold</strong>
<p>Controls speculative read of disk pages. If pages are read in close succession
from an extent of 256 consecutive pages, the system may decide to speculatively read
the entire extent.</p>
<p>ExtentReadStartupThreshold parameter value applies while the server is freshly
started and the buffer pool is not yet full. It can be set to preread more aggressively.</p>
<p>Default is 0.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ExtentReadStartupWindow</strong>
<p>Controls speculative read of disk pages. If pages are read in close succession
from an extent of 256 consecutive pages, the system may decide to speculatively read
the entire extent.</p>
<p>ExtentReadStartupWindow parameter value applies while the server is freshly
started and the buffer pool is not yet full. It can be set to preread more aggressively.</p>
<p>Default is 40000.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>AsyncQueueMaxThreads</strong>
<p>Sets the number of threads in a pool that is used for getting extra threads for running
queries and for aq_request. Each running statement has at least one thread that is not allocated
from this pool plus zero or more threads from this pool.</p>
<p>Setting the pool size to the number of cores plus a few is a reasonable default. On
platforms with core multithreading, one can count a core thread as a core for purposes of
this parameter.</p>
<p>If one expects to run many slow <a href="">aq_requests()</a>
(see <a href="">async_queue()</a>,
<a href="">aq_request()</a>, etc.),
then the number of threads should be increased by the number of slow threads one expects.
</p>
<p>Slow threads are typically I/O bound threads used for web crawling or similar long-latency,
low-CPU activity.</p>
<p>Note: Only effective with Virtuoso 7.0 and later.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ThreadsPerQuery</strong>
<p>This is maximum number of threads that can be claimed from the thread pool by a single
query. A value of one means that no query parallelization will take place, and all queries
will run single threaded.</p>
<p>The number of cores on the machine is a reasonable default if running large queries.</p>
<p>Note that since every query is served by at least one thread, a single query taking
all the extra threads will not prevent other queries from progressing.</p>
<p>Note: Only effective with Virtuoso 7.0 and later.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VectorSize</strong>
<p>This the number of simultaneous sets of query variable bindings processed at one time.
The default is 10,000, which is good for most cases.</p>
<p>If we are evaluating the query:</p>
<div>
<pre class="programlisting">
SELECT COUNT (*)
FROM t1 a,
t1 b
WHERE a.row_no + 1 = b.row_no
OPTION (LOOP, ORDER)
</pre>
</div>
<p>with vector size of 10,000, then 10,000 rows of t1 a will be fetched first;
1 will be added to the 10,000 row_no values;
and then the corresponding row of t1 b will be fetched for the 10,000 row_no of t1 a.
This process will repeat until enough batches of t1 a have been fetched to come to its end.</p>
<p>Note: Only effective with Virtuoso 7.0 and later.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>AdjustVectorSize</strong>
<p>Using a larger vector size when evaluating large queries with indexed random-access can
yield up to a 3x speed-up relative to using the default vector size. However, always using a
large vector size will prohibitively increase the overhead of running small queries. For this
reason, there is the option to adaptively select the vector size. Set AdjustVectorSize = 1
to enable this feature. The SQL execution engine will increase the vector size when it sees
an index lookup that does not get good locality, (e.g., after sorting the keys to look for,
too few consecutive lookups fall on the same page). Having more keys to look up increases
the chance that consecutive keys should be found on the same page, thus eliminating much of
the index lookup cost.</p>
<p>Note: Only effective with Virtuoso 7.0 and later.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxVectorSize</strong>
<p>When AdjustVectorSize is on, this setting gives the maximum vector size. The
default is 1,000,000 and the largest allowed value is about 3,500,000.</p>
<p>Note: Only effective with Virtuoso 7.0 and later.</p>
</div>
</li>
</ul>
<p>Note: The default for startup behavior is to always read full extents and the default for the normal
behavior is to trigger preread on the third read inside one second.</p>
<br />
<a name="ini_HTTPServer" />
<h6>[HTTPServer]</h6>
<p>Settings in this section control the web server component of the
Virtuoso Server.</p>
<ul>
<li>
<div class="formalpara">
<strong>ServerPort</strong>
<p>This specifies the initial HTTP listen port for the HTTP server.
Can be specified also as ipaddress:port. Once Virtuoso is started it is possible to create
multiple listeners using virtual directories.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPThreadSize = 120000</strong>
<p>
The stack size of the HTTP thread used for reading/processing HTTP client
requests and accepting connections. The default is 120,000 bytes. This
parameter cannot have value less than the default; if a smaller value is
specified the default will be used.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerThreads</strong>
<p>
This specifies the number of concurrently serviced HTTP requests.
If there are more concurrent requests, accepting the connections
will be deferred until there is a thread ready to serve each. If
an attempt to exceed the number of licensed connections is found.
the latter will be used. When the number of threads is low some of the
tutorials may perform poorly and appear not to work; this is due to the
demonstration licensing. When testing Virtuoso with the demonstration
license it can be quite easy to hit the limits unless remote connections
and HTTP connections are conserved. You may wish to either wait for
previous transactions to finish or restart the server to be sure.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPThreadSize = 120000</strong>
<p>
The stack size of HTTP thread used for reading/processing HTTP client requests
and accepting connections. The default setting, if not supplied, is 120,000 bytes.
The default value is the minimum; lesser values will be rounded up.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerRoot = ../vsp</strong>
<p>
This is the file system path of the root directory of files served by the Virtuoso
web server. The index.html in that directory will be served for the / URI. If relative,
the path is interpreted relative to the server's working directory.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerIdString = Virtuoso</strong>
<p>String passed as Server: header to HTTP client. This string is not
required, in its absence the above default will be assumed. This string
can be set to anything required.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ClientIdString = Mozilla/4.0 (compatible; Virtuoso)</strong>
<p>String passed as User-Agent: header to server by HTTP client.
This string is not required, in its absence the above default will be assumed.
This string can be set to anything required.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Charset = [CHARSET-NAME]</strong>
<p>Allows you to set the default server
character set. If no default is specified then ISO-8859-1 will be used automatically by
the server.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>EnabledGzipContent = 0</strong>
<p>This sets the default behavior of HTTP transmission. If set to 1
The Virtuoso HTTP server will send GZipped content to user agents. Otherwise
content will be sent as is. The function
<a href="fn_http_enable_gz.html">http_enable_gz()</a>
lets you change the server mode on the fly.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxKeepAlives = 10</strong>
<p>
Connections by HTTP 1.1 clients can remain open after the initial response has been sent.
This parameters sets a cap on how many socket descriptors will at most be taken by keep alive connections.
Such connections will be dropped by the server ahead of timeout if this number would be exceeded.
Thus the maximum number of open sockets for the Virtuoso HTTP server is this number plus the number of threads.
A keep alive connection is by definition not associated to any pending processing on any thread.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>KeepAliveTimeout = 10</strong>
<p>
This is a timeout in seconds before Virtuoso closes an idle HTTP 1.1 connection.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPProxyEnabled = 0</strong>
<p>Setting this to 1 activates Virtuoso proxy service capabilities. The
default value of 0 deactivates the proxy service.</p>
</div>
<div class="note">
<div class="notetitle">Note:</div>
<p>Ports on which proxying is enabled should not be presented to the
outside world under any situation.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPProxyServer = proxylocal:3128</strong>
<p>HTTP proxy server name and port</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPProxyExceptions = localhost:8890, 127.0.0.1:8890</strong>
<p>HTTP proxy exceptions name and port.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HTTPLogFile = log.out</strong>
<p>If specified, Virtuoso will produce an HTTP server log file with the date
appended to the name given in the parameter and the files extension as ".out".
The log file is rotated daily.</p>
</div>
<p>It will contain the following information:</p>
<p>
<span class="computeroutput">logDDMMYYYY.out</span> :-</p>
<ul>
<li>Client IP address</li>
<li>Date and time of request/response</li>
<li>Timestamp (milliseconds)</li>
<li>Request/response line</li>
</ul>
<p>An example of which is:</p>
<div>
<pre class="programlisting">
127.0.0.1 - - [12/Sep/2006:12:31:17 +0300] "GET /ods/ HTTP/1.1" 200 19453 "" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko.."
127.0.0.1 - - [12/Sep/2006:12:31:18 +0300] "GET /ods/default.css HTTP/1.1" 200 41389 "http://localhost:8894/ods/" "Mozilla/5.0 (Windows; U; ..."
127.0.0.1 - - [12/Sep/2006:12:31:19 +0300] "GET /ods/common.js HTTP/1.1" 200 7481 "http://localhost:8894/ods/" "Mozilla/5.0 (Windows; U; ..."</pre>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxCachedProxyConnections = 10</strong>
<p>
When Virtuoso is acting as a proxy or HTTP client, as is the case with the http_get or
the SOAP client functions, it caches connections to HTTP 1.1 servers. This is the maximum population of said cache.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ProxyConnectionCacheTimeout = 15</strong>
<p>
This is a timeout in seconds for dropping idle connections to other HTTP servers.
These result from http_get, SOAP client functions and proxying.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DavRoot = DAV</strong>
<p>
This specifies the root path of DAV resources. If DAV specific HTTP methods are used on Virtuoso,
these should only reference resources with paths starting with this. This is the top level DAV collection. Other paths can be declared as managed in DAV using the virtual directory mechanism, but this applies to the default virtual directory.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DAVQuotaEnabled = 1/0</strong>
<p>The Virtuoso administrator can enforce a quota on
all DAV accounts, apart from the "dav" administration user,
that restricts that amount of space a DAV
user can consume in their DAV home directory.
When this parameter is set to one (1) quotas are enabled,
when this parameter is set to zero (0), they are disabled.
The default quota value is five Megabytes (5MB) but can be
uniquely defined for each user. Every user has a quota except
"dav", which is unlimited.
Dav quotas are disabled if this parameter is not specified for
backwards compatibility.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DAVChunkedQuota = 1000000</strong>
<p>Virtuoso send resources to from WebDAV to the
requesting client in chunked encoding to save memory if
the request is in HTTP/1.1 and the files size is greater
then the <span class="computeroutput">DAVChunkedQuota</span>
number of bytes. The default chunked-quota value
is approximately one-megabyte.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLPort=4433</strong>
<p>this is the port on which SSL HTTPS connections will be accepted. Can be specified also as ipaddress:port. If unspecified
then the service will be disabled.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLCertificate=./virtuoso_cert.pem</strong>
<p>the option must point to the file with the server certificate in PEM format.</p>
<p>
If the option begins with 'db:' e.g. 'db:id_server', then certificate and key
will be loaded from DBA key space.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLPrivateKey=./virtuoso_key.pem</strong>
<p>points to the file containing the RSA private key in PEM format. </p>
<p>the certificate/key pair must be valid (eq. certificate is generated on base of key)</p>
<p>
If the option begins with 'db:' e.g. 'db:id_server', then certificate and key
will be loaded from DBA key space. Note: in the case of database stored key,
both SSLCertificate and SSLPrivateKey settings MUST be same.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>X509ClientVerify=0</strong>
<p>Whether the server will require X509 certificates from the browsers.</p>
</div>
<ul>
<li>
<strong>X509ClientVerify = 0</strong> - no certificate verification required</li>
<li>
<strong>X509ClientVerify = 1</strong> - ask for trusted certificates</li>
<li>
<strong>X509ClientVerify = 2</strong> - optionally ask for trusted certificates, if trusted certificate is presented it will be verified</li>
<li>
<strong>X509ClientVerify = 3</strong> - optionally accept any certificate including self-signed certificates</li>
</ul>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="odbcimplementation.html#secureodbcx509foafsll">WebID Protocol ODBC Login</a>
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>X509ClientVerifyDepth=1</strong>
<p>Specifies how deep the client certificate
verification process will traverse the Issuer chain before giving up. The default value of 0
means that only verification of the client certificate itself being in the server's CA list is
performed. Setting this option to -1 will ignore the depth checks</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>X509ClientVerifyCAFile=./calist.pem</strong>
<p>a PEM file of all the X509 certificates of the Certification Authorities (CA) which the server will use for verifying the
client's client certificate. A list of these will be sent to the client as a part of the
SSL handshake so the client will know which certificate to send.</p>
<p>
Note this option is in effect when X509ClientVerify is 1 or 2. If X509ClientVerify is 3 and X509ClientVerifyCAFile is given, server will send certificated during handshake, thus some clients may restrict user to send any certificate.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>POP3Port=1234</strong>
<p>Defines the TCP port number on which the Virtuoso POP3 server will listen.
The POP3 server will be disabled if this value is 0 or undefined.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>NewsServerPort=1235</strong>
<p>Defines the TCP port number on which the Virtuoso NNTP server will listen.
The NNTP server will be disabled if this value is 0 or undefined.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FTPServerPort</strong>
<p>The Virtuoso FTP server can be enabled by supplying this parameter with a
value. This value will then be the listening port of the FTP server. The usual
port for FTP is port 21.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FTPServerMinFreePort = 20000</strong>
<p>The Virtuoso FTP client and server use FTPServerMinFreePort and
FTPServerMaxFreePort parameters as lower and upper bounds for
active and passive connections. This parameters sets the lower bound.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FTPServerMaxFreePort = 30000</strong>
<p>The Virtuoso FTP client and server use FTPServerMinFreePort and
FTPServerMacFreePort parameters as lower and upper bounds for
active and passive connections. This parameters sets the upper bound.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FTPServerLogFile = ftpserver</strong>
<p>If specified Virtuoso will produce an FTP server log file with the date
appended to the name given in the parameter and the files extension as ".log".
The log file is rotated daily.</p>
</div>
<p>It will contain the following information:</p>
<p>
<span class="computeroutput">ftpserverDDMMYYYY.log</span> :-</p>
<ul>
<li>Client Host Name</li>
<li>Authorized User</li>
<li>Time</li>
<li>User Command</li>
<li>Server Response Code</li>
<li>Bytes Transferred</li>
</ul>
<p>An example of which is:</p>
<div>
<pre class="programlisting">
hostname anonymous [22/Oct/2003:15:21:43 +0300] "PASS user@domain.com" 230 0
hostname anonymous [22/Oct/2003:15:23:11 +0300] "LIST" 226 162
hostname dav [22/Oct/2003:15:25:00 +0300] "PASS <hidden>" 230 0
</pre>
</div>
</li>
<li>
<div class="formalpara">
<strong>FTPServerAnonymousLogin = 0</strong>
<p>Allows the FTP server to be accessible via the "anonymous" user login.
The anonymous user is not a real user, it has no SQL or DAV login ability.
The anonymous user can only access collections or resources that are set to public. </p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DefaultMailServer=localhost:25</strong>
<p>Default SMTP server name and port, this is used when the first parameter of the smtp_send function is omitted or NULL.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>TempASPXDir</strong>
<p>Allows you to choose what file system directory to be used for temporary
storage of ASPX files hosted in DAV.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PersistentHostingModules = [1/]0</strong>
<p>When set to "1" prevents Virtuoso from removing the plugin interpreters
from the HTTP threads after each request. The default setting is "0".</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HttpSessionSize = size in bytes, default 10000000</strong>
<p>The size threshold for large objects received by HTTP server. When this limit is exceeded
by incoming or outgoing data, this will be stored in a temp file, see 'TempSesDir' parameter in 'Parameters' section.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaintenancePage = file_name</strong>
<p>The name of a HTML page or other static content to be returned to the user agents when server is running in atomic mode.
Note that this page should be self-contained, i.e. no image or CSS not JavaScript references can be used.
</p>
</div>
</li>
</ul>
<br />
<a name="ini_URIQA" />
<h6>[URIQA]</h6>
<p>
The URIQA section sets parameters of both URIQA HTTP extension and URIQA web service.
Section <a href="uriqa.html">URIQA Semantic Web Enabler</a> contains detailed description of this functionality,
including more details about <a href="uriqa.html#uriqainifile">URIQA configuration parameters</a>.
This section should stay commented out as long as URIQA is not in use.
</p>
<ul>
<li>
<div class="formalpara">
<strong>DefaultHost = canonical name of the server, used by default for metadata retrieval, no default value</strong>
<p>The server name, including domain and port if needed, such as "www.example.com" or "www.example.com:8088".</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LocalHostNames = comma-delimited list of names of the server, that can be used for retrieval of metadata, no default value</strong>
<p>List of various allowed spellings of the server name, such as "www.example.com, mail.example.com, mail, localhost, localhost.localdomain".</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LocalHostMasks = comma-delimited list of name masks of the server, no default value</strong>
<p>List of various allowed spellings of the server name in form of pattern strings for SQL "like" operator, not in form of exact strings.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Fingerprint = unique fingerprint string of the server or group of identical servers, no default value</strong>
<p>Do not use this without an explicit need, to let server configure itself. Please refer to <a href="uriqa.html#uriqainifile">detailed description</a> before use.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DynamicLocal = 1/0 default 0, allow dynamic hostname translation in the IRIs</strong>
<p>If DynamicLocal is on and the host part of the IRI matches the Host header of the HTTP request in context or the DefaultHost if outside of HTTP context, then this is replaced with local: before looking up the IRI ID.</p>
</div>
</li>
</ul>
<br />
<a name="ini_SPARQL" />
<h6>[SPARQL]</h6>
<p>
The SPARQL section sets parameters and limits for SPARQL query protocol web service service.
This section should stay commented out as long as SPARQL is not in use.
</p>
Section <a href="rdfandsparql.html">RDF Data Access and Data Management</a> contains detailed description of this functionality.
<ul>
<li>
<div class="formalpara">
<strong>ExternalQuerySource = 1 or 0</strong>
<p>This controls processing of the "query-uri" parameter of the SPARQL query protocol webservice, means enable 1 or prohibited 0.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MinExpiration = 86400</strong>
<p>Sponger caching parameter in seconds. It will cause sponger to use this value as minimal
expiration of the pages, which would help in cases where source document's server do not
report expiration or it reports no caching at all.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxCacheExpiration = 1</strong>
<p>Cache Expiration time in seconds that overrides Sponger's default cache invalidation.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ExternalXsltSource = 1 or 0</strong>
<p>This controls processing of the "xslt-uri" parameter of the SPARQL query protocol webservice, means enable 1 or prohibited 0.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ResultSetMaxRows = number</strong>
<p>This setting is used to limit the number of the rows in the result, if this is higher than "maxrows" parameter this setting takes precedence.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DefaultGraph = IRI</strong>
<p>IRI of the default graph to be used if no "default-graph-uri" parameter is specified.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxQueryCostEstimationTime = seconds</strong>
<p>This setting is used to limit the estimate time cost of the query to certain number of seconds, the default is no limit.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxQueryExecutionTime = seconds</strong>
<p>
This setting is used to set the transaction execution timeout to certain limit in number of seconds, the default is no limit.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ImmutableGraphs = URI</strong>
<p>
IRI of graphs over which the sponger not to be able able to write.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PingService = URI</strong>
<p>
IRI of notification service to which the sponger results will be send.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DefaultQuery = SPARQL Query</strong>
<p>
Default SPARQL Query.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DeferInferenceRulesInit = 1 </strong>
<p>
Defer Loading of inference rules at start up.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MaxMemInUse = 0</strong>
<p>
Maximum amount of memory that is allowed for temporary
storing parts of a SPARQL query result. Default is zero for no limit.
ResultSetMaxRows is maximum length of a SPARQL query result. Default is
zero for no limit.
These two options may be useful for protecting public web service
endpoints against DOS attacks, but at the same time it may cause
returning incomplete results without reporting errors. When used, it is
strongly advised to set the value orders of magnitude larger than the
expected size of longest reply. As a rule of thumb, timeout should
happen before this limit has reached. Values less than 1000000 bytes are
impractical in all cases.
</p>
</div>
</li>
</ul>
<p>The SPARQL INI can be get as RDF via http://cname/sparql?ini service.</p>
<br />
<a name="ini_I18N" />
<h6>[I18N]</h6>
<ul>
<li>
<div class="formalpara">
<strong>WideFileNames = 1/2/3/0</strong>
<p>0: default value. It means not to normalize anything, so for ex.
"Jos"" and "Jose" are two distinct words.</p>
<p>1: Any pair of base char and combinig char (NSM, non-spacing
modifier) is replaced with a single combined char, so if character """ is written as a sequence
of "base" character "e" and a unicode char U +301 ("combining acute accent") then the pair will
be replaced with single U+00E9 ("latin small letter e acute").
</p>
<p>2: Any combined char is converted to its (smallest known) base. So
""" will lose its accent and become plain old ASCII "e".
</p>
<p>3: This is equl to 1|2 and when set then performs both conversions.
As a result, pair of base char and combinig char loses its second char and chars with accents will lose
accents.</p>
<p>If the parameter is required at all, the needed value is probably 3.
So the fragment of virtuoso.ini should be:</p>
<div>
<pre class="programlisting">
[I18N]
XAnyNormalization=3
</pre>
</div>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="virtuosotipsandtricks.html#virtuosotipsandtrickscontrolunicode3">How Can I Control the normalization of UNICODE3 accented chars in free-text index?</a>
</p>
</div>
</div>
</li>
<li>
<div class="formalpara">
<strong>WideFileNames = 1/0</strong>
<p>Default is 0. When 1 then file access and directory
listing functions may use wide strings as file names. If a file name
contains non-ASCII characters then directory listing result set will
contain a wide string instead of narrow string with question marks.
Note, however, that most of existing applications do not support wide
strings for that purposes, so the feature should be used with extreme
care.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VolumeEncoding</strong>
<p>Encoding identifier.
If set, file names are translated from wide strings and default server
encoding to the specified encoding before using them in file
manipulation BIFs. This is useful when server's filesystem is in some
national encoding or in UTF-8.
The translation is bi-directional: directory listing items are
translated from volume encoding to the default server encoding or wide.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VolumeEmergencyEncoding</strong>
<p>Encoding identifier.
If set, this encoding is used when the use of VolumeEncoding causes
encoding errors. The most popular case is when an UNIX filesystem tree
with UTF-8 contains some mounts of legacy storages with KOI or CP
encoding. Less popular is UTF-8 volume encoding and UTF-8QR as emergency
encoding to recover poorly encoded filenames.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VolumeEmergencyEncodingDirs</strong>
<p>List of directories in same syntax as
DirsAllowed, default is empty.
If set, all file names in the listed directories are supposed to be in
VolumeEmergencyEncoding and VolumeEncoding is even not tried. The most
popular case is when an UNIX filesystem tree with UTF-8 contains some
mounts of legacy storages with KOI or CP encoding.
</p>
</div>
</li>
</ul>
<br />
<a name="ini_Replication" />
<h6>[Replication]</h6>
<p>
The replication section sets the transactional replication parameters
for the server.
</p>
<ul>
<li>
<div class="formalpara">
<strong>ServerEnable=0/1</strong>
<p>A boolean parameter controlling whether a Virtuoso can or cannot act as a transactional replication publisher.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerName = log1</strong>
<p>
This identifies the server instance. The entries in SYS_REPL_ACCOUNTS where SERVER equals
this name are considered locally published accounts. This is the value returned by repl_this_server () SQL
function.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>QueueMax = 50000</strong>
<p>
This controls how much synchronized transactional subscribers may fall behind before
being disconnected. This controls how much memory the server will use to buffer undelivered
replication casts. If the queue exceeds this byte amount subscribers are disconnected and
must request re-synchronization. The byte count refers to the total length of the replay records
being buffered. The actual memory usage is somewhat greater.
</p>
</div>
</li>
</ul>
<br />
<a name="ini_Mono" />
<h6>[Mono]</h6>
<ul>
<li>
<div class="formalpara">
<strong>MONO_ROOT</strong>
<p>A path to the directory where the Mono system assemblies are
located. Usually it is a compile time setting, but the MONO_ROOT overrides it.
This ini setting overrides the Mono environment variable of the same name.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MONO_PATH</strong>
<p>A colon separated list of directories where the assemblies are
located to be found by Assembly.Load (equivalent to MS.NET Global assembly cache).
This ini setting overrides the Mono environment variable of the same name.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MONO_CFG_DIR</strong>
<p>A path where the 'machine.config' file is to be found while
running the ASPX code in Mono. This ini setting overrides an
environment variable of the same name. This is also usually a Mono
compile time default.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>virtclr.dll</strong>
<p>A fully qualified path and filename of the virtclr.dll
virtuoso helper assembly.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>MONO_TRACE = Off</strong>
<p>Mono debug tracing can be enabled by setting this parameter to
On. When tracing is On, Mono debug output with be sent to the Virtuoso
debug console.</p>
</div>
</li>
</ul>
<br />
<a name="ini_Client" />
<h6>[Client]</h6>
<ul>
<li>
<div class="formalpara">
<strong>SQL_QUERY_TIMEOUT=0</strong>
<p>This sets the initial value of the SQL_QUERY_TIMEOUT statement option
in connected clients. The ODBC standard value is 0, meaning indefinite, which is
impractical in many applications. This allows overriding the default. The timeout
is expressed in seconds. If the client application sets this option in a statement, this default
is overridden for the statement in question.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_TXN_TIMEOUT=0</strong>
<p>This is an ODBC extension option allowing setting a maximum duration
for a transaction. 0 means that there is no maximum. </p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_PREFETCH_ROWS=100</strong>
<p>For a forward only cursor, this option sets the number of rows prefetched at the
execute and on subsequent fetch requests. A high value will speed up long selects but
will be a disadvantage if only the first few rows are fetched from a cursor that has a large result set.</p>
<p>This should not be confused with the SQL_ROWSET_SIZE setting for scrollable
cursors.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_PREFETCH_BYTES=16000</strong>
<p>This option specifies the maximum number of bytes the server will send as prefetched rows on
a forward only cursor. If long rows are being prefetched this will cut off the prefetch after this many
bytes even if the number of rows is less than SQL_ROWSET_SIZE.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_NO_CHAR_C_ESCAPE=0</strong>
<p>This options is 0 by default and can be either 1 or 0. This option controls Virtuoso's interpretation of
the backslash in PL text which is normal interpreted as escaping rather than literal.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_UTF8_EXECS = 0</strong>
<p>Setting SQL_UTF8_EXECS = 1 enables UTF-8 identifier storage and retrieval, whereas
setting SQL_UTF8_EXECS = 0 disables it. The default setting is 0: disabled for
backwards compatible. See the <a href="wideidentifiers.html">Wide Character Identifiers</a>
section for more information</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_BINARY_TIMESTAMP = 1</strong>
<p>When SQL_BINARY_TIMESTAMP is set to 1 Virtuoso will describe all
TIMESTAMP columns as SQL_BINARY. If it is set to 0 then Virtuoso will report the
TIMESTAMP columns as SQL_TIMESTAMP</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_NO_SYSTEM_TABLES = 0</strong>
<p>This setting can be used to prevent SQLTables from returning system tables.
The default value of this setting is 0, which is allow system tables to be
returned in the normal way. Setting this to 1 will prevent system tables
from being returned from SQLTables. The client can also issue a
"set SQL_NO_SYSTEM_TABLE = 1/0" statement to set this in-connection.</p>
</div>
</li>
</ul>
<br />
<a name="ini_AutoRepair" />
<h6>[AutoRepair]</h6>
<ul>
<li>
<div class="formalpara">
<strong>BadParentLinks=0</strong>
<p>As a result of an internal error in the database the physical integrity of references may
be lost. Enabling this option causes the database to automatically repair such faults without
having to go through a crash dump plus restore. A value of 0 should be normally used.</p>
</div>
</li>
</ul>
<br />
<a name="ini_VDB" />
<h6>[VDB]</h6>
<ul>
<li>
<div class="formalpara">
<strong>ArrayOptimization=0/1</strong>
<p>Boolean parameter which allows the Virtuoso VDB to use Array parameters if the remote data source supports these.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>UseMTS = 0</strong>
<p>This parameter turns on/off MTS support in
Virtuoso. It is applicable to windows multithreaded
version of the Virtuoso server only.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>NumArrayParameters</strong>
<p>Specifies a size of the parameter batch used by the VDB (default = 10)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VDBDisconnectTimeout</strong>
<p>The time (in seconds) after which a VDB connection is considered timed-out and closed. Default : 1000</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>VDBOracleCatalogFix=0/1</strong>
<p>This setting can be enabled to improve
compatibility with the MS Oracle Driver which has problems with mixed case table names
in catalog calls.</p>
<p>Boolean parameter : allows a special mode for
the ORACLE ODBC drivers which do not return correct catalog data for mixed case tables. When
this is on and the return from the catalog functions is empty, then a
Catalog function is issued to take the same catalog data and the result
is filtered on the client side.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>AttachInAutoCommit</strong>
<p>An boolean parameter controlling whether the VDB Catalog functions called
while attaching a table will be called in AutoCommit mode (required by some Sybase drivers).</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>NumArrayParameters=10</strong>
<p />
</div>
</li>
<li>
<div class="formalpara">
<strong>ReconnectOnFailure=0 [1|0]</strong>
<p>The default setting of 0 instructs
the VDB layer to return underlying DB errors to the client rather than automatically
reconnecting without reporting the error, which would be setting 1.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>KeepConnectionOnFixedThread=1 [0|1|2|4|8]</strong>
<p>The default option is 1, this forces the VDB to map
a single thread to each ODBC session in the VDB rather than
multiple threads. Some ODBC drivers expect that calls concerning
a particular connection all take place on one thread.
For example, the Oracle 8.xx ODBC driver can produce
flaky crashes if this option is not set.
</p>
<p>0 - If a second request comes while a thread
is processing the epilogue or previous request for
same connection the second request is scheduled for
the same thread. This is called burst mode.
Once this mode is active all requests on this connection
will be processed in the same thread. until there is a break
at delay more than 'PrpcBurstTimeoutMsecs' elapses
between ending the last and receiving the next in same
connection. This break in activity return the thread to
the thread pool. The burst mode optimization saves thread
switching time and may improve performance by up to 30% in
situations involving short requests immediately
following each other.
</p>
<p>"ForcedFixedThread" mode (4).
This is the same as VDBFixedThread mode,
except that the thread enters this state immediately on
connection (as opposed to doing that on the first VDB activity).
Otherwise the scheduling stays the same as in VDB FixedThread.
</p>
<p>"Forced Burst" mode (8). When a connection is initially
made it's added to the select thread for monitoring.
When an RPC comes in it causes the
select thread to take the connection out of the select thread
and assign that connection to designated worker thread.
That thread from now on will read the data coming in from that
connection directly (not depending on the select thread to
detect the incoming data and wake the worker thread up).
That mode saves the thread switch done handling each RPC
(from the select thread to the worker thread).
When the connection terminates the thread is unbound from
it and returned to it's default idle state.
If an lengthy RPC is being processed the server will
switch the thread from burst mode to default mode while
running the RPC, so the asynchronous cancellation
requests can be processed.</p>
<p>
To disable ever going to burst mode, this option can be set to 2
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PrpcBurstTimeoutMsecs=100 (milliseconds)</strong>
<p>RPC burst mode timeout in milliseconds. (see above)</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SerializeConnect=0 [1|0]</strong>
<p>When enabled causes Virtuoso to wrap
SQLAllocConnect/SQLConnect calls sequence in a mutex, thus preventing abnormal program
exits when there are a large number of connection requests to the VDB. Certain ODBC drivers have been noted to produce flaky errors when a large number of threads is concurrently inside one of the ODBC connect calls.. </p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SkipDMLPrimaryKey=0 [1|0]</strong>
<p>This setting controls SQL compilation (not execution) for conditions
where rows in a local table are used to determine rows of a remote table
to be deleted (e.g. delete from rmt1 where rmc1 = 12 and rmc2 in (select lc1 from lt1))
where the remote table was linked without a Primary Key primary key
specified. Normally the primary key is used, even when not in the
original select select, to identify the rows to be deleted. When the
primary key is false or not available and SkipDMLPrimaryKey = 1
then the original where clause will be used instead.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>RemotePKNotUnique=0 [1|0]</strong>
<p>This option controls the SQL compiler ability to do some optimizations
when it knows it will receive 1 row from a remote table (using the applicable WHERE clause).
In some cases the remote is not providing any info about a primary key (or is providing
wrong data for the primary keys and unique indices of a table) through ODBC API and
the Virtuoso SQL optimizer may get fooled into thinking that a given SQL query
over a remote table will return no more than 1 row.
To avoid that turn on the above parameter to stop the compiler from doing that optimizations.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>UseGlobalPool=0 [1|0]</strong>
<p>This option controls the aggregation point of the VDB connection pools.
When it is off (=0, the default) the VDB connections that are no longer needed after transaction end
are collected server-side in a per-user connection cache. This (together with the
<a href="listitem_ini_VDB_KeepConnectionOnFixedThread.html">KeepConnectionOnFixedThread</a> INI option)
ensures conformance even with older ODBC drivers that require that connections must be used
only within the OS thread that initiated them. This however may result in somewhat low reuse rate
for the pooled connections. That's why when this option is 1 (On) the connections are stored
in a server-side per-DSN connection cache. Keep in mind that this may not work with all the ODBC
drivers, but it may provide lower ratio between virtuoso client connections and VDB-to-remote DBMS
connections, thus requiring smaller number of active connections on the remotes and faster
execution of VDB operations.</p>
</div>
</li>
</ul>
<br />
<a name="ini_Ucms" />
<h6>[Ucms]</h6>
<ul>
<li>
<div class="formalpara">
<strong>UcmPath</strong>
<p>
String parameter which specifies the path where UCM files are located.
If this parameter is not specified, UCM files cannot be loaded.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Ucm1, Ucm2,... Ucm99</strong>
<p>
Every UcmN parameter specifies one UCM file to load.
The value of UcmN is a pair of comma delimited strings.
The first string is a name of the UCM file to load, (relative to the path
specified in UcmPath), the second string is a name of the encoding as it was
used by the server. E.g. Ucm1 = java-Cp933-1.3-P.ucm,Cp933 will load the
encoding from the file java-Cp933-1.3-P.ucm and associate it with the
identifier 'Cp933'. You can register one encoding with more than one
name, if they are delimited by '|' (with no white spaces in the string).
e.g. Ucm2 = java-Cp949-1.3-P.ucm,Cp949|Korean will load the encoding from
the file java-Cp949-1.3-P.ucm and associate it with two identifiers,
'Cp949' and 'Korean'. See <a href="queryingxmldata.html#ucmencodings">UCM
Encodings</a> for more details.</p>
</div>
</li>
</ul>
<br />
<a name="ini_ZeroConfig" />
<h6>[Zero Config]</h6>
<ul>
<li>
<div class="formalpara">
<strong>ServerName</strong>
<p>Name used to advertise the Virtuoso ODBC service details in ZeroConfig.
This is the name that will be shown to clients amongst other ZeroConfig
datasources.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>ServerDSN</strong>
<p>An ODBC style connect string to preset the values of the
parameters when the ODBC service offered by this server is
selected by the Virtuoso ZeroConfig enabled clients.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLServerName</strong>
<p>Name used to advertise the Virtuoso ODBC SSL encrypted
service details in ZeroConfig.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SSLServerDSN</strong>
<p>An ODBC style connect string to preset the values of the
parameters when the ODBC SSL encrypted service offered by this
server is selected by the Virtuoso ZeroConfig enabled clients.</p>
</div>
</li>
</ul>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>The <a href="databaseadmsrv.html#rendezvous">Zero Config</a> section.</p>
</div>
<br />
<a name="ini_Plugins" />
<h6>[Plugins]</h6>
<ul>
<li>
<div class="formalpara">
<strong>LoadPath = /home/virtuoso/hosting</strong>
<p>The directory containing shared objects/libraries
for use as Virtuoso VSEI plugins.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Load<number> = <module type>, <module name></strong>
<p>
<span class="computeroutput"><number></span> is the
module load number, required and starting with 1.
<span class="computeroutput"><module type></span>
specifies the type of module that is to be loaded, and
hence how Virtuoso is to use it. So far "Hosting" and "attach" types exist.
<span class="computeroutput"><module name></span> is
the file name of the modules shared library or object.
</p>
<p>Example:</p>
<p>
<span class="computeroutput">Load6 = attach, libphp5.so</span>)</p>
<p>
<span class="computeroutput">Load7 = Hosting, hosting_php.so</span>)</p>
<p>"Attach" is used for now for the php library. It can be used to load other libraries in future too.
The reason is to load PHP5 functionality into virtuoso namespace, so when actually is loaded the hosting plugin,
it can bind to the already available symbols for php5.
</p>
</div>
</li>
</ul>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="">VSEI Plugins</a>
</p>
</div>
<br />
<a name="ini_Striping" />
<h6>[Striping]</h6>
<ul>
<li>
<div class="formalpara">
<strong>Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ]</strong>
<p><number> must be ordered from 1 upwards; The <size>
is the size of the segment which is equally divided across all stripes comprising
the segment.</p>
</div>
</li>
</ul>
<p>This section is only effective when <span class="computeroutput">Striping = 1</span> is
set in the <span class="computeroutput">[Database]</span> section. When
striping is enabled the Virtuoso database spans multiple segments where each
segment can have multiple stripes.</p>
<p>Striping can be used to distribute the database across multiple locations
of a file system for performance. Segmentation can be used for expansion or
dealing with file size limitations. To allow for database growth when a file
system is near capacity or near the OS file size limitation, new segments can
be added on the same or other file systems.</p>
<p>Striping only has a potential performance benefit when striped across
multiple devices. Striping on the same file system is needless and unlikely to
alter performance, however, multiple segments do provide convenience and
flexibility as described above. Striping across partitions on the same device is
likely to reduce performance by causing high unnecessary seek times on the
physical disk.</p>
<p>Database segments are pre-allocated as defined. This can reduce the
potential for file fragmentation which could also provide some performance
benefit.</p>
<p>Virtuoso striping alone does not allow for any fault tolerance. This is
best handled at the I/O layer or by the operating system. File system RAID with
fault-tolerant striping defined should be used to host the Virtuoso files if
striping based protection is desired.</p>
<p>The segments are numbered, their segment <number> must
be specified in order starting with segment1.</p>
<p>The <size> is the total size of the segment which is that will be
divided equally across all stripes comprising the segment. Its specification
can be in gigabytes (g), megabytes (m), kilobytes (k) or in
database blocks (b) the default.</p>
<div class="note">
<div class="notetitle">Note:</div>
<p>The segment size must be a multiple of the database page size which
is currently 8k. Also, the segment size must be divisible by the number of
stripe files contained in the segment.</p>
</div>
<p>Segments can be added to a database, however once defined they
should never be altered. Databases that were created without striping
cannot automatically be restarted with striping. You can convert a non-striping
database to striping by dumping the contents of the database to a transaction file
and replaying it with striping enabled. An on-line backup made with backup_online () can be restored on a database with a different striping configuration as long as the total number of pages is no less than the number of pages of the backed up database.
</p>
<p>Striping can be useful for the temporary objects database if large hash join temporary spaces or such are expected. This is enabled by the Striping setting in the Temp Database section of the ini file.
The stripes will be declared in the TempStriping section.
</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="databaseadmsrv.html#dbrebuild">Rebuilding A Database</a> in the Backup section.</p>
</div>
<br />
<br />
<a name="sampleinifile" />
<h5>6.1.1.5.2. Sample Configuration File ("virtuoso.ini")</h5>
<p>Following is the text of the sample virtuoso.ini file that comes with the distribution.
</p>
<div>
<pre class="programlisting">
;
; virtuoso.ini
;
; Configuration file for the OpenLink Virtuoso VDBMS Server
;
;
; Database setup
;
[Database]
DatabaseFile = virtuoso.db
TransactionFile = virtuoso.trx
ErrorLogFile = virtuoso.log
ErrorLogLevel = 7
FileExtend = 200
Striping = 0
; LogSegments = 1
; crashdump_start_dp = 0
; crashdump_end_dp = 0
; Log1 = log1.trx
;
; Server parameters
;
[Parameters]
ServerPort = 1111
ServerThreads = 10
CheckpointInterval = 60
CheckpointSync = 2
NumberOfBuffers = 2000
MaxDirtyBuffers = 1200
MaxCheckpointRemap = 2000
PrefixResultNames = 0
CaseMode = 1
;MinAutoCheckpointSize = 4000000
AutoCheckpointLogSize = 40000000
CheckpointAuditTrail = 1
[HTTPServer]
ServerPort = 1122
ServerRoot = ../vsp
ServerThreads = 2
MaxKeepAlives = 10
KeepAliveTimeout = 10
MaxCachedProxyConnections = 10
ProxyConnectionCacheTimeout = 15
DavRoot = DAV
[Replication]
ServerName = log1
Server = 1
QueueMax = 50000
[Client]
SQL_QUERY_TIMEOUT = 0
SQL_TXN_TIMEOUT = 0
SQL_PREFETCH_ROWS = 100
SQL_PREFETCH_BYTES = 16000
[AutoRepair]
BadParentLinks = 0
BadDTP = 0
;[Ucms]
;UcmPath = /ucm
;Ucm1 = java-Cp933-1.3-P.ucm,Cp933
;Ucm2 = java-Cp949-1.3-P.ucm,Cp949|Korean
;Ucm3 = java-ISO2022KR-1.3-P.ucm,ISO2022KR|ISO2022-KR
;
; Striping setup
;
; These parameters have only effect when Striping is set to 1 in the
; [Database] section, in which case the DatabaseFile parameter is ignored.
;
; With striping, the database spans multiple segments
; where each segment can have multiple stripes.
;
; Format of the lines below:
; Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ]
;
; <number> must be ordered from 1 up.
;
; The <size> is the total size of the segment which is equally divided
; across all stripes comprising the segment. Its specification can be in
; gigabytes (g), megabytes (m), kilobytes (k) or in database blocks
; (b, the default)
;
; Note that the segment size must be a multiple of the database page size
; which is currently 8k. Also, the segment size must be divisible by the
; number of stripe files constituting the segment.
;
; The example below creates a 200 meg database striped on two segments
; with two stripes of 50 meg and one of 100 meg.
;
; You can always add more segments to the configuration, but once
; added, do not change the setup.
;
[Striping]
Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
Segment2 = 100M, db-seg2-1.db
</pre>
</div>
<br />
<a name="indexdefragm" />
<h5>6.1.1.5.3. Index Defragmentation</h5>
<p>
When data is inserted into tables, database pages are split and typically pages end up not being fully utilized.
If data is inserted in ascending order of key value, which is often the case, space utilization is more efficient.
Still, gaps can be left by updates, deletions and page splitting.
Virtuoso has an autocompact feature which will take groups of adjacent dirty pages and see if it can fit
the same content on a smaller number of pages. It will rewrite the pages to save space before
writing these to disk. This operates automatically. Statistics on autocompaction are shown beside
the Read ahead status line of the result set of status ('');. The number of pages affected by
autocompaction and the number of resulting pages are shown. The numbers are cumulative since the start of the instance.
</p>
<p>
Autocompact is non-locking and if pages are busy, they will not be touched. Only relatively old dirty pages,
about to be written to disk are considered for compaction, not interfering with the hottest part of the working set.
</p>
<p>
The automatic compaction is not however effective if pages are updated singly, never making stretches of
consecutive dirty pages. Therefore a manual compaction function called DB.DBA.VACUUM () is also offered.
</p>
<p>
The vacuum stored procedure gets an optional table and index name and will read the index from beginning to end.
If neither argument is given, all indices of all tables will be compacted. If only the table is given, all
indices of this table will be compacted.
</p>
<p>
If consecutive leaves can be fit on fewer pages than they now occupy, this will rewrite them and free the pages left over.
This does however require transient space since the pages are not really replaced until the next checkpoint, hence a
vacuum operation can run out of disk. Using the checkpoint statement to force a checkpoint will free the space.
</p>
<p>
The effects and need for explicitly vacuuming a database can be assessed with the DB.DBA.SYS_INDEX_SPACE_STATS view.
</p>
<p>Running:
</p>
<div>
<pre class="programlisting">
select * from DB..SYS_INDEX_SPACE_STATS order by ISS_PAGES desc;
</pre>
</div>
<p>
produces a result set with the most space consuming index on top. ISS_PAGES is the total count of pages.
ISS_ROW_BYTES is the byte count of the rows. If dividing the total count of bytes by the count of pages is much
below 8172, (8K - 20), chances are that vacuuming the index may save space. Note that blobs are not affected by vacuuming.
If the blobs are small enough to fit on the row as normal strings they are already there.
Otherwise they occupy the needed number of pages and cannot be made more compact.
</p>
<p>
Note that querying the SYS_INDEX_SPACE_STATS view will always read through all the allocated pages of the database
and may take a while. The operation is not locking. Only the state as of the last checkpoint will be shown, hence
it is a good idea to run the checkpoint statement before querying this view.
</p>
<p>
Examples:
</p>
<div>
<pre class="programlisting">
DB..VACUUM ();
-- Compact all tables and indices of the Virtuoso instance
</pre>
</div>
<div>
<pre class="programlisting">
Db..VACUUM ('WS.%');
-- compact all tables of the WS. qualifier
</pre>
</div>
<div>
<pre class="programlisting">
DB..VACUUM ('DB.DBA.RDF_QUAD', 'RDF_QUAD_PGOS');
-- compact the rdf_quad_phgos index of the rdf_quad table.
</pre>
</div>
<p>
Virtuoso has an autocompact feature.
</p>
<br />
<br />
<a name="COMMANDLINE" />
<h4>6.1.1.6. Server Startup Command Line Options</h4>
<a name="SERVER" />
<h5>6.1.1.6.1. Virtuoso Server</h5>
<p>
This section presents the command line switches of the Virtuoso
server executable. Depending on the model and virtual database middleware the
server will have different names, all starting with virtuoso-. All these however
have the same options for UNIX systems and slightly different for Windows platform.
</p>
<p>
on Windows platform are available following server command line options:
</p>
<div>
<pre class="programlisting">
Usage:
virtuoso-odbc-t.exe [-clnCbDARf---dSIMKmrd] [+configfile arg] [+licensefile arg]
[+no-checkpoint] [+checkpoint-only] [+backup-dump]
[+crash-dump] [+crash-dump-data-ini arg]
[+restore-crash-dump] [+foreground] [+pwdold arg]
[+pwddba arg] [+pwddav arg] [+debug] [+service arg]
[+instance arg] [+mode arg] [+dumpkeys arg] [+manual]
[+restore-backup arg] [+debug]
+configfile specify an alternate configuration file to use,
or a directory where virtuoso.ini can be found
+licensefile specify an alternate license file to use,
or a directory where virtuoso.ini can be found
+no-checkpoint do not checkpoint on startup
+checkpoint-only exit as soon as checkpoint on startup is complete
+backup-dump dump database into the transaction log, then exit
+crash-dump dump inconsistent database into the transaction log,
then exit
+crash-dump-data-ini specify the DB ini to use for reading the data to dump
+restore-crash-dump restore from a crash-dump
+foreground run in the foreground
+pwdold Old DBA password
+pwddba New DBA password
+pwddav New DAV password
+debug allocate a debugging console
+service specify a service action to perform
+instance specify a service instance to start/stop/create/delete
+mode specify mode options for server startup (onbalr)
+dumpkeys specify key id(s) to dump on crash dump (default : all)
+manual specify when create a service to make it for manual startup
+restore-backup restore from online backup
+debug Show additional debugging info
The argument to the +service option can be one of the following options
start start a service instance
stop stop a service instance
create create a service instance
screate create a service instance without deleting the existing one
delete delete a service instance
list list all service instances
</pre>
</div>
<p>
The below are switches for server for UNIX platforms:
</p>
<div>
<pre class="programlisting">
Usage:
virtuoso-iodbc-t [-fclnCbDARwMKr---d] [+foreground] [+configfile arg]
[+licensefile arg] [+no-checkpoint] [+checkpoint-only]
[+backup-dump] [+crash-dump] [+crash-dump-data-ini arg]
[+restore-crash-dump] [+wait] [+mode arg] [+dumpkeys arg]
[+restore-backup arg] [+pwdold arg] [+pwddba arg]
[+pwddav arg] [+debug]
+foreground run in the foreground
+configfile use alternate configuration file
+licensefile use alternate license file
+no-checkpoint do not checkpoint on startup
+checkpoint-only exit as soon as checkpoint on startup is complete
+backup-dump dump database into the transaction log, then exit
+crash-dump dump inconsistent database into the transaction log, then exit
+crash-dump-data-ini specify the DB ini to use for reading the data to dump
+restore-crash-dump restore from a crash-dump
+wait wait for background initialization to complete
+mode specify mode options for server startup (onbalr)
+dumpkeys specify key id(s) to dump on crash dump (default : all)
+restore-backup restore from online backup
+pwdold Old DBA password
+pwddba New DBA password
+pwddav New DAV password
+debug Show additional debugging info
</pre>
</div>
<p>
The <strong>+crash-dump</strong> option will make use of the segmented log
defined in virtuoso.ini for storing the recovery log. See
<a href="databaseadmsrv.html#vdbrecovery">Crash Recovery</a> and virtuoso.ini below for
more information. The other options will not use the segmented log.
</p>
<p>
The <strong>+restore-crash-dump</strong> option will alter the server
startup sequence so that the recovery log produced by +crash-dump will
be re-played correctly.
</p>
<p>
The <strong>+mode</strong> option can be a combination of the following letters:
</p>
<ul>
<li>
<strong>o</strong> - only open the database and define the SYS_KEYS, SYS_COLS, SYS_KEY_PARTS, SYS_CHARSETS AND SYS_COLLATIONS system tables.</li>
<li>
<strong>n</strong> - leaves out the initialization of the system tables.</li>
<li>
<strong>b</strong> - do not process anything except the transaction log and system tables when restoring a crash dump (+restore-crash-dump).</li>
<li>
<strong>a</strong> - leaves out the initialization of the replication, users, compilation or stored and system procedures, as well
as the caching of the grants.</li>
<li>
<strong>l</strong> - write only the schema tables in the backup or crash dump.</li>
<li>
<strong>r</strong> - don't do the complete initialization (useful for performing a crash dump).</li>
</ul>
<p>
On Unix platforms the executable will detach itself from the console and run in the background
as a daemon unless the <strong>+foreground</strong> switch is specified.
</p>
<p>
For Windows NT and Windows 2000, the Virtuoso server will normally be installed
as a Windows service and can be started from the Control Panel or
automatically at system startup.
</p>
<p>
Ordinarily the Windows service will be a system process that runs in the background.
If you want the Virtuoso service on Windows to allocate a debugging
console the you can use the <strong>+debug</strong> (-d) switch.
This switch is only applicable to starting a service.
</p>
<p>
Virtuoso on Windows can be run directly from the command line using the
<strong>+foreground</strong> (-f) switch. The server will then start in
the foreground of the current "cmd" session. If this switch is not used
then the executable on Windows will assume that you are attempting to start
a Virtuoso service.
</p>
<p>
Windows services can be created and removed from the system as required. The
default installation under Windows will create a service by the name:
<strong>OpenLink Virtuoso VDBMS Server</strong>, and optionally another
service with the name <strong>OpenLink Virtuoso VDBMS Server [demo]</strong>.
The Demo service is a supplied demonstration database that can be installed.
</p>
<p>
The following options are available to the <strong>+service</strong>
switch for configuring Virtuoso services:
</p>
<ul>
<li>
<strong>start</strong> start a service instance</li>
<li>
<strong>stop</strong> stop a service instance</li>
<li>
<strong>create</strong> create a service instance</li>
<li>
<strong>screate</strong> create a service instance without deleting the existing one</li>
<li>
<strong>delete</strong> delete a service instance</li>
<li>
<strong>list</strong> list all service instances</li>
</ul>
<p>They are used with the <strong>+instance <name></strong>
where <strong><name></strong> is the instance name to configure a particular instance. All
instances are listed in the services applet, with their name in square brackets.</p>
<p>
<strong>+service list</strong> can be used to obtain the list
if services that are registered with Windows.</p>
<p>For each service listed you can start, stop, or delete the service.</p>
<p>
<strong>+service create</strong> can be used to create a new service.
In this case you also need to specify other start up options that would be
associated with the new service entry. If you were using an alternative
configuration file this must be specified using <strong>+configfile</strong>
switch.</p>
<div class="note">
<div class="notetitle">Note:</div>
<p>Make sure the Services Control Panel is closed, before attempting to
modify services from the command line, otherwise locking may occur.</p>
</div>
<br />
<br />
<a name="rendezvous" />
<h4>6.1.1.7. ZeroConfig ("Zero Configuration") Support</h4>
<p>The "ZeroConfig" protocol, also known as "Zeroconf" or
"Zero Configuration" is a protocol that allows discovery of services on the
network that are advertised by their hosts. It also has provisions for
automatic discovery of computers and various devices.
The main benefit of ZeroConfig is that it does not require DHCP, DNS or
directory servers.</p>
<p>ZeroConfig is an open protocol that Apple submitted to the IETF for
a standard-creation process.</p>
<p>The Virtuoso server and ODBC driver use the capabilities of ZeroConfig to
facilitate DSN (Data Source Name) setup and usage. This is divided in two parts:
Server-side and Client-side.</p>
<p>The Virtuoso server (Server-side) is configured via the Virtuoso INI
file to advertise its availability on a network with a given name. This allows
applications, and in particular the Virtuoso ODBC driver, to receive information
about a server, such as its network address, default login, etc, and use it
for configuring a data source or directory making a connection.</p>
<p>The Virtuoso ODBC driver (Client side) uses ZeroConfig to locate
the desired Virtuoso server during the set-up phase of a data source, and
determine available connection options such as:</p>
<ul>
<li>secure connection options</li>
<li>default database</li>
<li>default user </li>
<li>default password (if public/demo login is required)</li>
<li>default character set </li>
</ul>
<p>ZeroConfig provides the client with a service name, which must be bound to
the IP address/port of a host of the chosen service during DSN configuration.
This is used when existing DSN using a ZeroConfig name is used to connect,
it will map name with IP address and port before making a connection.</p>
<a name="rendezvousserverside" />
<h5>6.1.1.7.1. Setting-up the Server for Service Advertising</h5>
<p>The Virtuoso server is configured to advertise itself based on the
details specified in the <span class="computeroutput">[Zero Config]</span>
section of the Virtuoso INI file. Below is an example of such:</p>
<div>
<pre class="programlisting">
...
[Zero Config]
ServerName = Virtuoso Server
ServerDSN = UID=demo;PWD=
SSLServerName = Virtuoso Server (via SSL)
SSLServerDSN = UID=dba;PWD=;ENCRYPT=1
...
</pre>
</div>
<p>The ServerName and SSLServerName are human readable
strings chosen by the administrator to provide clients with a suitable
description of the service being provided.</p>
<div class="note">
<div class="notetitle">Note:</div>
<p>If the Virtuoso does not have the SSL listener enabled then the
SSL service will not be advertised automatically.
The SSL* keys will simply be ignored and do not need to be removed.
</p>
</div>
<p>The ServerDSN and SSLServerDSN are default connection
strings that can be used by clients to make the advertised connection. You only
need to to specify default username and password in these strings. The
default database can be specified or left to the setting for the username.
You cannot specify the server hostname, IP address or port number, these
are supplied by Virtuoso automatically.</p>
<p>ZeroConfig service advertising is multicast, hence it is advertised
on all available network interfaces.</p>
<br />
<a name="rendezvous" />
<h5>6.1.1.7.2. Using the Windows ODBC Driver with ZeroConfig</h5>
<p>Upon DSN set-up the ODBC driver listens for advertising servers, and
compiles a roster. This is displayed for the user to choose the desired
service to connect to. </p>
<p>If ZeroConfig is used to for data source set-up then the
set-up dialog will be initialized based on the details in the connection
string configured on the server.</p>
<p>When a DSN is configured based on a ZeroConfig service, the driver will
resolve the service name before making the connection to the server. The driver
does not store the network address or port number of the Virtuoso server, only
the ZeroConfig server name, so if the server's physical address is changed
the client DSNs associated with it do not all have to reconfigured; they will
resolve to the new address automatically on next use.</p>
<br />
<br />
<a name="DBSTAT" />
<h4>6.1.1.8. Server Status Monitoring</h4>
<p>The database status report is divided into 6 sections:</p>
<ul>
<li>Server</li>
<li>Database</li>
<li>Locks</li>
<li>Clients</li>
<li>Replication</li>
<li>Index Usage</li>
</ul>
<a name="Server" />
<h5>6.1.1.8.1. Server</h5>
<p>
This section shows how many connections are open and how many threads
the process has and how many are running at the present time. This also
displays the number of requests that have been received but are not yet
running on any thread.
</p>
<br />
<a name="Database" />
<h5>6.1.1.8.2. Database</h5>
<div>
<pre class="screen">
File size 203161600, 24800 pages, 259 free.
7000 buffers, 6987 used, 3884 dirty 8 wired down, repl age 8251 .
Disk Usage: 14246 reads avg 6 msec, 74% read last 10 s, 14457 writes,
4 read ahead, batch = 5.
Gate: 2729 2nd in reads, 0 gate write waits, 3372547 in while read 0 busy scrap.
Log = wi.log, 9851835 bytes
14950 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 18 connects, max 17 concurrent, 1024 licensed
RPC: 54441 calls, 17 pending, 17 max until now, 0 queued, 53988 burst reads (99%), 0 second
Checkpoint Remap 7646 pages, 0 mapped back. 0 s atomic time.
DB master 24800 total 259 free 7646 remap 3415 mapped back
temp 200 total 196 free
</pre>
</div>
<p>
The status consists of the following items:
</p>
<div class="formalpara">
<strong>File size:</strong>
<p>The database file size in bytes or 0 if the database consists
of statically allocated files. The total number of 8K database pages follows, then
the number of free pages. The number of buffers shown the total count
of 8K file cache buffers, followed by the number of used buffers and
the number of buffers that are dirty at the time. The wired down count
is normally zero but can be transiently other if pages are wired down
for processing by threads in the server.
</p>
</div>
<div class="formalpara">
<strong>Disk Usage:</strong>
<p>Shows the cumulative total number of reads and writes
and the average length of time spent inside the read system call for
the database files.</p>
<p>
The percentage is the percentage of the real time spent inside read
between this status report and the previous status report. This may exceed
100% if several reads are taking place concurrently on different stripes
in a multi-file database.
</p>
</div>
<div class="formalpara">
<strong>The Gate:</strong>
<p>Lists concurrent events. The <strong>2nd in read</strong> is
the count of concurrent requests for the same page, the
<strong>gate write waits</strong> is the count of times a modify operation had to wait for
exclusive access to a page being read by another thread, the <strong>in while read</strong>
is the count of file cache hits that have taken place while a read system
call was in progress on another thread.
</p>
</div>
<div class="formalpara">
<strong>Databases</strong>
<p>
Thus section shows the count of pages, free pages, checkpoint remap and mapped back for the main database and the space for temporary data such as sort results and hash indices.
The page count is the total size, the free count is the count of free pages, the checkpoint remap is the count of pages that occupy two pages in checkpoint space instead of one, the mapped back count is the number of pages that will return to their original place in checkpoint space at the next checkpoint. Understanding these is not necessary.
</p>
</div>
<div class="tip">
<div class="tiptitle">See:</div>
<p>The Disk Configuration section for a discussion of checkpoint remapping.</p>
</div>
<br />
<a name="Locks" />
<h5>6.1.1.8.3. Locks</h5>
<p>
The lock section shows various locking statistics accumulated since the
server was started. The deadlock count is divided into deadlocks caused
by a situation where several transactions read a page and one wants to
get write access and all other deadlock situations. The first is called
2r1w deadlock in the report.
</p>
<p>
The lock section also shows the total number of threads running,
i.e. engaged in performing some operation for a SQL or web client.
The number of threads waiting is the number of running threads that
are presently waiting for a lock. The number of threads in vdb is the
number of threads engaged in remote database operations or other
'slow' I/O, such as access to outside HTTP or SOAP services.
</p>
<p>
All locks currently in effect are listed with the owners (a) and possibly
waiting transactions. The transactions are named after their client. A
log or replication replay transaction is here named 'INTERNAL'.
</p>
<br />
<a name="Clients" />
<h5>6.1.1.8.4. Clients</h5>
<p>
Each connected client is listed with the number of bytes sent and received
from the client. The transaction status is either PENDING for OK or
BLOWN OFF or DELTA ROLLED BACK for a transaction killed by deadlock or
timeout. The locks owned by the transaction are listed following the
status. IE means exclusive and IS shared lock.
</p>
<br />
<a name="Replication" />
<h5>6.1.1.8.5. Replication</h5>
<p>
This section shows the server in question and a list of replication
accounts either provided or received by this server. Accounts where
the server name (left column) is the same as the server name are those
provided by this.
</p>
<p>
The columns are server name, account name, last transaction number and
status. The status is OFF for a local account or a replicated account
where the remote is not available. It is SYNCING if a resync is in
progress, IN SYNC if the account is up to date or REMOTE DISCONNECTED if
there was a connection to a remote party which subsequently disconnected.
</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="rdfgraphreplication.html">RDF Graph Replication</a>
</p>
</div>
<br />
<a name="indexusage" />
<h5>6.1.1.8.6. Index Usage</h5>
<p>
This part of the report summarizes the database's access statistics.
The output is a table with a row for each index in the database. Each
row is composed of the following columns:</p>
<div>
<pre class="screen">
Table The name of the table
Index The name of the index. Same as the table for primary key.
Touches The number of touches since startup. Each time the database
engine looks at an entry of the index is counted as a touch.
Not all touched entries are selected. For instance if the
engine scans a table with non-indexed selection criteria it
will touch each row but might select none.
Reads The number of disk reads caused by reading this index.
%Miss The percentage of touches that required a read. This can be
over 100% since getting one entry may required more than one
read if the top levels of the index are not in memory.
Locks The number of times a lock is set on an entry of the index.
Waits The number of times the engine has to wait for another
transaction to finish in order to set a lock on this index.
%W The percentage of waits of all locks set.
</pre>
</div>
<p>
In interactive SQL
</p>
<div>
<pre class="screen">
SQL> status();
</pre>
</div>
<p>Will print out the report.</p>
<br />
<br />
<a name="oemrelabeling" />
<h4>6.1.1.9. Re-labeling Server Executable on Win32 Platforms</h4>
<p>
The Virtuoso Service name can be altered using the key <strong>Win32ServiceName</strong> in the
<strong>Parameters</strong> section. The default name is 'OpenLink Virtuoso VDBMS Server'
</p>
<p>
To change the name of services:
</p>
<ul>
<li>stop the service</li>
<li>delete the service</li>
<li>change the name in file virtuoso.ini</li>
<li>create the service</li>
<li>start the service</li>
</ul>
<p>Services with old names must be deleted before creating service with the new name, i.e. with the
Win32ServiceName setting set to the current name of the service.
</p>
<p>
The name displayed in the ODBC Administrator, Setup and Configuration dialogs is taken from the driver section in the ODBCINST.
This can be directly edited in the registry using the regedt32 Windows utility, or a registry import file can be created which can be
applied by simply double-clicking the .reg file. Always exercise extreme caution when making changes to the registry.
</p>
<br />
<a name="dbsrcsecurity" />
<h4>6.1.1.10. Transport Level Security</h4>
<a name="srvadmencryption" />
<h5>6.1.1.10.1. Encryption</h5>
<p>
Virtuoso has the ability to encrypt it's CLI network connections using SSL.
The server listens on a separate port for SSL CLI connections and handles them just as
the normal CLI connections, vut now providing transport level security.
</p>
<a name="srvsidesupport" />
<h6>Server-side Support</h6>
<p>
Server side secure connections utilizes three parameters in the [Parameters] section of the virtuoso.ini:
</p>
<ul>
<li>
<strong>SSLServerPort</strong> - Specifies the port on which the server listens for incoming SSL CLI requests.</li>
<li>
<strong>SSLCertificate</strong> - The SSL certificate to use (same meaning as the SSLCertificate in HTTPServer section)</li>
<li>
<strong>SSLPrivateKey</strong> - The server's private key (same meaning as the SSLCertificate in HTTPServer section)</li>
</ul>
<p>
These parameters should be all set in order to enable the SSL CLI server.
</p>
<p>
If SSLServerPort is not specified, then the Virtuoso server ignores the other two and does not listen for
SSL CLI connections.
</p>
<p>
If a non-SSL connection is attempted to the SSL server port, the server rejects the connection.
If an SSL connection is attempted against the non-SSL port the server rejects the connection.
</p>
<br />
<a name="clisidesupport" />
<h6>Client-side Support</h6>
<p>
The client does not require any SSL-specific files (like Certificates or Private keys)
in the SSL connection process.
</p>
<div class="formalpara">
<strong>Native Clients (e.g. ISQL)</strong>
</div>
<p>
There is an custom ODBC connect option SQL_ENCRYPT_CONNECTION (=5004) supported by the Virtuoso CLI.
It should be set before issuing the SQLConnect call. Values are 'NULL' (no encryption - default), '1'
(encryption with no server X509 certificate checking and no X509 certificate sent to the server) and
a valid file path to a PKCS#12 certificate file (protected with the same password as the one used
to log in. Note that with the iODBC/ODBC clients this connect option is not applicable since the driver
managers don't cache or pass through the custom ConnectOptions set before connecting to the data source.
The ISQL has an additional option (-E) to do encrypted connects using the encryption option '1')
and -X <file> to set the above option to the file supplied.
</p>
<div class="formalpara">
<strong>ODBC & iODBC Driver</strong>
</div>
<p>
The drivers support an additional DSN attribute:
</p>
<div>
<pre class="programlisting">
ENCRYPT=<string>
</pre>
</div>
<p>
If this attribute is not specified then it defaults to "No".
</p>
<p>
It has the same meaning as the SQL_ENCRYPT_CONNECTION options (see above).
If this is not specified then it defaults to NULL.
</p>
<p>
The corresponding iODBC odbc.ini & ODBC Registry DSN attribute name is:
</p>
<div>
<pre class="programlisting">
"Encrypt"= <string>
</pre>
</div>
<p>
The Windows Connect & Setup dialogs have an additional wizard page to configure encryption.
</p>
<br />
<a name="x509certsupport" />
<h6>X509 Certificate Support</h6>
<p>
Virtuoso supports X509 certificate validation: server side for both ODBC and HTTPS
connections, and client side for the ODBC connections.
</p>
<div class="formalpara">
<strong>Server Side</strong>
</div>
<p>
To enable this option there are three new INI file parameters added the HTTPServer section for the HTTPS,
and the Parameters section for ODBC):
</p>
<ul>
<li>
<strong>X509ClientVerify</strong> - Whether the server will require
X509 certificates from the browsers. If set to 1 a client should send a X509 certificate which will be
validated against the server CA list.
</li>
<li>
<strong>X509ClientVerifyDepth</strong> - Specifies how deep the client certificate
verification process will traverse the Issuer chain before giving up. The default value of 0
means that only verification of the client certificate itself being in the server's CA list is
performed. Setting this option to -1 will ignore the depth checks.
</li>
<li>
<strong>X509ClientVerifyCAFile</strong> - a PEM (base64) file of all the
X509 certificates of the Certification Authorities (CA) which the server will use for verifying the
client's client certificate. A list of these will be sent to the client as a part of the
SSL handshake so the client will know which certificate to send.
</li>
</ul>
<div class="formalpara">
<strong>ODBC Client Side</strong>
</div>
<p>
In order for verification of the server certificate to take place a PKCS#12 file
should be supplied to the ODBC client. It will use the CA list in this PKCS#12 to
verify the server certificate. It will set the verification depth to -1 (unlimited)
while performing such a check.
</p>
<p>
If the server certificate is not verified correctly it will refuse to connect to the server.
When the ENCRYPT parameter is set to "1" (do SSL without X509 validation) the client will return a
SQL_SUCCESS_WITH_INFO in SQLConnect/SQLDriverConnect with the Server's certificate subject
and the verification result as the server will always send it's X509 certificate to the
client as a part of the SSL connect handshake.
</p>
<p>
If the PKCS#12 file is supplied the ODBC client will try to open it using the
login password. In order for the file to be successfully opened it should be encrypted with
the same password used for logging in.
</p>
<p>
Normally when exporting a PKCS#12 file from other programs it will contain only the
CAs of the Certificate validation chain. This means that client and server certificates should have
common CA in their certificate chains in order to be used for ODBC X509 validation. The
client certificate from the PKCS#12 file will not take place in the server
certificate validation process.
</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="fn_get_certificate_info.html">get_certificate_info()</a>
</p>
</div>
<br />
<br />
<a name="acl" />
<h5>6.1.1.10.2. File System Access Control Lists</h5>
<p>
Access Control Lists (ACL) are used to restrict file system access.
</p>
<p>
These lists are maintained in the Virtuoso INI file under the Parameters section with entries such as:
</p>
<div>
<pre class="programlisting">
DirsAllowed = <path> [, <path>]
DirsDenied = <path> [, <path>]
<path> := <absolute_path> or <relative_path>
</pre>
</div>
<div class="note">
<div class="notetitle">Note:</div>
<p>A relative path is relative to the servers current working directory.</p>
</div>
<p>The Virtuoso ISQL utility can be used to check the Server DirsAllowed params
as follows:</p>
<div>
<pre class="programlisting">
SQL> select server_root (), virtuoso_ini_path ();
</pre>
</div>
<p>The above should show in the result the server working directory and INI
file name.</p>
<p>Also you can check the relevant INI setting by running following
statement via ISQL command line utility:</p>
<div>
<pre class="programlisting">
SQL> select cfg_item_value (virtuoso_ini_path (), 'Parameters',
'DirsAllowed');
</pre>
</div>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>
<a href="databaseadmsrv.html#VIRTINI">Virtuoso INI File Configuration</a>
</p>
</div>
<p>
ACL's work in the following way:
</p>
<ul>
<li>All paths are converted from relative to absolute paths.</li>
<li>The path beginning with <http_root> is always allowed.</li>
<li>All DB files are always access denied (.db, db segments, .trx, log segments, .ini specified in INI file etc.)</li>
<li>If a path is not allowed or exists as denied then access to the file is rejected. </li>
<li>If a requested path is allowed and not in denied then access is allowed.</li>
<li>ACL's are inherited. If a directory allows access so does its subdirectories.</li>
</ul>
<p>
The following functions are restricted by file Access Control Lists (ACL) in the virtuoso.ini file:
</p>
<ul>
<li>
<a href="fn_file_to_string.html">file_to_string</a>
</li>
<li>
<a href="fn_file_to_string_output.html">file_to_string_output</a>
</li>
<li>sys_mkdir</li>
<li>sys_dirlist</li>
<li>
<a href="fn_string_to_file.html">string_to_file</a>
</li>
<li>cfg_write</li>
</ul>
<div class="note">
<div class="notetitle">Note:</div>
<p>the cfg_write function has restrictions against changing file access control lists in ini file</p>
</div>
<br />
<br />
<br />
<a name="vdbconcepts" />
<h3>6.1.2. Virtual Database</h3>
<a name="linkrmttableauto" />
<h4>6.1.2.1. Linking Remote Tables & Views</h4>
<p>The Virtuoso Server supports linking in of tables, views, procedures and
other components from a remote ODBC data-source, enabling them to be
queried as native objects in Virtuoso; this process is called
"attaching" or "linking". The easiest way to link to an external table is
to use the <a href="htmlconductorbar.html#remotetables">Linking Remote Tables Wizard</a>,
part of the Visual Server Administration Interface. Alternatively you can
attach these objects programmatically, as this section explains;
finally you can attach tables manually - see <a href="databaseadmsrv.html#MANSETRDS">Manually
Setting Up A Remote Data Source</a> which is useful for connections to
less-capable ODBC data-sources.</p>
<a name="ATTACH" />
<h5>6.1.2.1.1. ATTACH TABLE Statement</h5>
<div>
<pre class="programlisting">
ATTACH TABLE <table> [PRIMARY KEY '(' <column> [, ...] ')']
[AS <local_name>] FROM <dsn> [USER <uid> PASSWORD <pwd>]
[ON SELECT] [REMOTE AS <literal_table_name>]
</pre>
</div>
<table class="varlist">
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">table:</td>
<td>
<p>Adequately qualified table name of the form: identifier | identifier.identifier | identifier.identifier.identifier | identifier..identifier</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">column:</td>
<td>
<p>column to assume primary key</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">local_name:</td>
<td>
<p>fully qualified table name specifying local reference.</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">dsn:</td>
<td>
<p>scalar_exp</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">user:</td>
<td>
<p>scalar_exp</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">password:</td>
<td>
<p>scalar_exp</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">literal_table_name:</td>
<td>
<p>scalar_exp</p>
</td>
</tr>
</table>
<p>This SQL statement defines a remote data source, copies the schema
information of a given table to the local database and defines the table as a
remote table residing on the data source in question.
</p>
<p>
The table is a designation of the table's name on the remote data source dsn. It
may consist of an optional qualifier, optional owner and table names, separated
by dots. This must identify exactly one table on the remote dsn. The optional
local_name is an optionally qualified table name which will identify the table on
the local database. If qualifier or owner are omitted, these default to the current
qualifier 'dbname()' and the logged in user, as with CREATE TABLE. If the
local_name is not given it defaults to the <current qualifier>.<dsn>.<table name on
dsn>. The <dsn> will be the dsn with all alphabetic characters in upper case and
all non-alphanumeric characters replaced by underscores. The <table name on
dsn> will be the exact name as it is on the remote dsn, case unmodified.
</p>
<p>
The PRIMARY KEY option is only required for attaching views or tables where the primary key on the
remote table cannot be ascertained directly from the remote data source.</p>
<p>
If a dsn is not previously defined with vd_remote_data_source or ATTACH
TABLE, the USER and PASSWORD clauses have to be given.
</p>
<p>The <strong>REMOTE AS</strong> option allows you to provide a
string literal for referencing the remote table. This is useful when linking
tables from sources that do not support three-part qualification correctly.</p>
<br />
<a name="attachview" />
<h5>6.1.2.1.2. Attaching views</h5>
<p>
A view can be attached as a table if a set of uniquely identifying
columns is supplied.
</p>
<p>
This is done with the PRIMARY KEY option to ATTACH TABLE as follows:
</p>
<div>
<pre class="programlisting">
attach table T1_VIEW primary key (ROW_NO) from 'somedsn';
</pre>
</div>
<div class="note">
<div class="notetitle">Note:</div>
<p>Views cannot be attached unless the PRIMARY KEY options is used.</p>
</div>
<br />
<a name="linkrmttableautoexamples" />
<h5>6.1.2.1.3. Examples for Linking Remote Tables into OpenLink Virtuoso</h5>
<ul>
<li>
<a href="">Oracle</a>
</li>
<li>
<a href="">Progress</a>
</li>
<li>
<a href="">Ingres</a>
</li>
<li>
<a href="">IBM Informix</a>
</li>
<li>
<a href="">IBM DB2</a>
</li>
<li>
<a href="">Sybase</a>
</li>
<li>
<a href="">MySQL</a>
</li>
<li>
<a href="">PostgreSQL</a>
</li>
<li>
<a href="">JDBC</a>
</li>
<li>
<a href="">ODBC to ODBC</a>
</li>
<li>
<a href="">Firebird</a>
</li>
<li>
<a href="">Microsoft SQL Server</a>
</li>
</ul>
<br />
<br />
<a name="linkrmtprocauto" />
<h4>6.1.2.2. Linking Remote Procedures</h4>
<a name="ATTACHPROC" />
<h5>6.1.2.2.1. ATTACH PROCEDURE Statement</h5>
<div>
<pre class="programlisting">
ATTACH (PROCEDURE|FUNCTION) <proc_name> ([<parameter1>[,<parameter2>[...]]])
[ RETURNS <rettype> ] [AS <local_name>] FROM <dsn>
</pre>
</div>
<table class="varlist">
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">dsn:</td>
<td>
<p>scalar_exp</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">proc_name:</td>
<td>
<p>identifier | identifier.identifier | identifier.identifier.identifier | identifier..identifier</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">parameter1..parameterN:</td>
<td>
<p>parameters declaration (as in CREATE PROCEDURE)</p>
</td>
</tr>
<tr>
<td align="right" valign="top" class="varterm" nowrap="nowrap">local_name:</td>
<td>
<p>table</p>
</td>
</tr>
</table>
<p>The ATTACH PROCEDURE statement allows you to associate stored procedures
from remote datasources with Virtuoso so they can be used as if they were
defined directly within Virtuoso. Much like the ATTACH TABLE statement,
this SQL statement creates a local alias for a procedure on a given remote
data source so it can be considered locally defined. When this alias is called
called the procedure at the remote data source will actually be called.</p>
<p>Procedure generated result sets are not supported by the ATTACH PROCEDURE
statement. The only portable way to return values from a remote procedure is
to use INOUT or OUT parameters. Remote procedure result sets can be used by
combination of <span class="computeroutput">rexecute()</span> and Virtuoso PL, but this is
left for the user to implement as required.</p>
<p>The ATTACH PROCEDURE statement is not able to define new connections to
remote data sources, the connection should be defined prior using either
vd_remote_data_source or by attaching a table or view using the ATTACH TABLE
statement with USER/PASSWORD supplied.</p>
<p>Note that when generating pass-through statements to a given remote,
any procedure call for an attached procedure is passed through if the current
DSN is the same as the remote procedure's DSN.</p>
<p>The <strong>proc_name</strong> is the designation of the
procedure's name on the remote data source, DSN. The remote procedure
name supplied should always be fully qualified to avoid ambiguity, it may
consist of an optional qualifier/catalog, optional owner and finally procedure
name, separated by dots. This must identify exactly one procedure on the
remote data source.</p>
<p>The optional <strong>local_name</strong> is an optionally qualified
procedure name which will identify the procedure on the local Virtuoso
database. If the local_name is not given it defaults to the <current
qualifier>.<dsn>.<proc name on dsn>. The <dsn> will be
the data source name in upper case and with all non-alphanumeric characters
replaced by underscores. The <proc name on dsn> will be the exact
name as it is on the remote dsn, case unmodified.</p>
<p>If a dsn is not previously defined with vd_remote_data_source or ATTACH
TABLE, the ATTACH PROCEDURE will fail.</p>
<a name="VDOCS-TRANS-01" />
<div class="example">
<div class="exampletitle">Example:</div>
<p>
On remote Virtuoso (DSN name : remote_virt):
</p>
<div>
<pre class="programlisting">
CREATE PROCEDURE RPROC (IN PARAM VARCHAR) returns VARCHAR
{
return (ucase (PARAM));
}
</pre>
</div>
<p>
On the local virtuoso (DSN name : local_virt) :
</p>
<div>
<pre class="programlisting">
vd_remote_data_source ('remote_virt', '', 'demo', 'demopwd');
ATTACH PROCEDURE RPROC (IN PARAM VARCHAR) RETURNS VARCHAR from 'remote_virt';
</pre>
</div>
<p>
will result in creation of an procedure alias for RPROC in local_virt named DB.REMOTE_VIRT.RPROC
</p>
<p>
Calling it from the local_virt (using ISQL)
</p>
<div>
<pre class="programlisting">
select REMOTE_VIRT.RPROC ('MiXeD CaSe') as rproc_result;
rproc_result
---------------
MIXED CASE
1 rows
</pre>
</div>
</div>
<br />
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>The Virtuoso Visual Server Administration Interface provides a
graphical user interface for
<a href="htmlconductorbar.html#remoteprocedures">linking remote stored procedures</a>.</p>
</div>
<br />
<a name="TYPEMAPPING" />
<h4>6.1.2.3. Data Type Mappings</h4>
<p>
If a statement is passed through to a remote data source, the types
returned by SQLDescribeCol are taken directly from the remote prepare
and converted to the closest Virtuoso supported type.
</p>
<p>
If a statement involves both local and remote resources all types are
taken from the Virtuoso copy of the data dictionary.
</p>
<p>
In executing remote selects Virtuoso binds output columns according
to the type and precision given by SQLDescribeCol after preparing the
remote statement.
</p>
<p>
When a table is attached from a remote data source the catalog is read and
the equivalent entries are created in Virtuoso. Since the types present on different
DBMS's vary, the following logic is used to map ODBC types to Virtuoso types.
</p>
<table class="data">
<caption>Table: 6.1.2.3.1. Attach Table Type Mappings</caption>
<tr>
<th class="data">SQL Type</th>
<th class="data">Mapped Type</th>
</tr>
<tr>
<td class="data">SQL_CHAR</td>
<td class="data">varchar (precision)</td>
</tr>
<tr>
<td class="data">SQL_VARCHAR</td>
<td class="data">varchar (precision)</td>
</tr>
<tr>
<td class="data">SQL_BIT</td>
<td class="data">smallint </td>
</tr>
<tr>
<td class="data">SQL_TINYINT</td>
<td class="data">smallint</td>
</tr>
<tr>
<td class="data">SQL_SMALLINT</td>
<td class="data">smallint </td>
</tr>
<tr>
<td class="data">SQL_INTEGER</td>
<td class="data">integer </td>
</tr>
<tr>
<td class="data">SQL_BIGINT</td>
<td class="data">decimal (20)</td>
</tr>
<tr>
<td class="data">SQL_DECIMAL</td>
<td class="data" />
</tr>
<tr>
<td class="data">SQL_NUMERIC</td>
<td class="data">
<p>smallint if precision < 5 and scale = 0</p>
<p>integer if precision < 10 and scale = 0</p>
<p>double precision if precision < 16</p>
<p>decimal (precision, scale) otherwise</p>
</td>
</tr>
<tr>
<td class="data">SQL_REAL</td>
<td class="data">real</td>
</tr>
<tr>
<td class="data">SQL_FLOAT</td>
<td class="data">double precision</td>
</tr>
<tr>
<td class="data">SQL_DOUBLE</td>
<td class="data">double precision</td>
</tr>
<tr>
<td class="data">SQL_BINARY</td>
<td class="data">varbinary (precision)</td>
</tr>
<tr>
<td class="data">SQL_VARBINARY</td>
<td class="data">varbinary (precision)</td>
</tr>
<tr>
<td class="data">SQL_LONGVARCHAR</td>
<td class="data">long varchar</td>
</tr>
<tr>
<td class="data">SQL_LONGVARBINARY</td>
<td class="data">long varbinary</td>
</tr>
<tr>
<td class="data">SQL_DATE</td>
<td class="data">date </td>
</tr>
<tr>
<td class="data">SQL_TIME</td>
<td class="data">time</td>
</tr>
<tr>
<td class="data">SQL_TIMESTAMP</td>
<td class="data">datetime</td>
</tr>
</table>
<br />
<div class="note">
<div class="notetitle">Note:</div>
<p>The general case of decimal and numeric both revert to the Virtuoso
decimal type, which is a variable precision decimal floating point.</p>
</div>
<br />
<a name="TRANSMODEL" />
<h4>6.1.2.4. Transaction Model</h4>
<p>
One transaction on the Virtuoso VDBMS server may contain operations
on multiple remote data sources. As a general rule remote connections
are in manual commit mode and Virtuoso either commits or rolls back the
transaction on each of the remote connections as the main transaction
terminates.
</p>
<p>
ODBC does not support two phase commit. Therefore a transaction that
succeeds on one remote party may fail on another.
</p>
<p>
A transaction involving local tables and tables on one remote data source
will always complete correctly since the remote is committed before the
local and the local is rolled back if the remote commit fails.
</p>
<p>
Note that even though the client to Virtuoso connection may be in
autocommit mode the continuing connections will typically not be
autocommitting.
</p>
<p>
A remote connection is in autocommit mode if the Virtuoso connection
is and the statement is passed through unmodified. In all other cases
remote connections are in manual commit mode.
</p>
<p>Virtuoso supports 2PC - Two Phase Commit. See the
<a href="twopcimplementation.html">Distributed Transaction & Two Phase
Commit</a> section for more information.</p>
<br />
<a name="VDB_SQLFUNCTIONS" />
<h4>6.1.2.5. Virtual Database and SQL Functions</h4>
<p>
Different DBMS's support slightly different sets of SQL built-in functions
with slightly differing names. For example, what is called substring on Virtuoso
is called substr on Informix. Virtuoso allows declaring equivalences between local
user-defined or built-in functions and user defined or built-in functions on remote servers.
Knowing that a function exists on a remote database allows passing processing closer to the data,
resulting in evident gains in performance.
</p>
<p>
To declare that substring is called substr on DSN inf10, you can execute:
</p>
<div>
<pre class="programlisting">
db..vd_pass_through_function ('inf10', 'substring', 'substr');
</pre>
</div>
<p>
The first argument is the name of the remote database, as used with attach table and related statements.
If user defined functions with qualified names are involved, the names should be qualified in
the vd_pass_through_function call also. If many qualified or unqualified forms of the name are in use,
one should declare the mapping for them all.
</p>
<p>
To verify whether this declaration takes effect, one can use explain to see the query execution plan, for example:
</p>
<div>
<pre class="programlisting">
explain ('select substring (str, 1, 2) from inf10.sample_table');
</pre>
</div>
<p>
The declarations are persistent and can be dropped by using a last argument of NULL
for a given function. The declarations are kept at the level of a DSN and not at the level
of the type of DBMS because different instances can have different user defined functions defined.
</p>
<br />
<a name="VDB_SQLOPTSTATISTICS" />
<h4>6.1.2.6. Virtual Database and SQL Optimizer Statistics</h4>
<p>
If a query can be executed in its entirety on a single remote database, then optimizing this query
is exclusively the business of the remote database, as it gets the whole query text.
Virtuoso rewrites some things and suggests a join order but these are not binding on the remote database.
</p>
<p>
If a query involves tables from multiple remote databases or a a mix of local and remote tables,
knowing basic SQL statistics on the tables is essential for producing a meaningful query plan.
Virtuoso has information on indices existing on remote tables and if the remote table is attached
from a known type of DBMS, Virtuoso may read the DBMS specific statistics at attach time.
</p>
<p>
Note that the statistics of the remote database should be up to date before attaching.
</p>
<p>
The function sys_db_stat can be used for forcing a refresh of Virtuoso's statistics on remote tables.
</p>
<div>
<pre class="programlisting">
sys_db_stat (5, 0)
</pre>
</div>
<p>
will go through all tables, local and remote. For local tables, it will update statistics with a 5 percent
sampling rate and for remote tables it will refresh the statistics if the type of the host DBMS is among
the supported ones. If the remote DBMS is of an unknown type, Virtuoso will take the count of the remote
table and select the 1000 first rows to get a sample of data lengths and column cardinalities.
This is not very precise but will be better than nothing.
</p>
<p>
In order to force a full read of a remote table for statistics gathering, one can use
</p>
<div>
<pre class="programlisting">
db..sys_stat_analyze ('fully qualified table name', 0, 0);
</pre>
</div>
<p>
The table name is case sensitive, with all qualifiers, as it appears in SYS_KEYS and other system tables.
This will read the whole table.
</p>
<p>
Statistics on local as well as remote tables are kept in SYS_COL_STAT.
One may look at this table to see the effects of remote statistics collection.
In special cases, if a special effect is desired or the information is not otherwise available,
as in the case of a very large table on an unsupported type of server, it is possible to manually
update the contents of the table. Shutting down and restarting Virtuoso will force a reload of the statistics information.
</p>
<p>
Presently Oracle, Informix and Virtuoso are supported for direct access to the remote database's
statistics system tables. It is possible to define hook functions for accessing this same information
from any other type of DBMS. Please contact support for instructions on this.
</p>
<br />
<a name="VDB_DISTRQUERYOPTM" />
<h4>6.1.2.7. Distributed Query Optimization</h4>
<p>
When a query contains mixes of tables from different sources, the compiler must decide on an efficient
execution plan that minimizes the number of round trips to remote servers and evaluates query conditions
close to the data when possible. Additionally, any normal query optimization considerations such as choice
of join order and join type apply. See the section on SQL optimization and optimizer hints for more on this.
Additionally, the SQL optimizer uses round trip time statistics for the servers involved in the query.
</p>
<p>
In the following examples, we use the tables r1..t1, r2..t1 and t1, of which r1..t1 is on a server close by,
r2..t1 on a server farther away and t1 on the local server. The column row_no is a unique key and the string1
column is in indexed with 300 distinct values, the column fs1 has 3 distinct values. The tables all have 100000
rows. A round trip to r1 takes 10 ms and a round trip to r2 takes 100 ms.
</p>
<p>
Consider
</p>
<div>
<pre class="programlisting">
select * from r1..t1 a, t1 b where a.row_no = b.row_no and a.fs1 = 'value1';
</pre>
</div>
<p>
The compiler notices that 33333 rows will be selected from r1..t1 based on fs1. It will decide to read these
into a hash table, causing one linear scan of r1..t1 with relatively few round trips. Then it will read t1
locally and select the rows for which there is a matching entry in the hash. This is slightly faster
than doing 33333 random lookups of t1. If fewer rows were selected from r1..t1, the compiler would do a
loop join with the local t1 as the inner loop.
</p>
<p>
The absolute worst plan would be a loop join with t1 as the outer loop, with 100000 round trips to r1.
</p>
<p>
Now, if many tables are accessed from the same data source, the compiler tries to bundle these together into one statement. Thus, for:
</p>
<div>
<pre class="programlisting">
select * from r1..t1 a, r1..t1 b, t1 c where c.string1 = '111' and b.row_no = c.row_no and a.row_no = b.row_no + 1;
</pre>
</div>
<p>
The compiler will probably do the outer loop for t1, which is expected to select 100000/300 rows. Then it will do a
round trip to r1 with the statement.
</p>
<div>
<pre class="programlisting">
select * from t1 a, t1 b where a.row_no = b.row_no + 1 and a.row_no = ?.
</pre>
</div>
<p>
This is likely better than doing the remote part as an outer loop, bringing all the approx 100000 results in.
333 round trips selecting 1 row is better than 100000 rows transferred. If the data source were further away,
this could be otherwise, hence the importance of the round trip time statistic.
</p>
<p>
In distributed queries, the compiler will honor the option (order) and the join types e.hg. table option *(hash)
insofar the tables are local.
</p>
<p>
Thus, if we wrote
</p>
<div>
<pre class="programlisting">
select * from r1..t1 a, t1 b, r1..t1 c where c.string1 = '111' and b.row_no = c.row_no and a.row_no = b.row_no + 1 option (order);
</pre>
</div>
<p>
the compiler could not merge the two tablesfrom r1 into a single query because the order were given and there is
an intervening table not located on r1.
</p>
<br />
<a name="VDB_ARRAYPARAMETERS" />
<h4>6.1.2.8. Use of Array Parameters</h4>
<p>
ODBC and other data access API's usually offer a mechanism for executing a single parametrized
statement multiple times with a single client-server round trip. This is usually called support of array parameters.
</p>
<p>
Virtuoso can make use of array parameter support in ODBC drivers when available. Consider the statement:
</p>
<div>
<pre class="programlisting">
insert into r1..t1 select * from t1;
</pre>
</div>
<p>
Without array parameters, this would make a round trip to r1 for each row inn t1. With array parameters enabled,
with a batch size of 100, this would make only 1000 round trips for 100000 rows, resulting in dramatic increase
in performance. Typically, if the remote server is on the same machine, array parameters make such batch operations
about 3x faster. If the remote is farther away, the gain is greater.
</p>
<p>
Array parameters are used if the remote database and its ODBC driver support them. The feature is globally disabled
in the default configuration because some ODBC drivers falsely claim to support array parameters. To enable this
feature, the the ArrayOptimization entry in the [VDB] section of the ini file to 1. To set the batch size, use the
NumArrayParameters setting. 100 is a reasonable value.
</p>
<p>
Some ODBC drivers also support array parameters for select statements. To enable using this, you can set the
ArrayOptimization setting to 2. This may however not work with some drivers even if DML (insert/update/delete)
statements do work with array parameters.
</p>
<br />
<a name="TIMESTAMP_AUTOINCREMENT" />
<h4>6.1.2.9. Timestamps & Autoincrement</h4>
<p>
A transaction timestamp is not the same across the transaction
if the transaction has branches in different databases.
</p>
<p>
The data type and precision of a time stamp will also vary between
different types of databases.
</p>
<p>
Hence timestamp columns coming from tables on different servers are not
comparable for equality.
</p>
<p>
In inserts and updates of remote tables timestamps are assigned by
the database where the table in question is physically located.
</p>
<p>
Identity or autoincrement columns are likewise handled by the database holding the
remote table.
</p>
<p>
Note that MS SQL Server and Virtuoso describe a timestamp column as a binary column in ODBC
catalog and meta data calls. Thus remote SQL Server or Virtuoso timestamps will not appear as timestamps at all.
</p>
<p>
In the case of a Virtuoso remote database the binary timestamp can be cast into a DATETIME data type
and it will appear as a meaningful datetime.
</p>
<br />
<a name="vdbSTOREDPROCS" />
<h4>6.1.2.10. VDB Stored Procedures & Functions</h4>
<p>These procedures allow you to manually manage remote data sources and their tables.</p>
<ul>
<li>
<a href="fn_vd_remote_data_source.html">vd_remote_data_source()</a>
</li>
<li>
<a href="fn_vd_remote_table.html">vd_remote_table()</a>
</li>
<li>
<a href="fn_rexecute.html">rexecute()</a>
</li>
<li>
<a href="fn_rnext.html">rnext()</a>
</li>
<li>
<a href="fn_rmoreresults.html">rmoreresults()</a>
</li>
<li>
<a href="fn_rclose.html">rclose()</a>
</li>
<li>
<a href="fn_rstmtexec.html">rstmtexec()</a>
</li>
</ul>
<p>Functions capable of returning a result-set make use of the
<span class="computeroutput">results_set</span> parameter. To prevent them from returning
a result-set, the <span class="computeroutput">results_set</span> parameter should be set to
'null'. If Virtuoso finds an awaiting parameter to contain <span class="computeroutput">results_set</span>
it will fetch the result set regardless of <span class="computeroutput">cursor_handle</span>
parameter.</p>
<p>Unless explicitly granted, only the DBA group is permitted to use the
<span class="computeroutput">rexecute()</span> to maintain security. Caution is required
here since any user granted use of <span class="computeroutput">rexecute()</span> has
full control of the remote data source set-up by the DBA, however limited
to the overall abilities of the remote user on the remote data source.
Users can be granted and denied access to this function using the following
commands:</p>
<div>
<pre class="programlisting">
GRANT REXECUTE ON '<attached_dsn_name>' TO <user_name>
REVOKE REXECUTE ON '<attached_dsn_name>' FROM <user_name>
</pre>
</div>
<p>The following remote catalogue functions help you to obtain information about the
remote datasources that you are using. These could be especially useful in Virtuoso PL
later on if you are not able to know everything about the remote tables ahead of time for
the ATTACH TABLE statement</p>
<ul>
<li>
<a href="fn_sql_data_sources.html">sql_data_sources()</a>
</li>
<li>
<a href="fn_sql_tables.html">sql_tables()</a>
</li>
<li>
<a href="fn_sql_columns.html">sql_columns()</a>
</li>
<li>
<a href="fn_sql_statistics.html">sql_statistics()</a>
</li>
<li>
<a href="fn_sql_primary_keys.html">sql_primary_keys()</a>
</li>
</ul>
<br />
<a name="MANSETRDS" />
<h4>6.1.2.11. Manually Setting Up A Remote Data Source</h4>
<p>
Defining a remote table involves declaring the table as a local table
and then defining the data source if not already defined and associating
the new table with the remote data source.
</p>
<p>
The data source on which a table resides is declared at the table
level. This has no connection to the table's qualifier.
</p>
<p>
Assume a remote ODBC data source named test containing a table xyz
declared as follows:
</p>
<a name="VDOCS-TRANS-01" />
<div class="example">
<div class="exampletitle">Example:</div>
<div>
<pre class="programlisting">
CREATE TABLE XYZ (
A INTEGER,
B INTEGER,
C INTEGER,
PRIMARY KEY (A));
</pre>
</div>
</div>
<p>
To defined this as a remote table on the data source Virtuoso, first
define the table locally, using the above CREATE TABLE statement above.
</p>
<p>
Then define the data source:
</p>
<div>
<pre class="programlisting">
DB..vd_remote_data_source ('test', '', 'sa','');
</pre>
</div>
<p>
And the table:
</p>
<div>
<pre class="programlisting">
DB..vd_remote_table ('test', 'DB.DBA.XYZ', 'master.dbo.XYZ');
</pre>
</div>
<p>
This assumes that the remote data source has a login 'sa' with an empty
password and no special connection string options. The table names in
vd_remote_table have to be fully qualified. We here assume that the
Virtuoso table was created by DBA in under the default qualifier DB and
the remote XYZ was created by dbo in master.
</p>
<p>
The vd_remote_table declaration does not affect statements or procedures
compiled prior to the declaration.
</p>
<p>
Additional indices of remote tables may optionally be defined. They do
not affect the operation of the SQL compiler. The remote data source
makes the choice of index based on the order by clause in the statement
passed through.
</p>
<br />
<a name="BUGSlimits" />
<h4>6.1.2.12. Caveats</h4>
<ul>
<li>
<p>
Never attempt to attach a local table as a remote. The server will hang if it tries
to make a remote commit on itself.
</p>
</li>
<li>
<p>
If the schema of the remote table is changed it will need to be re-attached to Virtuoso.
</p>
</li>
<li>
<p>
The Virtuoso server treats dots (.) in the double-quotes escaped names as name element separators.
For example : the table name "a.b.c" is treated as "a"."b"."c" .
Because of this remote tables with dots in their table name (like tables from MS Text driver) require the dot
inside the table name to be replaced with the VDB "non-delimiting-dot" (\x0A) and the
vd_attach_table (in dsn varchar, in remote_name varchar, in local_name varchar, in uid varchar, in pwd varchar)
to be used instead of ATTACH TABLE statement.
</p>
<p>
The statement ATTACH TABLE "datafile.txt" as 'test' from 'text' user 'a' password 'b' should become :
</p>
<div>
<pre class="programlisting">
vd_attach_table ('text', 'datafile\x0Atxt', 'test', 'a', 'b');
</pre>
</div>
</li>
</ul>
<p>When Virtuoso interacts with a table or view attached from a remote
data source, it must be able to uniquely identify each row of the query.
At the attach time Virtuoso will query remote data source for the tables
primary keys and indices. These will be used to construct a copy of the
table definition in Virtuoso which is then used in reference to the remote
data source. At query time this information is used as much as possible.
This information may need to be supplemented by calls to SQLStatistics() for
further indicies or primary key information, as a last resort Virtuoso will
use SQLColAttribute() to determine which columns are SQL_DESC_SEARCHABLE.
</p>
<br />
<br />
<a name="usermodel" />
<h3>6.1.3. Virtuoso User Model</h3>
<p>The Virtuoso User Model is designed to support:</p>
<ul>
<li>Use of an external server for password and user account base maintenance
for example an LDAP server, another database server etc. This allows user accounts to be
verified against security information stored in some centralized repository,
allowing integration into existing security infrastructure .</li>
<li>Single namespace for users and groups for SQL and web service access.
In this way the local security info is stored in one place and allows
enabling SQL account to work as a web account or a web account as an SQL
account. This enforces unique names for users and roles in the database.</li>
<li>Extensibility of user information retrieval and checking.</li>
<li>Extensibility of user account and group attributes.</li>
</ul>
<p>There is a set of functions for administering the users and groups
(roles) of a Virtuoso database. All the user administration functions are restricted to
members of the dba group only.</p>
<div class="note">
<div class="notetitle">Note:</div>
<p>The terms 'role' and 'group' are identical in their usage in this document.
The terms security object or grantee refer collectively to users and
groups.</p>
</div>
<p>User manipulation functions do not generally return values. In case of
error an error condition will be signaled.</p>
<p>Users and roles are in the same namespace and are identified by name.
One name will not both designate a user and a group. A user or group may be
SQL enabled. A name that identifies a role (group) does not have a password
and is not a valid login name.</p>
<p>A user may belong to multiple roles. A role may inherit multiple other
roles. One role may be inherited or directly granted to a security object
multiple times. The order of granting is not relevant. The effective
privileges of a user are those granted to public, those granted to the user
plus those granted to a direct or indirect role of the user. The relationship
between users and roles is many to many. A role can be granted multiple other
roles as well as direct permissions. Cycles in granting roles are not allowed,
an error will be signaled if an operation would result in a cycle in the
role inheritance graph.</p>
<p>When SQL procedures or queries are executing, the effective privileges
are those granted to the owner of the procedure or view. A top level
dynamic SQL statement executes with the effective privileges of the logged in
user. If a SQL statement executes as a result of an HTTP request, the
applicable virtual directory specifies on behalf of which SQL account the SQL is
executed. A role cannot be the owner of procedures, views or other executable
SQL entities.</p>
<a name="vumsecobjects" />
<h4>6.1.3.1. Security Objects Management</h4>
<p>The following functions allow for creation and deletion of security objects
and roles, and for assigning or removing roles from security objects:</p>
<ul>
<li>
<a href="fn_USER_CREATE.html">USER_CREATE()</a>
</li>
<li>
<a href="fn_USER_ROLE_CREATE.html">USER_ROLE_CREATE()</a>
</li>
<li>
<a href="fn_USER_DROP.html">USER_DROP()</a>
</li>
<li>
<a href="fn_USER_ROLE_DROP.html">USER_ROLE_DROP()</a>
</li>
<li>
<a href="fn_USER_CHANGE_PASSWORD.html">USER_CHANGE_PASSWORD()</a>
</li>
<li>
<a href="fn_USER_SET_QUALIFIER.html">USER_SET_QUALIFIER()</a>
</li>
<li>
<a href="fn_USER_GRANT_ROLE.html">USER_GRANT_ROLE()</a>
</li>
<li>
<a href="fn_USER_REVOKE_ROLE.html">USER_REVOKE_ROLE()</a>
</li>
<li>
<a href="fn_USER_SET_OPTION.html">USER_SET_OPTIONS()</a>
</li>
<li>
<a href="fn_USER_GET_OPTION.html">USER_GET_OPTIONS()</a>
</li>
</ul>
<p>The security objects and roles data are contained in the system tables
described in the <a href="systemtables.html#UserSysTables">User System Tables</a> Section
of the Appendix</p>
<br />
<a name="vumuseroptions" />
<h4>6.1.3.2. User Options</h4>
<ul>
<li>
<div class="formalpara">
<strong>PASSWORD_MODE</strong>
<p>Function for checking a given password on SQL or DAV login. See below.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PASSWORD_MODE_DATA</strong>
<p>Application specific data for the Password Mode hook.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>LOGIN_QUALIFIER</strong>
<p>Default qualifier for SQL session.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>SQL_ENABLE</strong>
<p>If set SQL login is granted.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DAV_ENABLE</strong>
<p>If set the user account can be user for web authentication.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>DISABLED</strong>
<p>If set the user account is locked and cannot be used to
login as SQL or Web user (depends of SQL_ENABLE and DAV_ENABLE flags).
If the account in question is SQL enabled the DBA group can switch
the execution identity to it (see set_user_id () function).
This is useful when we need an account to execute Web pages (VSP/VSPX)
with some execution permissions but we do not
want to allow it to login via SQL/ODBC.
</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PRIMARY_GROUP</strong>
<p>This is the primary group of the user. This is no different from
other group memberships.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>GET_PASSWORD</strong>
<p>Function that will retrieve the password. If not defined the password
is assumed to be in the SYS_USERS table. This allows for custom encrypted
storage of passwords etc. This is simpler to use than the check hook.
Note that for security configurations where the server never does know the
passwords of user accounts, no digest based authentication schemes can be used,
including the HTTP digest authentication, since the digests cannot be computed
and checked without knowing the password. Possible users of this feature
are DBEV_LOGIN or HTTP login hooks.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>E-MAIL</strong>
<p>informative: e-mail of that user.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>FULL_NAME</strong>
<p>informative: full name of the user.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>HOME</strong>
<p>WebDAV home directory of the account, it is meaningful only if the
account is web enabled.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>PERMISSIONS</strong>
<p>WebDAV default permissions for new WebDAV objects created by the user.
This is only meaningful when web access is enabled.</p>
</div>
</li>
</ul>
<p>The functions for setting/getting these options will accept any
other named values, the above list only being those reserved for Virtuoso so far.</p>
<br />
<a name="vumloginexthook" />
<h4>6.1.3.3. Login Extension PL Hook</h4>
<p>
<span class="computeroutput">DB.DBA.USER_FIND
(<span class="paramdef">in <span class="parameter">name</span> varchar</span>);
</span>
</p>
<p>This is a user-defined PL function hook which, if it exists, will be
executed before doing the SQL/ODBC login. In this hook the user can find a
user account from some other server and register it in the local database. Or,
this can be used to perform some pre-login actions. It is similar to the
DBEV_LOGIN, but it does not change any account validation rule, it is
purely for pre-processing.</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>The <a href="hooks.html">Database Event Hooks</a> chapter.</p>
</div>
<a name="vumplhooksxmpls" />
<h5>6.1.3.3.1. PL Hooks Examples</h5>
<a name="vumqryldap" />
<div class="example">
<div class="exampletitle">Querying an LDAP Server</div>
<div>
<pre class="programlisting">
create procedure DB.DBA.LDAP_SEARCH (inout user_name varchar, in digest varchar)
{
whenever sqlstate '28000' goto ldap_validation_failure;
if (lcase(user_name) <> 'dba'))
{
ldap_search('foo.bar.com', 0, 'o=organization', '(cn=a*)',
sprintf('uid=%s,ou=users,o=organization', user_name), pwd_magic_calc(user_name,digest,1));
user_name := 'dba';
return 1; -- force validation as dba
}
else
{
-- bypassing ldap authentication for dba, let validation occur normally
return -1;
}
ldap_validation_failure:
return -1; -- try to validate normally
}</pre>
</div>
<div>
<pre class="programlisting">
create procedure DB.DBA.DBEV_LOGIN (inout user_name varchar, in digest varchar, in session_random varchar)
{
declare get_pwd varchar;
get_pwd := user_get_option (user_name, 'PASSWORD_MODE');
if (get_pwd is not null)
{
declare rc integer;
rc := call (get_pwd) (user_name, digest);
return rc;
}
return -1;
};</pre>
</div>
<div>
<pre class="programlisting">
user_create ('test_ldap', 'secret', vector ('PASSWORD_MODE', 'DB.DBA.LDAP_SEARCH'));
</pre>
</div>
</div>
<a name="vumuserfindhook" />
<div class="example">
<div class="exampletitle">USER_FIND PL Hook Example</div>
<div>
<pre class="programlisting">
create table
MY_DBA_USERS (M_NAME varchar primary key, M_PASSWORD varchar);</pre>
</div>
<div>
<pre class="programlisting">
create procedure
DB.DBA.USER_FIND (in name varchar)
{
-- do nothing for existing users
if (exists (select 1 from SYS_USERS where U_NAME = name))
return;
-- if there is in custom table
if (exists (select 1 from MY_DBA_USERS where M_NAME = name))
{
declare pwd varchar;
-- get password
select M_PASSWORD into pwd from from MY_DBA_USERS where M_NAME = name;
-- create a new SQL user based on external data
USER_CREATE (name, pwd, NULL);
}
};</pre>
</div>
</div>
<br />
<br />
<a name="vumrolesemantics" />
<h4>6.1.3.4. SQL Role Semantics</h4>
<p>The terms user group and role are used interchangeably. Roles can be
nested. There is a many to many relationship between users and roles. There
is likewise a similar, acyclic many to many relationship between roles. Each
role has a component role list of its granted (parent) roles, recursively,
no cycles allowed.</p>
<p>All role grants are listed in the roles system table whether they be
explicitly granted or only as a result of granting a group with groups
granted to it. The role grant graph has an explicit edge for each
role membership, direct or otherwise. The GI_DIRECT flag is true if
the grant is direct. Only direct role grants can be revoked.</p>
<div class="tip">
<div class="tiptitle">See Also:</div>
<p>The role system table description can be found in the appendix
under <a href="systemtables.html#UserSysTables">System Tables</a>.</p>
</div>
<p>The following SQL statements deal with roles. To create a new
role (group) object the following statement can be used:</p>
<div>
<pre class="programlisting">CREATE ROLE <NAME></pre>
</div>
<p>The <NAME> is a name of role to be created. It must be unique
in space of all security objects.</p>
<a name="vumcreaterole" />
<div class="example">
<div class="exampletitle">Creating a security role</div>
<div>
<pre class="programlisting">SQL> create role admins;</pre>
</div>
</div>
<p>Use the following statement to remove an existing role from the
security schema. </p>
<div>
<pre class="programlisting">DROP ROLE <NAME></pre>
</div>
<a name="vumrmrole" />
<div class="example">
<div class="exampletitle">Removing a security role</div>
<div>
<pre class="programlisting">SQL> drop role admins;</pre>
</div>
</div>
<p>The GRANT and REVOKE statements are used for controlling role membership as follows:
To assign a new
group, or list of groups (<ROLE>,...) to user <USER> use:</p>
<div>
<pre class="programlisting">GRANT <ROLE> [, <ROLE>] TO <USER> [WITH ADMIN OPTION];</pre>
</div>
<p>If the admin option is specified, the grantee can grant this same
privilege further to other grantees.</p>
<p>Roles can be revoked using:</p>
<div>
<pre class="programlisting">REVOKE <ROLE> [, <ROLE>] FROM <USER>;</pre>
</div>
<a name="vumgranting" />
<div class="example">
<div class="exampletitle">Granting & revoking security roles</div>
<div>
<pre class="programlisting">SQL> grant admins, users to demo;</pre>
</div>
<div>
<pre class="programlisting">SQL> revoke admins from demo;</pre>
</div>
</div>
<p>Only the dba group accounts may administer roles.</p>
<p>The dba group is not physically a role. When an empty database is created, it will have the dba account with full privileges. To grant these same full privileges to another user, the dba uses the grant all privileges to <grantee>. statement. This
will give the grantee full dba privileges, or in other words, add to the dba group. This may be reversed with the revoke all privileges from <grantee> statement.
</p>
<p>The GRANT statement accepts any valid SQL security object in the TO clause.
One cannot log in or perform any operations as a role. Roles are exclusively
shorthand for specific sets of permissions which are changed together and
are needed for multiple users. </p>
<br />
<br />
<a name="vaddistr" />
<h3>6.1.4. VAD - Virtuoso Application Distribution</h3>
<p>VAD provides a package distribution framework for installation, management,
dependency checking and un-installation of Virtuoso applications. A VAD package
contains all required Virtuoso components, which would constitute an application or
hosted solution, within a single distributable file. A VAD package cannot contain any
system parts independent of Virtuoso thus excluding operating system executables,
shared objects, installers or settings.</p>
<p>Virtuoso and VAD provide the following abilities: </p>
<ul>
<li>List all installed VAD packages.</li>
<li>List all completed operations over VAD packages.</li>
<li>Dependency checking - Check preconditions for installation of a package.</li>
<li>Install a VAD package.</li>
<li>Verification of installed package, compare to distribution state with ability
to build of list of locally changed parts of the package.</li>
<li>Check preconditions for un-installation of a package.</li>
<li>Uninstall a VAD package.</li>
</ul>
<a name="vadsummary" />
<h4>6.1.4.1. Summary of VAD Operations</h4>
<p>
The following is what the dba needs to know about VAD packages.</p>
<p>A VAD package is installed from a file with the db..vad_install SQL
function. The first argument is the file path, which must be in a
location that the server process can open, i.e. it is in the DirsAllowed list in the virtuoso.ini file. The second argument is 0, meaning that we are installing from a file.</p>
<div>
<pre class="programlisting">
SQL> vad_install ('conductor_dav.vad', 0);
</pre>
</div>
<p>is an example. If the package installation fails, the server exits
and will have to be restarted. No effects of a failed installation
will remain in the database after restart. Contact the supplier of the
VAD package for further instructions.</p>
<p>To know what is installed, do:</p>
<div>
<pre class="programlisting">
SQL> vad_list_packages ();
</pre>
</div>
<p>VAD package installations are not recorded in the transaction log.
Thus, if there is a backup followed by archived transaction logs
produced if CheckpointAuditTrail is on in virtuoso.ini, the VAD install
must be performed before replaying any logs that were made after the
VAD installation. The package installation must be just in the right
place in the replay sequence. In practice it is simplest to make an incremental backup after installing and packages, see backup_online () or the section on backing up.</p>
<p>For any further information, including how to make VAD packages, see the rest of this chapter.
</p>
<br />
<a name="vadpackcomposition" />
<h4>6.1.4.2. VAD Package Composition</h4>
<p>A VAD package has no developer tie-ins; it is built in a development environment
from source code that can be managed and versioned in the developers system of preference.</p>
<p>The VAD package is described by an XML structure called the 'VAD Sticker'. The
VAD sticker describes items to deploy, procedures to execute at install and uninstall time
and all conditions to be checked to ensure correct installation. The VAD Sticker consists
of the following:</p>
<ul>
<li>VAD package meta data
<ul>
<li>Names of package, developer, copyright holder etc. </li>
<li>Version number of package, build date, build number, build type (e.g.
sort of optimization performed). </li>
<li>Dependency information: minimal/maximal allowed version numbers of
Virtuoso server and depending VAD packages. Every required package may
include hint-text that may help the administrator determine (a) why the dependent
package is required, and (b) how to obtain the required package.</li>
<li>Information regarding known conflicts between packages; conflicting
package names and version number, with optional troubleshooting hints.</li>
<li>Ability to uninstall, a flag and list of reasons why it may be impossible
to uninstall the package.</li>
<li>Custom configuration data to be placed in the VAD Registry </li>
</ul>
</li>
<li>Locations of SQL files containing main and installation code:
<ul>
<li>Pre-install code, used to check application-specific installation preconditions.</li>
<li>Application specific table and view definitions.</li>
<li>Application specific stored procedure and trigger definitions. </li>
<li>Post-install (initialization) code, such as initial contents of tables.</li>
<li>Pre-uninstall code, used to check that it is safe to uninstall a package. </li>
<li>Post-uninstall code, used for removal of cached resources unusable or
meaningless without the package.</li>
</ul>
</li>
<li>Locations of Resources:
<ul>
<li>Documentation files. </li>
<li>Samples data for demonstration or package sanity check.</li>
<li>VSP/VSPX pages, related graphics, Java scripts, stylesheets, other
web content. </li>
<li>XML docs, XSLT sheets, DTDs and Schemas. </li>
</ul>
</li>
</ul>
<br />
<a name="vadpackversion" />
<h4>6.1.4.3. Package Versioning</h4>
<p>All required packages should be listed in the VAD sticker. Known conflict
may be listed in either of the conflicting VAD packages stickers, hence VAD stickers
of all installed packages should be checked.</p>
<p>Later versions of a package may be installed replacing earlier versions
of the same package. This however can be prohibited by listing either version
(or limit) as a known conflict in either VAD package sticker in the usual way.
Furthermore, it is possible to prevent re-installation of a package by stating that it
conflicts with itself. This provides some security against exploits involving attempts
to upgrade, downgrade or re-install a package, in the hope that the administrator
may corrupt the existing installation by installing new packages and working through
installing their dependencies.</p>
<p>Packages may differ in language and encoding of documentation and resource
files, even though the version number remains the same. If a package is sensitive to
internationalization issues, the developer should either assign different names to
various localizations of the package, or divide it into kernel package for any
language-independent parts and set of language-specific packages, with some
dependency between them.</p>
<br />
<a name="vadprocessres" />
<h4>6.1.4.4. Processing of Resources</h4>
<p>During creation of a VAD package, the "location" mentioned above may be
name of a file in file system or URI or DAV path. Upon package-time, URIs will be
resolved and resources under them will be copied into the package. The resulting
sticker will thus contain the location of resource within the package, the resource
itself, and the target location.</p>
<p>All SQL files have a specific order of loading. Tables, views etc. must be
defined before being referenced.</p>
<br />
<a name="vadunsupportfeat" />
<h4>6.1.4.5. Unsupported Features of VAD</h4>
<p>The VAD specification explicitly does not define the following:</p>
<div class="formalpara">
<strong>Method of development or environment</strong>
<p>There are no specific restrictions for the schema or Virtuoso/PL code of the
package. The VAD system does not make assumptions on the method of
software development.</p>
</div>
<div class="formalpara">
<strong>Method of source code control or versioning</strong>
<p>Version numbers used in the sticker have nothing common with tag labels
in a developers versioning system. Procedures edited directly within the
database using a web interface or CASE tools should be exported to a file for
inclusion in a VAD package. If the application developer uses some script to
export such code, this script is not usually part of sticker or the resulting package.</p>
</div>
<div class="formalpara">
<strong>Shipping/Deployment the VAD package from vendor to user</strong>
<p>VAD provides no methods for downloading dependent packages, or check
for package updates etc.</p>
</div>
<div class="formalpara">
<strong>Concurrent running of multiple versions of the same VAD
package on a single server</strong>
<p>There can be no guarantee that pre- or post-installation checks will
provide valid results if more than one VAD is being processed at the same
time. VAD does however guarantee that a package installation will be
either entirely successful or entirely rolled back.</p>
</div>
<div class="formalpara">
<strong>Installation or maintenance of non-Virtuoso hosted components</strong>
<p>Unlike Virtuoso-based packages, these components are usually operating
system specific, they may require some complex tuning, and their usage from
within Virtuoso applications may even require changes in virtuoso.ini configuration
file. VAD packages may contain test calls in pre-installation SQL procedures
to check that required external executables are available and provide
the functionality required.</p>
</div>
<div class="formalpara">
<strong>Data migration</strong>
<p>Some installations may require several days to complete migration/conversion
of stored data. Whilst it may be possible to provide a restricted service
during such time, VAD contains no tools to simplify such a process, this is
left to the administrator or developer. VAD completes its work right after
the execution of the post-installation code.</p>
</div>
<div class="formalpara">
<strong>Synchronous installation of a package on all hosts of a distributed system or cluster</strong>
<p>VAD has no standardized metadata regarding replication issues, hence
package-specific code may be required. Similarly, if a cluster uses
"round-robin" or a "director" loading management system and the server should
be stopped for VAD installation, the administrator should explicitly inform the
cluster manager about this event.</p>
</div>
<br />
<a name="vadsecurity" />
<h4>6.1.4.6. Security</h4>
<p>Since VAD packages are run by an administrator as the database DBA user,
care must be taken to ensure the package comes from a safe source. Any new package
installed may violate the security regulations of the target database and may even
inflict damage to files under the web-root of the Virtuoso Server or in directories
specified in the "DirsAllowed" parameter of the virtuoso.ini. If the
virtuoso.ini parameter "AllowOsCalls" is enabled then the installation procedures
of the package may call operating executables. It is the responsibility of the
database administrator to control this via the "AllowOsCalls", "SafeExecutables"
and "DbaExecutables" parameters of the virtuoso.ini.</p>
<p>VAD packages do not offer any automatic protection against unauthorized
modifications. Although ever VAD package contains a checksum, its purpose
is to guard against data transfer errors, it may not be sufficient to detect
unwanted modification.</p>
<br />
<a name="vadbuildingvadpacks" />
<h4>6.1.4.7. Building VAD Packages</h4>
<p>Initially, the VAD sticker and resources may reside in the file system,
DAV directory and or other locations available through the
<span class="computeroutput">DB.DBA.HTTP_URI_GET()</span> function.</p>
<p>The VAD creation operation parses the VAD sticker's XML description
and constructs the VAD file by calling <a href="fn_vad_pack.html">DB.DBA.VAD_PACK</a>.</p>
<p>This function reads the VAD sticker identified by the <span class="computeroutput">sticker_uri</span> which
contains the <span class="computeroutput">vad:package</span> root element.
Then the resources identified in the
sticker are retrieved. All resource URIs are interpreted in the context of the
<span class="computeroutput">base_uri_of_resources</span> and are parsed and checked
to be syntactically correct. Resources are appended to generated package that will
be stored at the <span class="computeroutput">package_uri</span>. <a href="fn_vad_pack.html">DB.DBA.VAD_PACK</a>
returns a human readable log of error and warning messages, it will signal
errors if any resource or database objects are unavailable at build time.</p>
<p>By convention, VAD package files have the extension '.vad'.</p>
<br />
<a name="vadutils" />
<h4>6.1.4.8. VAD Utilities</h4>
<p>An optional VAD package named VADutils provides various tools for capturing
changes made in the database after some point in time. The result of a capture consists of:</p>
<ul>
<li>Database object additions whose names match given patterns (e.g.
all tables and procedures within a particular catalog/qualifier).</li>
<li>Resource additions under particular locations.</li>
<li>Post-install local customizations of selected packages.</li>
</ul>
<p>The capture results may be useful for the following purposes:</p>
<ul>
<li>Archival of changes for replaying later.</li>
<li>Creating a special package of the changes for applying against a
fresh installation of the package. </li>
<li>Creating a new complete package containing both the original and
changes that will be included in the package sticker.</li>
</ul>
<p>These mechanisms provide good support for centralized development and custom
deployment methodology. If a site is localized to contain local links, graphics,
custom layout and such, then VAD capabilities offer help to the developer to
define the specific overlay of customizations over another VAD package. When
the underlying VAD package is updated the local customizations will be overwritten.
Being saved in a VAD package, customizations can be reapplied over the
updated base package.</p>
<br />
<a name="vadadminrspnslts" />
<h4>6.1.4.9. VAD Administrator Responsibilities</h4>
<p>VAD package installation, upgrade and uninstallation requires a
temporary break of service. The package checks may be performed on the
fly if it can be guaranteed that the resources being inspected will not be
altered by any users. The package check is a read-only process and operates
solely within the VAD Registry using read-only functions.</p>
<p>All VAD operations are logged in the server event log. All completed
operations are reflected in the <span class="computeroutput">DB.DBA.VAD_HISTORY</span>
system table.</p>
<p>The optional VADutils package provides some additional administrative tools,
mostly for troubleshooting. These include special installation and de-installation
functions that can ignore error signals, and provide an interactive editor for
the VAD Registry etc.</p>
<p>All operations described below require DBA access to the database.</p>
<p>Check if a VAD package may be installed by calling <a href="fn_vad_check_installability.html">DB.DBA.VAD_CHECK_INSTALLABILITY</a>.</p>
<p>Checks the presence and correct versions of required packages and of the
Virtuoso platform. It does not executes any pre-install Virtuoso/PL code from the
package, so there's no guarantee that installation will be successful if the
check found no error. If <span class="computeroutput">package_uri</span> is DAV
path, <span class="computeroutput">is_dav=1</span>, else <span class="computeroutput">is_dav=0</span>.</p>
<p>Perform VAD Package Installation by calling <a href="fn_vad_install.html">DB.DBA.VAD_INSTALL</a>.</p>
<p>If <span class="computeroutput">package_uri</span> is DAV path,
<span class="computeroutput">is_dav=1</span>, else <span class="computeroutput">is_dav=0</span>.</p>
<p>The administrator performs the following operations when installing:</p>
<ul>
<li>Invoke the install procedure from the web user interface or interactive SQL. This will perform the following:
<ul>
<li>Install documentation files.</li>
<li>Check for version and prerequisite package compatibility.</li>
<li>Disconnect SQL users and terminate web processing.</li>
<li>Make a database checkpoint.</li>
<li>Run the pre-install SQL script. </li>
<li>Load SQL code in the VAD package, in the order specified by the developer.</li>
<li>Copy web resources (VSP, VSPX, XSLT, etc.) into their designated places in WebDAV or file system Web root.</li>
<li>Run any post-install SQL code.</li>
</ul>
</li>
<li>If the installation was successful, the server will come back on-line. </li>
<li>If the installation was unsuccessful, e.g., mid-install failure due
to running out of disk space, or some other serious unrecoverable database error,
the Virtuoso server will exit. The administrator should consult the Virtuoso log
file to see what caused the failure. The installation can be completely undone
manually by halting the server (if not already stopped), and removing the
transaction log file (.trx). Upon Virtuoso restart, the server will continue from
the last checkpoint, made prior to install, as if the installation never took place.</li>
</ul>
<p>The return value of the <a href="fn_vad_install.html">DB.DBA.VAD_INSTALL()</a> function is
usually a sum of messages from pre- and post-installation procedures of the
package. It should normally contain at least the following:</p>
<ul>
<li>any errors and/or warnings encountered.</li>
<li>created users and catalogs/qualifiers </li>
<li>root VSP page for accessing the application, if applicable.</li>
<li>path to installed documentation files.</li>
<li>performance optimization hints.</li>
</ul>
<p>The VAD packages should be tested to install on an empty Virtuoso database,
after any required VAD packages. Installing a package on an empty server is
useful for determining that no other procedures or components were missed.
Since the application would normally run on the development machine where
the VAD package was built, it can be easy to overlook some components.
The completeness of the source archive of the application and its independence
from any ad hoc SQL objects is important, this is the only way the package can
be reliably versioned, tracked or uninstalled.</p>
<p>Check if a VAD package may be uninstalled by calling <a href="fn_vad_check_uninstallability.html">DB.DBA.VAD_CHECK_UNINSTALLABILITY</a>
</p>
<p>Performs a preliminary read-only checks to see whether the package given can
be uninstalled. This does not execute any pre-uninstall Virtuoso/PL code from within the
package at this stage. Hence, the success of this function does not guarantee
that uninstallation will be successful.</p>
<p>Perform VAD Package Uninstallation by calling <a href="fn_vad_uninstall.html">DB.DBA.VAD_UNINSTALL</a>.</p>
<p>The administrator will perform the following operations for the uninstallation process:</p>
<ul>
<li>Invoke the uninstall procedure from the web user interface or
interactive SQL. This will initiate the following:
<ul>
<li>Check that no other packages are using the package to be uninstalled. </li>
<li>disconnect SQL users and terminate web processing.</li>
<li>Make a database checkpoint.</li>
<li>Run the pre-uninstall SQL script.</li>
<li>Remove web resources installed by the package (all VSP , VSPX,
XSLT, etc files) in WebDAV or the filesystem under the web root.</li>
<li>Drop all SQL procedures and data. This is performed in reverse order to the install. </li>
<li>Run any post-uninstall SQL code.</li>
<li>Remove documentation files explicitly marked as removable. Usually
documentation would not be deleted as part of package uninstallation in case it is
needed e.g. if a set of documents is distributed as VAD package) </li>
</ul>
</li>
<li>If uninstallation was successful the server will come back on-line. </li>
<li>If uninstallation was unsuccessful, the server will exit. Uninstallation could
fail due to lack of disk space or some other serious unrecoverable database error.
The failed uninstallation attempt can be manually reversed by halting the
server (if not already) and deleting the transaction log file (.trx). Upon
server restart Virtuoso will continue from the last checkpoint, made prior to
uninstallation, as if the uninstallation was never attempted. The
administrator should consult the log file for clues to the failure. </li>
</ul>
<p>Check the state of VAD package installation by calling <a href="fn_vad_check.html">DB.DBA.VAD_CHECK</a>.</p>
<p>This checks to see if the elements of the package are as they are defined
in the original distribution. A list of differing elements is returned. Differences
revealed may not indicate a corruption, such changes could have been made
intentionally by another package, possibly a later version or upgrade that added
some columns to tables, and some resources may be customized by the user
post-installation.</p>
<p>This will check for the prior existence of tables, views etc owned by
other applications that are not compatible with this application. Any such
schema objects found are listed, the installation will not continue. These
may be dropped by the DBA to help the installation to succeed. Some such
elements may not be part of some other package, hence no package uninstall
would be available leading the DBA to drop them with the appropriate
SQL commands.</p>
<br />
<a name="vadpackageoverlap" />
<h4>6.1.4.10. Package Overlap</h4>
<p>Each package contains full definitions of all tables and indices.
Upon installing the following outcomes can occur:</p>
<ul>
<li>If a table already exists with the same primary key as the new
definition, additional columns are added to the table. If the primary keys
differ, the installation automatically fails. Note that a pre-install SQL script can
be defined to explicitly alter tables if consecutive versions of an application
use different primary keys. </li>
<li>Existing indexes are left untouched. New indices are added as specified
in the package. If indices should be modified or dropped, the pre-install
script is a reasonable place for dropping these.</li>
</ul>
<p>Thus the same SQL schema can be loaded twice without ill effect.</p>
<p>The post install script should be used to populate tables and such.
Inserts should be executed using the insert soft statement so that attempts to
insert duplicate are silently ignored without causing the installation to fail. The
post install script can perform any application level data format changes.</p>
<p>Packages should define their own distinct catalog or qualifier. They
should not overwrite another package unless upgrading a prior version.
Sometimes a package will require the use of another package's tables. This
should be achieved via grants issued in a pre-install script. A schema
element such as a table, view or procedure will always have at most one
owner package even though it may be referenced or even modified with additional
columns by another package installed later. These elements will only be
dropped when the owner package is dropped. Tables created ad-hoc from
interactive SQL do not have any owner package.</p>
<br />
<a name="vadsticker" />
<h4>6.1.4.11. VAD Sticker</h4>
<p>The VAD Sticker contains meta-data and descriptions of resources contained,
or to be contained, within a VAD package. Like any XML documents, the target
VAD package sticker can be sourced from more than one source file, which can
aid maintenance and development.</p>
<a name="vadstickerdtd" />
<h5>6.1.4.11.1. VAD Sticker DTD</h5>
<p>The namespace vad, used below, represents the URI
<span class="computeroutput">http://www.openlinksw.com/urn/vad</span>.</p>
<p>The top level element of a VAD Sticker is <sticker>. It must contain a
<caption> element and may contain <dependencies>,
<procedures>, <ddls> and <resources> elements.</p>
<div>
<pre class="programlisting">
<!--<<top>>-->
<!ENTITY % vad.source_sticker "INCLUDE">
<!ENTITY % vad.package_sticker "IGNORE">
<!ENTITY % vad.ecm.group_content "(dependencies | procedures | ddls | resources | registry)" >
<![%vad.source_sticker;[
<!ENTITY % vad.ecm.sticker "(caption, (group | %vad.ecm.group_content;)*)">
<!ELEMENT group ((group | %vad.ecm.group_content;)*) >
]]>
<![%vad.package_sticker;[
<!ENTITY % vad.ecm.sticker "(caption, %vad.ecm.group_content;)">
]]>
<!ELEMENT sticker %vad.ecm.sticker; >
<!ATTLIST sticker
version NMTOKEN #REQUIRED
xml:lang CDATA #REQUIRED
>
<!--<</top>>-->
<!--<<caption>>-->
<!ELEMENT caption (name, version)>
<!ELEMENT name ((prop)*)>
<!ATTLIST name
package NMTOKEN #REQUIRED
>
<!ELEMENT version ((prop)*)>
<!ATTLIST version
package NMTOKEN #REQUIRED
>
<!ELEMENT prop EMPTY>
<!ATTLIST prop
name NMTOKEN #REQUIRED
value CDATA #REQUIRED
>
<!--<</caption>>-->
</pre>
</div>
<p>The caption contains one name and one version element. These elements
have a package attribute for keeping requisites used by VAD procedures.
Other prop-s are for keeping admin-readable info, but they will not affect the
installer's behavior. Typical names of properties here are Vendor, Copyright,
Release+Date, Build, Language, Encoding, but any (even non-unique) names
are acceptable.</p>
<p>Sticker's elements for dependencies</p>
<div>
<pre class="programlisting">
<!--<<dependencies>>-->
<!ELEMENT dependencies ((require | allow | conflict)*) >
<!ATTLIST dependencies>
<!ENTITY % vad.ecm.version_list "((version | versions_earlier | versions_later)*)">
<!ELEMENT require (name, %vad.ecm.version_list;) >
<!ELEMENT allow (name, %vad.ecm.version_list;) >
<!ELEMENT conflict (name, %vad.ecm.version_list;) >
<!ATTLIST require
group NMTOKEN #IMPLIED
>
<!ELEMENT versions_earlier ((prop)*)>
<!ATTLIST versions_earlier
package NMTOKEN #REQUIRED
>
<!ELEMENT versions_later ((prop)*)>
<!ATTLIST versions_later
package NMTOKEN #REQUIRED
>
<!--<</dependencies>>-->
</pre>
</div>
<p>Element dependencies contains an list of packages related to given one.
For every version or range of versions of every package, developer may specify
whether the given version is required for the package, or allowed but not
required, or will cause some sort of troubles.</p>
<p>More precisely, to find information about some particular version of a
package, the list of children of dependencies element will be scanned from top
to bottom. If the first matching record is in conflict group, not in require or
allow, then installation is impossible. From other side, there must be at least
one installed package for every require section.</p>
<p>Element require may be labeled with optional group attribute. As an
exception from common rule, there must be at least one installed package for
every group of require sections with identical name. E.g. If an installation of
package B requires either of two interchangeable packages A1 and A2, sticker
should contain a pair of nodes in the same group:</p>
<div>
<pre class="programlisting">
<require group="G">
<name package="A1">...</name>
</require>
</pre>
</div>
<p>...</p>
<div>
<pre class="programlisting">
<require group="G">
<name package="A2">...</name>
</require>
</pre>
</div>
<div class="note">
<div class="notetitle">Note:</div>
<p>There are no methods to specify that exactly one package, either A1 or
A2, should be installed. It must be done by placing proper conflict
descriptions in stickers of A1 and/or A2, but not in the sticker of B.</p>
</div>
<p>Sticker's elements for procedures</p>
<div>
<pre class="programlisting">
<!--<<procedures>>-->
<!ELEMENT procedures ((sql)*)>
<!ATTLIST procedures
uninstallation (supported | prohibited) #REQUIRED
>
<![%vad.source_sticker;[
<!ENTITY % vad.sql.include "include CDATA #IMPLIED">
]]>
<![%vad.package_sticker;[
<!ENTITY % vad.sql.include "">
]]>
<!ELEMENT sql (#PCDATA)>
<!ATTLIST sql
purpose (install-check | pre-install | post-install | uninstall-check | pre-uninstall | post-uninstall) #REQUIRED
%vad.sql.include;
>
<!--<</procedures>>-->
</pre>
</div>
<p>Element procedures contains an list of Virtuoso/PL fragments, and every fragment
is tagged by one of four values of the purpose attribute. At every stage of install
or uninstall VAD procedure, a whole list of procedures will be scanned from the
beginning to the end, and all procedures of appropriate sort will be executed in
the same order as they are listed. In source sticker files, include attribute may
be used to insert text of some external file instead of having SQL code written
inside the element.</p>
<p>Sticker's elements for ddls</p>
<div>
<pre class="programlisting">
<!--<<ddls>>-->
<!ELEMENT ddls ((sql)*)>
<!ATTLIST ddls
>
<!--<</ddls>>-->
</pre>
</div>
<p>Element ddls is very similar to procedures and contains an list of
Virtuoso/PL fragments to create schemas etc.</p>
<p>Sticker's elements for resources</p>
<div>
<pre class="programlisting">
<!--<<resources>>-->
<!ELEMENT resources ((file | location)*)>
<!ATTLIST resources >
<![%vad.source_sticker;[
<!ENTITY % vad.file.source_uri "source_uri CDATA #IMPLIED">
]]>
<![%vad.package_sticker;[
<!ENTITY % vad.file.source_uri "">
]]>
<!ELEMENT file EMPTY>
<!ATTLIST file
type (doc | http | dav | code | special) #REQUIRED
source (http) "http"
target_uri CDATA #REQUIRED
makepath (yes | no | abort) "abort"
overwrite (yes | no | abort | equal | expected) "equal"
package_id CDATA #IMPLIED
location IDREF #IMPLIED
dav_owner CDATA #IMPLIED
dav_grp CDATA #IMPLIED
dav_perm CDATA #IMPLIED
%vad.file.source_uri;
>
<!ELEMENT location ((prop)*) >
<!ATTLIST location
id ID #REQUIRED
default_target_uri CDATA #REQUIRED
>
<!--<</resources>>-->
</pre>
</div>
<p>Element resources lists all files to be copied onto target box. For every
file, source and target URIs should be specified, and suggested behavior for
cases when a directory should be created or file should be overwritten. Target
URI may be relative to one of roots: for documentation, web-resources, DAV,
SQL code (it's where virtuoso.ini is located) and one of special locations,
additionally specified by location elements. (Installer may query administrator
to allow changing of locations' roots; in such case, information from
location's properties will be shown to the administrator.) By default, the
value of <span class="computeroutput">package_id</span> is a space delimited list of
type, location ID (if any) and target URI.</p>
<ul>
<li>dav_owner - DAV owner for file (used if type="dav", ignored if "filesystem");</li>
<li>dav_grp - DAV group for file (used if type="dav", ignored if "filesystem");</li>
<li>dav_perm - DAV permissions for file (used if type="dav", ignored if "filesystem").</li>
</ul>
<a name="ex_vadstickfile" />
<div class="example">
<div class="exampletitle">VAD installable file descriptions</div>
<p>To install files into DAV:</p>
<div>
<pre class="programlisting">
<file type="dav" source="http" target_uri="yacutia/yacutia_style.css" dav_owner='dav' dav_grp='administrators' dav_perm='111101101N' makepath="yes"/>
<file type="dav" source="http" target_uri="yacutia/yacutia_vdir_style.css" dav_owner='dav' dav_grp='administrators' dav_perm='111101101N' makepath="yes"/>
</pre>
</div>
<p>To install files into file system:</p>
<div>
<pre class="programlisting">
<file type="http" source="http" target_uri="yacutia/yacutia_style.css" makepath="yes"/>
<file type="http" source="http" target_uri="yacutia/yacutia_vdir_style.css" makepath="yes"/>
</pre>
</div>
</div>
<p>Sticker's elements for registry</p>
<div>
<pre class="programlisting">
<!--<<registry>>-->
<!ELEMENT registry ((record)*)>
<!ATTLIST registry >
<!ELEMENT record ANY>
<!ATTLIST record
key CDATA #REQUIRED
type (STRING | INTEGER | KEY | URL | XML) #REQUIRED
overwrite (yes | no | abort | equal | expected) "equal"
>
<!--<</registry>>-->
</pre>
</div>
<p>Element registry lists all branches to be defined in the VAD Registry. Every
record element contain data of one record. The first children of record element
(either a text or an element) will be serialized and stored as a value of
<span class="computeroutput">DB.DBA.VAD_REGISTRY.R_VALUE</span> cell. To prevent errors,
it is recommended to keep comments to the data outside the record element:
being in the wrong place inside, they may be stored in the registry instead
of actually needed data.</p>
<a name="ex_vadstickers" />
<div class="example">
<div class="exampletitle">Sample Stickers</div>
<p>A package that contains only some commonly useful ("exported") functions,
one table for internal purposes, a small sample VSP application, and small
set of documentation files.</p>
<div>
<pre class="programlisting">
<?xml version="1.0" encoding="ASCII" ?>
<!DOCTYPE sticker SYSTEM "vad_sticker.dtd">
<sticker version="1.0.010505A" xml:lang="en-UK">
<!-- Name and version; common data about the package -->
<caption>
<name package="rdf_lib">
<prop name="Title" value="RDF Support Library" />
<prop name="Developer" value="OpenLink Software" />
<prop name="Copyright" value="(C) 2003 OpenLink Software" />
<prop name="Download" value="http://www.openlinksw.com/virtuoso/rdf_lib/download" />
<prop name="Download" value="http://www.openlinksw.co.uk/virtuoso/rdf_lib/download" />
</name>
<version package="3.14">
<prop name="Release+Date" value="2003-05-05" />
<prop name="Build" value="Release, optimized" />
</version>
</caption>
<!-- This package requires no other packages,
but it conflicts with package virtodp of versions
from 1.00 to 2.17, inclusive -->
<dependencies>
<allow>
<name package="virtodp"></name>
<versions_earlier package="1.00"></versions_earlier>
</allow>
<conflict>
<name package="virtodp">
<prop name="Title" value="Virtuoso ODP Sample" />
</name>
<versions_earlier package="2.17">
<prop name="Date" value="2001-01-26" />
<prop name="Comment"
value="An incompatible version of RDF library is included in some old versions of virtodp " />
</versions_earlier>
</conflict>
</dependencies>
<!-- There are no installation procedures, other than DDLs -->
<procedures uninstallation="supported"></procedures>
<!-- There are some procedures, which may be re-applying and (maybe) reverted automatically -->
<ddls>
<sql purpose="pre-install">
"DB"."DBA"."VAD_CREATE_TABLE" ('DB', 'DBA', 'RDF_SCHEDULED_IMPORTS',
'ID integer,
URI varchar,
CALLBACK varchar,
VERSION varchar,
REPORT long varchar,
primary key (ID)');
</sql>
<sql purpose="post-install">
"DB"."DBA"."VAD_LOAD_RESOURCE" ('rdf_functions');
</sql>
</ddls>
<!-- Resources include... -->
<resources>
<!-- ...documentation, ... -->
<file type="doc" target_uri="rdf_lib/1.1/intro.dxt" />
<file type="doc" target_uri="rdf_lib/1.1/interface.dxt" />
<file type="doc" target_uri="rdf_lib/1.1/implementation.dxt" />
<file type="doc" target_uri="rdf_lib/1.1/sample_app.dxt" />
<!-- ...the file of commonly-useful functions, ... -->
<file package_id="rdf_functions"
type="code" target_uri="rdf_lib/1.1/rdf_lib.sql" />
<!-- ...pages of the sample application, named rdf_edit, ... -->
<file type="http" target_uri="rdf_lib/rdf_edit/default.htm" />
<file type="http" target_uri="rdf_lib/rdf_edit/browse.vsp" />
<file type="http" target_uri="rdf_lib/rdf_edit/edit.vsp" />
<!-- ...a DAV resource with sample RDF data, ... -->
<file type="dav" target_uri="rdf_lib/sample_odp_structure.rdf" />
<!-- ...two files of sample application's functions. -->
<file type="code" target_uri="rdf_lib/1.1/rdf_edit/content_level.sql" />
<file type="code" target_uri="rdf_lib/1.1/rdf_edit/view_level.sql" />
</resources>
<!-- There are no application-specific registry items in this package -->
</sticker>
</pre>
</div>
</div>
<br />
<br />
<br />
<a name="backup" />
<h3>6.1.5. Data Backup & Recovery</h3>
<p>Administering a database involves taking backups and having a readiness
to recover from backups and subsequent transaction logs.</p>
<p>Backups can be taken in two principal ways:</p>
<ul>
<li>Using the Virtuoso backup function/procedures.</li>
<li>Copying the database files.</li>
</ul>
<p>The Virtuoso backup functions can be used from any client directly, such
as ISQL. It is possible, and perhaps preferable, to create stored procedures
for performing the backup functions and scheduling these with the Virtuoso
scheduler.</p>
<p>The actual database file(s) can be copied while the database is running
so long as no checkpoint is made during the copy process. Checkpointing can
be disabled for this, but make sure it is re-enabled after the backup has been
made.</p>
<p>Making a full backup of a large database can take several hours if not
days just due to the speed of tapes or local area networks. Full backups
must in all cases be done without an intervening checkpoint. This is why
frequent full backups are not desirable. To ensure the possibility of full
recovery one must have the complete set of transaction logs (audit trail log)
since the last backup.</p>
<p>Restarting the database after restoring backed up files will show the
state in effect since the last checkpoint preceding the backup. Any
transaction log files made after the point of backup can be replayed to bring
the state up to the last recorded transaction.</p>
<a name="BACKUP_AUDIT" />
<h4>6.1.5.1. Log Audit Trail</h4>
<p>Virtuoso can maintain a transaction audit trail. This is enabled using
the <a href="server.html#fp_checkpointAuditTrail">CheckpointAuditTrail</a> setting
in the virtuoso.ini file. When this setting is non-zero, Virtuoso will
begin a new transaction log after each checkpoint. Thus one automatically gets
a full, unbroken sequence of transaction logs for the entire age of the database.
These logs are named as specified in virtuoso.ini and are suffixed with their creation
time.</p>
<p>Transaction logs older than the log that was current at the time of the
last backup are superfluous for recovery, since their transactions were
checkpointed before the backup started. Transactions of the log current at
the time of the backup are NOT in the backed up state since they were not
checkpointed, i.e. written into the read-only section of the database containing the data being backed up.</p>
<p>We strongly advise having the CheckpointAuditTrail enabled in any
production environment.</p>
<p>It is good practice to have at least two generations of full backup,
since the last backup may contain errors that were not known at the time of
its making. If such precaution is taken then only transactions logs older than
the oldest backup are safe to remove. If we needed to recover from the oldest
backup for any reason we would require all audit transaction logs created during
and after that backup.</p>
<p>A Virtuoso database can be restored from the last full backup and
all Audit Trail transaction files created during and after the backup. You
would need to start the database as normal with the backup version of the
database file. Once the database has been started, connect using iSQL.
You can then use the <a href="fn_replay.html">replay()</a> function to
replay the transaction files up to the required point. It is vital that
these files are replayed in the correct order.</p>
<br />
<a name="onlinebackups" />
<h4>6.1.5.2. On-Line Backups</h4>
<a name="usingbackuponline" />
<h5>6.1.5.2.1. Backup Using Backup_Online()</h5>
<p>Virtuoso is capable of performing online backups so that normal database
operation does not have to be disrupted in order to take backups.
The <a href="fn_backup_online.html">backup_online()</a>
can be used to backup the database in the state effective at the last checkpoint
to a series of backup files.</p>
<p>
The database storage is divided into a checkpoint space that is a read
only image from the time of the last checkpoint and thus can be safely
backed up anytime between checkpoints and the commit space where
updates subsequent to the last checkpoint data are stored.
Additionally, the database records what pages have changed since the
last checkpoint every time new checkpoint is made. This change
tracking makes it possible to make incremental backups. The first
time the backup_online function is called, it saves a compressed copy
of the then current checkpoint state into one or more files. The next
time it is called, it will write the changes that have come into the
checkpoint space since the last time backup_online was called. It is
possible to erase the change tracking data with the
backup_context_clear function. The next call to backup_online will
then make a full backup. Files generated by one or more calls to
backup_online without intervening backup_context_clear form a series
with distinct serial numbers and will be restored together. In order
to restore such files, the administrator must delete the previous
database files and start the server with a special flag and indicate
the location of the backup files. This will bring the database to the
state corresponding to the state as of the checkpoint immediately
preceding the last call to backup_online, i.e. the one that wrote the
newest of the backup files being restored. To restore onwards from
this state, the administrator must replay transaction logs, starting
with the log that was current when the last call to backup_online was
made. In order to preserve all such logs, one must run with the
CheckpointAuditTrail ini parameter set to 1.
</p>
<p>A database checkpoint cannot be performed while an online backup
is in progress. </p>
<a name="ex_anonlinebackup" />
<div class="example">
<div class="exampletitle">Performing an Online Backup</div>
<div>
<pre class="programlisting">
SQL> backup_context_clear ();
SQL> checkpoint;
SQL> backup_online ('virt-inc_dump_#', 150);
</pre>
</div>
</div>
<p>The <span class="computeroutput">backup_online()</span> procedure differs from the
the CheckPointAuditTrail mainly because it can be started from any point
in the database. Unless CheckPointAuditTrail was enabled when the database
was created, the database file at a particular state and all transaction logs
created by the AuditTrail since that state would be required to restore
the database. Only the backup set files would be required to restore
from <span class="computeroutput">backup_online()</span>. The <span class="computeroutput">backup_online()</span>
also makes a compressed backup, making it far more suitable for large
databases.</p>
<p>The last optional parameter allows to point the directory(ies) where the
backup files must be stored. See <span class="computeroutput">backup_online()</span> description
for details. </p>
<br />
<a name="rstrfrmbckuponline" />
<h5>6.1.5.2.2. Restoring From an Online Backup Series</h5>
<p>To restore from a backup series the administrator must first shutdown
the Virtuoso database server and move all database files (e.g. virtuoso.db
and virtuoso.trx) out of the database directory. It is recommended that copies
by taken rather than deleting them entirely. Then the command:</p>
<div>
<pre class="programlisting">
<virtuoso exe> +restore-backup <FILE_PREFIX>
</pre>
</div>
<div>
<pre class="programlisting">
-- for example:
virtuoso-iodbc-t +restore-backup dump-20011010_#
</pre>
</div>
<p>must be issued in the directory where the "*.bp" were stored. The
database will then be restored. The expression <virtuoso exe> above must
be replaced with the path and filename to the Virtuoso server executable used
on your system (e.g. ..\virtuoso-odbc-t.exe).</p>
<p>Each file in the series has a header containing a unique identifier,
for the backup set and the sequence number of the file in the backup set . If
an identifier in any file in the backup sequence differs from the identifier
contained in the first file, the restoration process will stop and report an
error, which is written to the Virtuoso log file.</p>
<p>At times the backup or restoration commands may return errors. Use the
following list to help diagnose and resolve them:</p>
<ul>
<li>
<div class="formalpara">
<strong>Timestamp [%lx] is wrong in file %s</strong>
<p>The unique identifier in header of the file differs from the identifier
of the first file. It is possible that the file was renamed or corrupt or
belongs to another backup set.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Number of file %s differs from internal number [%ld]</strong>
<p>The sequence number of the file does not correspond to the internal sequence number of file.
This could be caused by the file being renamed or corrupt.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Prefix is wrong in file %s</strong>
<p>The prefix of the file does not correspond to internal information.
Possible reason: file was renamed or corrupt or belongs to another backup set.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Could not begin online-backup</strong>
<p>Read error. Possible reason: Virtuoso database file was corrupt.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Seek/Read failure on stripe %s/database</strong>
<p>Read error. Possible reason: Virtuoso database file was truncated or hardware error.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Read of page %ld failed</strong>
<p>Read error. Possible reason: Virtuoso database file was truncated or hardware error.</p>
</div>
</li>
<li>
<div class="formalpara">
<strong>Backup file writing error</strong>
<p>Write error. Possible reason: disk is malfunctioning or full.</p>
</div>
</li>
</ul>
<a name="ex_anonlinebackuprestore" />
<div class="example">
<div class="exampletitle">Restoring an Online Backup</div>
<p>Following the online backup example above:</p>
<div>
<pre class="programlisting">
SQL> backup_context_clear ();
SQL> checkpoint;
SQL> backup_online ('virt-inc_dump_#', 150);
</pre>
</div>
<p>The following command could be used to restore the database from the
backup files created:</p>
<div>
<pre class="programlisting">
virtuoso-iodbc-t -c <db-ini-file> +restore-backup virt-inc_dump_#
</pre>
</div>
<p>or:</p>
<div>
<pre class="programlisting">
virtuoso-odbc-t.exe -c <db-ini-file> +restore-backup virt-inc_dump_#
</pre>
</div>
</div>
<br />
<br />
<a name="otherbackupmethods" />
<h4>6.1.5.3. Other Backup Methods</h4>
<p>
A possible way of making a full backup of a large databases is first to
turn off any automatic checkpoints and make a compressed copy of the
files. After the back up is complete, checkpointing should be re-enabled. The
files should be compressed to make efficient use of space, and should be copied
to a disk separate from the location of the database, and preferably to an external
backup medium such as tape.
</p>
<a name="manualbackup" />
<h5>6.1.5.3.1. Manual Backup</h5>
<p>
For a large database it is best to turn off any automatic checkpoints and copy the
database files to external storage. Checkpoints should be turned off by
issuing the command:
</p>
<div>
<pre class="programlisting">
checkpoint_interval (-1);
</pre>
</div>
<p>
at the SQL prompt. Checkpoints can be re-enabled in a post-backup
script by:
</p>
<div>
<pre class="programlisting">
checkpoint_interval (<n>);
</pre>
</div>
<p>
which sets the automatic checkpoint interval to <n> minutes. The backup will
be unusable if there are checkpoints made while it is in progress. Thus it is
important to guarantee that checkpoints do not occur. The only safe way of
doing this is the above, since it is in principle possible to have a server crash during the
backup and a roll forward following restart, all while the backup is in progress. If this
happens the backup will be readable and consistent with the state of the last checkpoint
if and only if there are no checkpoints between its start and completion.
Setting the interval to -1 will guarantee that the server, when starting after recovery
will not make a checkpoint.
</p>
<p>
The dba must make sure that clients do not issue checkpoint
or shutdown statements while a backup is in progress.
</p>
<p>The presence or absence of checkpoints at a given point in time can be
ascertained from the virtuoso.log event log file.
</p>
<br />
<br />
<a name="offlinebackups" />
<h4>6.1.5.4. Off-Line Backups</h4>
<p>
When Virtuoso is not running a complete and clean backup can be taken by making
a copy of the database file and transaction file(s) created after the last
checkpoint.
</p>
<p>
To get an up to the minute copy of a running database one can copy
the database file and the associated log, i.e. the file specified in
TransactionFile in the database's configuration file. When started, the log
will roll forward and restore the database to the state following the
last logged transaction.
</p>
<br />
<a name="vdbrecovery" />
<h4>6.1.5.5. Database Recovery</h4>
<a name="dbrebuild" />
<h5>6.1.5.5.1. Rebuilding A Database</h5>
<p>
The process of rebuilding a database consists of dumping its contents
into a large log file, or log files, and doing a roll forward from an empty database
with that log.
</p>
<p>
The general steps to rebuild a database are:
</p>
<ul>
<li>
<p>Shut down the running server, making a checkpoint. This is done with
the SHUTDOWN command from interactive SQL.</p>
</li>
<li>
<p>Make sure there is a log file specified in virtuoso.ini.</p>
</li>
<li>
<p>Start the server process virtuoso with the -b command line
option: e.g. % ./virtuoso -b (+backup_dump)</p>
<p>This will write the contents of the database into the log file specified
in virtuoso.ini and exit when complete.</p>
</li>
<li>
<p>Take a backup of the old database file.</p>
</li>
<li>
<p>Change the DatabaseFile setting in virtuoso.ini to a
non-existing database file or delete the old database file.</p>
</li>
<li>
<p>Start the server with the +restore-crash-dump option. The option is essential.</p>
</li>
<li>
<p>The server will build a new database file from the log and
once completed it will, by default, perform a checkpoint of the transactions to the
database file and start listening at the specified port. Virtuoso can be
started without a checkpoint using the -n (+no-checkpoint) option.</p>
</li>
<li>
<p>You may then connect to the database with interactive SQL
and if necessary make a checkpoint. The checkpoint will write freeze the state
following roll forward and delete the log used for the rebuild.</p>
</li>
<li>
<p>The database is now ready for normal use.</p>
</li>
</ul>
<span class="important">
<strong>Important:</strong>
<p>It is recommended you take a backup copy of the database file(s)prior to this procedure.
</p>
</span>
<p>
It may sometimes be useful to rebuild a database as above to save space.
Virtuoso does not relinquish space in the DB file back to the file system as
records are removed, however, Virtuoso does reuse pages that are made
available from a deletion of records. The steps above will build a new
compact database file. You would ordinarily not have to worry about this.
</p>
<br />
<a name="diagnosingrepairing" />
<h5>6.1.5.5.2. Diagnosing and Recovering a Damaged Database File</h5>
<p>
It is possible to recover data from a damaged Virtuoso database by
a procedure similar to rebuilding a database as described above. A
database file may be corrupt if the database repeatedly crashes during
a specific operation.
</p>
<p>
To determine whether a database is corrupt, you may use the backup to a null file
command in isql, for Unix platforms:
</p>
<div>
<pre class="programlisting">
SQL> backup '/dev/null';
</pre>
</div>
<p>For windows platforms you can use:</p>
<div>
<pre class="programlisting">
SQL> backup 'NUL';
</pre>
</div>
<p>This command will read through the database checking
its integrity. If the server crashes before completing the backup process, then
the database is indeed corrupt and needs to be recovered. No other
activity should take place while the command is executing.
</p>
<p>
To recover the database, follow the procedure for rebuilding it, except
use the -D 'capital D' or +crash-dump switch instead of -b. This will construct a
log file which you can replay to make a new database. The database
will contain the transactions that were committed as of the last
successful checkpoint. If the database altogether fails to open it
may be the case that the schema is damaged.
</p>
<p>
It is possible that the database to be recovered is too large
to fit in a single log file. The crash dump feature therefore allows segmenting the
recovery log into a number of files. See the virtuoso.ini configuration file documentation
for details. It is possible to make a crash dump in several pieces if there is not enough
total disk space to hold the dump on the system where the database is running.
</p>
<p>
If the recovery log is split over several files it is necessary to set the transaction file in the ini to point to the first of these files, delete the database file(s) and start the server with the +restore-crash-dump option. When the server comes online,
one can connect to it with isql and use the replay () function for replaying the remaining logs, one by one, in their original order.
</p>
<p>
For example,assuming the virtuoso.ini fragment:
</p>
<div>
<pre class="programlisting">
Log1 = rec-1.log 100M
Log2 = rec-2.log 100M
</pre>
</div>
<p>
we would make the dump with
</p>
<div>
<pre class="programlisting">
virtuoso +crash-dump
</pre>
</div>
<p>
and once the server has been started with +restore-crash-dump, with the ini setting TransactionFile set to rec1.log,
replay the remaining log with the isql commands:
</p>
<div>
<pre class="programlisting">
SQL> replay ('rec-2.log');
Done.
SQL> checkpoint;
</pre>
</div>
<div class="note">
<div class="notetitle">Note:</div>
<p>If the recovery is interrupted it can be restarted at the last checkpoint
made during the recovery. Note that a mid recovery checkpoint may take a very long time, e.g.
1 hour for a 10GB database, since it is possible that the delta since the
previous recovery checkpoint comprise almost all the database.
</p>
</div>
<br />
<a name="Xcrashrecovery" />
<h5>6.1.5.5.3. Crash Recovery When The Normal Crash Recovery Fails</h5>
<p>
When the schema tables (e.g. DB.DBA.SYS_COLS, DB.DBA.SYS_KEYS) have corrupt rows
the normal crash dump/crash restore procedure will not be possible because the server
relies on the schema tables to know the key layouts for reading the
data rows of other tables upon crash dump.</p>
<p>In such situations there is a special procedure to be followed to save as much data as
possible from the corrupt database. The general steps are:</p>
<ul>
<li>dump the intact schema table rows, and read them into a fresh database</li>
<li>read the schema from the fresh database in the normal way</li>
<li>proceed with dumping the rest of the tables from the corrupt database</li>
</ul>
<p>Thus the transaction log produced from the corrupt database, when replayed
on the new database file (the one holding the schema tables data) makes the closest
approximation to the corrupt database's data. However, this will not produce
a workable database by itself - it may possibly deny inserting of data into tables
with IDENTITY columns and will lose all the data within the Virtuoso registry
(accessible from <span class="computeroutput">registry_get()</span>/<span class="computeroutput">registry_set()</span>
functions).</p>
<p>Because of the very nature of the crash-restore process described here and
because of the fact that data is lost in the database schema, the server will not
attempt to dump tables whose schema description is lost. So care should be
taken when reading the data from the database.</p>
<p>This restoration procedure in no way replaces the regular database backup
procedures, it merely tries to save whatever reasonable data there may be left
from the database file.</p>
<p>
The recovery sequence is as follows:
</p>
<ol>
<li>
<p>
Do a crash dump of the schema tables (using the
'<span class="computeroutput">+crash-dump +mode oa +dumpkeys schema</span>'
virtuoso command line options).
</p>
</li>
<li>
<p>
Create a new INI file to describe the layout of the new database you'll use to temporarily fill up the restored data.
</p>
</li>
<li>
<p>
Move the transaction log file(s) produced in step 1 to the location required by the new INI file.
</p>
</li>
<li>
<p>
Replay the transaction log from step 1 on an empty database using the new INI file.
You will now have the schema tables readable in the new database (and nothing else):</p>
<div>
<pre class="programlisting">
Virtuoso options : -c <your new ini file> +restore-crash-dump -f
</pre>
</div>
</li>
<li>
<p>
Make a crashdump of the data in the non-schema tables of the old database while
having read the schema tables from the new database: </p>
<div>
<pre class="programlisting">
+crash-dump -c <your new ini file> +crash-dump-data-ini <your old ini file> +mode o -f
</pre>
</div>
</li>
<li>
<p>
Move the transaction log file(s) produced in step 1 to the location required by the new INI file.
</p>
</li>
<li>
<p>
Replay the transaction log from the previous step into the new db file using: </p>
<div>
<pre class="programlisting">
-c <your new ini file> +restore-crash-dump
</pre>
</div>
</li>
<li>
<p>
Do a normal crash dump of the new database:</p>
<div>
<pre class="programlisting">
-c <your new ini file> +crash-dump
</pre>
</div>
</li>
<li>
<p>
Move away (backup) the original (old) database files and put the transaction log
produced by the above step into the location specified in the original INI file.
You can also delete the rest of the DB files of the new database at that point.</p>
</li>
<li>
<p>
Replay the transaction log to make the old database afresh.
</p>
</li>
</ol>
<p>
To automate the above procedure, a sample Unix script follows that automates it somewhat.
This script expects the crashed database virtuoso.db.save and an appropriate INI file
(no striping, no log segmentation, transaction log file name virtuoso.trx)
in the current directory and creates the restored database.
It also expects the virtuoso-iodbc-t executable to be in the operating
system path. Also, make sure that you have a
suitable <span class="computeroutput">virtuoso.lic</span> license file in the current directory.
</p>
<div>
<pre class="programlisting">
#!/bin/sh
rm -rf xmemdump.txt virtuoso.trx virtuoso.tdb virtuoso.log virtuoso.db virtuoso.lck core.* new.lck new.log new.trx new.tdb new.db new.ini
cp -f virtuoso.db.save virtuoso.db
cat virtuoso.ini | sed 's/virtuoso\./new./g' > new.ini
virtuoso-iodbc-t -f +crash-dump +mode oa +dumpkeys schema
ls -la *.trx
mv virtuoso.trx new.trx
virtuoso-iodbc-t -c new -f -R
virtuoso-iodbc-t -c new +crash-dump +crash-dump-data-ini virtuoso.ini +mode o -f
ls -la *.trx
mv virtuoso.trx new.trx
virtuoso-iodbc-t -c new -R -f
virtuoso-iodbc-t -c new +crash-dump -f
rm -f virtuoso.trx virtuoso.tdb virtuoso.log virtuoso.db virtuoso.lck
ls -la *.trx
mv new.trx virtuoso.trx
virtuoso-iodbc-t -R -f
</pre>
</div>
<br />
<a name="XversionRecovery" />
<h5>6.1.5.5.4. Crash Recovery Across Virtuoso VDBMS Server Versions</h5>
<p>
If the database was created with a version prior to the one being used for
rebuilding, the system tables may be different. The creation here refers to
the first time the database was made, a crash recovery does not count as a
fresh start here.
</p>
<p>
If this is or may be the case, the first log must be rolled forward into the
empty database BEFORE the new and possibly incompatible system tables are created.
This is done by setting the TransactionFile parameter to the first of the
recovery logs and starting the server with the -R or +restore-crash-dump switch. For
good practice one should also specify the no checkpoint switch, so that the log
will in no case be damaged after the initial step of the roll forward.
After this initial step the system tables will be compatible and the
dba can proceed to replay the remaining recovery logs with the replay function.
</p>
<br />
<a name="procindexrecov" />
<h5>6.1.5.5.5. Backup and Restore individual table(s) and individual index(s) on a new fresh db</h5>
<p>This section describes how to get a part of db tables and restore on a new db.</p>
<p>Additionally, part of the steps from below can be used to backup separate table and recover
on same db.</p>
<p>Note: Only effective with Virtuoso 6.0 and later.</p>
<p>Basic steps:</p>
<ol>
<li>Create a function for dumping the key:
<div>
<pre class="programlisting">
create procedure bkp_key (in f any, in tb_name varchar, in key_name varchar)
{
backup_prepare (f);
backup_index (tb_name, key_name);
backup_flush ();
backup_close ();
}
;
</pre>
</div>
</li>
<li>On the source db execute:
<div>
<pre class="programlisting">
bkp_key ('mylog.txn', 'DB.DBA.T1', 'T1');
</pre>
</div>
<p>This will dump in a "mylog.txn" file the T1 table's primary key.</p>
</li>
<li>On source db stop server and do:
<div>
<pre class="programlisting">
virtuoso +backup-dump +foreground +mode l ## ( lower case L )
</pre>
</div>
<p>This will dump the schema tables only into the trx file.</p>
</li>
<li>On the target db make sure there is no db file and place the trx file produced
by previous step. Execute:
<div>
<pre class="programlisting">
virtuoso +restore-crash-dump +foreground ;
</pre>
</div>
<p>This will create a new db with same db schema as on the source db.</p>
</li>
<li>Start the target and do:
<div>
<pre class="programlisting">
replay ('mylog.txn') ;
</pre>
</div>
<p>This will insert the PK data into the table from the source db dump.</p>
</li>
<li>If the table in question has other indexes must drop them and re-create them, since they
are empty as in previous step we have been restoring only the PK.
</li>
</ol>
<p>
<strong>Note</strong>: following the steps from above can be dumped each index and
then replay. Also the steps may be combined for multiple tables and keys in the backup procedure -
just needs to be added the corresponding calls to the backup_index() function.</p>
<br />
<br />
<br />
<a name="perfdiag" />
<h3>6.1.6. Performance diagnostics</h3>
<p>This section provides a checklist for improving performance of a Virtuoso server.</p>
<p>If something does not work fast, this is mostly for the following reasons:</p>
<ul>
<li>Not enough memory</li>
<li>Not the right indices, missing statistics</li>
<li>Too much locking or too many threads on the same data</li>
<li>Bad disk layout, for example not striped or not enough file descriptors.</li>
</ul>
<p>Determining which is the case is simple. The result set returned by status ('') has most
of the information. Do this twice with some 10 seconds between the samples and see the second
result set.
</p>
<a name="perfdiagmemory" />
<h4>6.1.6.1. Memory</h4>
<p>If there is not enough memory, there will be frequent disk access. This is
seen from the buffers and disk usage lines.
</p>
<p>The very simplest test for this is looking at the CPU % of the process in top.
If there is constant load and the percent is low then the server is IO bound.
</p>
<p>If all memory is not in use, then memory cannot be the problem. This
is seen from the buffers line. If the used number is under 80% of the total or if the
replace age is several times larger than the count of buffers, then things are OK. If
the replace age is 0 then no buffers have ever been replaced and all that ever was
read is still in memory.
</p>
<p>If the replace age is less than or close to the buffer count times 4, then cache
replacement is frequent and adding buffers is advised.
</p>
<p>Adding more than 60-70% of system ram as buffers is not useful. The setting is
NumberOfBuffers in the ini file, count 9K per buffer.
</p>
<p>The disk access is summarized on the disk usage line. First is the
number of reads since start then the average latency in N ms. If %r is high, then a lot
of the time between the previous status and this is taken by disk. This can be over 100.
One thread that is waiting for disk all the time counts for 100. If the percent is
high then adding more disks and striping over them will be useful. Even with a single database
file, adding file descriptors (FDsPerFile setting in the ini) may be useful. If the average
read latency is 0 or close, then the data is cached by the OS. If it is high, then adding disks and striping may reduce it.
</p>
<p>The read ahead line will tell if there are sequential reads. These are
faster than random ones and can efficiently use striped disks.
</p>
<p>If the workload is random access, then a high number of read ahead means
that one might not have the required indices, thereby causing full table scans.
More on this in the query plans section below.
</p>
<br />
<a name="perfdiagswapping" />
<h4>6.1.6.2. Swapping</h4>
<p>Swapping is always bad. If swapping occurs, then one has too many buffers and
should decrease the number of buffers. Use an OS tool like top to see the size of the
database process and its virtual memory use. Having a resident size smaller than logical
size is not always bad since some code or data in the process may simply be unused but having,
after running in a steady state, i.e. all buffers used, a resident size less than the amount
of memory allocated for buffers is always bad. Before steady state, i.e. during cache warm-up,
the resident size is normally less than the buffer pool's size.
</p>
<p>
To see the count of major page faults. i.e. ones that read the disk, do:
</p>
<div>
<pre class="programlisting">
getrusage ()[]4];
</pre>
</div>
<p>through interactive SQL. The result is the count of major faults since
starting the process. The count should not vary between samples, at least not more
than a few units per minute.
</p>
<p>This function is not available under Windows. Use the task manager instead for tracking this.</p>
<p>There can be an actual memory leak, specially with plugins or hosting. See the
growth of the virtual size throughout the run, reaching full buffer utilization and doing
a couple of checkpoints. Past this, the process should not grow. You may also see if the
ThreadCleanupInterval or ResourcesCleanupInterval ini parameters have an effect. If in
spite of all the process grows there can be a genuine leak. It is normal for the size to
fluctuate somewhat due to varying amounts of uncommitted data, threads, connections and the like.
</p>
<br />
<a name="perfdiaglocking" />
<h4>6.1.6.3. Locking</h4>
<p>The lock status section indicates the count of deadlocks and waits. If the 2r1w
number is high then it means that the application should read for update. It gets shared
locks and cannot change them to exclusive. Adding the for update option to selects in the
right place will fix this, also setting DefaultIsolation in the ini to 2 for read committed
will be good. If the lock wait count increases fast, then locking may downgrade performance.
The threads line below shows how many threads have some task to do. The waiting is the number
of these threads that are waiting for a lock at the time of the status. The vdb number is the
number of threads waiting for network io, either from virtual database or any web protocol.
</p>
<p>Take a few samples and if these show few threads waiting and the waits or deadlocks
counts do not climb much locking will not be a problem. If these numbers are high, see
the sys_l_stat view for details.
</p>
<div>
<pre class="programlisting">
select sum (waits), sum (wait_msecs) from sys_l_stat;
</pre>
</div>
<p>for the totals. The first number is the count of times a thread waited for a lock,
the second is the sum of the real times spent waiting.
</p>
<div>
<pre class="programlisting">
select top 10 * from sys_l_stat order by waits desc;
</pre>
</div>
<p>shows the keys on which most waits take place. See also the deadlocks and the wait_msecs
columns. Numbers are cumulative.
</p>
<p>For details on disk, see:</p>
<div>
<pre class="programlisting">
select top 10 * from sys_d_stat order by reads desc;
</pre>
</div>
<p>These views are explained in more detail in the performance meters section.</p>
<p>If there is a multi-user application doing random access for read or write, it
is an idea to partition the data so that they do not hit the same page all the time.
For example, to allow for parallel insert of orders without contention, the TPC C
schema prefixes the order number with a warehouse and district number, so that simultaneous inserts
will most often not hit the same page.
</p>
<p>Even when there is no locking, there is still some serialization for database page
access. This is shown in the various wait columns of SYS_L_STAT. If the sum of the waits for
a key is over 1% of the touches for the key, as given in SYS_D_STAT, there is contention and a
performance penalty of maybe 10%.
</p>
<p>Such things can be improved by altering the schema design and configuration
parameters do not usually help. If there is disk access, having more memory always helps
because then locks will be in effect for less time.
</p>
<br />
<a name="perfdiagqueryplans" />
<h4>6.1.6.4. Query Plans</h4>
<p>To know if there is a bad query plan, see the explain output of the query.</p>
<p>Unless a full table scan is intended, a full table scan is pretty much always bad.</p>
<p>In the explain () result set, a table access is marked by a From table name.
Below is mentioned the driving index, the conditions on the index and the conditions that are not indexable.
</p>
<p>Joins are listed one after the other, the outermost first. The join order can
be read from the explain output going from top to bottom.
</p>
<p>If a table is read on a non primary key and columns not covered by the driving index
are accessed, there is a second key mentioned with a full match of the primary key columns as condition.
</p>
<p>A full table scan looks like this:</p>
<div>
<pre class="programlisting">
explain (select count (*) from t1');
...
from DB.DBA.T1 by STR2 8.4e+04 rows
Key STR2 ASC ()
Local Code
0: $30 "count" := artm $30 "count" + <constant (1)>
...
</pre>
</div>
<p>There are no conditions mentioned.</p>
<p>A lookup with index looks like this.</p>
<div>
<pre class="programlisting">
explain ('select fi2 from t1 where row_no = 11');
...
from DB.DBA.T1 by T1 Unique
Key T1 ASC ($26 ".FI2")
inlined <col=1694 ROW_NO = <constant (11)>>
...
</pre>
</div>
<p>The condition is shown on the line below the key.</p>
<p>A lookup with full scan testing every row looks like this:</p>
<div>
<pre class="programlisting">
...
from DB.DBA.T1 by T1 2.5e+04 rows
Key T1 ASC ($26 ".FI2")
row specs: <col=1699 FI2 > <constant (11)>>
...
</pre>
</div>
<p>The condition is shown after the heading row specs. The whole key mentioned
in the key will be read and the entries tested. if both indexed and row tests exist,
the indexed are done first as one would think.
</p>
<p>If your query has full table scans, consider adding an index.</p>
<p>If the index choice is not the right one, consider the following possibilities:</p>
<ul>
<li>Run statistics. This is not always necessary because the database takes dynamic samples but it can help in cases.</li>
<li>specify the desired index explicitly with table option.</li>
<li>Use literal constants instead of parameters, specially if the query is long running and values of the columns being compared are not evenly distributed.</li>
<li>To exclude index usage for a column, write col + 0 =xx.</li>
</ul>
<p>Hash joins can make full table scans. This is OK if the table scanned is small.</p>
<p>A hash join looks like this:</p>
<div>
<pre class="programlisting">
explain ('select count (*) from t1 a, t1 b where a.row_no = b.row_no + 1');
...
{
Fork
{
Fork
{
from DB.DBA.T1 by STR2 8.4e+04 rows
Key STR2 ASC ($26 "B.ROW_NO")
Local Code
0: $30 "temp" := artm $26 "B.ROW_NO" + <constant (1)>
4: BReturn 0
Current of: <$28 "<DB.DBA.T1 B>" spec 5>
Sort (HASH) ($30 "temp") -> ($26 "B.ROW_NO")
}
from DB.DBA.T1 by STR2 8.4e+04 rows
Key STR2 ASC ($37 "A.ROW_NO")
Current of: <$39 "<DB.DBA.T1 A>" spec 5>
Hash source ($37 "A.ROW_NO") -> ($26 "B.ROW_NO")
After code:
0: $44 "count" := artm $44 "count" + <constant (1)>
4: BReturn 0
}
Select ($44 "count", <$39 "<DB.DBA.T1 A>" spec 5>, <$28 "<DB.DBA.T1 B>" spec 5>)
}
</pre>
</div>
<p>First t1 is read from start to end and a hash is filled with row_no + 1.
Then t1 is read from start to end a second time. The hash source is the hash lookup.
A hash join, if there is no index or if the whole table or a large part thereof is
traversed is better than a loop join because it replaces random access with sequential.
The complexity is close to O(n + n) instead of O (n * log n). Plus sequential read makes
better use of read ahead.
</p>
<p>The corresponding loop join looks like this:</p>
<div>
<pre class="programlisting">
explain ('select count (*) from t1 a, t1 b where a.row_no = b.row_no + 1 option (loop)');
{
Fork
{
from DB.DBA.T1 by STR2 8.4e+04 rows
Key STR2 ASC ($26 "B.ROW_NO")
Current of: <$28 "<DB.DBA.T1 B>" spec 5>
Precode:
0: $29 "temp" := artm $26 "B.ROW_NO" + <constant (1)>
4: BReturn 0
from DB.DBA.T1 by T1 Unique
Key T1 ASC ()
inlined <col=1694 ROW_NO = $29 "temp">
Local Code
0: $35 "count" := artm $35 "count" + <constant (1)>
4: BReturn 0
Current of: <$31 "<DB.DBA.T1 A>" spec 4>
}
Select ($35 "count", <$31 "<DB.DBA.T1 A>" spec 4>, <$28 "<DB.DBA.T1 B>" spec 5>)
}
</pre>
</div>
<p>To prevent hash joins, use table option or option at the end of the
select as seen above. A hash join is very bad if a whole table is read for filling a
hash and then only a small number of entries are fetched from the hash. However, if
there is no index, then even this is better than a loop join.
</p>
<br />
<a name="perfdiagcheckpointduration" />
<h4>6.1.6.5. Checkpoint Duration</h4>
<p>A checkpoint can take a very long time in certain special conditions. Normally a
log checkpoint is about a minute if flushing several G worth of buffers to disk. The flushing
is mostly done online, after which there is an atomic time of a few seconds. After this operation
resumes. Applications do not notice the checkpoint except as a temporary increase in response delays.
</p>
<p>If spikes in response time are to be avoided, then making frequent checkpoints is
better than making infrequent ones.
</p>
<p>If there are long running transactions with many locks or uncommitted changes, then
the checkpoint interval should anyhow be longer than several times the expected duration of
such a transaction. If this is not so, the checkpoint can fall in the middle of the
transaction and will internally have to rollback and again reestablish the uncommitted state
so as to write a clean image on the new checkpoint. Doing this many times in the life of a
transaction is very inefficient. In generally do not make long transactions with locking. Preferentially
use read committed for anything long.
</p>
<p>A checkpoint's atomic time can be prohibitively long under the following circumstances:</p>
<ul>
<li>There are transactions with a lot of locks and uncommitted state at the time of the checkpoint.
Specially a transaction which has updated pages paged out to disk is bad. In general one should
not do transactions that update a large part of the disk cache, let alone exceed its size. Use
the row autocommit mode (log_enable (2)) for doing batch updates. Batch updates most often do
not require isolation anyway. If they do, then break the job into smaller transactions.
</li>
<li>The MaxCheckpointRemap parameter in the ini is too small and the database exceeds
the size of disk cache either in the Virtuoso process or the OS. This may increase the count
of used pages by the value of this parameter but will make for faster checkpoints. Set this
to up to 25% of the database page count.
</li>
</ul>
<br />
<br />
<a name="ptune" />
<h3>6.1.7. Performance Tuning</h3>
<a name="IO" />
<h4>6.1.7.1. I/O</h4>
<a name="DISKIO" />
<h5>6.1.7.1.1. Optimizing Disk I/O</h5>
<p>
Virtuoso allows splitting a database over several files that
may be on different devices. By allocating database fragments onto
independent disks I/O performance in both random and sequential database
operations can be greatly enhanced.
</p>
<p>
The basic unit of a database is the segment. A segment consists of an
integer number of 8K pages. A segment may consist of one or more files called
stripes. If a segment has multiple stripes these will be of identical
length and the segment size will be an integer multiple of the stripe size.
</p>
<p>
The size limit on individual database files is platform dependent, but 64 bit file offsets are used where available.
For large databases use of multiple disks and segments is recommended for reasons of parallelism even though a single database file can get very large.
A database can in principle grow up to 32TB (32-bit page number with 8KB per page).
</p>
<p>
When a segment is striped each logically consecutive page resides in a different
file, thus for a segment of 2 stripes the first stripe will contain
all even numbered pages and the second all the odd numbered pages. The
stripes of a segment should always be located on independent disks.
</p>
<p>
In serving multiple clients that do random access to tables the server can
perform multiple disk operations in parallel, taking advantage of the independent
devices. Striping guarantees a statistically uniform access frequency to all
devices holding stripes of a segment.
</p>
<p>
The random access advantages of striping are available without any specific configuring
besides that of the stripes themselves.
</p>
<br />
<a name="IOQS" />
<h5>6.1.7.1.2. Configuring I/O queues</h5>
<p>
Striping is also useful for a single client doing long sequential
read operations. The server can detect the serial nature of an operation, for
example a count of all rows in a table and can intelligently prefetch rows.
</p>
<p>
If the table is located in a striped segment then the server will read all
relevant disks in parallel if these disks are allocated to different I/O queues.
</p>
<p>
All stripes of different segments on one device should form an I/O queue.
The idea is that the database stripes that benefit
from being sequentially read form a separate queue. All queues are then read and written
independently, each on a different thread. This means that a thread
will be allocated per queue. If no queues are declared all database files, even if located
on different disks share one queue.
</p>
<p>
A queue is declared in the striping section by specifying a stripe
id after the path of the file, separated by an equal sign.
</p>
<div>
<pre class="programlisting">
[Striping]
Segment1 = 200M, disk1/db-seg1-1.db = iq1, disk2/db-seg1-2.db = iq2
Segment2 = 200M, disk1/db-seg2-1.db = iq1, (disk2/db-seg2-2.db = iq2
</pre>
</div>
<p>
In the above example the first stripes of the segments form one queue
and the second stripes form another. This makes sense because now all database files
on /disk1 are in iq1 and all on /disk2 are on iq2.
</p>
<p>
This configuration could have resulted from originally planning a 200 MB database
split on 2 disks and subsequently expanding that by another 200 MB.
</p>
<p>
The I/O queue identifier can be an arbitrary string. As many background I/O threads
will be made as there are distinct I/O queues.
</p>
<p>
Striping and using I/O queues can multiply sequential read rates by a factor almost
equal to the number of independent disks. On the other hand assigning stripes on one disk
to different queues can have a very detrimental effect. The rule is that all that is physically
accessed sequentially will be on the same queue.
</p>
<br />
<br />
<a name="SCHEMAS" />
<h4>6.1.7.2. Schema Design Considerations</h4>
<a name="DataOrg" />
<h5>6.1.7.2.1. Data Organization</h5>
<p>
One should keep the following in mind when designing a schema for
maximum efficiency.
</p>
<br />
<a name="IndexUsage" />
<h5>6.1.7.2.2. Index Usage</h5>
<p>
A select from a table using a non-primary key will need to retrieve the
main row if there are search criteria on columns appearing on the main row
or output columns that have to be fetched from the main row. Operations
are noticeably faster if they can be completed without fetching the main
row if the driving key is a non-primary key. This is the case when search
criteria and output columns are on the secondary key parts or primary
key parts. Note that all secondary keys contain a copy of the primary
key. For this purpose it may be useful to add trailing key parts to a
secondary key. Indeed, a secondary key can hold all the columns of a row
as trailing key parts. This slows insert and update but makes reference
to the main row unnecessary when selecting using the secondary key.
</p>
<p>
A sequential read on the primary key is always fastest. A sequential
search with few hits can be faster on a secondary key if the criterion
can be evaluated without reference to the main row. This is because a
short key can have more entries per page.
</p>
<br />
<a name="SpaceConsump" />
<h5>6.1.7.2.3. Space Consumption</h5>
<p>
Each column takes the space 'naturally' required by its value. No field
lengths are preallocated. Space consumption for columns is the following:
</p>
<table class="data">
<caption>Table: 6.1.7.2.3.1. Data type Space Consumption</caption>
<tr>
<th class="data">Data</th>
<th class="data">Bytes</th>
</tr>
<tr>
<td class="data">Integer below 128</td>
<td class="data">1</td>
</tr>
<tr>
<td class="data">Smallint</td>
<td class="data">2</td>
</tr>
<tr>
<td class="data">long</td>
<td class="data">4</td>
</tr>
<tr>
<td class="data">float</td>
<td class="data">4</td>
</tr>
<tr>
<td class="data">timestamp</td>
<td class="data">10</td>
</tr>
<tr>
<td class="data">double</td>
<td class="data">8</td>
</tr>
<tr>
<td class="data">string</td>
<td class="data">2 + characters</td>
</tr>
<tr>
<td class="data">NULL</td>
<td class="data">data length for fixed length column, as value of 0 length for variable length column. </td>
</tr>
<tr>
<td class="data">BLOB</td>
<td class="data">88 on row + n x 8K (see note below)</td>
</tr>
</table>
<br />
<p>
If a BLOB fits in the remaining free bytes on a row after non-LOBs are stored,
it is stored inline and consumes only 3 bytes + BLOB length.
</p>
<p>
Each index entry has an overhead of 4 bytes.
This applies to the primary key as well as any other keys. The
length of the concatenation of the key parts is added to this. For the
primary key the length of all columns are summed. For any other key the
lengths of the key parts plus any primary key parts not on the secondary
key are summed. The maximum length of a row is 4076 bytes.
</p>
<p>
In light of these points primary keys should generally be short.
</p>
<br />
<a name="PageAlloc" />
<h5>6.1.7.2.4. Page Allocation</h5>
<p>
For data inserted in random order pages tend to be 3/4 full. For data
inserted in ascending order pages will be about 90% full due to a
different splitting point for a history of rising inserts.
</p>
<br />
<br />
<a name="EfficientSQL" />
<h4>6.1.7.3. Efficient Use of SQL - SQL Execution profiling</h4>
<p>
Virtuoso offers an execution profiling mechanism that keeps track of the
relative time consumption and response times of different SQL statements.
</p>
<p>
Profiling can be turned on or off with the prof_enable function. When profiling is on, the
real time between the start and end of each SQL statement execute call is logged on the server.
When prof_enable is called for the second time the statistics gathered between the
last call to prof_enable and this call are dumped to the virtprof.out file in the server's
working directory.
</p>
<p>
Profiling is off by default. Profiling can be turned on with the statement:
</p>
<div>
<pre class="programlisting">
prof_enable (1);
</pre>
</div>
<p>
The virtprof.out file will be generated when prof_enable is called for the second time, e.g.
</p>
<div>
<pre class="programlisting">
prof_enable (0);
</pre>
</div>
<p>
will write the file and turn profiling back off.
</p>
<p>
Below is a sample profile output file:
</p>
<div>
<pre class="screen">
Query Profile (msec)
Real 183685, client wait 2099294, avg conc 11.428772 n_execs 26148 avg exec 80
99 % under 1 s
99 % under 2 s
99 % under 5 s
100 % under 10 s
100 % under 30 s
23 stmts compiled 26 msec, 99 % prepared reused.
% total n-times n-errors
49 % 1041791 7952 53 new_order
28 % 602789 8374 490 delivery_1
12 % 259833 8203 296 payment
5 % 123162 821 35 slevel
2 % 54182 785 0 ostat (?, ?, ?, ?)
0 % 11614 4 0 checkpoint
0 % 2790 2 1 select no_d_id, count (*) from new_orde
0 % 2457 3 1 select count (*) from new_order
0 % 662 2 0 status ()
0 % 11 1 1 set autocommit on
0 % 3 1 0 select * from district
</pre>
</div>
<p>
This file was produced by profiling the TPC C sample application for 3 minutes.
The numbers have the following meanings:
</p>
<p>
The real time is the real time interval of the measurement, that is the space in
time between the prof_enable call that started the profiling and the call that wrote the report.
The client wait time is the time cumulatively spent inside the execute call server side, only calls completely
processed between profiling start and end are counted. The average concurrency is the exec time divided by real time
and indicates how many requests were on the average concurrently pending during the measurement interval.
The count of executes and their average duration is also shown.
</p>
<p>
The next section shows the percentage of executes that finished under 1, 2, 5, 10 and 30 seconds
of real time.
</p>
<p>
The compilation section indicates how many statements were compiled during the
interval. These will be SQLPrepare calls or SQLExecDirect calls where the text of the statement
was not previously known to the server. The real time spent compiling the statements is added up.
The percentage of prepared statement reuses, that is, executes not involving compiling over all executes is
shown. This is an important performance metric, considering that it is always better to use prepared
statements with parameters than statements with parameters as literals.
</p>
<p>
The next section shows individual statements executed during the measurement interval sorted by
descending cumulative real time between start and completion of the execute call.
</p>
<p>
The table shows the percentage of real time spent during the calls to the statement as a percentage
of the total real time spent in all execute calls. Note that these real times can be higher than the measurement
interval since real times on all threads are added up.
</p>
<p>
The second column shows the total execute real time, the third column shows the count of executes of
the statement during the measurement interval. The fourth column is the count of executes that resulted in an error.
The error count can be used for instance to spot statements that often produce deadlocks.
</p>
<p>
Statements are distinguished for profiling purposes using the 40 first characters of their text.
Two distinct statements that do not differ in their first 40 characters will be considered the same for profiling.
If a statement is a procedure call then only the name of the procedure will be
considered, not possibly differing literal parameters.
</p>
<p>
The profiler will automatically write the report after having 10000 distinct statements in the measurement
interval. This is done so as to have a maximum on the profiling memory consumption for applications that always
compile a new statement with different literals, resulting in a potentially infinitely long list of
statements each executed once. It is obvious that such a profile will not be very useful.
</p>
<p>
It cannot be overemphasized that if an application does any sort of repetitive processing then this should be
done with prepared statements and parameters, for both reasons of performance and profilability.
</p>
<div class="note">
<div class="notetitle">Note:</div>
<p>
Note that measurements are made with a one millisecond precision. Percentages are rounded to
2 digits. Timing of fast operations, under a few milliseconds, will be imprecise as a result of the 1 ms resolution.
Also the cumulative compilation time may be substantially off, since the compilation may often take
less than 1 ms at a time. Also note that the precision may also vary between platforms.
</p>
</div>
<br />
<a name="METERS_SYSVIEWS" />
<h4>6.1.7.4. Meters & System Views</h4>
<a name="kdlstat" />
<h5>6.1.7.4.1. DB.DBA.SYS_K_STAT, DB.DBA.SYS_D_STAT, DB.DBA.SYS_L_STAT view</h5>
<p>
These views provide statistics on the database engine
</p>
<div>
<pre class="programlisting">
create view SYS_K_STAT as
select KEY_TABLE, name_part (KEY_NAME, 2) as index_name,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'n_landings') as landed,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'total_last_page_hits') as consec,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'page_end_inserts') as right_edge,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'page_end_inserts') as lock_esc
from SYS_KEYS;
</pre>
</div>
<div>
<pre class="programlisting">
create view SYS_L_STAT as
select KEY_TABLE, name_part (KEY_NAME, 2) as index_name,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_set') as locks,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_waits') as waits,
(key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_waits') * 100)
/ (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_set') + 1) as wait_pct,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'deadlocks') as deadlocks,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_escalations') as lock_esc
from SYS_KEYS;
</pre>
</div>
<div>
<pre class="programlisting">
create view sys_d_stat as
select KEY_TABLE, name_part (KEY_NAME, 2) as index_name,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'touches') as touches,
key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'reads') as reads,
(key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'reads') * 100)
> / (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'touches') + 1) as read_pct
from SYS_KEYS;
</pre>
</div>
<p>
These views offer detailed statistics on index access locality, lock contention and
disk usage.
</p>
<p>
'reset' specified as the stat name will reset all counts for the key in question.
</p>
<br />
<a name="keystats" />
<h5>6.1.7.4.2. SYS_K_STAT - Key statistics</h5>
<ul>
<li>
<p>KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES</p>
</li>
<li>
<p>INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.</p>
</li>
<li>
<p>LANDED The count of random accesses, including inserts. Any insert or select, whether empty, single line or multi-
line counts as one. Updates and deletes do not count, as they imply a select in the same or previous statement.</p>
</li>
<li>
<p>CONSEC The number of times a random access falls on the same page as the previous random access. This is always less than LANDED.
For repetitive access to the same place or an ascending insert, this will be near LANDED. For a totally
random access pattern this will be near 0.</p>
</li>
<li>
<p>RIGHT_EDGE The number of times an insert has added a row to the right edge of the page where the insert was made. </p>
</li>
<li>
<p>LOCK_ESC The count of lock escalations, see SYS_L_STAT.</p>
</li>
</ul>
<br />
<a name="lstats" />
<h5>6.1.7.4.3. SYS_L_STAT</h5>
<ul>
<li>
<p>KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES</p>
</li>
<li>
<p>INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.</p>
</li>
<li>
<p>LOCKS The number of times a lock has been set on the index. Making a new row or page lock
counts as one. Entering a row or page lock either after a wait or without wait (for a shared lock) counts as one.</p>
</li>
<li>
<p>WAITS The number of times a cursor reading this index waited for a lock. Note that this can be higher
than the number of locks set, e.g. a 'read committed' cursor may wait for a lock but will never make one.</p>
</li>
<li>
<p>WAIT_PCT The percentage of lock set events that involved a wait. </p>
</li>
<li>
<p>DEADLOCKS The number of times a deadlock was detected when trying to wait for a lock on this index.
Note that one deadlock event may involve locks on several indices. Each deadlock detection counts as one.</p>
</li>
<li>
<p>LOCK_ESC The number of times the set of row locks on a page of this index where escalated into one page lock
or a page lock was set initially. This is always less than LOCKS. This value will
be near LOCKS when there are many sequential selects which switch to page lock mode.
This happens when a cursor has performed over 2 lock escalations and the page being entered has no
locks, i.e. the lock can be set over the entire page.</p>
</li>
</ul>
<br />
<a name="dstats" />
<h5>6.1.7.4.4. SYS_D_STAT</h5>
<ul>
<li>
<p>KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES</p>
</li>
<li>
<p>INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.</p>
</li>
<li>
<p>TOUCHES The number of times a row is located on the index. Every row retrieved by a select or inserted counts as
one. All rows scanned by an select count or other aggregate counts as one. </p>
</li>
<li>
<p>READS The number of times a disk read was caused by a read operation on this index.
This may theoretically be higher than TOUCHES, since several levels of the index tree may have
to be read to get to a leaf.</p>
</li>
<li>
<p>READ_PCT The percentage of READS in TOUCHES.</p>
</li>
</ul>
<a name="" />
<div class="example">
<div class="exampletitle">Examples:</div>
<div>
<pre class="programlisting">
select index_name, locks, waits, wait_pct, deadlocks
from sys_l_stat order by 2 desc;
Get lock data, indices in descending order of lock count.
select index_name, touches, reads, read_pct
from sys_d_stat order by 3 desc;
Get disk read counts, index with most reads first.
select index_name, (consec * 100) / (landed + 1)
from sys_k_stat where landed > 1000 order by 2;
</pre>
</div>
</div>
<p>Get the percentage of consecutive page access on indices with over 1000 accesses so far,
most randomly accessed first.
</p>
<br />
<a name="statusfunc" />
<h5>6.1.7.4.5. status SQL function - status ();</h5>
<p>This function returns a summary of the database
status as a result set. The result set has one varchar column,
which has consecutive lines of text. The lines can be up to several hundred
characters.
</p>
<p>The contents of the status summary are described in
the <a href="databaseadmsrv.html#DBSTAT">Administrator's Guide</a>.</p>
<br />
<a name="statusfunc" />
<h5>6.1.7.4.6. Virtuoso db file usage detailed info</h5>
<p>All data in a virtuoso database are logically stored as database key rows.
Thus the primary key for a table holds the entire row (including the dependent
part) and the secondary keys just hold their respective key parts.
So the space that the table occupies is the sum of the space occupied by it's
primary key and all the secondary keys.</p>
<p>The main physical unit of allocation in a virtuoso db file is the database page
(about 8k in virtuoso 3.x).
So the server needs to map the key rows and outline blobs to database pages.</p>
<p>Virtuoso will store as many rows in a db page as it can, so usually one DB page
will contain more than 1 row of a given key. No page contains rows from more than one key. However
blobs (when not inlined on the row) will be placed in consecutive DB pages (up
to their size). In addition to the blob and key pages the Virtuoso DB will hold a number of
pages containing internal data. So the sum of the pages occupied by the key rows and the blobs is
less then the amount of occupied pages (as reported by the
<a href="fn_status.html">status()</a> BIF).</p>
<p>To provide detailed information about the space consumption of each key there's
a system view:</p>
<div>
<pre class="programlisting">
DB.DBA.SYS_INDEX_SPACE_STATS
ISS_KEY_TABLE varchar -- name of the table
ISS_KEY_NAME varchar -- name of the key
ISS_KEY_ID integer -- id of the key (corresponding to KEY_ID from DB.DBA.SYS_KEYS)
ISS_NROWS integer -- number of rows in the table
ISS_ROW_BYTES integer -- sum of the byte lengths of all the rows in the table
ISS_BLOB_PAGES integer -- sum of the blob pages occupied by the outline blobs on all the rows of the table
ISS_ROW_PAGES integer -- sum of all the db pages containing rows of this key
ISS_PAGES integer -- = ISS_BLOB_PAGES + ISS_ROW_PAGES (for convenience).
</pre>
</div>
<p>Each select on that view causes the server to go over all the db pages in the db file
(similarly to how the crash dump operates) and collect the statistics above. The pages are
traversed 1 time per select, but still on large database files that may take some time. </p>
<br />
<br />
<a name="TRANSACTION_ISOLATION_LEVELS" />
<h4>6.1.7.5. Transaction Metrics, Diagnostics and Optimization</h4>
<p>Bad design and implementation of transactions affects
applications in the following ways:</p>
<ul>
<li>Performance is lost by having to needlessly retry
transactions that are aborted by deadlock.</li>
<li>Concurrency is lost by having rows stay locked for too
long.</li>
<li>Memory is transiently consumed, adversely affecting
working set, by keeping data structures for too many simultaneous locks,
rollback records and uncommitted roll forward logs.</li>
</ul>
<p>The following rules should be observed when writing
transactions:</p>
<ul>
<li>Do not lock needlessly. For example, any report
transaction that reads the data once can always be done as read committed instead of
repeatable read without affecting semantics. Even if some data
is read multiple times, the repeatable read semantic is
typically not relevant for reports.</li>
<li>Lock for what is needed. If you mean to update later, do
the initial read with exclusive locks. See the for update clause in select,
for example.</li>
<li>Lock in constant order. If you must lock different
resources in one
transaction, lock them always in the same order. When updating
stock for an order, update the quantity on hand in increasing
order of item number, for instance.
</li>
<li>Lock the item with the least contention first. For
example, update the
detail before updating the summary. Update the quantity in
stock for the ordered items before updating the orders count of
the whole warehouse.
</li>
<li>Keep transactions short. Use stored procedures. Use the
explicit commit work statement.</li>
<li>For each transaction in a stored procedure, make sure
that if it is
deadlocked, the deadlocked transaction gets retried. For
example, have a "declare exit handler for sqlstate 40001" for
every transaction context. Make sure that a deadlocking
transaction is never retried endlessly. Two mutually
deadlocking transactions can keep retrying and again
deadlocking each other endlessly. To avoid this, have a maximum
count of retries and a random delay before restarting. The
restart delay should be between 0 and the expected duration of
the transaction.
</li>
<li>Always break batch updates into multiple transactions.
Update a few
thousand or tens of thousands of rows per transaction, never
more than that. Failing to do this makes for prohibitive cost
of retry with deadlocks and can cause swapping by keeping tens
or hundreds of megabytes in rollback state, locks and other
transaction temporary structures. This happens if one inserts,
updates, deletes several million rows in a single transaction.
If this is really needed and concurrency is no issue, use the
atomic mode, effectively making the server single user for the
transaction, thus having no locking or rollback. See the use of
the <strong>__atomic()</strong> function in the
<a href="fault.html#faultfaulttoleradmapi">Cluster Administration API</a>.
</li>
</ul>
<a name="ptuneprogvirtpl" />
<h5>6.1.7.5.1. Programming Virtuoso/PL</h5>
<p>The isolation level is set in Virtuoso/PL with the</p>
<div>
<pre class="programlisting">
set isolation := level;
</pre>
</div>
<p>statement, where level is one of 'serializable',
'repeatable', 'committed', 'uncommitted'. Example :</p>
<div>
<pre class="programlisting">
set isolation = 'serializable';
</pre>
</div>
<p>
The standard SQL syntax is also supported :</p>
<div>
<pre class="programlisting">
SET TRANSACTION ISOLATION LEVEL <isolation_level>
isolation level : READ UNCOMMITTED | READ COMMITTED | REPEATABLE READ | SERIALIZABLE
</pre>
</div>
<p>
Example :</p>
<div>
<pre class="programlisting">
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
</pre>
</div>
<p>The effect is for the rest of the procedure and any
procedure called from this procedure. The effect stops when the
procedure having executed the set isolation statement
returns.</p>
<br />
<a name="ptunesampledeadlockhandler" />
<h5>6.1.7.5.2. Sample Deadlock Handler</h5>
<p>The text for a deadlock handler is</p>
<div>
<pre class="programlisting">
declare retry_count int;
retry_count := 0;
retry:
{
declare exit handler for sqlstate '40001'
{
rollback work;
...
delay (rnd (2.5)); --- if 2.5 seconds is the expected duration of
the transaction.
...
retry_count := retry_count + 1;
if (retry_count > 5)
signal ("xxxxx", "Too many deadlock retries in xxxxx.");
goto retry;
}
-- do the operations. The actual working code here.
commit work;
}
</pre>
</div>
<p>An exclusive read is done with</p>
<div>
<pre class="programlisting">
select s_quantity from stock where s_i_id = 111 for update;
</pre>
</div>
<br />
<a name="ptuneodbciso" />
<h5>6.1.7.5.3. ODBC</h5>
<p>For the Virtuoso ODBC driver the isolation
is set by :</p>
<ul>
<li>connection option (in either <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqlsetconnectattr.asp">SQLSetConnectAttr ()</a> or <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqlsetconnectoption.asp">SQLSetConnectOption ()</a>)
<div>
<pre class="programlisting">
rc = SQLSetConnectOption (hdbc, SQL_TXN_ISOLATION, SQL_TXN_READ_COMMITTED);
</pre>
</div>
or
<div>
<pre class="programlisting">
rc = SQLSetConnectAttr (hdbc, SQL_TXN_ISOLATION, SQL_TXN_READ_COMMITTED, NULL);
</pre>
</div>
<p>Constants are : SQL_TXN_READ_UNCOMMITTED,
SQL_TXN_READ_COMMITTED, SQL_TXN_REPEATABLE_READ,
SQL_TXN_SERIALIZABLE</p>
</li>
<li>ODBC setup dialog
option : In Windows there is a drop-down combo box to set the
default transaction isolation level for a connection.</li>
<li>
<p>connection string element : You may specify the default
transaction isolation level for a given connection in it's
connect string (passed to the <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqldriverconnect.asp">SQLDriverConnect ()</a> ODBC API). Example
:</p>
<div>
<pre class="programlisting">
SQLDriverConnect (hdbc, hwnd,
"DSN=MyDSN;IsolationLevel=Repeatable Read;UID=dba;PWD=dbapwd", SQL_NTS,
NULL, 0,
NULL,
SQL_DRIVER_NOPROMPT).
</pre>
</div>
<p>The possible options for the connection string are : "Read
Uncommitted", "Read Committed", "Repeatable Read" and
"Serializable"</p>
</li>
</ul>
<br />
<a name="ptunejdbciso" />
<h5>6.1.7.5.4. JDBC</h5>
<p>In the Virtuoso JDBC driver the isolation is set by
the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/sql/Connection.html#setTransactionIsolation(int)">java.sql.Connection.setTransactionIsolation()</a>
JDBC API.</p>
<div>
<pre class="programlisting">
conn.setTransactionIsolation (java.sql.Connection.TRANSACTION_SERIALIZABLE)
</pre>
</div>
<p>The constants are described in the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/sql/Connection.html#field_summary">Java Docs</a>
</p>
<br />
<a name="ptunedotnetiso" />
<h5>6.1.7.5.5. .Net</h5>
<p>In the VirtuosoClient.NET provider the isolation is set by the
<a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemdataidbconnectionclassbegintransactiontopic2.asp">System.Data.IDbConnection.BeginTransaction Method (IsolationLevel)</a> function.</p>
<div>
<pre class="programlisting">
System.Data.IDBTransaction trx = conn.BeginTransaction (System.Data.IsolationLevel.ReadCommitted)
</pre>
</div>
<p>The constants are described <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemdataisolationlevelclasstopic.asp">here</a>
</p>
<br />
<br />
<a name="ptunemetricsdiag" />
<h4>6.1.7.6. Metrics and Diagnostics</h4>
<p>Metrics are presented at the server and the table level.</p>
<p>The first metric to check is the output of status ('');</p>
<p>The paragraph titled transaction status contains the
following:</p>
<ul>
<li>Count of deadlocks since server startup. There is a total
number of
deadlocks and the number of 2r1w deadlocks. The latter is a
special case where two transactions both hold a shared lock on
a resource and one tries to convert the lock to exclusive. This
situation can kill the transaction attempting to write. Such
deadlocks are essentially always needless. These are avoided by
reading for update when first reading the resource.
</li>
<li>Count of waits since server startup. This is incremented
every time
some operation waits for a lock, except if this wait leads into
a deadlock. If the number of deadlocks is high, let's say over
5% of the number of waits, transactions are likely badly
designed and deadlock too often, either because of not locking
for write at the get go (2r1w) or because of locking resources
in varying order.
</li>
<li>Count of threads running. This is the count of all
threads that are
somehow occupied, whether running or waiting. This count minus
the count of waiting minus the count of threads in vdb is the
count of threads that in principle could be on CPU.
</li>
<li>Count of threads waiting. This is the count of threads
that are
waiting for a lock. If this is a high percentage of the count
of threads running, say over 30%, resources are likely locked
inefficiently, keeping too many locked.
</li>
<li>Count of threads in vdb. This is the count of threads that
are at the time waiting for I/O either from a remote database or any sort
of network operation, including access to web services on other
serbers, access to web pages on other hosts etc.
</li>
</ul>
<p>The system view db.dba.sys_l_stat is used for locating
bottlenecks.</p>
<p>The columns are:</p>
<ul>
<li>index - The index being locked. Note that when reading on
non-primary
key, the lock is set on the index first, only then on the pk,
that is if the pk is accessed at all. For all updates however,
the pk will always be accessed. *locks - The count of times a
lock was set on this index.
</li>
<li>waits - The number of times there was a wait on this
index. There can
be more waits than locks because a read committed cursor can
wait but will not lock, thus all waits do not result in locks.
</li>
<li>wait_pct - The percentage of times setting a lock waited.
(100 * waits) / locks
</li>
<li>deadlocks - The number of times a deadlock was signalled
when attempting to wait for a lock on this index.
</li>
<li>lock_esc - The number of times a set of row locks was
converted into a page lock on this index.
</li>
<li>wait_msecs - The total amount of real time spent by some
thread waiting for a lock on this index. This may be greater than elapsed time
because many threads can wait at the same time.
</li>
</ul>
<p>All counts and times are cumulative from server startup.</p>
<p>The interpretation is as follows:</p>
<p>If deadlocks is high in relation to waits or locks, i.e.
over 5%, there are many deadlocks and the transaction profiles
may have to be adjusted. The table where deadlocks is
incremented is the table where the deadlock was detected but
the deadlock may involve any number of tables. So, if A and B
are locked in the order A, B half of the time and B, a the rest
of the time, then the deadlocks of the tables of A and B will
be about the same, half of the deadlocks being detected when
locking A, the other half when locking B.</p>
<p>If waits is high in relation to locks, for example 20%, then
there is probably needless contention. Things are kept locked
needlessly. Use read committed or make shorter transactions or
lock items with the less contention first.</p>
<p>Because transaction duration varies, the place with the
highest count of waits is not necessarily the place with the
heaviest contention if the waits are short. Use wait_msecs in
addition to waits for determining where the waiting takes
place.</p>
<p>To get a general picture, use the Conductor's Statistics
page or simply do</p>
<p>select top 5 * from sys_l_statt order by wait_msecs
desc;</p>
<p>to get a quick view of where time is spent. You can also
sort by waits desc or locks desc.</p>
<a name="ptunemetricdiagsqlissues" />
<h5>6.1.7.6.1. SQL Issues</h5>
<p>It is possible to get bad locking behavior if the SQL
compiler decides to make linear scans of tables or indices and
the isolation is greater than read committed. The presence of a
linear scan on an index with locking is often indicated by
having a large number of lock escalations. If lock_esc is near
locks then a large part of the activity is likely sequential
reads.</p>
<p>The general remedy is to do any long report type
transactions as read committed unless there are necessary
reasons to do otherwise.</p>
<p>To see how a specific query is compiled, one can use the
explain () function. To change how a query is compiled, one can
use the table option or option SQL clauses.</p>
<br />
<a name="ptunemetricdiagdynmicsobs" />
<h5>6.1.7.6.2. Observation of Dynamics</h5>
<p>Deadlocks and contention do not occur uniformly across time.
The occurrences will sharply increase after a certain
application dependent load threshold is exceeded.</p>
<p>Also, deadlocks will occur in batches. Several transactions
will first wait for each other and then retry maybe several
times, maybe only one succeeding at every round. In such worst
cases, there will be many more deadlocks than successfully
completed transactions. Optimize locking order and make the
transactions smaller.</p>
<p>Looking at how counts change, specially if they change in
bursts is useful.</p>
<br />
<a name="ptunemetricdiagsqlissues" />
<h5>6.1.7.6.3. Tracing and Debugging</h5>
<p>The most detailed picture of a system's behavior, including
deadlocks nd other exceptions cn be obtained with profiling. If
the application is in C, Java or some other compiled language,
one can use the language's test coverage facility to see
execution counts for various branches of the code.</p>
<p>For client applications, using the Virtuoso <a href="">trace()</a>
function can be useful for seeing which statements signal
errors. Also the profiling report produced by <a href="fn_prof_enable.html">prof_enable ()</a>
can give useful hints on execution times and error frequencies.
See Profiling and <a href="fn_prof_enable.html">prof_enable ()</a>.</p>
<p>For PL applications, Virtuoso provides a profiler and test
coverage facility. This is activated by setting PLDebug = 2 in
the Parameters section of the ini file and starting the server.
The functions cov_store () and cov_report are used for
obtaining a report of the code execution profile thus far. See
the documentation on "Branch Coverage" for this. The execution
counts of the lines for exception handler code will show the
frequency of exceptions. If in a linear block of code a
specific line has an execution count lower than that of the
line above it, then this means that the line with the higher
count has signalled as many exceptions as the difference of the
higher and lower count indicates.</p>
<p>The times indicated in the flat and call graph profile
reports for PL procedures are totals across all threads and
include time spent waiting for locks. Thus a procedure that
consumes almost no CPU can appear high on the list if it waits
for locks, specially if this is so on multiple threads
concurrently. The times are real times measured on threads
separately and can thus exceed any elapsed real tiime.</p>
<p>Single stepping is not generally useful for debugging
locking since locking issues are timing sensitive.</p>
<br />
<br />
<a name="clientlevelresourceaccounting" />
<h4>6.1.7.7. Client Level Resource Accounting</h4>
<p>Starting with version 6, Virtuoso keeps track of the count of basic database operations performed on behalf of each connected client.
The resource use statistics are incremented as work on the connection proceeds. The db_activity () function can be called to return the accumulated operation counts and to optionally reset these.
</p>
<p>The db_activity built-in function has one optional argument. The possible values are:</p>
<ul>
<li>0 - (default) - Return human readable string and reset the counts.</li>
<li>1 - return an array of numbers and reset the counts.</li>
<li>2 - return a human readable string and leave the counts.</li>
<li>3 - return an array of numbers and leave the counts.</li>
</ul>
<p>The human readable string is of the form:</p>
<div>
<pre class="programlisting">
22.56MR rnd 1.102GR seq 10P disk 1.341GB / 102.7K messages
</pre>
</div>
<p>The postfixes K, M, G, T mean 10^3 to 10^15, except when applied to bytes, where these mean consecutive powers of 1024.</p>
<p>The numbers, left to right are the count of random row lookups,
sequential row lookups, disk page reads, cluster inter-node traffic in
bytes and cluster inter-node message count as an integer number of
messages. If the configuration is a single server, the two last are 0.
</p>
<p>The random and sequential lookup counts are incremented regardless of whether the row was found or not or whether it matched search conditions.</p>
<p>If the information is retrieved as an array, the array contains integer numbers representing these plus some more metrics.</p>
<p>Index - Meaning</p>
<ul>
<li>0 - Random lookups</li>
<li>1 - sequential lookups</li>
<li>3 - lock waits</li>
<li>4 - total msec spent in lock wait on some thread. In a cluster situation, this may be more than elapsed real time.</li>
<li>5 - Disk reads</li>
<li>6 - Speculative disk reads. These are also counted in disk reads. A speculative read is a prefetch made on the basis of a read history of a disk extent.</li>
<li>7 - Cluster inter node message count</li>
<li>8 - Total bytes sent in cluster inter node traffic. A message is counted once, when sent.</li>
</ul>
<p>If the thread calling db_activity is a web server thread, the totals are automatically reset when beginning the processing of the current web request.</p>
<br />
<br />
<table border="0" width="90%" id="navbarbottom">
<tr>
<td align="left" width="33%">
<a href="server.html" title="Administration">Previous</a>
<br />Contents of Administration</td>
<td align="center" width="34%">
<a href="server.html">Chapter Contents</a>
</td>
<td align="right" width="33%">
<a href="htmlconductorbar.html" title="HTML based Administration Console (Conductor) Guide">Next</a>
<br />HTML based Administration Console (Conductor) Guide</td>
</tr>
</table>
</div>
<div id="footer">
<div>Copyright© 1999 - 2009 OpenLink Software All rights reserved.</div>
<div id="validation">
<a href="http://validator.w3.org/check/referer">
<img src="http://www.w3.org/Icons/valid-xhtml10" alt="Valid XHTML 1.0!" height="31" width="88" />
</a>
<a href="http://jigsaw.w3.org/css-validator/">
<img src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!" height="31" width="88" />
</a>
</div>
</div>
</body>
</html>
distrib
>
Fedora
>
14
>
x86_64
>
media
>
updates
>
by-pkgid
>
71d40963b505df4524269198e237b3e3
>
files
>
78
