.. -*- mode: rst -*-

.. _help-troubleshooting:

===============
Troubleshooting
===============

From time to time, Bcfg2 produces results that the user finds surprising.
This can happen either due to bugs or user error. This page describes
several techniques to gain visibility into the bcfg2 client and server
and understand what is going on.


Figure out if error is client or server side
============================================

* Cache a copy of the client configuration using ``bcfg2 -qnc /tmp/config.xml``
* Look in the file and search for the entry of interest
* If it looks correct, then there is a client issue
* If not, it is time to inspect things on the server

This file contains all aspects of client configuration. It is structured
as a series of bundles and base entries.

.. note::

    Most often the entry is not correct and the issue lies in
    the specification.

Review server log messages
==========================

The bcfg2-server process logs to syslog facility LOG_DAEMON. The server
produces a series of messages upon a variety of events and errors.

The server also supports two XML-RPC methods that can be used
to turn up the debug level in a live server:

* ``toggle_debug``: Turn debug on or off, depending on the current
  setting.
* ``set_debug``: Turn debug explicitly on or off.

These can be called with :ref:`bcfg2-admin xcmd <server-admin-xcmd>`,
e.g.::

    bcfg2-admin xcmd toggle_debug
    bcfg2-admin xcmd set_debug true

Each plugin also supports these two methods, which can be used to set
the debug level individually on a given plugin, e.g.::

    bcfg2-admin xcmd Packages.set_debug true
    bcfg2-admin xcmd Probes.toggle_debug

Finally, the File Activity Monitor has its own analogue to these two
methods, for setting the debug level of the FAM:

    bcfg2-admin xcmd toggle_fam_debug
    bcfg2-admin xcmd set_fam_debug false

Check if all repository XML files conform to schemas
====================================================

Bcfg2 comes with XML schemas describing all of the XML formats used in
the server repository. A validation command ``bcfg2-lint`` is
included with the source distribution and all packages.

If the bcfg2 server is not reflecting recent changes, try restarting the bcfg2-server process
=============================================================================================

If this fixes the problem, it is either a bug in the
underlying file monitoring system (fam or gamin) or a bug in
Bcfg2's file monitoring code. In either case, file a `ticket
<https://trac.mcs.anl.gov/projects/bcfg2/newticket>`_ in the tracking
system. In the ticket, include:

* filesystem monitoring system (fam or gamin)
* kernel version (if on linux)
* if any messages of the form "Handled N events in M
  seconds" appeared between the modification event and the client
  configuration generation request appeared in the server log
* which plugin handled the file in the repostiory (Cfg, Rules, Packages,
  SSHbase, Metadata, etc.)
* if a touch of the file after the modification causes the problem to
  go away

bcfg2-info
==========

Bcfg2 server operations can be simulated using the ``bcfg2-info`` command.
The command is interactive, and has commands to allow several useful
operations

* clients - Current client metadata (profile and group) settings
* groups - Current group metadata values
* mappings - Configuration entries provided by plugins
* buildfile [--altsrc=<altsrc>] <filename> <hostname> - Build a config
  file for a client
* buildbundle <bundle> <hostname> - Render a templated bundle for a client
* showentries <client> <type> - Build the abstract configuration (list
  of entries) for a client
* build <hostname> <output-file> - Build the complete configuration
  for a client

Type `help` in bcfg2-info for more information.

Error Messages
==============

The tables below describe error messages produced by Bcfg2 and steps that can
be taken to remedy them.

Client Errors
-------------

+------------------------------+----------------------------+--------------+
| Error                        | Meaning                    | Repair       |
+==============================+============================+==============+
| Incomplete information for   | The described entry is not | [c1]_        |
| entry <EntryTag>:<EntryName> | fully specified by the     |              |
| cannot verify                | server, so no verification |              |
|                              | can be performed.          |              |
+------------------------------+----------------------------+--------------+
| Incomplete information for   | The described entry is not | [c1]_        |
| entry <EntryTag>:<EntryName> | fully specified by the     |              |
| cannot install               | server, so no verification |              |
|                              | can be performed.          |              |
+------------------------------+----------------------------+--------------+
| The following entries are    | The client cannot figure   | [c2]_        |
| not handled by any tool:     | out how to handle this     |              |
| <EntryTag>:<EntryName>       | entry.                     |              |
+------------------------------+----------------------------+--------------+
| No ca is specified. Cannot   | The client is unable to    | [c3]_        |
| authenticate the server with | verify the server.         |              |
| SSL.                         |                            |              |
+------------------------------+----------------------------+--------------+
| GID normalization failed for | The client is unable to    | [c4]_        |
| FILENAME. Does group GROUP   | convert the group GROUP to |              |
| exist?                       | a usable GID.              |              |
+------------------------------+----------------------------+--------------+
| UID normalization failed for | The client is unable to    | [c5]_        |
| FILENAME. Does owner OWNER   | convert the owner OWNER to |              |
| exist?                       | a usable UID.              |              |
+------------------------------+----------------------------+--------------+
| SSL CA error                 | The CA certificate         | [c6]_        |
|                              | specified in bcfg2.conf is |              |
|                              | incorrect                  |              |
+------------------------------+----------------------------+--------------+


.. [c1] This entry is not being bound. Ensure that a version of this
        entry applies to this client.
.. [c2] It is possible that the type attribute for this generator entry
        is undefined. You may need to add the appropriate type attribute
        in the literal entry specification.
.. [c3] Copy the Bcfg2 server's CA certificate to the client and specify it
        using the **ca** option in the [communication] section of
        ``bcfg2.conf``
.. [c4] If the group doesn't exist, you need to specify the correct one
        in an :ref:`info.xml <server-info>` file or set the default group
        appropriately.
.. [c5] If the owner doesn't exist, you need to specify the correct one
        in an :ref:`info.xml <server-info>` file or set the default owner
        appropriately.
.. [c6] Check that the CA specified in bcfg2.conf is appropriate for the
        server you are attempting to access.

Server Errors
-------------

+------------------------------+---------------------+--------------+
| Error                        | Meaning             | Repair       |
+==============================+=====================+==============+
| Failed to bind entry:        | The server was      | [s1]_        |
| <EntryTag> <EntryName>       | unable to find a    |              |
|                              | suitable version of |              |
|                              | entry for client.   |              |
+------------------------------+---------------------+--------------+
| Failed to bind to socket     | The server was      | [s2]_        |
|                              | unable to bind to   |              |
|                              | the tcp server      |              |
|                              | socket.             |              |
+------------------------------+---------------------+--------------+
| Failed to load               | The server was      | [s3]_        |
| ssl key <path>               | unable to read and  |              |
|                              | process the ssl key.|              |
+------------------------------+---------------------+--------------+
| Failed to read file <path>   | The server failed   | [s4]_        |
|                              | to read the         |              |
|                              | specified file      |              |
+------------------------------+---------------------+--------------+
| Failed to parse file <path>  | The server failed   | [s5]_        |
|                              | to parse the        |              |
|                              | specified XML file  |              |
+------------------------------+---------------------+--------------+
| Client metadata resolution   | The server cannot   | [s6]_        |
| error for <IP>               | resolve the client  |              |
|                              | hostname or the     |              |
|                              | client is           |              |
|                              | associated with a   |              |
|                              | non-profile group.  |              |
+------------------------------+---------------------+--------------+
| Failed to decode <filename>  | The encoding being  | [s7]_        |
| Please verify you are using  | used is unable to   |              |
| the proper encoding          | decode the          |              |
|                              | character present   |              |
|                              | in this file.       |              |
+------------------------------+---------------------+--------------+
| Got unknown entries          | The Packages plugin | [s8]_        |
| [list of unknown entries]    | has no knowledge of |              |
|                              | the listed entries  |              |
+------------------------------+---------------------+--------------+
| Failed to import lxml        | The server cannot   | [s9]_        |
| dependency. Shutting         | import lxml         |              |
| down server.                 |                     |              |
+------------------------------+---------------------+--------------+
| You need to specify base64   | The server cannot   | [s10]_       |
| encoding for <path>          | send the file as    |              |
|                              | ascii text          |              |
+------------------------------+---------------------+--------------+
| ERROR: Error reading file    | The server cannot   | [s11]_       |
| '/path/to/schema': failed to | find the schema     |              |
| load external entity         | file                |              |
| "/path/to/schema"            |                     |              |
+------------------------------+---------------------+--------------+
| Packages: No matching        | None of the sources | [s12]_       |
| sources for client           | defined in the      |              |
| <clientname>; improper group | Package plugin's    |              |
| memberships?                 | ``sources.xml``     |              |
|                              | apply to the client |              |
+------------------------------+---------------------+--------------+


.. [s1] This entry is not being bound. Ensure that a version of this
        entry applies to this client.
.. [s2] Ensure that another instance of the daemon (or any other process)
        is not listening on the same port.
.. [s3] Ensure that the key is readable by the user running the daemon
        and that it is well-formed.
.. [s4] Ensure that this file still exists; a frequent cause is the
        deletion of a temp file.
.. [s5] Ensure that the file is properly formed XML.
.. [s6] Fix hostname resolution for the client or ensure that the profile
        group is properly setup.
.. [s7] Ensure the correct encoding is specified in the [components]
        section of ``bcfg2.conf``.
.. [s8] For packages listed other than **gpg-pubkey**, this error means
        that the Packages plugin is unable to find the package in any of
        the sources listed in ``Packages/sources.xml``. The issue often
        arises when the client is not in one of the groups necessary for
        the Source listed. In the case of gpg-pubkey, you can safely
        ignore the message as the Packages plugin has no knowledge of
        these packages (however, note that this package is most often
        specified as a BoundPackage entry).
.. [s9] Ensure that you have installed all the necessary
        :ref:`installation-prerequisites`.
.. [s10] You likely need to specify a base64 encoding using an
         :ref:`server-info` file for this entry.
.. [s11] Verify that you have the proper prefix set in bcfg2.conf.
.. [s12] Ensure that the client is a member of all the appropriate
	 :ref:`server-plugins-generators-packages-magic-groups` as
	 well as any additional groups you may have defined in your
	 :ref:`server-plugins-generators-packages` configuration.

FAQs
====

Why won't bcfg2-server start?
-----------------------------

If your server doesn't seem to be starting and you see no error
messages in your server logs, try running it in the foreground to see
why.

Why am I getting a traceback?
-----------------------------

If you get a traceback, please let us know by reporting it
on `Trac ticket tracker`_, via the mailing list, or on IRC. Your best bet
to get a quick response will be to jump on IRC during the daytime (CST).

.. _Trac ticket tracker: http://bcfg2.org

What is the most common cause of "The following entries are not handled by any tool"?
-------------------------------------------------------------------------------------

Often it corresponds to entries that aren't bound by the server (for which
you'll get error messages on the server). You should try inspecting the
logs on the server to see what may be the cause.