Sophie: openais-1.1.2-1mdv2010.1 x86

openais-1.1.2-1mdv2010.1.x86_64.rpm

AMF B.02.01 Implementation
--------------------------
The implementation of AMF in openais is directed by the specification
SAI-AIS-AMF-B.02.01, see http://www.saforum.org/specification/.

What does AMF do?
-----------------
The AMF has many major duties:
* issue instantiate, terminate, and cleanup operations for components
* assignment of component service instances to components
* executing of recovery and repair actions on fault reports delivered
by components (fault detection is a responsibility of all entities
in the system)

An AMF user has to provide instantiate and cleanup commands and a
configuration file besides from the binaries that represents the actual
components.

To start a component, AMF executes the instantiate command which starts
processes that are part of the component. AMF can stop the component
abruptly by running the cleaup command.

An service unit (SU) contains multiple components and represents a
"useable service" and is configured to execute on an AMF node. The AMF node
is mapped in the configuration to a CLM node which is "an operating system
instance". An SU is the smallest part that can be instantiated in a redundant
manner and can therefore be viewed as the unit of redundancy.

A service group (SG) contains multiple SUs. The SG is the unit that implements
high availability by managing its contained service units. An SG can be
configured to execute different redundancy policies.

An application contains multiple SGs and multiple service instances (SIs).

An SI represents the workload for an SU. An SI consists of one or more
component service instances (CSIs).

A CSI represents the workload of a component. The CSI is configured to include
a list of name value pairs through which the user can express the workload.

The AMF specification defines several types of components. The AMF
specification is exceedingly clear about which CLC operations occur for which
component types.

If a component is not sa-aware, the only level of high availability that
can be applied to the application is through execution of the CLC interfaces.

A special component, called a proxy component, can be used to present an
SA-aware component to AMF to manage a non-SA-aware component. This would be
useful, for example, to implement a healthcheck operation which runs some
operation of the unmodified application service.

Components that are SA-aware have been written specifically to the AMF
interfaces. These components provide the most support for high availability
for application developers.

When an SA-aware component has been instantiated it has to register within a
certain time. After a successful registration, AMF assigns workload to the
component by making callbacks once the service unit is available to take service.
There will be one callback for each CSI-assignment. Each CSI-assignment has
a HA state associated which indicates how the component shall act.
The HA state can be ACTIVE, STANDBY, QUIESCED or QUIESCING.

The number of CSIs assigned to a component and the setting of their HA state
is determined by AMF. In the configuration the operator specifies the preferred
assignment of workload to the defined SUs. The configuration specifies also
limits for how much work each SU can execute. If not the preferred distribution
of workload can be met due to problems in the cluster a reduction process with
6 levels of reduction will be executed by AMF. The purpose of the reduction
procedure is to come as close as possible to the preferred configuration without
violating any limits for how much workload an SU can handle. The reduction
procedure continues until there are no SUs in-service in the SG.

AMF supports fault detection through a healthcheck API. The user
specifies in the configuration file healthcheck keys and timing parameters.
This configuration is then used by the application developer to register
a healthcheck operation in the AMF. The healthcheck operation can be started
or stopped. Once started, the AMF will periodically send a request to the
component to determine its level of health. Optionally, AMF can be configured to
instead expect the component to report its health periodically.
The AMF reacts to negative healthchecks or failed healthchecks by executing
a recovery policy.

The AMF specification also includes an API for reporting errors with a
recommended recovery action. AMF will not take a weaker recovery action than
what is recommended but may take a stronger action based on the recovery
escalation policy.

There is a recovery escalation policy for the recomendations:
- component restart
- component failover

When AMF receives a recommendation to restart a component, the recovery policy
attempts to restart the component first. When the component is restarted and
fail a certain number of times within a timeout period, the entire service unit
is restarted. When the SU has been restarted a certain number of times within
a certain timeout period, the SU is failed over to a standby SU. If AMF fails
over too many service units out of the same node in a given time period as a
consequence of error reports with either component restart or component
failover recommended recovery actions, the AMF escalates the recovery to an
entire node fail-over.

What is currently implemented ?
-------------------------------

SA-aware components can be instantiated and assigned load according to the
configuration specified in amf.conf. Other types of components are currently
not supported. The processes of instantiation and assignment of workload are
both simplified compared to the requirements in the AMF specification.

Service units represented by their components can be configured to execute
on different nodes. AMF supports initial start of the cluster as well as adding
of a node to the cluster after the initial start. AMF also supports that a node
leave the cluster by failing over the workload to standby service units.

Healthchecks are implemented as specified with only a few details missing.

The error report API is implemented but AMF ignores the recommendation of
recovery action instead it will always try to recover by 'component restart'.

The error escalation mechanism up to SU failover is also implemented as
specified with a few simplifications.

Only redundancy model N+M is (partly) implemented.

You can find a detailed list of what is NOT implemented later in the README.

How to configure AMF
--------------------
The AMF specification doesn't specify a configuration file format. It does
however, describe many configuration options, which are specified formally in
SAI-Overview-B.02.01 chapter 4.5 - 4.11. The Overview can also be retrieved
from http://www.saforum.org/specification/.

An implementation specific feature of openais is to implement the configuration
options in a file called amf.conf. There is a man page in the /man directory
which describes the syntax of amf.conf and what configuration options which
are currently supported.

The example programs
--------------------
First the openais example programs should be installed. When compiling openais
in the exec directory a file called openais-instantiate is created. Copy this
file to a test directory of your own:

mkdir /tmp/aisexample

exec# cp openais-instantiate /tmp/aisexample

Copy also the script which implements the instantiate, terminate and clean-up
operations to your test directory:

exec# cp ../test/clc_cli_script /tmp/aisexample/clc_cli_script

Set execute permissions for the clc_cli_script

exec# chmod +x /tmp/aisexample/clc_cli_script

Copy the binary to be used for all components:
exec# cp ../test/testamf1 /tmp/aisexample/testamf1

Copy the amf example configuration files from the openais/conf directory to
your test directory.

exec# cp ../conf/*amf_example.conf /tmp/aisexample

set environment variables to the names of the configuration files:

setenv OPENAIS_AMF_CONFIG_FILE /tmp/aisexample/amf_example.conf
setenv OPENAIS_MAIN_CONFIG_FILE /tmp/aisexample/openais_amf_example.conf

You have to specify the host on which you would like to execute the AMF example.
Open the file 'amf_example.conf' and replace the line:

saAmfNodeClmNode=p01

in the following section in the cluster configuration:

safAmfNode = AMF1 {
saAmfNodeSuFailOverProb=2000
saAmfNodeSuFailoverMax=2
saAmfNodeClmNode=p01
}

p01 shall be replaced with the name of your host.

(You can obtain the name of your host by typing the command 'hostname' in a
shell.)

Modify the following rows of 'openais_amf_example.conf' so that they match your
user and group:

aisexec {
user: nisse
group: users
}

(One way to obtain your user and group is to type the command 'id' in a shell.)

Start aisexec by command:
./aisexec

aisexec will be run in the background.
Once aisexec is run using the example configuration file, 2 service units
will be instantiated. The testamf1 C code will be used for both component A
and component B of both SUs. The testamf1 program determines its
component name at start time from the saAmfComponentNameGet() api call.
The result is that 4 processes will be started by AMF.

Each testamf1 process will first try to register a bad component name and
there after register the name returned from saAmfComponentNameGet().
The testamf1 will be assigned CSIs after they execute a
saAmfComponentRegister() API call. Note that a successful registration causes
the state of the component and service units to be set to INSTANTIATED as
required by the AMF specification. The service instances and their names are
defined within the configuration file.

The component of type saAmfCSTypeName = B, which have the active HA state,
in this case, safComp=B,safSu=SERVICE_X_1,safSg=RAID,safApp=APP-1,
reports an error via saAmfErrorReport() after exactly 10 healthchecks.
The healthcheck period is configured to 1 second so one error report is sent
every 10th second.
This results in openais calling the cleanup handler, which for
an sa-aware component, is the CLC_CLI_CLEANUP command. This causes the cleanup
operation of the clc_cli_script to be run. This cleanup command then reads the
pid of the process that was stored to /var/run ( or /tmp) at startup of the
testamf1 program. It then executes a kill -9 on the PID. Custom cleanup
operations can be executed by modifying the clc_cli_script script program.

After this is done 2 times (configurable) the entire service
unit is terminated and restarted due to the error escalation mechanism. Once
this happens 3 times (also configurable), the code escalates to level 2 and a
failover of the SU takes place. After this testamf1 makes no more error
reports and nothing will happen until some problem is recognized (like the
process of one of the components stops executing).

The states of the cluster and its contained entities can be obtained by issuing
the following command in the shell:

pkill -USR2 ais

Some notes:
-----------
In the example, testamf1 is sending an error report at the 10th helthcheck.
This is actually controlled by the safCSIAttr = good_health_limit in
file amf_example.conf and can be changed as you like.

The file openais_amf_example.conf specifies logging to stderr.

If you would like to follow more closely the execution of the AMF in openais,
debug printouts can be enabled.

example:
logging {
fileline: off
to_stderr: yes
to_file: no
logfile: /tmp/openais.log
debug: off
timestamp: on
logger {
ident: AMF
debug: on
tags: enter|leave|trace1|trace2|trace3|trace4|trace6
}

Setting 'debug: on' generally gives many printouts all other parts of openais.

Run the example on a cluster with 2 nodes
-----------------------------------------

It is easy to run the example on more than one node.
Modify the file openais_amf_example.conf:

<1>
Replace the following line:
bindnetaddr: 127.0.0.0

bindnetaddr specifies the address which the openais Executive should bind to.
This address should always end in zero. If the local interface traffic
should be routed over is 192.168.5.92, set bindnetaddr to 192.168.5.0.

Modify amf_example.conf like this:
<1>
Remove the comment character '#' from the following lines:
# safAmfNode = AMF2 {
# saAmfNodeSuFailOverProb=2000
# saAmfNodeSuFailoverMax=2
# saAmfNodeClmNode=p02
# }
and replace p02 with the name of your second machine.
<2>
Locate the following two lines:
saAmfSUHostedByNode=AMF1
# saAmfSUHostedByNode=AMF2

Replace them with:

# saAmfSUHostedByNode=AMF1
saAmfSUHostedByNode=AMF2

Feedback
--------
Any feed-back is appreciated.

Keep in mind only parts of the functionality is implemented. Reports of bugs or
behaviour not compliant with the AMF specification within the implemented part
is greatly appreciated :-).

What is currently NOT implemented ?
-----------------------------------
The following list specifies all chapters of the AMF specification which
currently is NOT fully implemented. The deviations from the specification are
described shortly except in those cases when none of the requirements in the
chapter is implemented.

Chapter: Deviation:
--------- ----------
3.3.1.2 Administrative State Not supported (always UNLOCKED).
3.3.1.4 Readiness State State STOPPING is not supported.
3.3.1.5 Service Unitâs HA State ... State QUIESCING is not supported.
3.3.2.2 Operational State AMF does not detect errors in the
following cases:
â¢ A command used by the Availability
Management Framework to control the
component life cycle returned an
error or did not return in time.
â¢ The component fails to respond in
time to an Availability Management
Framework's callback.
â¢ The component responds to an
Availability Management Framework's
state change callback
(SaAmfCSISetCallbackT) with an error.
â¢ If the component is SA-aware, and it
does not register with the
Availability Management Framework
within the preconfigured time-period
after its instantiation.
â¢ If the component is SA-aware, and it
unexpectedly unregisters with the
Availability Management Framework.
â¢ The component terminates unexpectedly.
â¢ When a fail-over recovery operation
performed at the level of the service
unit or the node containing the
service unit triggers an abrupt
termination of the component.
3.3.2.3 Readiness State State STOPPING is not supported.
3.3.2.4 Componentâs HA State per ... State QUIESCING is not supported.
3.3.3.1 Administrative State Not supported (always UNLOCKED).
3.3.5 Service Group States Administrative state is not supported
(always UNLOCKED).
3.3.6.1 Administrative State Not supported (always UNLOCKED).
3.3.6.2 Operational State None of the rules for transition between states are implemented.
3.3.7 Application States Administrative state is not supported (always UNLOCKED).
3.3.8 Cluster States Administrative state is not supported (always UNLOCKED).
3.5.1 Combined States for Pre-Inst.... Only Administrative state = UNLOCKED is supported.
3.5.2 Combined States for Non-Pre-I... Not supported.
3.6 Component Capability Model Configuration of capability model is
ignored. AMF expects all components to
be capable to be x_active_or_y_standby.
3.7.2 2N Redundancy Model Not supported.
3.7.3.1 Basics Spare service units can not be handled
properly.
3.7.3.3 Configuration â¢ Ordered list of service units for a
service group: Not supported
(the order is unpredictable).
â¢ Ordered list of SIs: Neither ranking
nor dependencies among SIs are
supported. SIs are assigned to SUs in
any order.
â¢ Auto-adjust option: Not supported.
Auto-adjust is never done.
3.7.3.5.1 Handling of a Node Failure.. Not supported.
3.7.3.6 An Example of Auto-adjust Not supported.
3.7.4 N-Way Redundancy Model Not supported.
3.7.5 N-Way Active Redundancy Model Not supported.
3.7.6 No Redundancy Model Not supported.
3.7.7 The Effect of Administrative... Not supported.
3.9 Dependencies Among SIs, Compone.. Not supported.
3.11 Component Monitoring â¢ External Active Monitoring:
Not supported.
3.12.1.1 Error Detection AMF does not support that a component
reports an error for another component.
3.12.1.2 Restart â¢ AMF does not support terminating of
components by the terminate call-back
or the TERMINATE command.
â¢ AMF does not consider component
instantiation-level at restart.
â¢ The configuration option
disableRestart is not supported.
3.12.1.3 Recovery â¢ Component or Service Unit Fail-Over:
â¢ Component fail-over is not
implemented
â¢ Only SU fail-over is implemented and
the only way to trig that case is by
error escalation.
â¢ Node Switch-Over: Not implemented
â¢ Node Fail-Over: Not implemented
â¢ Node Fail-Fast: Not implemented
â¢ The configuration option
recoveryOnFailure is not handled,
i.e. is never evaluated.

3.12.1.4 Repair â¢ The configuration attribute for
automatic repair is not evaluated.
â¢ The administrative operation
SA_AMF_ADMIN_REPAIRED is not
implemented.
â¢ Repair after component fail-over
is not implemented.
â¢ Node leave while performing
automatic repair of that node,
is not implemented.
â¢ Service unit failover recovery:
Is implemented except that an attempt
to repair is always done (confi-
guration attribute is not evaluated).
â¢ Repair after Node Switch-Over,
Fail-Over or Fail-Fast
is not implemented.
3.12.1.5 Recovery Escalation The recommended recovery action is not
evaluated at the reception of an error
report.
3.12.2.1 Recommended Recovery Action The recommended recovery action is
never evaluated. Recovery action
SA_AMF_COMPONENT_RESTART is always
assumed.
3.12.2.2 Escalations of Levels 1 and 2 Is implemented with the following exception:
â¢ The configuration attribute
component_restart_max is compared to
the restart counter of the component
that has reported the error instead of
against the sum of all restart
counters of all components within
the SU.
3.12.2.3 Escalation of Level 3 Not implemented
4.2 CLC-CLI's Environment Variables Translation of non-printable Unicode
characters is not supported.
4.4 INSTANTIATE Command â¢ AMF does not evaluate the exit code of
the INSTANTIATE command as described
in the specification.
â¢ AMF does not supervise that an
SA-aware component registers itself,
within the time limit configured.
As a consequence, none of the recovery
actions described are implemented.
4.5 TERMINATE Command Not supported.
4.6 CLEANUP Command AMF does not evaluate the exit code of
the CLEANUP command and thus does not
implement any recovery action.
4.7 AM_START Command Not supported.
4.8 AM_STOP Command Not supported.
5 Proxied Component Management Not implemented.
7 Administrative API Not implemented
8 Basic Operational Scenarios Not implemented.
9 Alarms and Notifications Not implemented.

Appendix A: Implementation of CLC .. CLC-interfaces are partly implemented
for SA-aware components.
The terminate operation,
saAmfComponentTerminateCallback(),
is never called.
No CLC-interfaces are implemented for
any other type of component.

Appendix B: API functions in Unre.... AMF does not verify that the rules
described are fulfilled.

Which functions of the AMF API is currently NOT implemented ?
-------------------------------------------------------------

Function Deviation
-------- ---------
saAmfComponentUnregister() Is implemented in the library
but not in aisexec.

saAmfPmStart() Is implemented in the library
but not in aisexec.

saAmfPmStop() Is implemented in the library
but not in aisexec.

saAmfHealthcheckStart() This function takes a parameter
of type SaAmfRecommendedRecoveryT.
The value of this parameter is
supposed to specify what kind of
recovery AMF should execute if
the component fails a health
check. AMF does not read the
value of this parameter but
instead always tries to recover
the component by a component
restart.

void (*SaAmfCSIRemoveCallbackT)() AMF will never make a call-back
to this function.
void
(*SaAmfComponentTerminateCallbackT)() AMF will never make a call-back
to this function.
void
(*SaAmfProxiedComponentInstantiateCallbackT)() AMF will never make a call-back
to this function.
void
(*SaAmfProxiedComponentCleanupCallbackT)() AMF will never make a call-back
to this function.
saAmfProtectionGroupTrack() Is implemented in the library
but not in aisexec.

saAmfProtectionGroupTrackStop() Is implemented in the library
but not in aisexec.

void (*SaAmfProtectionGroupTrackCallbackT)() AMF will never make a call-back
to this function.

saAmfProtectionGroupNotificationFree() Not implemented.

saAmfComponentErrorReport() This function takes a parameter
of type SaAmfRecommendedRecoveryT.
The value of this parameter is
supposed to specify what kind of
recovery AMF should execute if
the component fails a health
check. AMF does not read the
value of this parameter but
instead always tries to recover
the component by a component
restart.

saAmfComponentErrorClear() Is implemented in the library
but not in aisexec.