% -*- latex -*-
##
#use "topfind";;
#require "caml2html";;
#require "atd";;
#require "unix";;
#use "../ag_version.ml";;
#use "macros.ml";;
##
\section{Introduction}
Atdgen is a command-line program that takes as input type definitions
in the \href{http://mjambon.com/atd}{ATD} syntax and produces OCaml
code suitable for data serialization and deserialization.
Two data formats are currently supported, these are
\href{http://mjambon.com/biniou.html}{biniou}
and \href{http://json.org/}{JSON}. Atdgen-biniou and Atdgen-json will
refer to Atdgen used in one context or the other.
Atdgen was designed with efficiency and durability in mind. Software
authors are encouraged to use Atdgen directly and to write
tools that may reuse part of Atdgen's source code.
Atdgen uses the following packages that were developed in conjunction
with Atdgen:
\begin{itemize}
\item \texttt{atd}: parser for the syntax of type definitions
\item \texttt{biniou}: parser and printer for biniou, a binary
extensible data format
\item \href{http://mjambon.com/yojson.html}{\texttt{yojson}}:
parser and printer for JSON, a widespread text-based data format
\end{itemize}
Atdgen does not use Camlp4.
\section{Command-line usage}
\subsection{Command-line help}
\begin{verbatim}
$ atdgen -help
\end{verbatim}
%$
## shell "cd ..; ./atdgen -help" ##
\subsection{Atdgen-biniou example}
\begin{verbatim}
$ atdgen -t example.atd
$ atdgen -b example.atd
\end{verbatim}
%$
Input file \texttt{example.atd}:
## atdgen_biniou_mli ~prefix:"example" "is used to produce
files \\texttt{example\\_t.mli},
\\texttt{example\\_t.ml},
\\texttt{example\\_b.mli} and
\\texttt{example\\_b.ml}.
This is \\texttt{example\\_b.mli}:" ##
type profile = {
id : string;
email : string;
~email_validated : bool;
name : string;
?real_name : string option;
~about_me : string list;
?gender : gender option;
?date_of_birth : date option;
}
type gender = [ Female | Male ]
type date = {
year : int;
month : int;
day : int;
}
## () ##
Module \texttt{Example\_t} (files \texttt{example\_t.mli} and
\texttt{example\_t.ml}) contains all OCaml type definitions that
can be used independently from Biniou or JSON.
For convenience, these definitions are also made available from the
\texttt{Example\_b} module whose interface is shown above.
Any type name, record field name or variant constructor can be
referred to using either module. For example, the OCaml
expressions \texttt{((x : Example\_t.date) : Example\_b.date)}
and \texttt{x.Example\_t.year = x.Example\_b.year} are both valid.
\subsection{Atdgen-json example}
\begin{verbatim}
$ atdgen -t example.atd
$ atdgen -j example.atd
\end{verbatim}
%$
Input file \texttt{example.atd}:
## atdgen_json_mli ~prefix:"example" "is used to produce
files \\texttt{example\\_t.mli},
\\texttt{example\\_t.ml},
\\texttt{example\\_j.mli} and
\\texttt{example\\_j.ml}.
This is \\texttt{example\\_j.mli}:" ##
type profile = {
id : string;
email : string;
~email_validated : bool;
name : string;
?real_name : string option;
~about_me : string list;
?gender : gender option;
?date_of_birth : date option;
}
type gender = [ Female | Male ]
type date = {
year : int;
month : int;
day : int;
}
## () ##
Module \texttt{Example\_t} (files \texttt{example\_t.mli} and
\texttt{example\_t.ml}) contains all OCaml type definitions that
can be used independently from Biniou or JSON.
For convenience, these definitions are also made available from the
\texttt{Example\_j} module whose interface is shown above.
Any type name, record field name or variant constructor can be
referred to using either module. For example, the OCaml
expressions \texttt{((x : Example\_t.date) : Example\_j.date)}
and \texttt{x.Example\_t.year = x.Example\_j.year} are both valid.
\subsection{Validator example}
\begin{verbatim}
$ atdgen -t example.atd
$ atdgen -v example.atd
\end{verbatim}
Input file \texttt{example.atd}:
## atdgen_validate_ml ~prefix:"example" "is used to produce
files \\texttt{example\\_t.mli},
\\texttt{example\\_t.ml},
\\texttt{example\\_v.mli} and
\\texttt{example\\_v.ml}.
This is \\texttt{example\\_v.ml}, showing how the user-specified
validators are used:" ##
type month = int <ocaml validator="fun x -> x >= 1 && x <= 12">
type day = int <ocaml validator="fun x -> x >= 1 && x <= 31">
type date = {
year : int;
month : month;
day : day;
}
<ocaml validator="Date_util.validate_date">
## () ##
\section{Default type mapping}
The following table summarizes the default mapping between ATD types and
OCaml, biniou and JSON data types. For each language more
representations are available and are detailed in the next section of this
manual.
\makebox[\textwidth]{%
\begin{tabular}{llll}
\hline
ATD & OCaml & Biniou & JSON \\
\hline
\tt unit & \tt unit & unit & null \\
\tt bool &\tt bool & bool & boolean \\
\tt int & \tt int & svint & number (int) \\
\tt float & \tt float & float64 & number (not int) \\
\tt string & \tt string & string & string \\
\tt option & \tt option & numeric variants (tag 0) & None/Some variants \\
\tt list & \tt list & array & array \\
\tt shared & no wrapping & shared & not implemented \\
variants & polymorphic variants & regular variants & variants \\
record & record & record & object \\
tuple & tuple & tuple & tuple \\
\hline
\end{tabular}%
}
Notes:
\begin{itemize}
\item The JSON null value serves only as the unit value and is
useful in practice only for instanciating parametrized types with
``nothing''. Option types have a distinct representation that does
not use the null value.
\item OCaml floats are written to JSON numbers with either a decimal
point or an exponent such that they are distinguishable from
ints, even though the JSON standard does not require a distinction
between the two.
\item The optional values of record fields denoted in ATD by a
question mark are unwrapped or omitted in both biniou and JSON.
\item JSON option values and JSON variants are represented in standard
JSON ({\tt atdgen -j -j-std}) by a single string e.g. {\tt "None"}
or a pair in which the
first element is the name (constructor) e.g. {\tt ["Some", 1234]}.
Yojson also provides a specific syntax for variants using edgy
brackets: {\tt <"None">}, {\tt <"Some": 1234>}.
\item Biniou field names and variant names other than the option types
use the hash of the ATD field or variant name and cannot currently
be overridden by annotations.
\item JSON tuples in standard JSON ({\tt atdgen -j -j-std}) use the
array notation e.g.
{\tt ["ABC", 123]}.
Yojson also provides a specific syntax for tuples using parentheses,
e.g. {\tt ("ABC", 123)}.
\item Types defined as {\tt abstract} are defined in
another module.
\end{itemize}
\section{ATD Annotations}
## annot_section "biniou" ##
## annot_field "repr" ##
\annotpar{Integers}
\annotposition after \texttt{int} type
\annotvalues \texttt{svint} (default), \texttt{uvint}, \texttt{int8},
\texttt{int16}, \texttt{int32}, \texttt{int64}
\annotsemantics specifies an alternate type for representing integers.
The default type is \texttt{svint}.
The other integers types provided by biniou are
supported by Atdgen-biniou.
They have to map to the corresponding OCaml types
in accordance with the following table:
\begin{tabular}{lll}
\hline
Biniou type & Supported OCaml type & OCaml value range \\
\hline
\tt svint & \tt int & \tt min\_int $\dots$ max\_int \\
\tt uvint & \tt int & {\tt 0 $\dots$ max\_int}, {\tt min\_int $\dots$ -1} \\
\tt int8 & \tt char & \tt '\bs000' $\dots$ '\bs255' \\
\tt int16 & \tt int & \tt 0 $\dots$ 65535 \\
\tt int32 & \tt int32 & \tt Int32.min\_int $\dots$ Int32.max\_int \\
\tt int64 & \tt int64 & \tt Int64.min\_int $\dots$ Int64.max\_int \\
\hline
\end{tabular}
In addition to the mapping above, if the OCaml type is \texttt{int},
any biniou integer type can be read into OCaml data regardless of the
declared biniou type.
\annotexample
## atdgen () ##
type t = {
id : int
<ocaml repr="int64">
<biniou repr="int64">;
data : string list;
}
## () ##
\annotpar{Floating-point numbers}
\annotposition after \texttt{float} type
\annotvalues \texttt{float64} (default), \texttt{float32}
\annotsemantics \texttt{float32} allows for a shorter serialized representation
of floats, using 4 bytes instead of 8, with reduced precision.
OCaml floats always use 8 bytes, though.
\annotexample
## atdgen () ##
type t = {
lat : float <biniou repr="float32">;
lon : float <biniou repr="float32">;
}
## () ##
\annotpar{Arrays and tables}
\annotposition applies to lists of records
\annotvalues \texttt{array} (default), \texttt{table}
\annotsemantics \texttt{table} uses biniou's table format instead of a
regular array for serializing OCaml data into biniou.
Both formats are supported for reading into OCaml data
regardless of the annotation. The table format allows
\annotexample
## atdgen () ##
type item = {
id : int;
data : string list;
}
type items = item list <biniou repr="table">
## () ##
## annot_section "json" ##
## annot_field "name" ##
\annotposition after field name or variant name
\annotvalues any string making a valid JSON string value
\annotsemantics specifies an alternate object field name or variant
name to be used by the JSON representation.
\annotexample
## atdgen_json () ##
type color = [
Black <json name="black">
| White <json name="white">
| Grey <json name="grey">
]
type profile = {
id <json name="ID"> : int;
username : string;
background_color : color;
}
## () ##
A valid JSON object of the \texttt{profile} type above is:
\begin{verbatim}
{
"ID": 12345678,
"username": "kimforever",
"background_color": "black"
}
\end{verbatim}
## annot_field "repr" ##
\annotpar{Association lists}
\annotposition after \texttt{(string * \_) list} type
\annotvalues \texttt{object}
\annotsemantics uses JSON's object notation to represent association
lists.
\annotexample
## atdgen () ##
type counts = (string * int) list <json repr="object">
## () ##
A valid JSON object of the \texttt{counts} type above is:
\begin{verbatim}
{
"bob": 3,
"john": 1408,
"mary": 450987,
"peter": 93087
}
\end{verbatim}
Without the annotation \texttt{<json repr="object">}, the data above
would be represented as:
\begin{verbatim}
[
[ "bob", 3 ],
[ "john", 1408 ],
[ "mary", 450987 ],
[ "peter", 93087 ]
]
\end{verbatim}
\annotpar{Floats}
\annotposition after \texttt{float} type
\annotvalues \texttt{int}
\annotsemantics specifies a float value that must be rounded to the
nearest integer and represented in JSON without
a decimal point nor an exponent.
\annotexample
## atdgen () ##
type unixtime = float <json repr="int">
## () ##
## annot_section "ocaml" ##
## annot_field "predef" ##
\annotposition left-hand side of a type definition, after the type name
\annotvalues none, \texttt{true} or \texttt{false}
\annotsemantics this flag indicates that the corresponding OCaml
type definition must be omitted.
\annotexample
## ocaml () ##
(* Some third-party OCaml code *)
type message = {
from : string;
subject : string;
body : string;
}
## () ##
## atdgen () ##
(*
Our own ATD file used for making message_of_string and
string_of_message functions.
*)
type message <ocaml predef> = {
from : string;
subject : string;
body : string;
}
## () ##
## annot_field "mutable" ##
\annotposition after a record field name
\annotvalues none, \texttt{true} or \texttt{false}
\annotsemantics this flag indicates that the corresponding OCaml
record field is mutable.
\annotexample
## atdgen () ##
type counter = {
total <ocaml mutable> : int;
errors <ocaml mutable> : int;
}
## () ##
translates to the following OCaml definition:
## ocaml () ##
type counter = {
mutable total : int;
mutable errors : int;
}
## () ##
## annot_field "default" ##
\annotposition after a record field name marked with a \texttt{\~{}}
symbol or at the beginning of a tuple field.
\annotvalues any valid OCaml expression
\annotsemantics specifies an explicit default value for a field of an
OCaml record or tuple, allowing that field to be omitted.
\annotexample
## atdgen () ##
type color = [ Black | White | Rgb of (int * int * int) ]
type ford_t = {
year : int;
~color <ocaml default="`Black"> : color;
}
type point = (int * int * <ocaml default="0"> : int)
## () ##
## annot_field "from" ##
\annotposition left-hand side of a type definition, after the type name
\annotvalues OCaml module name without the \texttt{\_t}, \texttt{\_b},
\texttt{\_j} or \texttt{\_v}
suffix. This can be also seen as the name of the original ATD file,
without the \texttt{.atd} extension and capitalized like an OCaml
module name.
\annotsemantics specifies the base name of the OCaml modules
where the type and values coming with that type are defined.
It is useful for ATD types defined as
\texttt{abstract} and for types annotated as predefined using
the annotation \texttt{<ocaml predef>}.
In both cases, the missing definitions must be provided by
modules composed of the base name and the standard suffix assumed by
Atdgen which is
\texttt{\_t}, \texttt{\_b},
\texttt{\_j} or \texttt{\_v}.
\annotexample
First input file \texttt{part1.atd}:
## atdgen () ##
type point = { x : int; y : int }
## () ##
Second input file \texttt{part2.atd} depending on the first one:
## atdgen () ##
type point <ocaml from="Part1"> = abstract
type points = point list
## () ##
## annot_field "module" ##
In most cases since Atdgen 1.2.0
\texttt{module} annotations are deprecated in favor of \texttt{from}
annotations previously described.
\annotposition left-hand side of a type definition, after the type name
\annotvalues OCaml module name
\annotsemantics specifies the OCaml module where the type and values
coming with that type are defined. It is useful for ATD types defined as
\texttt{abstract} and for types annotated as predefined using
the annotation \texttt{<ocaml predef>}.
In both cases, the missing definitions can be provided either by
globally opening an OCaml module with an OCaml directive or by specifying
locally the name of the module to use.
The latter approach is recommended because it allows to create
type and value aliases in the OCaml module being generated. It results
in a complete module signature regardless of the external
nature of some items.
\annotexample
Input file \texttt{example.atd}:
## atdgen () ##
type document <ocaml module="Doc"> = abstract
type color <ocaml predef module="Color"> =
[ Black | White ] <ocaml repr="classic">
type point <ocaml predef module="Point"> = {
x : float;
y : float;
}
## () ##
gives the following OCaml type definitions
(file \texttt{example.mli}):
## ocaml () ##
type document = Doc.document
type color = Color.color = Black | White
type point = Point.point = { x: float; y: float }
## () ##
Now for instance \texttt{Example.Black} and \texttt{Color.Black}
can be used interchangeably in other modules.
## annot_field "t" ##
\annotposition left-hand side of a type definition, after the type
name. Must be used in conjunction with a \texttt{module} field.
\annotvalues OCaml type name as found in an external module.
\annotsemantics This option allows to specify the name of an
OCaml type defined in an external module.
It is useful when the type needs to be renamed because its original
name is already in use or not enough informative.
Typically we may want to give the name \texttt{foo} to a type
originally defined in OCaml as \texttt{Foo.t}.
\annotexample
## atdgen () ##
type foo <ocaml_biniou module="Foo" t="t"> = abstract
type bar <ocaml_biniou module="Bar" t="t"> = abstract
type t <ocaml_biniou module="Baz"> = abstract
## () ##
allows local type names to be unique
and gives the following OCaml type definitions:
## ocaml () ##
type foo = Foo.t
type bar = Bar.t
type t = Baz.t
## () ##
## annot_field ~label:"field-prefix" "field\\_prefix" ##
\annotposition record type expression
\annotvalues any string making a valid prefix for OCaml record field names
\annotsemantics specifies a prefix to be prepended to each field of
the OCaml definition of the record. Overridden by alternate field
names defined on a per-field basis.
\annotexample
## atdgen () ##
type point2 = {
x : int;
y : int;
} <ocaml field_prefix="p2_">
## () ##
gives the following OCaml type definition:
## ocaml () ##
type point2 = {
p2_x : int;
p2_y : int;
}
## () ##
## annot_field "name" ##
\annotposition after record field name or variant name
\annotvalues any string making a valid OCaml record field name or
variant name
\annotsemantics specifies an alternate record field name or variant
names to be used in OCaml.
\annotexample
## atdgen () ##
type color = [
Black <ocaml name="Grey0">
| White <ocaml name="Grey100">
| Grey <ocaml name="Grey50">
]
type profile = {
id <ocaml name="profile_id"> : int;
username : string;
}
## () ##
gives the following OCaml type definitions:
## ocaml () ##
type color = [
`Grey0
| `Grey100
| `Grey50
]
type profile = {
profile_id : int;
username : string;
}
## () ##
## annot_field "repr" ##
\annotpar{Integers}
\annotposition after \texttt{int} type
\annotvalues \texttt{char}, \texttt{int32}, \texttt{int64}, \texttt{float}
\annotsemantics specifies an alternate type for representing integers.
The default type is \texttt{int}, but \texttt{char}, \texttt{int32},
\texttt{int64} or \texttt{float} can be used instead.
The three types \texttt{char}, \texttt{int32} and \texttt{int64} are
supported by both Atdgen-biniou and Atdgen-json but Atdgen-biniou
currently requires that they map to the corresponding fixed-width
types provided by the biniou format.
The type \texttt{float} is only supported in conjunction with JSON and
is useful when an OCaml float is used to represent an integral value,
such as a time in seconds returned by \texttt{Unix.time()}. When
converted into JSON, floats are rounded to the nearest integer.
\annotexample
## atdgen () ##
type t = {
id : int
<ocaml repr="int64">
<biniou repr="int64">;
data : string list;
}
## () ##
\annotpar{Lists and arrays}
\annotposition after a \texttt{list} type
\annotvalues \texttt{array}
\annotsemantics maps to OCaml's \texttt{array} type instead of \texttt{list}.
\annotexample
## atdgen () ##
type t = {
id : int;
data : string list
<ocaml repr="array">;
}
## () ##
\annotpar{Sum types}
\annotposition after a sum type (denoted by square brackets)
\annotvalues \texttt{classic}
\annotsemantics maps to OCaml's classic variants instead of
polymorphic variants.
\annotexample
## atdgen () ##
type fruit = [ Apple | Orange ] <ocaml repr="classic">
## () ##
translates to the following OCaml type definition:
## ocaml () ##
type fruit = Apple | Orange
## () ##
\annotpar{Shared values}
\annotposition after a \texttt{shared} type
\annotvalues \texttt{ref}
\annotsemantics wraps the value using OCaml's \texttt{ref} type,
which is as of Atdgen 1.1.0
the only way of sharing values other than records.
\annotexample
## atdgen () ##
type shared_string = string shared <ocaml repr="ref">
## () ##
translates to the following OCaml type definition:
## ocaml () ##
type shared_string = string ref
## () ##
## annot_field "validator" ##
\annotposition after any type expression except type variables
\annotvalues OCaml function that takes one argument of the given type
and returns a bool
\annotsemantics \texttt{atdgen -v} produces for each type named
\textit{t} a function \texttt{validate\_}\textit{t}:
## ocaml () ##
val validate_t : t -> bool
## () ##
Such a function returns true if and only if the value and all of its
subnodes pass all the validators specified by annotations of the form
\texttt{<ocaml validator="...">}.
\annotexample
## atdgen () ##
type positive = int <ocaml validator="fun x -> x > 0">
type point = {
x : positive;
y : positive;
z : int;
}
<ocaml validator="Point.validate">
(* Some validating function from a user-defined module Point *)
## () ##
The generated \texttt{validate\_point} function is equivalent to the
following:
## ocaml () ##
let validate_point p =
Point.validate p
&& (fun x -> x > 0) p.x
&& (fun x -> x > 0) p.y
## () ##
## annot_section ~label:"ocaml-biniou" "ocaml\\_biniou" ##
Section \texttt{ocaml\_biniou} takes precedence over section \texttt{ocaml}
in Biniou mode (\texttt{-b}) for the following fields:
\begin{itemize}
\item \texttt{predef} (see \ref{ocaml.predef})
\item \texttt{module} (see \ref{ocaml.module})
\item \texttt{t} (see \ref{ocaml.t})
\end{itemize}
## annot_section ~label:"ocaml-json" "ocaml\\_json" ##
Section \texttt{ocaml\_json} takes precedence over section \texttt{ocaml}
in JSON mode (\texttt{-j}) for the following fields:
\begin{itemize}
\item \texttt{predef} (see \ref{ocaml.predef})
\item \texttt{module} (see \ref{ocaml.module})
\item \texttt{t} (see \ref{ocaml.t})
\end{itemize}
\annotexample
This example shows how to parse a field into a generic tree
of type \texttt{Yojson.Safe.json} rather than a value of a specialized
OCaml type.
## atdgen () ##
type dyn <ocaml_json module="Yojson.Safe" t="json"> = abstract
type t = { foo: int; bar: dyn }
## () ##
translates to the following OCaml type definitions:
## ocaml () ##
type dyn = Yojson.Safe.json
type t = { foo : int; bar : dyn }
## () ##
Sample OCaml value of type \texttt{t}:
## ocaml () ##
{
foo = 12345;
bar =
`List [
`Int 12;
`String "abc";
`Assoc [
"x", `Float 3.14;
"y", `Float 0.0;
"color", `List [ `Float 0.3; `Float 0.0; `Float 1.0 ]
]
]
}
## () ##
Corresponding JSON data as obtained with \texttt{string\_of\_t}:
\begin{verbatim}
{"foo":12345,"bar":[12,"abc",{"x":3.14,"y":0.0,"color":[0.3,0.0,1.0]}]}
\end{verbatim}
## annot_section "doc" ##
Unlike comments, \texttt{doc} annotations are meant to be
propagated into the generated source code. This is useful for
making generated interface files readable without having to consult
the original ATD file.
Generated source code comments can comply to a standard format and
take advantage of documentation generators such as javadoc or
ocamldoc.
## annot_field "text" ##
\annotposition
\begin{itemize}
\item after the type name on the left-hand side of a type definition
\item after the type expression on the right hand of a type definition
(but not after any type expression)
\item after record field names
\item after variant names
\end{itemize}
\annotvalues UTF-8-encoded text using a minimalistic markup language
\annotsemantics The markup language is defined as follows:
\begin{itemize}
\item Blank lines separate paragraphs.
\item \verb!{{ }}! can be used to enclose inline verbatim text.
\item \verb!{{{ }}}! can be used to enclose verbatim text where
whitespace is preserved.
\item The backslash character is used to escape special character sequences.
In regular paragraph mode the special sequences are [\bs], [{{] and [{{{].
In inline verbatim text, special sequences are [\bs] and [}}].
In verbatim text, special sequences are [\bs] and [}}}].
\end{itemize}
\annotexample The following is a full example demonstrating the use of
\texttt{doc} annotations but also shows the full interface
file \texttt{genealogy.mli} generated using:
\begin{verbatim}
$ atdgen -b genealogy.atd
\end{verbatim}
%$
Input file \texttt{genealogy.atd}:
## atdgen_biniou_mli ~prefix:"genealogy"
"translates using \\texttt{atdgen -b genealogy.atd}
into the following OCaml interface
file \\texttt{genealogy\\_b.mli} with ocamldoc-compliant comments:" ##
<doc text="Type definitions for family trees">
type tree = {
members : person list;
filiations : filiation list;
}
type filiation = {
parent : person_id;
child : person_id;
filiation_type : filiation_type;
}
<doc text="Connection between parent or primary caretaker and child">
type filiation_type = {
?genetic : bool option;
?pregnancy : bool option;
?raised_from_birth : bool option;
?raised : bool option;
?stepchild : bool option;
?adopted : bool option;
}
<doc text="
Example of a father who raised his child from birth
but may not be the biological father:
{{{
{
genetic = None;
pregnancy = Some false;
raised_from_birth = Some true;
raised = Some true;
stepchild = Some false;
adopted = Some false;
}
}}}
">
type person_id
<doc text="Two persons with the same {{person_id}} must be the same
person. Two persons with different {{person_id}}s
may be the same person if there is not enough evidence to
support it."> = int
type person = {
person_id : person_id;
name : string;
~gender : gender list;
?biological_gender
<doc text="Biological gender actually used for procreating"> :
gender option;
}
type gender =
[
| F <doc text="female">
| M <doc text="male">
]
<doc text="Gender, definition depending on the context">
## () ##
\section{Library}
A library named \texttt{atdgen} is installed by the standard
installation process. Only a fraction of it is officially supported
and documented. The documentation is available online at
\url{##= odoc_url ##}.
