# createrepo_c
C implementation of createrepo
Run `createrepo -h` for usage syntax.
# Devel tips
## Building
Package build requires - Pkg name in Fedora/Ubuntu:
* bzip2 (http://bzip.org/) - bzip2-devel/libbz2-dev
* cmake (http://www.cmake.org/) - cmake/cmake
* expat (http://expat.sourceforge.net/) - expat-devel/libexpat1-dev
* file (http://www.darwinsys.com/file/) - file-devel/libmagic-dev
* glib2 (http://developer.gnome.org/glib/) - glib2-devel/libglib2.0-dev
* libcurl (http://curl.haxx.se/libcurl/) - libcurl-devel/libcurl4-openssl-dev
* libmodulemd (https://github.com/fedora-modularity/libmodulemd/) - libmodulemd-devel/
* libxml2 (http://xmlsoft.org/) - libxml2-devel/libxml2-dev
* python (http://python.org/) - python3-devel/libpython3-dev
* rpm (http://www.rpm.org/) - rpm-devel/librpm-dev
* openssl (http://www.openssl.org/) - openssl-devel/libssl-dev
* sqlite3 (https://sqlite.org/) - sqlite-devel/libsqlite3-dev
* xz (http://tukaani.org/xz/) - xz-devel/liblzma-dev
* zchunk (https://github.com/zchunk/zchunk) - zchunk-devel/
* zlib (http://www.zlib.net/) - zlib-devel/zlib1g-dev
* *Documentation:* doxygen (http://doxygen.org/) - doxygen/doxygen
* *Documentation:* sphinx (http://sphinx-doc.org/) - python-sphinx/python-sphinx
* **Test requires:** check (http://check.sourceforge.net/) - check-devel/check
* **Test requires:** python-nose (https://nose.readthedocs.org/) - python-nose/python-nose
* **Test requires:** xz (http://tukaani.org/xz/) - xz/
* **Test requires:** zchunk (https://github.com/zchunk/zchunk) - zchunk/
* **Experimental support:** drpm (https://github.com/rpm-software-management/drpm) - drpm-devel/
From your checkout dir:
mkdir build
cd build/
cmake ..
make
To build the documentation, from the build/ directory:
make doc
**Note:** For build with debugging symbols you could use (from the build/ directory):
cmake -DCMAKE_BUILD_TYPE:STRING=DEBUG .. && make
## Building from an rpm checkout
E.g. when you want to try weak and rich dependencies.
cmake -DRPM_PATH="/home/tmlcoch/git/rpm" .. && make
**Note:** The RPM must be built in that directory
Commands I am using for building the RPM:
cd /home/tmlcoch/git/rpm
CPPFLAGS='-I/usr/include/nss3/ -I/usr/include/nspr4/' ./autogen.sh --rpmconfigure --with-vendor=redhat --with-external-db --with-lua --with-selinux --with-cap --with-acl --enable-python
make clean && make
## Building with delta rpm support (drpm)
At first, you have to checkout the drpm library from
https://github.com/rpm-software-management/drpm.git and build it.
git clone https://github.com/rpm-software-management/drpm.git
cd drpm/
make
Then run ``cmake`` for createrepo_c with param ``-DDRPM_PATH="/home/tmlcoch/git/drpm"``
where the path is path to your build of drpm library.
cmake -DDRPM_PATH="/home/tmlcoch/git/drpm" .. && make
## Building for a different Python version
By default, cmake should set up things to build for Python 3, but you can do a build for Python 2 like this::
cmake -DPYTHON_DESIRED=2 .
## Other build options
### ``-DENABLE_LEGACY_WEAKDEPS=ON``
Enable legacy SUSE/Mageia/Mandriva weakdeps support (Default: ON)
### ``-DENABLE_THREADED_XZ_ENCODER=ON``
Threaded XZ encoding (Default: OFF)
Note: This option is disabled by default, because Createrepo_c
parallelizes a lot of tasks (including compression) by default; this
only adds extra threads on XZ library level which causes thread bloat
and for most usecases doesn't bring any performance boost.
On regular hardware (e.g. less-or-equal 4 cores) this option may even
cause degradation of performance.
### ``-DWITH_ZCHUNK=ON``
Build with zchunk support (Default: ON)
### ``-DWITH_LIBMODULEMD=ON``
Build with libmodulemd support (Default: ON)
Adds support for working with repos containing
[Fedora Modularity](https://docs.fedoraproject.org/en-US/modularity/) metadata.
## Build tarball
utils/make_tarball.sh [git revision]
Without git revision specified HEAD is used.
## Build Python package
To create a binary "wheel" distribution, use:
python setup.py bdist_wheel
To create a source distribution, use:
python setup.py sdist
Installing source distributions require the installer of the package to have all of the build dependencies installed on their system, since they compile the code during installation. Binary distributions are pre-compiled, but they are likely not portable between substantially different systems, e.g. Fedora and Ubuntu.
Note: if you are building a bdist or installing the sdist on a system with an older version of Pip, you may need to install the ```scikit-build``` Python package first.
To install either of these packages, use:
pip install dist/{{ package name }}
To create an "editable" install of createrepo_c, use:
python setup.py develop
Note: To recompile the libraries and binaries, you muse re-run this command.
## Build RPM package
Modify createrepo_c.spec and run:
utils/make_rpm.sh
Note: [Current .spec for Fedora rawhide](http://pkgs.fedoraproject.org/cgit/createrepo_c.git/plain/createrepo_c.spec)
## Testing
All unit tests run from librepo checkout dir
### Build C tests && run c and python tests
make tests && make test
Note: For a verbose output of testing use: ``make ARGS="-V" test``
### Run only C unittests (from your checkout dir):
build/tests/run_gtester.sh
Note: The C tests have to be built by ``make tests``)!
### Run only Python unittests (from your checkout dir):
PYTHONPATH=`readlink -f ./build/src/python/` nosetests -s tests/python/tests/
Or, for an alternative python version, specify the appropriate nosetests executable:
PYTHONPATH=`readlink -f ./build/src/python/` nosetests-3.4 -s tests/python/tests/
Note: When compiling createrepo_c without libmodulemd support add ``WITH_LIBMODULEMD=OFF``
### Links
[Bugzilla](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=MODIFIED&bug_status=VERIFIED&component=createrepo_c&query_format=advanced)
### Important notes
In original createrepo ``sha`` is a nickname for the ``sha1`` checksum.
Createrepo_c mimics this behaviour.
---------------------------------------------------
# Differences in behavior between createrepo_c and createrepo
## Checksums after update
### Use case:
- Repodata in repo/ are has checksum xxx
- Params: --update --checksum=yyy repo/
### createrepo_c result:
- All package checksums are recalculated into yyy
### original createrepo result:
- Only new and changed packages has yyy checksums
other packages has still xxx checksums
## Skip symlinks param
### Use case:
- Some packages in repo/ are symlinks
- Params: --skip-symlinks repo/
### createrepo_c result:
- Symlinked packages are ignored
### original createrepo result:
- Symlinked packages are processed
(https://bugzilla.redhat.com/show_bug.cgi?id=828848)
## Base path from update-md-path repo
### Use case:
- A somebody else's repo is somewhere
- The repo items have set a base path to http://foo.com/
- We want to create metadata for our repo
- Some packages in our repo are same as packages in somebody else's repo
- We want to speed up creation of our repodata with
combo --update and --update-md-path=somebody_else's_repo
- Params: --update --update-md-path=ftp://somebody.else/repo our_repo/
### createrepo_c results:
- All our packages have no base path set (if we don't set --baseurl explicitly)
### original createrepo result:
- Some packages in metadata (which was same in our repo and in somebody
else's repo) have base path set to http://foo.com/
- (https://bugzilla.redhat.com/show_bug.cgi?id=875029)
## Crippled paths in filelists.xml after update
### Use case:
- A repo with old metadata exists
- We want to update metadata
- Params: --update repo/
### createrepo_c results:
- All is fine
### original createrepo result:
- Some paths in filelists.xml are crippled
(https://bugzilla.redhat.com/show_bug.cgi?id=835565)
## --update leaves behind some old repodata files
### Use case:
- A repo with repodata created with --simple-md-filenames exists
- We want to update repodata to have checksums in filenames
- Params: --update repo/
### createrepo_c results:
- All repodata contains checksum in the name
### original createrepo result:
- All repodata contains checksum in the name
- There are old metadata without checksum in the name too
- (https://bugzilla.redhat.com/show_bug.cgi?id=836917)
## Modifyrepo_c
Modifyrepo_c is compatible with classical Modifyrepo except some misbehaviour:
* TODO: Report bugs and add reference here
### Batch file
When there is need to do several modification to repository (``repomd.xml``)
a batch file could be used.
> Batch file is Modifyrepo_c specific. It is not supported by the classical
Modifyrepo - at least not yet.
#### Example
# Add:
# [<path/to/file>]
# <options>
# Metadata that use a bunch of config options
[some/path/comps.xml]
type=group
compress=true
compress-type=gz
unique-md-filenames=true
checksum=sha256
new-name=group.xml
# Metadata that use default settings
[some/path/bar.xml]
# Remove:
# [<metadata name>]
# remove=true
[updateinfo]
remove=true
#### Supported options
| Option name | Description | Supported value(s) | Default |
|---------------|-------------|--------------------|---------|
| path | Path to the file. When specified it override the path specified in group name (name between [] parenthesis) | Any string | group name (string between '[' ']') |
| type | Type of the metadata | Any string | Based on filename |
| remove | Remove specified file/type from repodata | ``true`` or ``false`` | ``false`` |
| compress | Compress the new metadata before adding it to repo | ``true`` or ``false`` | ``true`` |
| compress-type | Compression format to use | ``gz``, ``bz2``, ``xz`` | ``gz`` |
| checksum | Checksum type to use | ``md5``, ``sha``, ``sha1``, ``sha224``, ``sha256``, ``sha384``, ``sha512`` | ``sha256`` |
| unique-md-filenames | Include the file's checksum in the filename | ``true`` or ``false`` | ``true`` |
| new-name | New name for the file. If ``compress`` is ``true``, then compression suffix will be appended. If ``unique-md-filenames`` is ``true``, then checksum will be prepended. | Any string | Original source filename |
#### Notes
* Lines beginning with a '#' and blank lines are considered comments.
* If ``remove=true`` is used, no other config options should be used
