pgcrypto - cryptographic functions for PostgreSQL
=================================================
Marko Kreen <markokr@gmail.com>
// Note: this document is in asciidoc format.
1. Installation
-----------------
Run following commands:
make
make install
make installcheck
The `make installcheck` command is important. It runs regression tests
for the module. They make sure the functions here produce correct
results.
Next, to put the functions into a particular database, run the commands in
file pgcrypto.sql, which has been installed into the shared files directory.
Example using psql:
psql -d DBNAME -f pgcrypto.sql
2. Notes
----------
2.1. Configuration
~~~~~~~~~~~~~~~~~~~~
pgcrypto configures itself according to the findings of main PostgreSQL
`configure` script. The options that affect it are `--with-zlib` and
`--with-openssl`.
When compiled with zlib, PGP encryption functions are able to
compress data before encrypting.
When compiled with OpenSSL there will be more algorithms available.
Also public-key encryption functions will be faster as OpenSSL
has more optimized BIGNUM functions.
Summary of functionality with and without OpenSSL:
`----------------------------`---------`------------
Functionality built-in OpenSSL
----------------------------------------------------
MD5 yes yes
SHA1 yes yes
SHA224/256/384/512 yes yes (3)
Any other digest algo no yes (1)
Blowfish yes yes
AES yes yes (2)
DES/3DES/CAST5 no yes
Raw encryption yes yes
PGP Symmetric encryption yes yes
PGP Public-Key encryption yes yes
----------------------------------------------------
1. Any digest algorithm OpenSSL supports is automatically picked up.
This is not possible with ciphers, which need to be supported
explicitly.
2. AES is included in OpenSSL since version 0.9.7. If pgcrypto is
compiled against older version, it will use built-in AES code,
so it has AES always available.
3. SHA2 algorithms were added to OpenSSL in version 0.9.8. For
older versions, pgcrypto will use built-in code.
2.2. NULL handling
~~~~~~~~~~~~~~~~~~~~
As standard in SQL, all functions return NULL, if any of the arguments
are NULL. This may create security risks on careless usage.
2.3. Security
~~~~~~~~~~~~~~~
All the functions here run inside database server. That means that all
the data and passwords move between pgcrypto and client application in
clear-text. Thus you must:
1. Connect locally or use SSL connections.
2. Trust both system and database administrator.
If you cannot, then better do crypto inside client application.
3. General hashing
--------------------
3.1. digest(data, type)
~~~~~~~~~~~~~~~~~~~~~~~~~
digest(data text, type text) RETURNS bytea
digest(data bytea, type text) RETURNS bytea
Type is here the algorithm to use. Standard algorithms are `md5` and
`sha1`, although there may be more supported, depending on build
options.
Returns binary hash.
If you want hexadecimal string, use `encode()` on result. Example:
CREATE OR REPLACE FUNCTION sha1(bytea) RETURNS text AS $$
SELECT encode(digest($1, 'sha1'), 'hex')
$$ LANGUAGE SQL STRICT IMMUTABLE;
3.2. hmac(data, key, type)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hmac(data text, key text, type text) RETURNS bytea
hmac(data bytea, key text, type text) RETURNS bytea
Calculates Hashed MAC over data. `type` is the same as in `digest()`.
If the key is larger than hash block size it will first hashed and the
hash will be used as key.
It is similar to digest() but the hash can be recalculated only knowing
the key. This avoids the scenario of someone altering data and also
changing the hash.
Returns binary hash.
4. Password hashing
---------------------
The functions `crypt()` and `gen_salt()` are specifically designed
for hashing passwords. `crypt()` does the hashing and `gen_salt()`
prepares algorithm parameters for it.
The algorithms in `crypt()` differ from usual hashing algorithms like
MD5 or SHA1 in following respects:
1. They are slow. As the amount of data is so small, this is only
way to make brute-forcing passwords hard.
2. Include random 'salt' with result, so that users having same
password would have different crypted passwords. This is also
additional defense against reversing the algorithm.
3. Include algorithm type in the result, so passwords hashed with
different algorithms can co-exist.
4. Some of them are adaptive - that means after computers get
faster, you can tune the algorithm to be slower, without
introducing incompatibility with existing passwords.
Supported algorithms:
`------`-------------`---------`----------`---------------------------
Type Max password Adaptive Salt bits Description
----------------------------------------------------------------------
`bf` 72 yes 128 Blowfish-based, variant 2a
`md5` unlimited no 48 md5-based crypt()
`xdes` 8 yes 24 Extended DES
`des` 8 no 12 Original UNIX crypt
----------------------------------------------------------------------
4.1. crypt(password, salt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
crypt(password text, salt text) RETURNS text
Calculates UN*X crypt(3) style hash of password. When storing new
password, you need to use function `gen_salt()` to generate new salt.
When checking password you should use existing hash as salt.
Example - setting new password:
UPDATE .. SET pswhash = crypt('new password', gen_salt('md5'));
Example - authentication:
SELECT pswhash = crypt('entered password', pswhash) WHERE .. ;
returns true or false whether the entered password is correct.
It also can return NULL if `pswhash` field is NULL.
4.2. gen_salt(type)
~~~~~~~~~~~~~~~~~~~~~
gen_salt(type text) RETURNS text
Generates a new random salt for usage in `crypt()`. For adaptible
algorithms, it uses the default iteration count.
Accepted types are: `des`, `xdes`, `md5` and `bf`.
4.3. gen_salt(type, rounds)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gen_salt(type text, rounds integer) RETURNS text
Same as above, but lets user specify iteration count for some
algorithms. The higher the count, the more time it takes to hash
the password and therefore the more time to break it. Although with
too high count the time to calculate a hash may be several years
- which is somewhat impractical.
Number is algorithm specific:
`-----'---------'-----'----------
type default min max
---------------------------------
`xdes` 725 1 16777215
`bf` 6 4 31
---------------------------------
In case of xdes there is a additional limitation that the count must be
a odd number.
Notes:
- Original DES crypt was designed to have the speed of 4 hashes per
second on the hardware of that time.
- Slower than 4 hashes per second would probably dampen usability.
- Faster than 100 hashes per second is probably too fast.
- See next section about possible values for `crypt-bf`.
4.4. Comparison of crypt and regular hashes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is a table that should give overview of relative slowness
of different hashing algorithms.
* The goal is to crack a 8-character password, which consists:
1. Only of lowercase letters
2. Numbers, lower- and uppercase letters.
* The table below shows how much time it would take to try all
combinations of characters.
* The `crypt-bf` is featured in several settings - the number
after slash is the `rounds` parameter of `gen_salt()`.
`------------'----------'--------------'--------------------
Algorithm Hashes/sec Chars: [a-z] Chars: [A-Za-z0-9]
------------------------------------------------------------
crypt-bf/8 28 246 years 251322 years
crypt-bf/7 57 121 years 123457 years
crypt-bf/6 112 62 years 62831 years
crypt-bf/5 211 33 years 33351 years
crypt-md5 2681 2.6 years 2625 years
crypt-des 362837 7 days 19 years
sha1 590223 4 days 12 years
md5 2345086 1 day 3 years
------------------------------------------------------------
* The machine used is 1.5GHz Pentium 4.
* crypt-des and crypt-md5 algorithm numbers are taken from
John the Ripper v1.6.38 `-test` output.
* MD5 numbers are from mdcrack 1.2.
* SHA1 numbers are from lcrack-20031130-beta.
* `crypt-bf` numbers are taken using simple program that loops
over 1000 8-character passwords. That way I can show the speed with
different number of rounds. For reference: `john -test` shows 213
loops/sec for crypt-bf/5. (The small difference in results is in
accordance to the fact that the `crypt-bf` implementation in pgcrypto
is same one that is used in John the Ripper.)
Note that "try all combinations" is not a realistic exercise.
Usually password cracking is done with the help of dictionaries, which
contain both regular words and various mutations of them. So, even
somewhat word-like passwords could be cracked much faster than the above
numbers suggest, and a 6-character non-word like password may escape
cracking. Or not.
5. PGP encryption
-------------------
The functions here implement the encryption part of OpenPGP (RFC2440)
standard. Supported are both symmetric-key and public-key encryption.
5.1. Overview
~~~~~~~~~~~~~~~
Encrypted PGP message consists of 2 packets:
- Packet for session key - either symmetric- or public-key encrypted.
- Packet for session-key encrypted data.
When encrypting with password:
1. Given password is hashed using String2Key (S2K) algorithm. This
is rather similar to `crypt()` algorithm - purposefully slow
and with random salt - but it produces a full-length binary key.
2. If separate session key is requested, new random key will be
generated. Otherwise S2K key will be used directly as session key.
3. If S2K key is to be used directly, then only S2K settings will be put
into session key packet. Otherwise session key will be encrypted with
S2K key and put into session key packet.
When encrypting with public key:
1. New random session key is generated.
2. It is encrypted using public key and put into session key packet.
Now common part, the session-key encrypted data packet:
1. Optional data-manipulation: compression, conversion to UTF-8,
conversion of line-endings.
2. Data is prefixed with block of random bytes. This is equal
to using random IV.
3. A SHA1 hash of random prefix and data is appended.
4. All this is encrypted with session key.
5.2. pgp_sym_encrypt(data, psw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pgp_sym_encrypt(data text, psw text [, options text] ) RETURNS bytea
pgp_sym_encrypt_bytea(data bytea, psw text [, options text] ) RETURNS bytea
Return a symmetric-key encrypted PGP message.
Options are described in section 5.8.
5.3. pgp_sym_decrypt(msg, psw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pgp_sym_decrypt(msg bytea, psw text [, options text] ) RETURNS text
pgp_sym_decrypt_bytea(msg bytea, psw text [, options text] ) RETURNS bytea
Decrypt a symmetric-key encrypted PGP message.
Decrypting bytea data with `pgp_sym_decrypt` is disallowed.
This is to avoid outputting invalid character data. Decrypting
originally textual data with `pgp_sym_decrypt_bytea` is fine.
Options are described in section 5.8.
5.4. pgp_pub_encrypt(data, pub_key)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pgp_pub_encrypt(data text, key bytea [, options text] ) RETURNS bytea
pgp_pub_encrypt_bytea(data bytea, key bytea [, options text] ) RETURNS bytea
Encrypt data with a public key. Giving this function a secret key will
produce a error.
Options are described in section 5.8.
5.5. pgp_pub_decrypt(msg, sec_key [, psw])
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pgp_pub_decrypt(msg bytea, key bytea [, psw text [, options text]] ) \
RETURNS text
pgp_pub_decrypt_bytea(msg bytea, key bytea [,psw text [, options text]] ) \
RETURNS bytea
Decrypt a public-key encrypted message with secret key. If the secret
key is password-protected, you must give the password in `psw`. If
there is no password, but you want to specify option for function, you
need to give empty password.
Decrypting bytea data with `pgp_pub_decrypt` is disallowed.
This is to avoid outputting invalid character data. Decrypting
originally textual data with `pgp_pub_decrypt_bytea` is fine.
Options are described in section 5.8.
5.6. pgp_key_id(key / msg)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pgp_key_id(key or msg bytea) RETURNS text
It shows you either key ID if given PGP public or secret key. Or it
gives the key ID that was used for encrypting the data, if given
encrypted message.
It can return 2 special key IDs:
SYMKEY::
The data is encrypted with symmetric key.
ANYKEY::
The data is public-key encrypted, but the key ID is cleared.
That means you need to try all your secret keys on it to see
which one decrypts it. pgcrypto itself does not produce such
messages.
Note that different keys may have same ID. This is rare but normal
event. Client application should then try to decrypt with each one,
to see which fits - like handling ANYKEY.
5.7. armor / dearmor
~~~~~~~~~~~~~~~~~~~~~~
armor(data bytea) RETURNS text
dearmor(data text) RETURNS bytea
Those wrap/unwrap data into PGP Ascii Armor which is basically Base64
with CRC and additional formatting.
5.8. Options for PGP functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Options are named to be similar to GnuPG. Values should be given after
an equal sign; separate options from each other with commas. Example:
pgp_sym_encrypt(data, psw, 'compress-algo=1, cipher-algo=aes256')
All of the options except `convert-crlf` apply only to encrypt
functions. Decrypt functions get the parameters from PGP data.
Most interesting options are probably `compression-algo` and
`unicode-mode`. The rest should have reasonable defaults.
cipher-algo::
What cipher algorithm to use.
Values: bf, aes128, aes192, aes256 (OpenSSL-only: `3des`, `cast5`)
Default: aes128
Applies: pgp_sym_encrypt, pgp_pub_encrypt
compress-algo::
Which compression algorithm to use. Needs building with zlib.
Values:
0 - no compression
1 - ZIP compression
2 - ZLIB compression [=ZIP plus meta-data and block-CRC's]
Default: 0
Applies: pgp_sym_encrypt, pgp_pub_encrypt
compress-level::
How much to compress. Bigger level compresses smaller but is slower.
0 disables compression.
Values: 0, 1-9
Default: 6
Applies: pgp_sym_encrypt, pgp_pub_encrypt
convert-crlf::
Whether to convert `\n` into `\r\n` when encrypting and `\r\n` to `\n`
when decrypting. RFC2440 specifies that text data should be stored
using `\r\n` line-feeds. Use this to get fully RFC-compliant
behavior.
Values: 0, 1
Default: 0
Applies: pgp_sym_encrypt, pgp_pub_encrypt, pgp_sym_decrypt, pgp_pub_decrypt
disable-mdc::
Do not protect data with SHA-1. Only good reason to use this
option is to achieve compatibility with ancient PGP products, as the
SHA-1 protected packet is from upcoming update to RFC2440. (Currently
at version RFC2440bis-14.) Recent gnupg.org and pgp.com software
supports it fine.
Values: 0, 1
Default: 0
Applies: pgp_sym_encrypt, pgp_pub_encrypt
enable-session-key::
Use separate session key. Public-key encryption always uses separate
session key, this is for symmetric-key encryption, which by default
uses S2K directly.
Values: 0, 1
Default: 0
Applies: pgp_sym_encrypt
s2k-mode::
Which S2K algorithm to use.
Values:
0 - Without salt. Dangerous!
1 - With salt but with fixed iteration count.
3 - Variable iteration count.
Default: 3
Applies: pgp_sym_encrypt
s2k-digest-algo::
Which digest algorithm to use in S2K calculation.
Values: md5, sha1
Default: sha1
Applies: pgp_sym_encrypt
s2k-cipher-algo::
Which cipher to use for encrypting separate session key.
Values: bf, aes, aes128, aes192, aes256
Default: use cipher-algo.
Applies: pgp_sym_encrypt
unicode-mode::
Whether to convert textual data from database internal encoding to
UTF-8 and back. If your database already is UTF-8, no conversion will
be done, only the data will be tagged as UTF-8. Without this option
it will not be.
Values: 0, 1
Default: 0
Applies: pgp_sym_encrypt, pgp_pub_encrypt
5.9. Generating keys with GnuPG
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Generate a new key:
gpg --gen-key
The preferred key type is "DSA and Elgamal".
For RSA encryption you must create either DSA or RSA sign-only key
as master and then add RSA encryption subkey with `gpg --edit-key`.
List keys:
gpg --list-secret-keys
Export ascii-armored public key:
gpg -a --export KEYID > public.key
Export ascii-armored secret key:
gpg -a --export-secret-keys KEYID > secret.key
You need to use `dearmor()` on them before giving them to
pgp_pub_* functions. Or if you can handle binary data, you can drop
"-a" from gpg.
For more details see `man gpg`, http://www.gnupg.org/gph/en/manual.html[
The GNU Privacy Handbook] and other docs on http://www.gnupg.org[] site.
5.10. Limitations of PGP code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- No support for signing. That also means that it is not checked
whether the encryption subkey belongs to master key.
- No support for encryption key as master key. As such practice
is generally discouraged, it should not be a problem.
- No support for several subkeys. This may seem like a problem, as this
is common practice. On the other hand, you should not use your regular
GPG/PGP keys with pgcrypto, but create new ones, as the usage scenario
is rather different.
6. Raw encryption
-------------------
Those functions only run a cipher over data, they don't have any advanced
features of PGP encryption. Therefore they have some major problems:
1. They use user key directly as cipher key.
2. They don't provide any integrity checking, to see
if the encrypted data was modified.
3. They expect that users manage all encryption parameters
themselves, even IV.
4. They don't handle text.
So, with the introduction of PGP encryption, usage of raw
encryption functions is discouraged.
encrypt(data bytea, key bytea, type text) RETURNS bytea
decrypt(data bytea, key bytea, type text) RETURNS bytea
encrypt_iv(data bytea, key bytea, iv bytea, type text) RETURNS bytea
decrypt_iv(data bytea, key bytea, iv bytea, type text) RETURNS bytea
Encrypt/decrypt data with cipher, padding data if needed.
`type` parameter description in pseudo-noteup:
algo ['-' mode] ['/pad:' padding]
Supported algorithms:
* `bf` - Blowfish
* `aes` - AES (Rijndael-128)
Modes:
* `cbc` - next block depends on previous. (default)
* `ecb` - each block is encrypted separately.
(for testing only)
Padding:
* `pkcs` - data may be any length (default)
* `none` - data must be multiple of cipher block size.
IV is initial value for mode, defaults to all zeroes. It is ignored for
ECB. It is clipped or padded with zeroes if not exactly block size.
So, example:
encrypt(data, 'fooz', 'bf')
is equal to
encrypt(data, 'fooz', 'bf-cbc/pad:pkcs')
7. Random bytes
-----------------
gen_random_bytes(count integer)
Returns `count` cryptographically strong random bytes as bytea value.
There can be maximally 1024 bytes extracted at a time. This is to avoid
draining the randomness generator pool.
8. Credits
------------
I have used code from following sources:
`--------------------`-------------------------`-------------------------------
Algorithm Author Source origin
-------------------------------------------------------------------------------
DES crypt() David Burren and others FreeBSD libcrypt
MD5 crypt() Poul-Henning Kamp FreeBSD libcrypt
Blowfish crypt() Solar Designer www.openwall.com
Blowfish cipher Niels Provos OpenBSD sys/crypto
Rijndael cipher Brian Gladman OpenBSD sys/crypto
MD5 and SHA1 WIDE Project KAME kame/sys/crypto
SHA256/384/512 Aaron D. Gifford OpenBSD sys/crypto
BIGNUM math Michael J. Fromberger dartmouth.edu/~sting/sw/imath
-------------------------------------------------------------------------------
9. Legalese
-------------
* I owe a beer to Poul-Henning.
* This product includes software developed by Niels Provos.
10. References/Links
----------------------
10.1. Useful reading
~~~~~~~~~~~~~~~~~~~~~~
http://www.gnupg.org/gph/en/manual.html[]::
The GNU Privacy Handbook
http://www.openwall.com/crypt/[]::
Describes the crypt-blowfish algorithm.
http://www.stack.nl/~galactus/remailers/passphrase-faq.html[]::
How to choose good password.
http://world.std.com/~reinhold/diceware.html[]::
Interesting idea for picking passwords.
http://www.interhack.net/people/cmcurtin/snake-oil-faq.html[]::
Describes good and bad cryptography.
10.2. Technical references
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
http://www.ietf.org/rfc/rfc2440.txt[]::
OpenPGP message format
http://www.imc.org/draft-ietf-openpgp-rfc2440bis[]::
New version of RFC2440.
http://www.ietf.org/rfc/rfc1321.txt[]::
The MD5 Message-Digest Algorithm
http://www.ietf.org/rfc/rfc2104.txt[]::
HMAC: Keyed-Hashing for Message Authentication
http://www.usenix.org/events/usenix99/provos.html[]::
Comparison of crypt-des, crypt-md5 and bcrypt algorithms.
http://csrc.nist.gov/cryptval/des.htm[]::
Standards for DES, 3DES and AES.
http://en.wikipedia.org/wiki/Fortuna_(PRNG)[]::
Description of Fortuna CSPRNG.
http://jlcooke.ca/random/[]::
Jean-Luc Cooke Fortuna-based /dev/random driver for Linux.
http://www.cs.ut.ee/~helger/crypto/[]::
Collection of cryptology pointers.
// $PostgreSQL: pgsql/contrib/pgcrypto/README.pgcrypto,v 1.18 2006/09/05 21:26:48 tgl Exp $
distrib
>
Mandriva
>
2009.0
>
x86_64
>
media
>
contrib-testing
>
by-pkgid
>
b7275baf04bd377dbb7da23de355da28
>
files
>
52
