The Missing Manual for Hacking Postgres

It’s probably obvious that Postgres is my favorite database. One minor grievance that I have with the project is that its documentation is almost entirely optimized for people who ultimately be users of the database rather than developers of it. An unfortunate side effect of this is that none of the repository’s standard files (e.g. README) give much insight into how to get started with the source code.

In numerous places, some references in files and in errors generated by make tasks are actively misleading in that they’ll reference an INSTALL for further instructions. Some investigation will reveal that INSTALL doesn’t actually exist on master; it’s only generated as part of a release.

The excellent Postgres does of course contain all the information needed to get started with development, but if its has one weakness, it’s that its overwhelming verbosity tends to obscure information.

Here I’ve tried to assemble some succinct instructions for getting started that are useful and more importantly, succinct. I don’t expect most of them to change all that much, but I’ll try to keep the document up-to-date in case they do.

The basics

Selecting prefix and port

It’s often desirable to have a stable release of Postgres running on your machine for day-to-day work along with your experimental build, so you may want to choose a non-standard install directory, data directory, and port for development.

A prefix is passed during configure to specify the target install directory. I use ./build in the current directory and name it $PG_BUILD_DIR. I call my data directory $PG_DATA_DIR.

A port can be overridden with a command line argument to a server or client command like psql. It can also be overridden for an entire session by setting the PGPORT environmental variable. I’ve chosen 5433 as my port.

I use the excellent direnv to manage these variables. It reads them out of an .envrc in the source directory:

export PG_BUILD_DIR="$PWD/build"
export PG_DATA_DIR="$PWD/data/primary"
export PGPORT=5433

(Be sure to direnv allow after saving the file.)

Building

Clone the repository:

git clone https://github.com/postgres/postgres.git

Run configure with a prefix pointing to your chosen target build directory:

./configure --prefix $PG_BUILD_DIR

Then build it. The -j option gives you some parallelism which will speed things up for any computer that’s still running today.

make -j16

Pick a number based off how many cores you have. I’m using an iMac Pro with an 8 cores, each of which is hyper-threaded, so I use specify a parallelism of 16.

Install the result to the prefix configured above:

make install

Running

Initialize a data directory and start an instance of Postgres right in your terminal. This is convenient because you can see any logging that it emits and you can restart it easily with Ctrl+C.

mkdir -p $PG_DATA_DIR

# initialize a data directory
$PG_BUILD_DIR/bin/initdb -D $PG_DATA_DIR

# start the server
$PG_BUILD_DIR/bin/postgres -D $PG_DATA_DIR -p $PGPORT

Now create a database and connect to it:

$PG_BUILD_DIR/bin/createdb -p $PGPORT brandur-test
$PG_BUILD_DIR/bin/psql -p $PGPORT brandur-test

Testing

Postgres doesn’t have much in the way of standard unit testing, but instead relies heavily on a thorough regression suite. Run it with:

make check

The command will start a new server, set it up, run the suite, and then tear it down. This is a reliable way to get consistent results, but is somewhat slow. A faster version is also provided which can use a server that you already have running elsewhere:

# requires $PGPORT to be set in the environment
make installcheck

There’s also a parallel version available to further improve speed (you should basically always prefer this variant):

# requires $PGPORT to be set in the environment
make installcheck-parallel

Optimizations

Building and testing Postgres is already pretty fast (with parallel commands, make for me takes ~30s from scratch and running the test suite takes ~15s), but if you’re going to be working with it heavily, you might want to take a few steps to make it even faster.

ccache

ccache is a clever little program that pretends to be your compiler target and caches results so that they can be returned immediately the next time it’s run with the same inputs.

It’s trivial to install (on Mac OS, I use a simple brew install ccache) and causes very few problems, so it’s a pretty easy enhancement.

Use it by telling configure that you want ccache as your C compiler:

./configure --prefix $PG_BUILD_DIR --with-CC="ccache gcc"

After warming up ccache by building once, then doing a make clean and building again, my runtime drops from 30s to less than 5s. Incremental compiles are even faster.

The gold linker

If you’re on Linux, you can try the gold linker, which is faster than the GNU linker. Unfortunately, it only supports ELF, so it’s not available to Mac OS users.

Just export it in your $CFLAGS before running configure:

export CFLAGS="-fuse-ld=gold"
./configure ...

pgindent

Postgres has a slightly unusual tradition of code indentation which seems to have evolved to maximize the number of bytes saved at a time when that mattered, and which continues through to this day. A program similar to Go’s gofmt called pgindent ships with the Postgres source to help automatically reformat source files that are inconsistent.

You may be asked to run pgindent if someone notices that your patch isn’t compliant, and it’s generally a good idea to run it on any sources files that you changed before producing a patch anyway.

A few dependencies need to be installed before pgindent can run. The most up-to-date instructions on how to do that can be found in its README.

After that’s done it can simply be run like so on a C file (where our current directory is the Postgres source root):

src/tools/pgindent/pgindent src/backend/utils/adt/mac.c

Given that pgindent is brittle Perl code and appears to have no test coverage whatsoever, I’d recommend committing changes before using it on any of your code.

Patch formatting

Changes to Postgres are submitted as patch file email attachments to the PG Hackers mailing list. Traditionally, Postgres required that patches were in a particular style called “context format” (as generated by the diff tool’s -c option), but that constraint has since loosened a bit as the “unified diff” (probably what you’re used to seeing from programs like git diff) has become widely considered to more legible.

One good method for producing a patch that will be acceptable on the mailing list is the use of git format-patch 1. This command formats each commit as a separate file named based on the commit message, and includes each entire commit message within the files for extra context. For example:

$ git format-patch master...
0001-Implement-SortSupport-for-macaddr-data-type.patch

Regardless of the tool you use, good commit hygiene is still of paramount importance, so remember to squash and fix using git rebase -i before producing patch files.

Running a local replica

If you need to test with a replica, it’s pretty easy to set that up by running a second Postgres instance listening on a different port and tweaking some configuration. Here’s a script that demonstrates how to do that:

#!/bin/sh

set -e

export PG_DIR="$PWD"

export PRIMARY_PORT=5433
export REPLICA_PORT=5434

read -p "Will delete $PG_DIR/data/{primary,replica}. Okay? [Ctrl+C cancels]" yn
rm -rf $PG_DIR/data/primary
rm -rf $PG_DIR/data/replica

# Initialize a new data directory for the primary, then use a bit of a shortcut
# by just copying it for use by the replica.
$PG_DIR/bin/initdb -D $PG_DIR/data/primary/
cp -r $PG_DIR/data/primary/ $PG_DIR/data/replica/

cat <<EOT >> $PG_DIR/data/primary/postgresql.conf
port=$PRIMARY_PORT
EOT

cat <<EOT >> $PG_DIR/data/replica/postgresql.conf
port=$REPLICA_PORT
shared_buffers=500MB
hot_standby=on
hot_standby_feedback=on
EOT

cat <<EOT >> $PG_DIR/data/replica/recovery.conf
standby_mode=on
primary_conninfo='host=127.0.0.1 port=$PRIMARY_PORT user=$USER'
EOT

cat <<EOT >> /dev/stdout
READY!
======

Start primary:
    $PG_DIR/bin/postgres -D $PG_DIR/data/primary

Start replica:
    $PG_DIR/bin/postgres -D $PG_DIR/data/replica

Create a database:
    $PG_DIR/bin/createdb -p $PRIMARY_PORT mydb

Connect to primary:
    $PG_DIR/bin/psql -p $PRIMARY_PORT mydb

Connect to replica:
    $PG_DIR/bin/psql -p $REPLICA_PORT mydb
EOT

1 Note that git format-patch is not officially endorsed and so your mileage with its usage may vary.

Article
The Missing Manual for Hacking Postgres

Published
August 17, 2016

Location
San Francisco

Find me on Twitter at @brandur.

Did I make a mistake? Please consider sending a pull request.