These CKAN Versions are exactely matched:

• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8
• 2.10.9
• 2.11.0
• 2.11.1
• 2.11.2
• 2.11.3
• 2.11.4
• 2.12.0
• 2.9.0
• 2.9.0a
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.13
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9

ckanext-datastore-dumper

When there is a high volume of requests on the datastore dump URL, it creates a heavy load on both the CKAN and Postgres database servers. In order to alleviate this load, this extension generates a blob file, uploads it to filestore, and then serves downloads directly from filestore.

How This Works

It runs a separate job each time when the datastore table is updated via the datastore create/update action. While the blob upload is in progress, the data is served via a generic datastore dump URL. Once the upload is completed, the data is served directly from the filestore. It also works with s3filestore.

Installation

To install ckanext-datastore-dumper:

1. Activate your CKAN virtual environment:

. /usr/lib/ckan/default/bin/activate

2. Clone the source and install it on the virtualenv:

git clone https://github.com/datopian/ckanext-datastore-dumper.git
cd ckanext-datastore-dumper
pip install -e .
pip install -r requirements.txt

3. Add datastore_dumper to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

4. Restart CKAN:

sudo service apache2 reload

5. Run CKAN background job worker:

# CKAN >= 2.9
paster --plugin=ckan jobs worker --config=/ckan.ini

# CKAN < 2.9
ckan -c ckan.ini jobs worker

Config Settings

If you are using ckanext-noanonaccess extension then allow this blueprint:

ckanext.noanonaccess.allowed_blueprint = datastore_dumper.dump

License

AGPL