Extension Datastore Dumper

These CKAN Versions are exactely matched:

• 2.0
• 2.0.1
• 2.0.2
• 2.0.3
• 2.0.4
• 2.0.5
• 2.0.6
• 2.0.7
• 2.0.8
• 2.1
• 2.1.1
• 2.1.2
• 2.1.3
• 2.1.4
• 2.1.5
• 2.1.6
• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8
• 2.10.9
• 2.11.0
• 2.11.1
• 2.11.2
• 2.11.3
• 2.11.4
• 2.12.0
• 2.2
• 2.2.1
• 2.2.2
• 2.2.3
• 2.2.4
• 2.3
• 2.3.1
• 2.3.2
• 2.3.3
• 2.3.4
• 2.3.5
• 2.4.0
• 2.4.1
• 2.4.2
• 2.4.3
• 2.4.4
• 2.4.5
• 2.4.6
• 2.4.7
• 2.4.8
• 2.4.9
• 2.5.0
• 2.5.1
• 2.5.2
• 2.5.3
• 2.5.4
• 2.5.5
• 2.5.6
• 2.5.7
• 2.5.8
• 2.5.9
• 2.6.0
• 2.6.1
• 2.6.2
• 2.6.3
• 2.6.4
• 2.6.5
• 2.6.6
• 2.6.7
• 2.6.8
• 2.6.9
• 2.7.0
• 2.7.1
• 2.7.10
• 2.7.11
• 2.7.12
• 2.7.2
• 2.7.3
• 2.7.4
• 2.7.5
• 2.7.6
• 2.7.7
• 2.7.8
• 2.7.9
• 2.8.0
• 2.8.1
• 2.8.10
• 2.8.11
• 2.8.12
• 2.8.2
• 2.8.3
• 2.8.4
• 2.8.5
• 2.8.6
• 2.8.7
• 2.8.8
• 2.8.9
• 2.9.0
• 2.9.0a
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.13
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9

When there is a high volume of requests on the datastore dump URL, it creates a heavy load on both the CKAN and Postgres database servers. In order to alleviate this load, this extension generates a blob file uploads it to filestore, and then serves downloads directly from filestore.

How this work

It runs separate job each time when the datastore table is updated via the datastore create/update action. While the blob upload is in progress, the data is served via a generic datastore dump URL. Once the upload is completed, the data is served directly from the filestore. It also works with s3filestorage.

Installation

To install ckanext-datastore-dumper :

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate

2. Clone the source and install it on the virtualenv

   git clone https://github.com/sagargg/ckanext-datastore-dumper.git
   cd ckanext-datastore-dumper 
   pip install -e .
   pip install -r requirements.txt
   

3. Add datastore_dumper to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

4. Restart CKAN. For example if you’ve deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload
   

5. Run ckan background job worker

   paster --plugin=ckan jobs worker --config=/ckan.ini # ckan >= 2.9
   ckan -c  ckan.ini jobs worker  # ckan < 2.9
   

Config settings

If you are using ckanext-noanonaccess extension then allow this blueprint

ckanext.noanonaccess.allowed_blueprint = datastore_dumper.dump

Developer installation

To install ckanext-datastore-dumper for development, activate your CKAN virtualenv and do:

git clone https://github.com/sagargg/ckanext-datastore-dumper.git
cd ckanext-datastore-dumper
python setup.py develop
pip install -r dev-requirements.txt

License

[AGPL](https://www.gnu.org/licenses/ag