These CKAN Versions are exactely matched:

• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8
• 2.10.9
• 2.11.0
• 2.11.1
• 2.11.2
• 2.11.3
• 2.11.4
• 2.12.0
• 2.9.0
• 2.9.0a
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.13
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9

ckanext-versioned-datastore

A CKAN extension providing a versioned datastore using MongoDB and Elasticsearch.

Overview

This plugin provides a complete replacement for CKAN’s datastore plugin and therefore shouldn’t be used in conjunction with it. Rather than storing data in PostgreSQL, resource data is stored in MongoDB and then made available to frontend APIs using Elasticsearch.

This allows this plugin to:

• provide full versioning of resource records - records can be updated when new resource data is uploaded without preventing access to the old data
• expose advanced search features using Elasticsearch’s extensive feature set
• achieve fast search response times, particularly when compared to PostgreSQL, due to Elasticsearch’s search performance
• store large resources (millions of rows) and still provide high speed search responses
• store complex data as both MongoDB and Elasticsearch are JSON based, allowing object nesting and arrays

This plugin is built on Splitgill.

Installation

Installing from PyPI

pip install ckanext-versioned-datastore

Installing from source

1. Clone the repository into the src folder: cd $INSTALL_FOLDER/src git clone https://github.com/NaturalHistoryMuseum/ckanext-versioned-datastore.git
2. Activate the virtual env: . $INSTALL_FOLDER/bin/activate
3. Install via pip: pip install $INSTALL_FOLDER/src/ckanext-versioned-datastore

Post-install setup

1. Add ‘versioned_datastore’ to the list of plugins in your config file: ckan.plugins = ... versioned_datastore
2. Install lessc globally: npm install -g "less@~4.1"

Other requirements

• MongoDB 4.x
• Elasticsearch 6.7.x (6.x is probably ok, but untested)
• CKAN’s job queue

Configuration

Required

Optional

Usage

The plugin automatically detects resources on upload that can be added to the datastore. Accepted formats: CSV, TSV, XLS, XLSX.

Data is added in two steps: 1. Ingesting records into MongoDB 2. Indexing documents from MongoDB into Elasticsearch

Search is available via datastore_search and datastore_search_raw actions.

Commands

• ckan -c $CONFIG_FILE initdb - ensure tables needed by this plugin exist
• ckan -c $CONFIG_FILE reindex $OPTIONAL_RESOURCE_ID - reindex a specific resource or all resources

Interfaces

• IVersionedDatastore - general interface for modifying data dicts, searches, results, fields, and index documents
• IVersionedDatastoreQuery - hooks for search queries
• IVersionedDatastoreDownloads - hooks for downloads