Extension Import Library


Extension Basics

Title
Import Library
Name
ckanext-importlib
Type
Public extension
Description
Library for importing datasets into CKAN using the API. Supports continuous/automated imports with duplicate detection and resource series grouping.
CKAN versions
Download-Url (zip)
Last commit
12 years ago (2014-01-28 17:03:18)
Url to repo
Category
Data Management & Quality


Background Infos

Description (long)
Show details

ckanext-importlib

Library for importing datasets into CKAN using the API.

Introduction

One-off imports of metadata into CKAN isn’t very hard, and using ckanclient directly is probably best for that. But when you are continuously importing you have some challenges which this library aims to help with:

  • When you reimport a dataset you want to check if it already exists in CKAN, using an ID stored in an extra field and possibly another extra field naming the source
  • You may import resources, which become grouped into datasets (e.g. time series data) - ResourceSeriesLoader
  • When you derive a unique name for a dataset from its title, you need to avoid clashes

ckanext-importlib was designed as a framework to be expanded, based on the needs of the data.gov.uk ONS importer.

Quickstart

To get the code:

hg clone https://github.com/okfn/ckanext-importlib.git

The code also requires installed: - importlib dependencies (pip-requirements.txt) - ckan - ckan dependencies (ckan/pip-requirements.txt)

To install the dependencies into a virtual environment:

virtualenv pyenv
pip -E pyenv install -e ../ckanext-importlib
pip -E pyenv install -e ckan
pip -E ../pyenv-ckanext-importlib install -r ../ckan/pip-requirements.txt
pip -E pyenv install -r pip-requirements.txt

Tests

To run the tests:

pip -E pyenv install -e nose
cd ckanext-importlib
nosetests --ckan ckanext/importlib/tests/
Version
0.1
Version release date
(not set)
Contact name
Open Knowledge Foundation
Contakt email
Contact Url
(not set)


Installation Guide

Configuration hints

Originally designed for data.gov.uk ONS importer. No CKAN plugin entry points defined - used as a library.

Plugins to configure (ckan.ini)
(not set)
CKAN Settings (ckan.ini)
# No specific environment variables required
DB migration to be executed
(not set)
<< back to Extensions