These CKAN Versions are exactely matched:

• 2.0
• 2.0.1
• 2.0.2
• 2.0.3
• 2.0.4
• 2.0.5
• 2.0.6
• 2.0.7
• 2.0.8
• 2.1
• 2.1.1
• 2.1.2
• 2.1.3
• 2.1.4
• 2.1.5
• 2.1.6
• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8
• 2.10.9
• 2.11.0
• 2.11.1
• 2.11.2
• 2.11.3
• 2.11.4
• 2.12.0
• 2.2
• 2.2.1
• 2.2.2
• 2.2.3
• 2.2.4
• 2.3
• 2.3.1
• 2.3.2
• 2.3.3
• 2.3.4
• 2.3.5
• 2.4.0
• 2.4.1
• 2.4.2
• 2.4.3
• 2.4.4
• 2.4.5
• 2.4.6
• 2.4.7
• 2.4.8
• 2.4.9
• 2.5.0
• 2.5.1
• 2.5.2
• 2.5.3
• 2.5.4
• 2.5.5
• 2.5.6
• 2.5.7
• 2.5.8
• 2.5.9
• 2.6.0
• 2.6.1
• 2.6.2
• 2.6.3
• 2.6.4
• 2.6.5
• 2.6.6
• 2.6.7
• 2.6.8
• 2.6.9
• 2.7.0
• 2.7.1
• 2.7.10
• 2.7.11
• 2.7.12
• 2.7.2
• 2.7.3
• 2.7.4
• 2.7.5
• 2.7.6
• 2.7.7
• 2.7.8
• 2.7.9
• 2.8.0
• 2.8.1
• 2.8.10
• 2.8.11
• 2.8.12
• 2.8.2
• 2.8.3
• 2.8.4
• 2.8.5
• 2.8.6
• 2.8.7
• 2.8.8
• 2.8.9
• 2.9.0
• 2.9.0a
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.13
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9

ckanext-datapreview

This CKAN extension supplies data from local storage or via a remote download, parses CSV/XLS and provides it at a URL that can be called by Recline or other CKAN data previewer.

e.g. for a spreadsheet it returns a JSON dict such as:

{
    "fields": ["Name", "Age"],
    "data": [["Bob", 42], ["Jill", 54]],
    "extra_text": "This preview shows only the first 10 rows",
    "max_results": 10,
    "length": 435,
    "url": "http://data.com/file.csv"
}

This extension is a modified, but local implementation of the OKFN dataproxy that runs as a CKAN extension rather than on Google AppEngine. This has been written to improve the performance on data.gov.uk and increase the maximum file size processed.

The interface to the extension:

/data/preview/<resource_id>?max-results=N&encoding=utf-8

is not exactly the same - dataproxy requires the URL instead of the resource id - the data returned is identical. Rather than always fetching the data from the remote site the new controller at the above route will first attempt to find the data in the ckanext-archiver’s local archive.

Installation

The most straightforward method of installation is:

git clone git://github.com/datagovuk/ckanext-datapreview.git
cd ckanext-datapreview
python setup.py develop

Or alternatively install directly using pip:

pip install -e git+https://github.com/datagovuk/ckanext-datapreview.git#egg=ckanext-datapreview

Once complete the datapreview should be added to your ckan.plugins property in the appropriate .ini file.

Config

In your CKAN config file, configure the following options:

limit

The ‘limit’ is the maximum size of a file downloaded or loaded into memory. If the data is not stored locally, then you don’t want to wait forever downloading it to be able to proxy it.

The limit is expressed in bytes, so the default of 5MB would be:

ckan.datapreview.limit = 5242880

Local CSV files are not subject to this limit because the first 100 rows can be loaded without loading the whole file into memory.

Requirements

• ckanext-archiver - for the resource cache
• messytables - (in setup.py)

Improvements

• Increases the limit on download size (doesn’t have the appengine download limit)
• Uses the local archive cache if it exists rather than hitting the remote site (only if ckanext-archiver has retrieved the file).

Note: This repository was archived by the owner on Jun 19, 2023. It is now read-only.