Extension Datapusher Plus


Extension Basics

Title
Datapusher Plus
Name
datapusher-plus
Type
Core extension
Description
A standalone web service that pushes data into the CKAN Datastore fast & reliably. It pushes real good!
CKAN versions

~2.10

Show details
Download-Url (zip)
Last commit
9 days ago (2026-01-12 19:14:03)
Url to repo
Category
Data Management & Quality


Background Infos

Description (long)
Show details

DataPusher+

NOTE: v2 is a major revamp. Documentation is currently WIP.

DataPusher+ is a fork of Datapusher that combines the speed and robustness of ckanext-xloader with the data type guessing of Datapusher - super-powered with the ability to infer, calculate & suggest metadata using Jinja2 formulas defined in the scheming configuration file.

https://github.com/user-attachments/assets/b2fc2c3a-d244-4d11-9cf3-8270f0e99162

The Formulas have access to not just the package and resource fields (in the same namespaces), it also has access to the following information in these additional namespaces that can be used in Jinja2 expressions: * dpps - with the “s” for stats.
Each field will have an extensive list of summary statistics (by default: type, is_ascii, sum, min/max, range, sort_order, sortiness, min_length, max_length, sum_length, avg_length, stddev_length, variance_length, cv_length, mean, sem, geometric_mean, harmonic_mean, stddev, variance, cv, nullcount, max_precision, sparsity, cardinality, uniqueness_ratio.) Check here for all other available statistics. * dppf - with the “f” for frequency table.
Each field will have its frequency table available sorted in descending order the top N (configurable, default 10) values, with a corresponding count & percentage. “Other (COUNT)” will be used as a “basket” for other values with COUNT set to the count of other values beyond the top N. ID fields will be indicated by “” in the table. * dpp - additional inferred/calculated metadata.
* ORIGINAL_FILE_SIZE (bytes) * `PREVIEW_FILE_SIZ

Version
3.0.0
Version release date
2026-01-06
Contact name
(not set)
Contakt email
(not set)
Contact Url
(not set)


Installation Guide

Configuration hints

). Note that when compiling from source, you may want to look into the Performance Tuning section to squeeze even more performance from qsv.

Also, if you get glibc errors when starting qsv, your Linux distro may not have the required version of the GNU C Library
(This will be the case when running Ubuntu 18.04 or older).
If so, use the `unknown-linux-musl.zip` archive as it is statically linked with the MUSL C Library.
Plugins to configure (ckan.ini)
other plugins
CKAN Settings (ckan.ini)
# ckanext.datapusher_plus.enable_druf = true
# ckanext.datapusher_plus.enable_form_redirect = true
# ckanext.datapusher_plus.api_token=$(ckan -c /etc/ckan/default/ckan.ini user token add CKAN_ADMIN dpplus | tail -n 1 | tr -d '\t')"
# ckanext.datapusher_plus.use_proxy = false
# ckanext.datapusher_plus.download_proxy = 
> ckanext.datapusher_plus.ssl_verify = false
# ckanext.datapusher_plus.upload_log_level = INFO
# ckanext.datapusher_plus.formats = csv tsv tab ssv xls xlsx xlsxb xlsm ods geojson shp qgis zip
# ckanext.datapusher_plus.pii_screening = false
# ckanext.datapusher_plus.pii_found_abort = false
# ckanext.datapusher_plus.pii_regex_resource_id_or_alias =
> ckanext.datapusher_plus.pii_show_candidates = false
DB migration to be executed
(not set)
<< back to Extensions