Common 2nd line support tasks for CKAN

Warning This document has not been updated for a while now. It may be out of date.

Last updated: 5 Jan 2021

CKAN is the publishing application for data.gov.uk. ckanext-datagovuk is the primary CKAN extension for data.gov.uk.

Support

If you find an issue with CKAN that you believe may be a potential security vulnerability please contact security@ckan.org rather than disclosing the information publicly (as per their security policy).

If you find an issue that does not appear to be a security vulnerability you should first search the issues of the upstream CKAN repository to see if it has already been reported and if there is a workaround. If there is not an issue, considering opening a new one or - if you know how to fix it - open a PR to resolve the problem.

Environments

There are three environments for CKAN:

You can SSH onto these machines in same way as all other GOV.UK AWS applications. The machine node class is ckan.

Managing CKAN

First check to see if it is possible to complete the task through the web interface (credentials are available in the govuk-secrets password store, under datagovuk/ckan).

For commands not available via the user interface you must connect to the server to perform these tasks. Most of the commands to interact with CKAN use a tool called paster. Many of these commands take a path to the config file with the -c option, which is located at /var/ckan/ckan.ini in our deployments.

On GOV.UK servers paster should be run with:

cd /var/apps/ckan
sudo -u deploy govuk_setenv ckan venv/bin/paster [COMMAND] -c /var/ckan/ckan.ini

Initialising the database

There may be times when you need to start with an empty database (e.g. on integration). The following commands will create the relevant schema for core CKAN and the harvesting extension on integration.

. /var/apps/ckan/venv/bin/activate
paster --plugin=ckan db init -c /var/ckan/ckan.ini
paster --plugin=ckanext-harvest harvester initdb -c /var/ckan/ckan.ini

Accessing the database

In order to access the CKAN database to run queries on the db_admin machine:

psql -U ckan -h postgresql-primary -p 5432 ckan_production

The password can be extracted from the configuration file on the ckan machine for the environment you are targeting:

more /var/ckan/ckan.ini | grep sqlalchemy

Accessing the CKAN API

There are times when it can be useful to access the CKAN API when debugging or resolving issues. Note that the responses will be different depending on your access permissions.

Queries can use the package ID or name (the slug) e.g.

https://data.gov.uk/api/3/action/package_search?q=id:93a39f01-7bba-430f-aa35-c30bf2d88b2f
returns the same as
https://data.gov.uk/api/3/action/package_search?q=name:north-lincolnshire-brown-field-register

Here are some more complex examples of using the API, as well as some where the documentation is more sparse:

# Retrieve full details about a package (dataset)
https://data.gov.uk/api/3/action/package_search?q=name:north-lincolnshire-brown-field-register

# Find all packages created during a specific timeframe
https://data.gov.uk/api/3/action/package_search?q=metadata_created:[2017-06-01T00:00:00Z%20TO%202017-06-30T00:00:00Z]

# Find all packages modified during a specific timeframe
https://data.gov.uk/api/3/action/package_search?q=metadata_modified:[2017-06-01T00:00:00Z%20TO%202017-06-30T00:00:00Z]

# Count all packages and list a sample
 https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id&q=state:active%20and%20dataset_type:dataset&limit=20

# Count harvested packages and list a sample
 https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,state,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id&q=state:active%20and%20extras_harvest_object_id:[%22%22%20TO%20*]%20and%20dataset_type:dataset&limit=20

# Count manually published packages and list a sample
 https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id&q=state:active%20and%20-extras_harvest_object_id:*%20and%20dataset_type:dataset&limit=20

# Count packages harvested in last 7 days and list a sample https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,state,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id&q=state:active%20and%20extras_harvest_object_id:[%22%22%20TO%20*]%20and%20dataset_type:dataset%20and%20metadata_modified:[NOW-7DAY/DAY%20TO%20NOW]&limit=20

# Count packages manually published in last 7 days and list a sample https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id&q=state:active%20and%20-extras_harvest_object_id:*%20and%20dataset_type:dataset%20and%20metadata_modified:[NOW-7DAY/DAY%20TO%20NOW]&limit=20

# Count packages from Daily harvest sources and list a sample
https://ckan.publishing.service.gov.uk/api/search/dataset?fl=id,metadata_modified,dataset_type,name,title,extras_guid,extras_harvest_source_title,extras_harvest_object_id,extras_frequency-of-update&q=state:active%20and%20extras_harvest_object_id:%5B%22%22%20TO%20*%5D%20and%20frequency-of-update:daily%20and%20dataset_type:dataset&limit=20

# List all publishers
https://data.gov.uk/api/3/action/organization_list

# View a publisher record
https://data.gov.uk/api/3/action/organization_show?id=government_digital_service

# View a user (e.g. to get CKAN API key for a publishing user)
https://data.gov.uk/api/3/action/user_show?id=user_d484581

Creating a system administrator account

paster --plugin=ckan sysadmin add USERNAME email=EMAIL_ADDRESS -c /var/ckan/ckan.ini

You will be prompted twice for a password.

Removing a system administrator account

paster --plugin=ckan sysadmin remove USERNAME -c /var/ckan/ckan.ini

Managing users

Listing users

paster --plugin=ckan user list -c /var/ckan/ckan.ini

Viewing a user

paster --plugin=ckan user USERNAME -c /var/ckan/ckan.ini

Adding a user

paster --plugin=ckan user add USERNAME email=EMAIL_ADDRESS -c /var/ckan/ckan.ini

Removing a user

paster --plugin=ckan user remove USERNAME -c /var/ckan/ckan.ini

Changing a user’s password

paster --plugin=ckan user setpass USERNAME -c /var/ckan/ckan.ini

Managing publishers

Change a publisher’s name

Change the name in the publisher page then reindex that publisher:

paster --plugin=ckan search-index rebuild-publisher [PUBLISHER] -c /var/ckan/ckan.ini

Managing datasets

Deleting a dataset

CKAN has two types of deletions, the default soft-delete, and a purge. The soft delete gives the option of undeleting a dataset but the purge will remove all trace of it from the system.

Where the following commands mention DATASET_NAME, this should either be the slug for the dataset, or the UUID.

Deleting a dataset:

Find the dataset in the CKAN UI
Click on the ‘Manage’ button, then the ‘Delete’ button at the bottom of the page

The ‘Delete’ button is currently not available for draft datasets. In order to soft-delete a draft dataset, follow the above steps, but manually change ‘edit’ to ‘delete’ on the ‘Manage’ page for the dataset.

Purging a dataset:

paster --plugin=ckan dataset purge DATASET_NAME -c /var/ckan/ckan.ini

There may be times when a large number of datasets must be deleted. This needs to be done on the CKAN machine, since the deletion API is protected from external access. Your API key is required, which can be obtained from your user profile on the web interface. Put a list of dataset slugs or GUIDs in a text file, with one dataset per line, then run the following.

while read p; do curl --request POST --data "{\"id\": \"$p\"}" --header "Authorization: <your_api_key>" http://localhost:<ckan_port>/api/3/action/package_delete; done < list_of_ids.txt

After deleting or purging a dataset, it will take up to 10 minutes to update on Find, due to the sync process.

Listing datasets

You can get a list all datasets (including those which have been deleted) for a particular organisation by exporting a CSV directly from the database.

First, find the id of the organisation you want via the API (here, finding the id of the Environment Agency)“

https://ckan.publishing.service.gov.uk/api/3/action/organization_show?id=environment-agency

Then follow the instructions to access the database The following query will list all datasets for the Environment Agency, with the number of records for each and export the result to a CSV file.

\copy (SELECT package.name, package.title, package.url, package.state, COUNT(resource.package_id) as num_resources FROM package LEFT JOIN resource ON (resource.package_id = package.id) WHERE package.owner_org='11c51f05-a8bf-4f58-9b95-7ab55f9546d7' GROUP BY package.id) to 'output.csv' csv header;

You can then use scp-pull from the govuk-connect tool to download the file to your machine.

Register a brownfield dataset

See the supporting manual.

Rebuilding the search index

CKAN uses Solr for its search index, and occasionally it may be necessary to interact with it to refresh the index, or rebuild it from scratch.

Refresh the entire search index (this adds/removes datasets, but does not clear the index first):

paster --plugin=ckan search-index rebuild -r -c /var/ckan/ckan.ini

Rebuild the entire search index (this deletes the index before re-indexing begins):

paster --plugin=ckan search-index rebuild -c /var/ckan/ckan.ini

Rebuilding the entire search index immediately removes all records from the search before re-indexing begins. No datasets will be served from the package_search API endpoint until the re-index has completed. This command should therefore only be used as a last resort since it will cause the sync process to assume there is no data for a period of time.

Only reindex those packages that are not currently indexed:

paster --plugin=ckan search-index -o rebuild -c /var/ckan/ckan.ini

`csw` endpoint

The csw endpoint should be available on https://data.gov.uk/csw which should redirect to https://ckan.publishing.service.gov.uk/csw.

`csw` endpoint unavailable

If it is not showing xml with an error Missing keyword: service you can check that it is running on the ckan machine:

$ sudo service pycsw_web-procfile-worker status
pycsw_web-procfile-worker start/running

$ $ ps aux | grep pycsw
root     29503  0.0  0.0  59652  2040 ?        Ss   06:04   0:00 sudo -u deploy -E sh -c PATH=/usr/lib/rbenv/shims:$PATH exec  unicornherder --gunicorn-bin ./venv/bin/gunicorn -p /var/run/ckan/pycsw_unicornherder.pid -- ckanext.datagovuk.pycsw_wsgi --bind localhost:${PYCSW_PORT} --timeout ${GUNICORN_TIMEOUT} --workers ${GUNICORN_WORKER_PROCESSES} --log-file /var/log/ckan/pycsw.out.log --error-logfile /var/log/ckan/pycsw.err.log 2>>'/var/log/ckan/procfile_pycsw_web.err.log' 1>>'/var/log/ckan/procfile_pycsw_web.out.log'
...

If no running processes are found then you can restart it using this command:

$ sudo service pycsw_web-procfile_worker restart

It is worth checking the pycsw logs to investigate why it failed:

$ tail -f /var/log/ckan/pycsw.err.log

You can get a summary of csw records available from this url https://ckan.publishing.service.gov.uk/csw?service=CSW&version=2.0.2&request=GetRecords&typenames=csw:Record&elementsetname=brief

Syncing the `csw` records with `ckan` datasets

Normally the sync between csw and ckan will start at 6 each day, but in case it should fail or if the sync needs to happen sooner you can manually trigger the sync after Solr has been reindexed.

$ paster --plugin=ckanext-spatial ckan-pycsw load -p /var/ckan/pycsw.cfg -u http://localhost:3220

Harvesting

Although harvesters can mostly be managed from the user interface, it is sometimes easier to perform these tasks from the command line.

List current jobs

Returns a list of currently running jobs. This will contain the JOB_ID necessary to cancel jobs.

paster --plugin=ckanext-harvest harvester jobs -c /var/ckan/ckan.ini

It may be faster to run a SQL query to get the ID of a specific harvest job. You can do this by running the command from Accessing the database and passing a -c argument:

<psql_ckan_production_command> -c "SELECT id FROM harvest_source WHERE name = '[NAME]'"

Find the harvest source of a dataset

The harvest source of a dataset can be found using the CKAN API. Once you have the slug of the dataset, you can go to the following URL:

https://ckan.publishing.service.gov.uk/api/action/package_show?id=<slug>

In the JSON response there should be a harvest field which will tell you the ID and name of the harvest source.

Get the status of a harvester

Login to CKAN as a sysadmin user (credentials are available in the govuk-secrets password store, under datagovuk/ckan).
Navigate to the relevant harvester (use the ‘Harvest’ button in the header).
You will see a list of the datasets imported by this harvest source.
Click the ‘Manage’ button to get the status.
A summary of the current status will be shown. Individual runs (and the error messages logged) can be access from the ‘Jobs’ tab.

Restart a harvest job

Follow the steps to get the status of a harvester.
If the harvester is currently running, click the ‘Stop’ button to stop it. Once it has stopped, or if it is not currently running, click the ‘Reharvest’ button. You will know if the harvester is running because the ‘Reharvest’ button will be disabled.

If the harvest job is hanging and the ‘Stop’ button is not responding, you will have to log on to the ckan machine to restart it:

SSH into the ckan machine with gds govuk connect -e production ssh ckan
Assume the deploy user - sudo su deploy
Activate the virtual environment - . /var/apps/ckan/venv/bin/activate
Run the harvest job manually - paster --plugin=ckanext-harvest harvester run_test <harvest source> -c /var/ckan/ckan.ini
- where harvest source is from the url when visiting the harvest source page, it will be something like cabinet-office

If the job fails to complete the ticket should be updated with comments and prioritised to low for the product owner to review.

Cancel a harvest job

Follow the steps to get the status of a harvester.
Click the ‘Stop’ button to stop it.

Sometimes a harvest job can get stuck and not complete, and it’s not possible to cancel it through the UI. You can get the JOB_ID from the harvest dashboard under "Last Harvest Job” (or from Listing current jobs).

Then cancel the job by running:

paster --plugin=ckanext-harvest harvester job_abort JOB_ID -c /var/ckan/ckan.ini

This can also be done by running SQL:

<psql_ckan_production_command> -c "UPDATE harvest_job SET finished = NOW(), status = 'Finished' WHERE source_id = '[UUID]' AND NOT status = 'Finished';"

Purging all currently queued tasks

It may be necessary, if there is a schedule clash and the system is too busy, to purge the queues used in the various stages of harvesting

WARNING

This command will empty the Redis queues

paster --plugin=ckanext-harvest harvester purge_queues -c /var/ckan/ckan.ini

Restarting the harvest service

The harvesting process runs as a single threaded program, if any harvesting process crashes by raising an exception, it will take out the entire process. We have configured Upstart to restart the process automatically, but if the service keeps crashing, Upstart will decide it’s unhealthy and stop that after a while.

You can check whether the process is still running by checking if entries are still being written to the log file on the ckan machine:

$ gds govuk connect -e production ssh ckan
$ sudo tail -f /var/log/ckan/procfile_harvester_fetch_consumer.err.log

Or you could check that the services are all showing as started on the ckan machine:

$ sudo initctl list | grep harvester

If any of the services are not showing as started, you will need to restart them. There are both ‘gather’ and ‘fetch’ jobs that may need restarting:

‘Gather’ jobs retrieve the identifiers of the updated datasets and create jobs in the fetch queue. To restart, run:

$ sudo initctl restart harvester_gather_consumer-procfile-worker

‘Fetch’ jobs retrieve the datasets from the remote source and perform the relevant updates in CKAN. To restart, run:

$ sudo initctl restart harvester_fetch_consumer-procfile-worker

Alternatively, if the server has stopped, there is a Fabric script that will restart it for you. It will only restart if it detects the harvesting process is no longer running, so is safe to run immediately if you suspect the process has crashed:

$ fab aws_production class:ckan ckan.restart_harvester

CKAN publisher on Staging environment responds with Nginx 504 timeout:

Sometimes CKAN publisher on Staging environment responds with a 504 from Nginx, this is due to it timing out when connecting to the database as there are too many connections, current limit is 1000, though under normal working the number of connections should be under 100 if the system is not under load testing.

SELECT rolname, rolconnlimit FROM pg_roles WHERE rolname='ckan';`

Changing the connection limit

The connection limit can be updated, but should not exceed the hard limit which is around 3000.

ALTER USER ckan WITH CONNECTION LIMIT 1001;

Trying to log on to the database will result in this error:

FATAL:  too many connections for role "ckan"

Log in as another user:

sudo psql -U aws_db_admin -h postgresql-primary --no-password -d ckan_production

View the number of connections and types of queries:

SELECT COUNT(*) FROM pg_stat_activity WHERE datname = 'ckan_production';

It has been observed that during an overnight sync one of the queries caused a large number of db connections:

The following query captures part of that query and uses it to target the pid:

SELECT pid FROM pg_stat_activity WHERE datname = 'ckan_production' and query LIKE 'SELECT "user".password AS user_password%';

To cancel these queries

SELECT pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE pid IN
(SELECT pid
FROM pg_stat_activity
WHERE datname = 'ckan_production' and query LIKE 'SELECT "user".password AS user_password%');

To identify long running queries

SELECT
pid,
now() - pg_stat_activity.query_start AS duration,
query,
state
FROM pg_stat_activity
WHERE
(now() - pg_stat_activity.query_start) > interval '5 minutes' AND
datname = 'ckan_production';

Cancel long running queries

This will take a few seconds to be processed

SELECT pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE
(now() - pg_stat_activity.query_start) > interval '5 minutes' AND
state = 'active' AND
datname = 'ckan_production';

If cancelling does not work you can terminate the query

Note - this must be used with caution as may cause the database to restart to recover consistency.

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE
(now() - pg_stat_activity.query_start) > interval '5 minutes' AND
state = 'active' AND
datname = 'ckan_production';