List of known problems and issues

After restart of cryoSPARC master there is "database: ERROR (spawn error)"

Sometimes when cryoSPARC master is shut down it can not restart due to database error. It leads to

cryoSPARC log with error
CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: ERROR (spawn error)

In such case you should check logs for errors

  • main log in  ~/.cryosparc/log/run/command_core.log

    database log
    less ~/.cryosparc/log/run/command_core.log

  • supervisor log in  ~/.cryosparc/log/run/supervisord.log

    database log
    less ~/.cryosparc/log/run/supervisord.log

  • database log in ~/.cryosparc/log/run/database.log

    database log
    less ~/.cryosparc/log/run/database.log
  • if there is information that database is  "node is not in primary or recovering state" i.e. below
    cryoSPARC log with error
    AssertionError: {'code': 500, 'data': None, 'message': "OtherError: node is not in primary or recovering state, full error:
{'ok': 0.0, 'errmsg': 'node is not in primary or recovering state', 'code': 13436, 'codeName': 'NotMasterOrSecondary'}

    then:

    backup of cryoSPARC projects

    Backup all your projects and data before removing cryoSPARC database. Usually it should not be a problem to import them without backup but you should backup projects just in case. You have to remember that each cryoSPARC database start naming projects from P1, therefore it is best to backup projects to i.e. $PLG_GROUPS_STORAGE/<your-plggteam> space.

    • you have to move current cryoSPARC database

      database log
      mv ~/.cryosparc/cryosparc_database ~/.cryosparc/cryosparc_database-$(date+"%Y-%m-%d")

    • rerun  configuration of cryoSPARC to rebuild database

    • import projects that you want working with

cryoSPARC master is consuming resources when no other jobs are running

You could run additional helper script called cryosparc_idle_watcher.sh to check whether there are any cryoSPARC worker jobs running and finish cryoSPARC master job when there aren't any jobs for some time. 

idle watcher
cryosparc_idle_watcher <timeout>

cryoSPARC master and workers on cluster differ in versions

After changing cryoSPARC master version (i.e. by changing module that you load) cluster lanes are still pointing to previous version of cryoSPARC workers. It could show as error:

Version mismatch
File "cryosparc_worker/cryosparc_compute/run.py", line 46, in cryosparc_compute.run.main 
AssertionError: Version mismatch! Worker version (v3.2.0+210629) and Master version (v3.2.0+210413) are not the same. Please update.

You have to update cluster lanes.

  1. Start interactive job

    Interactive job
    srun -p plgrid-services --nodes=1 --ntasks=1 --time=0-1 --pty bash

  2. Load cryoSPARC environment using module (remember to use correct version of cryoSPARC)

    Load cryoSPARC environment
    module add plgrid/apps/cryosparc/<version>

  3. Run script that update cluster lanes 

    cryoSPARC master job

    There should be only one job which run cryoSPARC master per user. Therefore you should stop job with cryoSPARC master before this step.

    Load cryoSPARC environment
    cryosparc_lanes_updater

  4. Stop cryoSPARC master

    run cryoSPARC master
    cryosparcm stop

  5. End interactive job

    end interactive job
    exit
  • No labels