# CIBRRIG
## :book: [ReadTheDocs](https://cibrrig.readthedocs.io/)
#### Support for extraction, preprocessing, sorting, analysis and plotting of physiology and Neuropixel recordings from rig to fig
## Description
Code to integrate hardware and software on the Neuropixel rig in JMB 971 at Seattle Childrens Research Institute, Center for Integrative Brain Research (SCRI-CIBR)
This code is maintained by Nick Bush in the Ramirez Lab and is subject to change.
The rig is designed to monitor breathing and behavior in a head-fixed mouse while recording from neuropixels throughout the brain. Rig is capable of hot-swap between awake and anesthetized preps.
Incorporates both custom code that is specific for the 971 Rig, and more general analyses that are applicable to Neuropixel recordings of respiratory/physiological systems.
**IMPORTANT**
This code is designed to work in conjunction with hardware in the the [pyExpControl](https://github.com/nbush257/pyExpControl) repository. Most functionality can be used independantly of this hardware, but the most critical piece is the automatically generated log file that is created during recording with this hardware.
The log file is a `.tsv` file with the name `_cibrrig_.g.t.tsv`. It has required columns:
`[label, category, start_time, end_time]`, and optional columns that describe parameters of the events (e.g., frequency, duration...). One could create these logfiles manually if desired, or ignore them entirely, but some functionality will fail.
---
## Installation
Create a virtual environment using mamba/conda.
>[!WARNING]
>If on SCRI networks it is critically important to specify the python version here. This circumvents the SSL issue we have been running into. BE SURE YOU HAVE MODIFIED YOUR .condarc file (in `C:/Users/`) appropriately
```
mamba create -n cibrrig python=3.12
mamba activate cibrrig
```
Then change directory to a place to install cibrrig locally.
>[!IMPORTANT]
>If you are on NPX 971 room computer, this has already been cloned and you should just install into your new venv.
>```
>cd C:/helpers/cibrrig
>git pull
>pip install -e .
>```
>(note the period)
>OTHERWISE, clone the repo:
>```
>cd
>git clone https://github.com/nbush257/cibrrig
>cd cibrrig
>pip install -e .
>```
>(note the period)
> **Once your virtual (mamba/conda) environment has been set up, `git pull` in the cibrrig directory will update `cibrrig` so you do not have to redo the pip install**
>[!WARNING]
>To do manual spike curation, you will need to install `phy` into a seperate conda/mamba environment due to some dependency issues at the moment
> See: https://github.com/cortex-lab/phy
Then, make sure the GPU is working for Kilosort (See [kilosort install instructions](https://github.com/MouseLand/Kilosort) steps 7 and 8):
>Next, if the CPU version of pytorch was installed (will happen on Windows), remove it with `pip uninstall torch`
>Then install the GPU version of pytorch `conda install pytorch pytorch-cuda=11.8 -c pytorch -c nvidia`
Make sure you are using the GPU by running the kilosort gui:`python -m kilosort` and confirming the PyTorch device is the GPU and not the CPU: 
Helper packages (Primarily matlab packages) should live in `C:/helpers` on the NPX computer so they are available to all users. Some functionality relies on these packages, but much is being phased out
These include:
- [Kilosort](https://github.com/MouseLand/Kilosort) (versions 2,3)
- Chronux http://chronux.org/
- Breathmetrics https://github.com/zelanolab/breathmetrics
- SALT ([Kvitsiani et al. 2013](https://www.nature.com/articles/nature12176))
---
## :exclamation: Quick start and Data structure
### From local computer :computer:
> :warning: This performs all processing on the local computer and ties up the resources. This workflow can get backed up if things go sideways.
> If you have recorded a dataset on the NPX computer you can simply open a command prompt and run:
>```
>mamba activate cibrrig
>npx_run_all
>```
> This will open a GUI that prompts you to choose some options and point to where you want thie files saved.
### From sasquatch (HPC) :monkey:
> :warning: Performing the computation on sasquatch keeps the acquisition rig cleaner
>**First**, compress and backup the dataset with:
>```
>mamba activate cibrrig
>backup
>```
>Example:
>```
>backup D:/Subjects/mickey_mouse \\baker.childrens.sea.kids/archive/ramirez_j/ramirezlab/alf_data_repo/ramirez/Subjects
>```
>**Second**, sign on to a sasquatch login node and run:
>```
>mamba activate iblenv
> pipeline_hpc --no-qc
>```
>N.B. This rsyncs the data to the sasquatch drive, submits SLURM jobs on sasquatch nodes, then moves the data to the ramirezlab alf repository.
>[!NOTE]
> There is incomplete code to run the pipeline via a series of SSH commands (`run_sasquatch.from_NPX`), but is not finished.
### Details:
Main entry points can be run from anywhere as long as the package has been pip installed
#### :arrow_right:Pipelines (Commands involved in end to end processing)
`npx_run_all` - Opens a GUI to performs backup, preprocess, and spikesorting\
`backup ` - Just performs backup\
`pipeline_hpc ` - Copy from run path to sasquatch tempdir, run pipeline, move to ramirezlab alf repo
#### Modules (Parts of the pipeline that can be run separately if needed)
`npx_preproc ` - Just performs preprocessing and extraction.\
`ephys_to_alf ` - Rename the recorded data to alf format\
`spikesort ` - run spikesorting\
`convert_ks_to_alf ` - convert sorted neural data from kilosort (i.e., phy) to ALF format. is the name of the sorting folder. Likely `kilosort4`\
`ephys_qc ` - Run IBL ephys qc and plots
In practice, it is easiest to simply run `npx_run_all` after recording. Previously run steps will be skipped or appropriately overwritten. Some users have shortcuts to batch scripts that activate the virtual environment and run this.
## Data structure
We save data in a way consistent with the **O**pen **N**europhysiology **E**nvironment ([**ONE**](https://github.com/int-brain-lab/ONE))
For a detailed description of filenames and structure see:[ONE Naming](https://github.com/int-brain-lab/ONE/blob/main/docs/Open_Neurophysiology_Environment_Filename_Convention.pdf)
Data should be organized with the following structure:
`.//Subjects///`
e.g.:
```
alf_data_repo/
├─ ramirez/
│ ├─ Subjects/
│ │ ├─ leonardo/
│ │ │ ├─ 2024-08-01/
│ │ │ │ ├─ 000/**<- SESSION_PATH**
│ │ │ │ ├─ 001/
│ │ │ ├─ 2024-08-02/
│ │ │ │ ├─ 000/
│ │ ├─ donatello/
│ │ │ ├─ 2024-03-05/
│ │ │ │ ├─ 000/
├─ sessions.pqt
├─ datasets.pqt
```
Data should have filenames like: `spikes.times.npy` of the form `