<aside> 💻

Pipelines that run entirely on a local machine, against data on that same machine. Mostly R scripts with intermediary Rds files stored on Box.

</aside>

Simple pipelines will not have version control but they may often be collaborative. For this reason there are a few standard practices that we employ, outlined below. Processed (small) data and pipelines should be organized and stored in the lab Box Data folder using a common structure.

Box data and script organization

An example structure within a designated Box /ZamanianLab/LabMembers/{Name}/{Project} sub-directory:

{Data Type}/
  ├── Master_summary.csv        [date, experimenter, other descriptive columns]
  ├── data/                     [data organized by date]
  │   └── YYYYMMDD/
  │   │   ├── YYYYMMDD.csv      [raw instrument output or csv data]
  │   │   └── Notes.txt         [assay description and additional details]
  │   └── (assay)_tidy.rds      [tidy/processed data]
  ├── code/                     [R script folder]
  │   ├── (assay)_tidy.R        [raw data > (assay)_tidy.rds]
  │   └── (assay)_analysis.R    [template: tidy data > analysis and plots]
  └── plots/                    [plot outputs]

• Raw data will either be stored in CSV files or as natively-exported instrument files within dated folders. Append a unique identifier to the end of the folder name if multiple unrelated outputs of that data type were generated on the same day (e.g., YYYYMMDD-p01 and YYYYMMDD-p02).
• Data tidying will be performed in scripts that are separate from those that perform analysis and visualization. Tidy data will be stored in .rds files (space-saving compared to .rda format).
• Columns and variable entries in data and metadata files should generally not include spaces or unnecessary case sensitivity. They should use a consistent dictionary of abbreviations for common variables (e.g., strain, stage, species, treatments, concentrations) shared across the laboratory. Here is a dictionary of full drug names and plot abbreviations in your metadata.