Data Management

§1 Core principles

Data management in a computational lab is the counterpart to sample handling in a wet lab. The same level of care is warranted. Work done on poorly organized, incompletely documented, or irreproducible data has limited scientific value regardless of how sophisticated the analysis is. The practices in this document protect the integrity and reproducibility of the lab's science.

Raw data is inviolable

Raw data — files as they came off the instrument, from the sequencing core, or from a collaborator — must not be modified. Ever. All processing lifts from the raw source and creates copies or derivatives. The original files remain intact, on backed-up storage, indefinitely. This is what makes it possible to return to the beginning of any analysis and reconstruct exactly what was done and why.

Scratch vs. permanent record

Not all data needs to be kept forever. A useful mental distinction: scratch data lives on HPCs and local machines during active work. It is temporary, exploratory, and disposable. The permanent record lives on the Raredon Lab server. Transfer clean, well-annotated files there — important intermediates, final processed objects, and anything you would need to reconstruct your analysis years later. The server is backed up; your laptop and HPC scratch space are not permanent.

Linear workflows are reproducible workflows

Every script should load its inputs from disk and save its outputs to disk. Dependencies on the working environment of a prior script — objects that were created but never saved — produce non-linear workflows that cannot be reproduced without running all prior scripts in the same session. See Document 04 §7 for the full standard and §7 of this document for a concrete failure example.

§2 What goes where

The lab uses three distinct storage locations. Each has a specific role. Understanding the difference prevents data loss, duplication, and confusion when multiple people need to access the same files.

§3 Raw data

Where raw data is stored

Labeling imaging data at acquisition

Poor file naming at the microscope is one of the most common and most costly data management failures in this type of lab. Microscopy data can accumulate rapidly into terabytes of files that become impossible to sort through without good names. Take a few minutes at the instrument to initialize file names that uniquely identify the sample, the experiment, the date, and any other parameters you will need later. You will not regret this investment. Expert microscopists are expert archivists: they keep detailed records that allow them to find and interpret images months or years after acquisition.

Build a README or markdown file for each imaging dataset describing what was collected, under what conditions, and where the files live. This is especially important for large imaging projects where no single person may remember all the details.

§4 Processed outputs and save decisions

What to save and what to discard

Every time a script runs, it produces outputs. Not all of them need to be kept. Too many intermediate files clutter the server, make navigation confusing, and create a false sense of progress. Too few, and you lose the ability to re-enter a pipeline midway without rerunning everything from scratch. Good judgment about save points — what the PI calls "elegant save points" — is one of the skills developed over time in this lab.

A useful test: if this file disappeared, would I have to rerun a substantial pipeline to recreate it? If yes, keep it. If it could be regenerated in minutes from a nearby upstream object, it is probably not worth the server space. Good judgment about what constitutes an elegant save point is one of the skills developed over time in computational work.

When in doubt, save. You can always delete a file later; you cannot always recreate it.

Metadata changes: save as CSV, not as a new object

One of the most common and most wasteful practices in single-cell analysis is re-saving a large data object (50+ GB) simply because its metadata has been updated: a new cell type annotation, a corrected sample label, an added QC column. The object itself has not changed — only the metadata attached to it has.

The correct practice is to save only the metadata as a CSV file, and to load both the base object and the metadata CSV at the start of any downstream script, merging them there. This approach saves an enormous amount of server space and makes the annotation history transparent and editable.

Versioning and post-publication housekeeping

During active development, maintain an archival record: keep intermediates and version trees that might need to be revisited. After a paper is published, do a round of computational housekeeping: discard intermediate files that became defunct, and preserve a clean, well-organized record showing the essential path from raw data to published result. The goal at publication is a pipeline that a new user could follow — linear, non-redundant, well-labeled — without extensive debugging.

When working through a complex pipeline, keep a README or markdown note in the data directory documenting what each major saved object contains, where it came from, and what it is used for. This navigation aid is invaluable when returning to a project after months away.

§5 Documentation

README and markdown files for data directories

Any data directory with more than a handful of files benefits from a README or markdown file describing its contents. Include: what each major file contains, where it came from, when it was created, and what downstream scripts use it. A few sentences per file is enough. This takes minutes to write and saves hours when someone (including yourself) needs to navigate the directory later.

The computational lab notebook

A highly recommended practice, especially for new lab members: keep a daily computational lab notebook. This is simply a text file or Word document where, at the end of each working day, you record:

• The date
• What you worked on
• Key file paths and script names used or created
• Decisions made and reasoning behind them
• Problems encountered and how they were resolved (or not)
• Open questions for the next session

This document is searchable. When you need to find a file you created six months ago, or reconstruct why a particular parameter was chosen, the notebook is where you look. The investment is a few minutes per day. The payoff compounds over time, and the habit is worth building early.

§6 Sharing data with external collaborators

Sharing large data objects with people outside the lab is a recurring practical challenge. The best option in most cases is SharePoint or OneDrive: files can be shared as links, accessed without VPN, and do not require the collaborator to have Yale credentials for a simple download.

Before sharing any data externally, confirm with the PI that the data is appropriate to share, that any relevant data sharing agreements are in place, and that the files are clearly labeled and documented so the recipient can use them without a lengthy explanation. Provenance information — where the data came from, how it was processed — should travel with the data.

§7 Common mistakes to avoid

Saving too many intermediate objects

Every iterative change to a data object does not need to be saved permanently. If you find yourself with dozens of near-duplicate files differing only in minor parameter choices, consolidate. Use good judgment about what represents a meaningful checkpoint versus a transient state. Make later users' lives easier, including your own.

Re-saving large objects for metadata changes

If only the metadata attached to a large object has changed — a new annotation column, a corrected label — save the metadata alone as a CSV. Do not save another copy of the full object. Load and merge at the start of downstream scripts. This single habit can save hundreds of gigabytes of server space across a project. See §4 for the correct pattern.

Poor labeling of imaging data at acquisition

Generic file names assigned by the microscope software ("image001.tiff", "scan_001") become impossible to interpret within weeks. Set meaningful file names at the instrument before you step away from the microscope. The few minutes this takes at acquisition saves many frustrated hours later.

Accumulating microscopy data without curation

Microscopy data can grow rapidly into terabytes. Be selective about what you acquire and retain. Think of yourself as a photographer: choose what to record thoughtfully, rather than recording everything and sorting it out later. Maintain a README for each imaging dataset so that others can understand what is there and why.

Non-linear workflows

A non-linear workflow is one where a script depends on the working environment of a previous script — on objects that were created but never saved to disk. The following is a concrete example of how this failure mode arises:

# Script 01: loads object, adds metadata column, but does NOT save
obj <- readRDS(path_input)
obj$new_annotation <- c(...)   # modifies metadata
# ...analysis continues, obj never saved to disk

# Script 02: assumes obj is already in the environment
# This script will fail in a clean session
result <- downstream_analysis(obj)  # obj does not exist

# Script 01: saves all outputs before finishing
obj <- readRDS(path_input)
obj$new_annotation <- c(...)
write.csv(obj@meta.data, path_metadata_out)   # save metadata
saveRDS(obj, path_obj_out)                    # save object if needed

# Script 02: loads from disk in a clean environment
obj      <- readRDS(path_obj_out)
metadata <- read.csv(path_metadata_out)
result   <- downstream_analysis(obj)

When you notice that a script only works after running another script in the same session, stop and fix the dependency before it becomes embedded in your workflow. Catch these early — they are much harder to untangle later.