Flywheel

From CNI Wiki

Jump to: navigation, search

The CNI uses Flywheel for data and computational management. Data from the scanner are automatically uploaded to the Flywheel database. The Flywheel system is on the Google Cloud Platform. Users download data either using a web-browser or another of the methods provided by Flywheel (e.g., command-line-interface or software-development-kit).


At the CNI, as data come into the Flywheel they are automatically processed according to rules determined by the CNI staff. For example, upon ingest PHI data are removed, DICOM and raw PFile data are converted to NIfTI format, and metadata about the scan parameters are read and inserted into the database. Some labs may configure initial processing of the data for individual projects differently. For help with customization groups can contact Flywheel directly or ask Michael Perry (see below).


This page describes Flywheel features that are specific to the CNI. You can learn more about Flywheel using the links below:

  • A 5 minute video overview of Flywheel is available on YouTube
  • Manual pages and basic material about the system are maintained by Flywheel at this site.
  • Links to videos and webinars are on the Flywheel Docs site


Log in to CNI's Flywheel site here: cni.flywheel.io

Authentication

Authentication to Flywheel at the CNI requires a valid (full-account) SUNetID.


New users should ask the CNI staff to create an account.

Data upload

Flywheel captures data directly from the MR scanner via its Connector technology. The scanner produces DICOM (and P-File) data, and those data are automatically packaged and uploaded to Flywheel. For a typical session, series data will be in Flywheel before the session has ended.

Flywheel Sort String (Good)

The Flywheel Connector determines where to place data in the hierarchy using information either entered into the MR console by the user or obtained from the DICOM data.


To make sure the data you collect is sent to the correct Flywheel project for your lab, you must enter a "Flywheel sort string" on the console.


Enter this string into the Patient ID field on the scanner console.


The string format is:

 <subject_label>@<group>/<project>

Flywheel Sort String (Bad)

If you do not enter the sort string correctly, the data will still be sent to Flywheel, but it will not be routed to the correct project. Instead, it will be assigned to the "unknown" group or "Unsorted" project, depending on what Flywheel can determine.


  • If the data are sent to the unknown group, you must ask the Site Admins to retrieve your data. (Michael, Laima).
  • If the data are sent to the "Unsorted" project (case sensitive), you can find it and move it yourself.


Thus, for a given piece of data coming in through a Connector:

 No Group, No Project --> group_id = "unknown", project = "Unsorted"
 No Group, Project --> group_id = "unknown", project = <Project>
 Group, No Project --> group_id = <Group>, project = "Unsorted"

Session Labels and Tags

In some cases, users want to control the name of the session rather than have Flywheel use the default name.


You can control the session name using the AdditionalPatientHistory section on the console.


Set the session label and tags using the following format (without the "<>"):

 label_<your_desired_session_label> 


Flywheel also lets you tag a session to make it easier to find with its search function.


You can set a tag by inserting this string in the AdditionalPatientHistory section.

 tag_<tag_name>

Protected Health Information

Protected Health Information (PHI) is considered High Risk Data according to the Stanford Data Classification Guidelines. PHI is defined as any information that can be used to identify an individual and may relate to their past, present, or future health. By law, this information must be encrypted by law and must be (a) stored in encrypted form, and (b) transmitted only through secure means.

Anonymized research data for publication can be shared without harm. See this Stanford site. Although CNI participants are not medical patients, we treat human subject data with the same PHI status as if they were patients. Specifically, data uploaded to Flywheel strips information from these fields:

  • Patient ID field
  • Date of Birth
  • Medical Record Number
  • Subject Name (first and last)

For PIs, please note we do look for PHI information and over the years we have found that users sometimes insert names, phone numbers and other information that might identify the subject. When we find such information, we delete it. The most likely places that RAs and others insert this information are these fields:

  • Subject code (part of Patient ID)
  • Exam description
  • Series description

Thus, we ask all CNI users: Never use any PHI identifiers when entering data into the scanner!

It is permissible to include the subject's weight and age up to some precision.

At present, to describe the PHI methods in your publications, you might find this summary paragraph helpful.

Data at Stanford's Center for Cognitive and Neurobiological Imaging are securely transferred from the MR scanner directly to a data management system (Flywheel.io) that is running within a Google platform space that is approved for research data. Prior to transfer the data file headers are stripped of fields that may contain subject information (patient id, DOB, MRN, name fields). These procedures meet the Stanford standard for anonymized research data for publication and can be shared without harm. See: this Stanford site.

N.B. Flywheel is planning to provide a substantial enhancement of the anonymization protocol so that each lab will have the ability to control the details of its own anonymization protocol. We will send a note describing the capabilities when the feature is released - which is expected first quarter 2021.

Data Processing on Upload

At upload time data are processed according to a given Project's "Gear Rules". For help with Gear rules please reach out to Michael Perry.

Downloading Data

There are several ways to download data from Flywheel, including via the WEB UI, the command line interface (CLI), and the Flywheel SDK (which is available for both Python and MATLAB). The links below can help you get started with the various methods of export:

UI Downloads


Flywheel SDK


Flywheel CLI

CLI: Tips and Best Practices for CNI Users

This section focuses on a few tips for using the CLI to download your data. For a complete overview of the Flywheel CLI, including how to get started, please look through the Flywheel CLI Documentation.


Tip: Download only the files you need When downloading data from Flywheel using the CLI you can greatly speed up your downloads by excluding data types which are not needed for your analysis.


Example: Exclude pfile and DICOM data from a container download: Most users do not need to download the raw scanner files (PFILES) or raw DICOM data. You can exclude certain data types from your downloads by using the `-e` flag with your CLI download, like so:

 fw download "cni/testproject/subject1/session1" -e pfile -e dicom


This tells the CLI to exclude any pfile and dicom files in the container. Note that you can use consecutive -e flags to exclude multiple data types.


Example: Download only NIfTI, BVEC, and BVAL files:
Most users are only interested in the data that will be input to their analysis pipelines. This is most often limited to three data types (nifti, bvec, and bval). You can use the following command with multiple include flags (`-i`) to accomplish exactly that:

 fw download "cni/testproject/subject1/session1" -i nifti -i bvec -i bval


This will generate an archive (.tar) file containing the requested hierarchy with only those files you explicitly need.


Example: Using quotes Often times your source-path (that is the group/project/subject/session string the describes the location of your data in Flywheel) will have one or more spaces or special characters in it. To properly address that location using the CLI it's important to use quotes around the source path, like so:

 fw download "cni/testproject/subject1/session1"


Example Download a single file with the CLI So you have navigated to a container and your only desire is to download a single file from that container, the best way to do that is using the CLI with the 'files' spec filter.


Example: Download a single NIfTI file from an acquisition container:

 fw download "test/Unsorted/s001/18591/T1w 1mm/files/18591_13_1.nii.gz"


The important bit here is the inclusion of "files" prior to the file name.

Using Sherlock and other Computers

We suggest that you use the Flywheel Command Line Interface (CLI) to transfer data to other compute resources, like Sherlock.


To download and install the CLI on Sherlock we use the `wget` tool, then `unzip` to extract the CLI resource package, and finally modify our `.bashrc` file to add the fw binary as an alias in our environment.


Step 1: Get the URL for the CLI package, which can be found on the "Profile" page within the Flywheel interface. To grab the URL, find the CLI section on the Profile page, right-click the Linux CLI Download link, and choose "Copy Link Address" (or similar). Once you have the download link address move to the next step.


Step 2: Log in to Sherlock and run the `wget` command, using the URL from Step 1 to download the CLI package.

 mkdir -p flywheel/cli
 cd flywheel/cli
 wget //storage.googleapis.com/flywheel-dist/cli/<version>/fw-linux_amd64.zip


Step 3: Unpack the CLI archive and cleanup the downloaded package.

 unzip  fw-linux_amd64.zip
 mv linux_amd64/fw .
 rmdir linux_amd64
 rm -f fw-linux_amd64.zip


Step 4: Modify your `.bashrc` file to add the fw CLI command to your environment, and source it to make the alias active. Note that this only needs to be done once.

 echo -e "alias fw='$HOME/flywheel/cli/fw'" >> $HOME/.bashrc
 source $HOME/.bashrc


Step 5: Once the above steps are complete, you should be able to log in using the CLI and use it as described in the official documentation: //flywheelio.zendesk.com/hc/en-us/sections/360001596834-Command-Line-Interface. The best way to do this is navigate to your profile page in Flywheel, make sure that you have generated an API Key, and use the login command text that is provided for you there.

 fw login <your API key>

Data Migration from NIMS

Before there was Flywheel, there was NIMS. Data have been preserved in NIMS and they can be migrated to Flywheel on an as-needed basis. Please inquire with Michael Perry for more information.

Support

Michael can help with most CNI Flywheel issues. If support is needed for Flywheel issues not related to the CNI, please email their help line: support@flywheel.io.

Personal tools