Archive for the ‘Computing’ Category

Flywheel: Part Four

August 29th, 2018

Flywheel

We are on track to switch over to Flywheel on Saturday, September 1.  As of that day, new data collected at CNI will not be placed in NIMS.  New data will be sent  to Flywheel directly (https://cni.flywheel.io).  NIMS will be kept online as legacy only.

Logins for the Flywheel site is managed by Stanford authentication, via Google. So to login click the ‘Sign in with Google’ button.

 

Login page of the Flywheel data management system

Login page of the Flywheel data management system

 

Freezing NIMS

Users often make changes to NIMS after data are uploaded, so we will permit edits (e.g., notes) until Saturday, Sept. 8th.  However, any edits made after August 31 will not be automatically carried over to Flywheel (which we have been doing).  After Sept. 8th we plan to freeze the NIMS database and make it read-only.

Data from October 1 2018 are already in the new Flywheel system.  NIMS contains about 7 years worth of legacy data.  Over the next year we plan to migrate all the NIMS data into Flywheel as well. If you would like specific Projects in NIMS integrated sooner, please let us know.

NIMS-FS

People who use NIMS-FS (through the CNI Linux Containers) should note that no new data will be accessible through that mechanism. Those wishing for “command line access” to CNI data will need to use a different technology.  The Flywheel CLI (command line interface) is an option for those of you who have used the features in NIMS-FS.  The CLI has methods for importing and exporting data from your authorized projects, and it runs on multiple platforms.  The CLI can be downloaded from your profile page within Flywheel. If you’re interested, please contact Michael Perry for instructions about how to install and use the Flywheel CLI.

- The CNI Team

 

Flywheel: Part Three

August 6th, 2018

Thanks to all of you who have started to take a look at the Flywheel data management system. A few notes and reminders.

1.  We are still on track for a September 1 change over.  Having said this, Michael is considering transitioning a little bit sooner.  The reason is that the very heavy use of the scanner this month is stressing the limits of our NIMS storage.  But we may be able to make it to September 1st with both NIMS and Flywheel running.

2. Starting this week, Michael will join Laima for a portion of the new user training to explain the Flywheel interface .  Knowledge is viral around this place – so check with new users, Michael, and Laima if you have questions.  And pass on what you learn.  There is also a plan to step up the documentation by mid-September.

3.  A number of people have asked us about the Flywheel technology.  Here are a few principles.

Cloud. The Flywheel system (https://cni.flywheel.io) is on the Google Cloud Platform.  This is an advantage to the CNI because we no longer have to maintain computers and back up the data.  In the cloud we can leverage (and pay for) more computing power for a short time if there is a big or rush job.  So far, our experience with pricing is good.

Data storage. The CNI has more than seven years of data including more than a 100,000 acquisitions.  The size of the data has increased dramatically with the new SMS sequences that many of you use (ask me why when you take my class). We have always worried about the cost of storing the data.  The largest files (e.g., the P-files) comprise almost 80% of our storage and these data are rarely used.  The Flywheel system uses ‘object storage’, which is the least expensive type of storage. Rarely accessed files drift off into ‘cold storage’, which is particularly inexpensive. This should prove to be cost effective for us while still allowing us to reconstruct data that was collected even a few years ago.  Which we have done for some users!

Computation.  The computations performed by the Center Edition run in  a simple framework that Flywheel calls a Utility Gear; this is a Docker Container coupled with a format for setting the program parameters.  Flywheel jobs, such as reconstructing SMS scans, or converting a file from DICOM to NIfTI, are carried out by Utility Gears.  In the cloud, we can invoke multiple Gears at once on many machines that are  provided temporarily (cloud scaling).  We pay for the machines only for the few minutes or hours they are in use.  Cloud-scaling is useful when we have to run a lot of jobs – such as re-running multiple recons to fix an error or improve an analysis. This functionality provides us with greater scalability and performance than we could provide through the computers we purchased with grant funding.

4. To install Flywheel, the CNI team had to keep NIMS running and develop a system to convert seven years of prior data; Michael has done a great job with those tasks. CNI is not the first site to run Flywheel on the Google Cloud because those sites had no legacy storage.  When we fully switch over, the CNI installation will be the largest by far, and we are determined to be the best! Go Cardinal!

 

NIMS transition to Flywheel: Part Two

July 17th, 2018

Main point

The Flywheel data management system has been running smoothly for several months at the CNI.  The main purpose of this message is to encourage you to explore Flywheel (https://cni.flywheel.io). A number of groups have already switched from NIMS to take advantage of Flywheel’s features. After Michael completes further testing, which we anticipate to be around September 1, we will stop sending new MRI data to NIMS and instead send data directly to Flywheel (see note). So this is a good time to learn about the new system.

During the Fall we will hold informal sessions at the CNI to explain the system and its features. We will also write more blog posts to explain the system and what motivated us to make the transition.

About Flywheel

Flywheel offers two levels of service, the Center and Lab Editions; the CNI runs the Center Edition. This version does all  the things that NIMS does, including automatically uploading data (e.g., P-files (raw data), DICOM data, and Physio data) and converting data to NIfTI format, which is what most users download.

The Center edition also has features that exceed what NIMS could do. Below is a partial list of Flywheel features that are relevant to CNI users.

  • Session Templates – For each project you can set up rules that automatically check that the expected data were collected, and flag sessions with missing data
  • Notes - Add notes about a session or acquisition (e.g., subject moved; life is good; buy milk)
  • Elastic Search – Find specific types of data, notes, and more
  • Visualization – Flywheel includes viewers for many types of data (NIfTI, DICOM, EEG, OBJ,  CSV, etc.)
  • Collections – Gather data from multiple projects to form a Collection of data
  • Permissions – Adding and removing access to users is simpler, and there are different permission levels available
  • Upload – You can add behavioral data, say from REDCap or or your own notes, to an existing session
  • Gears – Users can run one of a set of Utility Gears (Analysis Gears are available in the Lab Edition)
  • Gear Rules – Users can determine which set of Utility Gears are run on their data. For example, want to use dcm2niix for your DICOM data? Simply setup a Gear Rule to execute dcm2niix for all DICOM data coming in to a given project (ask Michael to show you how, or have a look at the doc).

There are many other features we think CNI users will benefit from. Please explore the interface and browse the documentation available in the UI.

Finally, please reach out to Michael Perry (lmperry@stanford.edu) if you have any questions or experience issues with the platform. In this early phase we expect things to come up and we welcome the opportunity to help navigate through this transition.

Note:  During this testing phase there is a delay before the data appear in Flywheel. Most data are uploaded first to NIMS, where we start most reconstructions and data processing. When testing is completed, in September, Flywheel will handle all reconstructions and data processing while NIMS will switch over to an archival mode.

The CNI Team

NIMS Transition to Flywheel: Part One

February 9th, 2018

BACKGROUND

When planning the CNI we committed to providing the community with data management services. As many of you know, most MRI centers do little more than hand the user a DVD at the end of the session, and wish them well. CNI users are supported much more extensively. Data acquired on our scanner are immediately transferred to the Neurobiological Image Management System (NIMS). Once in NIMS, the data can be reconstructed and converted into the formats that most of our community uses (e.g., NIfTI). These are the data that most users download and use in their research.

Over the years the CNI community has accumulated a great deal of data. NIMS contains the work of about 900 researchers. These data comprise more than 15,000 sessions from more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 100,000 anatomical scans, around 5000 diffusion scans and over 500 spectroscopy scans.

The CNI remains committed to supporting the research of its user community by providing state-of-the art data management. To this end we are excited about transitioning from NIMS, our home-grown data management system, to Flywheel.

THE ISSUES

Over the years there have been a series of updates to NIMS to accommodate the growing data set. In addition to updates to the NIMS software itself there have been several hardware upgrades over the last few years, including both increased storage and increased computational power; we were fortunate receive grants to support improvements from the Neurosciences Institute. At present NIMS is comprised of a 200TB main file server and a 200TB off-site backup file server, three compute servers (with more than 80 cores, 2TB of RAM, and 14TB of SSD scratch), and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and associated software are maintained by Michael and the CNI staff.

The support, both physical and financial, of this system is a burden on the CNI. Some of you have may have experienced the limitations of NIMS with regard to data reconstructions, or the limited feature set around search, sharing, and permissions, or downloading bulk datasets, or other missing features that fall into the “Wouldn’t it be nice if…” category.

THE SOLUTION

Over the last few years a number of us (Wandell, Schaefer, Dougherty, Perry, and others) were supported by the Simons Foundation to design the next generation of NIMS. This project (The Project on Scientific Transparency) has also been supported by and integrated with the work being done by Russ Poldrack and Chris Gorgolewski (OpenfMRI) – supported by the Arnold Foundation. This joint effort culminated in the development of an open-source framework to support the next generation of data management – SciTran.

Flywheel has adopted SciTran and engineered a modern user interface and infrastructure that makes it easy to use, maintain, and scale. Flywheel has a multitude of new features over NIMS; runs on Google Cloud Platform; and has a sustainable support strategy.

We will be describing Flywheel more fully in the coming weeks, first reaching out to lab managers and individual groups with the goal of rolling it out to the user community broadly in the coming months. We look forward to working with all of you to make the transition smooth and to help you take advantage of new opportunities that are enabled by the next generation system.

 

See you around the magnet!
- The CNI Team

Data management and storage at the CNI

May 31st, 2016

In planning the CNI environment, we made a risky decision: We committed to providing the community with data management services. Many of you know that most MRI centers do no more than hand the user a DVD at the end of the session, and wish them well. Or perhaps they allow the user to copy the data from the center to their lab over the Internet.  CNI users are supported much more extensively.

Data acquired on our scanner are immediately transferred from the GE system to the Neurobiological Image Management System (NIMS), a database. As they are placed in the database, the MRI data are converted into the formats (such as NIfTI) that most of our community uses.  These are the data that you typically download from a browser.

The full set of MRI data are kept online, backed-up, and available forever. The data are stored in an organized format that your collaborators can appreciate and understand. You can search through the entire database to discover what is there. You can perform simple visualizations and check image quality of your own data. The NIMS software you are using was designed and implemented by Bob Dougherty, Gunnar Schaefer and Reno Bowen.

In its 5th year of operation, the CNI has accumulated a great deal of data.  We are storing the work of about 700 people (trained by Laima). The data comprise more than 11,000 sessions and more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 10,000 anatomical scans, 5000 diffusion scans and 500 spectroscopy scans. You can search through the system and request access to scans carried out by other labs. Nobody is forced to share their data; but if you would like to share with another scientist, and your IRB permits it, then you can do so with a few clicks.

Over the years there have been a series of updates to the system to accommodate the growing data set.  There have been several hardware upgrades over the last few years, including both increased storage and increased computational power. The CNI data management system now includes a 200 Terabyte main file server with 200TB of off-site backup storage, three compute servers with a total of 80 cores, 1.9 TB of RAM, and 14 TB of fast SSD scratch storage, and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and software are being maintained by Michael Perry and Bob.

Over the last three years a number of us (Gunnar Schaefer, Michael Perry, Bob Dougherty, Renzo Frigato) have been supported by the Simons Foundation to design the next generation of this software.  This work has also been supported by and integrated with the work being done by Russ Poldrack and Chris Filo, supported by the Arnold Foundation.  The next generation of software has many new features, and we will start to tell the story of the next generation of data management software during the coming months.

 

SMS, EPI and tSNR (long)

March 15th, 2016

Bob, Hua and the CNI team are working with the community to improve our methods.  The text below - excerpted from a recent exchange with the Wagner lab – illustrates the tools we use to exchange data and computational methods.  This software infrastructure is an important part of the CNI research and development mission.

The story below illustrates how one group uncovered a problem and effectively communicate it to the CNI team.  They collaborated to develop a solution that improves on previous methods, and this will benefit the entire community.

In a subsequent post, we will describe our plan of action when we discover that a method can be improved. Such advances happen (routinely we hope!).  Our general plan is to share advances by taking advantage of the database (NIMS) so that we can identify data that could be improved and alert you to apply the improvements.

Bob, Brian and Michael

 

An SMS Story

Bob, Kangrong, Matt, Hua, and Adam have been developing simultaneous multislice methods (SMS) for use at the CNI. Variants of this sequence for rapid, whole-brain acquisitions have been developed and distributed on Siemens, but as far as we know, the CNI implementation is the first fully working system on the GE platform.  The SMS technology allows investigators to collect whole brain fMRI measurements with a repetition time of substantially less than 1 sec even with a resolution as low as 2mm isotropic. For diffusion imaging, SMS allows the scan time to be substantially reduced, allowing more diffusion directions and/or b-value shells to be measured in a reasonable scan time.

Our user community has been very actively involved in testing the CNI implementation.  As with the development of any new technology, feedback from the field is very important.  Since the initial roll out of the method, Karen LaRocque and the team in the Wagner lab have provided particularly useful feedback.

In February, Karen sent a message to Bob noting that

I’m analyzing some multiband (3 band) data and am noticing that the variance in the timeseries seems to vary systematically between the odd-even slices. It’s more apparent for some runs than for others, but is present in multiple runs / subjects.

Karen and her colleagues also shared links to data sets and Jupyter notebooks showing her calculations.

Bob responded with some ideas (the CNI team often uses “mux” as a synonym for SMS)

Hi Karen,

One hypothesis that I have is that this is related to the calibration scan. If the subject moves (even a little along z) during the calibration scan, it can create some inhomogeneity in the signal intensity in the calibration images and that will carry over into the mux scans.

Things to check: 

1. look at the quality of the calibration scan (should be the second volume in the nifti) and see if it correlates with the appearance of the artifact

2. is the tSNR variance caused by changes in the signal or the noise? Or both?

If it is a calibration scan issue, then we can try reconstructing some of of the bad scans using calibration data from a different scan. I will also talk to the MR physics group to see if there are any ideas for a fix in the reconstruction (e.g., smoothing the calibration images). Longer-term, we do hope to develop better calibration scan methods. But that will be some time away.

Also, can you tell me how serious you think this artifact is? Are you thinking that it makes data unusable in some cases? 

thanks,

bob

Hua pitched in, working with Bob to understand the problem.  Karen pointed Hua to their most problematic data set, which could be referenced by a link to the NIMS data base

From: Karen Fossum LaRocque <klarocqu@stanford.edu>
Sent: Wednesday, February 17, 2016 2:10 PM
To: Hua Wu
Cc: Anthony David Wagner; Michael Lawrence Waskom; Valerie Ann Carr; Ian Connors Ballard; Bob Dougherty
Subject: Re: multiband data: odd-even slice variance differences

Hi Hua,

This is my most problematic scan:  https://cni.stanford.edu/nims/auth/browse#main,exp=95441,sess=96169,epoch=96210

Thanks!
karen

After some analyses and experiments, the CNI team found an improved method.

From: Bob Dougherty <bobd@stanford.edu>

Sent: Monday, February 22, 2016 5:30 PM
To: Karen Fossum LaRocque
Cc: Anthony David Wagner; Michael Lawrence Waskom; Valerie Ann Carr; Ian Connors Ballard; Brian A Wandell; Hua Wu
Subject: Re: multiband data: odd-even slice variance differences

Hi Karen,

We’re trying out an alternative recon algorithm (split-slice GRAPPA) and some z-axis smoothing on the calibration images. The ssGRAPPA seems to make the tSNR more uniform across slices, and the smoothing may increase tSNR a bit as well. Here’s the comparison for subject you pointed us to (exam 7840, series 26):

heegieda

I’ve put both files up on your cni lx-container (cnic9) in /data/7840_26_1_weightedavg.nii.gz and /data/7840_26_1_spslgrappa.nii.gz. You should be able to scp the files with ‘scp klarocqu@cnic9:/data/7840_26_1_* .’ (for more info on the cni containers, see http://cni.stanford.edu/wiki/LXC). Note that the image orientation is messed up in these niftis because we’re using our bleeding-edge recon code, which isn’t fully debugged yet. But we hope to get that fixed this week. After more testing, we could be ready to start redoing recons in NIMS sometime next week.