Archive for the ‘Computing’ Category

Data management and storage at the CNI

May 31st, 2016

In planning the CNI environment, we made a risky decision: We committed to providing the community with data management services. Many of you know that most MRI centers do no more than hand the user a DVD at the end of the session, and wish them well. Or perhaps they allow the user to copy the data from the center to their lab over the Internet.  CNI users are supported much more extensively.

Data acquired on our scanner are immediately transferred from the GE system to the Neurobiological Image Management System (NIMS), a database. As they are placed in the database, the MRI data are converted into the formats (such as NIfTI) that most of our community uses.  These are the data that you typically download from a browser.

The full set of MRI data are kept online, backed-up, and available forever. The data are stored in an organized format that your collaborators can appreciate and understand. You can search through the entire database to discover what is there. You can perform simple visualizations and check image quality of your own data. The NIMS software you are using was designed and implemented by Bob Dougherty, Gunnar Schaefer and Reno Bowen.

In its 5th year of operation, the CNI has accumulated a great deal of data.  We are storing the work of about 700 people (trained by Laima). The data comprise more than 11,000 sessions and more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 10,000 anatomical scans, 5000 diffusion scans and 500 spectroscopy scans. You can search through the system and request access to scans carried out by other labs. Nobody is forced to share their data; but if you would like to share with another scientist, and your IRB permits it, then you can do so with a few clicks.

Over the years there have been a series of updates to the system to accommodate the growing data set.  There have been several hardware upgrades over the last few years, including both increased storage and increased computational power. The CNI data management system now includes a 200 Terabyte main file server with 200TB of off-site backup storage, three compute servers with a total of 80 cores, 1.9 TB of RAM, and 14 TB of fast SSD scratch storage, and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and software are being maintained by Michael Perry and Bob.

Over the last three years a number of us (Gunnar Schaefer, Michael Perry, Bob Dougherty, Renzo Frigato) have been supported by the Simons Foundation to design the next generation of this software.  This work has also been supported by and integrated with the work being done by Russ Poldrack and Chris Filo, supported by the Arnold Foundation.  The next generation of software has many new features, and we will start to tell the story of the next generation of data management software during the coming months.


SMS, EPI and tSNR (long)

March 15th, 2016

Bob, Hua and the CNI team are working with the community to improve our methods.  The text below - excerpted from a recent exchange with the Wagner lab – illustrates the tools we use to exchange data and computational methods.  This software infrastructure is an important part of the CNI research and development mission.

The story below illustrates how one group uncovered a problem and effectively communicate it to the CNI team.  They collaborated to develop a solution that improves on previous methods, and this will benefit the entire community.

In a subsequent post, we will describe our plan of action when we discover that a method can be improved. Such advances happen (routinely we hope!).  Our general plan is to share advances by taking advantage of the database (NIMS) so that we can identify data that could be improved and alert you to apply the improvements.

Bob, Brian and Michael


An SMS Story

Bob, Kangrong, Matt, Hua, and Adam have been developing simultaneous multislice methods (SMS) for use at the CNI. Variants of this sequence for rapid, whole-brain acquisitions have been developed and distributed on Siemens, but as far as we know, the CNI implementation is the first fully working system on the GE platform.  The SMS technology allows investigators to collect whole brain fMRI measurements with a repetition time of substantially less than 1 sec even with a resolution as low as 2mm isotropic. For diffusion imaging, SMS allows the scan time to be substantially reduced, allowing more diffusion directions and/or b-value shells to be measured in a reasonable scan time.

Our user community has been very actively involved in testing the CNI implementation.  As with the development of any new technology, feedback from the field is very important.  Since the initial roll out of the method, Karen LaRocque and the team in the Wagner lab have provided particularly useful feedback.

In February, Karen sent a message to Bob noting that

I’m analyzing some multiband (3 band) data and am noticing that the variance in the timeseries seems to vary systematically between the odd-even slices. It’s more apparent for some runs than for others, but is present in multiple runs / subjects.

Karen and her colleagues also shared links to data sets and Jupyter notebooks showing her calculations.

Bob responded with some ideas (the CNI team often uses “mux” as a synonym for SMS)

Hi Karen,

One hypothesis that I have is that this is related to the calibration scan. If the subject moves (even a little along z) during the calibration scan, it can create some inhomogeneity in the signal intensity in the calibration images and that will carry over into the mux scans.

Things to check: 

1. look at the quality of the calibration scan (should be the second volume in the nifti) and see if it correlates with the appearance of the artifact

2. is the tSNR variance caused by changes in the signal or the noise? Or both?

If it is a calibration scan issue, then we can try reconstructing some of of the bad scans using calibration data from a different scan. I will also talk to the MR physics group to see if there are any ideas for a fix in the reconstruction (e.g., smoothing the calibration images). Longer-term, we do hope to develop better calibration scan methods. But that will be some time away.

Also, can you tell me how serious you think this artifact is? Are you thinking that it makes data unusable in some cases? 



Hua pitched in, working with Bob to understand the problem.  Karen pointed Hua to their most problematic data set, which could be referenced by a link to the NIMS data base

From: Karen Fossum LaRocque <>
Sent: Wednesday, February 17, 2016 2:10 PM
To: Hua Wu
Cc: Anthony David Wagner; Michael Lawrence Waskom; Valerie Ann Carr; Ian Connors Ballard; Bob Dougherty
Subject: Re: multiband data: odd-even slice variance differences

Hi Hua,

This is my most problematic scan:,exp=95441,sess=96169,epoch=96210


After some analyses and experiments, the CNI team found an improved method.

From: Bob Dougherty <>

Sent: Monday, February 22, 2016 5:30 PM
To: Karen Fossum LaRocque
Cc: Anthony David Wagner; Michael Lawrence Waskom; Valerie Ann Carr; Ian Connors Ballard; Brian A Wandell; Hua Wu
Subject: Re: multiband data: odd-even slice variance differences

Hi Karen,

We’re trying out an alternative recon algorithm (split-slice GRAPPA) and some z-axis smoothing on the calibration images. The ssGRAPPA seems to make the tSNR more uniform across slices, and the smoothing may increase tSNR a bit as well. Here’s the comparison for subject you pointed us to (exam 7840, series 26):


I’ve put both files up on your cni lx-container (cnic9) in /data/7840_26_1_weightedavg.nii.gz and /data/7840_26_1_spslgrappa.nii.gz. You should be able to scp the files with ‘scp klarocqu@cnic9:/data/7840_26_1_* .’ (for more info on the cni containers, see Note that the image orientation is messed up in these niftis because we’re using our bleeding-edge recon code, which isn’t fully debugged yet. But we hope to get that fixed this week. After more testing, we could be ready to start redoing recons in NIMS sometime next week.