Archive for the ‘NIMS’ Category

NIMS Transition to Flywheel: Part One

February 9th, 2018

BACKGROUND

When planning the CNI we committed to providing the community with data management services. As many of you know, most MRI centers do little more than hand the user a DVD at the end of the session, and wish them well. CNI users are supported much more extensively. Data acquired on our scanner are immediately transferred to the Neurobiological Image Management System (NIMS). Once in NIMS, the data can be reconstructed and converted into the formats that most of our community uses (e.g., NIfTI). These are the data that most users download and use in their research.

Over the years the CNI community has accumulated a great deal of data. NIMS contains the work of about 900 researchers. These data comprise more than 15,000 sessions from more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 100,000 anatomical scans, around 5000 diffusion scans and over 500 spectroscopy scans.

The CNI remains committed to supporting the research of its user community by providing state-of-the art data management. To this end we are excited about transitioning from NIMS, our home-grown data management system, to Flywheel.

THE ISSUES

Over the years there have been a series of updates to NIMS to accommodate the growing data set. In addition to updates to the NIMS software itself there have been several hardware upgrades over the last few years, including both increased storage and increased computational power; we were fortunate receive grants to support improvements from the Neurosciences Institute. At present NIMS is comprised of a 200TB main file server and a 200TB off-site backup file server, three compute servers (with more than 80 cores, 2TB of RAM, and 14TB of SSD scratch), and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and associated software are maintained by Michael and the CNI staff.

The support, both physical and financial, of this system is a burden on the CNI. Some of you have may have experienced the limitations of NIMS with regard to data reconstructions, or the limited feature set around search, sharing, and permissions, or downloading bulk datasets, or other missing features that fall into the “Wouldn’t it be nice if…” category.

THE SOLUTION

Over the last few years a number of us (Wandell, Schaefer, Dougherty, Perry, and others) were supported by the Simons Foundation to design the next generation of NIMS. This project (The Project on Scientific Transparency) has also been supported by and integrated with the work being done by Russ Poldrack and Chris Gorgolewski (OpenfMRI) – supported by the Arnold Foundation. This joint effort culminated in the development of an open-source framework to support the next generation of data management – SciTran.

Flywheel has adopted SciTran and engineered a modern user interface and infrastructure that makes it easy to use, maintain, and scale. Flywheel has a multitude of new features over NIMS; runs on Google Cloud Platform; and has a sustainable support strategy.

We will be describing Flywheel more fully in the coming weeks, first reaching out to lab managers and individual groups with the goal of rolling it out to the user community broadly in the coming months. We look forward to working with all of you to make the transition smooth and to help you take advantage of new opportunities that are enabled by the next generation system.

 

See you around the magnet!
- The CNI Team

2017 Recap and Upcoming at CNI

January 12th, 2018

As we embark on a new year (Happy New Year!), the CNI team would like to take a moment to highlight some of the things that happened last year and to preview some of the things we’re working on for 2018.

Looking back at 2017

Personnel Changes

In May we said farewell to Bob Dougherty, who ventured off to a new life at a startup company here in the Valley. Bob remains close to the CNI and at the risk of being sappy, we’ll just say we miss him very much.

During his tenure at the CNI Bob had developed great working relationships with others on campus. For this reason, we were both confident and excited to welcome two new additions to the CNI team, both of whom were very active in CNI work. The first is Adam Kerr, who is the new Research Director at the CNI. Hopefully many of you have had a chance to meet and talk with Adam. Adam is a Stanford alum (PhD, EE) and a wonderful addition to the CNI staff.

We were also happy that Michael Perry could increase his efforts at the CNI to become our new IT Director. Michael fully supports the extensive compute infrastructure at the CNI, including the NIMS data management system. Michael is a Berkeley product, but he is excellent in many ways that compensate for this one flaw.

Policy updates

The Advisory Board made some adjustments to the scanning policy at the end of 2017. First, there was a change in how after-hours time (10pm – 7am) can be scheduled. Groups must first meet with Laima to address the increased risks of scanning after-hours.

Second, there was a change to how protocol development time can be used. There is now a cap of 10 hours per grant and the protocol development time for seed grants starting in FY18 will no longer be available.

Please take the time to read through all the scanning policies that are available on the Getting Started page of the WIKI.

DV26 upgrade

After weeks of testing, patching, and more testing, we upgraded the scanner software to DV26 on Monday, Nov 12, 2017. This brought along with it some small changes in operational protocol (e.g. SAR limit settings, spectroscopy prescriptions). For more information about the upgrade, please see this post.

EEG Refresh

This past year we sent out our EEG equipment for service. The EEG caps were refurbished and the software was upgraded to the current release. Please contact the CNI staff for information on how to get started if your group is interested in using the EEG equipment in your study.

Penetration Panels

We had two new 10” x 12” penetration panels installed in the scanner wall common to the equipment room. These penetration panels are used to support the introduction of new instrument connections into the scan room and were a critically required expansion as we had already exhausted all of our research penetration panel space.

Looking forward to 2018

Here are just a few things that we are planning for this next year.

New laptops and E-Prime upgrade

We have purchased new Windows and MAC laptops to support stimulus delivery at the CNI. In the upcoming weeks we will retire some of the old laptops. To help make this transition smooth, please migrate your scripts to the new workstations as soon as possible. Please get in touch with the CNI staff if you have concerns or issues transferring your scripts.

In addition to the workstations, we purchased E-Prime 3 upgrade licenses for our laptops. This version of E-Prime will be installed alongside existing versions, so code requiring older versions will continue to function.

Fee structure updates

Starting March 1, 2018 scan time will increase slightly. Fees will be as follows:

Time SlotHourly Rate

Peak: 8:00 AM to 6:00 PM weekdays $425
Off-hours: 6:00 PM to Midnight weekdays. 8:00 AM to Midnight weekends $325
Owl: Midnight to 8:00 AM weekdays. Midnight to 8:00 AM weekends $100
EEG: Anytime usage $50

CNI Innovation Seed Grants

This year we are again in the fortunate position of being able to support innovation. We now welcome applications for funds to purchase scan time at the CNI. Innovative projects that require up to 30 hours of scanner time for data acquisition will be considered. Note that these funds can only be used for scan time at the CNI. The deadline for CNI Innovation grant applications is Wednesday, January 31st, 2018; award announcements will be made in early February. You can apply for an award using the Google Form here.

NIMS transition to Flywheel

In October of last year we started mirroring NIMS on a new data management system, Flywheel. Data collected on or after October 1, 2017 are already in the Flywheel system, in addition to being in NIMS. This has allowed a few groups to begin testing this next generation system. While NIMS will continue to operate as usual for the time-being, we expect to transition to Flywheel during this year, with new data going only to Flywheel by the Summer 2018.

Flywheel is running on Google Cloud, which allows us to expand data storage without purchasing and installing new hardware. We also save costs of system management (e.g., backups, disk replacements, operating system upgrades). The hardware that we purchased in the past, and which is running out of warranty, will be maintained for the time being. But we expect that it will start to fail over the next few years and be replaced by Cloud systems.

The Flywheel system originated from NIMS, but it has a large number of features that make it feel different and also enable certain new capabilities. We believe the data management features include everything in NIMS but extend them significantly. Most importantly the Flywheel system will integrate tightly with scalable Cloud Computing. We anticipate that this system will support reproducible research and data sharing.

Expect to hear a lot from our team about the new Flywheel system during the upcoming weeks. We will offer group and individual training sessions that cover a number of topics, including:

  • Data Migration Efforts
  • DV26 Impact on reconstruction and data capture in Flywheel
  • Flywheel Center Edition training
  • Flywheel Lab Edition computational methods
  • User rights management training and demonstrations

If you or members of your group would like to be part of the pilot program with Flywheel, please contact Michael Perry, who is leading this effort.

See you around the magnet!

- The CNI Team

Updates for multiband reconstruction

March 15th, 2017

The CNI has recently introduced a new option for reconstructing SMS (aka, multiband or mux) scans. The default reconstruction method in the SMS reconstruction pipeline is currently 1D-GRAPPA (Blaimer M. et al. MRM 2013). Based on recent research and testing, we believe that the split-slice-GRAPPA (Cauley SF, et al. MRM 2014) reconstruction algorithm does a better job at unaliasing the simultaneously acquired slices, especially in cases where the calibration data are corrupted by subject motion. This more robust unaliasing will help reduce the chance of false correlations in fMRI scans by reducing signal leakage across aliased slices.

Another advantage of the split-slice GRAPPA method is that it is less dependent on the image contrast being consistent between the calibration data and the SMS data, therefore it allows more flexibility in choosing the best calibration scans. So, we have also introduced a new SMS calibration scan option – using a separate single-band scan as an external calibration for the target SMS scan. Most of you doing SMS are using internal calibration, i.e. the first few volumes integrated in the SMS scans are used as the calibration data. And a few of you are doing an external calibration that has the same SMS (mux) factor as the target scan. Compared to these calibration methods, the single-band external calibration has higher SNR in the calibration data and is less sensitive to subject motion during the calibration. Therefore we believe it is a more robust calibration method, and in combination with the split-slice-GRAPPA reconstruction method, is likely to produce better image quality for the SMS scans, especially when you have wiggly subjects.

Here is a compelling example of how the single-band calibration scan can reduce a particularly insidious artifact due to excessive eye motion during the calibration scans (this subject was instructed to intentionally move their eyes during the calibration scans). In these standard deviation maps of the BOLD timeseries, the aliased eye artifact is clearly visible in the 1D GRAPPA reconstruction (top). The split-slice GRAPPA reconstruction (middle) shows a reduced eye artifact in the white matter, but there is still significant aliasing of the eyes. This aliasing is substantially reduced or eliminated in the same data reconstructed using a single-band calibration scan with split-slice GRAPPA (bottom), even though that also had subject eye movement.

s1_grappa
s1_ssg
s1_sbref

However, in cases where there is no excessive motion during the calibration, all the methods are quite comparable:

s2_grappa
s2_ssg
s2_sbref

The CNI SMS data processing pipeline will by default keep using the original 1D-GRAPPA reconstruction method for continuity of ongoing studies. And if you have compliant subjects who remain still, ideally with eyes-closed during calibration, this method should be fine. However, if you think your subjects may move during calibration, then we recommend switching to split-slice GRAPPA for image reconstruction. And you may also consider adding a single-band calibration scan to your protocol. Any SMS scan that doesn’t have a separate calibration scan setup in the same scan session will also be reconstructed using the internal calibrations. In order to use the new methods, you need to do the following in your protocol:

To use the split-slice-GRAPPA reconstruction method, include the keyword “_ssg” at the end of each series description that you want to be reconstructed with split-slice-GRAPPA. Note that this also enables SENSE1 coil combination (Sotiropoulos et. al., MRM 2014).

To use the single-band calibration, you need to set up a separate scan with SMS (mux) factor = 1 (CV 22). Include the keyword “_sbref” in the series description. This single-band scan needs to have the same coverage as the multiband scan, i.e. the same FOV and matrix size but X times the prescribed number of slices, X being the SMS (mux) factor. You will need to increase TR (by a few seconds) in order to accommodate all the slices in one TR. You only need 4 or 5 phases for this scan. Because the TR will be longer, you can set the flip angle to 90 to optimize SNR.

For scans using the single-band calibration, the calibration volume is not included in the reconstructed images, i.e. the NIFTI files will only contain the SMS volumes. This information is also shown in the JSON file that now accompanies all SMS NIFTI files in NIMS. The JSON file contains a list of important parameters related to the scan and the reconstruction.

Please contact Hua or Bob if you have any questions and/or want help setting up these new SMS features in your protocol. Also, we owe thanks to the Wagner and Poldrack labs for help in testing these new methods.

Data management and storage at the CNI

May 31st, 2016

In planning the CNI environment, we made a risky decision: We committed to providing the community with data management services. Many of you know that most MRI centers do no more than hand the user a DVD at the end of the session, and wish them well. Or perhaps they allow the user to copy the data from the center to their lab over the Internet.  CNI users are supported much more extensively.

Data acquired on our scanner are immediately transferred from the GE system to the Neurobiological Image Management System (NIMS), a database. As they are placed in the database, the MRI data are converted into the formats (such as NIfTI) that most of our community uses.  These are the data that you typically download from a browser.

The full set of MRI data are kept online, backed-up, and available forever. The data are stored in an organized format that your collaborators can appreciate and understand. You can search through the entire database to discover what is there. You can perform simple visualizations and check image quality of your own data. The NIMS software you are using was designed and implemented by Bob Dougherty, Gunnar Schaefer and Reno Bowen.

In its 5th year of operation, the CNI has accumulated a great deal of data.  We are storing the work of about 700 people (trained by Laima). The data comprise more than 11,000 sessions and more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 10,000 anatomical scans, 5000 diffusion scans and 500 spectroscopy scans. You can search through the system and request access to scans carried out by other labs. Nobody is forced to share their data; but if you would like to share with another scientist, and your IRB permits it, then you can do so with a few clicks.

Over the years there have been a series of updates to the system to accommodate the growing data set.  There have been several hardware upgrades over the last few years, including both increased storage and increased computational power. The CNI data management system now includes a 200 Terabyte main file server with 200TB of off-site backup storage, three compute servers with a total of 80 cores, 1.9 TB of RAM, and 14 TB of fast SSD scratch storage, and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and software are being maintained by Michael Perry and Bob.

Over the last three years a number of us (Gunnar Schaefer, Michael Perry, Bob Dougherty, Renzo Frigato) have been supported by the Simons Foundation to design the next generation of this software.  This work has also been supported by and integrated with the work being done by Russ Poldrack and Chris Filo, supported by the Arnold Foundation.  The next generation of software has many new features, and we will start to tell the story of the next generation of data management software during the coming months.