NIMS | Center for Cognitive and Neurobiological Imaging

Archive for the ‘NIMS’ Category

Flywheel: Part Four

August 29th, 2018

Flywheel

We are on track to switch over to Flywheel on Saturday, September 1. As of that day, new data collected at CNI will not be placed in NIMS. New data will be sent to Flywheel directly (https://cni.flywheel.io). NIMS will be kept online as legacy only.

Logins for the Flywheel site is managed by Stanford authentication, via Google. So to login click the ‘Sign in with Google’ button.

Login page of the Flywheel data management system

Freezing NIMS

Users often make changes to NIMS after data are uploaded, so we will permit edits (e.g., notes) until Saturday, Sept. 8th. However, any edits made after August 31 will not be automatically carried over to Flywheel (which we have been doing). After Sept. 8th we plan to freeze the NIMS database and make it read-only.

Data from October 1 2018 are already in the new Flywheel system. NIMS contains about 7 years worth of legacy data. Over the next year we plan to migrate all the NIMS data into Flywheel as well. If you would like specific Projects in NIMS integrated sooner, please let us know.

NIMS-FS

People who use NIMS-FS (through the CNI Linux Containers) should note that no new data will be accessible through that mechanism. Those wishing for “command line access” to CNI data will need to use a different technology. The Flywheel CLI (command line interface) is an option for those of you who have used the features in NIMS-FS. The CLI has methods for importing and exporting data from your authorized projects, and it runs on multiple platforms. The CLI can be downloaded from your profile page within Flywheel. If you’re interested, please contact Michael Perry for instructions about how to install and use the Flywheel CLI.

- The CNI Team

Flywheel: Part Three

August 6th, 2018

Thanks to all of you who have started to take a look at the Flywheel data management system. A few notes and reminders.

1. We are still on track for a September 1 change over. Having said this, Michael is considering transitioning a little bit sooner. The reason is that the very heavy use of the scanner this month is stressing the limits of our NIMS storage. But we may be able to make it to September 1st with both NIMS and Flywheel running.

2. Starting this week, Michael will join Laima for a portion of the new user training to explain the Flywheel interface . Knowledge is viral around this place – so check with new users, Michael, and Laima if you have questions. And pass on what you learn. There is also a plan to step up the documentation by mid-September.

3. A number of people have asked us about the Flywheel technology. Here are a few principles.

Cloud. The Flywheel system (https://cni.flywheel.io) is on the Google Cloud Platform. This is an advantage to the CNI because we no longer have to maintain computers and back up the data. In the cloud we can leverage (and pay for) more computing power for a short time if there is a big or rush job. So far, our experience with pricing is good.

Data storage. The CNI has more than seven years of data including more than a 100,000 acquisitions. The size of the data has increased dramatically with the new SMS sequences that many of you use (ask me why when you take my class). We have always worried about the cost of storing the data. The largest files (e.g., the P-files) comprise almost 80% of our storage and these data are rarely used. The Flywheel system uses ‘object storage’, which is the least expensive type of storage. Rarely accessed files drift off into ‘cold storage’, which is particularly inexpensive. This should prove to be cost effective for us while still allowing us to reconstruct data that was collected even a few years ago. Which we have done for some users!

Computation. The computations performed by the Center Edition run in a simple framework that Flywheel calls a Utility Gear; this is a Docker Container coupled with a format for setting the program parameters. Flywheel jobs, such as reconstructing SMS scans, or converting a file from DICOM to NIfTI, are carried out by Utility Gears. In the cloud, we can invoke multiple Gears at once on many machines that are provided temporarily (cloud scaling). We pay for the machines only for the few minutes or hours they are in use. Cloud-scaling is useful when we have to run a lot of jobs – such as re-running multiple recons to fix an error or improve an analysis. This functionality provides us with greater scalability and performance than we could provide through the computers we purchased with grant funding.

4. To install Flywheel, the CNI team had to keep NIMS running and develop a system to convert seven years of prior data; Michael has done a great job with those tasks. CNI is not the first site to run Flywheel on the Google Cloud because those sites had no legacy storage. When we fully switch over, the CNI installation will be the largest by far, and we are determined to be the best! Go Cardinal!

NIMS transition to Flywheel: Part Two

July 17th, 2018

Main point

The Flywheel data management system has been running smoothly for several months at the CNI. The main purpose of this message is to encourage you to explore Flywheel (https://cni.flywheel.io). A number of groups have already switched from NIMS to take advantage of Flywheel’s features. After Michael completes further testing, which we anticipate to be around September 1, we will stop sending new MRI data to NIMS and instead send data directly to Flywheel (see note). So this is a good time to learn about the new system.

During the Fall we will hold informal sessions at the CNI to explain the system and its features. We will also write more blog posts to explain the system and what motivated us to make the transition.

About Flywheel

Flywheel offers two levels of service, the Center and Lab Editions; the CNI runs the Center Edition. This version does all the things that NIMS does, including automatically uploading data (e.g., P-files (raw data), DICOM data, and Physio data) and converting data to NIfTI format, which is what most users download.

The Center edition also has features that exceed what NIMS could do. Below is a partial list of Flywheel features that are relevant to CNI users.

Session Templates – For each project you can set up rules that automatically check that the expected data were collected, and flag sessions with missing data
Notes - Add notes about a session or acquisition (e.g., subject moved; life is good; buy milk)
Elastic Search – Find specific types of data, notes, and more
Visualization – Flywheel includes viewers for many types of data (NIfTI, DICOM, EEG, OBJ, CSV, etc.)
Collections – Gather data from multiple projects to form a Collection of data
Permissions – Adding and removing access to users is simpler, and there are different permission levels available
Upload – You can add behavioral data, say from REDCap or or your own notes, to an existing session
Gears – Users can run one of a set of Utility Gears (Analysis Gears are available in the Lab Edition)
Gear Rules – Users can determine which set of Utility Gears are run on their data. For example, want to use dcm2niix for your DICOM data? Simply setup a Gear Rule to execute dcm2niix for all DICOM data coming in to a given project (ask Michael to show you how, or have a look at the doc).

There are many other features we think CNI users will benefit from. Please explore the interface and browse the documentation available in the UI.

Finally, please reach out to Michael Perry (lmperry@stanford.edu) if you have any questions or experience issues with the platform. In this early phase we expect things to come up and we welcome the opportunity to help navigate through this transition.

Note: During this testing phase there is a delay before the data appear in Flywheel. Most data are uploaded first to NIMS, where we start most reconstructions and data processing. When testing is completed, in September, Flywheel will handle all reconstructions and data processing while NIMS will switch over to an archival mode.

The CNI Team

NIMS Transition to Flywheel: Part One

February 9th, 2018

BACKGROUND

When planning the CNI we committed to providing the community with data management services. As many of you know, most MRI centers do little more than hand the user a DVD at the end of the session, and wish them well. CNI users are supported much more extensively. Data acquired on our scanner are immediately transferred to the Neurobiological Image Management System (NIMS). Once in NIMS, the data can be reconstructed and converted into the formats that most of our community uses (e.g., NIfTI). These are the data that most users download and use in their research.

Over the years the CNI community has accumulated a great deal of data. NIMS contains the work of about 900 researchers. These data comprise more than 15,000 sessions from more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 100,000 anatomical scans, around 5000 diffusion scans and over 500 spectroscopy scans.

The CNI remains committed to supporting the research of its user community by providing state-of-the art data management. To this end we are excited about transitioning from NIMS, our home-grown data management system, to Flywheel.

THE ISSUES

Over the years there have been a series of updates to NIMS to accommodate the growing data set. In addition to updates to the NIMS software itself there have been several hardware upgrades over the last few years, including both increased storage and increased computational power; we were fortunate receive grants to support improvements from the Neurosciences Institute. At present NIMS is comprised of a 200TB main file server and a 200TB off-site backup file server, three compute servers (with more than 80 cores, 2TB of RAM, and 14TB of SSD scratch), and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and associated software are maintained by Michael and the CNI staff.

The support, both physical and financial, of this system is a burden on the CNI. Some of you have may have experienced the limitations of NIMS with regard to data reconstructions, or the limited feature set around search, sharing, and permissions, or downloading bulk datasets, or other missing features that fall into the “Wouldn’t it be nice if…” category.

THE SOLUTION

Over the last few years a number of us (Wandell, Schaefer, Dougherty, Perry, and others) were supported by the Simons Foundation to design the next generation of NIMS. This project (The Project on Scientific Transparency) has also been supported by and integrated with the work being done by Russ Poldrack and Chris Gorgolewski (OpenfMRI) – supported by the Arnold Foundation. This joint effort culminated in the development of an open-source framework to support the next generation of data management – SciTran.

Flywheel has adopted SciTran and engineered a modern user interface and infrastructure that makes it easy to use, maintain, and scale. Flywheel has a multitude of new features over NIMS; runs on Google Cloud Platform; and has a sustainable support strategy.

We will be describing Flywheel more fully in the coming weeks, first reaching out to lab managers and individual groups with the goal of rolling it out to the user community broadly in the coming months. We look forward to working with all of you to make the transition smooth and to help you take advantage of new opportunities that are enabled by the next generation system.

See you around the magnet!
- The CNI Team

2017 Recap and Upcoming at CNI

January 12th, 2018

As we embark on a new year (Happy New Year!), the CNI team would like to take a moment to highlight some of the things that happened last year and to preview some of the things we’re working on for 2018.

Looking back at 2017

Personnel Changes

In May we said farewell to Bob Dougherty, who ventured off to a new life at a startup company here in the Valley. Bob remains close to the CNI and at the risk of being sappy, we’ll just say we miss him very much.

During his tenure at the CNI Bob had developed great working relationships with others on campus. For this reason, we were both confident and excited to welcome two new additions to the CNI team, both of whom were very active in CNI work. The first is Adam Kerr, who is the new Research Director at the CNI. Hopefully many of you have had a chance to meet and talk with Adam. Adam is a Stanford alum (PhD, EE) and a wonderful addition to the CNI staff.

We were also happy that Michael Perry could increase his efforts at the CNI to become our new IT Director. Michael fully supports the extensive compute infrastructure at the CNI, including the NIMS data management system. Michael is a Berkeley product, but he is excellent in many ways that compensate for this one flaw.

Policy updates

The Advisory Board made some adjustments to the scanning policy at the end of 2017. First, there was a change in how after-hours time (10pm – 7am) can be scheduled. Groups must first meet with Laima to address the increased risks of scanning after-hours.

Second, there was a change to how protocol development time can be used. There is now a cap of 10 hours per grant and the protocol development time for seed grants starting in FY18 will no longer be available.

Please take the time to read through all the scanning policies that are available on the Getting Started page of the WIKI.

DV26 upgrade

After weeks of testing, patching, and more testing, we upgraded the scanner software to DV26 on Monday, Nov 12, 2017. This brought along with it some small changes in operational protocol (e.g. SAR limit settings, spectroscopy prescriptions). For more information about the upgrade, please see this post.

EEG Refresh

This past year we sent out our EEG equipment for service. The EEG caps were refurbished and the software was upgraded to the current release. Please contact the CNI staff for information on how to get started if your group is interested in using the EEG equipment in your study.

Penetration Panels

We had two new 10” x 12” penetration panels installed in the scanner wall common to the equipment room. These penetration panels are used to support the introduction of new instrument connections into the scan room and were a critically required expansion as we had already exhausted all of our research penetration panel space.

Looking forward to 2018

Here are just a few things that we are planning for this next year.

New laptops and E-Prime upgrade

We have purchased new Windows and MAC laptops to support stimulus delivery at the CNI. In the upcoming weeks we will retire some of the old laptops. To help make this transition smooth, please migrate your scripts to the new workstations as soon as possible. Please get in touch with the CNI staff if you have concerns or issues transferring your scripts.

In addition to the workstations, we purchased E-Prime 3 upgrade licenses for our laptops. This version of E-Prime will be installed alongside existing versions, so code requiring older versions will continue to function.

Fee structure updates

Starting March 1, 2018 scan time will increase slightly. Fees will be as follows:

Time SlotHourly Rate

Peak: 8:00 AM to 6:00 PM weekdays	$425
Off-hours: 6:00 PM to Midnight weekdays. 8:00 AM to Midnight weekends	$325
Owl: Midnight to 8:00 AM weekdays. Midnight to 8:00 AM weekends	$100
EEG: Anytime usage	$50

CNI Innovation Seed Grants

This year we are again in the fortunate position of being able to support innovation. We now welcome applications for funds to purchase scan time at the CNI. Innovative projects that require up to 30 hours of scanner time for data acquisition will be considered. Note that these funds can only be used for scan time at the CNI. The deadline for CNI Innovation grant applications is Wednesday, January 31st, 2018; award announcements will be made in early February. You can apply for an award using the Google Form here.

NIMS transition to Flywheel

In October of last year we started mirroring NIMS on a new data management system, Flywheel. Data collected on or after October 1, 2017 are already in the Flywheel system, in addition to being in NIMS. This has allowed a few groups to begin testing this next generation system. While NIMS will continue to operate as usual for the time-being, we expect to transition to Flywheel during this year, with new data going only to Flywheel by the Summer 2018.

Flywheel is running on Google Cloud, which allows us to expand data storage without purchasing and installing new hardware. We also save costs of system management (e.g., backups, disk replacements, operating system upgrades). The hardware that we purchased in the past, and which is running out of warranty, will be maintained for the time being. But we expect that it will start to fail over the next few years and be replaced by Cloud systems.

The Flywheel system originated from NIMS, but it has a large number of features that make it feel different and also enable certain new capabilities. We believe the data management features include everything in NIMS but extend them significantly. Most importantly the Flywheel system will integrate tightly with scalable Cloud Computing. We anticipate that this system will support reproducible research and data sharing.

Expect to hear a lot from our team about the new Flywheel system during the upcoming weeks. We will offer group and individual training sessions that cover a number of topics, including:

Data Migration Efforts
DV26 Impact on reconstruction and data capture in Flywheel
Flywheel Center Edition training
Flywheel Lab Edition computational methods
User rights management training and demonstrations

If you or members of your group would like to be part of the pilot program with Flywheel, please contact Michael Perry, who is leading this effort.

See you around the magnet!

- The CNI Team

Updates for multiband reconstruction

March 15th, 2017

The CNI has recently introduced a new option for reconstructing SMS (aka, multiband or mux) scans. The default reconstruction method in the SMS reconstruction pipeline is currently 1D-GRAPPA (Blaimer M. et al. MRM 2013). Based on recent research and testing, we believe that the split-slice-GRAPPA (Cauley SF, et al. MRM 2014) reconstruction algorithm does a better job at unaliasing the simultaneously acquired slices, especially in cases where the calibration data are corrupted by subject motion. This more robust unaliasing will help reduce the chance of false correlations in fMRI scans by reducing signal leakage across aliased slices.

Another advantage of the split-slice GRAPPA method is that it is less dependent on the image contrast being consistent between the calibration data and the SMS data, therefore it allows more flexibility in choosing the best calibration scans. So, we have also introduced a new SMS calibration scan option – using a separate single-band scan as an external calibration for the target SMS scan. Most of you doing SMS are using internal calibration, i.e. the first few volumes integrated in the SMS scans are used as the calibration data. And a few of you are doing an external calibration that has the same SMS (mux) factor as the target scan. Compared to these calibration methods, the single-band external calibration has higher SNR in the calibration data and is less sensitive to subject motion during the calibration. Therefore we believe it is a more robust calibration method, and in combination with the split-slice-GRAPPA reconstruction method, is likely to produce better image quality for the SMS scans, especially when you have wiggly subjects.

Here is a compelling example of how the single-band calibration scan can reduce a particularly insidious artifact due to excessive eye motion during the calibration scans (this subject was instructed to intentionally move their eyes during the calibration scans). In these standard deviation maps of the BOLD timeseries, the aliased eye artifact is clearly visible in the 1D GRAPPA reconstruction (top). The split-slice GRAPPA reconstruction (middle) shows a reduced eye artifact in the white matter, but there is still significant aliasing of the eyes. This aliasing is substantially reduced or eliminated in the same data reconstructed using a single-band calibration scan with split-slice GRAPPA (bottom), even though that also had subject eye movement.

However, in cases where there is no excessive motion during the calibration, all the methods are quite comparable:

The CNI SMS data processing pipeline will by default keep using the original 1D-GRAPPA reconstruction method for continuity of ongoing studies. And if you have compliant subjects who remain still, ideally with eyes-closed during calibration, this method should be fine. However, if you think your subjects may move during calibration, then we recommend switching to split-slice GRAPPA for image reconstruction. And you may also consider adding a single-band calibration scan to your protocol. Any SMS scan that doesn’t have a separate calibration scan setup in the same scan session will also be reconstructed using the internal calibrations. In order to use the new methods, you need to do the following in your protocol:

To use the split-slice-GRAPPA reconstruction method, include the keyword “_ssg” at the end of each series description that you want to be reconstructed with split-slice-GRAPPA. Note that this also enables SENSE1 coil combination (Sotiropoulos et. al., MRM 2014).

To use the single-band calibration, you need to set up a separate scan with SMS (mux) factor = 1 (CV 22). Include the keyword “_sbref” in the series description. This single-band scan needs to have the same coverage as the multiband scan, i.e. the same FOV and matrix size but X times the prescribed number of slices, X being the SMS (mux) factor. You will need to increase TR (by a few seconds) in order to accommodate all the slices in one TR. You only need 4 or 5 phases for this scan. Because the TR will be longer, you can set the flip angle to 90 to optimize SNR.

For scans using the single-band calibration, the calibration volume is not included in the reconstructed images, i.e. the NIFTI files will only contain the SMS volumes. This information is also shown in the JSON file that now accompanies all SMS NIFTI files in NIMS. The JSON file contains a list of important parameters related to the scan and the reconstruction.

Please contact Hua or Bob if you have any questions and/or want help setting up these new SMS features in your protocol. Also, we owe thanks to the Wagner and Poldrack labs for help in testing these new methods.

Data management and storage at the CNI

May 31st, 2016

In planning the CNI environment, we made a risky decision: We committed to providing the community with data management services. Many of you know that most MRI centers do no more than hand the user a DVD at the end of the session, and wish them well. Or perhaps they allow the user to copy the data from the center to their lab over the Internet. CNI users are supported much more extensively.

Data acquired on our scanner are immediately transferred from the GE system to the Neurobiological Image Management System (NIMS), a database. As they are placed in the database, the MRI data are converted into the formats (such as NIfTI) that most of our community uses. These are the data that you typically download from a browser.

The full set of MRI data are kept online, backed-up, and available forever. The data are stored in an organized format that your collaborators can appreciate and understand. You can search through the entire database to discover what is there. You can perform simple visualizations and check image quality of your own data. The NIMS software you are using was designed and implemented by Bob Dougherty, Gunnar Schaefer and Reno Bowen.

In its 5th year of operation, the CNI has accumulated a great deal of data. We are storing the work of about 700 people (trained by Laima). The data comprise more than 11,000 sessions and more than 6,000 subjects. There are more than 45,000 fMRI scans, nearly 10,000 anatomical scans, 5000 diffusion scans and 500 spectroscopy scans. You can search through the system and request access to scans carried out by other labs. Nobody is forced to share their data; but if you would like to share with another scientist, and your IRB permits it, then you can do so with a few clicks.

Over the years there have been a series of updates to the system to accommodate the growing data set. There have been several hardware upgrades over the last few years, including both increased storage and increased computational power. The CNI data management system now includes a 200 Terabyte main file server with 200TB of off-site backup storage, three compute servers with a total of 80 cores, 1.9 TB of RAM, and 14 TB of fast SSD scratch storage, and a powerful web server, all interconnected by a 10 Gigabit network. The NIMS hardware and software are being maintained by Michael Perry and Bob.

Over the last three years a number of us (Gunnar Schaefer, Michael Perry, Bob Dougherty, Renzo Frigato) have been supported by the Simons Foundation to design the next generation of this software. This work has also been supported by and integrated with the work being done by Russ Poldrack and Chris Filo, supported by the Arnold Foundation. The next generation of software has many new features, and we will start to tell the story of the next generation of data management software during the coming months.

Center for Cognitive and Neurobiological Imaging