Flywheel: Part Three

August 6th, 2018
August 6th, 2018

Thanks to all of you who have started to take a look at the Flywheel data management system. A few notes and reminders.

1.  We are still on track for a September 1 change over.  Having said this, Michael is considering transitioning a little bit sooner.  The reason is that the very heavy use of the scanner this month is stressing the limits of our NIMS storage.  But we may be able to make it to September 1st with both NIMS and Flywheel running.

2. Starting this week, Michael will join Laima for a portion of the new user training to explain the Flywheel interface .  Knowledge is viral around this place – so check with new users, Michael, and Laima if you have questions.  And pass on what you learn.  There is also a plan to step up the documentation by mid-September.

3.  A number of people have asked us about the Flywheel technology.  Here are a few principles.

Cloud. The Flywheel system (https://cni.flywheel.io) is on the Google Cloud Platform.  This is an advantage to the CNI because we no longer have to maintain computers and back up the data.  In the cloud we can leverage (and pay for) more computing power for a short time if there is a big or rush job.  So far, our experience with pricing is good.

Data storage. The CNI has more than seven years of data including more than a 100,000 acquisitions.  The size of the data has increased dramatically with the new SMS sequences that many of you use (ask me why when you take my class). We have always worried about the cost of storing the data.  The largest files (e.g., the P-files) comprise almost 80% of our storage and these data are rarely used.  The Flywheel system uses ‘object storage’, which is the least expensive type of storage. Rarely accessed files drift off into ‘cold storage’, which is particularly inexpensive. This should prove to be cost effective for us while still allowing us to reconstruct data that was collected even a few years ago.  Which we have done for some users!

Computation.  The computations performed by the Center Edition run in  a simple framework that Flywheel calls a Utility Gear; this is a Docker Container coupled with a format for setting the program parameters.  Flywheel jobs, such as reconstructing SMS scans, or converting a file from DICOM to NIfTI, are carried out by Utility Gears.  In the cloud, we can invoke multiple Gears at once on many machines that are  provided temporarily (cloud scaling).  We pay for the machines only for the few minutes or hours they are in use.  Cloud-scaling is useful when we have to run a lot of jobs – such as re-running multiple recons to fix an error or improve an analysis. This functionality provides us with greater scalability and performance than we could provide through the computers we purchased with grant funding.

4. To install Flywheel, the CNI team had to keep NIMS running and develop a system to convert seven years of prior data; Michael has done a great job with those tasks. CNI is not the first site to run Flywheel on the Google Cloud because those sites had no legacy storage.  When we fully switch over, the CNI installation will be the largest by far, and we are determined to be the best! Go Cardinal!

 


Comment

You must be registered (with a sunet id) and logged in to post a comment.