Speaker
Dr
Martin Rehr
(Niels Bohr Institute, University of Copenhagen)
Description
The Image Data Management System (IDMS) is a platform for data management, data analysis and data visualization. Storing, processing and visualizing scientific image data-sets is a challenge because the data streams from the large X-Ray and neutron facilities increase at a rate exceeding the rate in which generic disk drives grow. Furthermore processing and visualizing these image data-sets are just as big a challenge as storing the data due to the limited processing power of generic desktop workstations.
IDMS offers distributed storage with support for group collaborations. The offered storage scales horizontally with respect to capacity and new storage is added transparently to the users. IDMS storage is accessed either through a web interface or through IO services such as sftp, webdavs and ftps. On top of the storage capacity IDMS supports event based distributed computing mechanisms. This event system enables user-defined data processing pipelines based on storage system level events such as uploading directories or files to the system. Finally IDMS supports automatic preview generation of the scientific image data as well as in-browser 2D and 3D viewing of the created previews.
One example of an image storage, processing and previewing pipeline is CT-Reconstruction. The CT-Reconstruction task is compute intensive and well suited for distributed computing. Running it through IDMS using a number of distributed compute nodes decreases the reconstruction time significantly. If a CT-reconstruction work-flow is specified on a specific folder then previews are automatically generated for the projections uploaded to IDMS. When all projections are uploaded the system uses distributed compute nodes to perform a 3D reconstruction of the projections. Finally 2D slice and 3D volume previews are generated from the reconstructed tomogram. Once the processing pipeline is specified all these processing and preview generation steps occur automatically without any user interaction.
The generated previews are displayed through the web interface of the system along with basis statistics, histogram data and a contrast adjustment slider. This is shown for the projections in [figure 1]. Preview of the tomograms can be either 2D slices or 3D volume rendering shown in [figure 2] and [figure 3]. The 3D rendering is performed real-time and the user has the possibility to interact with the volume through the browser. Using IDMS for image previews has several advantages. First of all only the
domain experts that produce the image data need to know the raw binary format such as width, height, data-type and offset. Secondly users can eliminate the step of opening images through a third party application such as MatLab or ImageJ which is time consuming when dealing with a large number of different projects and images. Thirdly once the preview settings are in place it's easy to share image previews between collaborators within a group as the IDMS data URL pointing to the data is all that is needed.
[figure 1]: http://www.migrid.org/vgrid/eScience/Projects/NBI/IDMC/NOBUGS/figs/skull_proj.pdf
[figure 2]: http://www.migrid.org/vgrid/eScience/Projects/NBI/IDMC/NOBUGS/figs/skull_tomo_2D.pdf
[figure 3]: http://www.migrid.org/vgrid/eScience/Projects/NBI/IDMC/NOBUGS/figs/skull_tomo_3D.pdf
Primary author
Dr
Martin Rehr
(Niels Bohr Institute, University of Copenhagen)
Co-author
Mr
Jonas Bardino
(eScience, NBI, University of Copenhagen)
