ExaFEL

(LCLS/NERSC)

The objective of the ExaFEL project is to leverage exascale computing to reduce, from weeks to minutes, the time to analyze molecular structure x-ray diffraction data generated by LCLS.  Users of the LCLS require an integrated combination of data processing and scientific interpretation, where both aspects demand intensive computational analysis. The ultrafast x-ray pulses are used like flashes from a high-speed strobe light to produce “stop-action movies” of atoms and molecules. Data analysis must be carried out quickly to allow users to iterate their experiments and extract the most value from scarce beam time. Enabling new photon science from the LCLS will require near-real-time analysis (~10 min) of data bursts, requiring commensurate bursts of exascale-class computational intensities. 

The high repetition rate and ultra-high brightness of the LCLS make it possible to determine the structure of individual molecules, mapping out their natural variation in conformation and flexibility. Structural dynamics and heterogeneities, such as changes in size and shape of nanoparticles, or conformational flexibility in macromolecules, are at the basis of understanding, predicting, and eventually engineering functional properties in the biological, material, and energy sciences. The ability to image these structural dynamics and heterogeneities using noncrystalline-based diffractive imaging, including single-particle imaging (SPI) and fluctuation x-ray scattering, has been one of the driving forces behind the development of x-ray free-electron lasers. However, efficient data processing, classification of diffraction patterns into conformational states, and subsequent reconstruction of a series of 3D electron densities, which allow for visualization of how the structure is changing, are vital computational challenges in diffractive imaging. 

The ExaFEL challenge problem is the creation of an exascale-based data analysis workflow for serial femtosecond crystallography (SFX).  Here, the molecular structure is determined by merging the x-ray diffraction patterns from millions to billions of protein crystals exposed in random orientations.  X-ray free-electron lasers are uniquely suited for studying enzymatic reactions involving biomolecules, as the diffraction pattern is produced before the molecular structure is damaged by radiation.  Fast reaction triggering with optical lasers or rapid mixing allows the observation of time progression, giving a vastly improved understanding of the reaction chemistry.  Exascale computing serves two roles in this regard.  First, it will allow the diffraction pattern to be modeled with greatly enhanced detail, leading to very granular atomic resolution that will follow the path of single atoms reacting within a large molecular complex.  Second, by streaming the experimental data to a supercomputing facility in real time, diffraction quality can be assessed in a matter of minutes.  Such feedback into experimental decisions at the x-ray facility is critical, since the x-ray beam and the biological sample are both limited resources and quite valuable. New GPU-accelerated software (nanoBragg) has been developed to simulate diffraction patterns based on a physical model; while the remaining challenge will be to solve the inverse problem of adjusting the physical model to closely predict the observed data. 

The ExaFEL stretch goal is the creation of an automated analysis pipeline for imaging of single particles via diffractive imaging. This entails the reconstruction of a 3D molecular structure from 2D diffraction images using the new Multi- Tiered Iterative Phasing (MTIP) algorithm. In SPI, diffraction images collected from individual particles are used to determine molecular (or atomic) structure, even from multiple conformational states (or nonidentical particles) under operating conditions. Determining structures from SPI experiments is challenging because orientations and states of imaged particles are unknown and images are highly contaminated with noise. Furthermore, the number of useful images is often limited by achievable single-particle hit rates, currently between 1 and 10% of the machine rate. The MTIP algorithm introduces an iterative projection framework to simultaneously determine orientations, states, and molecular structure from limited single-particle data by leveraging structural constraints throughout the reconstruction, offering a potential pathway to increasing the amount of information that can be extracted from single-particle diffraction. 

Rapid feedback is crucial for tuning sample concentrations to achieve a sufficient single- particle hit rate, ensuring that adequate data are collected and available to steer the experiment. The availability of exascale computing resources and an HPC workflow that can handle incremental bursts of data in the analyses will allow for data analysis on the fly, providing immediate feedback on the quality of the experimental data while determining the 3D structure of the sample at the same time. 

To show the scalability of the analysis pipeline, the ExaFEL team is progressively increasing the fraction of the machine used for reconstruction while keeping constant the number of diffraction images distributed across multiple nodes. The goal is to distribute the images over an increasing number of nodes while reducing the overall reconstruction time up to the point where the analysis can keep up with data collection rates (5 kHz). 

We have recently developed GPU kernels to solve the nanoBragg inverse problem and solve for physics parameters.  This capability has been demonstrated on Summit using O(105) simulated diffraction patterns of randomly-oriented microcrystals generated in an XFEL beam. This represents a critical step towards the ability of using x-ray tracing in nanocrystallography reconstruction.   

We have further developed a new Cartesian/Nonuniform FFT formulation of M-TIP which allows SPI reconstruction to scale to a large number of nodes, retiring the risk of being able to analyze the massive volumes of data achievable by the LCLS upgrades. This ability of scaling SPI analysis, represents a revolution for free electron lasers as it provides the opportunity of exploiting the advantages that SPI offers over other techniques like CryoEM (e.g. ability of making measurements under physiological conditions, access to time domain) or crystallography (e.g. no need to crystallize molecules). We investigated acceleration for the new, scalable, M-TIP workflow along three critical calculations: orientation matching, and [inverse] non-uniform FFTs. Finally, we leveraged Summit GPUs to generate O(106) diffraction patterns of multiple conformations of a protein sample accounting for beam fluctuations, parasitic beamline scattering, and detector noise. These simulated images will be critical to characterize the performance of the new cartesian-basis M-TIP algorithm under realistic conditions. 

Finally, we designed and developed capabilities for streaming the science data from SLAC to the computing facility, starting the analysis job on the supercomputer and reporting the results of the analysis back to the experimenters in quasi real time, and integrated them with the LCLS data management system. This work was performed on Cori in collaboration with the NERSC and ESnet teams.