Search NASASearch

Engineering topics

Clune, Thomas

Publications and source records attributed to Clune, Thomas.

At least 19 records

The OpenSSP Snow Particle and Scattering Property Database: Current Status and Future Plans

Understanding the connection between microphysical properties of falling snow and remote sensing observations requires detailed models of snow particles and the related electromagnetic scattering properties. To this end, we have created a database of synthetically-grown snow particles along with electromagnetic scattering properties calculated using advanced computational methods. We showcase the breadth of the catalog, current work, applications of the catalog, and future plans.

Adams, Ian S.

Active and Passive Radiative Transfer Simulations for GPM-Related Field Campaigns

Using a three-dimensional radiative transfer model combined with cloud-resolving model output, we simulate active and passive sensor observations of clouds and precipitaiton. This combination of tools allows us to diagnose the contributions of various hydrometeor types. Radar multiple scattering is most closely associated with the presence of graupel. At Wband, massive amounts multiple scattering in deep convection can decorrelate the reflectivity profile from the vertical structure, but for less intense events, multiple scattering could be a useful indicator of riming. For passive sensors, polarization differences at 166 GHz indicate the presence of horizontally aligned frozen particles with pronounced aspect ratios, while high concentrations of more isotropic aggregates and graupel dampen the polarization difference while also contributing to the lowest brightness temperature depressions. The insights into remote sensing measurements will facilitate the development of improved algorithms and advanced sensors.

Adams, Ian S.

Active and Passive Radiative Transfer Simulations for GPM-Related Field Campaigns

Radiative transfer modeling is an important tool for interpreting remote sensing observations. It allows us to determine how sensor characteristics will impact observations, and it gives us a framework for us to test assumptions about the phenomena we are attempting to observe. In this work, we use cloud simulations for precipitation events observed during various GPM-related field campaigns. The simulations show how various properties of clouds and precipitation affect the measurements.

Adams, Ian S.

Three-Dimensional Sensor Forward Modeling of Clouds and Precipitation in the Multi-Instrument Inverse Solver Testbed (MIIST)

Sensor forward models are an essential tool for interpreting remote sensing observations and performing quantitative estimates of geophysical parameters. Our three-dimensional forward modeling and retrieval framework allows us to perform detailed analyses of NASA field campaign datasets for a deeper understanding of the remote sensing of clouds and precipitation. This presentation details the componenets of this radiative transfer model used to simulate active (radar) and passive (microwave radiometer) observations, and we give some relevant examples based on both model precipitation systems and actual observations.

Adams, Ian S.

Invariant Imbedded T-Matrix Method for Axial Symmetric Hydrometeors with Extreme Aspect Ratios

The single-scattering properties (SSPs) of hydrometeors are the fundamental quantities for physics-based precipitation retrievals. Thus, efficient computation of their electromagnetic scattering is of great value. Whereas the semi-analytical T-Matrix methods are likely the most efficient for nonspherical hydrometeors with axial symmetry, they are not suitable for arbitrarily shaped hydrometeors absent of any significant symmetry, for which volume integral methods such as those based on Discrete Dipole Approximation (DDA) are required. Currently the two leading T-matrix methods are the Extended Boundary Condition Method (EBCM) and the Invariant Imbedding T-matrix Method incorporating Lorentz-Mie Separation of Variables (IITM+SOV). EBCM is known to outperform IITM+SOV for hydrometeors with modest aspect ratios. However, in cases when aspect ratios become extreme, such as needle-like particles with large height to diameter values, EBCM fails to converge. Such hydrometeors with extreme aspect ratios are known to be present in solid precipitation and their SSPs are required to model the radiative responses accurately. In these cases, IITM+SOV is shown to converge. An efficient, parallelized C++ implementation for both EBCM and IITM+SOV has been developed to conduct a performance comparison between EBCM, IITM+SOV, and DDSCAT (a popular implementation of DDA). We present the comparison results and discuss details. Our intent is to release the combined ECBM IITM+SOV software to the community under an open source license.

Pelissier, Craig

Implementing Connected Component Labeling as a User Defined Operator for SciDB

We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDB's memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are "flattened" coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of O(1011) elements, with O(108) foreground cells and O(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.

UDO

Addressing the Big-Earth-Data Variety Challenge with the Hierarchical Triangular Mesh

We have implemented an updated Hierarchical Triangular Mesh (HTM) as the basis for a unified data model and an indexing scheme for geoscience data to address the variety challenge of Big Earth Data. We observe that, in the absence of variety, the volume challenge of Big Data is relatively easily addressable with parallel processing. The more important challenge in achieving optimal value with a Big Data solution for Earth Science (ES) data analysis, however, is being able to achieve good scalability with variety. With HTM unifying at least the three popular data models, i.e. Grid, Swath, and Point, used by current ES data products, data preparation time for integrative analysis of diverse datasets can be drastically reduced and better variety scaling can be achieved. In addition, since HTM is also an indexing scheme, when it is used to index all ES datasets, data placement alignment (or co-location) on the shared nothing architecture, which most Big Data systems are based on, is guaranteed and better performance is ensured. Moreover, our updated HTM encoding turns most geospatial set operations into integer interval operations, gaining further performance advantages.

SciDB

Evaluating the Impact of Data Placement to Spark and SciDB with an Earth Science Use Case

We investigate the impact of data placement for two Big Data technologies, Spark and SciDB, with a use case from Earth Science where data arrays are multidimensional. Simultaneously, this investigation provides an opportunity to evaluate the performance of the technologies involved. Two datastores, HDFS and Cassandra, are used with Spark for our comparison. It is found that Spark with Cassandra performs better than with HDFS, but SciDB performs better yet than Spark with either datastore. The investigation also underscores the value of having data aligned for the most common analysis scenarios in advance on a shared nothing architecture. Otherwise, repartitioning needs to be carried out on the fly, degrading overall performance.

Spark

ldentifying Episodes of Earth Science Phenomena Using a Big-Data Technology

A significant portion of Earth Science investigations is phenomenon- (or event-) based, such as the studies of Rossby waves, volcano eruptions, tsunamis, mesoscale convective systems, and tropical cyclones. However, except for a few high-impact phenomena, e.g. tropical cyclones, comprehensive records are absent for the occurrences or events of these phenomena. Phenomenon-based studies therefore often focus on a few prominent cases while the lesser ones are overlooked. Without an automated means to gather the events, comprehensive investigation of a phenomenon is at least time-consuming if not impossible. We have constructed a prototype Automated Event Service (AES) system that is used to methodically mine custom-defined events in the reanalysis data sets of atmospheric general circulation models. Our AES will enable researchers to specify their custom, numeric event criteria using a user-friendly web interface to search the reanalysis data sets. Moreover, we have included a social component to enable dynamic formation of collaboration groups for researchers to cooperate on event definitions of common interest and for the analysis of these events. An Earth Science event (ES event) is defined here as an episode of an Earth Science phenomenon (ES phenomenon). A cumulus cloud, a thunderstorm shower, a rogue wave, a tornado, an earthquake, a tsunami, a hurricane, or an El Nino, is each an episode of a named ES phenomenon, and, from the small and insignificant to the large and potent, all are examples of ES events. An ES event has a duration (often finite) and an associated geo-location as a function of time; it's therefore an entity embedded in four-dimensional (4D) spatiotemporal space. Earth Science phenomena with the potential to cause massive economic disruption or loss of life often rivet the attention of researchers. But, broader scientific curiosity also drives the study of phenomena that pose no immediate danger, such as land/sea breezes. Due to Earth System's intricate dynamics, we are continuously discovering novel ES phenomena. We generally gain understanding of a given phenomenon by observing and studying individual events. This process usually begins by identifying the occurrences of these events. Once representative events are identified or found, we must locate associated observed or simulated data prior to commencing analysis and concerted studies of the phenomenon. Knowledge concerning the phenomenon can accumulate only after analysis has started. However, as mentioned previously, comprehensive records only exist for a very limited set of high-impact phenomena; aside from these, finding events and locating associated data currently may take a prohibitive amount of time and effort on the part of an individual investigator. The reason for the lack of comprehensive records for most of the ES phenomena is mainly due to the perception that they do not pose immediate and/or severe threat to life and property. Thus they are not consistently tracked, monitored, and catalogued. Many phenomena even lack precise and/or commonly accepted criteria for definitions. Moreover, various Earth Science observations and data have accumulated to a previously unfathomable volume; NASA Earth Observing System Data Information System (EOSDIS) alone archives several petabytes (PB) of satellite remote sensing data, which are steadily increasing. All of these factors contribute to the difficulty of methodically identifying events corresponding to a given phenomenon and significantly impede systematic investigations. We have not only envisioned AES as an environment for identifying customdefined events but also aspired for it to be an interactive environment with quick turnaround time for revisions of query criteria and results, as well as a collaborative environment where geographically distributed experts may work together on the same phenomena. A Big Data technology is thus required for the realization of such a system. In the following, we first introduce the technology selected for AES in the next section. We then demonstrate the utility of AES using a use case, Blizzard, before we conclude.

Kuo, Kwo-Sen

Anticipated Changes in Conducting Scientific Data-Analysis Research in the Big-Data Era

A Big-Data environment is one that is capable of orchestrating quick-turnaround analyses involving large volumes of data for numerous simultaneous users. Based on our experiences with a prototype Big-Data analysis environment, we anticipate some important changes in research behaviors and processes while conducting scientific data-analysis research in the near future as such Big-Data environments become the mainstream. The first anticipated change will be the reduced effort and difficulty in most parts of the data management process. A Big-Data analysis environment is likely to house most of the data required for a particular research discipline along with appropriate analysis capabilities. This will reduce the need for researchers to download local copies of data. In turn, this also reduces the need for compute and storage procurement by individual researchers or groups, as well as associated maintenance and management afterwards. It is almost certain that Big-Data environments will require a different "programming language" to fully exploit the latent potential. In addition, the process of extending the environment to provide new analysis capabilities will likely be more involved than, say, compiling a piece of new or revised code.We thus anticipate that researchers will require support from dedicated organizations associated with the environment that are composed of professional software engineers and data scientists. A major benefit will likely be that such extensions are of higherquality and broader applicability than ad hoc changes by physical scientists. Another anticipated significant change is improved collaboration among the researchers using the same environment. Since the environment is homogeneous within itself, many barriers to collaboration are minimized or eliminated. For example, data and analysis algorithms can be seamlessly shared, reused and re-purposed. In conclusion, we will be able to achieve a new level of scientific productivity in the Big-Data analysis environments.

Kuo, Kwo-Sen

Tool for Automated Retrieval of Generic Event Tracks (TARGET)

Methods have been developed to identify and track tornado-producing mesoscale convective systems (MCSs) automatically over the continental United States, in order to facilitate systematic studies of these powerful and often destructive events. Several data sources were combined to ensure event identification accuracy. Records of watches and warnings issued by National Weather Service (NWS), and tornado locations and tracks from the Tornado History Project (THP) were used to locate MCSs in high-resolution precipitation observations and GOES infrared (11-micron) Rapid Scan Operation (RSO) imagery. Thresholds are then applied to the latter two data sets to define MCS events and track their developments. MCSs produce a broad range of severe convective weather events that are significantly affecting the living conditions of the populations exposed to them. Understanding how MCSs grow and develop could help scientists improve their weather prediction models, and also provide tools to decision-makers whose goals are to protect populations and their property. Associating storm cells across frames of remotely sensed images poses a difficult problem because storms evolve, split, and merge. Any storm-tracking method should include the following processes: storm identification, storm tracking, and quantification of storm intensity and activity. The spatiotemporal coordinates of the tracks will enable researchers to obtain other coincident observations to conduct more thorough studies of these events. In addition to their tracked locations, their areal extents, precipitation intensities, and accumulations all as functions of their evolutions in time were also obtained and recorded for these events. All parameters so derived can be catalogued into a moving object database (MODB) for custom queries. The purpose of this software is to provide a generalized, cross-platform, pluggable tool for identifying events within a set of scientific data based upon specified criteria with the possibility of storing identified events into a searchable database. The core of the application uses an implementation of the connected component labeling (CCL) algorithm to identify areas of interest, then uses a set of criteria to establish spatial and temporal relationships between identified components. The CCL algorithm is used for identifying objects within images for computer vision. This application applies it to scientific data sets using arbitrary criteria. The most novel concept was applying a generalized CCL implementation to scientific data sets for establishing events both spatially and temporally. The combination of several existing concepts (pluggable components, generalized CCL algorithm, etc.) into one application is also novel. In addition, how the system is designed, i.e., its extensibility with pluggable components, and its configurability with a simple configuration file, is innovative. This allows the system to be applied to new scenarios with ease.

Clune, Thomas

Challenges at Petascale for Pseudo-Spectral Methods on Spheres (A Last Hurrah?)

Conclusions: a) Proper software abstractions should enable rapid-exploration of platform-specific optimizations/ tradeoffs. b) Pseudo-spectra! methods are marginally viable for at least some classes of petascaie problems. i.e., GPU based machine with good bisection would be best. c) Scalability at exascale is possible, but the necessary resolution will make algorithm prohibitively expensive. Efficient implementations of realistic global transposes are mtricate and tedious in MPI. PS at petascaie requires exploration of a variety of strategies for spreading local and remote communic3tions. PGAS allows far simpler implementation and thus rapid exploration of variants.

Clune, Thomas

NASA's Integrated Instrument Simulator Suite for Atmospheric Remote Sensing from Spaceborne Platforms (ISSARS) and Its Role for the ACE and GPM Missions

Forward simulation is an indispensable tool for evaluation of precipitation retrieval algorithms as well as for studying snow/ice microphysics and their radiative properties. The main challenge of the implementation arises due to the size of the problem domain. To overcome this hurdle, assumptions need to be made to simplify compiles cloud microphysics. It is important that these assumptions are applied consistently throughout the simulation process. ISSARS addresses this issue by providing a computationally efficient and modular framework that can integrate currently existing models and is also capable of expanding for future development. ISSARS is designed to accommodate the simulation needs of the Aerosol/Clouds/Ecosystems (ACE) mission and the Global Precipitation Measurement (GPM) mission: radars, microwave radiometers, and optical instruments such as lidars and polarimeter. ISSARS's computation is performed in three stages: input reconditioning (IRM), electromagnetic properties (scattering/emission/absorption) calculation (SEAM), and instrument simulation (ISM). The computation is implemented as a web service while its configuration can be accessed through a web-based interface.

real-aperture instruments

Constraints and Opportunities in GCM Model Development

Over the past 30 years climate models have evolved from relatively simple representations of a few atmospheric processes to complex multi-disciplinary system models which incorporate physics from bottom of the ocean to the mesopause and are used for seasonal to multi-million year timescales. Computer infrastructure over that period has gone from punchcard mainframes to modern parallel clusters. Constraints of working within an ever evolving research code mean that most software changes must be incremental so as not to disrupt scientific throughput. Unfortunately, programming methodologies have generally not kept pace with these challenges, and existing implementations now present a heavy and growing burden on further model development as well as limiting flexibility and reliability. Opportunely, advances in software engineering from other disciplines (e.g. the commercial software industry) as well as new generations of powerful development tools can be incorporated by the model developers to incrementally and systematically improve underlying implementations and reverse the long term trend of increasing development overhead. However, these methodologies cannot be applied blindly, but rather must be carefully tailored to the unique characteristics of scientific software development. We will discuss the need for close integration of software engineers and climate scientists to find the optimal processes for climate modeling.

Schmidt, Gavin

Application-Controlled Parallel Asynchronous Input/Output Utility

A software utility tool has been designed to alleviate file system I/O performance bottlenecks to which many high-end computing (HEC) applications fall prey because of the relatively large volume of data generated for a given amount of computational work. In an effort to reduce computing resource waste, and to improve sustained performance of these HEC applications, a lightweight software utility has been designed to circumvent bandwidth limitations of typical HEC file systems by exploiting the faster inter-processor bandwidth to move output data from compute nodes to designated I/O nodes as quickly as possible, thereby minimizing the I/O wait time. This utility has successfully demonstrated a significant performance improvement within a major NASA weather application.

Clune, Thomas

Refactoring and Componentizing Legacy Codes: GMI Case Study

As Earth science applications grow increasingly complex over time, maintenance of such software becomes a significant challenge. In general, the software subsystems were designed, implemented and integrated (over an extended period of time) by different groups that did not follow the same software design standard. The objective was understandably to obtain scientific results quickly, but this goal was often achieved at the expense of good software practices. With that pattern of development, the cost of incremental changes to the applications grow ominously large. This increase in effort, tends to squeeze development resources and largely prevent any systematic attempt to reintroduce better software practices and thereby enhance the maintainability of the software. In this paper, we present the process we used to incrementally refactor and componentize the Global Modeling Initiative code to make it more understandable, maintainable, flexible and extensible. The resulting product not only preserves the scientific integrity of the original code but also extends the capabilities of the code for several applications.

Kouatchou, Jules

Advances in Modeling the Generation of the Geomagnetic Field by the Using of Massively Parallel Computers and Profound Optimization

At the Earth's surface, the magnetic field that is observed is similar to that that would be generated by a simple bar magnet running through the Earth's axis. This idea (permanent magnetism) was commonly believed a century ago. Because the temperature of the core is so high, permanent magnetism is not possible. Therefore, the magnetic field should decay, over tens of thousands of years. Since it does not, the field must be regenerating. Since the turn of the century, the idea that the core is molten iron which by moving generates a magnetic field arose. The set of equations to describe this are extremely non-linear and complex. Only in the last five to ten years have computers been able to solve these equations.

Clune, Thomas