Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,027 records · Page 57

Ride-hailing and taxi versus walking: Long term forecasts and implications from large-scale behavioral data

Introduction: Although ride-hailing and taxi trips can potentially reduce single-occupant vehicle trips and auto ownership, they can also replace pedestrian trips. Because physical activity is associated with improved health outcomes, the extent to which ride-hailing and taxi travel captures walking's mode share is of interest to policymakers. Methods: Based on large-scale behavioral data from the 2017 U.S. National Household Travel Survey, this paper reports on the development of a full Bayesian logistic regression model for determining the mode split between (1) ride-hailing and taxi and (2) walk while accounting for unobserved heterogeneity. The results from the stand-alone model inform two longer-term travel forecasting scenarios: a) higher risk of walk trips converting to ride-hailing and taxi, specifically in the future with high prevalence of automated vehicles, b) higher probability of such trips remaining as walking. Results: The results revealed that some of the important characteristics that increase the likelihood of a traveler using the ride-hailing and taxi mode versus walking include having a longer trip, using a smartphone to access the internet, having an interest in technologies, having a medical condition, and living in a metropolitan area with rail access. Further, the results from the first scenario suggest that an overall increase of up to 2.9% in the ride-hailing and taxi mode share may be expected. The second scenario shows that between 68% and 76% of ride-hailing and taxi trips could be diverted to walking if supportive pedestrian infrastructure were provided in the case study locations. The planning process can be adapted to consider not only congestion, crash, and emissions impacts of such shifts but also the effects of a loss of physical activity. Conclusions: The study findings show how the ride-hailing and taxi mode competes with walking. Further, the findings enable planners to update their regional travel forecasting models; policy makers can thus encourage active travel by prioritizing pedestrian infrastructure investments that may divert ride-hailing and taxi trips to walking. However, equity should be a key consideration to ensure that addressing the competition between these two modal choices does not hinder the provision of pedestrian facilities in communities that depend on walking.

99 GENERAL AND MISCELLANEOUS↗

Automated co-adding and energy calibration of large array microcalorimeter data with zero sample knowledge

State-of-the-art microcalorimeter spectrometers now contain large detector arrays with hundreds of individual pixels. Each individual pixel outputs a unique and non-linear response with respect to deposited energy. This work describes a pattern-recognition algorithm to combine these responses into a single energy-calibrated histogram, referred to as co-adding pixels. Photo-peaks from different pixels are matched together based upon how well the match aligns the centroids and heights of neighboring peaks. This usually results in around 100 co-adding calibration points from 30 to 300 keV for a several day acquisition of plutonium items with masses between 0.5 and 10 grams. An additional algorithm energy-calibrates this co-added spectrum using the fluoresced K x-ray emissions from a tantalum absorber and inherent x-ray escape peaks from the tin absorbers. Both algorithms operate without knowledge of the source and are fully automated. This work presents results from the acquisitions of high and low burnup plutonium, 10% enriched uranium, a 153 Gd calibration source, and a 57 Co+ 166m Ho calibration source. In all measurements, resolution defined as the full-width at half-maximum (FWHM) of photo-peaks is preserved between the individual pixel and co-added spectra at around 65 eV for incident photon energies between 60 and 208 keV. The energy calibration algorithm is approximate and yields a calibration curve off by an average of around 200 eV for incident photon energies between 60 and 208 keV.

47 OTHER INSTRUMENTATION↗

Three-dimensional displays for natural hazards analysis, using classified Landsat Thematic Mapper digital data and large-scale digital elevation models

Methods are described for using Landsat Thematic Mapper digital data and digital elevation models for the display of natural hazard sites in a mountainous region of northwestern Montana, USA. Hazard zones can be easily identified on the three-dimensional images. Proximity of facilities such as highways and building locations to hazard sites can also be easily displayed. A temporal sequence of Landsat TM (or similar) satellite data sets could also be used to display landscape changes associated with dynamic natural hazard processes.

Butler, David R.↗

Building a Large Scale Climate Data System in Support of HPC Environment

The Earth System Grid Federation (ESG) is a large scale, multi-institutional, interdisciplinary project that aims to provide climate scientists and impact policy makers worldwide a web-based and client-based platform to publish, disseminate, compare and analyze ever increasing climate related data. This paper describes our practical experiences on the design, development and operation of such a system. In particular, we focus on the support of the data lifecycle from a high performance computing (HPC) perspective that is critical to the end-to-end scientific discovery process. We discuss three subjects that interconnect the consumer and producer of scientific datasets: (1) the motivations, complexities and solutions of deep storage access and sharing in a tightly controlled environment; (2) the importance of scalable and flexible data publication/population; and (3) high performance indexing and search of data with geospatial properties. These perceived corner issues collectively contributed to the overall user experience and proved to be as important as any other architectural design considerations. Although the requirements and challenges are rooted and discussed from a climate science domain context, we believe the architectural problems, ideas and solutions discussed in this paper are generally useful and applicable in a larger scope.

54 ENVIRONMENTAL SCIENCES↗

Quality-Aware Data Management for Large Scale Scientific Applications

Increasingly larger scale simulations are generating an unprecedented amount of output data, causing researchers to explore new `data staging' methods that buffer, use, and/or reduce such data online rather than simply pushing it to disk. Leveraging the capabilities of data staging, this study explores the potential for data reduction via online data compression, first using general compression techniques and then proposing use-specific methods that permit users to define simple data queries that cause only the data identified by those queries to be emitted. Using online methods for code generation and deployment, with such dynamic data queries, end users can precisely identify the quality of information (QoI) of their output data, by explicitly determining what data may be lost vs. retained, in contrast to general-purpose lossy compression methods that do not provide such levels of control. The paper also describes the key elements of a quality-aware data management system (QADMS) for high-end machines enabled by this approach. Initial experimental results demonstrate that QADMS can effectively reduce data movement cost and improve the QoS while meeting the QoI constraint stated by users.

Computer Science↗

Quality-Aware Data Management for Large Scale Scientific Applications

Increasingly larger scale simulations are generating an unprecedented amount of output data, causing researchers to explore new data staging methods that buffer, use, and/or reduce such data online rather than simply pushing it to disk. Leveraging the capabilities of data staging, this study explores the potential for data reduction via online data compression, first using general compression techniques and then proposing usespecific methods that permit users to define simple data queries that cause only the data identified by those queries to be emitted. Using online methods for code generation and deployment, with such dynamic data queries, end users can precisely identify the quality of information (QoI) of their output data, by explicitly determining what data may be lost vs. retained, in contrast to general-purpose lossy compression methods that do not provide such levels of control. The paper also describes the key elements of a quality-aware data management system (QADMS) for highend machines enabled by this approach. Initial experimental results demonstrate that QADMS can effectively reduce data movement cost and improve the QoS while meeting the QoI constraint stated by users.

Zou, Hongbo↗

Causal discovery from data assisted by large language models

Knowledge-driven discovery of novel materials necessitates the development of causal models for property emergence. While in the classical physical paradigm, the causal relationships are deduced based on physical principles or via experiment, the rapid accumulation of observational data necessitates learning causal relationships between dissimilar aspects of material structure and functionalities based on observations. For this, it is essential to integrate experimental data with prior domain knowledge. Here, we demonstrate this approach by combining high-resolution scanning transmission electron microscopy data with insights derived from large language models (LLMs). By applying ChatGPT to domain-specific literature, such as arXiv papers on ferroelectrics, and combining the obtained information with data-driven causal discovery, we construct adjacency matrices for directed acyclic graphs that map the causal relationships between structural, chemical, and polarization degrees of freedom in Sm-doped BiFeO 3 . This approach enables us to hypothesize how synthesis conditions influence material properties and guides experimental validation. Furthermore, the ultimate objective of this work is to develop a unified framework that integrates LLM-driven literature analysis with data-driven discovery, facilitating the precise engineering of ferroelectric materials by establishing clear connections between synthesis conditions and their resulting material properties.

Causal inference↗

Deep learning for in situ data compression of large turbulent flow simulations

As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.

97 MATHEMATICS AND COMPUTING↗

The Fourth Catalog of Active Galactic Nuclei Detected by the Fermi Large Area Telescope: Data Release 3

Abstract An incremental version of the fourth catalog of active galactic nuclei (AGNs) detected by the Fermi Large Area Telescope is presented. This version (4LAC-DR3) derives from the third data release of the 4FGL catalog based on 12 yr of E > 50 MeV gamma-ray data, where the spectral parameters, spectral energy distributions (SEDs), yearly light curves, and associations have been updated for all sources. The new reported AGNs include 587 blazar candidates and four radio galaxies. We describe the properties of the new sample and outline changes affecting the previously published one. We also introduce two new parameters in this release, namely the peak energy of the SED high-energy component and the corresponding flux. These parameters allow an assessment of the Compton dominance, the ratio of the inverse-Compton to the synchrotron-peak luminosities, without relying on X-ray data.

79 ASTRONOMY AND ASTROPHYSICS↗

Data optimization for large batch distributed training of deep neural networks

Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.

Gahlot, Shubhankar↗

VisIRR: A Visual Analytics System for Information Retrieval and Recommendation for Large-Scale Document Data

In this paper, we present an interactive visual information retrieval and recommendation system, called VisIRR, for large-scale document discovery. VisIRR effectively combines the paradigms of (1) a passive pull through query processes for retrieval and (2) an active push that recommends items of potential interest to users based on their preferences. Equipped with an efficient dynamic query interface against a large-scale corpus, VisIRR organizes the retrieved documents into high-level topics and visualizes them in a 2D space, representing the relationships among the topics along with their keyword summary. In addition, based on interactive personalized preference feedback with regard to documents, VisIRR provides document recommendations from the entire corpus, which are beyond the retrieved sets. Such recommended documents are visualized in the same space as the retrieved documents, so that users can seamlessly analyze both existing and newly recommended ones. This article presents novel computational methods, which make these integrated representations and fast interactions possible for a large-scale document corpus. We illustrate how the system works by providing detailed usage scenarios. Finally, we present preliminary user study results for evaluating the effectiveness of the system.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Integrating large-scale functional genomics data to dissect metabolic networks for hydrogen production

The goal of this project is to identify gene networks that are critical for efficient biohydrogen production by leveraging variation in gene content and gene expression in independently isolated Rhodopseudomonas palustris strains. Coexpression methods were applied to large data sets that we have collected to define probabilistic causal gene networks. To our knowledge this a first systems level approach that takes advantage of strain-to strain variability to computationally define networks critical for a particular bacterial phenotypic trait.

08 HYDROGEN↗

Mapping Vegetation at Species Level with High-Resolution Multispectral and Lidar Data Over a Large Spatial Area: A Case Study with Kudzu

Mapping vegetation species is critical to facilitate related quantitative assessment, and mapping invasive plants is important to enhance monitoring and management activities. Integrating high-resolution multispectral remote-sensing (RS) images and lidar (light detection and ranging) point clouds can provide robust features for vegetation mapping. However, using multiple sources of high-resolution RS data for vegetation mapping on a large spatial scale can be both computationally and sampling intensive. Here, we designed a two-step classification workflow to potentially decrease computational cost and sampling effort and to increase classification accuracy by integrating multispectral and lidar data in order to derive spectral, textural, and structural features for mapping target vegetation species. We used this workflow to classify kudzu, an aggressive invasive vine, in the entire Knox County (1362 km2) of Tennessee (U.S.). Object-based image analysis was conducted in the workflow. The first-step classification used 320 kudzu samples and extensive, coarsely labeled samples (based on national land cover) to generate an overprediction map of kudzu using random forest (RF). For the second step, 350 samples were randomly extracted from the overpredicted kudzu and labeled manually for the final prediction using RF and support vector machine (SVM). Computationally intensive features were only used for the second-step classification. SVM had constantly better accuracy than RF, and the producer’s accuracy, user’s accuracy, and Kappa for the SVM model on kudzu were 0.94, 0.96, and 0.90, respectively. SVM predicted 1010 kudzu patches covering 1.29 km2 in Knox County. We found the sample size of kudzu used for algorithm training impacted the accuracy and number of kudzu predicted. The proposed workflow could also improve sampling efficiency and specificity. Our workflow had much higher accuracy than the traditional method conducted in this research, and could be easily implemented to map kudzu in other regions as well as map other vegetation species.

59 BASIC BIOLOGICAL SCIENCES↗

Big Data Challenges for Large Radio Arrays

Future large radio astronomy arrays, particularly the Square Kilometre Array (SKA), will be able to generate data at rates far higher than can be analyzed or stored affordably with current practices. This is, by definition, a "big data" problem, and requires an end-to-end solution if future radio arrays are to reach their full scientific potential. Similar data processing, transport, storage, and management challenges face next-generation facilities in many other fields.

Combining↗

The Role of the U.S. Electric Distribution System in Serving Data Center and Other Large Loads

The rapid expansion of data centers in the United States is reshaping how the electric distribution system must plan for and accommodate large load interconnections. This report evaluates the role of the distribution grid in serving these loads, from small edge facilities to hyperscale campuses. Using national datasets, utility filings, and industry studies, we assess demand growth, reliability requirements, interconnection thresholds, and infrastructure needs at substations and feeders. The analysis highlights the mismatch between fast data center development timelines and slower utility planning and construction cycles, as well as strategies such as phased energization, on-site generation, hosting capacity maps, and structured interconnection frameworks. While focused on data centers, the insights also apply to other high-density loads such as advanced manufacturing, hydrogen production, and electrified transportation. The report concludes with approaches to align planning processes, transparency tools, and regulatory frameworks so utilities can manage new large loads in ways that support a reliable and resilient grid.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A cross-dimensional analysis of data-driven short-term load forecasting methods with large-scale smart meter data

Electricity load forecasting is essential to utility operation and power grid stability. A wide spectrum of data-driven methods, ranging from linear regression models to more recent deep learning models have been adopted to forecast electric load over the years. However, there still lacks a holistic evaluation of the applicability of conventional statistical and machine learning based algorithms with respect to different temporal and spatial scopes, computational requirements, and sensitivity of model-tuning. Enabled by a large-scale electricity load profile dataset of over 40,000 residential customers in a utility region, we conducted a cross-dimensional analysis of data-driven load forecasting methods. Three regression-based and seven deep learning algorithms with different model configurations were evaluated in terms of their overall and peak load prediction accuracy, and training burdens, across spatial aggregation levels ranging from the transformer, feeder, substation, to neighborhood. We found, first, the load forecasting accuracy is constrained by a predictability boundary, influenced by the forecasting horizon and spatial aggregation level. Specifically, RandomForest, XGBoost, TFT, TSMixer, and TiDE models achieved less than 10 % prediction error for up to 96-h ahead forecasting for district, substation, and feeder levels, while other models struggle at long-horizon predictions; Second, for winter and summer peak load dates, most models were able to predict the peak demand timing within ± 1 h, but the prediction percentage error varied by models, with TFT and TiDE models being the top performers; Third, models with similar prediction accuracy can differ in training burden by an order of magnitude. Therefore, choosing model configurations that balance prediction performance and computational resource is an important practical consideration for large-scale deployment of the machine learning based load forecasting. The outcome of this study can guide researchers and practitioners to choose the proper load forecasting algorithms based on their problem scope, required accuracy, and available resources. The predictability boundary can serve as a benchmark for electricity load forecasting problems with new algorithms and datasets.

Li, Han↗

Experimental Data Curation at Large Instrument Facilities with Open Source Software

The National Synchrotron Light Source II operating at Brookhaven National Laboratory since 2014 for the US Department of Energy is one of the newest and brightest storage-ring synchrotron facility in the world. NSLS-II, like other facilities, provides pre-processing of the raw data and some analysis capabilities to its users. We describe the research collaborations and open source infrastructure developed at large instrument facilities such as NSLS-II for the purpose of curating high value scientific data along the early stages of the data lifecycle. Data acquisition and curation tasks include storing experiment configuration, detector metadata, raw data acquisition with infrastructure that converts proprietary instrument formats to industry standards. In addition, we describe a specific effort for discovering sample information at NSLS-II and tracing the provenance of analysis performed on acquired images. We show that curation tasks must be embedded into software along the data life cycle for effectiveness and ease of use, and that loosely defined collaborations evolve around shared open source tools. Finally we discuss best practices for experimental metadata capture in such facilities, data access and the new challenges of scale and complexity posed by AI-based discovery for the synthesis of new materials.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Velocity Structure at the Source Physics Experiment Phase I Site Obtained with the Large-N Array Data

The Source Physics Experiment (SPE) consists of a series of chemical explosions at the Nevada National Security Site. The goal of the SPE is to understand and model seismic-wave generation and propagation from these explosions. To achieve this goal, we need an accurate velocity model of the SPE site. A large-N seismic array deployed at the SPE site during one of the chemical explosions (SPE-5) provides great data for this purpose. The array consists of 996 geophones and covers an area of approximately 2 × 2.5 km. In addition to the SPE-5 explosion, the array recorded 53 large weight drops. Using the large-N seismic array recordings, we perform first-arrival analysis and obtain a 2D P-wave velocity model of the SPE site. We image a sharp transition from high-velocity Cretaceous granite to low-velocity Quaternary alluvium. Other geological units such as the Tertiary volcanic rocks and Paleozoic sedimentary rocks are also clearly shown. The results of this work provide important local geological information and will be incorporated into the larger 3D modeling effort of the SPE program to validate the predictive models developed for the site.

58 GEOSCIENCES↗