Search NASA⌕ Search

SEARCH · Search NASA

Results for “Heuristic algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Kalman Filtering with Inequality Constraints for Turbofan Engine Health Estimation

Kalman filters are often used to estimate the state variables of a dynamic system. However, in the application of Kalman filters some known signal information is often either ignored or dealt with heuristically. For instance, state variable constraints (which may be based on physical considerations) are often neglected because they do not fit easily into the structure of the Kalman filter. This paper develops two analytic methods of incorporating state variable inequality constraints in the Kalman filter. The first method is a general technique of using hard constraints to enforce inequalities on the state variable estimates. The resultant filter is a combination of a standard Kalman filter and a quadratic programming problem. The second method uses soft constraints to estimate state variables that are known to vary slowly with time. (Soft constraints are constraints that are required to be approximately satisfied rather than exactly satisfied.) The incorporation of state variable constraints increases the computational effort of the filter but significantly improves its estimation accuracy. The improvement is proven theoretically and shown via simulation results. The use of the algorithm is demonstrated on a linearized simulation of a turbofan engine to estimate health parameters. The turbofan engine model contains 16 state variables, 12 measurements, and 8 component health parameters. It is shown that the new algorithms provide improved performance in this example over unconstrained Kalman filtering.

Simon, Dan↗

Numerical arc segmentation algorithm for a radio conference: A software tool for communication satellite systems planning

The Numerical Arc Segmentation Algorithm for a Radio Conference (NASARC) provides a method of generating predetermined arc segments for use in the development of an allotment planning procedure to be carried out at the 1988 World Administrative Radio Conference (WARC) on the Use of the Geostationary Satellite Orbit and the Planning of Space Services Utilizing It. Through careful selection of the predetermined arc (PDA) for each administration, flexibility can be increased in terms of choice of system technical characteristics and specific orbit location while reducing the need for coordination among administrations. The NASARC software determines pairwise compatibility between all possible service areas at discrete arc locations. NASARC then exhaustively enumerates groups of administrations whose satellites can be closely located in orbit, and finds the arc segment over which each such compatible group exists. From the set of all possible compatible groupings, groups and their associated arc segments are selected using a heuristic procedure such that a PDA is identified for each administration. Various aspects of the NASARC concept and how the software accomplishes specific features of allotment planning are discussed.

Whyte, W. A.↗

An improved exploratory search technique for pure integer linear programming problems

The development is documented of a heuristic method for the solution of pure integer linear programming problems. The procedure draws its methodology from the ideas of Hooke and Jeeves type 1 and 2 exploratory searches, greedy procedures, and neighborhood searches. It uses an efficient rounding method to obtain its first feasible integer point from the optimal continuous solution obtained via the simplex method. Since this method is based entirely on simple addition or subtraction of one to each variable of a point in n-space and the subsequent comparison of candidate solutions to a given set of constraints, it facilitates significant complexity improvements over existing techniques. It also obtains the same optimal solution found by the branch-and-bound technique in 44 of 45 small to moderate size test problems. Two example problems are worked in detail to show the inner workings of the method. Furthermore, using an established weighted scheme for comparing computational effort involved in an algorithm, a comparison of this algorithm is made to the more established and rigorous branch-and-bound method. A computer implementation of the procedure, in PC compatible Pascal, is also presented and discussed.

Fogle, F. R.↗

Final Technical and Scientific Report: Advanced Control of the Azura Wave Energy Device

The objective of the project was to develop a model predictive controller and associated hydraulic PTO for the Azura wave energy device to improve the annual energy production (AEP) and reduce the levelized cost of electricity (LCOE) of the device. Performance of the controller and hydraulic PTO were validated in a 1/15th scale wave tank test performed at the University of Maine. Three controllers, with three corresponding PTOs, were developed and tested as part of the project. The three controllers are an ideal MPC controller paired with an ideal PTO, an MPC controller with a heuristic paired with a directional force control hydraulic PTO, and a preliminary study of an MPC controller applied to a variable force hydraulic PTO. Experimental data was used to estimate the controller paired with the directional force control would increase the annual energy production of the Azura at WETS by 12%, with a corresponding reduction in LCOE of 10.8%. Preliminary testing of the variable force hydraulic PTO paired with a MPC shows this configuration has the potential to increase the AEP of the Azura at WETS by at least 33%. Further design and development work is necessary to develop the PTO technology to achieve these gains. Additionally, all control schemes require additional work to address the short-term wave prediction method needed to implement the control. In addition to validation of these control methods, NWEI demonstrated careful design of a wave tank test model, PTO, and instrumentation yields high quality test data, and the ability to test advanced control algorithms and include complex PTO dynamics via hardware-in-the-loop testing techniques.

16 TIDAL AND WAVE POWER↗

Accelerating Continuous Variable Coherent Ising Machines Via Momentum

The Coherent Ising Machine (CIM) is a non-conventional architecture that takes inspiration from physical annealing processes to solve Ising problems heuristically. Its dynamics are naturally continuous and described by a set of ordinary differential equations that have been proven to be useful for the optimization of continuous variables nonconvex quadratic optimization problems. The dynamics of such Continuous Variable CIMs (CV-CIM) encourage optimization via optical pulses whose amplitudes are determined by the negative gradient of the objective; however, standard gradient descent is known to be trapped by local minima and hampered by poor problem conditioning. In this work, we propose to modify the CV-CIM dynamics using more sophisticated pulse injections based on tried-and-true optimization techniques such as momentum and Adam. Through numerical experiments, we show that the momentum and Adam updates can significantly speed up the CV-CIM’s convergence and improve sample diversity over the original CV-CIM dynamics. We also find that the Adam-CV-CIM’s performance is more stable as a function of feedback strength, especially on poorly conditioned instances, resulting in an algorithm that is more robust, reliable, and easily tunable. More broadly, we identify the CIM dynamical framework as a fertile opportunity for exploring the intersection of classical optimization and modern analog computing.

Ising Model↗

THE IMPORTANCE OF WIDE-FIELD FOREGROUND REMOVAL FOR 21 cm COSMOLOGY: A DEMONSTRATION WITH EARLY MWA EPOCH OF REIONIZATION OBSERVATIONS

In this paper we present observations, simulations, and analysis demonstrating the direct connection between the location of foreground emission on the sky and its location in cosmological power spectra from interferometric redshifted 21 cm experiments. We begin with a heuristic formalism for understanding the mapping of sky coordinates into the cylindrically averaged power spectra measurements used by 21 cm experiments, with a focus on the effects of the instrument beam response and the associated sidelobes. We then demonstrate this mapping by analyzing power spectra with both simulated and observed data from the Murchison Widefield Array. We find that removing a foreground model that includes sources in both the main field of view and the first sidelobes reduces the contamination in high k{sub ∥} modes by several per cent relative to a model that only includes sources in the main field of view, with the completeness of the foreground model setting the principal limitation on the amount of power removed. While small, a percent-level amount of foreground power is in itself more than enough to prevent recovery of any Epoch of Reionization signal from these modes. This result demonstrates that foreground subtraction for redshifted 21 cm experiments is truly a wide-field problem, and algorithms and simulations must extend beyond the instrument’s main field of view to potentially recover the full 21 cm power spectrum.

79 ASTRONOMY AND ASTROPHYSICS↗

TH-CD-207B-09: Task-Driven Fluence Field Modulation Design for Model-Based Iterative Reconstruction in CT

Purpose: There has been increasing interest in integrating fluence field modulation (FFM) devices with diagnostic CT scanners for dose reduction purposes. Conventional FFM strategies, however, are often either based on heuristics or the analysis of filtered-backprojection (FBP) performance. This work investigates a prospective task-driven optimization of FFM for model-based iterative reconstruction (MBIR) in order to improve imaging performance at the same total dose as conventional strategies. Methods: The task-driven optimization framework utilizes an ultra-low dose 3D scout as a patient-specific anatomical model and a mathematical formation of the imaging task. The MBIR method investigated is quadratically penalized-likelihood reconstruction. The FFM objective function uses detectability index, d’, computed as a function of the predicted spatial resolution and noise in the image. To optimize performance throughout the object, a maxi-min objective was adopted where the minimum d’ over multiple locations is maximized. To reduce the dimensionality of the problem, FFM is parameterized as a linear combination of 2D Gaussian basis functions over horizontal detector pixels and projection angles. The coefficients of these bases are found using the covariance matrix adaptation evolution strategy (CMA-ES) algorithm. The task-driven design was compared with three other strategies proposed for FBP reconstruction for a calcification cluster discrimination task in an abdomen phantom. Results: The task-driven optimization yielded FFM that was significantly different from those designed for FBP. Comparing all four strategies, the task-based design achieved the highest minimum d’ with an 8–48% improvement, consistent with the maxi-min objective. In addition, d’ was improved to a greater extent over a larger area within the entire phantom. Conclusion: Results from this investigation suggests the need to re-evaluate conventional FFM strategies for MBIR. The task-based optimization framework provides a promising approach that maximizes imaging performance under the same total dose constraint.

60 APPLIED LIFE SCIENCES↗

Inter-Domain Fusion for Enhanced Intrusion Detection in Power Systems: An Evidence Theoretic and Meta-Heuristic Approach

False alerts due to misconfigured or compromised intrusion detection systems (IDS) in industrial control system (ICS) networks can lead to severe economic and operational damage. However, research using deep learning to reduce false alerts often requires the physical and cyber sensor data to be trustworthy. Implicit trust is a major problem for artificial intelligence or machine learning (AI/ML) in cyber-physical system (CPS) security, because when these solutions are most urgently needed is also when they are most at risk (e.g., during an attack). To address this, the Inter-Domain Evidence theoretic Approach for Inference (IDEA-I) is proposed that reframes the detection problem as how to make good decisions given uncertainty. Specifically, an evidence theoretic approach leveraging Dempster–Shafer (DS) combination rules and their variants is proposed for reducing false alerts. A multi-hypothesis mass function model is designed that leverages probability scores obtained from supervised-learning classifiers. Using this model, a location-cum-domain-based fusion framework is proposed to evaluate the detector’s performance using disjunctive, conjunctive, and cautious conjunctive rules. The approach is demonstrated in a cyber-physical power system testbed, and the classifiers are trained with datasets from Man-In-The-Middle attack emulation in a large-scale synthetic electric grid. For evaluating the performance, we consider plausibility, belief, pignistic, and general Bayesian theorem-based metrics as decision functions. To improve the performance, a multi-objective-based genetic algorithm is proposed for feature selection considering the decision metrics as the fitness function. Finally, we present a software application to evaluate the DS fusion approaches with different parameters and architectures.

42 ENGINEERING↗

QASMTrans: A QASM Quantum Transpiler Framework for NISQ Devices

In quantum computing, transpilation plays a crucial role in converting high-level, machine-independent quantum circuits into circuits specially for a quantum device, considering factors such as basis gate set, topology, error profile, etc. Yet, the efficiency of transpilation remains a significant bottleneck, particularly when dealing with very large QASM level input files. In this paper, we present QASMTrans, a C++ based high-performance quantum transpiler framework that can demonstrate on average 50-100× speedups compared to the internal transpiler of Qiskit. Particularly, for large dense circuits such as ’uccsd n24’ and ’qft n320’ incorporating millions of gates, QASMTrans can successfully transpile in 69s and 31s, respectively, while Qiskit failed to finish in one hour. Using QASMTrans as the baseline, it becomes more feasible to explore much larger design space and impose more comprehensive compiler optimizations.

Hua, Fei↗

Commercial Non-Dispersive Infrared Spectroscopy Sensors for Sub-Ambient Carbon Dioxide Detection

Carbon dioxide produced through respiration can accumulate rapidly within closed spaces. If not managed, a crew's respiratory rate increases, headaches and hyperventilation occur, vision and hearing are affected, and cognitive abilities decrease. Consequently, development continues on a number of CO2 removal technologies for human spacecraft and spacesuits. Terrestrially, technology development requires precise performance characterization to qualify promising air revitalization equipment. On-orbit, instrumentation is required to identify and eliminate unsafe conditions. This necessitates accurate in situ CO2 detection. Recursive compensation algorithms were developed for sub-ambient detection of CO2 with commercial off-the-shelf (COTS) non-dispersive infrared (NDIR) sensors. In addition, the source of the exponential loss in accuracy is developed theoretically. The basis of the loss can be explained through thermal, Doppler, and Lorentz broadening effects that arise as a result of the temperature, pressure, and composition of the gas mixture under analysis. The objective was to develop a mathematical routine to compensate COTS CO2 sensors relying on NDIR over pressures, temperatures, and compositions far from calibration conditions. The routine relies on a power-law relationship for the pressure dependency of the sensors along with an equivalent pressure to account for the composition dependency. A Newton-Raphson iterative technique solves for actual carbon dioxide concentration based on the reported concentration. Moreover, first principles routines were established to predict mixed-gas spectra based on sensor specifications (e.g., optical path length). The first principles model can be used to parametrically optimize sensors or sensor arrays across a wide variety of pressures/temperatures/ compositions. In this work, heuristic scaling arguments were utilized to develop reasonable compensation techniques. Experimental results confirmed this approach and provided evidence that composition broadening significantly alters spectra when pressure is reduced. Consequently, a recursive compensation technique was developed with the Newton-Raphson method, which was subsequently verified through experimentation.

Swickrath, Michael J.↗

PROPERTIES OF SEQUENTIAL CHROMOSPHERIC BRIGHTENINGS AND ASSOCIATED FLARE RIBBONS

We report on the physical properties of solar sequential chromospheric brightenings (SCBs) observed in conjunction with moderate-sized chromospheric flares with associated Coronal mass ejections. To characterize these ephemeral events, we developed automated procedures to identify and track subsections (kernels) of solar flares and associated SCBs using high-resolution H{alpha} images. Following the algorithmic identification and a statistical analysis, we compare and find the following: SCBs are distinctly different from flare kernels in their temporal characteristics of intensity, Doppler structure, duration, and location properties. We demonstrate that flare ribbons are themselves made up of subsections exhibiting differing characteristics. Flare kernels are measured to have a mean propagation speed of 0.2 km s{sup -1} and a maximum speed of 2.3 km s{sup -1} over a mean distance of 5 Multiplication-Sign 10{sup 3} km. Within the studied population of SCBs, different classes of characteristics are observed with coincident negative, positive, or both negative and positive Doppler shifts of a few km s{sup -1}. The appearance of SCBs precedes peak flare intensity by Almost-Equal-To 12 minutes and decay Almost-Equal-To 1 hr later. They are also found to propagate laterally away from flare center in clusters at 45 km s{sup -1} or 117 km s{sup -1}. Given SCBs' distinctive nature compared to flares, we suggest a different physical mechanism relating to their origin than the associated flare. We present a heuristic model of the origin of SCBs.

79 ASTRONOMY AND ASTROPHYSICS↗

ALTSched: Improved Scheduling for Time-domain Science with LSST

Abstract Telescope scheduling is the task of determining the best sequence of observations (pointings and filter choices) for a survey system. Because it is computationally intractable to optimize over all possible multiyear sequences of observations, schedulers use heuristics to pick the best observation at a given time. A greedy scheduler selects the next observation by choosing whichever one maximizes a scalar merit function, which serves as a proxy for the scientific goals of the telescope. This sort of bottom-up approach for scheduling is not guaranteed to produce a schedule for which the sum of merit over all observations is maximized. As an alternative to greedy schedulers, we introduce ALTS ched, which takes a top-down approach to scheduling. Instead of considering only the next observation, ALTS ched makes global decisions about which area of sky and which filter to observe in, and then refines these decisions into a sequence of observations taken along the meridian to minimize airmass. We implement ALTS ched for the Large Synoptic Survey Telescope (LSST) and show that it equals or outperforms the baseline greedy scheduler in essentially all quantitative performance metrics. Due to its simplicity, our implementation is considerably faster than O p S im , the simulated greedy scheduler currently used by the LSST Project: a full 10-yr survey can be simulated in 4 minutes, as opposed to tens of hours for O p S im . LSST’s hardware is fixed, so improving the scheduling algorithm is one of the only remaining ways to optimize LSST’s performance. We see ALTS ched as a prototype scheduler that gives a lower bound on the performance achievable by LSST.

79 ASTRONOMY AND ASTROPHYSICS↗

Adaptable Constrained Genetic Programming: Extensions and Applications

An evolutionary algorithm applies evolution-based principles to problem solving. To solve a problem, the user defines the space of potential solutions, the representation space. Sample solutions are encoded in a chromosome-like structure. The algorithm maintains a population of such samples, which undergo simulated evolution by means of mutation, crossover, and survival of the fittest principles. Genetic Programming (GP) uses tree-like chromosomes, providing very rich representation suitable for many problems of interest. GP has been successfully applied to a number of practical problems such as learning Boolean functions and designing hardware circuits. To apply GP to a problem, the user needs to define the actual representation space, by defining the atomic functions and terminals labeling the actual trees. The sufficiency principle requires that the label set be sufficient to build the desired solution trees. The closure principle allows the labels to mix in any arity-consistent manner. To satisfy both principles, the user is often forced to provide a large label set, with ad hoc interpretations or penalties to deal with undesired local contexts. This unfortunately enlarges the actual representation space, and thus usually slows down the search. In the past few years, three different methodologies have been proposed to allow the user to alleviate the closure principle by providing means to define, and to process, constraints on mixing the labels in the trees. Last summer we proposed a new methodology to further alleviate the problem by discovering local heuristics for building quality solution trees. A pilot system was implemented last summer and tested throughout the year. This summer we have implemented a new revision, and produced a User's Manual so that the pilot system can be made available to other practitioners and researchers. We have also designed, and partly implemented, a larger system capable of dealing with much more powerful heuristics.

Janikow, Cezary Z.↗

Computing molecular excited states on a D-Wave quantum annealer

Abstract The possibility of using quantum computers for electronic structure calculations has opened up a promising avenue for computational chemistry. Towards this direction, numerous algorithmic advances have been made in the last five years. The potential of quantum annealers, which are the prototypes of adiabatic quantum computers, is yet to be fully explored. In this work, we demonstrate the use of a D-Wave quantum annealer for the calculation of excited electronic states of molecular systems. These simulations play an important role in a number of areas, such as photovoltaics, semiconductor technology and nanoscience. The excited states are treated using two methods, time-dependent Hartree–Fock (TDHF) and time-dependent density-functional theory (TDDFT), both within a commonly used Tamm–Dancoff approximation (TDA). The resulting TDA eigenvalue equations are solved on a D-Wave quantum annealer using the Quantum Annealer Eigensolver (QAE), developed previously. The method is shown to reproduce a typical basis set convergence on the example $$\hbox {H}_2$$ H 2 molecule and is also applied to several other molecular species. Characteristic properties such as transition dipole moments and oscillator strengths are computed as well. Three potential energy profiles for excited states are computed for $$\hbox {NH}_3$$ NH 3 as a function of the molecular geometry. Similar to previous studies, the accuracy of the method is dependent on the accuracy of the intermediate meta-heuristic software called qbsolv.

74 ATOMIC AND MOLECULAR PHYSICS↗

Analytic Theory for the Dynamics of Wide Quantum Neural Networks

Here, parametrized quantum circuits can be used as quantum neural networks and have the potential to outperform their classical counterparts when trained for addressing learning problems. To date, much of the results on their performance on practical problems are heuristic in nature. In particular, the convergence rate for the training of quantum neural networks is not fully understood. Here, we analyze the dynamics of gradient descent for the training error of a class of variational quantum machine learning models. We define wide quantum neural networks as parametrized quantum circuits in the limit of a large number of qubits and variational parameters. Then, we find a simple analytic formula that captures the average behavior of their loss function and discuss the consequences of our findings. For example, for random quantum circuits, we predict and characterize an exponential decay of the residual training error as a function of the parameters of the system. Finally, we validate our analytic results with numerical experiments.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A comprehensive formulation for volumetric modulated arc therapy planning

Purpose: Volumetric modulated arc therapy (VMAT) is a widely employed radiation therapy technique, showing comparable dosimetry to static beam intensity modulated radiation therapy (IMRT) with reduced monitor units and treatment time. However, the current VMAT optimization has various greedy heuristics employed for an empirical solution, which jeopardizes plan consistency and quality. The authors introduce a novel direct aperture optimization method for VMAT to overcome these limitations. Methods: The comprehensive VMAT (comVMAT) planning was formulated as an optimization problem with an L2-norm fidelity term to penalize the difference between the optimized dose and the prescribed dose, as well as an anisotropic total variation term to promote piecewise continuity in the fluence maps, preparing it for direct aperture optimization. A level set function was used to describe the aperture shapes and the difference between aperture shapes at adjacent angles was penalized to control MLC motion range. A proximal-class optimization solver was adopted to solve the large scale optimization problem, and an alternating optimization strategy was implemented to solve the fluence intensity and aperture shapes simultaneously. Single arc comVMAT plans, utilizing 180 beams with 2° angular resolution, were generated for a glioblastoma multiforme case, a lung (LNG) case, and two head and neck cases—one with three PTVs (H&N{sub 3PTV}) and one with foue PTVs (H&N{sub 4PTV})—to test the efficacy. The plans were optimized using an alternating optimization strategy. The plans were compared against the clinical VMAT (clnVMAT) plans utilizing two overlapping coplanar arcs for treatment. Results: The optimization of the comVMAT plans had converged within 600 iterations of the block minimization algorithm. comVMAT plans were able to consistently reduce the dose to all organs-at-risk (OARs) as compared to the clnVMAT plans. On average, comVMAT plans reduced the max and mean OAR dose by 6.59% and 7.45%, respectively, of the prescription dose. Reductions in max dose and mean dose were as high as 14.5 Gy in the LNG case and 15.3 Gy in the H&N{sub 3PTV} case. PTV coverages measured by D95, D98, and D99 were within 0.25% of the prescription dose. By comprehensively optimizing all beams, the comVMAT optimizer gained the freedom to allow some selected beams to deliver higher intensities, yielding a dose distribution that resembles a static beam IMRT plan with beam orientation optimization. Conclusions: The novel nongreedy VMAT approach simultaneously optimizes all beams in an arc and then directly generates deliverable apertures. The single arc VMAT approach thus fully utilizes the digital Linac’s capability in dose rate and gantry rotation speed modulation. In practice, the new single VMAT algorithm generates plans superior to existing VMAT algorithms utilizing two arcs.

60 APPLIED LIFE SCIENCES↗

Qubit Assignment Using Time Reversal

As quantum computers with large numbers of qubits become increasingly available, experiments executed on a given device may not utilize all available qubits. In this case, the outcome of executing a quantum program will depend on the ability to efficiently select a subset of high-performing physical qubits. For any given quantum program and device there are many ways to assign physical qubits for execution of the program, and assignments will differ in performance due to the variability in quality across qubits and entangling operations on a single device. Evaluating the performance of each assignment using fidelity estimation introduces significant experimental overhead and will be infeasible for many applications, while relying on standard device benchmarks provides incomplete information about the performance of any specific program. Furthermore, the number of possible assignments grows combinatorially in the number of qubits on the device and in the program, motivating the use of heuristic optimization techniques. We demonstrate a practical solution to the problem of qubit assignment by using simulated annealing with a cost function based on the Loschmidt echo, a diagnostic that measures the reversibility of a quantum process. We provide theoretical justification for this choice of cost function by demonstrating that the optimal qubit assignment coincides with the optimal qubit assignment based on state fidelity in the weak error limit, and we provide experimental justification using diagnostics performed on Google’s superconducting qubit devices. We then establish the performance of simulated annealing for qubit assignment using classical simulations of noisy devices as well as optimization experiments performed on a quantum processor. Our results demonstrate that the use of Loschmidt echoes and simulated annealing provides a scalable and flexible approach to optimizing qubit assignment on near-term hardware.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Epi Archive: Automated Synthesis of Global Notifiable Disease Data

ObjectiveLANL has built software that automatically collects global notifiable disease data, synthesizes the data, and makes it available to humans and computers within the Biosurveillance Ecosystem (BSVE) as a novel data stream. These data have many applications including improving the prediction and early warning of disease events.IntroductionMost countries do not report national notifiable disease data in a machine-readable format. Data are often in the form of a file that contains text, tables and graphs summarizing weekly or monthly disease counts. This presents a problem when information is needed for more data intensive approaches to epidemiology, biosurveillance and public health.While most nations likely store incident data in a machine-readable format, governments are often hesitant to share data openly for a variety of reasons that include technical, political, economic, and motivational issues1.A survey conducted by LANL of notifiable disease data reporting in over fifty countries identified only a few websites that report data in a machine-readable format. The majority (>70%) produce reports as PDF files on a regular basis. The bulk of the PDF reports present data in a structured tabular format, while some report in natural language.The structure and format of PDF reports change often; this adds to the complexity of identifying and parsing the desired data. Not all websites publish in English, and it is common to find typos and clerical errors.LANL has developed a tool, Epi Archive, to collect global notifiable disease data automatically and continuously and make it uniform and readily accessible.MethodsWe conducted a survey of the national notifiable disease reporting systems notating how the data are reported and in what formats. We determined the minimal metadata that is required to contextualize incident counts properly, as well as optional metadata that is commonly found.The development of software to regularly ingest notifiable disease data and make it available involves three or four main steps: scraping, detecting, parsing and persisting.Scraping: we examine website design and determine reporting mechanisms for each country/website, as well as what varies across the reporting mechanisms. We then designed and wrote code to automate the downloading of the data for each country. We store all artifacts presented as files (PDF, XLSX, etc.) in their original form, along with appropriate metadata for parsing and data provenance.Detecting: This step is required when parsing structured non-machine-readable data such as tabular data in PDF files. We combined the Nurminen methodology of PDF table detection with in-house heuristics to find the desired data within PDF reports2.Parsing: We determined what to extract from each dataset and parsed these data into uniform data structures, correctly accommodating the variations in metadata (e.g., time interval definitions) and the various human languages.Persisting: We store the data in the Epi Archive database and make it available on the internet and through the BSVE. The data is persisted into a structured and normalized SQL database.ResultsThe Epi Archive tool currently contains national and/or subnational notifiable disease data from twenty nations. When a user accesses the Epi Archive site, they are prompted with four fields: country, subregion, disease of interest, and date duration. Upon form submission, a time series is generated from the users’ specifications. The generated graph can then be downloaded into a CSV file if a user is interested in performing personal analysis. Additionally, the data from Epi Archive can be reached through a REST API (Representational State Transfer Application Programming Interface).ConclusionsLANL, as part of a currently funded DTRA effort, is automatically and continually collecting global notifiable disease data. While 20 nations are in production, more are being brought online in the near future. These data are already being utilized and will have many applications including improving the prediction and early warning of disease events.References[1] van Panhuis WG, Paul P, Emerson C, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014. 14:1144. doi:10.1186/1471-2458-14-1144[2] Nurminen, Anssi. "Algorithmic extraction of data in tables in PDF documents." (2013).

Khalsa, Hari S.↗