print font size font_plus font_minus font_reset

PUBLICATIONS

2017 | 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009 | 2008

The following articles and reports discuss topics relevant to systems engineering.
(See also ODASD(SE) Briefs; Policy and Guidance.)

2017

Digital Model-based Engineering: Expectations, Prerequisites, and Challenges of Infusion
Model-Based Systems Engineering (MBSE) Infusion Task Team, Interagency Working Group on Engineering Complex Systems (IAWG), March 2017.
DASD(SE), with support from other federal agencies, produced a paper discussing digital model-based engineering (DMbE) in an effort to identify what government organizations might expect in the course of moving to or infusing MBSE into their organizations. The MBSE Infusion Task Team developed the term "DMbE" to normalize, for the purposes of setting expectations and challenges, other similar initiatives, such as MBSE, model-based engineering, digital engineering, etc. This work should be viewed as a way to introduce the topic into an organization, with expectations for success, but without intruding on any specific implementation details of an organization or component. Please see also the Digital Engineering initiatives page on this website.

Digital Engineering Transformation Across the Department of Defense
Poster. Tracee Gilbert, Ph.D., et al., under contract with Office of the Deputy Assistant Secretary of Defense for Systems Engineering, 2017.

Model-Based Systems Engineering: Enabling the Digital Engineering Practice in the Department of Defense
Kristen Baldwin, Getting It Right 7(3), February 27, 2017: 1, 3.

2016

System-Aware Cyber Security: A Systems Engineering Approach for Enhancing Cyber Security.3
Barry M. Horowitz and D. Scott Lucero, INCOSE Insight 19(2) (July 2016): 39-42.
This article describes methods for using design patterns to add a layer of security to detect and deflect attacks that have successfully penetrated the perimeter of a cyber physical system, either from outside attacks or from supply chain or insider-initiated attacks. The article makes the case for trying new engineering processes in a prototyping environment, in addition to performing traditional technology development.

SD-22 – Diminishing Manufacturing Sources and Material Shortages (DMSMS): A Guidebook of Best Practices for Implementing a Robust DMSMS Management Program.
Alex Melnikow, Defense Standardization Program Office, January 2016.
Because Department of Defense (DoD) system life cycles are longer than technology life cycles, Diminishing Manufacturing Sources and Material Shortages (DMSMS) issues are inevitable. DoD cannot afford to be reactive in this area; reactivity may lead to a combination of schedule delays, readiness degradations, and higher cost. This guidebook provides best practices for robust and proactive DMSMS management. This version superseded the February 2015 version.

Defense System Complexity: Engineering Challenges and Opportunities6
Kristen Baldwin and D. Scott Lucero, The ITEA Journal of Test and Evaluation, March 2016: 10-16.
Although systems engineering and testing are among the DoD’s strengths, the traditional practice of engineering is challenged by many factors that translate into design and performance demands for defense engineers. The article discusses the challenges as well as DoD’s efforts to sustain and strengthen critical organic workforce capabilities; improve engineering, test, and evaluation methods and tools; and broaden partnerships with commercial and defense industry, universities, and research centers in a continued commitment to achieve superior capability for U.S. warfighters.

2015

Establishing the Technical Foundation: Materiel Solution Analysis Is More than Selecting an Alternative
Aileen G. Sedmak, Zachary S. Taylor, and Lt Col William A. Riski, USAF (Ret.), Defense Acquisition Research Journal 22(4) (October 2015): 364-393.
Several government and independent studies indicate effective systems engineering and program planning in the early stages of acquisition are essential to controlling costs and improving program results. To lay the foundation for successful and executable programs, this article describes the challenge of conducting good systems engineering and technical planning during the Materiel Solution Analysis (MSA) phase after completion of the Analysis of Alternatives and before Milestone A. It also presents the work of the DoD Development Planning Working Group to mitigate this challenge by describing the technical activities in the MSA phase to develop the level of knowledge and system concept maturity necessary to proceed into the next phase of acquisition. These technical activities are represented in a notional MSA Phase Activity Model.

Software and Hardware Assurance: DoD Establishes Federation of Software and Hardware Assurance
Tom Hurt and Ray Shanahan, CrossTalk: The Journal of Defense Software Engineering 28(5) (September/October 2015): 11-13.
Keeping DoD hardware and software technology secure is more critical than ever. In response to a mandate from Congress, Deputy Secretary of Defense Robert O. Work chartered the DoD Joint Federated Assurance Center (JFAC) as a federation of U.S. Military Department and agency software assurance (SwA) and hardware assurance (HwA) organizations and capabilities. According to this charter, the JFAC is charged with supporting program offices throughout the life cycle with SwA and HwA expertise, capabilities, policies, guidance, and best practices. The JFAC is responsible for coordinating with DoD organizations and activities that are developing, maintaining, and offering software and hardware vulnerability detection, analysis, and remediation support. Other responsibilities of the JFAC include (1) conducting SwA and HwA analyses and assessments in support of defense acquisition, operations, and sustainment activities; (2) advocating for the advancement of DoD interests in SwA and HwA research, development, and test and evaluation activities; and (3) building relationships with other communities of interest and practice in SwA and HwA such as other government organizations, academic environments, and private industry.

2014

Department of Defense Assured Microelectronics Policy in Response to Senate Report 113-85, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, July 2014.
This report on assured microelectronics policy is in response to Senate Appropriations Committee (SAC) 113-85, page 179, accompanying S. 1429, the Department of Defense Appropriations Bill, 2014, SAC-D, which states: "Assured Microelectronics.—The Committee understands that the Department of Defense issued an instruction which mandates assurance measures for all information and weapons systems that are national security systems, mission assurance category one, or are otherwise critical military and intelligence systems. The Committee directs the Department to deliver a report within 180 days of the enactment of this act on the progress implementing this assured microelectronics policy." The policy referred to in the SAC report is Department of Defense (DoD) Instruction (DoDI) 5200.44, "Protection of Mission-Critical Functions to Achieve Trusted Systems and Networks (TSN)," November 5, 2012. The Instruction provides a strategy for acquisition programs to integrate robust systems engineering, supply chain risk management (SCRM), security, counterintelligence (CI), intelligence, information assurance, software assurance, and hardware assurance (with an emphasis on microelectronics) for managing risks to system integrity and trust. The Instruction provides guidance for managing the risk that a foreign intelligence or other hostile elements could exploit supply chain vulnerabilities to sabotage or subvert mission-critical functions, system designs, or critical components (CC).

2013

Transforming the Practice of Engineering for Large Complex Systems
Inter-Agency Working Group on the Engineering of Complex Systems, December 2013.
The complexity of engineered systems has swelled in the last several decades, and this trend is likely to continue for the foreseeable future. While projects are becoming more complex, current engineering practice has largely evolved from a top-down approach that is the legacy of past successes. A fundamental rethinking of engineering methodologies is urgently needed if our nation is to ensure that the large complex systems critical to our national security, economy, and quality of life are resilient in the face of natural disasters, creative adversaries, and an unforeseeable future. In June 2012, an Inter-Agency Working Group (IAWG) on Engineering Complex Systems convened to explore these issues from the perspectives of several government agencies that represent diverse missions. The IAWG seeks to stimulate a dialogue that will help usher the engineering community toward the next generation of research and practice.

Requirements Challenges in Addressing Malicious Supply Chain Threats3
Paul R. Popick and Melinda Reed, INCOSE Insight 16(2) (July 2013): 23-27.
In today’s environment of cyber attacks and exploitation of system vulnerabilities, the systems engineer needs to be more aware of security during the system specification and design stage. This article discusses the U.S. Department of Defense (DoD) state of practice for incorporating trusted system and network security requirements into the specifications for large, complex systems. The article describes the current environment, the trends that are influencing the need for system security engineering, and the types of system security requirements and analysis techniques the DoD is using.

Towards Affordably Adaptable and Effective Systems2
Robert Neches and Azad M. Madni, Systems Engineering 16(2) (Summer 2013): 224–234.
Resilience means different things in different disciplines. From a systems engineering perspective, we define resilience as the ability of a system to adapt affordably and perform effectively across a wide range of operational contexts, where context is defined by mission, environment, threat, and force disposition. A key issue in engineering resilient systems is the lengthy and costly upfront engineering process, which program managers justifiably find unacceptable. This paper presents how advances in computational technology can potentially transform the system development process in new and novel ways to enable fast, efficient, and inexpensive upfront engineering—the key to engineering resilient systems. These processes, in turn, can enable rapid development, deployment, and operation of affordably adaptable and effective systems.

2012

Executive Summary and Addendum: Report on Trusted Defense Systems in Response to National Defense Authorization Act, Section 254, December 22, 2009; addendum January 2010–November 2012.
In accordance with Section 254 of Public Law 110-417, “Duncan Hunter National Defense Authorization Act for Fiscal Year 2009” (FY09 NDAA Section 254), the Department of Defense (DoD) completed three vulnerability assessments on selected acquisition programs, completed a study of techniques for verifying trust in integrated circuits, and validated and continued to implement the Department’s Strategy for Systems Assurance and Trustworthiness using the results from the assessments and study. The Department issued policy and conducted activities designed to assure trust in integrated circuits, software, and other electronic components. The executive summary and addendum provide highlights of the report on these DoD efforts.

The United States Department of Defense Revitalization of System Security Engineering Through Program Protection1
Kristen Baldwin, Paul R. Popick, John F. Miller, and Jonathan Goodnight, Proceedings of the IEEE Systems Conference 2012, Vancouver, British Columbia, March 19-22, 2012.
This paper discusses system security engineering (SSE), specifically the Department's policies and techniques to integrate security into systems engineering through the program protection process. Although SSE is normally viewed as a specialty engineering area, the paper emphasizes the need to more tightly integrate SSE with overall systems engineering.

2011

Systems of Systems and Security: A Defense Perspective3
Kristen Baldwin, Judith Dahmann, and Jonathan Goodnight, INCOSE Insight 14(2) (July 2011): 11-13.
The U.S. Department of Defense (DoD) is working to integrate security into the systems engineering tradespace for designing and acquiring military systems. This article examines the current United States defense approach to security and the challenges posed by systems of systems. IT intentionally does not address challenges that are not unique to systems of systems. Although the situation described is specific to the defense domain, the authors believe that the issues also apply beyond the defense community to other areas including integrated transportation systems, financial systems, and other critical infrastructure.

Implications of Systems of Systems on System Design and Engineering1
Judith Dahmann, Ph.D., and Kristen Baldwin. Proceedings of the IEEE 6th International Conference on System of Systems Engineering, Albuquerque, NM, June 27-30, 2011.
Over the past 10 years there has been a steady growth in attention to issues related to systems of systems (SoS) and systems engineering, particularly in Defense in the United States. This attention has focused on how to apply SE principles and practices to SoS, considering the differences between systems and SoS. For many organizations, however, despite recognition of SoS considerations, the focus of investment and development continues to be on individual systems. This paper looks at SoS and SE from the perspective of constituent systems and examines impacts on systems engineering of systems in light of the increased prevalence of SoS. The paper addresses these issues based on the experience and viewpoint of the U.S. Department of Defense and identifies areas for further attention in systems engineering research and practice.

An Implementers’ View of Systems Engineering for Systems of Systems1
Judith Dahmann, Ph.D., et al. Proceedings of the IEEE Systems Conference 2011, Montreal, Quebec, April 4-7, 2011.
This paper builds on and extends U.S. Department of Defense guidance on systems engineering of systems of systems by developing and presenting a view of SoS SE that translates the SoS SE core elements, their interrelationships, and SoS decision-making artifacts and information from a “trapeze” model to a more familiar and intuitive time-sequenced “wave” model representation. The information is thus rendered in a form more readily usable by SoS SE practitioners in the field and one that corresponds with incremental development approaches that are the norm for SoS capability evolution. The paper describes and motivates the development of the wave model, discusses its key characteristics, and provides examples of SoS efforts that reflect this view of SoS SE. The paper describes how the information critical to successful SoS SE is created, where it fits into the wave model, how it evolves over time, and in which artifacts the information is normally contained.

2010

Proceedings of the 1st Workshop on U.S. Undergraduate Systems Engineering Programs, Colorado Springs, Colorado, April 7–8, 2010, Sponsored by the United States Air Force Academy and Director, Defense Research and Engineering, USD(AT&L)
Workshop Materials: Quick Look | Agenda | Final Report | Compressed Briefs
The goals of the workshop were to: (1) analyze the current state of bachelor’s degree programs in Systems Engineering across the United States, (2) explore where those programs are headed, and (3) propose actions that could be taken by academia, industry, and government to strengthen the value those programs offer to students, universities, employers, and the nation.

System Engineering Artifacts for SoS1
Judith Dahmann, Ph.D., et al. Proceedings of the IEEE Systems Conference 2010, San Diego, April 6, 2010.
This paper describes system of systems (SoS) systems engineering (SE) artifacts, compares and contrasts them with similar ones developed and used for individual systems, and explains how they are used to guide SoS engineering processes. The paper concludes with next steps for using SoS artifacts to continue maturing the understanding of SoS SE in an international cooperative effort with the United Kingdom, Australia, and Canada.

2009

Department of Defense Report to Congress Implementation of Recommendations on Total Ownership Cost on Major Weapon Systems, Office of the Director for Defense Research and Engineering/Systems Engineering, September 2009.
The Department of Defense (DoD) submitted this Report to Congress in response to Section 818 of the National Defense Authorization Act for FY 2008 (Pub. L. 110-181), Report on Implementation of Recommendations on Total Ownership Cost for Major Weapon Systems. All GAO recommendations have been implemented. At the Department level, policy and guidance is in place to set requirements for acquisition program reliability and ownership cost, to develop systems using a reliability growth program as an integral part of design and development, and to suggest including best practices for reliability in defense contracts for major systems acquisitions. The OSD acquisition and test oversight processes emphasize reliability. Follow-through across the military departments varies in terms of component acquisition policy and implementation. Workforce education, personnel resources, contracting, system design tradeoffs, and acquisition decision making are areas for continued emphasis. As the department continues efforts to improve system reliability and the resulting total cost of ownership, it looks forward to opportunities to report progress to Congress.

Systems Security Engineering: A Critical Discipline of Systems Engineering3
Kristen Baldwin, INCOSE Insight 12(2) (July 2009): 11-13.
This article addresses the comprehensive set of threats that the United States Department of Defense (DoD) must consider with respect to its acquisition programs resulting in the need to recognize systems security engineering as a critical element of systems engineering. Security specialties that have emerged over time as responses to new threats and risks include information security to protect information and information systems from unauthorized access, use, disclosure, disruption, modification or destruction; physical and personnel security to protect information and other valuable assets physically stored within facilities and installations; and communications and network security to protect electronic information in transit over networks. Security has now become a system-level risk. This article provides a summary of DoD's system assurance activities, the state of the practice, and the Defense Departments's response and planned way ahead for systems security engineering.

New Methods to Measure the Safety of Military Equipment5
William Edmonds, Jenelle Hirano, and Elizabeth Rodriguez-Johnson, National Defense, May 2009: 20.
This guest commentary discusses the Defense Department's new tool for acquisition programs that was designed to gauge the safety of weapons systems. It is called the "system safety metrics method" tool, or SSMM. SSMM can be especially useful for programs that are driven by an urgent need, such as the mine resistant ambush protected vehicle. The model would help to properly identify potential hazards to the operators of the vehicle — including overheating or rollovers. Other programs that would benefit from SSMM are commercial off-the-shelf procurements such as body armor. The equipment may be effective in protecting against ballistic threats or fragments, but that does not ensure it is safe for the user because a provider may not have tested or documented flammability or ventilation issues. Identifying safety vulnerabilities early in a program's life cycle is imperative for protecting troops and reducing preventable accidents. In addition, detecting safety weaknesses early on helps to save costs. The earlier problems are discovered, the better they can be addressed and remedied before the government invests heavily in development, testing and deployment.

Value Engineering and Services Contracts
Jay Mandelbaum, Ina R. Merson, Danny L. Reed, James R. Vickers, and Lance M. Roark. IDA Document D-3733, June 2009, 25p.
Value Engineering (VE) generates more than a billion dollars in savings and cost avoidance annually for the Federal Government. Most VE savings, especially those that are contractor initiated, are based on savings in acquisition of hardware. However, the government now predominantly spends its contract dollars on services. The document therefore identifies significant opportunities for using VE to save the government money in service contracts and to suggest ways to capitalize on these opportunities. The document also gives examples of the difficulties in applying VE to service contracts under the current Federal Acquisition Regulation (FAR) VE clause, and demonstrates that workarounds are possible. Some recommendations are made on actions that could be taken to improve the use of VE for service contracts.

Value Engineering Throughout a Defense System's Life Cycle
Danny Reed, Ph.D., and Jay Mandelbaum, Ph.D. Defense Acquisition, Technology and Logistics, May-June 2009: 52-59.
DoD policy recognizes the VE methodology as a systems engineering tool for making a significant contribution toward greater economy in developing, acquiring, operating, and supporting the products necessary to fulfill its mission. This article provides greater detail on the phases of the DoD acquisition process, the role of systems engineering within those phases, and the potential contributions the VE methodology can make to the systems engineering processes.

Systems of Systems and Net-Centric Enterprise Systems
Judith Dahmann, Ph.D., et al. Proceedings of the Seventh Conference on Systems Engineering Research (CSER 2009), Loughborough University, England, April 20–23, 2009.
This paper describes the characteristics of systems of systems (SoS) and net-centric enterprise (NCE) systems, examines their similarities and differences and highlights the implications for systems engineering and acquisition. The paper begins with a review of our current understanding of SoS and the implications for SE. It then looks at the characteristics of NCE systems and compares them with the characteristics of SoS. The paper closes with a discussion of the implications for systems engineering and acquisition.

Reliability, Maintainability, Supportability: Emergent Properties of Complex Adaptive Systems4
Robert M. Flowe, RMS Partnership Newsletter 13(2) (April 2009): 1-4.
This article discusses the complexity of interactions among the elements of joint capabilities which may confound traditional system- and program-centric acquisition management methods. The author contends that emergent properties such as capability-level reliability, maintainability, and supportability (RMS) are particularly susceptible to these dynamics. However, by changing the terms of reference from system-centric to explicitly considering the entities and their formally-defined interactions, we open the door to examining the emergent properties and dynamic behaviors from a network analysis perspective, and thereby can use these principles to gain some insight into potential emergent properties. In this way, the author feels that we can influence the behaviors of individual entities to support the desired Joint Capabilities objective with the underlying strategic RMS that these capabilities demand.

Business Models to Advance the Reuse of Modeling and Simulation Resources, Dennis P. Shea and Kenneth L. Graham, Center for Naval Analyses (CNA) Report, March 2009.
Today, modeling and simulation (M&S) is employed throughout the acquisition process and in almost every acquisition program, by government and industry, and is an integral part of military training programs and defense analyses. These efforts have produced a rich infrastructure of valuable intellectual resources, including models, simulations, databases, scenarios, threat libraries, verification and validation histories, environmentals, and others. Unfortunately, relatively few of the M&S resources developed in prior efforts are reused during the life cycle of an acquisition program or shared and reused by other programs, services, or organizations outside the original sponsor and developer. This paper addresses the reasons limiting reuse, investigates economic business models that could overcome these barriers and advance reuse, and provides recommendations for how the Defense Department could become a more consummate and savvy consumer of M&S goods and services, including understanding its property rights in M&S developed by industry, negotiating to obtain best value for M&S investment dollars, and ensuring that future users will be able to discover, assess, and use the M&S from today’s investments.

Implementation of the Sustainment Key Performance Parameter and Related New Reliability, Availability, and Maintainability Policies in DoD Acquisition Programs4
Grant R. Schmieder and Gordon M. Kranz, RMS Partnership Newsletter 13(1) (February 2009): 1-4.
This article discusses the mandatory Sustainment Key Performance Parameter (KPP), Materiel Availability (denoted by AM), and two supporting Key System Attributes (KSAs), Materiel Reliability (RM) and Ownership Cost (OC) implemented in the Chairman of the Joint Chiefs of Staff (CJCS) Instruction 3170.01 series, Joint Capabilities Integration and Development System (JCIDS). The authors describe the Reliability, Availability, Maintainability, and Cost Rationale Report (RAM-C Rationale Report) Manual, currently in coordination, to assist requirement developers and program managers in performing the analyses and trade-offs required to implement the Sustainment KPP.

2008

Report on Systemic Root Cause Analysis of Program Failures
National Defense Industrial Association Systems Engineering Division in conjunction with Office of Under Secretary of Defense Acquisition, Technology & Logistics, Systems & Software Engineering, Arlington, VA, December 2008.
Since 2004, the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD(AT&L)), Systems and Software Engineering/Assessments and Support (SSE/AS) Directorate has been conducting Program Support Reviews (PSRs) for major defense programs to help identify and resolve program issues and risks; and ultimately improve the probability of program success. Through analysis of the PSR data, SSE/AS has identified systemic issues seen across Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAIS) that impede acquisition success. The National Defense Industrial Association (NDIA) Systemic Root Cause Analysis (SRCA) Task Group was formed to analyze the data and attempt to extract the lowest level root causes of program failures. The Group used information generated from SSE/AS’s analysis to derive a joint government-industry set of recommendations to address the systemic issues and improve the execution discipline of acquisition programs. Although the analysis focused on Acquisition Category I (ACAT I) programs, the results are scalable and can be applied to most acquisition programs.

Report of the Reliability Improvement Working Group, September 4, 2008
RIWG Transmittal Memo, September 4, 2008 | Final Report Volume I | Final Report Volume II (Appendices)
The Reliability Improvement Working Group (RIWG) was chartered by the Director, Operational Test and Evaluation and the Deputy Under Secretary of Defense (Acquisition and Technology), in February 2008, to implement recommendations by the Defense Science Board (DSB) documented in their report on Developmental Test and Evaluation (T&E) of May 2008. This report summarizes what the Components achieved during this period, and what remains to be done in order to fully realize the DSB recommendations. It is therefore a record of progress in steps taken by each Component and OSD, and a guide to what additional next steps could be taken by either a Component, after recognizing what have components have done and achieved, or by OSD.

A Model for Systems Engineering in a System of Systems Context
Judith Dahmann, Ph.D., et al. Proceedings of the Sixth Conference on Systems Engineering Research (CSER 2008), Redondo Beach, CA, April 4, 2008.
Systems engineering is a key enabler of defense system acquisition. Current Department of Defense (DoD) systems engineering policy and guidance focus on the engineering of new systems. At the same time, the defense environment is increasingly characterized by networks of systems which work together to meet user capability needs. Individual systems are no longer considered as individual bounded entities, but rather as components in larger, more variable, ensembles of interdependent systems which interact based on end-to-end business processes and networked information exchange. This paper presents a model of systems engineering which provides a framework for supporting the systems engineer in this systems-of-systems (SoS) environment.

Preliminary Observations on DoD Software Research Needs and Priorities: A Letter Report, Committee on Advancing Software-Intensive Systems Producibility, National Research Council, 2008.
The Committee on Advancing Software-Intensive Systems Producibility was appointed by the National Research Council (NRC) and convened under the auspices of the NRC's Computer Science and Telecommunications Board (CSTB) to assess the nature of the national investment in software research and, in particular, to consider ways to revitalize the knowledge and human resource base needed to design, produce, and employ software-intensive systems for tomorrow’s defense needs. This letter report provides preliminary feedback from the committee regarding its observations on Department of Defense (DoD) needs and priorities for software research as well as suggestions for a research agenda that would be executable within the DoD’s Science and Technology framework.

Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition, National Research Council Air Force Studies Board, 2008.
The ability of U.S. military forces to field new weapons systems quickly and to contain their cost growth has declined significantly over the past few decades. There are many causes including increased complexity, funding instability, bureaucracy, and more diverse user demands, but a view that is gaining more acceptance is that better systems engineering (SE) could help shorten development time. To investigate this assertion in more detail, the US Air Force asked the NRC to examine the role that SE can play during the acquisition life cycle to address root causes of program failure especially during pre-milestone A and early program phases. This book presents an assessment of the relationship between SE and program outcome; an examination of the SE workforce; and an analysis of SE functions and guidelines. The latter includes a definition of the minimum set of SE processes that need to be accounted for during project development.

1 Reprinted with permission of the IEEE Systems Council.
2 This is the pre-peer reviewed version of the following article: Neches, R. and Madni, A. M. (2013), Towards affordably adaptable and effective systems. Syst. Engin., 16: 224–234. doi: 10.1002/sys.21234, which has been published in final form at http://onlinelibrary.wiley.com/doi/10.1002/sys.21234/abstract
3 Reprinted with permission of the International Council on Systems Engineering (INCOSE).
4 Reprinted with permission of the Reliability, Maintainability, and Supportability Partnership (RMS/P).
5 Reprinted with permission of the National Defense magazine.
6 © 2016 International Test and Evaluation Association. Reprinted by permission.