Abstracts for TMC 2008
The F-35 Lighting II Program
Dr. Jim Alper, Joint Strike Fighter Program
The F-35 Lightning II Program (also known as the Joint Strike
Fighter Program) is the Department of Defense's focal point for
defining affordable next generation strike aircraft weapon
systems for the Navy, Air Force, Marines, and our allies. The
F-35 is the next generation strike fighter bringing cutting-edge
technologies to the battlespace of the future. The JSFs advanced
airframe, autonomic logistics, avionics, propulsion systems,
stealth, and firepower will ensure that the F-35 is the most
affordable, lethal, supportable and survivable aircraft ever to
be used by so many warfighters across the globe. As the program
plans for Follow on Development, we continue to seek
technologies that will improve affordability and enhance our
current capabilities.
Achieving Better Acquisition
Outcomes: GAO’s 2008 Assessment of Weapon Programs
Ms. Cheryl K. Andrew, U.S. Government Accountability Office
DOD’s investment in weapon systems represents one of the largest discretionary items in the U.S. budget. The department expects to invest about $900 billions (fiscal year 2008 dollars) over the next 5 years on development and procurement with more than $335 billion, or 37 percent, going specifically for new major weapon systems. Over the past 6 year, GAO has reported on individual programs as well as many crosscutting problems with the acquisition process and has offered numerous recommendations on how DOD could improve acquisition outcomes. This presentation provides information on the technology, design, and manufacturing maturity of 72 weapon programs and assesses overall trends in DOD acquisition outcomes.
Using the Advancement Degree of
Difficulty (AD2) as an Input to Risk Management
Mr. James W. Bilbro, JB Consulting International
The Advancement Degree of Difficulty (AD2) is a concept for
developing an understanding and quantifying the difficulties
associated with maturing technology to a point where it can be
successfully incorporated into a program within available cost,
schedule and performance constraints. AD2 is not a substitute
for formal risk assessment, but it is intended to provide early
insight into the critical areas for which formal risk and cost
assessments should be performed. The presentation relates the
output of the AD2 to the concepts of variation, foreseen
uncertainty, unforeseen uncertainty and chaos as developed by De
Meyer et.al. and to the 5x5 risk matrix use in formal risk
assessment processes.
The Use of TRLs in an IRAD and MTEC
Program
Mrs. Melissa C. Butler, Northrop Grumman Corporation,
Intelligence Group (TASC)
The Northrop Grumman Corporation Intelligence Group (TASC)
Independent Research and Development (IRAD) and Methods and
Tools for Enhanced Capabilities (MTEC) program began to use
Technical Readiness Levels (TRLs) in September 2007 with the
approval of the Intellectual Property Capture Process. This
document is used to define what TRLs are, how they are used on
the program, and what documentation and artifacts are required
at each TRL. The definitions of the TRLs used are directly
derived from the U.S. Department of Defense (DoD) Technology
Readiness Assessment (TRA) Deskbook.
The TRL ratings are used in combination with weighted Technical
Transactions as a method to help determine the overall technical
progress of a project. IRAD Technical Transactions are defined
as “a communication with a person or group outside of the
project with the objective of promoting the project for future
collaboration or contract agreements.” MTEC transactions are
defined as a “communication with a person or group outside of
the project with the objective of creating or enhancing our
policies, procedures, and tools in order to make significant
improvements in our operational effectiveness and efficiency.”
All transactions are counted on weighted scale ranging from .5
to 2.0 depending on the impact the transaction has on the
receiver.
Currently there are 45 projects in the IRAD and MTEC program,
with focuses on hardware, software, and process improvement. The
TRL for each project is determined by the Principle Investigator
(PI) of the project with input from the CTO and the CTO Council.
The TRL and Technical Transactions are collected and
consolidated on a monthly basis and the data is used to create
“Bubble Charts”. There are 3 Bubble Charts, Information IRADs,
Hardware and Systems IRADs, and Process and Tool MTECs. The
Technical Transactions are considered to have a 6 month
relevancy. Therefore, after 6 months, the first month’s
transactions are removed and the seventh month’s transactions
are added. Each bubble in the chart represents one project. The
size of the bubble is relative to its funding level and the
color is the overall status as determined by the PI. See example
below.
The IRAD and MTEC program runs on a yearly cycle, with each
project required to submit a Final Report, including a final TRL,
at the end of each calendar year. Each project wishing to
continue into the next year must also submit a proposal
describing the updated goals of the project. A mid-year call for
proposals also takes place; which is used to bring in new
projects to replace those which have been removed from the
program. Projects may be removed from the program if steady
progress is not being made; this determination is made in part
due to the projects TRL and number and quality of Technical
Transactions.
In summary, the use of TRLs in combination with Technical
Transactions has allowed the IRAD and MTEC program to begin a
quantitative measurement program which is now used to help
determine the overall success of the individual projects and the
program as a whole.
Return to Agenda
Defining Integration Readiness Levels
and Its Application in the Capability Based Acquisition Process
Dr. Fereshteh Farzad, Booz Allen Hamilton Inc.
Technology Readiness Level (TRL) has been successfully utilized
in the technology development as a standard tool in the defense
acquisition process. TRL is an excellent measure of the maturity
of individual technologies, but it lacks appropriate metrics to
evaluate the challenge of the components or subsystems
integration into an operational system. Inefficient system
integration readiness has led to significant program
difficulties for decades. In order to unify the process of
system integration, a system focused metric is needed to
complement the technology readiness assessment process.
We propose a maturity scale, Integration Readiness Level (IRL),
to accurately assess the progress of integration steps for a
system under development by providing a clear sequencing for
different development activities throughout the program
life-cycle. IRL evaluates the current and future status of a
system for delivering the anticipated program’s mission
capabilities defined in its Capability Development Document (CDD)
and the Concept of Operations (ConOps). IRL measures the level
of system’s capabilities completion process and could vary for
different mission capabilities. It highlights the critical risks
items and can be applied to systems with various levels of
complexities and systems-of-systems. An integration readiness
level of a system provides insight over the system status and
enables timely decisions for eliminating programmatic risks. IRL
could prove to be a basis for a reliable and effective
assessment tool to guide future system integration activities
and acquisition process.
In this presentation we will describe the levels of the IRL
scale and discuss the results of its implementation to a virtual
case study
Return to Agenda
Technology Maturity and Technology
Transition; Relationships and Best Practices
Mr. Lance Flitter, Naval Surface Warfare Center, Carderock
Division
There has been a great deal of work and thought put into metrics and approaches to technology maturity assessment in recent years. While there is a general understanding of the value of technology maturity assessment within the context of the acquisition of systems, the broader context of how technology maturity assessment relates to another key objective of DoD and federal government, technology transition, is frequently not addressed. This presentation will consider technology transition, what it is and challenges and best practices for achieving transition. Within this context the role of technology maturity assessments will be considered with the goal of addressing the key questions, how do technology maturity assessments relate to and support technology transition?
Software Technology Maturity is NOT
Software Product Maturity
Dr. Suzanne Garcia, Senior Member, Technical Staff SEI/CMU
Software Technology Readiness Levels (TRLs) are promoted as a valuable way of characterizing the technology risk associated with software technologies being used in mission-critical systems. TRLs - as they are currently defined - are distinct maturity plateaus and represent phase gates/end-criteria for technology development life cycle phases. However, particularly at higher TRLs, the technology development life cycle can be easily confused with the product development life cycle, because the demonstration of such TRLs expects the availability of a gradually evolving objective system.
The Program Manager's central concern is to determine if the
evolving software PRODUCT is mature enough to keep the program
risk reasonable, including technology risks. Unfortunately,
confusing software technology maturity with software product
maturity creates its own set of problems. This presentation will
provide some reasoning mechanisms to help identify when this
conflation is happening, and suggest some approaches to keeping
the attention on software technology where that is the
appropriate focus.
DOE-EM Technology Readiness
Assessment (TRA) Update
Mr. Kurt Gerdes, U.S. Department of Energy, Office of Waste
Processing (EM-21)
The Department of Energy (DOE) Office of Environmental
Management (EM) will update its progress on the application of
the TRA/TMP process to its nuclear waste clean-up program. DOE-EM
will also describe the integration of the process into its
project management structure and the impact the process has had
on project cost, schedule, scope, and execution. It will also
provide insight as to how the process is viewed by upper DOE-EM
management and external stakeholders. DOE-EM is also working
with the United Kingdom’s Nuclear Decommissioning Authority (UK
NDA) to collaborate on a TRA to be conducted at Sellafield in
the fall of 2008.
Return to Agenda
From Advanced Concept Demonstrator to
Transformational Weapon System: A Short History of the Predator
Unmanned System
Mr. Frank Grimsley, Director of Engineering, 303rd
Aerospace Wing
The purpose of this presentation is to present the developmental history of the Predator Unmanned System (UAS). The presentation will discuss some the successes/problems associated with the rapid fielding of the system. The problems include both technical and programmatic issues such as lack of test assets and incomplete product definition. In addition, the presentation will discuss issues associated with “product ionizing” an Advanced Concept Demonstrator and an improved process to address those issues. The main issue concerns the lack of airworthiness requirements (especially for UAS) in the use of company developed aircraft. The presentation will also discuss some of Aeronautical Systems Center’s efforts to improve the situation and further work that is required. In spite of all these issues, the Predator has become one of the most transformational weapon systems ever produced.
DOE-EM Technology Readiness Assessment
(TRA)/Technology Maturation Plan (TMP) Process Guide and Plans
for TRA Training
Dr. Steven Krahn, U. S. Department of Energy
In 2006-2007 the Department of Energy (DOE) Office of
Environmental Management (EM) piloted more than a half dozen
Technology Readiness Assessments (TRAs). The TRA methodology was
adapted from the DoD Deskbook and utilized William Nolte’s TRL
calculator modified to fit radioactive waste processing. Based
on the pilot TRAs and follow-up Technology Maturation Plans,
DOE-EM has developed the “Technology Readiness Assessment (TRA)/Technology
Maturation Plan (TMP) Process Guide. This paper will describe
the TRA/TMP Process Guide (issued in March 2008), its
development, and the structures and processes DOE-EM has put
into place to carry out and provide oversight for future TRAs.
The paper will also describe progress on a training course being
developed for TRA leaders and participants and plans for future
development in the areas of ground water processing and waste
processing software.
AFSO21 / D&SWS / Tech
Development: Air Force Initiative – High Confidence Technology
Transition Planning Through the Use of Stage-Gates (TD-13)
Dr. Claudia V. Kropas-Hughes, Deputy Chief, Technology
Transition Division, AFMC/A5S
Since July 2006, the Air Force has been undergoing
transformation to reduce costs and streamline processes
throughout the Air Force. In these hard fiscal times, it is
necessary to ensure that every “blue” dollar is spent in the
best interest of the warfighter. A key problem to be addressed
was highlighted in a September 2006 GAO report entitled
“Stronger Practices Needed to Improve DoD Transition Processes”.
This report states that immature technology often drive up
weapon system costs. Specifically, the GAO recommends a rigorous
“gated-maturity” review process that will “...ensure a
technology’s relevancy and feasibility and enlist product line
commitment to use the technologies once the labs are finished
maturing.”
The Air Force initiative “High Confidence Technology Transition Planning Through the Use of Stage-Gates” is being implemented to address this key problem. The primary goals of this initiative are: 1) improve the transition planning process leading to an increased probability and speed of technology transition and, 2) improve the confidence in acquisition programs and direct transitions to service. The approach to achieve these goals and to better inform decision makers is based on industry and government Best Practices; specifically the use of a gated technology development and transition process that aligns and integrates with the Air Force acquisition process. The stage-gating development process is an approach to manage technology transition through predetermined stages - from idea to fielding. Each stage-gate is a point in the development process where the technology must pass pre-determined criteria to begin the next phase of development. This is a point at which go/no-go decisions will be made by the team or management.
SemanticTRL Part 1: The SemanticTRL
Community Approach for Collaborative
Portfolio Management Across the Global Information Grid (GIG)
Robert Kruse, VenLogic LLC
Since the President Signed the eGov Act of 2002 into Law, the GAO and Executive Branch Management Scorecard have tracked how well the Federal Departments and 28 Major Agencies have managed over $1.9B spent on eGov initiatives. Multiple directives emerged to set policy for the development of the Global Information Grid (GIG). Today, the GIG is a key backbone infrastructure for enabling the president’s vision for a more NetCentric, collaborative, sharing government. With IPv6 coming on this year, the GIG has the capability to integrate literally billions of sensors and links that “touch everything,” according to DISA.
To date, the GIG applications have been primarily focused on the
warfighter’s needs for data sharing. However, GIG’s semantic
architecture harnesses the potential to offer new collaborative
applications for enabling Labs and Customers to see inside of
each other’s pipeline, verify requirements, describe funding
needs, and alert transition readiness for more timely insertion
into Programs of Record (POR). Likewise, portfolio managers can
track their
investments, funding gaps, and co-funding partners easier and
faster across multiple agencies, providing critical leverage of
taxpayer dollars. The SemanticTRL enables managers to manage
both the funding process and the customer requirements process,
making better use of the pre-existing, built-in semantic savvy
GIG architectures and current standards.
By taking advantage of the existing, approved standards for semantic collaboration, investors, portfolio managers, lab program managers, program offices, MAJCOMs, PORs and industry can use the familiar TRL to more accurately and conveniently move technology through the transition process faster.
We will introduce the concept of a SemanticTRL as a mechanism for performing advanced collaboration and community-building services across the secure GIG and the Internet. By converting the traditional TRL into the more advanced Semantic design language of the GIG, a SemanticTRL can close the Tech Transition Gap between the labs, the Acquisition community and their warfighter customers. The semantic approach can also be used to synchronize the various maturity benchmarks very quickly, using the GIG framework, thereby allowing rapid changes in assessment criteria for TRL, MRL, etc to be shared and collaboratively updated in near real time across multiple agencies, industries, and countries. User TRL data, milestones, action plans and process models for tracking funding, cost, schedule and risk are now globally interoperable with AF-GCCS, NATO, and much more.
The TRL has yet to be integrated into a modern collaborative platform for net-centric use on the GIG until now. The SemanticTRL now offers the most promising advancement in collaborative assessment methodologies since the concept was introduced by VenLogic at TMC 2007. Managers of the TRL should consider ensuring that the TRL and the respective variants remain compliant with Semantic GIG Standards. SemanticTRL meets that requirement.
We propose to demonstrate the power of the SemanticTRL using a
simulated use case involving multiple government agencies,
touching nearly every organization represented at this
conference. Engaging a different, similar collaborative approach
using legacy relational databases would cost tens of millions in
IT design expenses over many years. We aim to show how using the
SemanticTRL with the GIG, users can experience enhanced
interoperability benefits for a fraction of the cost in much
less time, starting in Q4 2008. The future is now.
SemanticTRL Part 2: Readiness
Assessment Tool Synchronization using the
Collaborative SemanticTRL Community Approach
Since the TRL’s adoption by DoD in 1996, there has been an
expanding number of assessment tools for measuring maturity and
assessing risk. NATO, Canada, UK, DOD and DOE all use variations
of the TRL originally proposed by NASA. There are now
assessment tools for measuring Manufacturing Readiness (MRL),
Biotech Readiness (BRL), Logistics (LRL), Systems Integration (SRL),
Systems of Systems (SoS), Venture Capital (VMI), Business
Readiness (BRL), Software Maturity (CMMI) just to name a few.
While the MRL has succeeded in rising towards formal acceptance
alongside TRLs, so many other factors contribute to Tech
Transition Failure that are not being addressed by these two
narrowly focused tools.
Therefore, it’s no surprise that decision makers charged with
the task of performing risk assessments as part of their Tech
Transition Plans (TTP), Tech Transition Agreements
(TTAs), and/or Tech Readiness Assessment (TRA) reports seek
alternative ways to identify and report areas of potential risk
not identified by the TRL or MRL.
Many organizations use the highly referenced NASA scale and Nolte TRL Calculator for establishing a baseline within their Technology Readiness Assessment (TRA) report, and then justify their deviations from the baseline to articulate direct relevance to their situation. Thus a new TRL variant is born. A variant that is not only used to make continued investment decisions, but is not formally documented, not recognized outside of the immediate user group, and is not easily accessed for comparison to similar variants for consistency checking, citation, and improvement by the next user. Each new variant results in a certain level of “reinventing the wheel,” unnecessary rework that could be streamlined if the relevant data were made available.
The US Dept. of Energy (DoE) provides a case in point, where the DoD TRL originally used for hardware and software cases,was modified for use with Nuclear Waste cleanup (ref: Herb Sutter, DoE TRA SPD-07-195, July 2007). There is no easy way for DoE TRA Report analysts to quickly query DOD TRL criteria across multiple agencies and generate new criteria that is consistent with the family of TRL predecessors. Nor is it easy for DOE Nuclear Energy analysts to share their TRL derivative in a common format for DoD users to access at a future date so that others may save time learning from their best practices.
10 USC 2366a and the TRA Deskbook mandate the TRL’s use by law in the DOD 5000.1 acquisition directives. However, the law is not only difficult to apply in practice, but even more difficult to resolve discrepancies and enforce across multiple agencies. It appears that a “Family Tree” of TRL variants has extended beyond the control of the authorities that mandate its use. The TRL is essentially built upon the notion of “best practices” which continue to evolve faster than the guidelines for using them. Sorting out and reconciling the meaning behind the TRLs applied to a given use case quickly becomes an analytical exercise itself and a key contributor to the Tech Transition Gap, instead of accelerating Tech Transition as it was intended.
While the TMC conference is the only known collaboration environment for benchmark practitioners to share their views on maturity best practices across agencies, the conference sessions do not provide enough time to get into the details of synchronizing TRL 9 with MRL9 vs. MRL 10, for example. Since DoD has no provisions for an equivalent TRL10, this synchronization problem creates unnecessary confusion that delays technology seeking to be delivered to the warfighter.
This presentation will offer a novel means for developers and users of TRLs, MRLs, IRLs, LRLs, BRLs, SoS, VMI and every other maturity benchmark tool – to track, synchronize, collaborate upon, and cite their unique risk assessment methodologies across a distributed, Global Information Grid (GIG) on a daily basis, if they choose.
We propose to demonstrate the SemanticTRL applied to synchronizing the TRL and MRL assessment frameworks. We will show how a Semantically-enabled Community approach can be used by admins, investors, portfolio managers, developers, customers, and end users to provide necessary guidance, oversight, and mediation of problems that naturally arise in the development and implementation of organically derived TRL variants. The net result will provide evidence that more variants are better and the real solution resides in how the assessment frameworks are managed and evolved by the community over time.
Measuring Business Risk with
Business Readiness Level (BRL)
Mr. Scott Leonard, Northrop Grumman Technical Services
Bad business often kills good technology. Broad experience with
the SBIR, ManTech, and DPA Title III programs shows that science
& technology (S&T) development companies are often more
challenged by business issues than technology or manufacturing
issues. Unless the technology under development can eventually
be commercialized as a self-sustaining business (or business
unit), achieving TRL and MRL goals won’t be enough to transition
the technology to the warfighter.
The Business Readiness Level (BRL) tool adds a third dimension
to program risk management, supplementing Technology Readiness
Level (TRL) and Manufacturing Readiness Level (MRL) in the
program manager’s toolkit. The BRL tool defines twelve business
risk factors for which a company and their technology can be
assessed. Similar to TRL and MRL, the risk factors are
objectively scored by evaluators who are familiar with the
business and technology being assessed.
Unlike TRL and MRL, the twelve scores are reported individually
(rather than as one combined score) on a scale from 1-10. The
lowest scores show the program manager which business risks most
need to be managed. For programs that tolerate higher risk, such
as the SBIR and DARPA programs, low scores may just be “yellow
flags” to be monitored. More risk-averse programs (such as those
seeking near-term technology transition) may view low scores as
“red flags.” Red flags would encourage the program manager to
develop mitigation and/or contingency plans.
The BRL tool was developed under contract to the Air Force
Manufacturing SBIR Program managed by Air Force ManTech. The
Program is currently using this new risk assessment dimension to
address the critical business issues that could inhibit
technology transition to the warfighter. Although the BRL tool
has SBIR program origins, it is broadly applicable to any
Federal S&T development program that wants to manage business
risk associated with its contractors and their technologies.
This presentation describes the purpose of the BRL tool, the
twelve risk factors, and how BRL integrates with TRL and MRL as
a third dimension in the program manager’s risk measurement and
management toolkit.
Technology Readiness Assessments
for Systems of Systems
Dr. Jay Mandelbaum, Institute for Defense Analyses
This presentation describes why and how the Technology Readiness Assessment (TRA) process should be changed for a system of systems (SoS) and its associated systems engineering processes. There are two basic reasons behind the “why.”
- From a Critical Technology Element (CTE) identification perspective, the full set of CTEs for a system or the SoS as a whole, may not be known or knowable at the time a Milestone B TRA is conducted. CTEs must be considered tentative prior to completion of overall SoS system engineering and then individual system(s) engineering.
- Once a CTE is identified, assessing its maturity involves a demonstration in a relevant environment. Demonstrating (or even quantifying) the technical performance of an individual system needed to meet operational requirements may be problematic because of the difficulties in allocating SoS requirements to associated systems or subsystems.
The “how” is discussed for the following TRA activities.
- Describing the SoS
- Identifying the SoS environment(s) and interfaces
- Identifying SoS CTEs and their associated relevant/operational environments
- Conducting the SoS TRA
- Documenting and coordinating the SoS TRA
The presentation concludes with a discussion of SoS TRA
updates.
Return to Agenda
Technology Maturation Planning for the
Autonomous Approach and Landing Capability (AALC) Program
Mr. Robert McCarty, SynGenics Corporation
A multi-directorate Air Force Research Laboratory team has
demonstrated technologies for a manned aircraft autonomous
approach and landing capability (AALC). The focus of the AALC
demonstration was to pave the way for an Air Mobility Command
(AMC) capability of landing transportation aircraft in
visibility reducing weather conditions without ground based
navigation support. AALC will enhance base opening operations,
and expedite force deployments by using multiple airfields in
the objective areas unhindered by low ceiling and poor
visibility conditions. A Technology Maturation Plan (TMP) was
developed in order to address deficiencies uncovered during the
demonstration and to guide technology development required to
achieve an Acquisition Milestone B Decision for System
Development and Demonstration (SDD) leading to an Initial
Operational Capability for the Air Force. The TMP recommends
composition of an Integrated Product Team to enable successful
technology development for SDD. Requirements were generated
during the AALC demonstration to describe the capability desired
by AMC. These requirements were used to shape the TMP which
recommends a multi-phase approach. The first phase would involve
updating customer requirements as well as the follow-on team’s
knowledge of sensors, sensor fusion, displays, and related
technology maturity. The first step in the first phase would be
to conduct requirements and technology analysis using a Systems
Engineering Tailored for Science and Technology (SETFST)
process. The second phase would be to contract for the
technology maturation required to meet the requirements
generated in phase I. The third phase would be to use the
technologies matured in phase II to conduct a flight test
demonstration. The fourth and final phase would be to provide
the resulting information to the appropriate systems groups for
transition of the technology to operational aircraft. The AALC
TMP is structured to address three primary aspects of successful
technology maturation: Science and Technology (S&T) community
teaming with acquisition and sustainment program managers to
address user needs in order to guide development of technologies
required for successful transition, program office planning for
integration of transitionable technologies into their
acquisition programs, and overall planning for full integration
or bridging of S&T technology development planning into target
acquisition program strategies. Multiple dimensions of
successful technology maturity assessment are outlined in the
AALC TMP including identification of all key stakeholders,
thorough and traceable definition and documentation of
requirements and exit criteria, value analysis to enable
selection of technology options/products best suited to meet
defined requirements, identification of and engagement with
target acquisition programs, identification of other programs
that could contribute to AALC success, definition of major
technology development and maturation milestones, and
delineation of a strategy for risk analysis/management. The need
for key functional strategies is also indicated in the TMP to
address technical, business, financial, logistics,
manufacturing, and test aspects of technology maturation. This
presentation will describe the framework of the AALC TMP,
identify lessons learned in the AALC demonstration program, and
illustrate how the program will move forward to provide the
required warfighter capability.
Manufacturing Readiness Levels (MRLs)
and Manufacturing Readiness Assessments (MRAs)
Mr. James T. Morgan, AFRL/RXMT
Manufacturing readiness, like technology readiness, is critical
to the successful introduction of new products and technologies.
Manufacturing Readiness Levels (MRLs) represent a new and
effective tool for the S&T and acquisition communities to
address manufacturing risks. MRLs assess the manufacturing
maturity of a hardware-intensive technology or product, as well
as the level of manufacturing risk associated with producing a
weapon system using the technology. Although they were only
recently introduced, MRLs have already gained wide acceptance
throughout government and industry. MRLs have been developed to
be congruent with TRLs and to serve a similar purpose, which is
to provide a widely-accepted common language and measurement
scale to ensure weapon systems meet warfighter requirements. Mr.
Morgan will discuss the Air Force MRL implementation program,
describe the criteria being used and present some examples of
how MRLs can be used to guide development programs.
Successful (and not so Successful!)
Technology Transition on the C-17 Aircraft
Mr. Richard Reams, ASC/ENFS
This presentation will explore several aircraft structural
technology projects that were successfully transitioned from the
laboratory to production on the Boeing C-17 aircraft, and focus
on the specific reasons for the successful transition. The
presentation will also review other projects that did not
successfully transition into production and will elaborate upon
the inhibiting factors that prevented production incorporation.
For each structural project, an examination into the structural
/ materials technology involved, the maturity of the technology,
and the airworthiness certification steps required by the
program office will be made. Additionally, the principal factors
involved in bridging the gap between the laboratory and the
production floor and the lessons learned during the transition
process will be provided.
Return to Agenda
A Probabilistic Approach to Maturity Assessment
Dr. Brian Sauser, Stevens Institute
Since NASA introduced Technology Readiness Level (TRL) as a
contracting tool, it has been adopted by various agencies as the
metric for assessing the maturity of the critical technology
elements of systems under development (e.g. Department of
Defense (DoD), Department of Energy (DoE), National Air and
Space Intelligence Center). Over the years, the measurement of
TRL has evolved into a structured approach with the goal of
producing a quantitative value that accurately represents the
maturity of the technology. Currently, the most common methods
for determining a TRL are done by: (1) Individual Expert: an
expert in the technology or TRLs assesses its state of maturity;
(2) Group: the maturity is determined through a discussion among
the technology’s stakeholders; and (3) Assessment Tool: the use
of guidance documentation or a software tool that directs the
maturity assessment (e.g. the DoD Technology Readiness
Assessment Deskbook; the AFRL TRL calculator).
However, these methods of estimation are human-intensive and are
therefore subjective. The resulting TRL values are the product
of the trade-offs among several contractors or stakeholders. In
addition, for any decision making approach, important
considerations for judging its overall value have to be judged
again factors such as time, cost, nature of the problem, and
satisfaction and commitment to the collection, analysis and
results of the data.
Since different opinions always exist among individuals, it is
inevitable that variations will exist in the TRL estimates
produced by the evaluators. These variations can provide more
precise insights into the real maturity of the technologies,
which could not be captured by some unique and discrete values
that the methodologies above can produce.
To capture these insights, we apply a basic assumption - that
the evaluation of the TRL follows a probabilistic form. We
propose to incorporate the complete information that the
stakeholders provided by using the relative frequency of the TRL
values generated as a probability distribution to combine every
evaluator’s judgment of the readiness of the technology. That
is, the dispersion in the TRL estimates can be represented in
the values that are calculated so that the technology managers
can make more informed decisions vis-à-vis their tolerance for
and the magnitude of the risks associated with the project.
We will demonstrate this approach with TRL and then illustrate
how it can be applied to the evaluation of the integration of
technologies for determining a system’s maturity (i.e. a System
Readiness Level as described in A System Maturity Index for
Decision Support in Life Cycle Acquisition, 2007 Technology
Maturity Conference). We will then use case examples to
illustrate how this approach and the resulting analysis can be
used to make lifecycle decisions during system development.
Return to Agenda
A Quick-Look Design-of-Experiments
(DOE) Orientation
Mr. John Sparks, SynGenics Corporation
DOE is one of the core “Six-Sigma” methodologies used to
systematically analyze the nature and cause of measured
variation per controlled testing. Using DOE, one can select
“statistically rich” points in the design space that allow for
maximum information mining from the investment made. An example
of information mining is the linking of cause to effect by
establishing the coefficients for pre-determined ‘best-fit’
models such as linear models, piecewise linear models, or
general quadratic (response surface) models. Test programs built
upon sound DOE principles are efficient in that the program
itself tends to be compressed—saving both time and money. The
test programs are also highly effective in that the data
generated is multipurpose and supportive of a variety of
analysis techniques.
Return to Agenda
Mission Assurance through Improving
Software TRAs for NSS Systems
Mr. Daniel Winton, The Aerospace Corporation
Frequent instances of software development delays in National Security Space (NSS) systems indicate that systems software is a high risk area in any NSS development. This presentation is based on the premise that improving how systems software is addressed in Technology Readiness Assessments (TRA) improves the likelihood of success of NSS system acquisitions. It outlines an approach for better addressing systems software in TRAs by providing an alternative definition of software Critical Technology Elements (CTE) and providing a new systems software focused Technology Readiness Level (TRL) scale. It provides recommendations on how systems software development risk can be reduced and how the state of systems software readiness can be assessed. Further, identification of an integrated architecture encompassing both systems and software dimensions is foundational to the approach. Through adoption of the recommendations in this presentation, it is envisioned that systems software issues will be identified earlier in the acquisition lifecycle, allowing more time for correction and resulting in lower risk and higher probability of mission success.
Status of the Air Force’s Improved
Technology Maturity Assessment Project
Dr. Kyle Yang, MIT Lincoln Laboratory
As part of the Air Force’s Develop and Sustain Warfighting Systems (D&SWS) Process, the Technology Development (TD) subprocess has the goal of improving the efficiency with which the Air Force develops and fields new systems and technologies. Within TD, there is a project named, "Improved Technology Maturity Assessment, TD-1-12,” which seeks to establish metrics to measure and gauge technology maturity, using a more comprehensive set of criteria than typically covered using Technology Readiness Levels. Part of this includes the usage of Manufacturing Readiness Levels. However, the intent is also to account for integration issues, maintainability, and other "ilities." Methods developed through TD-1-12 are intended to be useful both for a program office's own efforts, as well as its ability to report upwards, such as at key milestones or even monthly reports. Development of the methodology has involved surveying available methods as well as gathering lessons learned from past projects. This briefing will present the methodology of the TD-1-12 team as well as provide a preview of the expected methodology.
 |
|
|
If you have any problems accessing these pages, send an e-mail to Jennie.Hempstead@gdit.com for help or call Ms. Jennie Hempstead at (937)254-7950 ext. 1150. Please specify in the e-mail which operating system you are using, which version of Windows you are using (if you are running Windows), and which type and version of browser you are using. This page is best viewed using Microsoft Internet Explorer 6.0 or Netscape 6.2 or later. The use of other browsers may result in incorrect formatting of the page. |