THE TECHNOLOGY INVESTMENT DECISION MODEL, DOES IT WORK?

Katherine J. Owens
Paragon Professional Associates
Idaho Falls, Idaho

ABSTRACT

The word "model," when used in context of a system or theory, is defined as: 1 ) a tentative description accounting for all known properties, or 2) a preliminary pattern serving as the plan from which an item not yet constructed will be produced (American Heritage Dictionary, 2nd College Edition). Given these definitions, one could conclude that any unknown variable could negatively influence the overall effectiveness, accuracy, or usefulness of the model. Therefore, prior to accepting a model as a standard worthy of imitation, sufficient data must support the models validity in the application for which it is intended (i.e., the model must be validated against test data). This paper presents a case study of one such test. A test was conducted to determine if a market based decision methodology known as the Technology Investment Decision Model is a valid tool for managing technology development in the U.S. Department of Energy Mixed Waste Focus Area. This paper summarizes the origins of the model and the methodology for validating it' s application in the Mixed Waste Focus Area.

INTRODUCTION

In August, 1994, the United States General Accounting Office (GAO) issued a report to the United States Secretary of Energy summarizing the results of a management review of the Department of Energy (DOE). The objective of the review was to evaluate the internal and external barriers inhibiting the use of innovative technologies in DOE environmental cleanup operations. The GAO found that, "although DOE has spent a substantial amount (of money) to develop waste cleanup technology, little new technology finds its way into the agency's cleanup actions ..... even where new technology has been successfully demonstrated, agency officials are reluctant to try new approaches..." (1)

The following problems were identified as contributing to this finding: 1 ) innovative technology is not being used to clean up contaminated sites, 2) many barriers limit the use of new and innovative technology (discussed later in this paper), 3 ) DOE program offices are not working together effectively, and 4) flawed decision-making process. The GAO review team acknowledged the new strategies implemented by the DOE Office of Environmental Management (EM)^a to correct coordination problems and eliminate duplication of effort, but found that," insufficient emphasis is given to ensuring that all parties - at the level where decisions are made - are knowledgeable about the strengths of the technological innovations being studied" and recommended the following: "1 ) fully involve regulators and other stakeholders in making decisions at the local level about the technology to be selected, and 2)formally include OTD^b staff in the evaluation and selection of technologies to be used to clean up DOE sites." (2)

In response to those recommendations the Deputy Assistant Secretary of the EM Office of Science and Technology (OST), initiated several corrective actions, one of which was the development of a process to appropriately link technology development with DOE cleanup operations. The underlying premise of this objective is a customer oriented approach to innovative technology development involving regulators^c and stakeholders^d.(3)

BACKGROUND

The Technology Investment Decision Model

In 1994, a team of individuals from government and industry convened to examine business practices associated with successful product development and to incorporate those practices into a model for guiding innovative technology development within DOE. The product of that endeavor is called the Technology Investment Decision Model (TIDM).^e The model's utility is described as, "a framework for addressing market factors and transfer issues as early as possible in the technology development process," and is intended to help decision makers in the DOE Office of Environmental Management to better manage technology development programs, thereby expediting the use of innovative technologies. (4)

The TIDM differs from traditional DOE decision processes in that it provides a holistic approach to technology development. As shown in Fig. 1, the model presents a staged process for technology development with several distinguishing features. First, the model is structured around a concept known as the "Innovation Cycle" which is defined as "that portion of the life of a product that precedes its manufacture on a full scale," (5) and consists of several stages of development beginning with basic research and ending with full scale demonstration. The transition between each stage is designated as a "gate." At each gate, a set of decision points or criteria determines whether a technology may proceed to the next stage of development. Second, the model incorporates non-technical factors as part of the decision criteria. These non-technical factors, such as market niche, return on investment, regulatory/stakeholder acceptance, and environmental, health and safety, are known to be the major factors influencing technology implementation. (6) These non-technical factors allow for a broader range of participation in criteria development and evaluation. Third, the model advocates "life cycle management." Life cycle management provides a mechanism for defining the full spectrum of customer requirements (inputs), activities (functions), and products (outputs). Criteria definition and progress assessment is accomplished through cross-functional (horizontal) and interdepartmental (vertical) cooperation.

The TIDM design team claims that the process better aligns EM technology development activities with the needs and requirements of the customers within the DOE user community by providing a mechanism for integrating "EM technology development efforts with it's cleanup operations and other participants in the technology maturation (innovation) process, e.g., industry, tribal/government regulators, and concerned citizens." (7) The model also serves as a tool for R&D managers to conduct comprehensive assessments of relevant risk factors (e.g. regulatory, economic, social, political, etc.), and to anticipate at what stages in the development cycle specific resources are needed (e.g., staffing, equipment, facilities, and expertise). The result being a process facilitating earlier customer involvement in defining product requirements in all stages of the development cycle. Furthermore, the model implies that earlier customer involvement in product life cycle planning results in better-cost benefit analyses and more prudent investment decisions, thereby, expediting the transfer of innovative technologies. However, at the beginning of fiscal year 1996 (October 1, 1995), there was no documented evidence supporting these claims.

Although the TIDM was developed in 1994, there was no indication that any of the EM Technology Development Programs had actually implemented the model. Several concept papers were drafted suggesting the methodology for implementing the model, but no real test of those concepts is documented. ^f Then in the Fall of 1995 two events in the Mixed Waste Focus Area (MWFA) provided the needed impetus for testing those concepts: 1) Implementing a technology development process borne of the TlDM was a dear expectation of OST management. (8) Since the MWFA is on a compressed schedule, in order to realize any of the benefits advocated of the model's utility, the product of its design needed to be in place by the start of the next fiscal year, October 1, 1996. 2) Faced with impending funding cuts, compressed development schedules, and increased political and public scrutiny, technology development managers in the MWFA needed a systematic and defensible decision process that would enhance the transfer of innovative technologies to the end-user.

In January 1996^g the MWFA approved a task to investigate the validity of the TIDM as a tool for managing the development of innovative technologies for the treatment, storage, and disposal of mixed wastes. This tool would serve as the framework for a systematic decision process to manage, evaluate, and transfer innovative technologies to the end-user. This paper describes the methodology used in developing that decision process and how the model was proven to be a valid management tool for the MWFA.

THE PROBLEM

Since there is no evidence of the model' s implementation in DOE technology development programs, the models utility as a management tool within the operational construct of the MWFA was unknown. So in early 1996, a team was convened to carry out the task of determining if the TIDM could be used as a framework upon which to develop a systematic and defensible decision process for technology development in the MWFA. The team members^h provided expertise in the major disciplines advocated in the model - technical, regulatory, financial/business, and communication/education. Systems engineers provided guidance and services in conceptualizing complex systems, integrating parallel pathways, and defining terms and requirements. A preliminary assessment of the TIDM revealed that it isn't the methodology and design of the TIDM that needs validation, but rather the architecture for its implementation.

THE HYPOTHESIS

If the reality of the MWFA system architecture can perform the functional elements of TIDM while maintaining the model's essential principles, the mechanism for the model's implementation should be revealed. Once this mechanism is known, the architecture for a technology decision process can be developed, and its application can be articulated through some form of written guidance.

METHODOLOGY

Given the complexity of the MWFA organizational structure and the diversity of the technology development projects under its purview, a systems engineering methodology was utilized. The reference for this methodology is the DOE Life Cycle Asset Management Guidance (DOE Order 0 430.1). Definitions of key terms are provided to aid in nomenclature used in this discussion. Engineering is, "the activity that implements the planning guidance based on management direction." Systems engineering is, "the integration of principles, processes, and products for planning, management, engineering, specialty integration, and reviews through the identification of functions, requirements, and architecture." Architecture is defined as, "a name given to the conceptualization of an end-product, wherein all functions are performed and all requirements (principles) are met. A function is, "a task that must be done." A requirement "must be measurable and states how well an architecture must perform a function." 9 So, to prove the hypothesis: 1) define the requirements, 2) determine the functions, 3) identify the engineering mechanism, 4) develop the architecture, 5) verify that requirements are met, and 6) write the guidance for the system.

Step 1. Define the Requirements of the TIDM

The TIDM advocates four key principles that must be maintained when implementing the model:

• Developers need to understand and address the needs and dynamics of the marketplace early in the innovation process.
• Decision Criteria must encompass both technical and non-technical factors.
• Formal decision points should provide the mechanism for determining investments in selected projects.
• Decisions should reflect an Environmental Management Research and Development Strategy.

The challenge for the MWFA is assuring that all four principles are preserved regardless of how the model is adapted and identifying the functional elements in the MWFA system architecture that satisfy those principles.

Step 2. Review the MWFA System Architecture

To assist the team in gaining a better understanding of the various decision pathways in the MWFA organization the system engineers developed a logic diagram conceptualizing how the MWFA will meet its mission. The relationship between the program's inputs (sources, requirements, criteria, etc.), functions (activities and actions), and outputs (products, deliverables, results) was discussed.

Step 3. Develop a Glossary of Terms

During the review of the MWFA System Architecture a general disagreement prevailed on the interpretation of the MWFA mission. Many saw the MWFA simply as a technology provider wherein individual technologies are developed to meet an identified need; others saw the program as a provider of technology components wherein technologies are developed to fill gaps in larger systems; and still others saw the MWFA as a provider of technology systems wherein a fully functional system is developed for a specific purpose. In fact, each interpretation is correct. The systems engineering approach dictates that development must support the end-use, therefore, customer requirements will dictate the design, schedule, and complexity of the development effort. Also, there was general disagreement of the interpretation of the term "demonstration," another important feature of technology development in the MWFA. Some viewed demonstration as the test of a full scale operational system, while others interpreted the term to mean any type of test to prove the technology' s ability to meet a set of requirements, regardless of scale. Since the systems engineers have a keen perception of the criticality of accurately defining requirements toward achieving expected results, they provided assistance in the development of a MWFA Glossary of Terms. Until the program could communicate in consistent language, defining accurate requirements was impossible. A glossary of terms was drafted and distributed throughout the program for review and comment and after several iterations the MWFA Glossary of Terms was finalized.

Step 4. Overlay MWFA System Architecture onto TIDM

A comprehensive analysis was performed on the TIDM to assess the model' s compatibility with the MWFA system architecture. Since the design of the model mirrors the systems engineering process practiced by the MWFA it was deemed an appropriate framework for the program's technology decision process. Other functional elements of the TIDM (listed below) were modified to better depict the MWFA system.

• Development Stages - One major modification was made to the TIDM development stages. The Demonstration Stage was eliminated, as were references to "scale" as a criterion in the other stages. The MWFA holds the position that "demonstration" per se, can occur in different stages at various scales. Marketability and full system functionality is demonstrated at whatever scale necessary in the Engineering Development Stage.
• Stage/Gate Expectations - The gate expectations as presented in the TIDM were deemed appropriate but additional expectations were added at the later stages to include customer procurement strategies and customer/funding commitments. A summary of the MWFA gate expectations is presented below: 10
• I - Basic Research (Pre-MWFA) - is the development stage during which fundamental research is undertaken to build core scientific or engineering knowledge; no specific need, end-user, or application of this knowledge is necessary.
• 2 - Applied Research - is the development stage during which knowledge gained from basic research is applied to problems related to potential end-user needs, and practical application of this knowledge is identified and agreed to by the MWFA.
• 3 - Exploratory Development - is the development stage during which one or more alternative technologies, methods, or devices are evaluated for potential application to a specific problem. Work performed during this stage results in defined application concepts.
• 4 - Advanced Development - is the development stage that focuses the technology to a specific application resulting in technology specifications, more detailed performance requirements, proof of practice, and sufficient detail to provide a basis for cost-benefit analysis. Funding is shared by the customer, end-user, or commercial partner.
• 5 - Engineering Development - is the development stage where prototype equipment is built to test design features and performance limits. Bench scale or pilot scale systems may be built and RCRA treatability studies are conducted independent of the physical scale of the equipment. Technology (but not necessarily an operating system) is ready for the end-user with test data providing performance and cost data able to support the conceptual design and/or capital for full scale production. Funding is shared by the customer, end-user, or the commercial partner.
• 6 - Implementation (Post MWFA) - is the post development stage when the end- user or commercial partner has assumed full ownership and utilizes the technology for its intended purpose. During implementation the end-user has a fully functional system where operating permits have been obtained and regulatory/tribal and stakeholder issues have been resolved.
• Transition Criteria - "Go/No-Go" Decision Points
• to Paladino, "The absolute key to the effectiveness of the model remains with the establishment of criteria...that the use of well-articulated criteria formed through a consensus process will result in the vertical and horizontal alignment of expectation." (11) The MWFA agrees with this assessment. The draft MWFA transition was distributed to a broad representation of stakeholders including, the MWFA Waste Type, Management, and Technical Teams, Tribal and Stakeholder Involvement Steering Committee, Environmental Management Site Representatives, and the Idaho Site Team for the Development of On-Site Innovative Technologies (DOIT). Fig. 2 illustrates the technology development transition criteria as modified by the MWFA.

Information and Documentation Requirements

This activity proved to be the critical juncture toward solving the problem. As the information requirements were being defined for evaluating the transition criteria, a discussion ensued as to how that information is documented and managed. Interface Control Documents (ICD) are the design and management tool formalizing an agreement between two or more entities having resources that functionally and physically connect. For complex projects, interfaces are often defined through the use of Interface Control Working Groups (e.g. , MWFA Waste Type Teams), wherein, representatives from both sides develop documentation that establishes the requirements to which the interface will be designed and developed. ICDs may either be a drawing, document, or standardized form. (12) A review of the MWFA Systems Architecture revealed the Interface Control Documents (ICD) for, the technical requirements of the project (e.g. Technology Development Requirements Documents, and MWFA Integrated Master Schedule), the non-technical requirements of the project (e.g. Records of Regulatory and Public Involvement, Systems Studies, etc.), and overall project performance (e.g. Technology Performance Reports). But what the team didn't see was the project development and demonstration pathway leading to the final technology review, or the ICD for coordinating the various inputs to the project. Therefore, an ICD called the Technology Development Plan (TDP) was designed to coordinate the information from all the input sources, including the TDRD and supporting studies and analyses, and describe the integrated life-cycle planning for a specific technology development. The TDP contains descriptions of development activities, performance objectives, schedules, transition and success criteria, and life-cycle cost, and integrates the activities of multiple contributors over several funding cycles. The TDP is the engineering mechanism for the MWFA to manage a portfolio of technology development options. (13) It is through this document that the functions and principles of the TIDM are integrated into the MWFA system. Fig. 3 illustrates the MWFA System with the TDP and the Project Development and Demonstration pathway. The pivotal role of the TDP sets it apart as the link between program activities and project activities. Table .I illustrates the relationship between the MWFA development stages, documentation requirements, and responsible party.

Table I. MWFA Technology Development Document Responsiblility Matrix

• Gate Procedure
• discovery of the engineering mechanism for implementing the technology decision process provided the architecture for adapting the TIDM to the MWFA. Fig. 4 illustrates the MWFA gate procedure or "project review cycle," and shows the relationship between the ICDs including the Technology Development Requirements Document, System Studies, Technology Development Plan, Test Plans, Test Results, Regulatory and Public Involvement Plans, Market Analyses, and the Technology Performance Report. The functional elements connecting these products comprise the MWFA Project Review Cycle.

Step 5. Identify how the MWFA Preserves the Principles of the TIDM

The architecture for the MWFA Project Review Cycle shows how each of the four essential principles of the TIDM are preserved.

• Developers need to understand and address the needs and dynamics of the market place early in the innovation process.
• discussed earlier in this paper the DOE is recognized as the primary market for the MWFA. Since the MWFA is governed by the conditions set forth in the FFC Act Site Treatment Plans (STPs), the information provided in the STPs, the Site Needs Assessments, and the MWFA Integrated Master Schedule comprise the MWFA Technology Development Requirements Documents. The TDRDs along with the other site integration activities (e.g. regulatory and public involvement) feed into the Technology Development Plan which establishes the development and demonstration parameters of every technology selected for development. The TDRDs and the TDP are revised after every project review cycle to reflect any changes in the market.
• Decision Criteria must encompass both technical and non-technical factors.
• TDRDs include technical and non-technical requirements as delineated in the Site Treatment Plans. In addition, the MWFA external liaison staff conducted site surveys verifying the non-technical requirements and incorporated those requirements into the language of the TDRDs. The decision criteria in the MWFA Project Review Cycle were developed with guidance from regulators, tribal members, financial/business experts, and other interested stakeholders. Those non-technical requirements that cannot be translated into quantifiable measures for the TDRD are addressed at the project level (site level) and are qualified through the various interface activities that are outlined in the TDP. To assist in that translation a crosswalk between non-technical issues and technical requirements was developed. I
• Formal decision points should provide the mechanism for determining investments in selected projects.
• MWFA Project Review Cycle transition criteria establish the "go/no-go" decision points at every gate review. If a project is unable to meet those criteria, or the data is incomplete, the project is re-scoped if the requirements can be revised, if not, the project is terminated.
• Decisions should reflect an EM R&D investment strategy.
• level MWFA/DOE program strategy is integrated with project development strategy via the Technology Development Plan (TDP). The TDP(s) reflect the MWFA investment strategy and are presented at the Environmental Management Internal Review Board for approval every fiscal year. As these strategies change over the course of the technology development process, the TDP and subsequent activities are revised to reflect those changes.

Step 6. Write MWFA Technology Development Transition Guidance

Since the MWFA Project Review Cycle performs the functions and satisfies the principles of the TIDM, the team was confident that it represents the product they were chartered to deliver. Although the product was completed, the task was not. The key mechanism of the MWFA Project Review Cycle is the Technology Development Plan, but since it was not part of the original MWFA system architecture a document was drafted to provide the basic criteria from which to prepare the TDP and a description of the decision process within which it functions. The name of this document is called the Mixed Waste Focus Area Technology Development Transition Guidance (INEL- 96/0424). The final document was published by Lockheed Martin Idaho Technologies Company and implemented by the MWFA in October 1996.

CONCLUSIONS

The scientific method was used to prove the hypothesis that the key to implementing the Technology Investment Decision Model does not rest in its design, but rather in the architecture that performs its functions and satisfies its principles. The systems engineering process was used in adapting the model to the Mixed Waste Focus Area, revealing a critical gap in the MWFA System Architecture that once filled, proved to be the engineering mechanism for performing the functions and integrating the principles of the TIDM. That engineering mechanism, now known as a Technology Development Plan links technology requirements to technology performance in the MWFA Project Review Cycle (Technology Decision Process). The Mixed Waste Focus Area Technology Development Transition Guidance is the document that provides the basic criteria for developing Technology Development Plan, and describes the framework upon which to evaluate technology development progress. This guidance document is patterned after the architecture of the TIDM, thus proving that the model is valid as a management tool for technology development in the U. S. Department of Energy.

ACKNOWLEDGMENTS

The author wishes to thank the other members of the team for their dedication, tenacity, and perseverance especially during those times of adversity. A special thank you is extended to the MWFA systems engineers for shedding light on systems theory and the meaning of a requirement. My appreciation is extended to Dr. Paul Wichlacz for sharing the vision that guided my way, and to Ms. Susan Prestwich for her boundless enthusiasm and encouragement. And finally, a personal sense of deepest gratitude is preserved in the memory of the late Christine Bonzon whose insight and convictions helped shape the development of this study.

LITERATURE CITED

1. U.S. General Accounting Office, A Report to the Secretary of Energy - Management Changes Needed to Expand Use of Innovative Technologies, GAO/RCED-94-205, p. 1 (August 1994).
2. Ibid. pp. 9-10
3. U.S. Department of Energy, A New Approach to environmental Research and Technology Development at the U.S. Department of Energy, Action Plan, (January 1994).
4. PALADINO, JOSEPH and PAUL LONGSWORTH, A Decision Model for Technology Development in the Department of Energy's Environmental Clean-Up Program, U.S. Department of Energy, Office of Environmental management, Office of Technology Development, p. 2, (1995).
5. GARRETT, LEONARD, J., and MILTON SILVER, Production Management Analysis, 2nd Edition, Harcourt, Brace, and Jovanovich, New York, N.Y., p. 88, (1973).
6. National Environmental Technology Applications Center, Barriers to Environmental Technology Commercialization. Prepared for Burns and Roe Services Corporation for presentation to the U.S. Department of Energy, (April 1995).
7. See Number 4.
8. U.S. Department of Energy, Environmental management Office of Science and Technology FY96 Program Execution Guidance, (1995).
9. U.S. Department of Energy, Life Cycle Asset Management (LCAM) DOE Order 0 430-1 - Project Execution and Engineering Management Planning Guide (Draft), p. 50, (August 1995).
10. Mixed Waste Focus Area Technology Development Transition Guidance, INEL-96/0424, Lockheed Martin Idaho Technologies, Prepared for the U.S. Department of Energy, pp. 12-17, (October 1996).
11. See Number 4, p. 11.
12. See Number 9, pp. 64-65.
13. See Number 10, p. 2.

a Originally established as the Office of Environmental Restoration and Waste Management. The name was changed in 1994.

b Office of Technology Development. The name was changed to the Office of Science and Technology in 1995.

c Recent legislation grants regulatory authority to Indian Tribes. References to regulators also includes Indian Tribes.

d For the purpose of this paper the term "stakeholder" is defined as any person, group, organization having a vested interest in the outcome.

e Currently acknowledged by DOE as the "Technology Decision Process," it is also known as the "Technology Gates Model"," Paladino Model", "Paladino Gates ", and "Stage/Gates Model."

f As of this writing, Revision 6 of the OST Standard Operating Procedures (Dec. 1996) includes a "preliminary procedure" for implementing the Technology Decision Process.

g This task was originally proposed in March 1995 and was included in the official FY96 Program Execution Guidance for the MWFA, however, congressional continuing resolution of the FY96 federal budget delayed funding until Jan. 1996.

h Besides the author, the team members were, Jim Herzog and John Belief of Lockheed Martin Idaho Technologies Company and Peggy Hinman of the law firm Carter, Brock, and Hinman. Systems engineering support was provided by Don Norman and George Beitel of Lockheed Martin Idaho Technologies Company.