     
» Demos» Published Papers » Tech FAQ's        |
| PUBLISHED PAPERS |
|
OTC 14049 Monitoring Underwater Operations with Virtual Environments PDF version of this paper Juan C. Santamaria & Augusto Opdenbosch / XYZ Solutions, Inc. 
Copyright 2002, Offshore Technology Conference
This paper was prepared for presentation at the 2002 Offshore Technology Conference held in Houston, Texas U.S.A. , 69 May 2002. This paper was selected for presentation by the OTC Program Committee following review of information contained in an abstract submitted by the author(s). Contents of the paper, as presented, have not been reviewed by the Offshore Technology Conference and are subject to correction by the author(s). The material, as presented, does not necessarily reflect any position of the Offshore Technology Conference or its officers. Electronic reproduction, distribution, or storage of any part of this paper for commercial purposes without the written consent of the Offshore Technology Conference is prohibited. Permission to reproduce in print is restricted to an abstract of not more than 300 words; illustrations may not be copied. The abstract must contain conspicuous acknowledgment of where and by whom the paper was presented. Abstract This paper introduces a new set of software tools that integrate near-real-time visualization with a publish-and-subscribe mechanism to achieve remote monitoring and control of dynamic objects in an underwater scene. The approach proposed in this paper involves the integration of existing technologies to produce a powerful and flexible solution easily adapted to the extensive and diverse set of situations encountered in underwater construction, underwater surveying, and maritime navigation. Users can define elaborate virtual scenes that can accurately represent all the relevant elements associated with an underwater construction job, including complex structures and dynamic objects, such as ROVs, vessels, etc. Real-time data from instruments and positioning sensors is made available by using a publishing mechanism and a remote data server. Users with Internet or intranet access can subscribe to any real-time data field being published and receive updates each time the information changes, allowing them to monitor and log the events of the underwater job at the same time they are taking place. The use of advanced cueing techniques and multi-resolution rendering make it possible to achieve interactive frame rates without sacrificing accuracy and realism. The concept of 3D modules will also be introduced. These powerful modules can be linked to live data and attached to any element in the virtual environment including dynamic ones. This flexibility allows the users to monitor the data, not only as it changes, but also within the spatial context that makes the most sense. This paper focuses on the visualization components of the proposed solution, while providing a general description of the other two components (data acquisition, data distribution), and describes the Ehime Maru recovery mission as an example of an underwater job that benefited by using this technology. Introduction Any scenario that deprives or diminishes our senses will always produce challenges because our ability to make good decisions lessens with poor perception. The ocean is without a doubt a perfect example of such a place. Depths of only a few hundred feet already pose serious challenges for any kind of operation. The drastic loss of visibility associated with depth, combined with the enormous pressures and low temperatures makes it a place where only tele-operated robots can function. These robots provide limited feedback to the people that operate them, making underwater construction a very expensive and time-consuming process. There are several factors responsible for the lack of useful feedback, many of which are bound by the laws of physics. Communication technologies that thrive in air simply fail to work in water (e.g., radio waves). Position technologies such as GPS or laser tracking cannot be used underwater. Light can only travel a limited distance in water. As a result, the sensors currently available provide limited accuracy and frequency. The cameras available today can only provide an image of the immediate vicinity even under good visibility conditions. To complicate things even further, the data collected by all these sensors and cameras is often scattered across many systems, making its perception and analysis very difficult. All these factors lead to working scenarios where the people involved must make decisions with very little information and scattered data. Only when poor perception conditions can be improved to provide better awareness through the use of visualization and data consolidation and management, the resulting increase in awareness will lead to better decisions, hence, better job performance. This paper describes a suite of software tools, collectively known as XYZ World, designed to enhance the awareness of underwater operations and allow operators to make more informed decisions, in less time, at lower cost. XYZ World has been successfully applied in a wide range of underwater operations including the recovery of the Ehime Maru. This paper focuses on the topic of visualization of three-dimensional scenes and it is structured in four sections. The first section defines the concepts and provides an overview of the entire system. The second section describes the implementation of the visualization component in detail. The third section provides a summary of the application of XYZ World in the recovery of the Ehime Maru. The last section concludes the paper and provides references to real-world applications and companies that have benefited from the use of these tools. Definitions and Concepts The main objective of the XYZ World solution is to increase the awareness of operators and involved parties so that they can make more informed decisions in less time and at lower cost. To accomplish this objective, the solution follows two central tenets: (1) to maintain a world model that accurately mimics the real one and (2) to allow operators to observe and interact with the real world indirectly through a virtual environment based on the world model. Thus, operators and involved parties can safely observe and act on events that occur in remote, dangerous, and often humanly inaccessible locations such as underwater scenes. For example, an operator can visualize the placement of a tool on the sea floor as seen from a "virtual" camera located at a fixed point near the target location. In addition, the virtual environment can depict crucial information that is usually implicit or invisible in the real world and that can aid operators in their task. For example, an operator can observe a "virtual" ray from the ROV to the target location indicating the distance to go and adjust the navigation control of the ROV accordingly. A solution with these characteristics needs to perform the following three fundamental functions: real-time data acquisition and processing, robust data distribution, and interactive visualization of three-dimensional scenes. A brief description of each function follows. The section concludes with an overview of the architecture of the complete system. Real-Time Data Acquisition and Processing In most underwater job operations, the data is scattered across several systems, many of which are isolated from each other. The protocols used by sensors and surveying equipment change from one system to another, making data consolidation a real challenge. The solution can be easily configured to adapt to any data stream or protocol will have the power to gather all the relevant information, even if it comes from different sources, including databases. In addition, it is often common that the most relevant information is not implicitly defined in the original data source; thus, the solution provides the capability of deriving and computing additional information from the raw data and making it available to the entire system. Good examples include getting rid of noise through the use of filtering algorithms, computing distances among several dynamic objects, and simulating data fields of a vehicle or subcomponent. The derived information is invaluable during the stages of the job where operators require such information to make effective decisions. Robust Data Distribution Mechanism Often, the data originates from disparate sources and becomes available at different frequencies. In order to provide an accurate representation of an underwater scene, it is of paramount importance to maintain and distribute its current state. The solution provides a mechanism that can easily distribute this information to all relevant destination applications. This distribution mechanism is flexible enough to adapt to the wide number of possible sources and destination combinations, and it is independent of the data sources, data types, and destination applications. It also takes advantage of the current Internet and intranet infrastructures in order to make the information available to remote viewers. Visualization of Three-Dimensional Scenes A three dimensional virtual world is a tool that has the power to assist and improve the perception and understanding of an underwater scene. When the elements of a three dimensional environment are linked with real-time sensor data, another dimension is added into the equation, allowing the information to be presented not only in a spatial but also in a temporal context. Since complexity is often associated with the components of underwater operations, a useful virtual environment system needs to display the virtual scene at interactive frame rates while maintaining the integrity of the scene. This is a very big challenge when high polygon count scenes are involved. There are several options to deal with large data sets but they can all be summarized in one sentence: Show only as many polygons as you need. This is a simple concept to understand but a difficult one to develop. Fortunately, scene optimization is a need that is shared by the video game industry and that has boosted the research and development of multi-resolution rendering techniques. The definition or creation of a virtual scene is another subject that is often neglected. It is common to find cases where scene definition is performed by a different type of user. This can present some logistic challenges when a virtual scene has to be edited during a job. In an ideal world, the issue should be constrained by the access clearance of the user instead of system usability. This is why the definition of a virtual scene must be a simple task that requires very little skill and time. The scene description should also be usable and manageable during all the phases of the job. System Architecture XYZ World's system architecture is simple and concise. The architecture consists of three families of network-enabled applications and services: data distribution, data acquisition, and scene visualization. Each family or suite of applications has been designed to address each of the three main functions described above. The core of the data distribution suite consists of a real-time database server and a publish-and-subscribe service library. The real-time database server is responsible for maintaining an accurate representation (i.e., world model) of all the elements that compose the underwater scene. The publish-and-subscribe library allows all other applications to synchronously and concurrently receive update notifications and query information about the world model. The data acquisition suite consists of applications customized to gather data from specific sources and publish the information to the real-time database server. This suite of applications also includes database access stubs and general-purpose simulators. Together, the data acquisition applications are responsible for updating the world model so that it accurately represents the underwater scene. The scene visualization suite consists of applications that subscribe to the real-time database server, receive updates every time the state of the world model changes, and present the most current state of the scene to the user using 2D or 3D perspectives. In this manner, different viewers at different locations in the network can display the state of the underwater scene in a synchronous fashion. Figure 8 shows a diagram illustrating the components that perform the three main functions of the system. The next section discusses the design of the scene visualization tools, especially the XYZ 3D Viewer, which is the application that allows operators to observe and interact with the real world model through a virtual environment constructed from the world model. The application allows operators to improve the perception and understanding of underwater scenes where near real-time data is available. The visualization tools can be used in all the stages of a job and provide a common framework that increases awareness and promotes better decision-making. Scene Visualization This section describes the suite of tools that support the visualization function of the XYZ World. It is divided into three parts: Data Model, Architecture, and Implementation. The Data Model describes the elements that compose the world model and virtual environment. The Architecture section contains an overview of the structure of the solution and the rationale behind its design. The Implementation section describes the actual components that make up the XYZ 3D Viewer. Data Model The XYZ World represents a virtual environment scene using an efficient data structure termed the world model, which consists of a list of entities with properties designed to represent their real-world counterparts in an underwater scenario. This model is expandable and flexible enough to adapt to the unpredictable nature of sub sea tasks. Figure 9 shows the data model diagram illustrating the entities, properties, and their relationships. A scene in the XYZ World is made of five types of entities: Surfaces: Due to the amount of points that surveying instruments can produce it was necessary to use a surface model capable of representing surfaces with hundreds of millions of polygons yet fast enough to render them at acceptable frame rates. A multi-resolution approach accomplishes both of these requirements by adjusting the amount of detail according to the location of the observer in the virtual scene [2]. The multi-resolution surface model used in XYZ World can be updated in near-real time making it useful for surveying applications and navigation as well as underwater construction. Objects: Static and dynamic objects are represented using CAD geometry or basic shapes (e.g., cubes, cylinders, spheres, cones, etc.). Complex objects with high polygon counts can be handled through the use of interactive level of detail (LOD) management. Dynamic objects are updated through the use of bindings that link objects in the virtual environment with their counterparts in the world model. These objects can have multiple cameras, multiple lights and multiple indicators. In addition, parent/children relations between objects can also be modeled. Cameras: This entity does not have a real-world counterpart, but it is used to represent the concept of a camera in the virtual environment. They can be attached to moving objects and can be configured to track entities as well. Indicators: These entities are used to represent the value of a field or property according to some predefined behavior and/or appearance. These entities can also represent a conceptual property that exists in the real world; for example, the distance between two objects or a projection distance between an object and a surface. Lights: These entities may not have a real-world counterpart in most scenarios, but they are used to represent the concept of a light source in the virtual environment. Architecture The Scene Visualization tools are a collection of interfaces and modular components that define the elements of a virtual environment, such as surfaces, static and dynamic objects, cameras, lights, and indicators. There are two modules that provide support for surfaces. One provides a Triangulated Irregular Network (TIN) framework while the other one supports a Multi-resolution grid (MRGrid). The TIN module is ideal for surfaces with very sparse and irregular points that define a static surface. The MRGrid module needs regularly spaced points but offers real-time update capabilities. The object module provides methods to open, manipulate and render 3-D geometric models, from different file formats, in the virtual scene. In addition, it provides level of detail management tools that can switch between multiple resolution versions of the same shape based on the distance between the viewer and the object. The camera module defines a viewing point in the scene that responds to several input devices. Virtual cameras can be created using this module and attached to moving objects to provide multiple vantage points during a job. The lights module provides methods to manipulate the lighting conditions in the scene. This includes light placement, light properties and light attachments. The indicators are simple modular gadgets that can be attached to objects and display relevant information in real time. Since the management of all these components is a demanding development task, a management module is also available. The scene manager module provides a framework where supported entities (i.e., surfaces, objects, lights, cameras, and indicators) can be added, deleted, queried and manipulated through a common and simple interface. In addition, the scene manager handles entity-to-entity interactions such as references and attachments. Another tedious development effort is the configuration of the application framework. In order to expedite this task, XYZ World also provides Microsoft Foundation Classes (MFC) Document/View derived classes with all the basic components needed to launch a virtual scene. Figure 1 illustrates the three development levels (referred to as "gears") of the XYZ World libraries.
The next subsection covers the implementation details of the modules that compose the data model in XYZ World. Implementation The XYZ 3D Viewer is an application that allows an operator to interactively see a virtual 3D scene including digital terrain models, static and dynamic 3D objects, cameras, light sources and indicators. It couples real-time data coming from the Data Server, with static and moving objects, and indicators in the virtual scene. Each of the entities in the data model described above is realized using software components (see [1,3]). The following subsections provide some examples and images of all the different entities supported in this viewing application. Object Displays the geometry of an object in the virtual environment and represents a real-word object. The object component can render geometry descriptions of different formats (e.g., 3D studio, VRML, Wavefront, or AutoCAD).
LOD Object Displays the geometry of an object using multiple resolutions according to the viewer's location. The object definition consists of several geometry descriptions with different levels of detail and the criteria used to switch among the geometry descriptions. For example, when the point of view of the camera is far away from the object, then the component renders the object with the lowest level of detail, and as the point of view moves closer to the object, the component renders the object using geometries with increasingly higher levels of detail.
Surface Displays a surface in the virtual environment and represents real-world surfaces. The multi-resolution capabilities allow the application to render high polygon count surfaces at interactive frame rates.
Indicator Displays the value of a field or property according to a predefined representation. This entity represents a conceptual property that exists in the real world; for example, the distance of one object with respect to another object or a surface.
Light Represents the concept of a light source in the virtual environment. There can be multiple light instances.
Camera The user navigates the scene using virtual cameras that can be manipulated with a mouse, a joystick, and/or a spaceball. Cameras attached to moving objects can be further adjusted to provide the most useful vantage point possible. The application supports multiple views, including stereoscopic ones, to give users a complete understanding of the virtual scene.
The next section briefly discusses the use of XYZ World during the Ehime Maru recovery mission that ended successfully on November 26, 2001 when the Ehime Maru was brought to its final resting place. Ehime Maru Recovery This section illustrates the use of XYZ World during the Ehime Maru recovery (see also [5]). The underwater job involved many different tasks including underwater navigation, construction, and surveying, and therefore showcases the use of XYZ World as a decision-making support tool. The first subsection describes the project and its objectives. The second subsection provides a detailed description of how the XYZ World tools were used in each phase to improve the decision process. Project Description The Ehime Maru salvage showcases all the concepts presented in this paper. The mission objective was to recover the vessel that rested on the seabed at approximately 2000 feet. The project consisted of four stages. The first stage was a reconnaissance survey of the area to establish the exact location of the vessel, to collect the bathymetry of the area, and prepare the vessel for lifting. Stage two involved the installation of a sensor array, the removal of debris from the area, and the installation of a lifting harness around the vessel. The third stage encompassed the lifting and towing of the vessel to a shallow water area where it was accessible to Navy divers for the search and recovery of 8 victims. The fourth stage consisted of the relocation of the Ehime Maru to its final resting place. The XYZ World was put to use during the planning, execution, and evaluation phases of the first and second stages of the recovery mission. During the second stage, the system consisted of 5 viewing stations located around the recovery vessel (Rockwater 2). The location of the computers were as follows: one at the bridge to support the captain of the Rockwater 2, one at the logistics room to support the project manager and other observers, one at each of the three ROV vans to support the operators of the ROVs (Manta, XL16, and Quest). The next subsections provide a brief description of how the XYZ World tools where used in each phase to improve the decision process. Figure 10 shows snapshots of the XYZ World and real photos taken during the recovery job. Planning The key concept that allows XYZ World tools to be used during the planning phase is the fact that the server and the viewer applications are independent of the source of the information. Simulated positions, orientations, and other variables can be published, visualized, and logged in the same fashion as the real data. This allows users to plan the job using the same environment they will later use during the execution of the project. The only difference between planning and execution is that during execution, the sources of the data are real sensors instead of simulators. It is important to note that the usefulness of this analysis depends on the accuracy and fidelity of the simulators that are used. In this particular job, the bathymetry data collected during the first stage of this project played an important role in the planning phase. Once the survey was completed, a more in depth analysis was conducted to determined optimal paths for ROV and vessel navigation, task durations, risk of collisions, etc. An additional application of a simulated scene was to present ideas to all the project members and clients. An interactive three-dimensional scene provided an understanding that otherwise could require time to explain. Execution During the execution of this job, position sensors for the ROVs involved collected their position and orientation and published it in real-time to the server. The five viewing stations subscribed to the server and obtained current data for all the objects in the world model. Virtual cameras were placed in key locations to provide feedback to the people operating the ROVs, the vessel and other pieces of equipment such as cranes and drillers. Distance indicators designed to measure clearances in critical locations were invaluable and alerted viewers of potential collisions before they happened. During the towing phase, GPS position was published and used in the viewer stations to place the Rockwater 2 within the scene and show the crew what lies beneath the surface of the water. This allowed them to maneuver the Rockwter 2 and raise or lower the Ehime Maru. Evaluation Since all the data published to the server was logged, a total reconstruction of the entire operation could always be accomplished. Visualization of the logged data was performed through the same viewer used during the planning and execution phases. The data was logged by the server with time stamps that can be used during playback to reconstruct the events with temporal accuracy. Since navigation within the viewer is independent of the events taking place, the logged events can be viewed and studied from new locations and perspectives. Furthermore, televiewing can also take place, allowing viewers in remote sites to discuss and analyze the operation without having to travel to a meeting location. Conclusion All the concepts introduced in this paper have been validated in the field by our clients. Applications in the areas of marine navigation, underwater construction and underwater surveying have been developed using XYZ World technology. Kongsberg Simrad's SPS2000 digital chart navigator uses a 3D viewer that displays a 3D surface of a digital chart area along with a virtual representation of the boat and surrounding objects (e.g., lighthouses, buoys, underwater structures, etc.). The view provided by the 3D display improves the perception of the boat's location in relation to the seabed and allows the crew to make better decisions while sailing. QPS's 3D Viewer uses a multi-resolution grid to visualize multi-beam data as it is collected, allowing the surveyors to visualize the area of interest and make sure that the target zone is covered. This prevents having to go back to the site due to missing spots. Several survey companies use XYZ's data collectors to publish data from sensors to an XYZ Data Server. By subscribing to it from multiple locations using XYZ 3D Viewers, engineers can visualize the entire operation as it takes place from local or remote locations. This system has allowed survey companies to visualize not only the execution phase of projects but has assisted the company during the planning phases of key projects. In addition, the data logged during the execution phases of these projects has been useful to present results to the survey companies' clients and keep a record of the tasks completed. Existing clients have validated the usefulness of XYZ World tools in a variety of scenarios. The quality and performance of the jobs they have accomplished while using the XYZ World tools improved dramatically. They have attributed the reason for their success to the new level of awareness provided by these tools, which have led them to make time and cost-saving decisions in their projects. References 1. D. Box, Essential COM. Addison Wesley, 1998. 2. Lindstrom, P., Koller, D., Ribarsky, W., Hodges, L., Faust, N., and Turner, G. "Real-time, continuous level of detail rendering of height fields," In Computer Graphics (SIGGRAPH '96 Proceedings) (1996), 109-118. 3. M. Kirtland, Designing Component-Based Applications. Microsoft Press, 1999. 4. Opdenbosch, A. and Santamaria, J.C. "Remote Visualization and Management Tools for Underwater Operations". In Proceedings MTS/IEEE Oceans 2001 (2001), Vol. 3, 1953-1959. 5. Canyon Offshore. "The Recovery of the Ehime Maru". Ocean News & Technology. Nov./Dec. 2001. 34-36. Technology Systems Corporation, 2001. |
