     
» Demos» Published Papers » Tech FAQ's        |
| PUBLISHED PAPERS |
Remote Visualization and Management Tools for Underwater Operations
prepared for presentation at the Oceans 2001 Conference PDF version of this paper
Abstract This paper introduces a new set of software tools that integrate near-real-time visualization with a publish and subscribe mechanism to achieve remote monitoring and control of dynamic objects in an underwater scene. The approach proposed in this paper involves the integration of existing technologies to produce a powerful and flexible solution to problems in which enhanced awareness of the situation can lead to improvements in performance. These tools can be easily adapted to the extensive and diverse set of situations encountered in underwater construction, underwater surveying and maritime navigation. Real-time data from instruments and positioning sensors is made available by using a publishing mechanism and a remote data server. Users with Internet or intranet access can subscribe to any real-time data field being published and receive updates every time the information changes. Underwater infrastructure information resulting from construction, maintenance and inspection is often stored in databases. The software libraries introduced in this paper also provide a full set of tools to store and retrieve information from such databases. The database access tools use the same publish and subscribe mechanism involved in real-time data acquisition. This similarity makes it very easy for developers to integrate historic and live data in a single monitoring application. The visualization tools presented in this paper enable developers to define virtual scenes that can display all the relevant elements associated with an underwater construction job, including complex structures and dynamic objects; i.e., ROVs, vessels, etc. The use of advance cueing techniques and multi-resolution rendering make it possible to achieve satisfactory interactive frame rates without sacrificing accuracy and realism. The concept of 3D indicator modules will also be introduced. These powerful modules can be linked to live data using the subscribe tools and attached to any element in the environment including dynamic ones. This flexibility allows the users to monitor the data, not only as it changes, but also in the 3D location that makes the most sense. The three components of the tools being introduced in this paper (data acquisition, data distribution, and scene visualization) will be fully described along with examples of applications that take full advantage of the integration of all three components. I. Introduction Any scenario that deprives or diminishes our senses will always produce challenges because our ability to make good decisions lessens with poor perception. The ocean is without a doubt a perfect example of such a place. Depths of only a few hundred feet already pose serious challenges for any kind of operation. The drastic loss of visibility associated with depth, combined with the enormous pressures and low temperatures makes it a place where only tele-operated robots can function. These robots provide limited feedback to the people that operate them, making underwater construction a very expensive and time-consuming process. There are several factors responsible for the lack of useful feedback, many of which are bounded by the laws of physics. Communication technologies that thrive in air simply fail to work in water (e.g., radio waves). Position technologies such as GPS or laser tracking cannot be used underwater. Light can only travel a limited distance in water. As a result, the sensors currently used provide limited accuracy and frequency. The cameras available today can only provide an image of the immediate vicinity even under good visibility conditions. To complicate things even further, the data collected by all these sensors and cameras is often scattered across many systems, making its perception and analysis very difficult. All these factors lead to working scenarios where the people involved must make decisions with very little information and scattered data. Only when poor perception conditions can be improved to provide better awareness through the use of visualization and data consolidation and management, the resulting increase in awareness will lead to better decisions, hence, better job performance. This paper is structured as follows. Section II outlines the requirements of a solution that can accomplish these goals and therefore significantly improve the performance of underwater construction, underwater surveying, and navigation jobs. Section III describes a proposed solution that fully addresses the requirements described in Section II. Section IV illustrates a hypothetical sample project (Salvage Operation). This particular job has been designed to showcase all the concepts presented in this paper. Even though the project itself is not real, it is based on several projects where this technology has been used in the manner exposed. Section V concludes the paper and provides references to real-world applications and companies that have benefit with the use of these tools. II. Solution Requirements The main objective of the solution is to increase awareness of operators and involved parties so that they can make better decisions. An ideal solution should improve and facilitate the decision-making processes in all phases of a job. For example, during the planning phase of an underwater construction job, the solution should allow engineers to perform tasks that can aid decisions at the strategic level and help them share information about the job. During this phase the solution should allow engineers to perform tasks such as visualizing and considering different scenarios for positioning the construction tools in the underwater scene and using simulators to debug operation details and/or brief operators about the specifics of the job. During the execution phase, the solution should allow operators, engineers and other interested parties to watch the progress of the job in real-time and provide relevant and effective information that can aid decisions at the control level. Finally, during the evaluation phase, the solution should enable information sharing and learning among interested parties by providing flexible and interactive views of the underwater scene that were crucial to the success or failure during different stages of the job. It is important to note that while most complex operations commonly require planning, execution, and evaluation phases, the tools associated with each of these phases not only differ but also lack the capabilities for sharing information. The processes associated with underwater operations are no exception to this problem. In many instances the contractors involved in the different stages of a project use different data gathering systems, communication protocols, and storage schemes, making it difficult for other parties and even the clients to take advantage of the information available. This problem also affects information sharing across the project phases even within a single company. A solution that can be used in all stages of a project will not only create a link across all the parties involved but could also provide a standard for data acquisition, visualization, and analysis. Such a solution must comply with the following three groups of requirements: A. Real-Time Data Acquisition and Processing In most underwater job operations, the data is scattered across several systems, many of which are isolated from each other. The protocols used by sensors and surveying equipment change from one system to another, making data consolidation a real challenge. A solution that can be easily configured to adapt to any data stream or protocol will have the power to gather all the relevant information, even if it comes from different sources including databases. In addition, it is often common that the most relevant information is not implicitly defined in the original data source; thus, the solution should provide the capability of deriving and computing additional information from the raw data and making it available to the entire system. Good examples include getting rid of noise through the use of filtering algorithms, computing distances among several dynamic objects, and simulating data fields of a vehicle or subcomponent. The derived information is invaluable during the stages of the job where operators require such information to make effective decisions. B. Robust Data Distribution Mechanism Often, the data originates from disparate sources and becomes available at different frequencies. In order to provide an accurate representation of an underwater scene, it is of paramount importance to maintain and distribute its current state. The solution must provide a mechanism that can easily distribute this information to all relevant destination applications. This distribution mechanism must be flexible enough to adapt to the wide number of possible sources and destination combinations, and it must be independent of the data sources, data types, and destination applications. It should also take advantage of the current Internet and intranet infrastructures in order to make the information available to remote viewers. C. Visualization of Three-Dimensional Scenes A solution that can present all the information in a spatial context in near-real time will eliminate confusion while providing a common place for analysis, inspection, collaboration, verification and validation. A three dimensional virtual world that can display all the relevant components of a job as a dynamic system will have the power to assist and improve the perception and understanding of an underwater scene. Since complexity is often associated with the components of underwater operations, it is important that a solution can display all the elements while maintaining interactive frame rates. This must be done without sacrificing the integrity of the scene. The definition of a virtual scene must be a simple task that requires very little skill and time. The scene description should also be usable and manageable during all the phases of the job. Furthermore, this scene description needs to reflect the state of the job as a function of time regardless of whether that time is simulated, current, or logged. The next section will introduce a suite of tools that address all these requirements to provide a comprehensive solution that can be used to improve the perception and understanding of underwater scenes where near real-time data is available. The proposed solution can be used in all the stages of a job and provides a common framework that increases awareness and promotes better decision-making. III. Proposed solution: XYZ World This paper introduces a new suite of tools, henceforth XYZ World, designed to integrate data acquisition, data distribution and scene visualization. This section is divided in two parts: Architecture and Implementation. The Architecture section contains an overview of the structure of the solution and the rationale behind its design. The Implementation section describes the actual components that make up the XYZ World. A. Architecture XYZ World's architecture is simple and concise. The architecture consists of three families of network-enabled applications and services: data distribution, data acquisition, and scene visualization. Each family or suite of applications has been designed to address each of the three requirements described in Section II. The core of the data distribution suite consists of a real-time database server and a publish-and-subscribe service library. The real-time database server is responsible for maintaining an accurate representation or world model of all the elements that compose the underwater scene. The publish-and-subscribe library allows all other applications to synchronously and concurrently receive update notifications and query information about the world model. The data acquisition suite consists of applications customized to gather data from specific sources and publish the information to the real-time database server. This suite of applications also includes database access stubs and general-purpose simulators. Together, the data acquisition applications are responsible for updating the world model so that it accurately represents the underwater scene. The scene visualization suite consists of applications that subscribe to the real-time database server, receive updates every time the state of the world model changes, and present the most current state of the scene to the user using 2D or 3D perspectives. In this manner, different viewers at different locations in the network can display the state of the underwater scene in a synchronous fashion. The following three sections describe the three components of the system architecture in more detail. 1) Data Distribution The main objective of the real-time database server is to maintain and distribute an accurate representation of the underwater scene. The server represents the scene using an efficient data structure termed the world model, which consists of a list of entities with properties designed to represent their real-world counterparts in an underwater scenario. This model is expandable and flexible enough to adapt to the unpredictable nature of sub sea tasks. A scene in the XYZ World is made of five types of entities: Surfaces: Due to the amount of points that surveying instruments can produce it was necessary to use a surface model capable of representing surfaces with hundreds of millions of polygons yet fast enough to render them at acceptable frame rates. A multi-resolution approach accomplishes both of these requirements by adjusting the amount of detail according to the location of the observer in the virtual scene [2]. The multi-resolution surface model used in the XYZ World can be updated in near-real time making it useful for surveying applications and navigation as well as underwater construction. Objects: Static and dynamic objects are represented using CAD geometry or basic shapes (e.g., cubes, cylinders, spheres, cones, etc.). Complex objects with high polygon counts can be handled through the use of interactive level of detail (LOD) management. Dynamic objects are updated through the use of bindings that link objects in the virtual environment with their counterparts in the world model. These objects can have multiple cameras, multiple lights and multiple indicators. In addition, parent/children relations between objects can also be modeled. Cameras: This entity does not have a real-world counterpart, but it is used to represent the concept of a camera in the virtual environment. They can be attached to moving objects and can be configured to track entities as well. Indicators: These entities are used to represent the value of a field or property according to some predefined behavior and/or appearance. These entities can also represent a conceptual property that exists in the real world; for example, the distance between two objects or a projection distance between an object and a surface. Lights: These entities may not have a real-world counterpart in most scenarios, but they are used to represent the concept of a light source in the virtual environment. 2) Data Acquisition The main objective of the data acquisition applications is to update the state of the world model by acquiring and publishing the data originating from disparate data sources. Specifically, each application represents the interface between a given data source and the world model since its main function is to acquire the data corresponding to the source and update the relevant components in the world model by publishing such data. There are three different groups of data acquisition applications: Sensor gathering: These applications interface directly with the sensors that provide the data. Common examples range from simple embedded microcontrollers with Analog-to-Digital (A/D) converters to sophisticated survey computers communicating through serial cables. Data processing: These applications commonly generate and publish new information by subscribing to the data gathered and published by other applications. Common examples are data filters and general-purpose simulators. Database stubs: These applications serve as gateways to high-end databases and they are responsible for publishing information that is relevant in the world model. 3) Scene Visualization These tools are a collection of specialized component-based modules, called 3D Gear, designed to shorten the development cycle of complex virtual environment applications [1]. XYZ's 3D Gear provides three different levels of abstraction. The first level of abstraction (first gear) provides direct access to the software components that model the key elements of a virtual scene. At this level, the 3D Gear is a collection of interfaces and components that define the elements of a virtual environment, such as surfaces, static and dynamic objects, cameras, lights, and indicators. The second level of abstraction (second gear) consists of managing components that create, destroy, manipulate, and track all the components in the world model. The third level of abstraction (third gear) includes specialized versions of Microsoft Foundation Classes (MFC) for the Document and View classes that can give a custom application instant access and visualization of a world model. B. Implementation This section describes the current state of the suite of applications that constitute XYZ World as of Summer 2001. However, efforts continue to expand the suite, support new types of data sources, viewers, and data distribution schemes. The suite consists of XYZ Serial (Data Acquisition), XYZ Data Server (Data Distribution), and XYZ 3D Viewer (Scene Visualization). 1) XYZ Serial The XYZ Serial program receives external sensor data and publishes them to the XYZ Data Server program where they are subsequently sent on to the XYZ 3D Viewers. The XYZ Serial program can receive data two ways: From a serial port. Typically, survey computers are capable of receiving real-time raw positioning data from the sensors on a device like an ROV, calculating actual X, Y, and Z coordinates, and assembling that data into a survey string that is output from the survey computer on its serial port. From a file. The XYZ Serial program is capable of opening and reading a text file of data values. This is useful to test or simulate the survey data string prior to receiving live data from the serial interface of the survey computer. Since there are almost an infinite number of message formats and fields that can be sent, a sample of each data string is obtained in advance and analyzed. The anticipated formats are entered into the Message Definition File and saved. This Message Definition File is an Excel spreadsheet. Once prepared, the user runs the XYZ Serial program and then loads the Message Definition File. This file tells the XYZ Serial program how to publish the data it is receiving. 2) XYZ Data Server The XYZ Data Server is a COM+ application [3] that allows network users of the XYZ World to share distributed data in near real-time. The application efficiently manages the sharing of data by allowing programs to notify interested parties of changes in the data. Programs that provide notification to other programs, such as the XYZ Serial, are termed Publishers and programs that receive data from publishers and act on it, such as XYZ 3D Viewer, are termed Subscribers. The XYZ Data Server maintains an external database (i.e., COM+ Catalog) containing a list of the different event types that applications can use to publish and subscribe data. Subscribers can read this list and pick the events types that they want to hear about. When a publisher wants to fire an event, it looks in this database, finds the event type and activates it. Then, the XYZ Data Server finds all interested subscribers and notifies the new event to every interested subscriber. The XYZ Data Server has the following features: Event notifications: Each event consists of a list of fields, each having three members (i.e., a triplet): id, a value, and unit. The id is the name of the field, value is the numeric quantity of the field, and unit is the unit of measure of the numeric quantity. Event Filtering: This feature allows the XYZ Data Server to enable or disable notification only if the event parameters had certain values. For example, if a certain numeric field is within a range. Monitoring. The ability to display the subscribers and publishers and monitor their communication. Security: The XYZ Data Server uses the COM+ authentication system. The administrator can enable or disable the right to use different notification types for domain groups or users. 3) XYZ 3D Viewer This application allows a user to interactively see a virtual 3D scene including digital terrain models, static and dynamic 3D objects, cameras, and light sources. It couples real-time data coming from the Data Server with static and moving objects, and indicators in the virtual scene. The user navigates the scene using virtual cameras that can be manipulated with a mouse, a joystick, and/or a spaceball. Cameras attached to moving objects can be further adjusted to provide the most useful vantage point possible. The application supports multiple views, including stereoscopic ones, to give users a complete understanding of the virtual scene. The next section will discuss the use of XYZ World introduced in this section using a sample marine application as a target project. IV. Sample Project (Salvage Operation) This section illustrates how all phases of an underwater job could benefit by using the tools introduced in this paper. We have selected a salvage operation since it incorporates navigation, underwater tasks and surveying, and therefore showcases all the marine application types discussed previously (construction, surveying, and navigation). First we will describe the project and its objectives. This will be followed by a detailed description of how the XYZ World tools are used in each phase to improve the decision process. A. Project Description This particular salvage example has been designed to showcase all the concepts presented in this paper. Even though the project itself is not real, it is based on several projects where this technology has been used in the manner exposed. The mission objective is to recover a vessel that lies on the sea bed at 3000 feet. The project will be executed in three stages. The first stage will consist of a recognizance survey of the area to establish the exact location of the vessel and to collect the bathymetry of the area. Stage two will involve the installation of a sensor array, the removal of debris from the area, and the installation of a lifting harness around the vessel. The final stage will encompass the lifting and towing of the vessel to a shallow area where it will later become a diving attraction. The following section will discuss the use of XYZ World tools during the planning phase. This will be followed by a discussion about the use of the tools during the execution of the salvage operation. The final section will suggest possible applications of the same tools in an evaluation phase. B. Planning The key concept that allows XYZ World tools to be used during the planning phase is the fact that the server and the viewer applications are independent of the source of the information. Simulated positions, orientations, and other variables can be published, visualized, and logged in the same fashion as the real data. This allows users to plan the job using the same environment they will later use during the execution of the project. The only difference between planning and execution is that during execution, the sources of the data are real sensor instead of simulators. It is important to note that the usefulness of this analysis depends on the accuracy and fidelity of the simulators that are used. In this particular example, the bathymetry data collected during the first stage of this project can play an important role in the planning phase. Once the survey is completed, a more in depth analysis can be conducted to find out optimal paths for ROV and vessel navigation, task durations, risk of collisions, etc. An additional application of a simulated scene is to present ideas to all the project members and clients. An interactive three-dimensional scene provides an understanding that otherwise could require time to obtain. C. Execution During the execution of this job, position sensors for the ROVs involved can collect their position and orientation and publish it in near real-time to the server. Viewers (local and remote) can subscribe to the server and obtain current data for all the objects in the world model. Virtual cameras can be placed in key locations to provide feedback to the people operating the ROVs, the vessel and other pieces of equipment such as cranes and drillers. Distance indicators designed to measure clearances in critical locations can be invaluable and would alert viewers of potential collisions before they happen. If variables such as age of data (AOD) are published, the viewer can display the affected objects in a different fashion to alert users that what they are seeing is not up-to-date. During the towing phase, GPS position can be published and used in the viewer to place the salvage vessel within the scene and show the crew what lies beneath the surface of the water. This will allow them to maneuver the salvage vessel and raise or lower the towed boat. Tension data for the crane wires can be displayed in the viewer using colors to show the stress and warn the crew about a possible failure. D. Evaluation Since all the data published to the server is logged. A total reconstruction of the entire operation can always be accomplished. Visualization of the logged data is performed through the same viewer used during the planning and execution phases. The data is logged by the server with a time stamp that can be used during playback to reconstruct the events with temporal accuracy. Since navigation within the viewer is independent of the events taking place, the logged events can be viewed and studied from new locations and perspectives. Furthermore, tele viewing can also take place, allowing viewers in remote sites to discuss and analyze the operation without having to travel to a meeting location. V. Conclusion All the concepts introduced in this paper have been validated in the field by our clients. Applications in the areas of marine navigation, underwater construction and underwater surveying have been developed using XYZ Technology. Kongsberg Simrad's SPS2000 digital chart navigator uses a 3D viewer that displays a 3D surface of a digital chart area along with a virtual representation of the boat and surrounding objects (e.g., lighthouses, buoys, underwater structures, etc.). The view provided by the 3D display improves the perception of the boat's location in relation to the seabed and allows the crew to make better decisions while sailing. QPS's 3D Viewer uses a multi-resolution grid to visualize multi-beam data as it is collected, allowing the surveyors to visualize the area of interest and make sure that the target zone is covered. This prevents having to go back to the site due to missing spots. Several survey companies use XYZ's data collectors to publish data from sensors to an XYZ Data Server. By subscribing to it from multiple locations using XYZ 3D Viewers, engineers can visualize the entire operation as it takes place from local or remote locations. This system has allowed survey companies to visualize not only the execution phase of projects but has assisted the company during the planning phases of key projects. In addition, the data logged during the execution phases of these projects has been useful to present results to the survey companies' clients and keep a record of the tasks completed. Existing clients have validated the usefulness of XYZ World tools in a variety of scenarios. The quality and performance of the jobs they have accomplished while using the XYZ World tools improved dramatically. They have attributed the reason for their success to the new level of awareness provided by these tools, which have led them to make time and cost-saving decisions in their projects.
References [1] D. Box, Esential COM. Addison Wesley, 1998. [2] Lindstrom, P., Koller, D., Ribarsky, W., Hodges, L., Faust, N., and Turner, G. "Real-time, continuous level of detail rendering of height fields," In Computer Graphics (SIGGRAPH '96 Proceedings) (1996), 109-118. [3] M. Kirtland, Designing Component-Based Applications. Microsoft Press, 1999. PDF version of this paper |
