eScience Lectures Notes : From QuickTime to MPEG4

Slide 1 : Promises and Challenges of Networked Virtual Environments

From QuicktimeVR to MPEG4

Introduction

3D File Format, Graphic Library/programming Langage

QuicktimeVR

The graphics libraries from SGI

DirectX

VRML : Virtual Reality Modeling Langage

VRML like...

Java 3D

Communications Middleware

Living Worlds

Open Community

Virtual Communities

High Level Architecture (HLA)

Java Shared Data Toolkit (JSDT)

CavernSoft

DIVE

Development Toolkit, all in one !...

MPEG4

DIS

Virtools worldtoolkit

Tiwi

 

Slide 2 : From QuicktimeVR to MPEG4

From QuicktimeVR to MPEG4

Cheap Networked VR :

Without or with low level of sharing

With low level of immersion

From simple tools to basic bricks to build complex NVR

One aspect (will be presented by Hugh Fisher) : "without any graphics, you may experience the feeling of community" (Multi User Donjon)

What we may begin with... "without complex sharing, we may experience a feeling of exploration of virtual world"

File Format, Graphic Library/programming Langage, Development Toolkit ...

Slide 3 : 3D File Format, Graphic Library/programming Langage

3D File Format, Graphic Library/programming Langage

Rendering and Graphics

These standards are mainly concerned with graphics and rendering of 3D scene graphs.
They have very little or no consideration about communication between users and other networking related issues.

QuicktimeVR

The graphics libraries from SGI

DirectX

VRML : Virtual Reality Modeling Langage

VRML like...

Java 3D

 

Slide 4 : QuicktimeVR

QuicktimeVR

Realistic Immersion

Cheap

360ish degrees

QuicktimeVR is a technology that allows users to look around and from within a cylinder or a sphere of stitched 2D pictures in order to feel inside somewhere.

Creation

Terrace, Pool, Reception, Bar, Restaurant and magnificent views of Dunk Island and the Coral Sea.

At 2228 meters above sea level Mount Kosciuszko is Australia's highest peak.

 

Imagina 2000

Online : http://imagina.ina.fr/Imagina/2000/3D/QuickTimeVR/scene_1_vers_2.fr.html

Local : /INA/Imagina/2000/3D/QuickTimeVR/scene_1_vers_2.fr.html

Slide 5 : SGI

The graphics libraries from SGI

OpenGL

Open Inventor

The OO format of SGI

 

OpenGL Performer

OpenGL Performer provides a powerful and extensible programming interface (with ANSI C and C++ bindings) for creating real-time visual simulation and other interactive graphics applications.

OpenGL Multipipe
OpenGL Multipipe SDK
OpenGL Optimizer
OpenGL Performer
OpenGL Shader
OpenGL Volumizer
OpenGL Vizserver

OpenGL is a graphical library introduced by Silicon Graphics in 1992 to allow developers to write a single piece of code, based on the OpenGL API2, which is supposed to run in various platforms (as long as they have an OpenGL library implementation).

Since its inception OpenGL has been controlled by an Architectural Review Board3 whose representatives are from the following companies: 3DLabs, Compaq, Evans & Sutherland (Accelgraphics), Hewlett-Packard, IBM, Intel, Intergraph, NVIDIA, Microsoft, and Silicon Graphics.

Most of the Computer Graphics research (and implementation) broadly uses OpenGL which has became a de facto standard. Virtual Reality is no exception to this rule.

OpenGL Architecture Review Board, http://www.opengl.org

There are many options available for hardware acceleration of OpenGL based applications. The idea is that some complex operations may be performed by specific hardware (an OpenGL accelerated video card such as those based on 3DLab’s Permedia series or Mitsubishi’s 3Dpro chipset for instance) instead of the CPU which is not optimized for such operations. Such acceleration allows low-end workstations to perform quite well yet at low cost.

We will see that such is the importance of OpenGL that both VRML and Java3D are built on top of it, i.e. if a given workstation has hardware support for OpenGL the VRML browser and Java3D, will also benefit from it.

These standards are mainly concerned with graphics and rendering of 3D scene graphs. They have very little or no consideration about communication between users and other networking related issues.

Slide 6 : DirectX

DirectX

Windows specific

Some original things, patented by microsoft

(Vector mapping ?)

Microsoft DirectX® is a group of technologies designed by Microsoft to allow Windowsbased computers to run and display applications rich in multimedia elements such as fullcolor graphics, video, 3-D animation, and surround sound. DirectX is an integral part of Windows 98 and Windows 2000, as well as Microsoft® Internet Explorer 4.0. DirectX components may also be installed in Windows 95 as an optional package.

DirectX allows a compliant application to run in any Windows based system, independent of particularities of hardware of each system. In some sense it seems similar to OpenGL; however there is a logical limitation in disponibility as it is a Windows specific component.

DirectX accomplishes its task via a multilayered structure. The Foundation layer is responsible for resolving any hardware dependent issue. DirectX also allows developers to deploy creation and playback of multimedia content via DirectX’s Media layer. A third layer, Component, complets the high level protocol layer stack.

Some VRML browsers also provide a Direct3D based version (as well as the common
OpenGL). A good example is blaxxun’s Contact 4.04

Slide 7 : VRML

VRML : Virtual Reality Modeling Langage


VRML 2.0, which is the latest version of the well-known VRML format, is an ISO standard (ISO/IEC 14772-1:1997). Having a huge installed base, VRML 2.0 has been designed to support easy authorability, extensibility, and capability of implementation on a wide range of systems. It defines rules and semantics for presentation of a 3D scene.

Using any VRML 2.0 compliant browser, a user can simply use a mouse to navigate through a virtual world displayed on the screen. In addition, VRML provides nodes for interaction and behavior. These nodes, such as TouchSensor and TimeSensor, can be used to intercept certain user interactions or other events which then can be ROUTed to corresponding objects to perform certain operations.

Moreover, more complex actions can take place using Script nodes which are used to write programs that run inside the VRML world. In addition to the Script node, VRML 2.0 specifies an External Authoring Interface (EAI) which can be used by external applications to monitor and control the
VRML environment. These advanced features enable a developer to create an interactive 3D environment and bring the VRML world to life.

Slide 8 : VRML like...

VRML like...

After the fall of interest in VRML....

Important directions ?

Slide 9 : Java 3D

Java 3D

Java 3D is part of the Java Media APIs developed by Javasoft. Providing developers with high level constructs for creating and manipulating 3D geometry in a platformindependent way, the Java 3D API is a set of classes for writing three-dimensional graphics applications and 3D applets.

Since a Java3D program runs at the same level as any other Java program/applet in the virtual machine, controlling the virtual world
becomes very easy through calling the Java3D API from any Java program. Although Java3D and VRML both appear to target the same application area, they have fundamental differences. VRML is aimed at a presentational application area and includes some support for runtime programming operations through its External Authoring Interface and the Script node, as mentioned earlier. Java 3D; however, is specifically a Java language API, and is only a runtime API. Java 3D does not define a file format of its own and is designed to provide support for applications that require
higher levels of performance and interactivity, such as real-time games and sophisticated mechanical CAD applications. In this sense, Java 3D provides a lower-level, underlying platform API.

Many VRML implementations can be layered on top of Java 3D. In fact, it is possible to write a VRML browser using Java 3D, such as the browser developed by VRML consortium’s Java3D and VRML Working group.

Slide 10 : Communications Middleware

Communications Middleware

Standards in this category focus on the issues concerning connecting users together on
the network to create shared worlds. Although some of them also provide graphical
capabilities, their main target is networking, world status updating, and object-sharing
capabilities.

Slide 11 : Living Worlds

Living Worlds


Living Worlds (LW) is a Working Group of the VRML Consortium, supported by a large number of organizations. The LW effort aims to define a set of VRML 2.0 conventions that support applications which are multiuser and interoperable. “Scenesharing", which is concerned with the coordination of events and actions across the network, is one of the main elements of LW. In short, LW is a first attempt to devise a common VRML 2.0 interface to support basic interaction in multi-user virtual scenes and enables each participant to know that someone has arrived, departed, sent a message or changed something in the scene.

Although LW specifies rules for object sharing and exchanging update messages across the network, it is not a communications middleware
and the reason it is being presented in this section is that it is also not concerned with graphics and rendering. In fact LW does not care about the actual technical implementation of the communications system that enables world sharing. Referred to as the Multi User Technology (MuTech), the actual system that runs on the network and is responsible for message passing among clients can be developed by any technology as long as its interface to the VRML world follows the LW specifications. Currently no implementations of LW are publicly available.

http://www.vrml.org/WorkingGroups/living-worlds/

Slide 12 : Open Community

Open Community

A proposal of standard for multiuser enabling technologies from Mitsubishi Electric Research Laboratories (MERL)

Spline (Scalable Platform for Large Interactive Networked Environments) is an implementation compliant with OC which provides a library with ANSI C and Java API.

.... 1997

Open Community (OC) is a proposal of standard for multiuser enabling technologies from Mitsubishi Electric Research Laboratories. Spline (Scalable Platform for Large Interactive Networked Environments) is an implementation compliant with OC which provides a library with ANSI C and soon Java API. Such library provides very detailed and essential services for real-time multi-user cooperative applications. For its communication, Spline uses the Interactive Sharing Transfer Protocol (ISTP) which is a hybrid protocol supporting many modes of transportation for VR data and information, namely

Spline uses the Interactive Sharing Transfer Protocol (ISTP) which is a hybrid protocol supporting many modes of transportation for VR data and information:

The last two subprotocols are build upon the other three. ISTP does not provide videostreaming capability to date, however such support could be provided by extending ISTP with an extra appropriated subprotocol.

http://www.opencommunity.com/

Slide 13 : Virtual Communities, Games

Virtual Communities

Active Worlds : http://www.activeworlds.com/ and Alpha World

Paris Second World http://www.2nd-world.fr/

SCOL de Cryo Networks http://www.cryo-networks.com/ and Cryopolis...

Blaxxun contact

Deep Matrix

La solution Open Source de Geometrek est peut être une alternative ou un point de départ pour ceux qui veulent mettre au point une communauté virtuelle sans avoir à être dépendant d'un éditeur. Le serveur et le client sont développés en JAVA, Deep Matrix est compatible avec les plug in VRML les plus courants (Cortona, Blaxxun, et CosmoPlayer), plug in qu'il utilise comme simple moteur 3D. Vous l'avez compris, la solution est très ouverte, elle est en plus gratuite pour une utilisation non commerciale.

VNET

Voici un autre serveur multi-utilisateurs Open Source. Il est bâti sur la même architecture que Deep Matrix : un serveur en Java, un client Java avec un plug in VRML pour afficher le monde virtuel. Le protocole de partage de monde (déplacement d'objets, message privé, événements partagés) est intéressant. VNET vous est fourni avec ses sources.
Actualité :
VNET et Deep Matrix sont des solutions qui sont abandonnées depuis l'année 1999.
Suite aux problèmes financiers de Blaxxun, beaucoup de créateurs de mondes virtuels ne savent pas comment continuer à partager leurs créations, si Blaxxun venait à fermer son serveur gratuit. Certains groupes d'utilisateurs s'organisent actuellement pour "déterrer" VNET et DeepMatrix. Ces deux solutions sont effet stables et pleines de promesses, elles pouraient constituer une alternative à la solution gratuite de Blaxxun. Espèrons tout de même que Blaxxun survive à la tempête !

Games

Doom, Quakes... STU

EverQuest : http://everquest.station.sony.com/

XPilot http://www.xpilot.org/ (2D only)

Slide 14 : High Level Architecture (HLA)

High Level Architecture (HLA)

HLA is the spiritual successor to DIS, although it focuses more closely on the
problems of arranging very large-scale simulations rather than the run-time distribution of data.

DIS / HLA - military standards tailored to the requirements of simulation and war games

Framework for distributed simulation systems developed by the U.S. Defense Modeling and Simulation Office

Defines standard services and interfaces to be used by all participants in order to support efficient information exchange

The High Level Architecture (HLA) is a software architecture for creating computer simulations out of component simulations. The HLA provides a general framework within which simulation developers can structure and describe their simulation applications.

HLA is a framework for distributed simulation systems developed by the U.S. Defense Modeling and Simulation Office (DMSO). HLA attempts to provide a very generic environment that any virtual object can attach to in order to participate in a simulation. It is a very well-thought architecture that defines standard services and interfaces to be used by all participants in order to support efficient information exchange. HLA is adopted as the facility for Distributed Simulation Systems 1.0 by the Object Management Group (OMG) and is now in the process of becoming an open standard through the IEEE.
HLA's Runtime Infrastructure (RTI) is a set of software components that implement the services specifies by HLA. Today, a few RTI implementations for different platforms are available.

If you want to learn HLA... http://www.ecst.csuchico.edu/~hla/courses.html


DIS / HLA - military standards tailored to the requirements of simulation and war games. DIS is an
efficient, if inflexible, protocol for medium scale simulation. The imaginatively named Higher Level
Architecture (HLA) is the spiritual successor to DIS, although it focuses more closely on the
problems of arranging very large-scale simulations rather than the run-time distribution of data.
HLA remains a hot topic in defence circles. Limitations of the specification provide great
opportunities for the lab, with the prospect of serious defence dollars on offer for a full simulation
infrastructure

Slide 15 : Java Shared Data Toolkit (JSDT)

Java Shared Data Toolkit (JSDT)

Provides an abstract model of the network, designed specifically to support collaborative applications

Provides different modes of transportation

Pure Java, and hence 100% portable

Not specifically designed for 3D simulations and virtual environments
not tuned for high performance or low latency

Java Media Framework.

Allows you to manage audio and video streams within a Java program

Issue : most of the "real" work is done in native code (not portable).

Although not specifically designed for 3D simulations and virtual environments, JSDT is part of the Java Media APIs developed by Javasoft and provides real-time sharing of applets and/or applications. JSDT provides many facilities such as tokens that can be used for coordinating shared objects. It also provides different modes of transportation including a reliable socket mode, RMI mode which uses Remote Method Invocation, and a multicast mode that makes use of the Lightweight Reliable Multicast Protocol (LRMP) and is useful for shared application with a large number of participants.

Java and the Network...

Java and it’s associated media streaming and networking packages represent an alternative basis for developing collaborative virtual environments. The core class libraries provide socket and distributed object (RMI) models of communication. In addition, there are a number of standard Java extensions that provide specialised data distribution mechanisms. Two extensions are especially relevant to the task of building collaborative virtual environments: the Java Shared Data Toolkit and the Java Media Framework.

The Java Shared Data Toolkit (JSDT) provides an abstract model of the network, designed specifically to support collaborative applications.

"The JavaTM Shared Data Toolkit software is a development library that allows developers to easily add collaboration features to applets and applications written in the Java programming language."

"This is a toolkit defined to support highly interactive, collaborative applications written in the Java programming language."

The fundamental abstraction used in the JSDT is that of a shared byte buffer. These buffers are un-typed blocks of data, but with the use of Java object serialisation can be used to replicate complete graphs of objects. Buffer replication can be performed over a range of different network transports, including a reliable multicast system as well as RMI, sockets and even http. A great strength of JSDT is that it is pure Java, and hence 100% portable. The main limitation is that it is not tuned for high performance or low latency, so it’s suitability for building collaborative VEs is an open question. If object serialisation is used then JSDT will also have problems inter-operating with non-Java clients.

The second relevant extension is the Java Media Framework.

"The Java Media Framework (JMF) is an application programming interface (API) for incorporating time-based media into Java applications and applets."
"The Java Media Framework API specifies a unified architecture, messaging protocol and programming interface for playback, capture and conferencing of compressed streaming and stored timed-media including audio, video, and MIDI across all Java Compatible platforms."

Essentially, JMF allows you to manage audio and video streams within a Java program. The crucial point about JMF is that most of the "real" work is done in native code. Native platform codecs are used throughout which means that performance is excellent, but compatibility is harder to achieve.
The combination of the core Java networking classes, JSDT, JMF, a portable execution format (the Java class file) and automatic platform neutral serialisation mechanism make Java a very attractive base on which to build collaborative VEs. As with Java3D the principle concern is the runtime performance of Java. However, experiments conducted by the authors with early implementations of Java indicate that network bandwidth is a more common cause of performance bottlenecks : even a simple interpreted JVM can saturate a 155Mbit ATM network. Interoperability problems not-withstanding, Java should be considered an excellent environment for networking applications.

Texte extract from a non published paper by Sam Taylor and Hugh Fisher

Slide 16 : CavernSoft

CavernSoft

CAVERN, the CAVE Research Network, is an alliance of industrial and research institutions equipped with immersive equipment and high-performance computing resources all interconnected by high-speed networks to support collaboration in design, training, visualization, and computational steering in virtual reality.

C++ hybrid-networking/database library optimized for the rapid construction of collaborative Virtual Reality applications

VR developers can quickly share information between their applications with very little coding

The Information Request Broker (IRB) is the nucleus of all CAVERN-based client and server applications

An IRB is an autonomous repository of persistent data driven by a database, and accessible by a variety of networking interfaces

CAVERNsoft is most succinctly described as a C++ hybrid-networking/database library optimized for the rapid construction of collaborative Virtual Reality applications. Using CAVERNsoft VR developers can quickly share information between their applications with very little coding. CAVERNsoft takes over the responsibility of making sure data is distributed efficiently.

The IRB

The Information Request Broker (IRB) is the nucleus of all CAVERN-based client and server applications. An IRB is an autonomous repository of persistent data driven by a database, and accessible by a variety of networking interfaces. The goal is to develop a hybrid system that combines a distributed shared memory model with distributed database technology and realtime networking technology under a unified interface to allow the construction of arbitrary collaborative VR (CVR) topologies.

http://www.evl.uic.edu/cavern/cavernsoft/

Slide 17 : DIVE

DIVE

The Distributed Interactive Virtual Environments project from the Swedish Institute of
Computer Science is one of the original attempts to develop collaborative VE.

Allows its participants to navigate in 3D space and see, meet and interact with other users and applications.

Extensive use of Tcl

Provides persistence through the use of a centralised database server

Still under active development, DIVE is a well-known and widely respected toolkit. It is designed around the notion that the environment should serve both for interactions and for developing new material. It makes extensive use of Tcl to provide extensibility and provides persistence through the use of a centralised database server.

 

Slide 18 : MPEG4

MPEG4

 

MPEG-4 has future potential for use in CVEs, but is not a viable technology in the short to medium
terms, for the following reasons:

Slide 19 : DIS

VRML-DIS-Java

A possibly successful approach may be to adopt the model of the VRML-DIS-Java working group of the Web3D consortium. This group has attempted to adapt the existing Distributed Interactive Simulation (DIS) military protocols, for use in building shared VRML worlds. They have made extensive use of Java to achieve this integration, and their results are quite compelling. The limitation of this approach is the same as that of any DIS based CVE : DIS is an excellent protocol for distributing information about a simulated battlefield, but it is utterly inadequate when interactions do not involve high-velocity projectiles, explosions or weapons of mass virtual destruction.

Slide 20 : What Else

What Else ?

COVEN is a four-year European project that was launched with the objective of comprehensively exploring the issues in the design, implementation and usage of multi-participant shared virtual environments, at scientific, methodological and technical levels.

Avocado - developed at GMD, Avocado builds a field network abstraction over the Performer scene
graph. Distribution is achieved primarily through the use of networked routes between fields. This is
a very simplistic model for distribution, but one that works well for small applications. Avocado is
built on the Ensemble toolkit from Cornell

WorldToolkit by Sens8
"WorldToolkit is the leading cross-platform real-time 3D development tool."
(World2World, the Client-Server... WorldUp)

 

Virtools worldtoolkit

Tiwi : only for the Wedge...

3D WORKING GROUP : Parallel Graphics

The absence of a flexible, standard 3-D format represents one of those key barriers. By working with industry leaders and graphics experts, we plan to create a format that will do for 3-D on the Web what the JPEG format did for digital photography on the desktop."
>Parallel Graphics


ParallelGraphics a rejoint le 3D Working Group dont nous vous parlions il y a quelques jours. Ce groupement d'éditeurs devrait produire un standard web 3D pour le monde de la CAO. Voici une partie des menbres : Intel Corporation, 3Dlabs Inc., Actify Inc., Adobe Systems, ATI Technologies, The Boeing Company, Dassault Systemes, Lattice Technology, Mental Images, Microsoft Corporation, Naval Postgraduate School, National Institute of Standards and Technology (NIST), SGDL Systems, Inc., i3Dimensions, Tech Soft America/OpenHSF.
Patrick Gelsinger (Intel vice president and chief technology officer) à déclaré à propos de cette alliance :"The absence of a flexible, standard 3-D format represents one of those key barriers. By working with industry leaders and graphics experts, we plan to create a format that will do for 3-D on the Web what the JPEG format did for digital photography on the desktop."