eScience Lectures Notes : .


Slide 1 : 1 / 34 : Introduction to Virtual Reality

Virtual Reality

Source of information for this chapter

Le traité de la réalité virtuelle
By Philippe Fuchs, Guillaume Moreau, Jean-Paul Papin
Préfacé par Alain Berthoz, Professeur au Collège de France
Le traité de la réalité virtuelle
Volume 1 : Fondements et interfaces comportementales

Le traité de la réalité virtuelle
Volume 2 : Création des environnements virtuels & Applications

For free for students : see : http://caor.ensmp.fr/interlivre/index.php

INTRODUCING VIRTUAL ENVIRONMENTS / NCSA : http://archive.ncsa.uiuc.edu/Cyberia/VETopLevels/VR.Overview.html

Virtual Reality: A Short Introduction by K.-P. Beier : http://www-vrl.umich.edu/intro/index.html


Slide 2 : 2 / 34 : Introduction

Introduction to Virtual Reality

First let's have a look at a little video.

For the display device presentation, have a look at the Computer Graphics matching presentation

What is VR ?

Some idea...

"The experience of venturing inside a computer-generated "virtual" world, of immersion in data, is what happens in "virtual reality" or "VR." Through various display devices, data are transformed into 3D images you not only see but also hear or even touch."

"Virtual Environments: New Ways to Perceive and Explore Data..."

"In immersive VR, the user becomes fully immersed in an artificial, three-dimensional world that is completely generated by a computer."

"An artificial environment created by computers, in which people can immerse themselves and feel that this artificial reality really does exist." (Whatis.com, USA).

Various names : Virtual Worlds, Virtual Environments, Immersive VR...

Cyberspace. A consensual hallucination experienced daily by billions legitimate operators, in every nation, by children being thaught mathematical concepts... A graphic representation of data abstracted from the banks of every computer in the human system.
Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding...

Neuromancer by William Gibson

 


Slide 3 : 3 / 34 : First Examples

First Examples

See at a different scale, at a different time...

National Center for Supercomputing Applications (NCSA)

Become smaller than an atom or larger than the universe. Journey back in time and space to the birth of galaxies, watch ripples of gravity as black holes collide or travel through the human bloodstream. Build a city of cathedrals, mosques, and pyramids or listen to the sounds of chaos. All of these are possible within virtual environments.

 


Slide 4 : 4 / 34 : First Examples (2)

First Examples (2)

From abstract data visualisation to data exploration and to interaction with data

Human Interface Technology Laboratory

Computer visualization is reshaping how scientists evaluate and explore their data and promises to transform how students learn. Within the past four decades, dramatic advances in graphic display and high-end computing have made it possible to transform billions of bits of data into interactive, three-dimensional images that you can manipulate in real time. Here, all the display and feedback devices that make this possible are collectively termed "Virtual Environments. "

 


Slide 5 : 5 / 34 : First Examples

First Examples (3)

To Build or to Sell ?

Detroit Midfield Terminal Project

 

 


Slide 6 : 6 / 34 : First Examples (4)

First Examples (4) : Training

Ecole des Mines de Paris / SNCF

 

 


Slide 7 : 7 / 34 : History

History : terminology and popularisation

The term 'Virtual Reality' (VR) was initially coined by Jaron Lanier, founder of VPL Research (1989). Other related terms include 'Artificial Reality' (Myron Krueger, 1970s), 'Cyberspace' (William Gibson, 1984), and, more recently, 'Virtual Worlds' and 'Virtual Environments' (1990s).

THE LAWNMOWER MAN : 1992


* * * STARRING: Jeff Fahey, Pierce Brosnan, Jenny Wright, Mark Bringleson, Geoffrey Lewis, Jeremy Slate, Dean Norris
1992, 105 Minutes, Directed by: Brett Leonard

Oh dear. Virtual reality finally hits the big screen - but not exactly in the way sci-fi fans or virtual reality boffins would have liked it. VR boffins complained that Lawnmower Man created unrealistic expectations amongst the public of what to expect from their "LSD of the 1990s" (as once 'Sixties drug guru Timothy Leary dubbed it - talk about creating unrealistic expectations!). Sci-fi fans complained that the movie was mostly cack.
They had it right: the plot, a mishmash of horror and the latest in technological buzzwords, doesn't quite make the grade. Effects wise, this film deserves all the accolades it can get. The computer graphics are spectacular and this movie is the first time cyberspace really makes it to the big screen. However, one cannot help but leave the cinema feeling cheated, feeling that all those incredible graphics might have been better served by a better screenplay - perhaps William Gibson's Neuromancer...

 

JOHNNY MNEMONIC : 1995 ( From William Gibson : "Neuromancer" : 1984)


* * * STARRING: Keanu Reeves, Dolph Lundgren, Takeshi, Ice-T, Dina Meyer, Udo Kier, Denis Akiyama, Henry Rollins, Tracy Tweed, Don Francks
1995, 98 Minutes, Directed by: Robert Longo

Cyberpunk godfather William Gibson's work finally makes it to the big screen. Unfortunately it is too late and one wonders what would have happened had his Hugo award-winning Neuromancer been made into a movie shortly after its publication. The sad truth is that we have already had buzzwords such as virtual reality and cyberspace explained in (albeit lesser) films such as

A far more general and openminded Timeline

http://www.artmuseum.net/w2vr/overture/looking.html

The same, put in images, and in a still more open context : The cybernetics

http://www.artmuseum.net/w2vr/timeline/timeline.html

Summary

 


Slide 8 : 8 / 34 : Prehistory

Prehistory

 


Slide 9 : 9 / 34 : What it is NOT

What it is NOT

When 'Virtual Reality' is used in a confusing and misleading manner...

Usual image :

What the media usually show : a person with a visiocasque and a dataglove...


From : The Virtual Reality Homebrewer's Handbook by Robin Hollands


Photo by W. Fisher / S. Fisher
VIEW (Virtual Interface Environmental Workstation)
Courtesy of Scott Fisher, NASA-Ames Research Center

The Lawnmower Man

Synopsis:
A slow-witted teenager develops a super intellect when he becomes a guinea pig for a scientist experimenting with virtual reality.

1992, 105 Minutes, Directed by: Brett Leonard

JOHNNY MNEMONIC


* * STARRING: Keanu Reeves, Dolph Lundgren, Takeshi, Ice-T, Dina Meyer, Udo Kier, Denis Akiyama, Henry Rollins, Tracy Tweed, Don Francks

1995, 98 Minutes, Directed by: Robert Longo

 

The book :

Neuromancer

By the Cyberpunk godfather William Gibson

It could be, but it is not "only that", nor "always that"


Head-Mounted Display (HMD)

 


Slide 10 : 10 / 34 : TechnicalDefinition

Philippe Fuchs' Technical definition or VR

You may talk about VR for a system if and only if ...

The man/user is an observer and actor (meaning he has an action on ...)

Something to explore, fictitious or not

User action, Navigation; RT : the user perception

The user should  not perceive the delay between its action (motor) and the sensory response from the system. The max delay is about 100 ms (depends on which direction)(25 fps <=> 40ms)

Feeling of being somewhere

The "natural" is what you have learnt in the real life ... Even if we once achieve to reach the same level of complexity (and some think it is not possible), one may wonder if we really need to reach the perfection to achieve efficiency in regards to the expected result.

Input and Output, sensory-motor interface, Metaphors

To make a difference from the set "screen, keyboard and mouse" which match with the Human Computer Interface field ...

A Behavioural Interface is an apparatus that involve a human behaviour, natural and without (or with short) learning period.

All the senses may be taken into account, not all are needed for all the applications

NB : No "Why" (Answer : the Applications)

Other definitions often mix purpose, what it is and what it is for (applications). Moreover, one should not define VR by the tools it uses (Head mounted display, dataglove ...)

Not these all 4 points will be perfectly done in all VR application, but the 4 should just appear if we want to be able to talk about a VR project.


Slide 11 : 11 / 34 : Purpose

Purpose of Virtual Reality

To allow one or more participants to get involved into sensorimotor activities and therefore mental ones within an artificial world which is

Which activities will depend on the application. The higher degree of realism is not always what should be targeted. Less realism but better (intuitive and/or easy to dominate) interaction would be a better choice.

NB1 : This is not the answer to the question: "what is the use of it ?".

NB2 : Fictitious does not mean that this would be only for game (teaching/training situation).

NB3 : In Augmented Reality the artificial world is what is added.

The Purpose of VR is to allow one or several people to do sensorimotor, and thus mental, experiments in an artificial world, which is either imaginary, or a simulation of some aspects of the real world.


Slide 12 : 12 / 34 : Applications : a taxonomy

Applications

The Why Question, what sort of things we want to do with VR

Phippe Fuch propose a taxonomy

VR is of no interest at all if you strictly and exactly reproduce each aspect of the real world.

You modify at least one of these 3 main aspects

Time

Place

"Realistic vs Fictive" Artificial World and InterAction

NB


Slide 13 : 13 / 34 : Application examples 1 : IAr

Application Examples 1 : IAr

IAr.T0.L0 : Virtual Activity

Training or fun. Game, sport, surgery, army

IAr.T0.L-> : Virtual Transfer

Tourism

IAr.T0.L~ : Virtual Microscope or Telescope

Science

IAr.T+.L0 : Virtual Design

Architecture, Fashion, Product Design

IAr.T-.L-> : Virtual Event

Archeology, Police Investigation

 


Slide 14 : 14 / 34 : Application examples 2 : IAr

Application Examples 2: IAi

IAi.T0.L0 : Virtual Creation

Art, Database visualisation ...

IAi.T0.L-> : Fictive navigation

Structure (Directory Tree) visualisation. FSN !!!

IAr.T0.LU : Televirtuality

IVR, where your being is replaced by an avatar

 


Slide 15 : 15 / 34 : Main Scientific Issues

Main Scientific Issues

To allow one or more participants to get involved into sensorimotor activities and therefore mental ones within an artificial world which is

 


Slide 16 : 16 / 34 : Information Transfert within the Human Body

Information Transfert within the Human Body


Slide 17 : 17 / 34 : 5, 6, 7 ... senses

Five senses... but they were 7... or more ?

hearing

sound

ears,body


The ability to hear; the auditory faculty; SYN. audition, auditory sense, sense of hearing, auditory modality.
ouie

sight

image

eyes

The ability to see; the faculty of vision; SYN. vision, visual sense, visual modality. vue

touch

surface / temperature

skin

The faculty of touch; SYN. sense of touch, skin senses, touch modality, cutaneous senses. toucher

smell

odour

nose

The faculty of smell; SYN. sense of smell, olfaction, olfactory modality. odorat

taste

savour, flavour

tongue/nose

The faculty of taste; SYN. gustation, sense of taste, gustatory modality. gout

kinesthesia

position, movement, muscular tensions

muscles

The perception of body position and movement and muscular tension etc; SYN: kinaestesia, feeling of movement kinesthesie

proprioception

balance, acceleration, position, location, orientation, movement of the body

ear

The ability to sense the position and location and orientation and movement of the body and its parts. proprioception

 

 


Slide 18 : 18 / 34 : Anthropocentric User-Diagram

Anthropocentric User-Diagram

Or "how we perceive the world and how I am in the world"

Or "I feel therefore I act" ;-)

Immersion

 

 

Input

 

Perception

Interactivity

 

 

Output

 

Action

 

 


Slide 19 : 19 / 34 : Human Behavior in a virtual world

Human Behavior in a virtual world

...should follow the same principle than in a real world :

Immersion and Interactivity

3 level of Analysis


Slide 20 : 20 / 34 : First Level

First Level of Immersion / Interactivity

Low level, Physical level

Because the computer is physically connected to the human body by its senses and its motor responses.

The physical devices, Which interface to use ?

You have to know that not only Sight interface exist (HMD...), but lots of other, you should have an idea of all the others

What Hardware (interface, computer) ?

What Software (Driver, Toolkit, Programming Language...) ?

TECHNOCENTRIC DESIGNER DIAGRAM

NB : By now (and still in the future), human characteristics are higher than what the technical VR device may provide to the user...

Notion of Transparency : with (touch) or without (sight) material medium

Target : try to minimise the delay for the feedback loop : "Action / Perception"

Are the metrology characteristics in adequation with the psychophysics characteristics ?

(example of the Head Mounted Display)

 


Slide 21 : 21 / 34 : Sensory Interface

Sensory Interface


Slide 22 : 22 / 34 : Motor Interface

Motor Interface


Slide 23 : 23 / 34 : Sensorimotor Interface

Sensorimotor Interface

Usually, force feedback interface (kinesthesia sens)

Receptor inside the muscles

Another example : eye driven pointer


Slide 24 : 24 / 34 : Canesta Keyboard : another kind of input device

Canesta Keyboard : another kind of input device

http://www.canesta.com/products.htm

The Canesta Keyboard is the world's first projection keyboard capable of being fully integrated by OEMs into smart phones, cell phones, PDAs, or other mobile or wireless devices (view clip) . When equipped with the Canesta Keyboard, the OEM device uses a tiny laser "pattern projector" -- also developed by Canesta -- to project the image of a full-sized keyboard onto a convenient flat surface between the device and the user, such as a tabletop or the side of a briefcase. The user can then type on this image and Canesta's electronic perception technology will instantly resolve the user's finger movements into ordinary serial keystroke data that is easily utilized by the wireless or mobile device. The Canesta Keyboard Perception Chipset™ includes all the modules needed for a projected keyboard product including: the Canesta Keyboard Sensor Module, the Canesta Keyboard Light Source, and the Canesta Keyboard Pattern Projector.
The Canesta Keyboard is an important new application that resolves the "missing link" with mobile and wireless devices -- the ability to do "true" data input. Current input solutions such as thumb keyboards or handwriting recognition, though popular, are limited in their ability to support typing-intensive applications such as document and memo creation as well as email composition. An integrated projection keyboard means that the mobile or wireless device can now support applications that previously would have only been practical with a full-sized, mechanical keyboard. This is good news for OEMs that wish to differentiate their products with important, new mobility applications, and good news for service providers, that now can offer value added services to their subscribers, including "leave your notebook PC at home."


Slide 25 : 25 / 34 : First Level

First Level of Immersion / Interactivity

Low level, Physical level

Because the computer is physically connected to the human body by its senses and its motor responses.

The physical devices, Which interface to use ?

You have to know that not only Sight interface exist (HMD...), but lots of other, you should have an idea of all the others

What Hardware (interface, computer) ?

What Software (Driver, Toolkit, Programming Language...) ?

TECHNOCENTRIC DESIGNER DIAGRAM

NB : By now (and still in the future), human characteristics are higher than what the technical VR device may provide to the user...

Notion of Transparency : with (touch) or without (sight) material medium

Target : try to minimise the delay for the feedback loop : "Action / Perception"

Are the metrology characteristics in adequation with the psychophysics characteristics ?

(example of the Head Mounted Display)

 


Slide 26 : 26 / 34 : Second Level of Immersion / Interactivity

Second Level : Mental Immersion / Interactivity of the user

Triadic diagram for the behavioural interfacing

you may just hear a sound (Schema by substitution) instead of feel the touch of an object


Slide 27 : 27 / 34 : Sensorimotor Schema

Sensorimotor Schema

What is natural seems simple and it has already been learnt

For Piaget, a schema is a mental structure which enables to repeat the same action or to apply it to new actions.

(a behavioural brick ...)

Piaget : specialist in Child learning behaviour, the birth of the intelligence 1979

Imagine that you say to somebody that if he wants to stop his computer, he first has to click on a start button ... that is not natural at all, it is counterintuitive !

There is in it some parameter that describe a general situation where you may want to use that schema

Ex : When you grab an object, the same scheme may be used in different situation

The behavioural interface is a mixed entity :


Slide 28 : 28 / 34 : Different levels of abstraction

Different levels of abstraction

In VR Schema are often slightly modified

Ex : when you grab and move around a position tracker instead of an object and you see the object on the screen, not in your hand.

Schema by substitution

When you hear a sound instead of touching or when a colour tell you that it is hot or dangerous.

Metaphor

When you show where you want to go (The superman Metaphor)

When you say to the user, to get that, you should do that, because it is as if you were in such another situation (The "world in your hand metaphor")

Driving a car could either be a schema, a schema by substitution (you say : "drive me to the biggest house") or a metaphor ("use the keyboard to steer and change the speed and you will move as if driving a car")

When using what ?

The best is of course the Schema (Natural immersion), but it is not always possible or of good quality, and often the other solutions are then better (more efficient)

When a metaphor become a schema ...

From the desktop to the web crawling ("click on the blue text to get" or ... to "get the page") and to the move around ...


Slide 29 : 29 / 34 : BSA : On the Virtual World Side

On the Virtual World Side

BSA = Behavioural Software Assistance

When the virtual world guess what you are trying to do and help you to do it

When you play a card game and the software put the card exactly at the right position, even if you put it 1 mm away.

From Hugh :

Real world signposts are often difficult to read due to angle or darkness. In a VE they can be billboarded so the signpost is always facing the user, no matter where they are.

Example : icon alignment

3 stages of the computer interface evolution by Dr. Kim Silverman

Dr. Kim Silverman
Principal Research Scientist
Manager, Spoken Language Technologies
Apple Computer

VBP = Virtual Behavioural Primitive

Everything that you will ever do in a VR is a combination of one of the following actions / VBP


Slide 30 : 30 / 34 : Second Level of Immersion / Interactivity

Second Level : Mental Immersion / Interactivity of the user

Triadic diagram for the behavioural interfacing

you may just hear a sound (Schema by substitution) instead of feel the touch of an object


Slide 31 : 31 / 34 : Second Level Graphe

Second Level Graphe

BPVR, VBP = navigation, object handling, visual observation, locomotion, orientation, etc.

BSA = Behavorial Software Assistance

 


Slide 32 : 32 / 34 : Technocentric Designer Diagram

Technocentric Designer Diagram


Slide 33 : 33 / 34 : Final evaluation of a virtual world

Final evaluation of a virtual world

Have you had the feeling that you were completly immerse inside another world ?

VS.

Did the user behave the same way (or with enough efficiency) than in the real world, regarding what we where looking at ?

(Move, Act is different from behave : result centric)


Slide 34 : 34 / 34 : Glossary

Glossary

French English Comment (Most of the time, "Ultralingua dictionary" definition)
Interface Interface

+. (Computer science) Hardware and associated circuitry that links one device with another (especially a computer and a hard disk drive or other peripherals);
+. (Computer science) A program that controls a display for the user (usually on a computer monitor) and that allows the user to interact with the system;

In VR field, it is a bit more complex, and it is one of the subject of this lecture to define it.

A tool that allows a person to interact with a computer. For example, a mouse is an interface device that allows you to put information into a computer. Virtual reality includes interface devices such as head-mounted displays that transmit sensations of the artificial world, as well as transmitting information into the computer.

Interfacage Interfacing The fact to set up a interface
Sensorial Sensory Involving or derived from the senses; "sensory experience"; "sensory channels"; SYN. sensorial.
Sensorimoteur Sensorimotor Of or relating to the sensory and motor co-ordination of an organism or to the controlling nerves.
Motrice Motor Involving or relating to movements of the muscles; "motor co-ordination"; "a motor reflex"; "motor paralysis."
Comportemental Behavioural Of or relating to behaviour; "behavioural sciences";
Comportement Behaviour The action or reaction of something (as a machine or substance) under specified circumstances; "the behaviour of small particles can be studied in experiments";
Teleoperation Teleoperation, Remote Control Remote control of robot manipulators
Simulation Simulation. A computer model of a real phenomenon or system. The system is described by a set of mathematical formulae or models in a computer program. Running the computer program shows how the system works and, by changing variables, it is possible to make predictions about how the real system will change. When there are many variables, simulation is often the only way to reasonably predict an outcome.
Interaction Interaction A mutual or reciprocal action; interacting.
Interactivite Interactivity The quality of that which is interactive.
Effecteur Effector
1. A nerve fiber that terminates on a muscle or gland and stimulates contraction or secretion.
2. An organ that becomes active in response to stimulation.
metrologie metrology The science of, or a system of, weights and measures; also, a treatise on the subject.
psychophysique psychophysics The branch of psychology concerned with quantitative relations between physical stimuli and their psychological effects.
scheme schema An internal representation of the world; an organisation of concepts and actions that can be revised by new information about the world.

artefact

artifact

A man-made object;