Virtual Reality

Course overview

Course Summary

Duration: 12 weeks, 84 hours

Project will be written after course completion

Course Description

OpenGL and WebGL, real-time rendering, 3D display systems, display optics & electronics, IMUs and sensors, tracking, haptics, rendering pipeline, multimodal human perception and depth perception, stereo rendering, presence. Emphasis on VR technology. Hands-on programming assignments. The 3-unit version requires a final programming assignment in which you create your own virtual environment.


Strong programming skills.


Basic computer graphics / WebGL / OpenGL.

Course Catalog

Topics include:

  • Introduction to computer graphics
  • The graphics pipeline
  • OpenGL, WebGL, and GLSL shader programming
  • JavaScript with Three.js
  • Stereoscopic perception and rendering
  • Head mounted display optics and electronics
  • Inertial measurement units: gyros, accelerators, magnetometers
  • Sensor fusion: complementary filter, Kalman filter
  • Human perception: visual, audio, vestibular, tactile, ...
  • Guest lectures from VR experts (e.g., Oculus Research)
  • ... more interesting topics.

Helpful Background

This course requires programming experience (especially JavaScript, C or C++) as well as basic knowledge of linear algebra. We will review most of the required mathematical concepts, but we assume that you have strong programming skills. Homework require JavaScript and GLSL programming. Even though we don't assume prior experience in JavaScript and GLSL, you should be able to understand object-oriented programs and modify them. Previous knowledge of computer graphics or computer vision would be helpful but is not absolutely required. 

Course Goals & Equipment

This is a technical class. Students will learn about all hardware (optics, electronics, display, microcontroller, ...) and software (JavaScript, WebGL, GLSL) aspects of Virtual Reality (VR). The goal for this class is to learn all of these aspects in a hands-on manner. Each assignment is a small piece of a bigger project. The goal for each student or small team of students is to build a fully functional head mounted display, including optics, display, IMU, rendering, lens distortion shader, model loader etc., from off-the-shelf parts. The HMD we will build is inspired by the Oculus DK1. We will NOT learn game development, but study the fundamental building blocks of VR systems and implement all of them. 

Tentative Syllabus

Lecture 1: Introduction to VR and AR

Overview of class, logistics, history of VR/AR

Lecture 2: The Graphics Pipeline and OpenGL I: Overview and Transformations

Rotation, translation, scaling, modelview matrix, projection matrix

Lecture 3: The Graphics Pipeline and OpenGL II: Lighting and Shading

Lecture 4: The Graphics Pipeline and OpenGL III: OpenGL Shading Language (GLSL) GLSL vertex and fragment shaders

Lecture 5: The Human Visual System

Perception of depth, color, contrast, resolution

Lecture 6: The Graphics Pipeline and OpenGL IV: Stereo Rendering

Lecture 7: Head Mounted Display Optics I

Magnifier designs, stereo rendering for HMDs, lens distortion correction

Lecture 8: Head Mounted Display Optics II

Advanced HMD optics

Lecture 9: Inertial Measurement Units I

Gyros, accelerometers, magnetometers

Lecture 10: Inertial Measurement Units II

Sensor fusion, complementary filter, arduino

Lecture 11: Positional Tracking I

Tracking with the light house

Lecture 12: Positional Tracking II

Advanced positional tracking

Lecture 13: Spatial Sound

Lecture 14: Haptics

Lecture 15: Panoramic Imaging and Cinematic VR

Lecture 16: VR Engines and Other Aspects of VR

(Latency, eye tracking, post-rendering warp,...)