For many of us, the term virtual reality conjures images from popular culture, such as the gleaming techno-world in the Tron films or the hyper-realistic environment in The Matrix and its sequels. Or it may make us think of next-generation gaming, which promises to bring us headsets that allow users to step right into the game.
Some version of each of these is in our future, with little doubt. The recent purchase by Facebook of Oculus, maker of the Oculus Rift device, which has earned rave reviews from beta users, demonstrates that the tech giants are anticipating and positioning themselves for an explosion of VR that is likely to occur over the next half-decade. Google Glass, which is technically an “augmented reality” device – allowing overlay of digital images rather than substitution of the entire field of vision – also points in this direction. Google and others are experimenting with contact lens that can convey visual prompts and perform other tasks, and many companies are working on devices that provide more intuitive and responsive interactions with computer systems, so it is not difficult to imagine a future where the devices for delivering a VR-like experience are nearly invisible and ubiquitous.
While gaming and the ability to explore dreamed-up worlds will likely drive a great deal of VR development, also interesting to consider is how VR will infiltrate many of our daily activities. Video conferencing is an obvious entry point. Slightly less obvious (but not, apparently, if you are Facebook) is the opportunity to interact virtually with your online social network. Education? Wouldn’t an art teacher in Kansas jump at the chance to allow his or her students to stroll through the halls of the Louvre? Virtual attendance at sporting events? Check. Participating in a political rally halfway across the globe, or visiting several vacation spots you’ve been dreaming about, in order to choose one to actually visit? Check, and check.
Google Street View
Wait a minute, you say – I can already visit many locations on Earth online, to some extent. With the “Street View” function on Google Maps or Google Earth, anyone can view ground-level, 360-degree images of a staggering number of highways and byways, including most of North America and Western Europe, Japan, Australia and New Zealand, and portions of Central and South America, Africa, Asia, the Middle East and Eastern Europe, for a total of nearly 50 countries worldwide (see this map for current coverage). While the images are static, the ability of a user to “look” in each direction, and to see buildings and landscapes from multiple angles, is a hint of things to come.
So, is Street View a form of “virtual reality”? Perhaps, in a rudimentary form, because it simulates a user’s presence in a rendered world. But the type of VR that is anticipated to become more ubiquitous over the next decade is far more immersive. I tend to like the simple definition NASA came up with many years ago in connection with its virtual windtunnel project: “Virtual reality is the use of computer technology to create the effect of an interactive three-dimensional world in which the objects have a sense of spatial presence.” A two-dimensional image such as that rendered in Street View, in its current form, does not provide much of sense that the objects in the images are real objects.
Street View images are captured by a special rig consisting of nine directional cameras mounted on top of various types of vehicles. The rig also houses three lasers that allow Google to calculate distances to objects in the images, in order to create temporary 3D models that facilitate the transformation of the individual photographs into a “stitched” 360-degree image. (In the past Google’s rigs also controversially collected data from unencrypted wifi networks, including the content of emails; a class action lawsuit alleging violations of the Wiretap Act is ongoing, and Google recently sought certification to the U.S. Supreme Court after the 9th Circuit ruled against it.)
Google’s vehicles with Street View camera rigs have been prowling the streets since at least 2007. Google has introduced similar camera mounts on special “trikes” (pedal tricycles), boats, snowmobiles, and even backpacks, to cover harder-to-reach locations. Google Earth also offers some underwater locations, as well as locations on the moon and Mars, although the images for these locations were not collected by Google. Its “Cities in 3D” option allows users to view three-dimensional images of buildings in many large cities, automatically generated from aerial photography. And, of course, Google offers satellite imagery in its Maps and Earth programs. Finally, the company encourages third parties to contribute imagery to be presented in its programs.
Google’s Street View program began with a partnership with a company known as Immersive Media, which – as its name suggests – was and is known for pioneering 360-degree video capture and playback. However, Google later jettisoned Immersive Media and created its own camera rigs and software. Interestingly, Immersive Media’s cameras collected video rather than static images; Google’s public statements about its current program suggest that it only collects static images.
Google introduced its “Street View” option to users in May 2007. No doubt the motivation was in large part to increase the value of its existing map programs to users, and in turn deliver more viewers to the ads that appear within the programs. But, whether by intention or inadvertence, Google also has quietly been doing something else: It has been warehousing a vast amount of data about the physical world that might be used to recreate that world in virtual reality.
The data
Combine all of Google’s imagery of the physical world, and you can put together a pretty good portrait of many permanent and semi-permanent features of many locations on Earth. There is also a record of more transient items – such as people and automobiles – at the moment the photos were snapped (Google applies software that automatically blurs faces and license plates in the images it makes available in its application, to address privacy concerns).
Could you then combine those various pieces into a convincing VR environment? Techniques certainly exist that allow photographs to be used as a basis for creating three-dimensional models of what is seen in the photograph, especially when there are multiple images snapped of the subject matter, from different perspectives. Indeed, as noted above, Google measures distances with lasers and actually creates rough 3D models in the course of stitching its photos together, so this may aid in rendering virtual environments from the data. Computing speed should ultimately not prove problematic, either, as suggested by a recent Microsoft demonstration of code that could convert a virtual scene into a convincing Oculus Rift demonstration, all processed in the cloud.
The point is that if it so desired, Google likely could take the data it has collected and convert it into more fully-rendered 3D visuals for other purposes, or sell its data to others that want to do so.
And it can do so for periods of time back to at least 2007. This is important. Even if other companies are mapping areas of the world in detail for use in 3D rendering (and some are), Google has something no one else possesses: extensive coverage of the world through time, for the past seven years. Recently, it began making this archival Street View imagery available in Google Maps, allowing users to view past Street View images using a “time slider” feature (it already provides historical satellite imagery through a similar feature). While Google cannot go back in time and record the world digitally prior to 2007, chances are good that Google will maintain this partial monopoly on extensive worldwide imaging data for many years.
As an aside, one might also note the variety of mass digitization efforts the company has been undertaking in other areas (such as the Google Books project) and consider that its efforts to collect as much historical data as it can across many disciplines may be, at least in part, with an eye towards eventual deployment in VR environments as well.
The use of Street View imagery for VR is not hypothetical – it is already occurring, in limited ways. Programmers have begun developing preliminary Oculus Rift environments around Street View.
What does Google really own?
But Google cannot really own its digitization of the physical world, can it? The answer is: No, and sort of.
No, Google cannot obtain any rights to the objects or places in its photographs, any more than you own the Eiffel Tower after snapping a picture of it with your camera. Nor can Google prevent anyone else from taking their own pictures of the same vistas.
Indeed, under copyright law, there is a very good argument that Google does not even acquire a copyright interest in the photographs themselves. Copyright protects works of authorship, including photographs, that display a minimal amount of creativity. For photographs, dating back to 1884 the U.S. courts have said that this creativity arises from a photographer’s choice of lighting, camera angle and composition, etc. – in other words, copyright protection extends to those creative elements provided by the photographer, not to the underlying subject matter reproduced in the image. As a practical matter most photographers, by virtue of their creative choices, hold a sufficient copyright interest to prevent others from reproducing or otherwise exploiting their photos without permission.
However, this protection does not extend to works that are the result of “slavish” imaging – for instance, using a digital scanner to copy something does not give the operator of the scanner a copyright in the resulting image (notably, this may not be true in countries other than the United States – Italy, for instance, allows photographers to claim protection even in highly faithful photographs of old, public domain artwork). Thus images captured by traffic cameras and webcams likely are not protectable by the owners of those cameras, under copyright law. One would expect the law to treat Google’s Street View images – automatically taken from a camera rig attached to the roof of a vehicle – the same way.
Interestingly, Google itself states that Street View imagery may not be incorporated into a third-party “project” unless “the imagery comes directly from” the Google Maps program or “the imagery is embedded or linked to on your website using the HTML and URL provided on Google Maps.” Google states that users cannot print copies of Street View imagery except for personal use; 10 or fewer images may also be printed for academic, non-commercial use. While such restrictions may not be enforceable under copyright law, they may be enforceable under contract law – once a user accesses the images through Google’s programs, the user generally must abide by Google’s terms of use, which impose restrictions on use.
What Google likely does own, under copyright law, is protection for the organization and arrangement of its database of information collected through its Street View and other image collection efforts, assuming the organization of that data is itself sufficiently “creative.” Data organization that simply follows a logical or natural organizational scheme – to take a simple example, if the data is merely organized by latitude and longitude and camera direction – likely would not be protectable. But a more complex organizational structure does provide at least “thin” copyright protection for that structure, as highlighted by the Supreme Court in the Feist Publications v. Rural Telephone Service Co. case which found protection for certain aspects of printed telephone directories.
Copyright protection for databases does not extend to the data itself; rather, copyright law treats facts about the physical world as non-copyrightable subject matter. This means that the visual “record” of where items in the physical world are situated relative to one another, stored as a string of ones and zeroes in computer memory, is not protectable by copyright law. If someone had access to that raw data, presumably they could recreate the relative physical locations and sizes on a three-dimensional grid without fear of an infringement claim by Google.
It’s that “access” which may prove tricky.
Presumably more significant to Google than any intellectual property protection is simply its physical ownership of the database itself. No one else possesses Google’s server farms stuffed with the data gathered from the Street View mapping project. No one can legally harvest that data from Google’s map programs without Google’s permission. Google would also likely have other legal theories beyond breach of contract that it could pursue if someone tried to scrape its imagery and harvest the underlying data on a massive scale, including trespass or theft of trade secrets.
To summarize, Google does not “own” the virtual world. But it has a nice head start on being positioned to render significant portions of the real world virtually.
What’s next?
Google’s efforts are merely an indication of what’s in our near future. We will see many, many efforts to map our world for conversion into VR, as well as efforts to map existing fictional and artistic spaces for the same purpose. Indeed, crowdsourcing of VR landscapes surely will begin occurring in earnest as well; ball cameras and other personal devices for capturing 360-degree images are in the works.
Photographers are already aware of the legal issues that may arise around the use of photographs that contain images of individuals, or buildings, or artwork, or branded products – these concerns will spill over and perhaps be amplified in a VR environment, especially given the ability for users to interact more directly with such items in VR. For instance, if I am in a crowd in Times Square when it is being mapped photographically, and an avatar with my face and body is generated in a VR application, one can imagine many uses to which that avatar might be put that would be objectionable to me.
What about physical objects converted into VR objects? Is a representation of a patented machine that “functions” in VR like the real-world object an infringement, despite the fact that the physical mechanisms that are present in the real-world object do not exist in the virtual representation? (Imagine a VR stapler: it appears to function like a real stapler but likely has no actual internal parts.) Is a VR version of copyrighted sculpture an infringement of the original sculpture?
Turning to fictional works, where a VR implementation incorporates narrative elements from original work, or visuals from the original work, clearly permission from the original authors will almost certainly be needed to create the virtual embodiment of those works. But in some cases a virtual world that is merely reminiscent of the original work might be created without using much if all of the original content, leading to questions of whether the new work is a fair use under copyright law because it “transforms” the original. And immersive environments will often need significant additional, original content in order to fully realize the virtual world, raising complicated issues of ownership of elements of the new worlds.
Already, an Oculus Rift developer has created a virtual version of Jerry Seinfeld’s apartment from the television series Seinfeld. This environment does not include the characters from the show – Kramer does not burst into the room and Newman is not lurking in the corner – nor does it recreate any narrative from the show, so in many respects it simply looks like an ordinary New York City apartment. But would this constitute a derivative work based on the set design for the original show (in which case permission likely would be needed to create it), or would it be sufficiently “transformative” to be considered a fair use? Would a VR environment that closely imitates the look and feel of, say, Salvador Dali’s surrealist art be actionable? What about the insertion of characters from prior works? Or “mash-ups” of multiple fictional environments, or of fictional and “real” environments? Or VR environments based on historical photographs?
Google may be leading the first charge into this uncharted (pun intended) territory, but the frontier is wide open.
