Years ago I had built a hacked-up set of Virtual Windows out of engineering scrap.  And it sort-of worked which led to dreams of full motion HD video on high quality displays with properly hidden cables.  Combined with a motivation to remodel the room, plans began in the spring of 2009 to build the best Winscape I could.  Modeling began in Sketchup.  Wives were consulted.  Cocoa, Quicktime, OpenGL, and Bluetooth programming started to be learned.  Tools were bought.


Fast Forward over a lot of room remodeling: the fireplace brick face was removed, the fireplace was sealed up, a new wall was built over the fireplace, ceiling lighting was installed, ethernet was run, the cracked concrete floor was leveled, and a wood floor was installed.

History:

Embedding two 80-pound plasma screens inside of a wall has some unique challenges.  Namely, how does one access them if they need servicing, and how does one keep them from overheating?


Access is very troubling, indeed.  After lots of ideas and CAD modeling, I built the facade wall in sections so that if I really needed to, I could crack the plaster joint and remove a section of the wall.  Thinking with more optimism, I designed each window cavity with a narrow margin through which a few sets of fingers could possibly grip a display enough to lift and tilt it out of the cavity.  I won’t find out if the design works until a display needs servicing.

Display Challenges:

Tracking a viewer:

There are various options for tracking an object or person in a room.  Each has their advantages and disadvantages, but some of Johnny Chung Lee’s work with the Wii remote (Wiimote) makes it a very compelling option.  The tracking of multiple IR points is built into the Wiimote firmware.  The Wiimote has a fairly well understood interface via Bluetooth thanks to the reverse engineering efforts by the internet community of enthusiasts like Johnny Chung Lee.  It’s also relatively inexpensive.  I suppose you could even use multiple units if you wanted to.  The Wiimote results are not very accurate and this impedes the illusion being presented to the viewer.  A pricier, more accurate tracker would make the illusion perfect.  Someday I’ll have to work on that.


The Wii remote needs 3.0 volts of DC which is what it gets off of batteries.  I don’t want to have to change batteries, so a simple LM1117-based voltage regulator circuit takes a 3-5v wall wart source and brings it down to a steady 3v. 


Starting with the code from the Darwiinremote project, I hacked it up until it behaved like I needed.  I removed the auto-discover mode that the project likes to use and instead initiated a connection to a known MAC address over Bluetooth.  The discovery process was never reliable for me.  I changed the Bluetooth communication calls to use async callback functions so that the main thread executing Bluetooth communications wouldn’t hold back my video rendering.


OpenGL and Quicktime:

To display a HD video onto two different screens, one can share the Quicktime context with the OpenGL contexts so that they would all have access to the frames at the same time.  This works out great and uses hardware acceleration as much as possible.


The user’s head location (provided by IR via the Wiimote) is used to calculate a proper viewing frustum in OpenGL.  Since the view vector is non-orthogonal to the image plane, we have to do some extra work in OpenGL.  It’s just like rendering the wall of a CAVE. 


My OS X application’s preferences panel allows the user to specify the physical layout of the displays in relation to the tracker origin (Wiimote) in a table format.  This way, any gaps between the displays are automatically calculated out of the rendered views and the tracking data is now in relation to the displays.


The video card in the Mac Pro supports OpenGL textures up to 4096x4096.  This is great for high-res still photos, but as far as I can tell, there’s no hope for sending 4K video over the bus in real-time.  If someone knows otherwise, please let me know.

Power:

HDTVs aren’t as smart as computer monitors when it comes to power savings.  Computer monitors can sense a signal on their input and automatically turn on/off based on that signal.  HDTVs aren’t as convenient.  I use a Smart Strip to automatically turn off power to the displays, Wiimote, audio amp, and fans whenever the computer is asleep or off.  While the Smart Strip worked far better than the APC Power-saving Surge Arrest did at detecting the power draw of the Mac Pro tower, it still isn’t 100% reliable at detecting the difference between sleep and wake power draw.  One approach to make this more reliable would be to replace the relay in the Smart Strip with a lower voltage relay capable of being driven by SATA power.  This way the strip doesn’t have to sense power draw.


The displays are set to automatically turn on whenever power is restored to them.   The Mac is doing an OK job at handling the displays in this manner.  I’m having some difficulty with what I guess to be the computer waking from an HDMI signal glitch as the displays power off.

High Resolution Video:

After experimenting with consumer HD video equipment, I was disappointed in the resulting lack of detail.  While the Winscape cannot currently handle 4K video to the displays, they do greatly benefit from 1080p video with as much detail as possible.  After getting excited by the capabilities of the RED One camera, I hired a camera rental for a day in San Francisco.  The results are 4K video that can be down-sampled to 1080p with much higher detail than I could get with even broadcast-level professional HD video equipment.  I’m using a Canon 7D now, but it of course can’t compete with the RED One.


Check out the scenes page for tips on shooting footage and examples of the scenes I’m using.

 
I created a head tracking class in my OS X application that performs sanity checking on the data retrieved from the Wiimote.  If the X,Y,Z data was outside of the pre-configured bounds of the room, the data would be clipped.  Since Z data (distance to the IR points) is calculated based on the distance between X,Y data of two points, the Z data is pretty noisy.  I added a simple temporal filter to smooth out the Z data.


The circuit to the left simulates a Wii sensor bar.  Two clusters of IR LED emitters spread out the IR light via nine IR LEDs so that the “bar” is visible to the Wiimote from just about any normal orientation of the viewer.  A switch and red LED indicate whether the circuit is powered or not.  This board hangs from the viewer’s neck like a necklace.

 

Mounting the displays to the wall in portrait orientation was also a challenge since all HDTV viewing is done in landscape orientation.  With some minor modifications for bolt width, brackets from Peerless Mounts did the trick (PLP-V4X3 with SF16D). 


Plasma displays consume between 250 and 430 watts of power each and therefore generate quite a lot of heat.  That heat needs to be vented from the cavity to avoid damage to the electronics and fire risk.  I mounted two PVC plastic pipes above the displays that punch through the wall into the garage in the next room.  The pipes have very quiet fans on the end of them to pull the warm air out of the cavity into the garage.  The pipe below the displays serves as a wiring conduit to connect the displays and peripherals to the computer in the garage.  Cool air can enter the cavity via the wiring conduit or via slats underneath the bookshelf.