Project JabberWocky

Home entertainment center – remoteless control

Dec 2013 – Late on a cold Sunday afternoon the battery on the receiver remote had run out and my football team was’nt doing too well, started hunting around for a pair of AA’s. Without fail had a carton full of AAA’s but not a single AA. So grabbed some paper/pen, a pot of coffee and wrote down ideas I wanted to do.. Started browsing a bunch of youtube videos for inspiration. Imagine something like this you walk into a room and with a few voice commands can bring the whole entertainment system to life (TV, Roke, Cable, receiver, blu ray players, ) and being able to control every single one of them through gestures/voice seamlessly. Surprisingly no one seems to have implemented a system like this yet – there is Kinect which can control only Kinect but not really one system to rule them all.  Welcome to Project Jabberwocky – a full remoteless control system for home entertainment.

Project Jabberwocky – Why the name ?

There is an episode of ABC’s short run “Better of Ted” series called “Jabberwocky”. Ted moves some money around and creates this fictitious project called Jabberwocky to explain it. It culminates into a series of funny presentations and wrong turns. If you have’nt seen it you should 🙂 Am sure google would come up with a more profound meaning for Jabberwocky.


Jabberwocky

 

2008 – Some background on me :  I have been working on and off robotics projects for a while as a hobby primarily on Vision Recognition (OpenCV, Emgu CV), experimented with some Lidar equipment from Hokuyo, overlays of video on top of depth data from Hokuyo etc. All of these were with various degrees of success but expensive to become reality for the masses. Fastforwarding to 2011 became interested in the latest Kinect/Prime Sense/OpenNI stack this was a true sub $300 pseudo Lidar equipment with stable libraries and APIs. Anyone who has done any work on OpenCV knows the complexity of visual processing. This is a field comprising of  advanced math, fnding the best algorithm to do the job, image processing structures, transient states and so many other real world variables and is a true software engineering challenge. OpenCV was also how I got in touch with classmates from 20 years ago who were the best buds in high school, lo and behold he was an OpenCV Contributor.

Shout out to Shameem
http://www-personal.umich.edu/~shameem/haarcascade_eye.html

 

 

In the real world I work in the advertising industry and have been doing so for the past 10 years of my 14+ year career so far.  Project Jabberwocky is a two part system – a Home Remoteless control solution and a secure safe data aggregator solution for Linear/Digital Advertising.

With an Idea and not that big of a budget started planning, first investment was to buy a 7ft white board,  I still find the need to have physical things to doodle on.

Planning

Planning was the fun part about Organizing and Finding what the current state of things, even basics such as choosing what OS to use.

Sensors Evaluated /API :

  • PrimeSense Carmine/OpenNI
  • Kinect/Kinect SDK.
  • Creative Labs Senz 3d /Intel Perceptive Computing API
  • Soft Kinetic DS 311
  • LeapMotion
  • OpenCV/Emgu CV – Without Depth Sensing.

PrimeSense Carmine – I’d have ended up using this if Apple had not bought them out, current state is that the support of API’s is still up in the air and even the OpenNI stuff seems to be in a state of DisArray.

Creative Labs Senz 3d – Bought one of these cameras off of Amazon without reading the full specs of the camera. After it came, realized this was a short range precision sensor with about a 3.5 ft range and inadequate for what I had in mind, this is competing technology with Leapmotion and PC Control devices. I must mention that the Intel Perceptual computing APIs and sample code was the fastest I have seen , optimized and engineered well. All of Intels three letter abbreviations are used Intel TBB , Intel IPP , OpenCL, GPU offloading and so on..

SoftKinetic DS311 – SoftKinetic is probably the only company left not owned by some big corp and I would’nt be surprised if they got snapped up pretty soon. They have their own SDK and an IISU Middleware for gesture recognition. The Sensor DS311 matched the specs I was looking for exactly but I dint end up choosing it for no other reason than not wanting to spend the time and money evaluating the sensor  and SDK 🙁 (if anyone else has tried this pl send me a note on your thoughts) .

LeapMotion – Leapmotion is another short range pc control device and not much suited to the range parameters and specs I need. It seems to have an active thriving community and most of the uses I can think of don’t go past the “thats cool” stage. Probably another couple of years till these kind of inputs are seen as a real viable solution for real world problems  and HCI. The Creative Senz 3d is a direct competitor to what this can do.

ps1080
Kinect/Kinect SDK – Kinect is one of the most successful Depth ranging cameras with a range of 4 meters. It is well known the kinect device was a primesense device and it carried the same PS1080 depth ranging chip technology, newer versions may be more accurate evolutions of the technology. This suited the range and accuracy I needed for Project Jabberwocky. Ended up using the MS SDK and should say they have done a great job on the SDK with the examples and great architecture on the Image processing pipeline. The SDK protects the developer to a certain extent from doing something stupid but also gives enough flexibility to hack it when needed.

OpenCV /EmguCV  – This is the granddaddy of all image processing frameworks and has been my all time favorite on visual cog processing on anything from stereo vision to applying shaders on image sets. But as I get older, am finding have less patience on starting off projects from the nuts and bolts. Getting an assembled engine is better (my 2cs at my age – but if you are a grad this may be exactly what you are looking for) Nowadays just don’t have the time to be a grease monkey, to tune it,  find cam positioning etc.

Ok ..  So how is this all Different from what you can do on the XBox with Kinect ?

  • First off Kinect on Xbox does’nt let you go free of the remotes. You still need a remote for your receiver , a remote for your TV or an universal remote and so on.
    the solution? Project Jabberwocky
  • Xbox Gestures are localized to Xbox Console, you can do everything within xbox to control but none of the other equipment
  • In the real world you have a mixture of devices, that are IR controlled, Bluetooth Controlled, API/Network Controlled (Roku), Project Jabberwocky lets you stitch all of this together.

What Can Jabberwocky currently do :

  • Voice Controlled Complex commands – when gestures are not enough.
  • Gestures for basic control of devices.
  • Learn your existing remotes IR codes.
  • Arrange Contexts of Gestures to Actions.
  • Fast threaded non-blocking control of devices.
  • Data metrics gathering.

Current Limitations :

  • Control limited to 12 feet (Kinect Limitation) .
  • You have to gesture to mute to issue voice commands sometimes.

 Where can I try this out ?

  • Pl watch this area for a download link soon.
  • Can I get the source code? Will not be providing the source code at this time but may decide to do it later.
  • What do I need to Run this?
    * Windows Kinect
    * USB UIRT IR Transmitter. (In future will support Iguana IR)
    * Kinect Drivers – Windows SDK 1.8
     http://www.microsoft.com/en-us/kinectforwindowsdev/Downloads.aspx

Contact : sundru@gmail.com