What is CV (Computer Vision)?

"Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images and, in general, high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions." (source)

What is OpenCV (Open Source Computer Vision Library)?

"OpenCV is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications...

The library has more than 2500 optimized algorithms... These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million...

It has C++, C, Python, Java and MATLAB interfaces and supports Windows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications and takes advantage of MMX and SSE instructions when available... OpenCV is written natively in C++ and has a templated interface that works seamlessly with STL containers." (source)

'OpenCV for Processing' (Do not mistake for 'OpenCV')

"OpenCV for Processing is based on OpenCV's official Java bindings. It attempts to provide convenient wrappers for common OpenCV functions that are friendly to beginners and feel familiar to the Processing environment." (source)

What can it do:

1. Face Detection
2. Brightness Contrast
3. Filter Images
4. Find Contours
5. Find Edges
6. Find Lines
7. Brightest Point
8. Region Of Interest
9. Image Diff
10. Dilation And Erosion Thin
11. Working With Color Images
12. Background Subtraction
13. Color Channels
14. Find Histogram
15. Hue Range Selection
16. Calibration Demo
17. Histogram Skin Detection
18. Depth From Stereo
19. Warp Perspective
20. Marker Detection

LINK TO LIBRARY
*IMPORTANT NOTE: OpenCV will not run on 32bit (OS 10.6+) OpenCV will not work with Processing 3

Statistics

• cv.jit.mean Calculates the mean value of a matrix over time.
• cv.jit.ravg Calculates the running average of a matrix over time. cv.jit.sum Sums all the pixels in a plane, works for any type/dim.
• cv.jit.variance Estimates the variance of a matrix over time. cv.jit.stddev Estimates the standard deviation of a matrix over time.

Motion Analysis

• cv.jit.opticalflow Estimates the optical flow using various algorithms.
• cv.jit.LKflow Estimates the optical flow using the Lucas-Kanade technique.
• cv.jit.HSflow Estimates the optical flow using the Horn-Schunk technique.
• cv.jit.track Track the position of up to 255 individual pixels.
• cv.jit.features2track Initialize cv.jit.track to easiest pixels to track.
• cv.jit.framesub Difference between consecutive frames.
• cv.jit.shift Region tracking using the MeanShift and CAMShift algorithms.
• cv.jit.touches Track multiple regions at a time. (Optimized for multi-touch interfaces.)

Binary Images

• cv.jit.threshold Adaptive thresholding.
• cv.jit.canny Extract binary edges from a greyscale image.
• cv.jit.binedge Returns only edge pixels.
• cv.jit.dilate Turns a pixel ON if at least one neighbour is ON.
• cv.jit.erode Removes the edge pixels from an image. c
• v.jit.open Erode followed by dilate.
• cv.jit.close Dilate followed by erode.

Image Segmentation

• cv.jit.floodfill Isolates a single connected component.
• cv.jit.label Gives each connected component a unique value.
• cv.jit.blobs.bounds Find bounding boxes for each connected component.
• cv.jit.blobs.centroids Find center of mass for each connected component.
• cv.jit.blobs.direction Find direction each connected component points to.
• cv.jit.blobs.elongation Calculate elongation for each connected component.
• cv.jit.blobs.moments Calculate moments of inertia for each connected component.
• cv.jit.blobs.orientation Measure angle of main axis for each connected component.
• cv.jit.blobs.recon Carry out pattern recognition on each connected components.
• cv.jit.blobs.sort Re-arrange labels so that each connected component keeps the same from frame to frame.

Shape Analysis

• cv.jit.mass Returns the number of non-zero pixels.
• cv.jit.centroids Calculates the center of mass for an image.
• cv.jit.moments Computes various invariant shape descriptors.
• cv.jit.orientation Calculates a shape’s main axis.
• cv.jit.direction Calculates the direction a shape points to.
• cv.jit.perimeter Counts the number of edge pixels.
• cv.jit.elongation Estimates how thin a shape is.
• cv.jit.circularity Estimates how compact a shape is.
• cv.jit.undergrad Performs simple pattern recognition.
• cv.jit.learn Performs pattern analyis and recognition on an incoming list.
• cv.jit.faces Finds human faces in an image.
• cv.jit.features Finds areas of high contrast, pixels that are easy to track.
• cv.jit.lines Finds straight lines.
• cv.jit.hough Compute Hough space. (by Christopher P. Baker and Mateusz Herczka.)
• cv.jit.hough2lines Find straight lines in Hough space.
• cv.jit.snake Fit a point sequence to image edges.

Miscelaneous

• cv.jit.grab Cross-platform wrapper for jit.qt.grab/jit.dx.grab.
• cv.jit.changetype Change the type of a matrix without changing other attributes.
• cv.jit.resize Anti-aliased matrix resize.
• cv.jit.cartopol Treats the data in two matrices as cartesian coordinates and translates to polar data. cv.jit.poltocar …and vice versa.

Drawing and display

• cv.jit.track.draw Visualize output of cv.jit.track.
• cv.jit.lines.draw Visualize output of cv.jit.lines.
• cv.jit.features.draw Visualize output of cv.jit.features.
• cv.jit.faces.draw Visualize output of cv.jit.faces.
• cv.jit.centroids.draw Visualize output of cv.jit.centroids.
• cv.jit.blobs.orient.draw Visualize output of cv.jit.blobs.orientation.
• cv.jit.blobs.elongation.draw Visualize output of cv.jit.blobs.elongation.
• cv.jit.blobs.direction.draw Visualize output of cv.jit.blobs.direction.
• cv.jit.blobs.centroids.draw Visualize output of cv.jit.blobs.centroids.
• cv.jit.blobs.bounds.draw Visualize output of cv.jit.blobs.bounds.
• cv.jit.blobs.color Visualize output of cv.jit.label.
• cv.jit.shift.draw Drawing utility for cv.jit.shift.
• cv.jit.flow.draw Display optical flow using hue and saturation.
• cv.jit.touches.draw Drawing utility for cv.jit.touches.

Obsolete objects

• cv.jit.covariance Computes the covariance matrix of a vector.
• cv.jit.mahalanobis Computes the Mahalanobis metric.
• cv.jit.hmean Calculates the harmonic mean over time.
• cv.jit.gmean Calculates the geometric mean over time.
• cv.jit.trackpoints Display utility for cv.jit.track.
• cv.jit.trackgroup Manager utility for cv.jit.track.
• cv.jit.shapeinfo Wrapper for cv.jit.moments.
  
  

LINK TO LIBRARY
*IMPORTANT NOTE: Patch requires Kinnect 1, Max/ Msp, Jitter, CNMAT Externals, tapTools, freenectlib, oscP5

Physical Setup:

• Kinect hooked up on top the table proving RGB normal video plane matrix, IR matrix and depth map
  
• Short throw projecting on top of the table

Mapping:

• Projection mapping using syphon client mapped through MadMapper
  

Max/ Jitter:

• Kinect hooked up using jit.freenect (freenect kinect library for Max)
  
• Freenect in mode 1 (depth map) set at 5” depth cut off from table surface (like a scan)
• Background subtraction + slide down to only see new objects on the surface
• Using computer vision library’s jit.blobs.centroid, iterate through every blob’s x, y, z position and mass
• For every frame scale, prepend and route tracking info via Open Sound Control protocol (OSC library
• for Max by CNMAT), achieved through identifying a port on local host

In Processing:

• Using OSCp5 library in processing receive OSC message and control the location of attractor, repellor,
  
• generator and etc. as part of the tangible physics engine/ particle system.

I want to create an application that will enable me to bridge the line between the physical world and the virtual world by tracking real-life objects and assigning them to objects in the application. By doing so, the chosen objects become registered as object-orient objects within the code and are able to function as them. This allows for all sorts of phenomena, events and special effects to occur not on the screen but in your immediate built environment, real-time. Furthermore, if additional real-life objects are placed onto the frame, new interactions/ relations can occur between them, demonstrating the code’s ability to control action flow whilst encouraging participation! To prototype this application I will create a 'tangible physics engine' revolved around physical objects in space using OPENCV and releasing particles onto the space via Leap Motion's gesture mapping.

A Table Where Little People Live
By: Team Lab

Reality Editor:
a new kind of tool for empowering you to connect and manipulate the functionality of physical objects.
By: Fluid Interfaces MIT Media Lab

The Giver of Names:
is quite simply, a computer system that gives objects names.
By: David Rockeby

Form + Code
By: Casey Reas