What is CV (Computer Vision)?
"Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images and, in general, high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions." (source)
What is OpenCV (Open Source Computer Vision Library)?
"OpenCV is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications...
The library has more than 2500 optimized algorithms... These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million...
It has C++, C, Python, Java and MATLAB interfaces and supports Windows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications and takes advantage of MMX and SSE instructions when available... OpenCV is written natively in C++ and has a templated interface that works seamlessly with STL containers." (source)
"Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images and, in general, high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions." (source)
What is OpenCV (Open Source Computer Vision Library)?
"OpenCV is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications...
The library has more than 2500 optimized algorithms... These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million...
It has C++, C, Python, Java and MATLAB interfaces and supports Windows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications and takes advantage of MMX and SSE instructions when available... OpenCV is written natively in C++ and has a templated interface that works seamlessly with STL containers." (source)
"Making Things See" by Greg Bordenstein (click for source)
'OpenCV for Processing' (Do not mistake for 'OpenCV')
"OpenCV for Processing is based on OpenCV's official Java bindings. It attempts to provide convenient wrappers for common OpenCV functions that are friendly to beginners and feel familiar to the Processing environment." (source)
What can it do:
LINK TO LIBRARY
*IMPORTANT NOTE: OpenCV will not run on 32bit (OS 10.6+) OpenCV will not work with Processing 3
"OpenCV for Processing is based on OpenCV's official Java bindings. It attempts to provide convenient wrappers for common OpenCV functions that are friendly to beginners and feel familiar to the Processing environment." (source)
What can it do:
- Face Detection
- Brightness Contrast
- Filter Images
- Find Contours
- Find Edges
- Find Lines
- Brightest Point
- Region Of Interest
- Image Diff
- Dilation And Erosion Thin
- Working With Color Images
- Background Subtraction
- Color Channels
- Find Histogram
- Hue Range Selection
- Calibration Demo
- Histogram Skin Detection
- Depth From Stereo
- Warp Perspective
- Marker Detection
LINK TO LIBRARY
*IMPORTANT NOTE: OpenCV will not run on 32bit (OS 10.6+) OpenCV will not work with Processing 3
cv.jit Computer Vision for Jitter (Max/ Msp)
Statistics
Motion Analysis
Binary Images
Image Segmentation
Shape Analysis
Miscelaneous
Drawing and display
Obsolete objects
*IMPORTANT NOTE: Patch requires Kinnect 1, Max/ Msp, Jitter, CNMAT Externals, tapTools, freenectlib, oscP5
- cv.jit.mean Calculates the mean value of a matrix over time.
- cv.jit.ravg Calculates the running average of a matrix over time. cv.jit.sum Sums all the pixels in a plane, works for any type/dim.
- cv.jit.variance Estimates the variance of a matrix over time. cv.jit.stddev Estimates the standard deviation of a matrix over time.
Motion Analysis
- cv.jit.opticalflow Estimates the optical flow using various algorithms.
- cv.jit.LKflow Estimates the optical flow using the Lucas-Kanade technique.
- cv.jit.HSflow Estimates the optical flow using the Horn-Schunk technique.
- cv.jit.track Track the position of up to 255 individual pixels.
- cv.jit.features2track Initialize cv.jit.track to easiest pixels to track.
- cv.jit.framesub Difference between consecutive frames.
- cv.jit.shift Region tracking using the MeanShift and CAMShift algorithms.
- cv.jit.touches Track multiple regions at a time. (Optimized for multi-touch interfaces.)
Binary Images
- cv.jit.threshold Adaptive thresholding.
- cv.jit.canny Extract binary edges from a greyscale image.
- cv.jit.binedge Returns only edge pixels.
- cv.jit.dilate Turns a pixel ON if at least one neighbour is ON.
- cv.jit.erode Removes the edge pixels from an image. c
- v.jit.open Erode followed by dilate.
- cv.jit.close Dilate followed by erode.
Image Segmentation
- cv.jit.floodfill Isolates a single connected component.
- cv.jit.label Gives each connected component a unique value.
- cv.jit.blobs.bounds Find bounding boxes for each connected component.
- cv.jit.blobs.centroids Find center of mass for each connected component.
- cv.jit.blobs.direction Find direction each connected component points to.
- cv.jit.blobs.elongation Calculate elongation for each connected component.
- cv.jit.blobs.moments Calculate moments of inertia for each connected component.
- cv.jit.blobs.orientation Measure angle of main axis for each connected component.
- cv.jit.blobs.recon Carry out pattern recognition on each connected components.
- cv.jit.blobs.sort Re-arrange labels so that each connected component keeps the same from frame to frame.
Shape Analysis
- cv.jit.mass Returns the number of non-zero pixels.
- cv.jit.centroids Calculates the center of mass for an image.
- cv.jit.moments Computes various invariant shape descriptors.
- cv.jit.orientation Calculates a shape’s main axis.
- cv.jit.direction Calculates the direction a shape points to.
- cv.jit.perimeter Counts the number of edge pixels.
- cv.jit.elongation Estimates how thin a shape is.
- cv.jit.circularity Estimates how compact a shape is.
- cv.jit.undergrad Performs simple pattern recognition.
- cv.jit.learn Performs pattern analyis and recognition on an incoming list.
- cv.jit.faces Finds human faces in an image.
- cv.jit.features Finds areas of high contrast, pixels that are easy to track.
- cv.jit.lines Finds straight lines.
- cv.jit.hough Compute Hough space. (by Christopher P. Baker and Mateusz Herczka.)
- cv.jit.hough2lines Find straight lines in Hough space.
- cv.jit.snake Fit a point sequence to image edges.
Miscelaneous
- cv.jit.grab Cross-platform wrapper for jit.qt.grab/jit.dx.grab.
- cv.jit.changetype Change the type of a matrix without changing other attributes.
- cv.jit.resize Anti-aliased matrix resize.
- cv.jit.cartopol Treats the data in two matrices as cartesian coordinates and translates to polar data. cv.jit.poltocar …and vice versa.
Drawing and display
- cv.jit.track.draw Visualize output of cv.jit.track.
- cv.jit.lines.draw Visualize output of cv.jit.lines.
- cv.jit.features.draw Visualize output of cv.jit.features.
- cv.jit.faces.draw Visualize output of cv.jit.faces.
- cv.jit.centroids.draw Visualize output of cv.jit.centroids.
- cv.jit.blobs.orient.draw Visualize output of cv.jit.blobs.orientation.
- cv.jit.blobs.elongation.draw Visualize output of cv.jit.blobs.elongation.
- cv.jit.blobs.direction.draw Visualize output of cv.jit.blobs.direction.
- cv.jit.blobs.centroids.draw Visualize output of cv.jit.blobs.centroids.
- cv.jit.blobs.bounds.draw Visualize output of cv.jit.blobs.bounds.
- cv.jit.blobs.color Visualize output of cv.jit.label.
- cv.jit.shift.draw Drawing utility for cv.jit.shift.
- cv.jit.flow.draw Display optical flow using hue and saturation.
- cv.jit.touches.draw Drawing utility for cv.jit.touches.
Obsolete objects
- cv.jit.covariance Computes the covariance matrix of a vector.
- cv.jit.mahalanobis Computes the Mahalanobis metric.
- cv.jit.hmean Calculates the harmonic mean over time.
- cv.jit.gmean Calculates the geometric mean over time.
- cv.jit.trackpoints Display utility for cv.jit.track.
- cv.jit.trackgroup Manager utility for cv.jit.track.
- cv.jit.shapeinfo Wrapper for cv.jit.moments.
*IMPORTANT NOTE: Patch requires Kinnect 1, Max/ Msp, Jitter, CNMAT Externals, tapTools, freenectlib, oscP5
Tangible Physics Engine Prototype Setting & Processing Pipeline
Physical Setup:
Mapping:
Max/ Jitter:
In Processing:
- Kinect hooked up on top the table proving RGB normal video plane matrix, IR matrix and depth map
- Short throw projecting on top of the table
Mapping:
- Projection mapping using syphon client mapped through MadMapper
Max/ Jitter:
- Kinect hooked up using jit.freenect (freenect kinect library for Max)
- Freenect in mode 1 (depth map) set at 5” depth cut off from table surface (like a scan)
- Background subtraction + slide down to only see new objects on the surface
- Using computer vision library’s jit.blobs.centroid, iterate through every blob’s x, y, z position and mass
- For every frame scale, prepend and route tracking info via Open Sound Control protocol (OSC library
- for Max by CNMAT), achieved through identifying a port on local host
In Processing:
- Using OSCp5 library in processing receive OSC message and control the location of attractor, repellor,
- generator and etc. as part of the tangible physics engine/ particle system.