Spatial Media: draw/erase/space

 

draw/erase/space is a site-specific art piece designed to increase public interaction, while at the same time questioning how art is created and destroyed, and how individuals regard others' creations.  Designed to be installed in a public subway, the installation gives multiple users the option to "draw" or erase" within a given space on the subway platform.  Their creation or destruction of art is then displayed on the opposite platform on a projected screen.

The project draws much of its artistic interpretation from two pieces:  Robert Rauschenberg's "Erased deKooning", and Brian Eno & Robert Fripp's "No Pussyfooting".  Both of these pieces wrestle with the complexities of creation and collaboration, and whether destruction is actually a form of creation as well.  draw/erase/space attempts to embrace this spirit by providing individuals with an environment where they can confront these issues head on, simultaneously other hopeful "artists".

Using a ceiling mounted kinect and projector, the piece projects an guideline for where the users are able to stand in order either "erase" or "draw". The floor projection provides boundaries for where users can stand to interact with the piece itself.

The interaction occurs via the overhead mounted Kinect, in combination with a front facing projector.  The user raises their hand above their head, and the hand is detected via blob tracking.  The front facing projection then takes these blobs as input to either erase or draw with a texture on a permanent background.

 

The interaction succeeds by using the Kinect's depth map and feeding it to OpenCV.  By setting a height threshold, it doesn't detect movement below a certain height.  As the user raises their hand, it detects which side they are on (draw/erase) and draws a cursor (green/red) to indicate where their "brush" is located. These cursors are mapped to a larger space, as they only take up half of the kinect window, but need to be able to traverse the whole of the projected image.

  

As users interact in the space, the image becomes a combination of their drawing and erasing "work", and reflects their behavior over time.  The space supports multiple users in each area, and can also support a variety of backgrounds and brushes.  

code: Drawerasespace.java

Nature Of Code Final: Game Of Lifetris

As I originally outlined in my proposal, for my Nature Of Code final I decided to create an implementation of Conway's game of life, and combine it with the classic computer game Tetris.  The result was one that, while not terribly fun to play, actually does go a long way in demonstrating that the game of life is more complex than it appears, and despite its simple rules, offers of a large degree of randomness.

I started by trying to discern the basics of the game, and came to the conclusion that I would need to start with some pieces/organisms already on the board.  The game of life tends towards decimating populations, so having no starting population lends itself to a quickly emptied (and boring) board.  Moreover, given the game's natural tendencies, I decided that I would make the aim of the game to allow pieces to live, rather than to destroy them (as was true in the original tetris).

I took this implementation and coded a simple version that was in black and white and played out a generation of the game of life after each tetris piece was dropped.  Once this was complete, I noticed a major problem:  Despite my hopes for the contrary, there's little way to discern what will happen in the next round of the game.

To work my way around this, I next tried implementing a "future" system, where the game would calculate the results of your current piece placement ahead of time, and attempt to show them to you in a translucent clear projection on top of the current board.  The problem with this is that, while interesting, it's still extremely difficult to tell whether you're creating or destroying pieces.

Continuing on my quest for at least some level of usability, I reached for one final implementation.  This time I maintained the "future" board concept, but also colored the tiles more strategically.  This time I made tiles that stayed alive black, tiles that died red, and tiles that were born green.  In this way, it's easy to try and maximize green and minimize red as you place your piece.

Once the interface was intuitive, I added a scoring system based on how many organisms were alive after each turn, and played the game a bit.  The unfortunate result is that it's not very fun.  While tetris allows you to create strategies and begin to intuit moves, the game of life is simply too random to be able to do anything but try and pick correct placement on a turn by turn basis.

Despite this fact, I think there could still be room for using the game of life mechanism to make for a fun game implementation.  Some cases that still seem to hold water did come to mind:  a game where there was a desire to finish with a very specific number of organisms, a game where all the organisms needed to be eliminated in a certain number of turns, or basically any game where the strategy was less about time and movement, and more about specific placement.

In short, the game of life is too random of an algorithm to function desirably in a time and movement based game.  I believe it would hold up far better under slower, ore strategic circumstances, and that tetris is certainly not.

Sound And The City: Grid Music And Notation

Over the past few weeks, I've taken the idea put forth with orchestraSeating, and modified it significantly.  The modifications have been an effort to reduce the deployment overhead, increase the quality of musical delivery, and make for a musical system that was less spatially specific.  The result is a new version of the installation that I call, for reasons that will become obvious, Grid Music.

orchestraSeating was built around the premise of physical sensors, in a specific site, playing back multi-tracked versions of classical orchestra music.  While this premise is interesting, it is lacking on a number of fronts.  For one, it begs for a site-specific, resource-heavy installation (for example, the cafe at Alice Tully hall, above).  For another, it constrains the piece to the domain of classical music, and therefore requires that the installation and the piece has some level of synchronicity.

A grid overlaid on the Alice Tully seating plan

By contrast, gridMusic uses a grid overlaid on a public space to fuel a generative music engine.  The engine uses an overhead camera in concert with the grid to monitor activity, and then follow a set of rules related to that activity.  The rules are "activated" by movement within the space, and once activated, they cause playback of recorded clips, which yields the generative composition.

Much like Terry Riley's "In C", there are a set of clips (in this case, 20) that the algorithm has to choose from.  Also similar to Riley, the parts must be played in order.  However, the manner in which they are selected is based not on the personal preferences of the players, but by movement within the grid.  When movement is detected in a square of the grid, that square begins playback of Part 1 will continue looping until movement ceases in that square of the grid, and then begins again.  When movement restarts in a given square of the grid, that square is advanced to the next part.

This set of rules allows for easy visualization and state of the piece.  In other words: notation.  The squares that are active are colored, and indicate which clips are currently playing.  Every time any square on the grid changes, a new file representing the grid is saved, complete with a timestamp.  With the traversal of the various grids and their associated time stamps, one can easily discern the composition that was played back at the site for a given performance.

Because of the visually pleasing nature of the grid, it would not be unheard of to involve the grid in some way at the site.  This could be done as a projection or display on a monitor, and would perhaps help to invite the participation of individuals as they acted in the space.

As activity increased, the grid would become progressively more active, with colors varying and changing as per the algorithm's specification.  This would create a lively, interactive visual that would compliment the audio portion of the installation.  Moreover, if the composition were broadcast live to the web, it could be similarly accompanied by the progression of the grids.

As each square reached it's final "switch" from the black "part", it would return to it's original white, inactive state.  It would remain this way until all the squares came to rest, at which point after a duration of 5 minutes down time, the piece would begin again.

Pixel By Pixel: Multiple Perspectives In 2D

 This week in Pixel By Pixel, we were asked to take various painting styles discussed in class to inspire an interactive piece.  As such, I decided to leverage last week's work in image banding, and attempt to emulate Picasso's attempts at forcing multiple perspectives into a single 2D plane.

Picasso's "Portrait Of Dora Maar"

 

In order to accomplish this, I modified my program to accept two camera inputs, one for each perspective.  I then added a keyboard interaction to allow the user to modify the banding resolution.  The result is a program that can look at two perspectives, and divide them amongst image bands accordingly.

 In order to achieve an effect similar to Picasso's, it's necessary that the user carefully align their camera angles so that the perspective in question is mutually centered in each camera.  This will ensure the unity necessary in the merging of the 2D planes into one.  In the image above, I've centered myself in the frame of both camera perspectives.

As the user interacts with the banding resolution, a variety of effects occur.  Here, we can see that at low banding resolutions, there are very obvious differentiations between the two images, resulting in recognizable pieces of each image, and a feeling of displacement for those pieces.

As the banding resolution is increased, the feeling of displacement is reduced.  Instead the image yields more of a feeling of simultaneous existence, with both images occupying the same space.  This is due to the increased resolution revealing a more evenly distributed rendering of each perspective, despite using exactly the same number of pixels.  At these higher resolutions, there is less necessity for the object or scene to be centered, as the increased clarity allows for both perspectives to be seen regardless of positioning. (This effect can be seen at the top of this entry)

Because the image banding controls are distinct, they can also be used to combine the two effects.  This can result in a striped pattern that allows for some of the best of both worlds.  The increased resolution in one dimension increases clarity, while the lower resolution in the other dimension allows for the feeling of displacement, and the clear existence of two perspectives.

This two camera application of image banding and perspective is clearly in the early stages.  Most notably, a fixed and aligned camera configuration might yield more consistent images and interaction.  Additionally, a much larger number of cameras could be used, resulting in further displacement and perspective collisions.  For example, four cameras aligned on an x-y axis could result in a 2D image that showed perspective on an object or scene from all sides.  Alternatively, motorized camera mounts could allow the user to control perspective, and thus control the displacement in the 2D image they were creating.

Nature Of Code: Final Project Proposal, Tetris and Conway's Game Of Life

Our Nature of Code final project is about as abstract as you can get, with the option to leverage just about any and all sides of the various phenomena.  What's more, the visualization (or lack thereof) is also completely open ended.  In short, we were asked to look at the huge amount of material covered this semester, and get inspired.

For me, that meant childhood video games - Specifically, Tetris.  While reading this article on Conway's Game Of Life, I was struck by how similar the cellular shapes were to Tetris pieces.  This got me thinking as to whether there might be a way to combine the two into a video game that yielded novel game play, driven by cellular automata.

While I'm still fleshing out the idea, you can check out my thoughts so far in the proposal below.

ppt: Nature of Code Final Project Proposal - Tetris and Conway's Game Of Life

Pixel By Pixel: Pixel Transformation

This week in Pixel By Pixel, we were asked to delve into the world of pixel transformation.  Put simply, taking the individual pixels in an image, and processing them to change location in the grid. The goal was a result that was "dynamic and interactive".

Part 1: Reflected Pixels
I began the exercise by subdividing both horizontal and vertical pixels by two, and reflecting the pixels to the remaining three quadrants.  The result was a program that is both entertaining and visually dynamic.  As can be seen from the images below, it allows for (particularly with facial anatomy) imagery that instinctually feels deformed or distorted.  However, it can feel pretty simplistic, along the lines of a house of mirrors.

Part 2: Banding
While experimenting with modifying the reflection algorithm, I modified the factor by which the horizontal reflector was divided.  This resulted in the banding pattern seen above.  Observing the pattern caused me think that intentional banding could create compelling imagery, based around the concepts of repetition, patterns, and blending.

Part 3: Banding Grids
I decided to pursue the banding algorithm, and to do so in both the vertical and horizontal dimensions.  I achieved the desired result by grabbing a section of pixels, and then reapplying it using modulo math.  Since the modulo resets every time its argument is reached, the algorithm would then start anew and redraw the desired band.  The result is a "banding" grid of the desired area, as can be seen above.

Part 4: Abstract Banding Grids And Resolution
While a low resolution banding grid can be more compelling in terms of subject recognition, a higher resolution grid can actually yield more unusual and abstract patterns, particularly when it comes to subject blending, and perceived appearance of the subject.

Part 5: Subject Recognition In High Resolution Banding Grids
Perhaps the most counter intuitive part about the high resolution grids is that despite their complexity and abstraction (see above), they are actually simply a grid of a repeated image.  What this means is that if great care is taken, the subject can actually still be recognized, as can be seen from the recognizable lettering in the image below.

The idea of banding grids manages to take a variable chunk of an existing screen, and repurpose it as a tool to build patterns that can be simultaneously abstracted and recognizable.  What's more, it takes a group of pixels and repurposes them on a macro level such that the group itself becomes the implementation of the pixel.  Further developments in this area might include using the mouse to move the selected band, implementing a constant-movement band that iterated across the image, or intelligently selecting bands of the implemented resolution to recreate the image itself, but in recursive bands that had in fact been sourced by the image.

Sound And The City: Orchestra Seating Mk. 2

This week in Sound And The City finds us re-presenting our final project proposals, this time with the addition of an exterior critique.  As such, I've modified my initial proposal to include a specific site, and a more specific implementation plan.  I've also implemented a rudimentary demo in Logic Pro to simulate the effect of the installation.

ppt: orchestraSeating mk. 2