There are plenty of ways to get interactive graphics on a screen, but as with most things, it’s best to use the right tool for the job. We could try writing our own graphics framework, touch-sensing drivers, and 3D engines, but the best use of our time isn’t solving problems that have already been solved by others. We prefer to stand on the shoulders of those who have worked on those tough, fundamental problems, and focus our time on the unique constraints and opportunities of the project at hand.Here we’ll compare some of the tools we’ve used, outline what they’re best at, and show off the work we’ve produced with each.
Although its hardware products have yet to garner rave consumer reviews, Microsoft does two things amazingly well: development tools and frameworks. Back in 2006, they introduced Windows Presentation Foundation, which paired a markup language called XAML with .NET’s C# programming language. Originally designed for rapid development of business applications on the desktop, it evolved to include fairly advanced graphics features, support for touch sensors, and advanced typography. WPF was reworked for the web as Silverlight, and for mobile devices as Windows 8’s WinRT framework.WPF and WinRT are well-suited for applications with heavy UI and text requirements. Any sort of button, scrolling view, grid, or layout container you might need is in there, and if it’s not, you can probably extend the existing ones to get what you need. The look of each control is completely customizable, so what might start as a generic-looking app can quickly adopt the style of the brand or environment it lives in. Silverlight has many of the same capabilities as the other XAML dialects, but further development on it seems to have stopped, so it can’t be recommended for new projects.While these XAML frameworks provide an efficient environment to a developer that’s experienced with them, there are certain drawbacks. There are many building blocks that can be put together in innumerable ways to get to the solution you’re after, but keeping track of the vast number of controls and their capabilities can be daunting. The cost of this incredible flexibility is significant complexity, which may only become clear over the course of multiple projects. There can also be runtime performance costs associated with such a high-level managed framework, and since the source code isn’t available, these issues can be tough to track down. The XAML frameworks are also lacking in 3D support, and are really only a good option for purely 2D applications.[one_half]https://vimeo.com/34800305[/one_half][one_half_last]https://vimeo.com/59677761[/one_half_last]https://vimeo.com/43428799
Recent advances of web technologies have started to make them a good fit for installation projects. JavaScript engines have gotten faster, programming patterns have matured, and Canvas and WebGL have enabled lower-level 2D and 3D graphics capabilities in the browser. Many of the effects that were once only possible with C++ can now be achieved with JavaScript in the browser. Authoring tools for web technologies tend to be open-source as well, so it’s easier to understand and tweak them.Deciding to build using web technologies is only the first step of the architecture process for a project. After that, there are a hundred choices for application frameworks and graphics utilities. jQuery remains an essential tool, but then there’s Angular or Backbone for application logic, Two or Three or Easel or Voodoo for graphics, and styling tools like SASS and Bourbon. Then you can play on the server as well -- are you going to go old-school with Apache, or mess with Node and Socket for real-time interaction with clients on other platforms?Not everything is easy on the web though, yet. Cross-browser issues can still be a big pain, but hopefully for an installation you have control over the browser and display hardware. JavaScript is still a weird language that takes some getting used to, and while it’s getting faster, it still won’t run as fast as native code. Browsers also aren’t really designed to have very large canvases, so they’re probably not the right choice for rendering multi-monitor video wall projects. They’re also heavily sandboxed for security reasons, so you won’t have much luck accessing files on the local machine or integrating with external sensors.[one_half]https://vimeo.com/77623506[/one_half][one_half_last]https://vimeo.com/14982601[/one_half_last]
Among the tools available to the creative coding community, vvvv is something of an odd bird. Its active user community has shaped it into a hydra that can talk to almost any sensor and render almost any kind of visual with high performance and without writing any code, if you manage to bend your mind around the way it works. Its node-based paradigm is a good fit for visual thinkers, but things that are relatively simple to do in code can get a bit messy very quickly.[caption id="attachment_1725" align="alignnone" width="770"]
A small portion of the vvvv program that made up the Intel CES booth.[/caption]vvvv is built around the notion of patches which can be put together in any number of ways, with the end result being passed to a DirectX surface. Patches are created and modified in a purely visual editor, which can be less intimidating to those who are not programmers. It does offer a relatively simple way to write plugins in C#, which we’ve found to be a good way to implement application logic which can be difficult or messy to describe with boxes and lines.One very compelling feature of vvvv is boygrouping, which is a cheekily-named way to run the same application across multiple machines and keep the state of each machine 'N Sync with the others. This kind of thing is handy when you’ve got 12 machines outputting one 3D scene to 24 projectors.Aside from the programming model, the framework is closed source and support can be kind of iffy. It is also not free, or cheap, for commercial use, so be sure to include that in your project budget.https://vimeo.com/35707460
Our most recent projects are large scale and have intense runtime performance requirements. In order to get that sort of performance, we need to communicate more directly with input devices and output hardware than most frameworks allow. We still don’t want to work completely from scratch, however, so we’ve chosen to work with Cinder. Cinder is mature, battle-tested, open-source, and is supported by an active community that adds new functionality all the time.Cinder provides a higher-level wrapper around OpenGL for graphics and Boost for networking and other features. Cinder apps are written in C++, and much of the application code is reusable across Windows, OS X, and iOS. There is a fairly high learning curve (and a deep lack of documentation), but our team has gotten up to speed and become productive fairly quickly. It doesn’t have the boygrouping feature of vvvv, but that and much more can be implemented with a node.js server to synchronize clients.The high-performance nature of C++ has its own drawbacks, however. Because it’s not a managed language like C# or JavaScript, you have to do your own memory bookkeeping. This can be a hit to productivity, but it actually gives a developer more control over how and when memory is used. Since there isn’t a native editor for Cinder apps (like vvvv), or a human-readable markup language to define interface elements (like XAML), it can be difficult to involve the creative team directly in the development process.[one_half]https://vimeo.com/76738105[/one_half][one_half_last]https://vimeo.com/86557707[/one_half_last]
Above are the frameworks that we have used successfully on projects in the past, but there are other options that haven’t yet found a solid match with our project mix. openFrameworks is a C++ wrapper around OpenGL much like Cinder, but with a more open-source, community-focused structure. Unity is a professional development tool focused mainly on 3D gaming, but has a large community and excellent documentation. Touch Designer creates high-quality video effects with a focus on live performance, but can be adopted to traditional software development as well. Cornerstone is designed specifically for Multitaction display hardware. Greenhouse takes a unique approach to the creation of “spatial interfaces.” New tools are popping up all the time.
The answer to “What framework should I use?” tends to be “Well, that depends.” There are tradeoffs to be made when comparing frameworks such as performance vs ease-of-use. Or perhaps your particular hardware is only supported in a particular framework. If you’re able to prioritize the attributes of your project that are most critical, the right tool should bubble to the top of the list of options.WPF/WinRTHTML/JS/CSSvvvvCinderAbstraction levelhighhighmediumlowPopularitymediumhighlowmediumMaturityhighhighmediumhighPerformancelowmediumhighhighEase-of-usehighhighmediumlowOpen SourcenoyesnoyesSensor supportWinRT: noWPF: yesnoyesyes
