This week I got to talk to one of the best in the business for TouchDesigner work and interactive installations, the talented and humble Matthew Ragan. You’d be hard pressed to find someone as talented as him, with as much experience as him, who is such a nice individual to chat with and be around. It’s rare and he’s a gem to the community, before even starting to think about all the things he’s contributed back to the TouchDesigner community! Matthew works at Obscura Digital in San Francisco, a familiar name if you’re interested in interactive and immersive media, and can be generally found pushin buttons and pushin pixels on his website or working on such iconic things as MGM Cotai, Salesforce, AT&T Stadium, Shantou University, The Mart, his work with transparent displays, and eglass at Obscura Digital.
We decided to talk about some things that everyone imagines but might not get a chance to be involved in: mega-projects. Full disclosure: I just made this term up, because I didn’t know another to describe the type of projects we’re talking about. What I mean when I talk about mega-projects are projects of enormous scale in terms of infrastructure (30+ outputs/screens/projectors or hectic server infrastructure or etc), client spend in the multi-millions on interactive alone (aka crazy demands), long timelines (unlimited revisions, hooray!), and general difficulty/complexity (on the other side of the world, new techniques, language barriers, etc). When you combine a bunch of these into a project, you’d get what I call a mega-project. Matthew, having worked at Obscura Digital for years now, has experienced a lot of these types of projects, and at nVoid we have probably 30% of our projects being mega-projects, while the other 70% are more tame and straight forward blasts. So without further ado, I give you the marathon of wisdom and experience we discussed.
1) What’s it like working on mega-projects? What’s the best part?
Elburz: Your first project of mega proportions is a trip for sure. The stakes immediately feel higher, the scope can feel overwhelming, for some reason the email chains have 30 people on them, the conference calls have a ton of silent members listening, you get emails at really weird hours, your calendar has a needless amount of check-ins schedule, and the list goes on. “Strict timelines” become a thing of the past, and you find it more reasonable to book expensive last minute travel than you do trying to book early only to find out you need to reschedule everything a few times. There’s a ton more co-ordination involved than on smaller projects. Everyone seems to want to be sure about everything before boots hit the ground, unlike smaller projects where it’s often fine to just hit the ground and go with the flow on stuff. Unlike regular projects where you might do 1 site visit then 1 trip to install, mega-projects will have a bunch of site visits, sometimes just to discuss changes or alternatives. The wealthier the clients, the less they’ll want to talk via email and phone. All that aside, the mega-projects feel pretty fulfilling from a “muscle-flexing” perspective. Those are the projects that you brag about or write case studies for, tell everyone you know how you just did XYZ and “omg it was so hard…but we killed it!!” They can be rewarding creatively, but it’s a toss up depending on what you’re working on. Accomplishing the challenge is pretty great though and you’re sure to learn a bunch of new tricks every time that you’ll remember for the next project. It also makes your regular projects feel much more relaxing.
Matthew: Everything you said, 100 times over. Mega projects always end up involving a set of integrations that are where the rubber really meets the road – the system you’re building might be the single turn key tool that has to be the everything machine, or you might be just part of a bigger content process and you need mechanisms for talking to a content management system. At the front end of the project scope is likely to be all over the map – and likely to change a good number of times before things finally settle. You have meetings to vet pie-in-the-sky ideas that have been pitched, build decks, establish the MVP (minimum viable product) for the job, and dream big about building the structure of the project to be scale-able for future revisions. You suddenly worry about logging, remote management, media distribution, and source control more than you ever imagined you might. You pour over signal flow diagrams from the client, server room rack diagrams, and gantt charts to make sure you understand what pieces of the project depend on others – where there are likely to be blocking puzzle pieces, and how to use your time and resources efficiently.
Every detail matters and having good documentation suddenly becomes of paramount importance to you. I also find that more and more on these projects I’ll take the time to build unit tests for ideas – does this rendering technique really scale, and how do we test that idea; does this communication protocol change meet our needs; how do I make sure that I understand the real challenges of a particular problem? My favorite parts of these challenges have certainly come from the project building practices that emerge from having to think big – I’ve spent a lot of time thinking carefully about how machine configuration works, and how to make this part of the puzzle flexible. We’ve mostly landed on a re-usable framework for our projects, and where that becomes exciting is the consistency it brings to the work. It means that for many of our projects even if a developer hasn’t worked on it directly, the structure is familiar enough that someone can jump in and start working and troubleshooting once they’re familiar with our project structure. Knowing that you’ve got a handle on some of the big problems means that the next project (or the v2 of the project you’re working on) only gets better. Really the best part of one of these huge projects is to see it running – somehow the blood, sweat, tears, sleepless nights, and anxiety filled testing phases are worth it when you see the beast of a system up and running. Nothing quite describes that feeling of control when you know exactly how to conjure forth beautiful artwork spread across 60+ outputs. Those moments feel like black magic, like you’ve conjured real magic and suddenly wield a power summons art from ideas.
2) What was the hardest thing to learn when working on these kinds of projects?
Elburz: For me, the two things were learning when I was/wasn’t allowed to speak and how hierarchies work on big projects. The first might sound really aggressive/oppressive but you have to know your role on large projects, and it really doesn’t matter if you have some amazing idea on how to do some other part of the project, it’s not your job to just blurt stuff out. Especially if we’re talking specifically about the TouchDesigner developer role, your role is so low in the food chain you should almost always default to saying nothing in a room unless you’re directly asked about something. If you have seniority or relationships with other parties, that’s different, but I think most people are safer just keeping their mouth shut for the most part, especially in large meetings. If you have something to contribute outside your scope, bring it up to the person directly above you in the hierarchy privately and let them figure out the best way pass that along or introduce you in a larger meeting.
Matthew: Working at Obscura, I’ve been lucky to be a part of an organization that already has some structure, so some of the questions of when to speak or not speak are often filtered by a producer. That being said, navigating the politics and personalities of these jobs can be daunting and hard to understand. You might have a great new realtime scene or new technique that you’re excited about that the client isn’t at all interested in hearing you ramble about – or maybe your Executive Producer (EP) is trying to keep one of those items in their back pocket to surprise the client with some features that weren’t included in the original statement of work, and you just let the cat out of the bag. For these very reasons I always like to do a rehearsal demo or run through with a producer or with the EP before talking to the client – just so you know exactly the boundaries of what you should / shouldn’t talk about.
Elburz: Rehearsals for demo’s are so often skipped, that’s a really good point you bring up. I think most people don’t expect that folks like us, who’ve been on big projects for years and years, like to do a quick run through still and figure out what’s what before the big clients come in the room. It’s oddly reassuring and calming for me. Which also leads into the second thing, which is knowing the hierarchies. Regular projects will have a few parties max: client, agency, creative team (might be agency), integrator (possibly AV consultant), and you’ll be somewhere in there. With these mega projects you’ll have all of the above + representatives from other organizations with a stake in it (possibly even political), international/local variants of each of the above, people flying in and out for the project, people that don’t speak your language, etc etc. Having a really rough high level understanding of who works for whom can be critical to make sure your work goes smoothly and you know who to talk to to get something you need. Ask the wrong person and it doesn’t happen, ask the wronger person and everyone will land in deep shit because they probably weren’t supposed to even know!
Matthew: So far this has all been a dodge from your question though.. what is the hardest thing to learn when working on these mega projects? Everything, lol. In all honesty, I think the hardest piece is really understanding the nature of distributed real-time rendering. It’s a tricky beast and often defies what feels logical. Having a firm handle on understanding if an approach is going to be deterministic is harder than it looks on the surface – and can really get you into a pickle. For the readers who are wondering what we mean by Deterministic – that’s a term that’s often used in graphics programming to describe if the results of a process can be determined ahead of time, or if the results arise from the process itself. Noise functions are often deterministic, whereas something that involves feedback requires access to the data itself (and is mostly a poor candidate for distributed rending – though there are some neat tricks you can get away with).
Perlin noise, even though it feels organic and random, has a consistent result if you plug in all the same values. Because of that, you can use noise functions across machines with consistent results – provided that every machine has all the same inputs. Unlocking this idea makes it possible to do all sorts of fascinating and huge work, BUT you have to have a strong handle on what’s feeding your systems, which in turn means that you need an iron grip on your approach to state management. That seems like an obvious thing to say, but what happens at start-up? what happens at shut-down? What happens if a machine goes off-line then comes back online? What if you drop a packet, what’s the recovery and fail over solution to ensure that machines stay in sync? The questions just keep going here, which is why I think this was such a challenging lesson to learn. Coordinating multiple machines means making no assumptions, and it means you should be ready to be humbled by that concept that you thought you understood – only to find out that you’re missing some of the key ideas that make that idea really work.
3) What was the funniest/most interesting experience you’ve had on these kinds of projects?
Matthew: Luckily this isn’t the case anymore, but my first job in China had beds that were very firm. So firm that they felt more like sleeping on cardboard covered cinder blocks than on anything that might actually be a mattress. I complained about it so much that my partner finally convinced me to take an inflatable camping mattress with me on a subsequent trip – which made a world of difference. It also made me the envy of the group that was sharing an apartment. I couldn’t help but laugh that about the fact that part of my kit for this HUGE job was a camping mattress – 20+ servers, millions of dollars worth of projectors, and me huffing and puffing to fill up a mattress to catch some semi-comfortable sleep.
Elburz: Amazing!!! Mine is also a Chinese story, mainly because I think to your point, working really really far away from home is always interesting. We did a retail experience in Shanghai, and after we left, the staff were experiencing some audio issues. So I kept getting Skype calls at like 3 or 4am my time (which would be store hours in Shanghai) asking me what to do about audio, and in my sleepy daze I’d have no idea what they’re talking about and would just say something like “restart it?” and then I’d fall back asleep. Classic!
4) What’s a mistake you made early on with mega-projects?
Elburz: I didn’t read every document that made it through my email. It can be time consuming, but someone on your team should probably be reading everything that comes down the pipe. Little things can change and be tweaked slowly and in a subtle way through document revisions or email chains, and if you assume that every document that comes down is the same as the last or doesn’t concern you since no one said your name on the thread, prepare for some hilarious surprises come install time!
Matthew: I would definitely second your point here about documentation. It’s hard to do as you go, but it makes a huge difference. Anything you can do to help yourself remember what you were thinking is worth doing. Once a project gets sufficiently large it becomes too hard to keep all the details in your head, this is double true if you’re working on a team – anything you can look over to brief you about how features work, or what a particular call does helps in the long run. I think this goes back to that point earlier about how you maintain projects – sometimes a client has the budget for a project up front, but not for a sustained service contract. There’s no problem with that, and while it’s usually in their best interest to keep some kind of service going, it’s not always something they opt in for. Until it’s time to upgrade, or something goes wrong. Without fail you get that call right about the time the details for a project have become fuzzy, or you’ve forgotten how something works competently. Then you’re cursing your past self – and trying to figure out why on earth you did something in some seemingly backwards way; the only memory you have is one where you recall needing to some awkward work around – but not why you needed it. So you make all the same mistakes again, find a slightly better work around, and burn through all your time solving the problem and don’t write down any solutions – again. Rinse repeat. Any notes or documentation is better than nothing – take it from folks who have learned the hard way.
The other mistake I made early in mega-project building was being too trusting. I trusted that ideas that made their way into decks were already vetted and considered, not just provocations for discussion. Consequently, there were a few occasions where I was looking at some seemingly impossible asks and trusted that they were already set in stone. Decks are wonderful tools for presenting ideas and starting conversations, but developers often need a feature list or a spec sheet. What is this thing supposed to do exactly? What features are in the MVP (minimum viable product), what are the mechanics of that idea, what’s the interface, what’s the user experience, who are the primary stakeholders? It’s easy to trust that there are answers to these questions, or that all of the pieces have been thought through carefully, but that’s not always the case. When in doubt, even just a little, ask – or find out who you need to have that offline conversation with, and dive deep. Python has a programming principle that rings true here, “Explicit is better than implicit.” Make sure you know exactly what’s going on, when in doubt, ask and make sure all of the stakeholders are on the same page. Never make assumptions if you can avoid it – it’s only more headache later on down the line.
Elburz: I think you hit the nail on the head about assumptions. It’s really the underlying truth to my thought on reading emailed decks, because if you make assumptions it could cost you and your team big. On smaller projects, “assumption recovery” is generally something you can cover from your own budgets, but when we’re talking multi-million dollar projects where the computer budget alone might be $100,000 USD, if you got the wrong video cards because of an assumption….you’re a bit boned!
5) Do you prefer the mega-projects or the “regular” ones?
Matthew: The regular ones… oh my god, give me a regular one. I say that after really banging my head against some of the big ones. If I’d never done a mega-project I’d always dream of bigger. Having done several now, the constraints imposed can be really frustrating. The final scale is amazing, but the process requires significantly more discipline, planning, and a very close eye on your work. It has to live beyond the launch, it has to be maintainable, predictable, and stable. On top of that, chances are you’ll have to train someone else to keep it running. You’ll have to train someone that’s hired by the client, or that you sub-contract, or that your company hires as a part of the support team – any one of those is a possibility, but if you (the developer) become the primary person who maintains and services the project then you’ve got a limit to how much work you can make. And I get that someone is shaking their head while reading that, but it’s true – at least in my experience. Big projects don’t need full time attention, but they do need an occasional baby-sitter and troubleshooter, and if you don’t make a plan for that situation, then you can only make so many of these kinds of pieces before the maintenance consumes the majority of your time. There are worse things that can happen, but if you thrive on change and experimentation, that reality can feel artistically confining. That’s not to say that you shouldn’t do one if you have the chance! Just that you should know what you’re getting yourself into before you jump.
Elburz: I agree, if I had never done a mega project, I’d probably never be happy with regular projects, but doing a few a year is more than enough for me. I feel like there’s generally more creative room to play on projects that aren’t as insane. Once a project becomes insanely technical or hectic, the creative part kind of takes a back seat to “what’s still possible on this thing after all the shenanigans we’ve been through?” Also I lose less hair on regular projects, so that’s a plus!
6) How does travel impact you on these mega-projects? Pros? Cons?
Elburz: Pros: I’m a gypsy at heart, I think. I don’t like staying in a place for too long, and constantly moving around for projects and work is refreshing for me. I like the constant change of scenery. Food is a big thing for me, and I love eating whatever I find when I’m traveling, so new cuisines are a blast. So there’s a good set of pros for me. I don’t have a partner or dependents to worry about or miss, so traveling is great!
Matthew: Seeing the world while also getting to practice your craft is a pretty amazing deal – hands down. On top of that getting to work on projects that are close to the edge of possible in these places is pretty stellar. I love the moments where you get to watch the users / spectators for one of these projects. The moments where art transcends language and place remind me about why I do this work, to see that in a place far from home is all the more exciting. I also love being places that are different – it changes the way you see the world, and forces you to question values and perspectives that you’ve previously held. Change is good for one’s creative center, so jumping on a plane is always fun and exciting.
Elburz: Although with that in mind, I will say, there are some real cons to the whole thing. As I get older, there’s certain things that get a little harder/more required. Airplane food I don’t really eat anymore. I used to think “oh sick, free food!” now I think “hmm, I think I’ll just have a clif bar and nap.” I find in hotels I try to find a place with a decent desk, I used to be able to work kind of anywhere in a bed, on the ground, some weird position, etc, but now I find if my hotel sucks, my work suffers a bit and I end up immediately looking up a nearest library or co-working type space that I can hole up in. The physical strains of traveling are also starting to show, I’ve had a few times where I didn’t get enough sleep en-route and ended up basically getting sick while on the plane and having a shitty time on the other side.
Matthew: Boy oh boy do the little things matter as you get older. Work space and time differences are huge. It’s easy to think that you’ll be able to rally for that conference call that starts as soon as you land – chances are you’ll be travel dazed and less-than-fresh, not the best qualities if you need to be on your A-game for an important call. I also think having a clear understanding of your access to the work-space and displays at the job site is important to have clearly outlined. I’ve had a few jobs now where prior to arrival things seemed like they’d be easy access, only to find that that we could be in the basement work space, but couldn’t see anything live except for specific hours. This can be particularly precarious if the client wants to see you working, but your hours for review are overnight. I had one job where I essentially worked 9 am – 4 am most days for a month. The client wanted to “see” us working during the day, but we didn’t have access to displays until overnight. Those circumstances can be grueling and demoralizing – so finding ways to clarify working hours or split shifts can be critical to being productive.
The other con for me with travel is keeping my home in order. I share life with a wonderful partner and two cats – leaving them behind is never fun, and I worry about making sure I’m communicating and taking care of my responsibilities with respect to them. For the cats that means making sure there’s someone to check in on them, clean the liter boxes, and give them treats. I worry that they’ll forget about me, or destroy the couch, or slip into kitty depression. They’re always fine, but I still worry about them. Maintaining a relationship is a little harder, finding time to talk finding something to bring home, and making sure the person on the other end feels like a priority is important. Finding the time for that is hard, but well worth it.
7) Any pro travel tips?
Matthew: Sign up for a frequent travel program – bank those miles, and hotel stays whenever you can. If you can upgrade to premiere economy or even business (which is a game changer) do it – that little bit of extra leg room, or the ability to lie down transforms how you feel after you get off the plane. Travel is a waiting game interrupted by moments of frantic rushing – catch that transfer, or get through customs to meet the client, and and and, or or or. Anything you can do to help yourself feel more comfortable or relaxed is worth it. I’m the kind of person who would rather be 2 hours early for the flight and sit with a glass of wine while I catch up on emails, than to be the person pleading to cut through security so I don’t miss my flight. Pack snacks, and plan your day so you don’t have to stress more than necessary. You’ll be happier for it on the other end.
Elburz: Agreed, I don’t know when this habit started, but I always carry 4x Clif Bars on me at all times, I think as a result of traveling so much and making sure I have some good kind of snack for when times get tough or food is scarce or expensive or in a different currency that you don’t want to deal with for the 1 hour you’re in that layover country. Travel is also very hard on your body, so rest before hand, sleep during, and even sleep as soon as it’s acceptable on the other end! The best way to counteract jet lag is to immediately start eating meals on a regular schedule when you land. No midnight snacks or weirdly timed meals. Hit the ground and if it’s dinner, you eat, even if you’re not hungry. The energy spikes will help set you on the local time. If the timezone change is great, you would ideally not work on the first day you land, can maybe walk around and just regain your bearings.
8) Why do you spend so much time on education (knowing that it’s your own personal time)?
Elburz: I was fortunate enough to learn from very smart people around me who were willing to share their knowledge (shout-outs to the Derivative team), so there’s a natural passing on of knowledge that I feel I should be a part of. If I didn’t, I mean a lot of that accumulated knowledge would probably just be lost eventually. Especially in our field, where it’s hard to find single resources that have accumulated all the disparate elements of our work into a unified thing (inside my brain), so I think there’s value there that would be a shame if it was lost without being shared. That pay-it-forward kind of element works well with lots of recent business trends where educational content and gaining mind-share through education helps boost business overall. And then there’s the context of time for me, when I started working with TouchDesigner there wasn’t nearly as many developers are there are now, so doing educational stuff was also a way of creating more skilled workforce that I could tap into, as well as the saying “the rising tide floats all boats”, where the more people there are using TouchDesigner, the more clients would be interested and would want to use it, meaning there’d be more gigs, etc. It’s definitely a tough thing to maintain for a long period, so I try lately to find the most efficient things I could educate on that maybe other people aren’t talking about (i.e. business, on-site things, soft-skills, etc).
Matthew: I’m right there with you – I was lucky enough to have smart and generous people who helped me at the beginning, and I think it’s important to give back to the community with that same spirit. I also promised my adviser Darrell Hucks that I would write about my grad school experience. Before I started my MFA he told me that the best thing I could do in grad school was to write about what I was doing – even if it seemed obvious, or not particularly useful, still write it down and write often. That discipline forced me to keep working and exploring, and it also meant that I committed to learning in front of people. Some of those earlier tutorials aren’t that great, but I know that folks would be sad to see them go. Besides that they stand as a testament to what a Touch programmers arc might look like.
For me, the other big piece here is that having freely available learning materials helps break the barrier to entry for making digital art. There’s a lot of talk about the democracy of this work, but it’s often hard to find good learning materials that are made for designers or artists – not programmers. There are already enough barriers to entry when it comes to making this kind of art, and I hope that a set of free tutorials and learning materials helps encourage diversity in this community. It was hard for me to get a foot hold on some of these concepts and some of these techniques, and the last thing I want is for other perspectives to be kept out of this field.
In all honesty, the other reason I do it is because I too forget how to do things. So, just like in grad school, I write down what I’m doing. Somewhere between a tutorial and a journal entry, sometimes it’s a means for me to remember and share an approach or practice that is not obvious or was a real head scratcher for me. A good example there would be using inheritance in projects. I’ve thought about it a good deal in the past, even kicked the tires and taking it for a spin – but every time I’ve felt like it was more than I really needed. Until I gave it a closer look as a way to separate out python pieces that are for a job from python bits that need to persist from job to job. Suddenly, ah-ha a use case that made sense, and where this approach had some real legs. Reading through the forum and the slack group I also kept stumbling on challenges with how to approach switch statements in python. Not at all sexy as a topic, but a HUGE time saver once you get a handle on the idea and technique – so much so that it’s been a game changer for some of the otherwise fussy if, elif, else hell that we’ve encountered on past projects. So, I could hoard my gold and keep this to myself… and that would be great for me, but not for the community. Which then touches on your other point – training future collaborators. I also selfishly write some of these materials to help focus new and intermediate Touch developers on solid working approaches and techniques. If someone can step into a job already having a handle on good pythonic practices for TouchDesigner, and a sense of using a version control system it’s much easier to get them up and running fast.
Elburz: It’s definitely hard to keep up though, even writing this blog happens outside of work hours because I have too much to do during work hours. It’s Saturday at 11:20am and I’m travelling for work, and instead of seeing the city or what not, I’m putting this together because it’s something I want to keep doing that has value to me, and I know these kind of things are part of your core values. I can also reveal behind the curtains a bit, we did this exchange over the course of a whole week of chatting via email while you’re on your way from San Francisco to Asia and I was wrapping up an installation and getting ready to do the cross-world flight myself. So even in all that hectic chaos, time change (for you), and canceled flight scrambles (for me), we kept putting in the effort to make this happen. I’m not sure if those kind of things cross most reader’s minds, that we’re basically doing these things instead of having free time or lives, haha!
9) What would you be your one wish on the educational side of things?
Matthew: More time. I love making tutorials and hosting workshops, but it’s not always easy to fit into life. I’d love to have 20 hours a month to dedicate just to creating new materials, it’s just hard to rally at the end of a long week. I’m also past the point of being able to remember what’s really hard for new programmers – it’s funny how time does that to you. These days it’s hard for me to see my own blindspots, so I’d love to have better sense of what concepts need a better metaphor or more detailed explanation.
Elburz: Exactly that last thing you mentioned, I’d wish for more active consumption. It’s so hard to create content over a long period of time when everyone is just a passive consumer. I don’t know which direction to go in, if certain things are working or not, or if anyone is even reading/consuming the stuff! The few times I hear from readers or similar about something they really enjoyed is so valuable in guiding the direction of the output. As much as I might have a plan for certain things, I’m not on an unstoppable war-path here, so hearing from people what kind of things would be more helpful to them would very quickly change my approach to better server readers. It can also help give us fuel to keep going, like you said after a long week of trying to climb interactive Mt. Everest, sometimes I do think to myself “I mean should I even bother? Will anyone notice if I didn’t write anything for the next few weeks? Maybe not!” So in that sense, it does help keep motivations up in tired times.
I can’t thank Matthew enough for this time. I thank him obviously for taking the time to dig into stuff with me, but I also thank him on behalf of the community as a whole (I guess I can do that?) for all the time he’s put into sharing his knowledge, experience, tools, guidance, and good nature on the Slack, Derivative forum, the TouchDesigner Help Group on Facebook, and anywhere else he might have had the chance to cross your path. He’s invaluable and the TouchDesigner community wouldn’t be anywhere near where it is today without him. So thanks, Matthew.
As I mentioned above, we were both pretty busy this week and he still made time for me to chat about our experiences. I think you will rarely see such high-end developers find a way to easily give you these experiences. Maybe over a drink at a conference once a year if you’re lucky enough to pull them away, but if you don’t know them that well, you may be out of luck. So I hope that with this interview we can give you some glimpses into the interesting world of mega-projects, the ups and downs of travel, and a peek into why ourselves (and many other familiar names you know from Slack or the forums) put in time to help others. If you want to learn more about the legendary Matthew Ragan – pushin buttons, pushin pixels, by all means please check out his iconic work that he has done at Obscura Digital in San Francisco for MGM Cotai, Salesforce, AT&T Stadium, Shantou University, The Mart, his work with transparent displays, and eglass.
It seems to be a thing now to send me funny pictures along with headshots, so here’s a pretty amazing gallery of the dreamy Matthew Ragan.