The Development Process Behind Painting VR

Introduction

80.lv: Please introduce yourselves. Where did you study? What companies have you worked for? What projects have you contributed to?

Xander Clerckx, Head Developer: I finished “Digital Arts and Entertainment” in 2011. Then I started working in the field of 3d animation, mainly as a rigger and 3d generalist. I worked on many short films, feature films, and some commercials. I always kept experimenting in my free time (and on jobs where I could) with making interactive projects with things like OpenFrameworks, processing, and Unity. 

Wim Reygaert, Founder: Hey, I’m Wim Reygaert. I’ve been a film director, writer, and producer for 25 years and shifted lanes in 2018 when I founded Oisoi, one of the first VR production companies in Belgium. The idea was to do for-hire work for about two years to get as good as we can at developing for VR, and then start concentrating on our own IP.

Xander and I met at a poorly attended game jam about 3 months after I fired up the company and we started working together two days later. Since then, we have developed experiences for IMEC, the United Nations, the aviation industry, the National Theatre of Ghent, and several individual artists. Covid-19 added an extra year to those two but also raised the interest for spatial computing in general, so actually, everything is going well and as planned.

I’ve had my own companies or worked as an independent director for production companies in Belgium and abroad since my early twenties, and I’ve directed for a lot of musicians and artists like Soulwax, Michael Borremans, Green Velvet, Chvrches, and many others so creativity and experiment are part of my natural habitat. 

Painting VR

80.lv: How did you start your Painting VR app? What inspired you? What was the goal? 

Wim Reygaert: When we started thinking about coming out with our own app, we were focussing on a different idea. Much bigger and more intricate than Painting VR, but also a creation game in VR. We soon realized that it would cost a ton of money to make it and didn’t fit in our plan to release something we could finance ourselves. There’s no way we could have managed or paid for a production like that with the two of us.

Around that time we were developing a multi-user WebGL environment for the release of an album by Philadelphia hip-hop producer NAH and he asked Xander to find a way to let the users draw on the wall with blood, which he did. Xander had just helped one of his students with figuring out how she could make a zen-garden where you rake the sand in VR, and somehow – don’t ask me how, he’s a magician – brought those two together to make Painting VR.

Well, it wasn’t called that back then, he’d just been prototyping a very impressive and near-realistic painting simulation on the Oculus Quest in his spare time, and when he showed it to me I was blown away. It was clear that we had something very immersive and effective we could work with instead of our first idea, so we sat down and started brainstorming on how we could fit this into a concept. I’d calculated that we could work on this for about 50% of our time, so we were still doing for-hire work when we started working on it at the beginning of this year. 

Oculus

80.lv: Why did you choose Oculus as your main platform? What makes it perfect for you? What are the advantages? 

Wim Reygaert: There are a couple of reasons for that. First, it was the platform we’ve been developing for the last year before we started, so it kind of came naturally. We had just finished a prototype for a multi-user VR training for aviation technicians, and it was an absolute nightmare to get everything synchronized, so Xander had to do a lot of research and managed to come up with a decent version in the end. 

For the record: we want everybody to be able to play Painting VR, so we’re not a ‘Quest only’ game. But when we started looking at the numbers, it was very obvious that the biggest group of people already in virtual reality – that was our target audience description, so to speak – were using a Quest. And probably not because it was the absolute best, but it sure was the cheapest, and it was wireless. 

Being a small team, we had to choose on to start with, and because the Oculus Quest is actually the most limited one when it comes to capacity and computing power, we decided to go for it because once we have everything right on the Quest, it shouldn’t be too hard or frustrating to bring it to PCVR. Which we want to do earlier as planned because a lot of people are asking for it. 

Production Process

80.lv: Let’s discuss the production. What was the first step? How did you create different canvases? How does the whole painting work behind the scene?

Xander Clerckx: The core of the whole thing is of course to get the paint from the brush to the canvas This was literally the first thing I set up. For this, I started from another project for which I made a wall that bleeds where you touch it. So the basics were already there. This is basically done with something Unity calls a RenderTexture which allows us to write colors super fast( with the GPU) to a texture. Then it was just a matter of defining what could and what couldn’t be registered on that texture and voila paint was born. 

From there it was just a matter of adding extra features like the texture and paint buckets and some chill tunes to get inspired by while painting. That's where the playing began. Any object could become paint just by following a few simple steps. Soon we had a huge list of ideas, a lot of which we haven't even tried yet. Like painting with fire or slimy blobs, or with a video 

After that, the motherload of work began. Removing a ton of bugs, making everything look nice and making it run on Quest, adding decent grab mechanics (I started out with an asset called AutoHand), and just a whole list of small things I didn’t anticipate.

Mixing Colors

80.lv: The app allows users to mix colors in an incredibly realistic way. How did you set up this mechanic? What were the challenges?

Xander Clerckx: The challenges were some tricky code (again, done on the GPU side). For this, I really had to do quite some research and I was super excited when I finally got it working. And I also ended up with a nice set of recordings of some funky glitches which made great paintings as a result. 

What we still need to improve is the actual “real” color mixing instead of the digital mixing which we have now. But that shouldn’t be too hard to achieve. 

Brush Strokes

80.lv: How did you deal with brush strokes and make them feel real? The task seems incredibly complex, especially in VR. 

Xander Clerckx: The main trick here was to get the Normal Map on the canvas right. I’m using a black and white map on the brush hairs which gives some variation and after it's applied on the canvas I turn it into a Normal Map in real-time. 

I also mix in the original Normal Map of the canvas relief. Which becomes less visible when thicker paint is applied. On top of that, I define which areas should get more shiny and reflective. If we get it performant enough we can even introduce things like metallic or glowing paints. 

Interface

80.lv: Please discuss your work on the app’s interface. How did you make it comfortable and flexible enough for users? 

Xander Clerckx: The UI as well went through a few stages. But overall it was just a matter of adding to it every time we thought of a new feature, when it became too bulky we redesigned it a bit.

But when we couldn’t avoid it we would use a big enough and readable font. We also tried to give physical mechanics as an alternative to the UI because it can be more intuitive to some, and it’s just more fun. We want it to be possible for Painting VR to be played without opening the UI at all. And we’ll work on that more and more in the future. 

Challenges

80.lv: What were the main challenges during production? What made you try again and again? Is there something you would like to improve?

Xander Clerckx: The Spray Can! It has been through so much already. It kept me awake at night, I took extra showers just to get a shower idea on how to fix some issues. And it's still not the way I want it. Luckily we recently had some progress which will benefit the Spray Can a lot. 

Future Plans

We launched our virtual reality art painting simulation game in early access for $9,99 on May 5, 2021. Now, after about five months, we have over 10000 users from all over the world. Now we want to bring those people together in virtual reality. People from the USA, Japan, Canada, South Korea, Brazil, Mexico, the UK, and several other countries came together on our Discord to show their work. Before we knew it, an international community was born around the art created in Painting VR. It's the logical next step to bring this community together in virtual reality.

A multi-user functionality has always been on our roadmap, as a way of letting the user organize studio visits and show their work to others. But during the weeks after the launch, we realized that a multi-player functionality has a promise of an exciting new world in it, and is much more than just a feature. We're already forging connections between like-minded people from all walks of life, gathered on our Discord server around a shared interest: painting. It makes sense to bring this international scene of artists, gamers, and hobbyists together in VR

Painting VR is running a Kickstarter campaign to bring multi-user to the game and bring Painting VR to PCVR.

Xander Clerckx and Wim Reygaert, Developers of Painting VR

Interview conducted by Arti Sergeev