LogInTechnical Artist Nils Bakker has shared the workflow behind his recently-showcased game prototype that lets one explore the entire planet Earth in 3D, explaining how it was made using Google's Photorealistic 3D Tiles, ChatGPT, and Unreal Engine 5.
In case you missed it
You may find these articles interesting
Introduction
Hey there! I'm Nils, and I'm all about diving into new tools and techniques, especially in the world of interactive 3D and motion graphics. The cool tech we have these days makes it feel like we're living in an awesome playground with endless possibilities popping up every minute.
So, how did I get into technical art? Well, it all started with the hype around 360-degree videos. I got curious and jumped right in. Then came the rise of VR and AR, and I couldn't resist exploring that too. That's when I stumbled upon Unity and realized the mind-blowing potential of real-time rendering. Being a motion graphics enthusiast, I was stoked to find myself creating digital experiences.
And then some time later, COVID hit, and everything went digital. Talk about timing! Just before the pandemic, I was working on this prototype for virtual events and conferences. I wanted to push the boundaries and see events happening in both the real world and the virtual space. I found myself captivated by the idea of a remote future, where events unfolded effortlessly across various planets, leaving behind the notion of physically convening in one room.
But here's the twist: the pandemic had different plans for us, and we were all stuck at home. It was during this unexpected turn of events that the concept of a virtual studio suddenly sparked in my mind. Collaborating with some colleagues at FX Agency, this awesome digital agency, I initially started building the virtual studio in Unity. However, as I delved deeper into the project, I couldn't resist the gravitational pull of Unreal Engine and its unrivaled graphical power and fidelity. So, I made the bold decision to switch gears and continue developing the virtual studio using Unreal Engine, taking advantage of its impressive capabilities.
Unreal Engine blew my mind! It allowed me to prototype lightning fast, even with my limited coding skills. As a 3D Artist, I was able to create interactive experiences that not only looked dope but also provided a killer user experience.
I've worked on some pretty cool projects, like setting up an interactive showroom for big-name clients such as Cupra. Lately, as part of the OMM Agency, I've been diving into the sports and fashion world, collaborating with cool brands like Nike and Tommy Hilfiger. Trust me, it's an insanely interesting field with endless possibilities for innovation. I'm just excited to be a part of it all!
Getting Started With Unreal Engine
The primary driving force behind my decision to learn Unreal Engine was its artist-friendly nature. I was absolutely captivated by the ability to create stunning interactive 3D worlds. Unity is undeniably a fantastic piece of technology, particularly in the field of AR. However, the way Unreal Engine empowers artists with limited coding knowledge to develop interactive experiences is simply mind-blowing. Coming from a background in 3D motion graphics, I adore the instantaneous feedback and high-quality visualizations that Unreal Engine offers, eliminating any rendering delays. It genuinely feels like stepping into the future.
I owe a huge debt of gratitude to the invaluable tutorials provided by Winbush, Yu Fujishiro, and William Faucher. Winbush and William Faucher's tutorials proved instrumental in guiding motion graphics enthusiasts like myself into the world of Unreal. On the other hand, Yu Fujishiro's expertise lies in crafting beautifully immersive experiences with unique user interactions, making their tutorials an absolute goldmine of knowledge.
The Prototype
It all began on Monday, the 15th, when the idea sparked in my mind. I was determined to make ChatGPT work in Unreal Engine, simply to understand how it functions and have the ability to incorporate it into my future projects, should the need arise.
After successfully getting it to function as intended, I wanted to take it a step further and create something enjoyable out of it. However, earlier that week, I had watched the Google I/O and caught a glimpse of Google Photorealistic 3D Tiles. Intrigued, I looked it up once again and decided to register for API access, thinking that maybe I could integrate it into my ChatGPT project. And that's when everything began to fall into place.
I added the functionality of Photorealistic 3D Tiles to Unreal Engine and, along the way, came up with a fun concept of sending postcards across the world, but in the form of paper airplanes.
With a holiday planned for the following week, I challenged myself to complete this project within a week. As a result, I finished it on Thursday and recorded the process, which I then shared on Friday. Although I wasn't entirely satisfied with the outcome, I made a promise to myself to post it on LinkedIn, and so I did before embarking on my vacation.
Never did I expect it to resonate with so many people. It's incredible to witness the joy it brings to others and how it inspires them to explore Google 3D tiles. I quickly shared my workflow so that others could replicate it and set it up for themselves, which you can find on my website.
Developing the Prototype
When it comes to side projects, I usually follow a workflow that involves diving right in without spending days on research, planning, and overthinking. Since I had already begun implementing ChatGPT into Unreal Engine, it was a seamless process to connect the Photorealistic 3D Tiles and explore how they worked together. Once I figured out how to incorporate the 3D Tiles data, I started brainstorming cool ideas to combine ChatGPT and Photorealistic 3D tiles into a simple yet captivating concept.
That's when the concept of teleporting around the world emerged. I thought, how amazing would it be to soar through the skies as a paper airplane? Inspired by this notion, I envisioned a concept of sending postcards across the globe. Instead of writing traditional postal addresses, users could enter anything as the address, and ChatGPT would find the "best" address for their postcard.
Combining Photorealistic 3D Tiles and ChatGPT APIs
The gathering of photorealistic 3D tiles data is facilitated by the Cesium Plugin for Unreal, which proves to be an excellent and user-friendly tool. Obtaining the 3D data for a specific location simply requires inputting the latitude and longitude coordinates. To engage ChatGPT, a prompt is sent using a specific format that guides its response to adhere to a predetermined structure.
The prompt used:
Answer the following question in the following format.
<< Latitude, Longitude >> ((City Name)) [Country name] ^^a fun fact about this place^^
nothing else. Use that format include the <, > and {} and ().
[USER INPUT]
Setting Up the Paper Plane
Setting up the primary movement mechanics for the paper airplane was a relatively swift and straightforward process. Leveraging the Unreal Physics system made it incredibly convenient to establish basic airplane controls. Due to time constraints, I couldn't invest in developing an advanced control system for the paper plane, which made maneuvering the aircraft quite challenging. Consequently, I opted to avoid flying too close to buildings, as I frequently encountered crashes with buildings and trees. Given more time, I would undoubtedly enhance this aspect and potentially incorporate additional mouse controls. This would enable users to freely look around and even zoom in for a more detailed view of the surroundings.
The Purpose of the Teleportation Sequence
To conceal the transition and mitigate the loading time of the 3D tiles, I implemented a teleportation sequence with Niagara, drawing inspiration from the iconic warp speed effect seen in Star Wars. Once the user enters the prompt on the postcard, the Niagara particle system is spawned, gradually filling the screen with particles. As the outside world becomes obscured, the player and the Niagara system are teleported to the new destination.
Upon arrival at the new position, the system verifies whether the Photorealistic 3D tiles have finished loading. As I couldn't find an existing function to determine the proper loading of the tiles, I devised a simple blueprint solution. By casting a ray from the player in both upward and downward directions, it checks for collisions with the 3D Tiles. A hit indicates that the tiles have spawned. Initially, the tiles spawn with a very low level of detail (LOD), allowing a brief moment for additional resolution to load before checking for another hit. This precaution ensures that no tall buildings or mountains abruptly appear right after the first hit.
The raycast hit also serves another purpose: determining the ground position. By utilizing this information, the player's position is adjusted above the ground, preventing any intersections with buildings or mountains. Additionally, the player is slightly offset backward to provide a clear view of the landmark initially entered in the prompt when exiting the teleportation tunnel.
Conclusion
Currently, I am actively researching the potential of Photorealistic 3D Tiles. If time permits, my plan is to make this prototype open source and share it on my Gumroad page, so that others can explore and interact with it.
Furthermore, I am in the process of assembling a team comprising other developers and 3D artists. Together, we aim to explore ways to enhance this experience and make it accessible to a wider audience. One avenue we are exploring is developing a web-based experience rather than a standalone Unreal Engine game. We are also investigating various non-gaming sectors where this technology could find valuable applications, such as tourism, urban planning, and more.