A Deep Dive Into Generative Art

Introduction

Hi, I'm Ishan Verma. I’m a Lead Material Artist with four years of AAA experience, as well as a self-taught artist and Substance 3D Designer enthusiast. As of the latest, I have moved on to an indie studio known as Antariksha Sanchar which are recipients of an Epic MegaGrant. I will be leading the materials and textures teams for their upcoming amazing Indian-themed video game.

In this article, I will talk about my experience with creative AI, my approach towards it, how AI helped me to improvise and create art, various workflows for both 2D and 3D art using available AI methods, and discuss some very interesting topics.

What is AI Art?

Deep learning algorithms such as Generative Adversarial Networks (GANs) and transformers are used to generate art by following a set of rules, but instead of using a set of rules, they “learn” a specific aesthetic by analyzing thousands of images and producing results based on that analysis.

Using the aesthetics it has learned, the algorithm tries to create new images that stick to the aesthetic, later computer systems that comprehend human language, learn from experience, make more predictions and mimic the thinking process of humans, and produce rich outputs.

AI for Artists

AI has made it easier for artists to make their artistic expression without having to go through the process of creating their artwork. AI can either be a benefit or a hindrance depending on how it is going to be used. The role of AI in art creation and the relationship between the artist and AI are still evolving.

A lot of critics are of the opinion that AI will disrupt the art industry since it makes it easier for artists to express themselves.

The Current State of Creative AI

In terms of artificial intelligence, the state of the art is Deep Neural Networks and Predictive Analysis, since Deep Learning models deal with items such as voice recognition and image recognition. These features give the machine an intelligent feature that can recognize objects. Computer systems can grasp the meaning of human language, learn from experience, and make predictions. Later those predictions are processed and diverse art is produced.

Text-to-Visual AI Tools

The art community has access to a variety of text-to-image generators, and it is getting crazy day by day how people are using them to produce fantastic results.

Midlourney AI, DreamAI by Wombo, StarryAI, DALL-E Mini, Pixel Ray Replicate, and Disco Diffusion are some of the great AI tools I have tried.

Here are four different examples of neural diffusion art created with Pixel Ray and Disco Diffusion. The diffusion process stems from the text I prompted, and then AI detected it and started predicting it using neural networks and worked step by step to create the results.

Artificial intelligence is becoming more prevalent in everyday life, presenting the art industry with a new opportunity to utilize AI-generated art for the purpose of creation and market growth. It is received with excitement, as well as agitation. 

Some think we are in the embryonic stages of the next major art movement, but others are skeptical that machines can access creativity in the same way humans do.

No matter what, AI-generated art raises a plethora of unanswered questions and unexplored territories with respect to authorship, ownership, and how much importance should be ascribed to the process over the final result.

Enabling Creative Workflows Using AI

Being a Material and Texture Artist myself, the very first experiment I did with AI was to create a practical use of it where I could derive textures and drive it later using available suites to make it more usable. I used very specific, defined text prompts to get a diverse number of textures from multiple AI platforms such as DreamAI and StarryAI.

My process was quite basic, first I triggered a texture based on a custom text prompt using AI and later drove it to Substance 3D Sampler where using Stamp Clone and Content Aware Fill tools I fixed the unnecessary data and seams to make it tileable. After that, I exported Height, Normal, Ambient Occlusion, and Base Color Maps to Substance 3D Designer. Later fixed the Base Color values in accordance with the PBR validation, plus created a custom roughness using available maps. Then I took all the maps to Unreal Engine to showcase and render.

Midjourney

Here I created a couple of Viking wood materials using Midjourney, the process was very simple, first, the image texture was generated using AI, later to Substance 3D Sampler and Designer, and then to Unreal Engine 5 for rendering.

High-quality renders can be seen here and here.

Here’s another example of an AI-based texture I created using Midjourney. I also did a vertex paint blend using Unreal Engine 5:

DreamAI

Here I created a couple of marble finish materials using the same method as above.

HQ renders here.

StarryAI

Here I created a stylized wood planks material using the same method.

Although it works like magic in terms of texture creation, it is still a long way from reaching the goal to adapt human thoughts to achieve more precise results.

AI in 3D Asset Creation

Following the texture authorization, I wanted to be more creative to use AI but to make practical stuff out of it. So I thought why not produce some photographs using a text prompt, but the likely results were not precisely what I wanted, so I just created a few surreal poltergeist-themed images in DreamAI.

Later, I imported these images to Substance 3D Designer and converted these images into actual photographs using the Photograph Generator I created a while back.

Then I exported the textures and created a simple mesh for each picture in 3ds Max, did quick UV mapping, and then made renders using Unreal Engine 5.

I also made a concept of an old rotten chainsaw using AI. I tried to produce both a front and back view for it to get more details out of the concept. Somehow most of the details are adequate and can be used to create a 3D model using it

Micro details such as surface details, text, what kind of machinery or blade it's using, etc are pretty diffused but can be cleaned up by a Concept Artist to get more out of it.

To add more of the same, I tried to dig into creating concept art for more game assets. Below is an example of an ammo crate box I created using a prompt. Likely it's not adequate but still gives a diverse output all the same.

Creative Concepts

It does require another powerful touch to generate tex- prompt-based art which can boost a lot to produce instant artistic results in quick seconds. Below is an image of a monastery I created based on the text prompt “Colorful monastery with cloudy ambient look” and AI produced creative results.

Although not being way precise in terms of following art rules but good enough to give valid visual language to the art.

Later, my curiosity led me to dive in more and to think about why not to write a small narrative and conceptualize its key art using AI. Glad it turned out to be a fun game, and the results were quite mesmerizing. Below I have shared an example I created using a short narrative and the magic of AI:

1. People gathered from their homes when they saw a bright trail of light in the sky.

2. Being packed together all of them started to approach the direction of the trail in the sky.

3. People start to approach the open field nearby to get a clear look to understand more about what those trails are.

4. Suddenly, the whole place gets a full ambient from light emitted by the spaceship hovering up on a wheat field, and people closely noticed the flying object.

5. With a sudden flash and boom from a flying object, people panicked and that led to chaos.

The narrative above gave a powerful sight of the AI, this shows how useful the tool can be for Art Directors and likely roles to initiate their thoughts to carve out quick sketches instantly and boost their production workflow to imitate their thoughts in instant. 

It can also help various artists to create vital key concepts related to their artworks, video games, film production, and even for writing any content based on such art.

Character Concepting

My experiments with AI continued and I went into сreating concepts for characters. I started with creating portraits, which were eccentric but gave a lot to start from as a base or to get inspired from. My prompt consisted of cultural diversities and regional bases, which gave adequate results in terms of appearance, facial adaptation, and even their outfits. I did a few more portraits on a gender basis and the results were enthralling.

For portraits, I used Indian, Russian, Viking, and Japanese as example prompts to get diverse looks.

Later I started digging into character conceptualization. My main motto was to get precise results but it did act up in terms of respect to actual human anatomy and gave passable results for a base which Concept Artists can start from or get inspired to create something out of it.

Below I showcased two different concepts for characters:

Indian Armored Warrior Character Concept

Here, for the visuals, I can mark out that it's not exactly a character concept but a thin line where it's carving the actual colorful silhouette with maximized details to get inspired from. At the current phase, it’s unusable in terms of practical workflow but Concept Artists can use it as a base or as a reference.

Dark Plague Doctor Character Concept

Like the warrior concept, here most of the stuff created by AI is still diffused but likely the shapes and the doctor’s helmet is way detailed and can be used for the creation of actual 3D models.

Different Art Styles

Initially, I was interested in seeing if AI could handle diverse art styles, but using the same art or prompt or not, so I turned to ideas. As a result, I used DreamAI to create over 90 artworks using a diverse number of text prompts, which answered my questions on pushing for different art styles.

Below the collage shows the maximum effort given by AI and the prompts given by artists to machines that can produce insane results. It contains themes like Marine, Steampunk, Fantasy, Horror, Stonework, Architecture, etc. In the picture it is just a small glimpse of Meet Mat meets DreamAI but a wide chart of 90 artworks with a diverse number of themes and specified art can be found here.

For the secondary tests, I used Midjourney for acquiring the same method but the results were way passable. The prompt I used for this one was “Sailing ship was attacked by Sea Monster”, in addition, I highlighted the kind of art I want it to be, for example, WaterColor based.

I produced six different art styles using the same prompt but in the end changed its style method, which shows how AI can be helpful to converse or create diverse levels of painting methods in a stint of time.

The Role of an Artist

Generated images by AI were surprising but at the same time unconventional, as most art created by AI doesn't follow the artistic rules or mimic the exact thoughts of humans, which makes the art kinda serendipitous at first glance but somehow a bit eccentric.

Art generated by AI is not the actual focus of the artist nor will be of the machine, but what has been learned by the neural networks over time and the human input to it, it just reflects the same which can be spat out some surprising deformities that can be either marvelous or totally out of visual language.

Artists are individuals who create works of art from scratch. They may produce paintings, drawings, pottery, photographs, films, video games, animation, etc. They have the potential to express their emotions through their art using colors, gestures, themes, and such. They are the medium between emotions and art, without them it’s hard to contribute to give life to the art.

Technology can be of great value and aesthetic beauty, but paintings and crafts have what I call ‘artistic achievement by humans’. Art is really all about human creativity. Technology is a help to it, not a replacement.

Artists won’t be replaced by AI, but it will redefine our conception of the word artist. AI generators make it possible for artists to discover unforeseen forms of imaginative creation – like giving visual meaning to the most beautiful prompted text.

80.lv: Should Substance 3D and other software developers introduce more AI-driven features in your opinion?

Ishan Verma: At the current phase, AI can help developers to enable a computer to do fast and accurate testing that reduces the failure rate and shortens the development process. It will be a more streamlined process and computers can easily handle the repetitive processes. It can also be efficient for AI that can study the feedback given by users globally and fix or improvise the stuff to enhance the software itself. In other words, AI engineering might focus on models that make computerized systems offer tasks in a predicted and better pattern over time to boost actual production.

Now, If I talk about Substance 3D and other similar software companies, being a material artist and their vital user, I would like to say that AI can initiate and improvise a lot of stuff for sure to boost the production as well as give more to users in terms of AI-based features, that can be nodes or workflow prediction while working on texture creation, in application AI-driven web tools to grab stuff on the go, as well capabilities to read out user pattern while using the tool, later predict and help us to introduce shortcut methods based on our every usage or to adjust the software according to our needs.

80.lv: How will such approaches influence art and creativity? Are there some drawbacks? What should the industry be aware of?

Ishan Verma: AI generators provide a new wave of tools that will help artists uncover unseen forms of imaginative creation – like giving a visual meaning to the most beautiful English words. When it comes to influencing, at the current pace, It’s just a pleasing art creation tool for now, but in terms of the upcoming generation, it can blur the line for art learners to learn the art from the core and pretty much rely on machinery methods. This will affect their creativity and artistic skills both in terms of physical and thinking.

I will say that Industries should take AI as a tool but not as an entity, as it should not replace the actual physical talent out there globally but to aid their artistic capability and to ensure to produce better workflows, techniques, and reliable methods so that artists can use AI in an effective manner without getting actually replaced.

80.lv: Some artists noted they don’t feel AI works are their own pieces. What’s your take? Where is this thin line?

Ishan Verma: Some artists describe creating art as a form of self-expression and as a means to communicate their feelings and thoughts by capturing the atmosphere of a scene, second or emotions, etc. However, many artists believe that machines could hinder the essence of art and could even replace humans as creators if they become reliant on them. 

Here the thin line is that AI artists feel that machine-generated art is a new opportunity to grasp the term called Artist but rather than that it’s just a simple process that it’s becoming an art form in itself. AI won’t be replacing artists, but just going to redefine the understanding of the word artist.

To be honest, I don't know if AI is destroying a lot of the physical creative process, and that most of the work in the future will be cleaned up one by the artists. But yes, It definitely takes away a lot of creative work from other parts of production (if it really happens!).

Conclusion

New algorithms and high-performance tools have been developed as a result of recent scientific advances in artificial intelligence. Artists can now think about creative and reflective processes from a new perspective when humans are interacting with artificial intelligence.

Just like how we train our hands to draw what we think of, AI in art creation is simply an instrument with creative potential and it depends on how humans utilize it as it could hinder the essence of art and could even replace humans as creators if they become reliant on them. 

In the end, I would like to thank 80 Level for giving me such an opportunity for this article. Feel free to reach out to me on my ArtStation, Instagram, and LinkedIn.

Ishan Verma, Lead Material Artist at Antariksha Sanchar LLP