Category: Experimental Animation

I think, in AI generating workflow, The most worthy of study is the promts, so I will mostly build my video around the promts.

Prompt it up! (script)

The overall concept of the script is to follow our workflow and explain how to combine pictures and prompts at each stage to achieve satisfactory results.

I use Ae to add some small animation and sketches to the video essay, to make it easier for people to take a look at it.

Since generating each shot separately often caused inconsistencies in the environment, lighting, character design, and overall style, I decided to generate the entire shot sequence together instead.

I first organised the sequence using the storyboard, then wrote a detailed prompt describing the timing, camera movement, character actions, and transitions between shots. By uploading both the storyboard and the prompt, the AI had a clearer understanding of how the whole sequence should develop.

This method did not remove every problem, but it made the shots feel more connected and visually consistent.

Storyboard

discribe each shot to chatgpt and ask it to give me a story board with style frame and guideline.

Use the storyboard to generate the prompts

Results

This week I try to generating each shot with styleframe, character sheets and environment references.

Style prompts for every shot.

Result

Problems: have some inconsistancy problems. The scene and the bottle don’t match.

The first two shots were generated as one sequence, and they did not have the same consistency problems. The background, lighting, and overall visual style stayed much more stable between the shots.

This suggests that, for shots where the environment is clearly visible, it may be better to group them into one continuous sequence and generate them together. This gives the AI more context about the location and helps it keep the background design more consistent.

However, this method also requires a clearer storyboard and a more detailed prompt. The timing, camera movement, character actions, and transitions between shots all need to be planned in advance.

We try to go for a sketchy style. However during our tests, when the character moves, the brush strokes would gone missing

I try to add some nagetive prompts to help with the consistancy.

Originally, we use Chinese for environment generation. But during experiment, there appear to be some inaccuracy. I wonder if it has somthing to do with language.

After research, it appears language will affact the model to some extent.

Language and Prompting in Image Generation

• Language can influence image results.
  Even if two prompts have the same meaning, different languages may lead to slightly different styles, details, or levels of accuracy.
• English style keywords are often more stable.
  Many image models respond well to common English visual terms such as painterly brushstrokes, hand-painted texture, low saturation, and dusty atmosphere.
• Chinese is still very useful.
  Chinese can express mood, intention, and subtle creative ideas clearly, especially if the user is more comfortable writing in Chinese.
• Direct translation is not always enough.
  A word like “笔触感” can mean brushstroke texture, painterly texture, or rough brushwork, and each may produce a different result.
• A mixed-language workflow works best.
  The user can first describe the idea in Chinese, then refine the prompt with specific English visual keywords.
• Overall, prompting is about clarification.
  The goal is not just to translate, but to turn creative ideas into clear visual instructions that the model can follow.

For image gerneration, the most popular three AI are image-2, nano bananas and midjourney

Midjourney:

The pictures it generates are more aesthetically pleasing, with better color schemes and lighting effects. However, it is very difficult to modify and control the picture. Greater freedom is needed for it to generate better pictures. A good thing is that it can easily

Image2 & nano bananas

Both of these models will be more rigorous and easier to modify. In comparison, the image generated by “image2” will look more attractive.

I try to remodel the character i already have, because the face structure is not very satisfying.

I will try to use these character for animation reference. But it seems like ai have some trouble identify the character’s movements.

At this process I also try some ai generating tools

I generate a high poly model and then make some adjustments ant then do the topology. Then the model would be ready for rigging.

Model made in this process maybe only enough for reference purposes.

In order to cover up the instability of AI generation. We will use a more sketchy style.

Testing

Reference

Outcome

I feel the image style would rely more on the reference picture instead of the prompts. So it will be pretty important to find enough references.