Overview

Algorithmic Muse is an AI-driven drawing system that learns to create abstract representations of the human figure through a sequence of individual brush movements. Instead of producing a complete image in a single computational process, the system constructs each drawing gradually by deciding where the brush should move, how long each movement should be, and whether the brush should touch or leave the canvas.

Unlike conventional generative image systems based on diffusion and the iterative denoising of pixels, Algorithmic Muse treats drawing as a continuous process of embodied decision-making. The artificial agent does not directly generate a finished raster image. Instead, it develops a drawing strategy that can be translated into physical coordinates and executed by a robotic arm.

During a physical drawing session, the system can respond to an existing or live input image, calculate a new sequence of strokes, and translate these virtual movements into the movements of the robotic drawing mechanism. This enables the robotic arm to work autonomously, but also creates the possibility of an alternating drawing process in which a human and the AI respond to one another on the same sheet of paper.

A more detailed account of the system, its training process, and the accompanying evaluation can be found in the published research paper.

Training Process

The drawing agent was trained through Reinforcement Learning using the Proximal Policy Optimization implementation provided by Stable-Baselines3. For this purpose, a custom drawing environment was developed with the Gymnasium framework, allowing the agent to interact with a virtual canvas through the same basic actions that would later be translated into movements of the physical robotic arm.

Within this environment, the agent repeatedly creates drawings, receives feedback about the visual result, and gradually adapts its behaviour. Instead of being shown one correct sequence of strokes, it must discover through experimentation which combinations of movements are more likely to produce recognisable and aesthetically coherent representations of human figures.

Drawing Mechanics

At every step of the drawing process, the agent chooses the direction and distance of its next movement. It also decides whether the virtual brush should remain on the canvas and create a visible line or whether it should be lifted and repositioned without leaving a mark.

These simple actions form the basic vocabulary of the drawing system. Through longer sequences of decisions, the agent can produce continuous contours, disconnected forms, overlapping lines, and more complex compositions. The resulting image is therefore not generated as a fixed arrangement of pixels, but emerges from the temporal movement of a virtual drawing tool.

Because the virtual drawing mechanics correspond to the physical capabilities of the robotic arm, the learned actions can later be converted into coordinates and motor movements. The same underlying system can therefore connect the simulated training environment with the material drawing process.

Reward Function

After each action, the current state of the drawing is evaluated. The agent receives positive or negative feedback depending on whether its latest movement improves the visual qualities that the system has been trained to recognise.

The reward is based on changes in the evaluation of the current image. If a new stroke makes the drawing more similar to a meaningful human-figure representation, the agent receives a positive reward. If the movement reduces this similarity or produces an undesirable result, the reward is smaller or negative.

An additional coverage mechanism encourages the agent to explore different regions of the canvas. Without this component, the agent could learn to repeatedly draw within a very small area. By rewarding spatial exploration, the system is encouraged to develop compositions that make use of a larger part of the available drawing surface.

GAN-Inspired Architecture

The overall training setup follows a principle that is comparable to a Generative Adversarial Network. One component produces images through sequential drawing actions, while another component evaluates the visual results and provides a signal that guides the learning process.

However, the architecture is not a conventional GAN in which a generator and discriminator are trained directly against one another. Instead, a separately trained YOLOv8 image classifier evaluates several categories of images, including completed human-figure drawings, partial drawing fragments, random actions, and unsuccessful outputs produced during earlier training stages.

The classifier's evaluation of the drawing is converted into a reward for the reinforcement learning agent. When the classifier becomes more confident that the evolving image resembles a meaningful drawing, the agent receives positive feedback. In this way, visual classification becomes part of the learning environment and helps the agent discover successful drawing behaviours.

Reinforcement Learning

Proximal Policy Optimization allows the agent to improve its drawing strategy through repeated interaction with the custom environment. During training, the agent produces a sequence of actions, observes the resulting canvas, receives a reward, and updates its internal policy based on the outcome.

Over hundreds of thousands of training steps, the agent gradually learns which movements and stroke sequences are more likely to result in successful drawings. The process does not require a predefined path for every image. Instead, the agent develops a general policy for navigating the canvas and responding to different visual states.

This also means that the resulting movements are not simply copied from a fixed dataset. Each drawing is generated as a new sequence of decisions, allowing variations and unexpected forms to emerge from the interaction between the learned policy, the input image, and the physical constraints of the drawing mechanism.

Paintings

As part of the study, 203 participants compared and evaluated images created by Algorithmic Muse alongside drawings produced by human artists. The survey examined how participants perceived the visual and aesthetic qualities of the generated drawings.

In an optional final section, 169 participants selected the Algorithmic Muse drawing they preferred most. They were also able to choose a color in which the selected image would be physically reproduced by the robotic arm.

The resulting paintings translate the digitally learned movements into physical marks on paper. Although the same underlying drawing was selected by different participants, the physical execution, chosen color, and material behaviour of the robotic arm give each painting an individual character.

Scroll through the resulting paintings in the gallery below:

←

→

1 / 33

Contributions

Special thanks go to Prof. Dr. Daniel D. Hromada and Dipl.-Ing. Christian Schmidts for their guidance, technical expertise, support, and valuable insights throughout the development of the project.