Left: We train a pSp-StyleGAN2 pipeline that learns a 16-layer latent space where appearance factors such as gloss and artistic style emerge in a disentangled and hierarchical manner. Specific layers specialize in different attributes, e.g., Layer 6 captures gloss and Layer 8 captures style. Right: This learned space enables intuitive control of appearance. Given a reference geometry (edges) and optional albedo map, the user can employ our diffusion-based pipeline to transfer the gloss and style of an input drawing (inset) to new objects, or traverse the gloss dimension to obtain predictable variations from matte to glossy while keeping other factors stable.
Abstract
Humans can infer material characteristics of objects from their visual appearance, and this ability extends to artistic depictions, where similar perceptual strategies guide the interpretation of paintings or drawings. Among the factors that define material appearance, gloss, along with color, is widely regarded as one of the most important, and recent studies indicate that humans can perceive gloss independently of the artistic style used to depict an object. To investigate how gloss and artistic style are represented in learned models, we train an unsupervised generative model on a newly curated dataset of painterly objects designed to systematically vary such factors. Our analysis reveals a hierarchical latent space in which gloss is disentangled from other appearance factors, allowing for a detailed study of how gloss is represented and varies across artistic styles. Building on this representation, we introduce a lightweight adapter that connects our style- and gloss-aware latent space to a latent-diffusion model, enabling the synthesis of non-photorealistic images with fine-grained control of these factors. We compare our approach with previous models and observe improved disentanglement and controllability of the learned factors.
Method
Our method is built in two stages. First, we train a StyleGAN2-ADA generator on our stylized gloss dataset and then learn a pSp encoder that projects an input image into the generator's extended W+ latent space in a single forward pass. This produces one style vector per generator layer, creating a hierarchical representation in which global structure is handled by early layers, gloss and artistic style emerge in the intermediate layers, and color is mostly captured in later layers. Because the encoder is trained to reconstruct images through the frozen generator, the resulting latent space remains both interpretable and directly editable.
We then reuse that disentangled representation to guide a diffusion model. The diffusion pipeline conditions Stable Diffusion XL with the intermediate W+ embeddings, which carry the appearance factors of interest, especially gloss and style, while the text prompt specifies complementary attributes such as geometry, illumination, and color. A ControlNet branch further constrains spatial structure from edge maps, and an optional albedo input can refine color control. This combination preserves the flexibility and image quality of diffusion models while inheriting the continuous, predictable appearance control learned in the StyleGAN latent space.
Results
Layer-Wise Information
How different visual factors are distributed across the W+ layers.
Gloss Encoding
Gloss is captured as a smooth, ordered latent dimension, making it easy to control continuously.
Pipeline Inputs
Different inputs can used by the diffusion pipeline to guide appearance, structure, and color.
Comparison with Prior Work
Comparison with various baselines of different modalities, including general-purpose text-to-image models, and specialized stylization methods.
Resources
arXiv:
Paper link
Dataset:
Hugging Face dataset
Code (GitHub):
Repository
User study:
Stimuli and data
BibTeX
todo
Acknowledgements
This work has been supported by grant PID2022-141539NB-I00, funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU. This work has also received funding from the Gobierno de Aragon through the project “HUMAN-VR” (PROY_T25_24), and from the European Union (ERC grant number 101220555, PROXIE). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Additionally, Santiago Jimenez-Navarro was supported by a Gobierno de Aragon predoctoral grant (year 2025-2029). Special thanks to the reviewers and to everyone who participated in the user study.