Generative art with language+diffusion models

September 16, 2022 — March 5, 2024

computers are awful
generative art
machine learning
making things
neural nets
photon choreography
Figure 1

Generative art using modern diffusion-backed image generators. The name-brand models are DALL-E 2, stable diffusion, Midjourney etc, which are diffusion + transformer models.

CLIP presumably goes somewhere near here.

For audio stuff see music diffusion.

1 DIY models


Civitai is a labor of love from a small team. After being inspired daily by the incredible progress of the Stable Diffusion community and the explosion of custom fine-tuned models, textual inversions, and more, we wanted to see if we could create something that would continue to help the community grow and thrive.

After seeing a gap around sharing the custom models that were being made by the community, we decided to try our hand a putting together a tool that would make it easy for anyone to share, find, and review models. While there were existing services like HuggingFace that allowed users to expose their models as repositories, we felt that it was missing a few key features that would really allow it to serve as a home for the growing community and use case:

  • A way for creators to tag models with things that make sense to the SD community

  • A good way for people interested in the model to review and share their creations

  • A simpler upload and download interface (how many of us are really familiar with code repos)

  • An indexed and visual browsing experience of all the models available

  • An API that can be used by SD tools to tap into the growing library of models, embeds, aesthetic gradients, and hyper networks available

  • About the Project · civitai/civitai Wiki

2 Punditry

3 Mechanics

4 UIs

5 Incoming

6 References

Dhariwal, and Nichol. 2021. Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat].
Dutordoir, Saul, Ghahramani, et al. 2022. Neural Diffusion Processes.”
Han, Zheng, and Zhou. 2022. CARD: Classification and Regression Diffusion Models.”
Ho, Jain, and Abbeel. 2020. Denoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat].
Hoogeboom, Gritsenko, Bastings, et al. 2021. Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat].
Nichol, and Dhariwal. 2021. Improved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat].
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat].
Song, Yang, and Ermon. 2020a. Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems.
———. 2020b. Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems.
Song, Jiaming, Meng, and Ermon. 2021. Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs].
von Platen, Patil, Lozhkov, et al. 2022. Diffusers: State-of-the-Art Diffusion Models.”
Yang, Zhang, Hong, et al. 2022. Diffusion Models: A Comprehensive Survey of Methods and Applications.”