Generative art with language+diffusion models

3.1 Feature matrix

Speed is relative and hardware-dependent; published benchmarks usually run on a desktop NVIDIA GPU, which is faster than most laptop or Apple-Silicon setups. For live quality rankings — Elo scores from blind human votes, updated as new models ship — check the Artificial Analysis text-to-image leaderboard.

3.2 Structural control

A ControlNet lets us pin down the composition of an image — where a figure stands, the pose it strikes, the outline of a scene — separately from what the prompt asks for. We hand the model a “hint” image that encodes the structure we want, and it renders a finished picture that obeys it. This is how we get a character in a chosen pose, or a building with a given silhouette: spatial details more exact than we would describe with words.

The hint comes in a few flavours. A pose skeleton is a stick figure marking where the head, shoulders, elbows and knees go — read off a reference photo by a pose-detector like OpenPose (Cao et al. 2019), or posed by hand. An edge map traces the outlines of a reference as pale lines on black, either hard-edged (Canny, after the classic edge-detection algorithm) or soft-edge, a blurrier trace that leaves the model more room to interpret. A scribble is just a rough sketch we draw ourselves. Either way the model takes the hint as scaffolding it must follow, and invents everything else from the prompt.

Which base models support this? The technique was invented on Stable Diffusion 1.5, so that older model still has the widest menu of control types — if we want some unusual kind of conditioning, it most likely exists there first. Newer models caught up by bundling the common controls (pose, edges, depth) into a single all-in-one model:

• SD 1.5 — the original, and still the most varied.
• SDXL — community all-in-one models.
• Flux.1 dev — an official set from Black Forest Labs (Flux Tools) and a community all-in-one.
• Qwen-Image — gained its own all-in-one in late 2025.

A ControlNet is tied to a single base model, the way a LoRA is. The clients page walks through the workflow — ComfyUI is the most thorough, Draw Things the easiest on a Mac.

4.1 Starting points by goal

• Photorealistic output for publication → FLUX.1 dev (or dev + LoRA)
• Fast iteration on 16 GB, no licence constraint → Z-Image Turbo, FLUX.1 schnell GGUF, or FLUX.2 [klein] 4B
• Specific aesthetic that probably exists as a fine-tune → check SDXL on CivitAI first
• Anime, creature design, dramatic poses, negative-affect content → Pony V6 XL, hopefully other fine-tunes
• Text legible inside the image → Ideogram (API) or FLUX.1 dev, or the Alibaba models (Z-Image Turbo/Qwen-Image)
• Just want something that looks good without thinking about it → Midjourney

4.2 The rest of the menu

Open Draw Things and the model list is loooong — names like LTX, Wan 2.2, Hunyuan, ERNIE, HiDream-I1, Cosmos. Most of that length is the menu collapsing three different kinds of model into one alphabetical list. Video models (Wan 2.2, LTX-Video, Hunyuan) are a separate modality. Cosmos is NVIDIA’s world foundation model for robotics and autonomous-vehicle simulation — it generates video, but as synthetic training data for embodied AI rather than as art.

Among the image models proper, two more have ecosystems forming around them. HiDream-I1 (HiDream.ai, 17B, MIT) topped the Artificial Analysis board for a stretch of 2025 and has a modest but active LoRA following, with the lineage continuing in the O1 successor. ERNIE-Image (Baidu, 8B, Apache 2.0) is newer still — it occupies the text-rendering niche and picked up ComfyUI workflows and community quantizations within days of release.

One prominent name is absent from the community hype: Stable Diffusion 3 / 3.5. SD3’s June 2024 launch went badly — weak anatomy, and a licence so restrictive that CivitAI temporarily banned all SD3 resources — and although Stability later revised the licence and shipped the improved 3.5, the community had already decamped to Flux. In 2026 its LoRA ecosystem is still thin beside SDXL or Flux.

5.1 Historical engravings and prints

• YFG Albrecht Dürer Engraving Style (Flux dev) — captures some of Dürer’s burin line work (swelling/tapering strokes, cross-hatched form modelling) rather than the muddy “old print” pastiche most engraving LoRAs settle for.
• WOOD ENGRAVING Style for FLUX (Flux dev) — clean large-block woodcut, not mezzotint sludge. More Gustave Doré than Piranesi.
• gokaygokay/Flux-Engrave-LoRA (Flux dev) — gokaygokay is a well-known HuggingFace style-LoRA trainer with consistent technical quality.

5.2 Scientific diagrams

• YFG Patents (Flux dev) — trained on patent figures with numbered callouts, dashed hidden lines, exploded views. Most “blueprint” LoRAs only render whole machines; this one captures the schematic call-out idiom itself.
• Anatomica: Chalk (Flux dev) — anatomical chalk-plate aesthetic, and the creator notes the style transfers to non-anatomical subjects, so we get “Vesalius treats a toaster”.
• Century Botanical Illustration (Flux dev) — 18th-century plate aesthetic (stipple shading, latin labels, ivory ground). Almost every “botanical” LoRA returns modern watercolour; this one does engraved plates.
• Anatomical Surrealism (Flux dev) — Haeckel-meets-Da-Vinci hybrid of plate-style anatomy with mechanical/botanical.

Homework for the reader: scientific-diagram coverage is thinner than one might hope. There is room here for a fine-tune on Ramón y Cajal-style neural plates or period microscopy. And every option above is built on Flux.1 dev — there is still no Qwen-Image patent or diagram LoRA. This is an odd gap given that Qwen’s text rendering is exactly what a callout-laden schematic wants. The raw material is sitting in the open: DeepPatent2 is 2.7M public-domain patent drawings already labelled with object names, so a Qwen patent LoRA is waiting for whoever trains it first — a cloud-GPU job, since Draw Things’ trainer doesn’t reach Qwen.

5.3 Emoji and stickers

• fofr/sdxl-emoji (SDXL) — the Apple-emoji LoRA. Lightweight, well-known, transparent-bg-friendly via post-matte.
• starsfriday/Kontext-Emoji-LoRA (FLUX.1-Kontext-dev) — different value proposition: style transfer to emoji. Feed a portrait, get the emoji version of that person. Kontext-native.

There is, as far as I can tell, no great isolated-icon-with-transparency model right now. Generate on a flat colour and matte out with rembg, BiRefNet, or SAM as a separate step. Surely Qwen-Image-Layered should be perfect for this?

5.4 Other specialist idioms

• Stained Glass Window Style (FLUX) (Flux dev).

7.1 Captioning

A text-to-image model was trained on (image, caption) pairs, so to teach it something new we feed it more of the same. The caption does two jobs.

First, it is the handle: the words we type later to summon what we trained, usually a rare trigger token — ohwx woman, sks dog — that means nothing to the base model yet, so the new concept has no preconceptions attached to it.

Second, it tells the trainer what to ignore. Whatever we name in the caption that the model already understands we can factor out; the residual we leave unnamed is what gets absorbed into the LoRA. This is the lever for controlling what the LoRA actually learns: for a style we caption the content and let the style be the residual; for a character we caption the pose, outfit and background so the character identity attaches to the trigger token. This is some weird multimodal voodoo.

Which dialect of caption to use depends on the base model:

• Natural-language captions — full sentences, “a photograph of a woman standing in a field at dusk” — for Flux, SD3.5, Qwen-Image, anything with a strong text encoder.
• Booru tags — comma-separated danbooru-style tags, 1girl, standing, field, sunset — for the SD 1.5 / SDXL / Pony anime lineage, the score_9, anthro, … convention Pony uses.

We rarely type these from scratch: bootstrap with a VLM (a captioner like BLIP, or a “WD14 tagger” for booru tags) and hand-correct.

7.2 Training software

The mature trainers are CUDA-first command-line tools (most with a GUI wrapper). These can be run locally, but usually we rent a GPU by the hour on a cloud provider.

• ai-toolkit (ostris) — the current default for Flux, FLUX.2 and Qwen-Image; config-file driven with an optional web UI. What most current tutorials assume.
• kohya-ss/sd-scripts — a highly configurable veteran working with SD 1.5, SDXL and Flux; most of the older training GUIs are wrappers around it.
• DiffSynth-Studio (ModelScope) — Qwen-Image-first, with a layer-by-layer offload path that makes it feasible to train large models on consumer devices, somehow.
• OneTrainer — seems very approachable: a proper desktop GUI covering LoRAs, full fine-tunes and embeddings, with dataset and labelling built in, and support for a wide range of models.
• SimpleTuner — seems optimised for large datasets and multiple GPUs, production-scale runs rather than a quick character LoRA.

7.3 Where to run it

Training is hungrier than inference. The VRAM bar is high — Flux dev LoRA training wants roughly 24 GB and is comfortable at 48 GB, while FLUX.2 dev needs an 80 GB-class card — so the options run roughly in order of how much of the stack we manage ourselves:

• Our own GPU (CUDA): any of the trainers above, locally, given a large enough NVIDIA card.
• Rent a GPU by the hour: RunPod or Massed Compute — spin up a CUDA pod and run ai-toolkit or kohya on it, the middle ground between local and fully managed.
• Fully hosted (upload images, get a LoRA back): fal (a serverless-GPU host) via its flux-lora-fast-training, Replicate’s Flux trainers, and CivitAI’s on-site trainer (priced in Buzz credits — around 2000 for a Flux LoRA). For Qwen specifically, fal, WaveSpeed and ModelScope’s AIGC Central all run trainers.

On a Mac the on-device option is Draw Things’ LoRA training, which covers SDXL, Flux, Kwai Kolors and SD3.5 but not Qwen-Image or Z-Image. Even a large-memory Mac usually trains in the cloud anyway, because the mature trainers want CUDA.

8.1 Bypassing the blandness post-training

The blandness in layer 1 comes from post-training. Most production-facing models get an RLHF (reinforcement learning from human feedback) or DPO (direct preference optimization) (Wu et al. 2026) pass that aligns outputs to human preference ratings — Diffusion-DPO (Wallace et al. 2023) is a representative public example of the technique. Those ratings correlate with positive affect and conventional aesthetics (McCormack et al. 2024), and against content that disquieted US-market human reviewers during training, so the models drift towards cute and pretty, emotionally beige. The same base weights with different post-training produce different behaviour — which is part of why the open ecosystem exists: edgier, more expressive options are very much in demand.

The positive-affect bias shows up most on underspecified prompts. This has theoretical backing in the preference alignment literature (Wallace et al. 2023) but AFAICT has not been studied specifically for image emotional valence. Community prompt engineering guides report that explicit physical descriptors work better than emotional labels: “tears streaming, jaw clenched, hollow eyes” rather than “sad character.” Adding positive-affect terms to the negative prompt — happy, smiling, cheerful, resolved, peaceful — is a commonly-reported workaround; the mechanism would be steering the sampler away from the part of latent space reinforced by positive preference ratings.

A fine-tune trained on raw character or creature art, without the RLHF passes of a production model, keeps a wider emotional and compositional range — negative affect, dramatic poses, non-cute creatures. All of this is content a default checkpoint would render as cheerful regardless of the prompt. Pony Diffusion V6 XL is the canonical example of a model optimized for drama (and, tbh, horniness).

8.2 Remove the safety checker

The second layer is a filter in the client pipeline — typically a CLIP-based classifier (CLIP being the image-text matching model that scores how well an image matches a text label) that fires after the image is generated and blacks out or blurs the result if it is too edgy. Unlike the weights, this is client software and comes off easily: in diffusers-based pipelines the safety checker is a separately-loaded model component that can simply not be loaded; it is not part of the generation weights.

The same filter is what fires on politically or historically charged prompts — mushroom clouds, battle scenes, burned buildings. It acts on the output image, not the prompt, and the model weights themselves often carry no corresponding restriction. Framing such prompts in historical or clinical registers (“nuclear test, Bikini Atoll, 1952 archival photograph, monochrome, fallout plume”) is anecdotal community knowledge — I have seen it reported to help bypass output classifiers, but have no systematic evidence for it, and how durable it is against classifier updates is unknown.

8.3 Licensing

Permissive licensing. Latitude in the licence is a different axis from latitude in the weights, and three regimes are in play:

• Apache 2.0 (FLUX.1 schnell, FLUX.2 [klein] 4B, Z-Image Turbo) places no restriction on subject matter at all.
• OpenRAIL — and its SD-family variant CreativeML OpenRAIL-M, which Stable Diffusion 1.5 used and Pony inherits — keeps the weights open but attaches a use-restriction clause prohibiting a listed set of harmful applications (child sexual abuse material, certain disinformation uses). It restricts what the model is used for, not who uses it.
• Non-commercial (FLUX dev, FLUX.2 [klein] 9B and [dev]) restricts commercial use of the weights, and Black Forest Labs ships a separate content-filter module alongside — often conflated with the weights, but distinct from them.

So the most permissively-licensed open weights are the Apache-2.0 ones, while the most permissive in content are the RLHF-skipped fine-tunes — and those need not be the same model.

HuggingFace hosts relatively anodyne fine-tunes, while CivitAI hosts the edgier long tail. Most uploads inherit their backbone’s licence, so an SD-family fine-tune is usually CreativeML OpenRAIL-M whether or not its page says so.

9.1 Hygiene

Several things seem to reduce legal exposure, per practitioners and lawyers in this area.

Documenting purpose before generating — a paragraph in a lab notebook or file header — creates contemporaneous evidence of intent. Fair-use and fair-dealing analysis in all four jurisdictions makes purpose and intent explicit factors.

Keeping prompt logs for published work has become a practical precaution. Courts have begun ordering preservation of AI generation records specifically: the SDNY’s May 2025 order in New York Times v. OpenAI required OpenAI to preserve “all output log data on a going forward basis”; a National Law Review analysis (Feb 2026) notes that GenAI prompts, outputs, and logs are now treated as discoverable ESI (electronically stored information) under standard rules. Retrospective reconstruction is harder to defend than contemporaneous records.

Embedding C2PA provenance metadata at generation time — mandatory in the EU from August 2026 — is useful elsewhere for the same reason: it makes the origin of an image legible without relying on the viewer to trust a caption. Draw Things and some ComfyUI node packs support it; the c2patool CLI works on any file.

Distributing NSFW or dramatically charged research imagery only within a research team or behind access controls is a common precaution. All four jurisdictions hinge liability primarily on distribution, not generation, which means the same image can be lower-risk as an unpublished research artefact than as a blog illustration.

Identifiable real persons in embarrassing or intimate contexts carry civil and — in some jurisdictions — criminal exposure regardless of artistic intent.

If our work may reach multiple jurisdictions, using EU Art. 50 requirements as a floor (watermark + human-readable “AI-generated” disclosure) satisfies all four regimes simultaneously.