Getting Started with Hugging Face Diffusers

Hugging Face Diffusers is a Python library for running image, video and audio diffusion models on your own hardware. It packages the text encoder, scheduler, denoising network and VAE into a single pipeline object that you call with a prompt.

Hosted image APIs are easy to start with. Then the per-image bills arrive, rate limits slow down your prototype, and you want to know what the model is actually doing. Running the model locally fixes all three, but most tutorials assume you already know PyTorch.

This guide is written for JavaScript developers. It takes you from installation to a first image, explains the settings that matter, shows how to fit the model on smaller GPUs, and finishes with a FastAPI endpoint that your frontend calls with fetch(). The examples target Diffusers v0.42.1.

What Is Hugging Face Diffusers?

Diffusers is Hugging Face’s library for diffusion models. Its central abstraction is DiffusionPipeline, which bundles the components a model needs:

• one or more text encoders that turn the prompt into embeddings
• a scheduler that controls the denoising algorithm
• a UNet or diffusion transformer that predicts the noise at each step
• a VAE that decodes the result into pixels

If you’re new to the Hub, Inference Providers or Spaces, read A Quick Guide to Hugging Face for Developers first. If you want inference with no Python backend at all, Transformers.js runs models directly in the browser, although it covers a different set of models.

What Hardware Does Diffusers Need?

Diffusers runs on three kinds of machine. An NVIDIA GPU is the fastest by a wide margin, an Apple Silicon Mac works through PyTorch’s "mps" backend, and a plain CPU works but is very slow.

The Quickstart lists "cuda", "mps", "xpu" and "cpu" as device targets. Diffusers also has a dedicated Apple Silicon (MPS) guide. This helper picks the right device and dtype for the current machine:

import torch

if torch.cuda.is_available():
    device, dtype = "cuda", torch.float16
elif torch.backends.mps.is_available():
    device, dtype = "mps", torch.float16
else:
    device, dtype = "cpu", torch.float32

Install Diffusers and Generate a First Image

Generating a first image with Hugging Face Diffusers takes three steps: install Diffusers with PyTorch, load SDXL, then call the pipeline with a prompt and save .images[0].

Diffusers requires Python 3.11 or newer since v0.42.0, and is tested with PyTorch 2.6 or newer. Follow the installation guide to set up a virtual environment, then install the packages. SDXL’s text encoders are CLIP models from Transformers, so install that library as well:

uv pip install "diffusers[torch]>=0.42.1" transformers
# or: pip install "diffusers[torch]>=0.42.1" transformers

Using the device and dtype from the helper above, generate an image:

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    dtype=dtype,
).to(device)

prompt = "isometric pixel-art coffee shop at dusk, warm window light, rain on the street"
image = pipeline(prompt).images[0]
image.save("first.png")

image is a PIL image, so .save() writes a PNG to disk. The model id is the checkpoint’s full Hub name, including the stabilityai/ organization prefix.

Licence: SDXL base 1.0 is released under the CreativeML Open RAIL++-M License. Read it before you use generated images in a product, and check the licence of any other model you swap in.

Which SDXL Settings Matter Most?

Four SDXL pipeline arguments control most of what you see in the output: num_inference_steps, guidance_scale, generator, and height / width. Fix the seed first, then change one argument at a time.

generator = torch.Generator().manual_seed(42)

image = pipeline(
    prompt,
    num_inference_steps=30,
    guidance_scale=7.0,
    height=1024,
    width=1024,
    generator=generator,
).images[0]

• num_inference_steps sets how many denoising passes run. More steps take longer and usually add detail, until the gains level off.
• guidance_scale sets how closely the model follows the prompt. Higher values stick closer to the text, and values that are too high can look overcooked.
• generator fixes the starting noise. With the same model, settings and hardware, a seeded generator gives you the same image again.
• height / width set the output size in pixels. Larger images need more memory and more time.

The SDXL pipeline reference lists every argument along with its default value.

Swap the Scheduler

The scheduler decides how noise is removed at each step, and different schedulers trade speed against quality. Replace it on a loaded pipeline with from_config(), passing in the existing config so the settings that match the checkpoint stay aligned:

from diffusers import DPMSolverMultistepScheduler

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)

generator = torch.Generator().manual_seed(42)
image = pipeline(prompt, num_inference_steps=20, generator=generator).images[0]

The Schedulers guide uses this exact pattern with SDXL. The DPMSolverMultistepScheduler reference states that it produces high-quality samples in around 20 steps. Keep the seed fixed when you compare schedulers, so the scheduler is the only thing that changes between the two images.

How Do You Fit SDXL on Smaller GPUs?

If SDXL hits out-of-memory errors, try these four Diffusers memory fixes in order: model CPU offload, VAE tiling, VAE slicing and 4-bit quantization. Each step saves more memory and costs more speed.

1. Model CPU offload. Skip .to(device) and call pipeline.enable_model_cpu_offload(). Each whole model sits on the GPU only while it runs.
2. VAE tiling. pipeline.vae.enable_tiling() decodes large images in tiles. The memory guide notes that this can cause slight tone variation between tiles.
3. VAE slicing. pipeline.vae.enable_slicing() decodes a batch one image at a time. It only helps when you generate several images in a single call.
4. 4-bit quantization. Install bitsandbytes and add a short config:

from diffusers.quantizers import PipelineQuantizationConfig

quant_config = PipelineQuantizationConfig(
    quant_backend="bitsandbytes_4bit",
    quant_kwargs={"load_in_4bit": True, "bnb_4bit_quant_type": "nf4", "bnb_4bit_compute_dtype": torch.float16},
    components_to_quantize=["unet"],
)
pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    dtype=torch.float16,
    quantization_config=quant_config,
)
pipeline.enable_model_cpu_offload()

The quantization overview documents PipelineQuantizationConfig and its backends. Diffusers added bitsandbytes quantization on MPS in v0.39.0, so this approach isn’t limited to CUDA. On Apple Silicon, bitsandbytes needs PyTorch 2.9 or newer, which is higher than the PyTorch 2.6 baseline above.

Avoid enable_sequential_cpu_offload() unless nothing else fits. It saves the most memory, but the memory guide calls it extremely slow. Also, a pipeline loaded with device_map must call reset_device_map() before you use .to() or either offload method.

Calling Hugging Face Diffusers From a Web App

To use Diffusers from a frontend, wrap the pipeline in a small HTTP server. The server should load the model once at startup and return PNG bytes. Install fastapi and uvicorn, then save this as main.py:

import io, threading
from contextlib import asynccontextmanager
import torch
from diffusers import DiffusionPipeline
from fastapi import FastAPI, Response
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

state = {}
lock = threading.Lock()

@asynccontextmanager
async def lifespan(app: FastAPI):
    state["pipe"] = DiffusionPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0", dtype=torch.float16
    ).to("cuda")  # or "mps"
    yield
    state.clear()

app = FastAPI(lifespan=lifespan)
app.add_middleware(CORSMiddleware, allow_origins=["http://localhost:5173"],
                   allow_methods=["POST"], allow_headers=["*"])

class GenerateRequest(BaseModel):
    prompt: str
    seed: int | None = None

@app.post("/generate")
def generate(req: GenerateRequest):
    gen = torch.Generator().manual_seed(req.seed) if req.seed is not None else None
    with lock:
        image = state["pipe"](req.prompt, generator=gen).images[0]
    buf = io.BytesIO()
    image.save(buf, format="PNG")
    return Response(content=buf.getvalue(), media_type="image/png")

Run it with uvicorn main:app. The code is built from three FastAPI features:

• The lifespan handler loads the model once, before the server accepts any requests. from_pretrained() is far too slow to call per request.
• Because generate is a plain def, FastAPI runs it in a threadpool instead of blocking the event loop.
• The lock makes requests use the GPU one at a time.

CORSMiddleware allows your dev frontend’s origin to call the endpoint.

On the frontend, use fetch() to request the image, read the body as a Blob and display it with URL.createObjectURL():

async function generate(prompt, seed, img) {
  const res = await fetch("http://localhost:8000/generate", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt, seed }),
  });
  if (!res.ok) throw new Error(`Generation failed: ${res.status}`);
  if (img.src.startsWith("blob:")) URL.revokeObjectURL(img.src);
  img.src = URL.createObjectURL(await res.blob());
}

Generation takes seconds, not milliseconds. Show a loading state, disable the submit button while a request is in flight, and pass an AbortController signal if users can cancel. Before you serve the generated PNGs to real visitors, shrink them with the OpenReplay image compressor.

Next Steps

Once the pipeline is loaded on the right device, Diffusers needs very little code: a model id, a dtype, a prompt and a seed. Generate a first image with SDXL, fix the seed and adjust one setting at a time. If memory runs short, work through the offload and quantization steps in order. Then put the FastAPI endpoint behind your frontend. From there, the memory guide and the Schedulers guide cover the trade-offs that come up as you tune for production.

FAQs

Where does Diffusers store downloaded model weights?

from_pretrained() saves model files in the Hugging Face Hub cache and reuses them on later runs, so it does not download them again. The cache defaults to ~/.cache/huggingface/hub. Set the HF_HUB_CACHE environment variable to move only the model cache. Set HF_HOME to move all Hugging Face data, which places the cache at HF_HOME/hub. You can also pass a cache_dir argument to choose a location for a single load.

Why does the same seed give a different image on another machine?

A seed only fully reproduces an image on the same hardware with the same PyTorch version. CUDA GPUs use a different random number generator from CPUs, and GPU matrix multiplication is less deterministic. The Diffusers reproducibility guide recommends passing a CPU generator when results must match, which is what torch.Generator() creates by default. Even so, it does not promise identical output across different GPU models or PyTorch versions.

Does 4-bit bitsandbytes quantization work on an Apple Silicon Mac?

Yes, with recent versions. Diffusers added bitsandbytes quantization on MPS in v0.39.0, and bitsandbytes 0.50.0 made all of its 4-bit and LLM.int8() configurations work on MPS. The bitsandbytes MPS path needs PyTorch 2.9 or newer, which is higher than the PyTorch 2.6 minimum for Diffusers. On macOS 26 and later, installing the kernels package enables faster Metal kernels.