refactor: centralize model factories and defaults

docs/model_config.md

@@ -1,7 +1,7 @@
 # Model Configuration Conventions
 
-This document describes the conventions for model configuration structs and
-weight-based configuration detection.
+This document describes model configuration structs, weight-based configuration
+detection, and the definitions used to integrate models into the runtime.
 
 ## Config Types
 
@@ -116,3 +116,113 @@ struct.
 If a model has no weight-derived config today, it may still provide
 `detect_from_weights` for API consistency, but it should not print a config
 detection log unless it actually derives values from weights.
+
+## Model Integration Definitions
+
+Every `SDVersion` has an explicit `ModelDefinition` in
+`src/model/integrations/`. Related variants share an integration file and runner
+factory. Network blocks and their weight-derived configuration remain under
+`src/model/diffusion/`; reusable text encoder implementations remain under
+`src/model/te/`. Weight-name conversion remains in `src/name_conversion.cpp`.
+
+To add a model:
+
+1. Add its identity to `SDVersion` without renumbering existing values.
+2. Implement the network and `*Config::detect_from_weights` following the
+   conventions above, or reuse an existing runner.
+3. Add a definition in its integration file, including its component
+   factory, traits, geometry, and sampling policy.
+4. Add the family's explicit registration in `src/model_registry.cpp`. For a new
+   variant of a registered family, add its definition to that family's list.
+5. Add version detection signatures and any required weight-name mappings in
+   `src/model_loader.cpp` and `src/name_conversion.cpp` respectively. Update shared
+   conditioning or runtime code only when existing behavior cannot serve the model.
+
+Use the existing family integration as the starting point for a variant. For a
+new family, `src/model/integrations/moss_image.cpp` is a compact example of a
+conditioner/diffusion factory and its definition. Definitions collect model
+construction and defaults; they do not replace version detection, name conversion,
+or model-specific inference algorithms.
+
+The registry checks for missing, duplicate, and invalid version registrations.
+Explicit references keep integrations reachable in static-library builds. The
+existing CMake glob includes integration sources automatically. No legacy
+builder, display-name table, or prediction-default fallback needs updating.
+
+### Detection And Weight Conversion
+
+Version detection remains in `ModelLoader::get_sd_version()` in
+`src/model_loader.cpp`. Add model signatures there and preserve the order of
+overlapping checks. Detection uses tensor names and metadata, and can run before
+and after name conversion, so signatures must work at both stages. Do not load
+tensor data to identify an architecture. The registry looks up definitions by
+the detected version; it does not perform detection.
+
+Weight-name mappings and their version-based dispatch remain centralized in
+`src/name_conversion.cpp`. Add new mappings there when needed. Inference, GGUF
+export, quantization, and LoRA conversion share the `convert_tensor_name` entry
+point; preserve its prefix normalization and component conversion order.
+
+### Components And Conditioning
+
+The core factory receives `model_builders::Context`, the tokenizer configuration,
+and the output runners. Backend checks, optional IP-Adapter setup, scale
+overrides, and tokenizer usage validation remain in the common builder. A
+factory returns failure if any required additional backend cannot initialize.
+
+LLM architecture selection, tokenizer requirements, prompt preparation, and
+encoding remain centralized in `src/conditioning/conditioner.hpp`. Integration
+factories instantiate the conditioner with the model version; models requiring
+new LLM conditioning behavior still need changes in that shared implementation.
+
+`vae_kind` is the single source of truth for the VAE family; compatibility
+helpers derive their results from it. `KL`, `FLUX`, and `FLUX2` share the
+`AutoEncoderKL` implementation while retaining their distinct latent formats.
+The related traits have separate responsibilities:
+
+- `supports_video_generation` describes the model's video generation capability.
+- `video_tae` selects `TinyVideoAutoEncoder` instead of `TinyImageAutoEncoder`.
+  Image models can use a video TAE, so this is independent of video generation.
+- `fake_vae` selects FakeVAE for pixel-space models. It is separate from
+  `vae_kind`, which also controls latent-format compatibility helpers.
+- `vae_format_override` allows PiD's VAE format override.
+
+`configure_vae` handles model-specific KL adjustments such as
+PixArt's scale and SDXL's convolution scale. VAE compatibility traits also drive
+latent transformations and previews through the existing helper functions.
+
+### Sampling And Geometry
+
+`ModelSamplingPolicy` groups the defaults first: `default_prediction`,
+`default_flow_shift`, `default_sample_method`, and `default_scheduler`. Set only
+the values that differ from the struct defaults. `prediction_from_weights` and
+`flow_shift_from_weights` preserve checkpoint-dependent defaults during automatic
+prediction selection; `probe_v_prediction` enables SD2's runtime v-prediction
+probe. Explicit prediction settings take precedence over automatic selection.
+MOSS's default unconditional skip layer stays
+in the diffusion engine, with active unconditional SLG taking precedence.
+
+The resolved `prediction_t` selects the denoiser directly. Flux3, MiniMax H3 AV,
+and Ming-Image use `FLUX3_FLOW_PRED`, `H3_AV_FLOW_PRED`, and
+`MING_IMAGE_FLOW_PRED`; their CLI overrides are `flux3_flow`, `h3_av_flow`, and
+`ming_image_flow`. Explicit `sd3_flow` always selects `DiscreteFlowDenoiser`,
+and `flux_flow` always selects `FluxFlowDenoiser`, regardless of model version.
+Automatic selection retains each model's default implementation. New prediction
+values are appended before `PREDICTION_COUNT`, preserving existing valid values;
+clients using the automatic-selection sentinel must rebuild with the new header.
+
+`ModelGeometry` declares `diffusion_down_factor`, `latent_channels`, and
+`image_channels`. `ref_image_preset` defaults to `"default"`; set it only for models
+that need another reference-image preset.
+
+The common pipeline executes these policies; specialized video, multimodal,
+cache, and extension algorithms remain in their existing runtime components.
+Model-specific diffusion extra parameters are assembled in the diffusion engine.
+Video frame conversion and alignment also remain in the diffusion engine.
+`get_vae_scale_factor()` retains PiD's pixel-space special case and otherwise
+uses the VAE's scale factor.
+
+Validate migrations against the previous implementation using synthetic metadata,
+name conversions, prompt preparation, default/override selection, and focused
+builds. Real generation checks are still needed to validate numerical behavior
+when changing network or encoding algorithms.

examples/common/common.cpp

@@ -818,7 +818,7 @@ ArgOptions SDContextParams::get_options() {
          on_sampler_rng_arg},
         {"",
          "--prediction",
-         "prediction type override, one of [eps, v, edm_v, sd3_flow, flux_flow, sefi_flow]",
+         "prediction type override, one of [eps, v, edm_v, sd3_flow, flux_flow, sefi_flow, minit2i_flow, sensenova_u1_flow, flux3_flow, h3_av_flow, ming_image_flow]",
          on_prediction_arg},
         {"",
          "--lora-apply-mode",

examples/server/api.md

@@ -447,10 +447,6 @@ image establishes the generation dimensions if `size` is omitted. Input
 geometry follows `image_preprocess`: references preserve their dimensions by
 default, while init and mask use the generation canvas preset.
 
-When the request starts without preconfigured reference images, the first uploaded image becomes the img2img init image if the loaded model does not support reference-image conditioning, the final reference list is empty, a mask is present, or an explicit numeric `strength` override is provided. Otherwise, only the final reference list is used for image conditioning, without initializing the img2img latent. Requests with preconfigured references retain their existing init-image selection.
-
-Routing uses the reference list and mask after applying `sd_cpp_extra_args`. Both `ref_images: []` and `ref_images: null` clear the reference list; `mask_image: null` clears an uploaded mask. An explicit `init_image` in `sd_cpp_extra_args`, including `null`, takes precedence over the implicit init-image selection.
-
 Reference encoding then follows model presets and `ref_image_args`. To skip
 input geometry for references and disable resizing before VAE encoding, include
 this in `prompt`:

examples/server/routes_openai.cpp

@@ -7,7 +7,6 @@
 #include "common/common.h"
 #include "common/media_io.h"
 #include "common/resource_owners.hpp"
-#include "openai_edit_policy.h"
 
 static std::string extract_and_remove_sd_cpp_extra_args(std::string& text) {
     std::regex re("<sd_cpp_extra_args>(.*?)</sd_cpp_extra_args>");
@@ -158,24 +157,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.height      = height;
     request.gen_params.batch_count = n;
 
-    std::string sd_cpp_extra_args_str        = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
-    const json extra_args                    = sd_cpp_extra_args_str.empty()
-                                                   ? json::object()
-                                                   : json::parse(sd_cpp_extra_args_str, nullptr, false);
-    const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image");
-    const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images");
-    const bool has_extra_strength_override   = extra_args.is_object() &&
-                                             extra_args.contains("strength") &&
-                                             extra_args["strength"].is_number();
-    const bool has_extra_mask_image = extra_args.is_object() &&
-                                      extra_args.contains("mask_image") &&
-                                      !extra_args["mask_image"].is_null();
-    const bool supports_ref_image_conditioning = sd_ctx_supports_ref_image_conditioning(runtime.sd_ctx);
-    const bool may_need_uploaded_init_image    = !has_extra_init_image_override &&
-                                              (!supports_ref_image_conditioning || !mask_bytes.empty() ||
-                                               has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override);
-    SDImageOwner uploaded_image_for_init;
-    bool uploaded_image_was_first_reference = false;
+    std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
     for (auto& bytes : images_bytes) {
         int img_w            = 0;
         int img_h            = 0;
@@ -190,17 +172,9 @@ static bool build_openai_edit_request(const httplib::Request& req,
             continue;
         }
 
-        const bool is_first_ref_image = request.gen_params.ref_images.empty();
         SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, (uint32_t)resolved_channel, raw_pixels});
         request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
 
-        if (is_first_ref_image) {
-            uploaded_image_was_first_reference = true;
-            if (may_need_uploaded_init_image) {
-                uploaded_image_for_init = image_owner;
-            }
-        }
-
         request.gen_params.ref_images.push_back(std::move(image_owner));
     }
 
@@ -231,21 +205,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
         return false;
     }
 
-    const auto init_image_action = apply_openai_edit_init_image_policy(
-        request.gen_params.init_image,
-        uploaded_image_for_init,
-        uploaded_image_was_first_reference,
-        supports_ref_image_conditioning,
-        !request.gen_params.ref_images.empty(),
-        request.gen_params.mask_image.get().data != nullptr,
-        has_extra_init_image_override,
-        has_extra_strength_override);
-    if (init_image_action == OpenAIEditInitImageAction::UseUploaded &&
-        request.gen_params.init_image.get().data == nullptr) {
-        error_message = "could not allocate init image";
-        return false;
-    }
-
     // Intentionally disable prompt-embedded LoRA tag parsing for server APIs.
     if (!request.gen_params.resolve_and_validate(IMG_GEN, "", runtime.ctx_params->hires_upscalers_dir, true)) {
         error_message = "invalid params";

include/stable-diffusion.h

@@ -94,6 +94,9 @@ enum prediction_t {
     SEFI_FLOW_PRED,
     MINIT2I_FLOW_PRED,
     SENSENOVA_U1_FLOW_PRED,
+    FLUX3_FLOW_PRED,
+    H3_AV_FLOW_PRED,
+    MING_IMAGE_FLOW_PRED,
     PREDICTION_COUNT
 };
 
@@ -486,8 +489,6 @@ SD_API int32_t sd_get_num_physical_cores();
 SD_API const char* sd_get_system_info();
 SD_API bool sd_ctx_supports_image_generation(const sd_ctx_t* sd_ctx);
 SD_API bool sd_ctx_supports_video_generation(const sd_ctx_t* sd_ctx);
-/** Returns whether the loaded model consumes ref_images as conditioning inputs. */
-SD_API bool sd_ctx_supports_ref_image_conditioning(const sd_ctx_t* sd_ctx);
 
 // ControlNet hot-swap APIs are not safe to call while generation is in flight.
 SD_API bool sd_ctx_load_control_net(sd_ctx_t* sd_ctx, const char* path);