feat(vllm): extract tensor_parallel_size and max_model_len from hf_overrides as server flags

[Ravijangid820]

Summary

vLLM ignores server parameters like tensor_parallel_size and max_model_len when passed via --hf-overrides. This PR extracts these known server parameters from hf_overrides and passes them as proper --tensor-parallel-size and --max-model-len flags to the vLLM serve command.

Closes #912

Changes

• pkg/inference/backends/vllm/vllm_config.go: After serializing hf_overrides, extract max_model_len and tensor_parallel_size and append them as proper vLLM serve flags
• pkg/inference/backends/vllm/vllm_config_test.go: Add 4 test cases covering the new extraction behavior

Testing

• All existing tests pass
• New tests for tensor_parallel_size extraction
• New tests for max_model_len extraction
• New tests for both parameters together
• New test verifying backend context size takes precedence over hf_overrides max_model_len
• go test -race ./pkg/inference/... passes

🤖 Generated with Claude Code

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="pkg/inference/backends/vllm/vllm_config.go" line_range="107" />
<code_context>
+			}
+			if tpSize, ok := config.VLLM.HFOverrides["tensor_parallel_size"]; ok {
+				if tp, ok := toInt(tpSize); ok {
+					args = append(args, "--tensor-parallel-size", strconv.Itoa(tp))
+				}
+			}
</code_context>
<issue_to_address>
**Explicit runtime settings are overridden**

When runtimeFlags and HFOverrides specify different values for the same extracted server parameter, `GetArgs` appends the HF-derived flag after `config.RuntimeFlags`, so vLLM receives duplicate options and uses the later HF-derived value. The caller's explicit runtime setting is ignored.

Avoid appending an HF-derived flag when RuntimeFlags already specifies that option, so the explicit runtime setting takes precedence.

Also at `pkg/inference/backends/vllm/vllm_config.go:97-102`.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[Copilot]

🟡 Changes recommended

Numeric conversion and short runtime-flag alias handling can produce incorrect server arguments.

3 open findings

• Recognize -tp alias to avoid conflicting tensor parallel settings · New
• Reject non-integral float tensor parallel values · New
• Extract all validated integer types with overflow checks · New

What changed in this PR

Extracts vLLM server-specific settings from Hugging Face overrides into command-line flags.

Changes:

• Adds extraction and runtime-flag precedence handling.
• Adds coverage for extraction and precedence scenarios.

File	Description
vllm_config.go	Converts selected overrides into vLLM server flags.
vllm_config_test.go	Tests extraction and precedence behavior.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ericcurtin]

Thanks @Ravijangid820! model-runner is being deprecated in favour of https://github.com/llmmanorg/llmman, so please open PRs there instead. If continuing here, please fix the gci lint failure and Copilot's findings (non-integral floats, -tp alias, int overflow). Setting to draft, please reopen when ready for re-review.