fix(sglang): force reasoning when the template prefills the think tag - #11790
Open
pos-ei-don wants to merge 1 commit into
Open
fix(sglang): force reasoning when the template prefills the think tag#11790pos-ei-don wants to merge 1 commit into
pos-ei-don wants to merge 1 commit into
Conversation
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
localai-org-maint-bot
approved these changes
Aug 31, 2026
localai-org-maint-bot
left a comment
Collaborator
There was a problem hiding this comment.
Good to merge from my review. The prefilled-think-tag detection is scoped correctly, grammar-constrained output is preserved as content, and both parser paths share the same construction logic. Regression coverage, Python syntax checks, and diff checks look clean. @mudler
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Qwen3-style chat templates append the opening
<think>tag to the prompt when thinkingis enabled. The model therefore never generates it and emits only the reasoning text plus the
closing
</think>.sglang's
ReasoningParserkeys off the opening tag:With such a template the entire completion — reasoning, the raw
</think>, and the answer —comes back as
content, andreasoning_contentstays empty no matter howreasoning_parseris configured:
sglang's own OpenAI server solves this with
This backend has no template manager, so the same signal is derived from the rendered prompt:
if it ends with the detector's
think_start_token, the tag was prefilled and the parser isconstructed with
force_reasoning=True.Notes for Reviewers
Why this is conditional rather than always on. Forcing unconditionally breaks the
thinking-off case — a completion with no tags at all is then filed entirely as reasoning and
the answer disappears:
force_reasoningis only passed when it is meant to beTrue, so detector defaults(DeepSeek-R1 already defaults to
True) are untouched.Structured decoding is the exception, and it matters. A grammar applies from the first
token, so the model cannot emit the closing tag even though the template opened the block.
The whole completion is schema output and belongs in
content; forcing there files it asreasoning and returns an empty answer. Measured against a JSON-schema code audit:
10107 characters of "reasoning", zero content. sglang's own server keeps the two apart
for the same reason — its grammar backend owns the reasoning prefix when a reasoning parser
is configured. Hence the
grammar_constrainedguard, with a regression test for it.The construction is factored into
_new_reasoning_parser()so the streaming andnon-streaming paths, which previously built the parser separately, cannot drift apart.
Tested against Qwen3-style models on an sglang backend;
backend/python/sglang/test.pycoversthe prefilled-tag case, the thinking-off case and the grammar case.
Signed commits