Skip to content

fix(sglang): force reasoning when the template prefills the think tag - #11790

Open
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:sglang-force-reasoning-prefilled-think-tag
Open

fix(sglang): force reasoning when the template prefills the think tag#11790
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:sglang-force-reasoning-prefilled-think-tag

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Description

Qwen3-style chat templates append the opening <think> tag to the prompt when thinking
is enabled. The model therefore never generates it and emits only the reasoning text plus the
closing </think>.

sglang's ReasoningParser keys off the opening tag:

in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
    return StreamingParseResult(normal_text=text)

With such a template the entire completion — reasoning, the raw </think>, and the answer —
comes back as content, and reasoning_content stays empty no matter how reasoning_parser
is configured:

ReasoningParser("qwen3", stream_reasoning=False)                        reasoning    0 chars
ReasoningParser("qwen3", stream_reasoning=False, force_reasoning=True)  reasoning  746 chars
                                                                        content     28 chars

sglang's own OpenAI server solves this with

force_reasoning = (self.template_manager.force_reasoning
                   or self._get_reasoning_from_request(request))

This backend has no template manager, so the same signal is derived from the rendered prompt:
if it ends with the detector's think_start_token, the tag was prefilled and the parser is
constructed with force_reasoning=True.

Notes for Reviewers

Why this is conditional rather than always on. Forcing unconditionally breaks the
thinking-off case — a completion with no tags at all is then filed entirely as reasoning and
the answer disappears:

19 chars of output, no </think>
  without force : reasoning   0    content  19   "2,3,5,7,11,13,17,19"
  with    force : reasoning  19    content   0   ""      <- answer gone

force_reasoning is only passed when it is meant to be True, so detector defaults
(DeepSeek-R1 already defaults to True) are untouched.

Structured decoding is the exception, and it matters. A grammar applies from the first
token, so the model cannot emit the closing tag even though the template opened the block.
The whole completion is schema output and belongs in content; forcing there files it as
reasoning and returns an empty answer. Measured against a JSON-schema code audit:
10107 characters of "reasoning", zero content. sglang's own server keeps the two apart
for the same reason — its grammar backend owns the reasoning prefix when a reasoning parser
is configured. Hence the grammar_constrained guard, with a regression test for it.

The construction is factored into _new_reasoning_parser() so the streaming and
non-streaming paths, which previously built the parser separately, cannot drift apart.

Tested against Qwen3-style models on an sglang backend; backend/python/sglang/test.py covers
the prefilled-tag case, the thinking-off case and the grammar case.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.

sglang's ReasoningParser keys off the opening tag:

    in_reasoning = self._in_reasoning or self.think_start_token in text
    if not in_reasoning:
        return StreamingParseResult(normal_text=text)

so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.

sglang's own OpenAI server handles this via

    force_reasoning = (self.template_manager.force_reasoning
                       or self._get_reasoning_from_request(request))

This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.

Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.

force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.

The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to merge from my review. The prefilled-think-tag detection is scoped correctly, grammar-constrained output is preserved as content, and both parser paths share the same construction logic. Regression coverage, Python syntax checks, and diff checks look clean. @mudler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants