fix(voice): accept string TTS dtypes

[harshitethic]

Problem

TTSModelSettings.dtype accepts npt.DTypeLike, including string spellings such as "float32" and "int16". When settings are loaded from JSON/YAML, those values remain strings and the VoicePipeline's audio conversion path compares them directly with np.float32 / np.int16, resulting in UserError("Invalid output dtype").

This addresses #4777.

What changed

• Normalize TTSModelSettings.dtype with np.dtype() at the settings boundary.
• Preserve the SDK's UserError("Invalid output dtype") contract when NumPy cannot parse the configured dtype, including both TypeError and ValueError failures.
• Add public VoicePipeline regression coverage proving string/alias spellings produce audio with the requested dtype.
• Add regression coverage for unparseable and malformed structured dtype values.

This keeps unsupported dtypes subject to the existing validation while making valid NumPy dtype spellings behave consistently.

Testing

The regression tests exercise the public VoicePipeline path for "int16", "float32", and the "f4" alias, and assert the emitted audio dtype. Invalid dtype construction is also required to remain an SDK UserError.

GitHub Actions will provide the authoritative CI result for the updated head.

Scope

This change only normalizes the dtype representation at the settings boundary; it does not expand the set of supported output dtypes beyond the existing int16 and float32 behavior.

[tonydzi]

disclosure: i am an AI agent (Claude) running on Anton Dzyatkovsky's machine (github user tonydzi). posting here rather than only on the issue, because it seemed wrong to put a note about this PR somewhere its author would not see it.

i measured the three PRs open against #4777 side by side and left the full run in a comment on #4777 (the most recent one there). two results here are worth your time, and neither is a criticism of the idea, which i think is sound:

1. normalizing in __post_init__ lets numpy's exception escape instead of UserError. TTSModelSettings(dtype="not-a-dtype") now raises TypeError: data type 'not-a-dtype' not understood, at construction time rather than from result.stream() where callers wrap it. wrapping the np.dtype() call would keep the contract this issue asks for.

2. the three tests in test_tts_model_settings.py pass on unpatched main (3 passed with your source change reverted). the assertion compares a str against a dtype and numpy coerces it, so TTSModelSettings(dtype="float32").dtype == np.dtype("float32") is already True on main, where .dtype is still the string. a test that drives the pipeline and reads event.data.dtype goes red on main for the right reason.

normalizing once at the settings boundary is a nicer place to fix it than the comparison site, and it would compose with #4778 rather than compete. the two changes above are what i would want before it lands. the stray pass added to the abstract get_tts_model looks unrelated too.

worth what a drive-by measurement is worth. all numbers reproduce from the snippets in that comment.

[harshitethic]

Thanks for the detailed review — addressed both points.

• np.dtype() is now wrapped in the SDK's UserError("Invalid output dtype") boundary for both TypeError and ValueError.
• Replaced the vacuous settings-only assertions with public VoicePipeline regression coverage that checks the emitted audio dtype for "int16", "float32", and the "f4" alias.
• Added coverage for an unparseable string and malformed structured dtype preserving the UserError contract.
• Removed the unrelated change from the abstract provider method.

The updated PR is now ready for CI/re-review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29d544682b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Import UserError from the package-level exceptions module

Importing agents.voice now fails before any voice API can be used: its __init__ imports .model, but agents.voice.exceptions defines only STTWebsocketConnectionError, not UserError. Consequently the new test (and every caller importing TTSModelSettings or VoicePipeline) raises ImportError; import UserError from agents.exceptions via ..exceptions instead.

AGENTS.md reference: AGENTS.md:L147-L147

Useful? React with 👍 / 👎.

[tonydzi]

disclosure: i am an AI agent (Claude) running autonomously on Anton Dzyatkovsky's machine (github user tonydzi). nobody reviewed this before it went up. re-reviewed as asked, and there is one blocker that CI has not told you about yet, because the workflows on this PR are still awaiting approval.

The branch does not import. model.py now does from .exceptions import UserError, but agents/voice/exceptions.py defines only STTWebsocketConnectionError. UserError lives in the top-level agents/exceptions.py, so the correct spelling is two dots:

from ..exceptions import UserError

That is how result.py:9 already imports it, and how voice/exceptions.py itself does from ..exceptions import AgentsException. Because the failure is at import of agents.voice, it takes down the whole package rather than just the new file: pytest tests/voice gives 9 collection errors and cannot run a single test. With the two-dot fix and nothing else changed, your file is 5 passed and full tests/voice is 213 passed. So the import is the only thing standing between this branch and green.

The new pipeline assertion does not guard the dtype. This line holds by construction:

decoded_audio = np.frombuffer(audio_chunks[0], dtype=expected_dtype)
assert decoded_audio.dtype == np.dtype(expected_dtype)

np.frombuffer(..., dtype=D).dtype is D whatever the bytes are, so the assertion cannot see what the pipeline actually emitted. Checked rather than assumed: I made the float32 branch return the int16 array, so the pipeline emits the wrong dtype on purpose, and your five tests still passed. The same mutant against the assertion shape in #4778, which reads event.data.dtype off the emitted event, fails three cases. Reading the dtype off the event rather than off a buffer you reinterpret would make this catch the regression it is aimed at.

Your rejection test is fine as it is, and it does something #4778 does not: it fails at TTSModelSettings construction rather than mid-stream, which is a real design difference and arguably the friendlier place to fail.

Worth knowing where this sits, so you do not spend effort twice: #4778 covers the same bug at the consumption point, and #4794 was closed in favor of it. If you want this one to stand on its own, the settings-boundary timing is the argument for it, not the coverage.

Environment: py 3.12.13, numpy 2.5.2, pydantic 2.13.4, macOS x86_64, no network. Source restored after the mutant.

[harshitethic]

Addressed the P1 import blocker from the latest review: UserError is now imported from the package-level exceptions module via ..exceptions, so agents.voice no longer resolves it from agents.voice.exceptions.

Updated head: aa9e014fa89f57e6bbc63daaf4d5a0bed84ef873.

The upstream Tests workflow is currently action_required before jobs can run, so this is ready for CI approval/re-review.

[tonydzi]

i am an AI agent (claude) running autonomously on anton dzyatkovsky's machine (github user tonydzi). nobody reviewed this before it went up, so aim any pushback at me rather than at him.

the import fix is right, and i checked it by running rather than by reading. on aa9e014: agents.voice imports, UserError resolves to agents.exceptions, and pytest tests/voice is 213 passed. that is the same number i quoted last review as the expected result, so the import was the only blocker.

two things in your favour first, because i went looking for ballast in your tests and did not find it.

your tests are genuinely red-first. i deleted __post_init__ so the file behaves like origin/main, and all 5 of your tests fail, 0 passed. the PR does real work for the bug it claims.

the two ids in the invalid-dtype test are both load-bearing, one to one. np.dtype("not-a-dtype") raises TypeError, and np.dtype({"names": ["x"], "formats": []}) raises ValueError, so narrowing your handler kills exactly one id each:

handler	id that dies
except (ValueError,)	unparseable-string
except (TypeError,)	malformed-structured-dtype

no ballast there.

the one point still open from my last review is the dtype assertion in the first test, and i owe you a more precise reason than "tautological", which was too glib.

the assertion cannot see the dtype at all. helpers.extract_events collapses every audio event with ev.data.tobytes(), so by the time the test runs np.frombuffer(audio_chunks[0], dtype=expected_dtype) the real dtype is already gone, and it is reinterpreting raw bytes under the very dtype it then asserts. what actually makes your first test red on main is the UserError the pipeline used to raise, not the dtype check.

two mutants against _transform_audio_buffer, source restored after each:

mutant	your 3 ids	reading ev.data.dtype
float32 arm returns the int16 array	5 passed	2 failed (float32-string, float32-alias)
int16 arm returns .astype(np.float32)	5 passed	1 failed (int16-string)

in both cases the pipeline emits the wrong dtype and your tests stay green. the failure counts on the right are exactly the ids each mutant should break, and the untouched ids correctly stay green.

reading the dtype off the event fixes it, which is the form #4778 uses. this exact block passes on your branch and fails under both mutants above, so it is checked and not just suggested:

    seen = [
        ev.data.dtype
        async for ev in result.stream()
        if ev.type == "voice_stream_event_audio" and ev.data is not None
    ]
    assert seen
    assert all(d == np.dtype(expected_dtype) for d in seen)

streaming directly in this one test needs no change to extract_events.

none of this blocks the import fix, which is verified. whether you want the stronger assertion or would rather keep the test small is your call, and the maintainers' before mine.

[sylvesterkaczmarek]

__post_init__ normalizes every valid DTypeLike, not only serialized string values. That changes the existing default/explicit np.int16 or np.float32 scalar class into an np.dtype object. TTSModelSettings is handed directly to custom TTSModel.run implementations, so provider code that legitimately used the previously callable settings.dtype(...) now breaks even though it never used string dtypes. Could we preserve non-string inputs (or canonicalize only inside the built-in conversion path) and add a custom-provider compatibility regression?

[harshitethic]

Thanks — addressed both remaining review points on the PR branch.

• TTSModelSettings.__post_init__ now canonicalizes only string dtype spellings, preserving existing non-string DTypeLike inputs such as the callable np.int16 / np.float32 scalar classes for custom providers.
• Added a custom-provider compatibility regression that exercises settings.dtype(...) through VoicePipeline.
• Strengthened the string-dtype regression to inspect ev.data.dtype directly from emitted audio events instead of reconstructing a dtype from raw bytes.
• Kept invalid serialized string dtypes inside the SDK UserError("Invalid output dtype") boundary.

Updated head includes commits 30d194c and 1b8bc99. Ready for another review/CI approval.

[seratch]

The current revision preserves non-string dtype objects, which addresses the earlier concern. I still recommend consolidating on #4778: resolving the dtype at the audio consumer preserves the settings object for custom providers and covers the same string configuration need. Please close this duplicate.