Fix adding authorized keys in Kubernetes and Slurm - #4235
Merged
Conversation
The Kubernetes jump pod added the user's key with `grep -qvF KEY file`, which asks whether some line does not contain the key, not whether no line contains it. The jump pod's `authorized_keys` holds the project key, so the condition held whether or not the user's key was already there, and every run appended another copy of it. Users keep the same key across runs, so the file of a jump pod, shared by the whole namespace, grew by one duplicate per run. Had the file been empty, the condition would have been false and the key never added at all. Slurm got the condition right, but matched the whole entry, comment included, so the same key submitted with a different comment was added a second time. All three implementations -- SSH fleets, Kubernetes, Slurm -- are replaced with `get_add_authorized_keys_script()`, extracted from the SSH fleet one, the only one of the three that was correct. It returns the script instead of running it, as the call sites execute it in three different ways: paramiko, `ssh` over the jump pod, and the Slurm client. * Keys reach the script as heredoc data instead of being interpolated into its commands, so the shell never parses anything that came from a key. * Entries are rebuilt from `parse_public_key()`, a new helper parsing a public key in OpenSSH disk format, so that nothing unvalidated is written to the file. A key that fails to parse is skipped with a warning: one bad key does not keep the rest out of the file. * Entries are deduplicated on the key blob alone, since neither the comment nor the `command="/bin/false"` options Kubernetes and Slurm prepend are a part of the identity of a key. * Kubernetes and Slurm entries now carry the `# added by dstack` marker the SSH fleet entries have carried since #4179. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Kubernetes jump pod added the user's key with
grep -qvF KEY file, which asks whether some line does not contain the key, not whether no line contains it. The jump pod'sauthorized_keysholds the project key, so the condition held whether or not the user's key was already there, and every run appended another copy of it. Users keep the same key across runs, so the file of a jump pod, shared by the whole namespace, grew by one duplicate per run. Had the file been empty, the condition would have been false and the key never added at all.Slurm got the condition right, but matched the whole entry, comment included, so the same key submitted with a different comment was added a second time.
All three implementations -- SSH fleets, Kubernetes, Slurm -- are replaced with
get_add_authorized_keys_script(), extracted from the SSH fleet one, the only one of the three that was correct. It returns the script instead of running it, as the call sites execute it in three different ways: paramiko,sshover the jump pod, and the Slurm client.parse_public_key(), a new helper parsing a public key in OpenSSH disk format, so that nothing unvalidated is written to the file. A key that fails to parse is skipped with a warning: one bad key does not keep the rest out of the file.command="/bin/false"options Kubernetes and Slurm prepend are a part of the identity of a key.# added by dstackmarker the SSH fleet entries have carried since Don't add duplicate keys while provision SSH instance #4179.