feat(oci): acquire CNCF ModelPack artifacts via llmman serve

[ericcurtin]

Description

The oci:// scheme extracts the reference as a container image: ExtractOCIImage writes the layers to a docker-save tar and unions that filesystem into the destination. A CNCF ModelPack artifact is not a runnable image -- its layers are typed application/vnd.cncf.model.{weight,weight.config,doc,code,dataset}.v1.{raw,tar,tar+gzip,tar+zstd}, an uncompressed layer is a bare file rather than a tar, and there is no filesystem to union. So oci:// today works for a model baked into a container image, but not for a model artifact.

name: my-model
parameters:
  model: oci://ghcr.io/org/model:tag

Implementation

Detection stays local (pkg/oci/modelpack.go): artifactType == application/vnd.cncf.model.manifest.v1+json, falling back to the config descriptor's mediaType for registries and clients that predate or strip artifactType. v1.Manifest has no ArtifactType field, so the raw manifest is parsed.

Acquisition is delegated to a running llmman serve rather than hand-rolled here. llmman already implements the ModelPack media types, registry auth, resumable blob download and a content-addressed store -- that is registry protocol code LocalAI has no particular interest in owning, and it covers layouts a hand-rolled reader would have to grow one at a time (.raw vs .tar vs .tar+gzip vs .tar+zstd, filepath annotations, image indexes).

New pkg/llmman is the client:

• GET /api/version probes reachability and identity -- a server answering without a version field is reported as "not an llmman daemon", which is worth distinguishing from nothing listening.
• POST /api/pull streams NDJSON so a multi-gigabyte fetch is not silent; status/completed/total map onto the existing downloadStatus callback. An error arrives in-band at HTTP 200, and a stream that simply ends without success is also a failure -- both are treated as errors rather than a completed pull.
• llmman resolve --no-pull reports where the bytes landed. The daemon deliberately exposes no local path (/api/show returns only a digest and size), so the CLI is the documented interface for this; --no-pull guarantees it only reports on what /api/pull already fetched, keeping the daemon the only thing that touches the network.

So a pull needs both the daemon reachable and the binary on PATH (or LOCALAI_LLMMAN_BIN); each missing piece has its own actionable error. LLMMAN_HOST is honoured with the same parsing llmman's own clients use, including rewriting a wildcard bind (0.0.0.0, [::]) to loopback.

Files are hard-linked out of llmman's store where possible, falling back to a copy across filesystems, so a model shared with llmman costs its bytes once rather than twice.

pkg/downloader/uri.go gains one branch after signature verification. A manifest that is not ModelPack takes exactly the path it did before, so container-image behaviour is bit-for-bit unchanged.

Testing

Verified, actually executed:

$ go test ./pkg/oci/ ./pkg/llmman/ ./pkg/downloader/
ok  github.com/mudler/LocalAI/pkg/oci         188.364s
ok  github.com/mudler/LocalAI/pkg/llmman
ok  github.com/mudler/LocalAI/pkg/downloader    6.198s

• pkg/llmman (13 tests, against a real httptest server -- no daemon needed): endpoint defaults and every LLMMAN_HOST form incl. wildcard-to-loopback rewriting; /api/version accepted, a non-llmman server rejected, nothing-listening reported actionably; pull success with forwarded byte progress; in-band error at HTTP 200; stream ending without success; non-OK status; a non-JSON diagnostic line tolerated; the full resolve output contract incl. leaked-diagnostic and unknown-field tolerance and six malformed cases; LOCALAI_LLMMAN_BIN default/override/empty-override; missing-binary error.

• pkg/downloader (4 tests): linkOrCopyTree over a nested directory, asserting st_ino equality to prove the file was hard-linked rather than copied; a single-file (GGUF) payload; overwriting a stale destination; a missing source.

• pkg/oci (5 specs): detection by artifactType, by config mediaType, a plain container image not claimed, neither-discriminator not claimed, unparseable manifest errors.

• go build ./pkg/..., go vet, gofmt -l all clean on touched files (note: pkg/downloader/cancel_test.go is unformatted on master already; untouched here)

Not verified here, flagged rather than implied: no run against a real registry serving a ModelPack artifact, and no end-to-end model load against a live llmman serve. Coverage is at the client/protocol level with a stub server.

Notes for Reviewers

Two judgement calls worth a look:

1. Why a daemon and a binary, rather than just one. llmman serve owns the download but exposes no path; llmman resolve knows the path but would pull in-process. Using /api/pull + resolve --no-pull keeps all network I/O in the daemon (one shared cache, one place to configure registry credentials) while still learning where the files are. The cost is two dependencies instead of one, which is why each has a distinct error message.
2. Hard-linking rather than copying. This assumes LocalAI's models directory and llmman's store are usually on the same filesystem. When they are not, it degrades to a copy silently, which is the pre-existing behaviour anyway.

[X] Yes, I signed my commits.

Disclosure: this change was written with AI assistance. I have reviewed it.

[localai-org-maint-bot]

Thanks for the detailed implementation and test notes. This needs a few repository-policy changes before it can merge:

1. pkg/llmman/client.go constructs raw http.Client values in DefaultClient and ProbeClient. All outbound HTTP in LocalAI must use pkg/httpclient (httpclient.New(...) for the streaming pull and httpclient.NewWithTimeout(...) for the version probe) so redirects and TLS use the project hardening policy.

2. The new tests in pkg/llmman/client_test.go and pkg/downloader/llmman_test.go use t.Fatal, t.Fatalf, and t.Errorf. New Go tests must use Ginkgo v2/Gomega; these calls are explicitly rejected by the repository lint policy. Please convert these cases to the existing package suites (or add a Ginkgo suite bootstrap where the package does not have one).

3. This adds user-facing behavior and two configuration variables (LLMMAN_HOST and LOCALAI_LLMMAN_BIN) but no page under docs/content/. The docs-with-code rule requires the setup, daemon/binary dependency, configuration, and oci:// ModelPack example in this PR.

Once those are addressed, the protocol and filesystem behavior can get a focused follow-up review.