Uploading

upload_artifact hashes your inputs, uploads whatever bytes the server doesn’t already have (as a single PUT or a parallel multipart upload, whichever the server directs), commits a new immutable version, and optionally points aliases at it. It returns the committed version’s VersionInfo.

metrana.upload_artifact(
    files,
    *,
    type,                 # e.g. "model", "dataset"
    name,
    workspace=None,       # defaults to the active run's workspace
    aliases=None,
    metadata=None,
    description=None,
    references=None,
    storage=None,         # "s3://my-bucket/prefix" to upload to a bucket you own
    created_by_run=None,
    client_token=None,
    api_key=None,
    base_url=None,
    concurrency=None,
) -> VersionInfo

What you can upload

files accepts a single path, or a list of paths, each of which is a file or a directory:

# A single file — stored under its basename ("model.safetensors").
metrana.upload_artifact("model.safetensors", type="model", name="llm")

# A directory — walked recursively; each file keeps its path relative to the directory.
# checkpoints/step-5000/{weights.bin, config.json} -> "weights.bin", "config.json"
metrana.upload_artifact("checkpoints/step-5000/", type="model", name="llm")

# A mix — files land under their basename, directories under their relative contents.
metrana.upload_artifact(
    ["config.json", "weights/", "tokenizer.model"], type="model", name="llm",
)

The in-artifact path is what download_artifact recreates and what list_artifact_files reports. Two inputs that resolve to the same in-artifact path (for example a file and a directory that both yield config.json) are rejected with a MetranaArtifactValidationError before anything uploads.

Metadata and description

Attach queryable metadata and a human-readable description at upload time. Both are recorded on the version and returned in its VersionInfo:

metrana.upload_artifact(
    "checkpoints/step-5000/",
    type="model", name="llm",
    description="step 5000, eval loss 1.83",
    metadata={"step": 5000, "eval_loss": 1.83, "base": "llama-3-8b"},
)

Setting aliases on upload

Pass aliases to point one or more user aliases at the new version immediately after it commits:

metrana.upload_artifact(
    "checkpoints/step-5000/", type="model", name="llm",
    aliases=["best-eval", "v2.1"],
)

The latest alias is managed automatically and cannot be set here.

Aliases are applied after commit

The version commits first; aliases are a separate, best-effort follow-up. If the upload succeeds but an alias can’t be applied, upload_artifact raises MetranaArtifactAliasError — but the version is safe and already committed. The error carries version_index, applied, and failed, so you can re-point the failed aliases with set_artifact_alias without re-uploading:

from metrana.utils.exceptions import MetranaArtifactAliasError

try:
    metrana.upload_artifact("ckpt/", type="model", name="llm", aliases=["production"])
except MetranaArtifactAliasError as e:
    for alias in e.failed:
        metrana.set_artifact_alias(
            type="model", name="llm", alias=alias, version_index=e.version_index,
        )

References: catalogue bytes you already store

A reference records a file that lives in your own bucket. Metrana stores its path, digest, and size but never copies or serves the bytes — so you can catalogue large existing datasets without re-uploading them. Provide references as a list of dicts alongside your managed files:

metrana.upload_artifact(
    "index.json",                         # at least one managed file is required
    type="dataset", name="webcrawl-2024",
    references=[
        {
            "path": "shards/000.tar",
            "storage_uri": "s3://my-bucket/webcrawl/000.tar",
            "digest": "<sha256-hex>",
            "size": 1073741824,
            "content_type": "application/x-tar",   # optional
            "etag": "…",                            # optional
        },
    ],
)

This path pairs references with managed files: you still upload at least one file to Metrana (upload_artifact requires a non-empty files), and each reference points at bytes you placed in your bucket yourself. On download, references are fetched only if you opt in with storage= — see Downloading. To store all of an artifact’s bytes in your own bucket instead, use storage= (below).

Bring your own bucket

Pass storage="s3://<bucket>/<prefix>" to upload the bytes to an S3 bucket you own, using the AWS credentials in your environment. Metrana stores only the metadata (paths, digests, and the s3:// URIs) and never touches the bytes — the objects flow directly from your machine to your bucket. This is what you want when data-residency or egress rules mean artifacts must stay in your own account.

metrana.upload_artifact(
    "checkpoints/step-5000/",
    type="model", name="llm",
    workspace="acme",
    storage="s3://acme-artifacts/metrana",
)

This needs the s3 extra — pip install 'metrana[s3]' (see Installation) — and standard AWS credentials in your environment (the usual boto3 chain: environment variables, shared config, or an instance/pod role). Your credentials never leave your machine; Metrana neither receives nor stores them.

Bytes are written content-addressed at <prefix>/<workspace>/blobs/<sha256>, so identical bytes are stored once and re-running an upload skips objects that are already present. Because the files are recorded as references, downloading them back needs the same storage= opt-in.

storage= vs references=

references= catalogues bytes you already placed in your bucket, alongside at least one Metrana-managed file. storage= makes the SDK upload your files to your bucket for you, with no Metrana-managed bytes at all. Either way the files are recorded server-side as references.

Recording run lineage

If the artifact is the output of a training run, pass that run’s id as created_by_run to record lineage:

metrana.upload_artifact("ckpt/", type="model", name="llm", created_by_run="<run-uuid>")

Note

created_by_run is not auto-populated from an active metrana.init(...) run yet — pass it explicitly if you want the lineage recorded.

Idempotency and resuming

Every call uses a client_token to make the upload idempotent. By default a fresh random token is generated per call. Pass your own token to resume the same draft — if a long upload is interrupted, re-running with the same token picks up where it left off instead of starting a new version, and already-transferred bytes (content-addressed) are not re-sent:

TOKEN = "nightly-export-2024-07-14"
metrana.upload_artifact("big-dataset/", type="dataset", name="corpus", client_token=TOKEN)
# interrupted? re-run with the same TOKEN to resume the same draft.

Tuning throughput

concurrency caps the number of simultaneous byte transfers (default 16). Files upload in parallel, and a single large file is split into parts that upload in parallel too. Raise it on a fat pipe, lower it if you’re saturating a shared link:

metrana.upload_artifact("big-dataset/", type="dataset", name="corpus", concurrency=32)

See Configuration for api_key, base_url, and how the workspace is resolved.

Errors

upload_artifact raises the MetranaArtifact* hierarchy: MetranaArtifactValidationError for empty or duplicate inputs, MetranaArtifactQuotaError if the upload would exceed a storage ceiling, MetranaArtifactAliasError for the alias case above, and MetranaArtifactError (the base) for any other API or transfer failure.