Backing up an AI server: what is unique, what can be rebuilt, and how to prove the restore works

What lives on an AI server

An AI server holds three kinds of data: files that can be downloaded again, files that exist nowhere else, and databases that give a consistent copy only when asked.

Our classification; the sources are in the sections below.

Base weights: a download only while the revision exists

Llama 3.3 70B in NVIDIA’s FP8 checkpoint is 15 safetensors files, 72,656,585,760 bytes on Hugging Face: 72.7 GB, the 68 GiB used across this site. Downloading it again restores it only while three conditions hold.

The revision exists. Hugging Face selects a version with the revision parameter, a branch, tag or commit hash; only a commit hash pins the version, and it “must be the full-length hash”; vLLM’s revision setting takes the same values. But owners can rewrite history: Hugging Face calls its super-squash “a destructive operation that cannot be undone”, after which “commit history will be permanently lost”.

Your access stands. Meta’s Llama 3.3 70B Instruct repository is gated with manual approval, and Hugging Face states that model authors “can decide at any time to block your access to the model without prior notice”. Access goes to individual users, not to organisations, so the token that pulls a gated model belongs to a person, and the restore path can leave with them.

The files match. Hugging Face lists a SHA-256 value for each weight file, and hf cache verify checks a local directory against “their checksums on the Hub”. Both depend on the Hub, so keep your own copy of the files you serve, with a SHA-256 manifest stored away from the server.

Adapters and fine-tuned weights: unique and tied to one base

PEFT saves a LoRA adapter as adapter_model.safetensors and adapter_config.json, and the saved state “only contains the parameters of the adapter module, not the base model”. The configuration names the base model and has a revision field for it; in NVIDIA’s NemoGuard content-safety adapter for Llama 3.1 8B that field is null, PEFT’s value when none is given, and the base, Meta’s Llama 3.1 8B Instruct, is gated. Record the base commit yourself.

Size follows rank, target layers and precision. By our arithmetic a rank 16 adapter on every attention and MLP projection of Llama 3.3 70B has 207.1 million parameters: 828 MB in FP32 or 414 MB in BF16. NVIDIA’s adapter, likewise rank 16, is one 1.23 GB file; its configuration also targets lm_head, which PEFT counts among the embedding layers it can save “in addition to adapter weights”. A full fine-tune is a whole new checkpoint: 141.1 GB for the 70B in BF16.

Your own adapters and fine-tunes cannot be downloaded again, and retraining is no reliable way to recreate them: PyTorch does not promise completely reproducible results “across PyTorch releases, individual commits, or different platforms”. Back up each adapter or checkpoint with its training data, evaluation set, configuration and base revision, as one restore unit.

Vector stores: let the database make the copy

Copying a running database’s data directory gives files, not necessarily a consistent state. Each store documents its own route.

PostgreSQL 18 documentation; pgvector README, v0.8.6; Qdrant snapshot documentation; Milvus Backup documentation and README, v0.5.16; Milvus release notes (3.0.0 of 29 July 2026); all read in September 2026.

pgvector keeps vectors as ordinary PostgreSQL data, 4 bytes per dimension plus 8 per vector, and “uses the write-ahead log (WAL), which allows for replication and point-in-time recovery”. pg_dump “makes consistent exports even if the database is being used concurrently”; file-level backups and continuous archiving, by contrast, are “extremely server-version-specific”. A dump carries only a CREATE EXTENSION command, so the target needs pgvector’s “control, script, and other files”, and it rebuilds every HNSW index on restore, faster “when the graph fits into maintenance_work_mem”. A base backup restores index files as they were and allows point-in-time recovery, but only for “the entire database cluster”.

Qdrant snapshots are “tar archive files that contain data and configuration of a specific collection on a specific node”, so a cluster needs one per node, taken with a POST to /collections/{collection_name}/snapshots; full storage snapshots suit single-node deployments only. Recovery defaults to priority replica, which will “prefer existing data over the snapshot”: a new collection recovered that way comes back empty, so “you need to set the priority to snapshot”. Milvus Backup, for example milvus-backup create -n my_backup, copies metadata and data into a backup root path while the instance stays “fully functional”.

Rebuilding the index works while the source documents and the exact embedding model survive; switching models means new vectors, and Qdrant’s advice is to “re-embed points”. Copying is cheap by comparison: 10 million chunks at 1,024 dimensions are 41 GB of vector data before indexes and row overhead, by our arithmetic. The access rights stored with each chunk are only as current as the last sync: re-sync them from the source systems before users query a restored index.

Virtual machines, passthrough GPUs and Kubernetes

Image-level VM backup rests on snapshots: “When you back up a VM, Veeam Backup & Replication requests VMware vSphere to create a VM snapshot.” With a GPU in passthrough that route is closed while the VM runs: Broadcom’s vSphere 9.0 documentation lists snapshots among the features “unavailable for virtual machines configured with DirectPath”, and a Broadcom knowledge-base article for ESXi 8.0 adds that “Snapshots succeed on powered-off VMs even with passthrough devices present”. A running passthrough VM, like a bare-metal GPU server, is protected from inside: a guest agent such as Veeam Agent for Linux, built for “physical endpoints and virtual machines running Linux-based operating systems”, plus the application-level copies above. For vGPU VMs we found no snapshot restriction in current NVIDIA or Broadcom documentation; prove the route with one test backup and restore.

On Kubernetes, Veeam Kasten (release 9.0.6 in September 2026) captures an application’s namespaced resources such as ConfigMaps and Secrets, its workloads, Helm v3 release information and “all persistent storage resources” associated with the workloads; a model cache on the node rather than in a persistent volume is outside that list. Kasten also warns that “catastrophic storage system failure will destroy your snapshots along with your primary data”, so export them to an external location. Pin every image by digest, <image-name>@<digest>: in Kubernetes’ words, “An image digest uniquely identifies a specific version of the image”.

Where the copies go, and how long they take

Protect the NVMe tiers differently: the model store is large and read-mostly, so copy it once per new revision, not nightly. Adapters, evaluation sets, configuration and logs are small and unique: every backup takes them, and training data gets its own schedule. One copy should be out of reach of every administrator account: Veeam’s 3-2-1-1-0 rule adds one immutable or air-gapped copy and zero errors after recovery verification to three copies on two media, one of them off-site. Our article on immutable backup covers the mechanics.

At line rate, before protocol overhead, disk speed and the backup software’s own work, the network sets this floor:

Our arithmetic: bytes × 8 ÷ line rate. Sizes from the Hugging Face file listings of NVIDIA’s FP8 checkpoint (15 files) and Meta’s BF16 checkpoint (30 files), September 2026. Lower bounds for the copy alone.

A copy time is one term of a recovery, not an estimate of it: our articles on backup and disaster recovery and on RPO and RTO cover the rest of the clock and how to set the targets. We recommend a second 25 GbE port to keep backup traffic and model pulls off the network that serves users.

Restore tests that prove something

A green restore job proves that bytes arrived. Three tests show that the AI service came back, each against a baseline recorded while everything works.

Files. Check every restored weight, adapter and tokenizer file against the SHA-256 manifest, for example with sha256sum --check, which exits with a non-zero status on a mismatch. For Hub models, hf cache verify with --revision, --local-dir and --fail-on-missing-files, without which it only warns about a missing file, compares a restored directory with the Hub’s values.

Answers. Run the fixed evaluation set that accepted the model on the restored stack, against a pass mark set in advance. Do not expect identical text: vLLM does not promise reproducible results by default, “for the sake of performance”, and even with its reproducibility settings “only provides reproducibility when it runs on the same hardware and the same vLLM version”. Grade answers, not strings.

Retrieval. Run fixed queries against the restored index and compare the top results with the baseline. pgvector’s README notes that, unlike typical indexes, “you will see different results for queries after adding an approximate index”, so set the pass mark in advance: the same IDs from restored index files, an agreed overlap from an index rebuilt by a dump restore or from re-embedded chunks. Add one query per access level to confirm that permissions came back.

Regular test restores to verify backup integrity are part of the cyber resilience service our engineering partner Vixen.UNO delivers; on an AI server, write these three checks into the test plan.

What we supply

Eurokommerz supplies AI servers built to order, with NVMe tiers for scratch, model store and indexes and network cards from 25 to 400G, under manufacturer warranty with an EU contract and invoicing. The backup side is engineered by the Vixen.UNO team, our engineering partner, under the same contract: its cyber resilience service covers Veeam-based backup with a recovery site in Baltneta’s Tier-3 data centres in Lithuania, regular test restores and an incident response plan with roles, actions and deadlines; capacity, support terms and recovery targets are fixed in the SLA. Virtual-machine replication to a recovery site, where systems come up from replicas and run while your primary site is down, is the disaster recovery service.

FAQ