Artifact Management & Release Engineering

Packaging Across Ecosystems

18 min Lesson 4 of 28

Packaging Across Ecosystems

A DevOps engineer must ship artifacts in whatever format the consumer ecosystem demands — a Java service into a Maven repository, a Python library onto PyPI, an internal CLI as a Debian package, and a microservice as an OCI container image. Each format carries its own metadata conventions, dependency resolution model, signing contract, and failure mode. Understanding the common patterns — not just the commands — is what separates an engineer who can debug a broken release at 2 AM from one who cannot.

JVM Artifacts: JARs, WARs, and Maven Coordinates

Java artifacts published to Maven-compatible repositories (Nexus, Artifactory, GitHub Packages, Maven Central) are identified by three coordinates: groupId:artifactId:version (GAV). The repository lays files out deterministically at groupId/artifactId/version/artifactId-version.jar, which allows any build tool to fetch a reproducible dependency graph.

A production Gradle build publishing to an internal Artifactory instance looks like this:

# build.gradle.kts — publishing a library JAR to Artifactory
plugins {
    `maven-publish`
    `java-library`
}

group = "com.acme.platform"
version = providers.environmentVariable("RELEASE_VERSION").orElse("0.0.0-SNAPSHOT").get()

publishing {
    publications {
        create<MavenPublication>("mavenJava") {
            from(components["java"])
            pom {
                name.set("ACME Platform Core")
                description.set("Internal shared utilities")
                licenses {
                    license { name.set("Apache-2.0") }
                }
            }
        }
    }
    repositories {
        maven {
            name = "artifactory"
            url = uri(System.getenv("ARTIFACTORY_URL") ?: "https://artifactory.internal/libs-release-local")
            credentials {
                username = System.getenv("ARTIFACTORY_USER")
                password = System.getenv("ARTIFACTORY_PASS")
            }
        }
    }
}

# Publish from CI
./gradlew publishMavenJavaPublicationToArtifactoryRepository \
  -PRELEASE_VERSION=2.4.1

SNAPSHOT vs. release repositories are separate by design. Artifactory and Nexus enforce this by rejecting SNAPSHOT uploads to release repos and vice versa. Never publish a -SNAPSHOT version to a release repository — snapshots are mutable by definition, which means the same coordinate can resolve to different bytes on different days. Release artifacts must be immutable. Mixing them causes non-reproducible builds that are almost impossible to audit after an incident.

Python Wheels: Why Wheels Beat Source Distributions

Python packaging has two artifact types: source distributions (.tar.gz, sdist) and binary wheels (.whl). A wheel is a ZIP archive with a name encoding the Python version, ABI, and platform: mylib-1.2.0-cp311-cp311-manylinux_2_28_x86_64.whl. Pip installs a wheel by unzipping it — no compilation step, no build toolchain required at install time. This matters enormously for CI speed and for reproducibility in production deploys.

Publishing an internal library to a private PyPI (Nexus, Artifactory, or Google Artifact Registry) with twine:

# Build source dist + wheel
python -m build          # produces dist/mylib-1.2.0.tar.gz and dist/mylib-1.2.0-py3-none-any.whl

# pyproject.toml (PEP 517/518 — the modern standard)
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.backends.legacy:build"

[project]
name = "mylib"
version = "1.2.0"
requires-python = ">=3.11"
dependencies = [
    "httpx>=0.27",
    "pydantic>=2.7",
]

[project.optional-dependencies]
dev = ["pytest", "mypy", "ruff"]

# Upload to private index (TWINE_PASSWORD is an API token)
TWINE_USERNAME=__token__ \
TWINE_PASSWORD="$PYPI_API_TOKEN" \
twine upload \
  --repository-url https://artifact-registry.internal/simple/python/ \
  dist/*

# Consume from private index in requirements.txt
--extra-index-url https://__token__:${PYPI_API_TOKEN}@artifact-registry.internal/simple/
mylib==1.2.0

Use manylinux wheels for compiled extensions. If your library wraps a C extension, build inside the official manylinux_2_28 Docker image and use auditwheel repair to bundle the exact shared libraries into the wheel. This produces a self-contained artifact installable on any glibc-compatible Linux without requiring the end user to have matching system libraries — the same principle as statically linking a Go binary.

npm Packages: Shrinkwrap, Provenance, and Scope Hygiene

npm packages published to a registry (npmjs.com or Verdaccio/Artifactory internally) are tarballs with a package.json manifest. The key operational concern in production is supply chain integrity — npm's public registry has been a vector for typosquatting and dependency confusion attacks. Scoping all internal packages under a private namespace and pointing that namespace to your internal registry completely closes the dependency confusion surface.

# .npmrc — route @acme/* packages to internal registry; public packages to npmjs
@acme:registry=https://registry.internal.acme.com/
//registry.internal.acme.com/:_authToken=${NPM_INTERNAL_TOKEN}
registry=https://registry.npmjs.org/

# package.json for an internal shared library
{
  "name": "@acme/design-tokens",
  "version": "3.0.0",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
  "files": ["dist/"],
  "scripts": {
    "build": "tsc --project tsconfig.build.json",
    "prepublishOnly": "npm run build"
  },
  "publishConfig": {
    "registry": "https://registry.internal.acme.com/",
    "access": "restricted"
  }
}

# Publish from CI (token scoped to publish only)
npm publish --provenance   # attaches SLSA provenance attestation (npm >= 9.5)

# Lock exact versions for production deploys
npm ci                     # installs from package-lock.json exactly; fails if lock is stale

Debian Packages: deb for System-Level Artifacts

For agents, daemons, CLIs, and anything installed on a bare VM or Debian-based container, .deb packages are the gold standard. They integrate with apt for dependency resolution, support pre/post install hooks (preinst, postinst) for user creation and service registration, and handle config file management (conffiles) so upgrades do not overwrite operator edits. Building debs with nfpm (a modern, config-driven alternative to dpkg-buildpackage) is common in Go and Rust shops.

# nfpm.yaml — package config for a Go daemon
name: "myagent"
arch: "amd64"
platform: "linux"
version: "${RELEASE_VERSION}"
maintainer: "Platform Team <platform@acme.com>"
description: "ACME monitoring agent"
section: "utils"
priority: "optional"

contents:
  - src: ./dist/myagent-linux-amd64
    dst: /usr/bin/myagent
    file_info:
      mode: 0755
  - src: ./configs/myagent.yaml
    dst: /etc/myagent/myagent.yaml
    type: config|noreplace    # preserves operator edits on upgrade
  - src: ./systemd/myagent.service
    dst: /lib/systemd/system/myagent.service

scripts:
  postinstall: ./scripts/postinstall.sh   # systemctl daemon-reload && systemctl enable myagent
  preremove: ./scripts/preremove.sh       # systemctl stop && systemctl disable myagent

# Build all formats in one shot
nfpm package --config nfpm.yaml --packager deb --target dist/
nfpm package --config nfpm.yaml --packager rpm --target dist/

# Sign with GPG and publish to an APT repo (Aptly or Pulp)
dpkg-sig --sign builder dist/myagent_1.0.0_amd64.deb
aptly repo add stable dist/myagent_1.0.0_amd64.deb
aptly publish update stable

Container Images: OCI Standard, Layers, and Attestations

OCI container images are the universal packaging format for server-side workloads. At big-tech scale, the discipline is not "build a Dockerfile" — it is minimizing attack surface, build time, and layer cache invalidation simultaneously.

Container image layers: only changed layers are rebuilt; stable dependency layers are cached and reused across builds.

The critical discipline for production container images is layer ordering and multi-stage builds. Place the most stable layers (base OS, OS packages, language runtime) at the top of the Dockerfile so they are cached across builds. Copy dependency manifests and install them before copying application code. This means a code-only change skips all the slow package install steps. Use multi-stage builds so the final image contains only the runtime artifact, not the compiler, test tools, or build cache.

Pin by digest, not by tag. A tag like node:20-alpine is mutable — the registry owner can push a different image to the same tag at any time. In production CI, pin base images by their immutable SHA-256 digest: FROM node:20-alpine@sha256:a1b2c3.... This guarantees your builds are hermetic: the same Dockerfile always produces the same intermediate layers, making security audits meaningful and rollbacks predictable.

The Common Pattern: Content-Addressable, Signed, and Attested

Despite the surface differences between a .jar, a .whl, an npm tarball, a .deb, and an OCI image, every modern ecosystem converges on the same three guarantees at the artifact level:

Content-addressed storage — the artifact is identified by a cryptographic hash of its contents (SHA-256), not by a mutable name. If two artifacts share a hash, they are byte-for-byte identical.
Signing — a private key signs the artifact or its manifest; the public key is distributed out-of-band. Consumers verify the signature before trusting the artifact. OCI uses cosign; Maven uses GPG; deb repos use APT key infrastructure; npm provenance uses OIDC-based Sigstore attestations.
SBOM (Software Bill of Materials) — a machine-readable manifest of every dependency bundled into the artifact. Required for supply chain audits, CVE scanning, and SLSA level 3+ compliance. Tools: syft for container images, cyclonedx-maven-plugin for JARs, cyclonedx-bom for Python.

One registry to rule them all. At scale, run a single proxy + host + virtual repository in Artifactory or Nexus that aggregates Maven Central, PyPI, npmjs, Docker Hub, and your internal repos behind one URL. Every build pulls through this proxy, which caches upstream packages. If an upstream registry goes down (Docker Hub rate-limits, npm has an outage), your builds continue from cache. This single change eliminates an entire class of CI failures and is one of the first things a Platform Engineering team sets up.