Linux Fundamentals

Environment Variables & Configuration

22 min Lesson 9 of 26

Environment Variables & Configuration

Every process on Linux inherits a set of environment variables — named key-value pairs that configure its runtime behaviour without hard-coding values into the program itself. Environment variables are how you tell a database client which host to connect to, how you inject secrets into containers without committing them to source control, and how the shell knows where to find every command you type. Understanding them deeply is non-negotiable for production DevOps work.

What Environment Variables Are and How They Work

When the kernel forks a new process, it passes two things to that process: its arguments (argv) and its environment (envp) — a null-terminated array of KEY=VALUE strings. The child process can read any of these strings, and it inherits a full copy of the parent's environment. Critically, a child process can only see its own copy; changes a child makes do NOT propagate back to the parent. This is a fundamental Unix design principle: environment is inherited downward, never upward.

Read and manipulate environment variables with these essential commands:

# Print all environment variables in the current shell
printenv

# Print a single variable
printenv HOME
echo $HOME          # shell expands the variable

# Set a variable in the current shell session (not exported to child processes)
MY_VAR=hello
echo $MY_VAR        # hello

# Export it so child processes inherit it
export MY_VAR=hello
export DB_HOST=postgres.internal

# Unset a variable
unset MY_VAR

# Run a single command with a temporary environment override
# (does NOT modify the current shell)
DB_HOST=staging.internal ./my-app

# See the exact environment a running process has (replace PID)
cat /proc/12345/environ | tr '\0' '\n'

Key insight: VAR=value command (no export) sets the variable only for that one command invocation. It is the cleanest way to test with a different config without polluting your shell session. Production scripts use this pattern extensively.

The PATH Variable — The Most Important Variable on Any System

PATH is a colon-separated list of directories the shell searches, in order, when you type a command name without a path. When you type docker, the shell walks each directory in PATH and executes the first match it finds. If no match exists, you get command not found.

# Inspect your PATH
echo $PATH
# /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

# Find exactly which binary will run for a given command name
which python3         # /usr/bin/python3
type python3          # python3 is /usr/bin/python3  (shell built-in check too)
command -v python3    # /usr/bin/python3  (POSIX-portable; use in scripts)

# Add a directory to the front of PATH (higher priority)
export PATH="/opt/myapp/bin:$PATH"

# Common locations you will add in a DevOps career:
# /usr/local/bin          -- homebrew/snap/custom installs
# $HOME/.local/bin        -- user-installed tools (pip install --user, etc.)
# /opt/homebrew/bin       -- macOS Apple Silicon brew
# $HOME/go/bin            -- Go binaries
# $HOME/.cargo/bin        -- Rust binaries (cargo install)
# /snap/bin               -- Ubuntu Snap packages

Production pitfall — PATH order matters: Prepending to PATH means your custom binary shadows a system binary of the same name. This is intentional when you need a newer version, but dangerous if done carelessly. A common incident: a developer installs a local curl that has debug flags enabled and prepends its directory, then a systemd service that calls curl picks up the debug build and fails silently. Always know which binary wins with which <command>.

Shell Profiles and Configuration Precedence

The shell reads different configuration files depending on how it is invoked. Getting this wrong is the source of many "works on my machine, breaks in cron / CI / SSH" bugs. There are two axes:

Login shell vs. non-login shell: A login shell is the first shell session that starts when you authenticate — e.g., SSH login, su - username, or a terminal emulator set to login mode. A non-login shell is any subsequent shell — e.g., opening a new terminal tab, running bash from within another script, or a subshell.
Interactive vs. non-interactive shell: An interactive shell has a terminal attached and prompts the user. A non-interactive shell runs a script and exits — this is what cron jobs and CI runners use.

Shell profile load order by session type. Non-interactive shells (cron, CI) do NOT load ~/.bashrc — a common source of "works in terminal, fails in cron" bugs.

The practical rules for where to put things:

/etc/profile and /etc/profile.d/*.sh: System-wide settings applied to every user at login. In production, configuration management tools (Ansible, Puppet) write here to configure PATH additions or proxy settings fleet-wide. Never put secrets here.
~/.bash_profile (or ~/.profile): Per-user, login shell only. The canonical place to set and export environment variables that tools need everywhere (e.g., GOPATH, JAVA_HOME, NVM_DIR). By convention, this file sources ~/.bashrc so interactive shells also benefit.
~/.bashrc: Per-user, non-login interactive shell. Aliases, shell functions, prompt customisation, and completions go here. Do NOT put long-running commands or heavy computation here — it runs on every new terminal tab.
/etc/environment: PAM-read file. Key-value pairs, no shell syntax. Variables set here are available to ALL processes started by PAM (login, SSH, graphical sessions) regardless of which shell they use. Ideal for LANG, TZ, proxy settings, or feature flags that every process on the host must see.

# /etc/environment -- plain KEY=VALUE, no export keyword, no shell expansion
LANG=en_US.UTF-8
TZ=UTC
HTTP_PROXY=http://proxy.internal:3128
HTTPS_PROXY=http://proxy.internal:3128
NO_PROXY=localhost,127.0.0.1,.internal

# ~/.bash_profile -- sources ~/.bashrc, then sets login-time exports
[[ -f ~/.bashrc ]] && source ~/.bashrc

export GOPATH="$HOME/go"
export PATH="$GOPATH/bin:$HOME/.local/bin:$PATH"
export JAVA_HOME="/usr/lib/jvm/java-21-openjdk-amd64"
export EDITOR=vim
export PAGER=less

# ~/.bashrc -- interactive-only: aliases, prompt, completions
alias ll='ls -lah --color=auto'
alias k='kubectl'
alias tf='terraform'

# Git prompt (shows branch name in PS1)
parse_git_branch() { git branch 2>/dev/null | sed -n 's/\* //p'; }
export PS1='\u@\h:\w \[\e[32m\]$(parse_git_branch)\[\e[0m\]$ '

Applying Changes Without Restarting

After editing a profile file, the current shell session does not automatically see the changes. You have two options:

source ~/.bashrc (or the dot shorthand: . ~/.bashrc) — re-executes the file in the current shell. Changes take effect immediately without opening a new terminal.
Open a new shell — a fresh login shell will read all profiles from scratch.

In production automation (e.g., a CI job that installs a tool and then uses it in the same script), you must source the profile or explicitly set the variable — a new subprocess does not inherit changes the parent script made to a file it hasn't sourced.

Environment Variables in Production Systems

In containerised and cloud-native environments, environment variables are the primary mechanism for injecting runtime configuration. The Twelve-Factor App methodology (the design philosophy behind most modern cloud services) mandates that all configuration live in the environment, not in config files baked into the image.

# Dockerfile: never bake secrets into the image
# Use ARG for build-time values, ENV for runtime defaults
FROM ubuntu:24.04
ARG APP_VERSION
ENV APP_ENV=production \
    LOG_LEVEL=info \
    PORT=8080

# docker run: inject at runtime
docker run -e DB_HOST=prod-db.internal \
           -e DB_PASSWORD="$(cat /run/secrets/db_pass)" \
           -p 8080:8080 \
           myapp:latest

# Kubernetes: ConfigMap for non-sensitive config, Secret for credentials
# env from a ConfigMap key
env:
  - name: LOG_LEVEL
    valueFrom:
      configMapKeyRef:
        name: app-config
        key: log_level
  # env from a Secret (base64-encoded in etcd, plaintext to the container)
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: password

Pro practice — never echo secrets: In shell scripts, avoid echo $DB_PASSWORD or set -x (which prints every command and its expanded variables). Secrets leaking into CI logs is one of the most common security incidents in engineering organisations. Use printenv DB_PASSWORD | wc -c to verify a secret is set without revealing its value.

Debugging Environment Problems

When a service or script cannot find a binary, fails to connect to a database, or behaves differently in CI than on your laptop, environment variables are almost always the cause. A systematic debug approach:

# 1. Check what a running process actually sees (not what you think it exported)
cat /proc/$(pgrep -f myapp)/environ | tr '\0' '\n' | sort

# 2. Run a command under the exact same conditions as the failing service
sudo -u deployuser env -i HOME=/home/deployuser PATH=/usr/local/bin:/usr/bin:/bin bash -l -c 'which python3 && python3 --version'

# 3. Print ALL variables at the top of a failing script to capture the snapshot
env | sort > /tmp/env-snapshot-$(date +%s).txt

# 4. Verify variable expansion is what you expect (common pitfall: spaces around =)
# WRONG:  export MY_VAR = "hello"    # bash treats = as a command argument
# RIGHT:  export MY_VAR="hello"

# 5. Check if a required variable is unset vs. empty
[[ -z "${DB_HOST}" ]] && echo "DB_HOST is unset or empty" && exit 1
# Use parameter expansion for default fallback:
DB_HOST="${DB_HOST:-localhost}"

Common failure mode — cron jobs and systemd units: Cron runs jobs with a near-empty environment: no PATH beyond /usr/bin:/bin, no user profile sourced, no HOME in some versions. A script that calls docker or terraform without a full PATH will fail silently. Fix it by either sourcing the profile at the top (source /etc/profile) or using absolute paths. Similarly, systemd unit files do not inherit the interactive user's environment; set Environment= or EnvironmentFile= directives in the [Service] block explicitly.

Sensitive Variables and Security Hygiene

Environment variables are visible to anyone who can read /proc/<pid>/environ, which by default is restricted to the process owner and root. However, they can leak through:

Verbose logging (e.g., set -x in shell scripts, debug middleware in web frameworks)
Error pages that dump the environment
ps aux output if the secret was passed as a command-line argument rather than as an env var
Docker inspect output and Kubernetes pod descriptions (for env vars, not secrets mounted as files)

For anything truly sensitive (passwords, API tokens, private keys), prefer secrets management systems — HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets backed by an encrypted secrets store — and inject them as files into /run/secrets/, not as environment variables, where possible. At a minimum, never commit .env files to version control.