Skip to content

Global & user configuration

Deployment settings — where products are written, and the CRAB compute knobs — are kept out of the production setups, so a setup is backend-agnostic and not tied to one user. They follow the same pattern as FLAF: a committed config/global.yaml with the defaults, merged with a config/user_custom.yaml holding your user-specific values.

config/user_custom.yaml is layered on top of config/global.yaml: scalars and lists override, nested maps are deep-merged (see dsprod/config.py).

user_custom.yaml is not in the repository

It is user-dependent, so it is git-ignored and you create it yourself after cloning (see the example below). global.yaml is committed and holds only settings that are the same for everyone.

Create your config/user_custom.yaml

Optional: global.yaml already points fs_default at the shared production area at FNAL, so a fresh checkout writes there. Create user_custom.yaml when you want your own space instead — always do this for tests, so you do not write into the production tree:

# config/user_custom.yaml
fs_default: davs://eoshome-k.cern.ch:8444/eos/user/k/kandroso/DSProd/

Replace the host and path with your own EOS area. You can override any global.yaml value in the same file, e.g. the CRAB processing site:

fs_default: davs://eoshome-k.cern.ch:8444/eos/user/k/kandroso/DSProd/

crab:
  whitelist: [ T2_CH_CERN ]
  max_cores: 1

fs_default

fs_default uses the FLAF path notation: one URI carrying protocol, host and base path. All product paths are relative to it, so the storage area lives in exactly one place.

fs_default: davs://eoshome-k.cern.ch:8444/eos/user/k/kandroso/DSProd/   # remote (WLCG/gfal)
fs_default: /eos/user/k/kandroso/DSProd/                                # local file system

A value starting with / selects a local file system; anything else is treated as a remote (WLCG) one, accessed through the gfal-CLI interface.

One file system for every backend

fs_default is used by all backends. A production submitted with --workflow crab writes exactly where the same setup would write with --workflow local or htcondor — there is no separate CRAB output location. (Later, more granular fs_* keys can be added, as in FLAF.)

A production writes to <fs_default>/<output>, where output is named by the production setup; --test writes to <output>_test instead. The layout inside it is described in Architecture.

fs_watchdog is an optional second file system holding the CRAB watchdog's heartbeat flags; when it is unset they go to fs_default like everything else. Both take the same FLAF-style URI, and a plain /... path selects a local file system.

config/global.yaml (committed defaults)

crab:
  max_cores: 4
  # memory_mb: 0                    # optional; default per-job request in MB (0 = CRAB's own max)
  # mb_per_core: 2500               # optional; CRAB's own limits, override only if CMS changes them
  # mb_single_core: 3000
  # whitelist: [ T2_CH_CERN, ... ]  # optional; unset (default) = every tier T1_*/T2_*/T3_*
  # blacklist: [ ... ]              # optional; exclude misbehaving sites
  # ignore_global_blacklist: true   # optional; waive CMS's known-broken-site list (not recommended)
  # parallel_jobs: 5000             # optional; jobs per CRAB task / in flight
  # refill_fraction: 0.2            # optional; min wave size, as a fraction of parallel_jobs
  # retry_release_minutes: 45       # optional; release a parked retry after this long
  # auto_blacklist: { ... }         # optional; see Backends -> Failing sites (on by default)

Cores and memory are asked for separately

Each task declares its own n_cpus and memory (in MB) next to its max_runtime; RunProd asks for 4 cores and 10000 MB, NanoMergeTask for 1 CPU and 5000 MB. Those are DSProd's defaults — a production setup sets its own in a resources: block, since what a job needs belongs to the production rather than to the framework. --<task>-crab-memory overrides the memory for one run. A memory of 0 falls back to crab.memory_mb, and then to CRAB's own max(3000, 2500 * numCores); that is the framework's marker for "work it out", which a setup expresses by naming no memory at all rather than by writing 0.

CRAB sells memory only in per-core units, so max_cores bounds both

CRAB accepts only 1, 2, 4 or 8 cores and refuses any task above max(3000, 2500 × numCores) MB. So a memory request larger than the task's cores can hold raises numCores for the CRAB submission (HTCondor still gets n_cpus) — and if max_cores cannot hold it, the task refuses to submit rather than quietly asking for less. That is deliberate: on CRAB the number is a kill threshold (a job over it is removed with exit 50660 and never retried), so a silently shrunk request is a silently dead branch.

max_cores still caps each task's n_cpus as well, so a value below a task's own n_cpus makes it run with fewer threads on CRAB than on HTCondor.

The crab: block holds compute settings only — CRAB never stages out (DSProd owns all I/O), so there is no CRAB output location to configure. These can also be overridden per run on the command line (e.g. --crab-whitelist, --crab-memory).

Every site by default. DSProd jobs have no real input dataset, so they can run at any CMS processing site. The CRAB client insists on a Site.whitelist whenever ignoreLocality is set, so an unset one becomes T1_*, T2_*, T3_* — the widest pool it accepts; configuring one can only narrow it. See Backends for when restricting is worthwhile, and Job waves for parallel_jobs/refill_fraction/retry_release_minutes.

On grid workers

Being git-ignored does not mean being absent from jobs: the CRAB code tarball ships the whole config/ directory, so your user_custom.yaml travels with the job and grid workers resolve fs_default exactly as your local runs do.

law.cfg

config/law.cfg holds only law/luigi framework settings (scheduler, job dir, modules). Storage and CRAB settings are not there — they are in global.yaml / user_custom.yaml.