Command arguments

A reference for the options you pass to law run. The common ones are defined on FLAF's base task classes (FLAF/run_tools/law_customizations.py), so they work on every FLAF task. LAW also provides built-in options for status and cleanup.

Underscores become dashes on the command line

A parameter named transfer_logs in the code is --transfer-logs on the CLI; anaTuple_version is --anaTuple-version, and so on.

Boolean options take an explicit value

LAW makes boolean options explicit: --bundle on its own means True, and an option that is on by default is switched off with False, e.g. --transfer-logs False.

Common task options

Option	Default	Meaning
--version	(required)	Label that namespaces this run's outputs. Different versions never collide.
--era	(required)	The era, e.g. Run3_2022, or several separated by commas (Run3_2022,Run3_2022EE): one run over all of them, with one workflow per stage; see Running several eras and below. A task that does not support several eras refuses them with <Task> takes one era. Or a meta-era of era_groups:, given alone (Run3_22_23): HistMergerTask sums its eras and HistPlotTask plots the sum, every other stage runs over its eras; see Meta-eras.
--workflow	htcondor	local (this machine), htcondor (CERN batch), or crab (WLCG). Without the option LAW picks the first workflow a task declares, which for FLAF tasks is HTCondor — so always pass it. InputFileTask and AnaTupleFileListTask always run locally. See HTCondor and CRAB.
--branches	(all)	Which branches of the launched task to run, e.g. 0, 0,2, 5:8 (a start:end range excludes its end, so this is 5, 6, 7). Upstream tasks are then asked only for the branches these need — once their branch maps exist; on a from-scratch run the upstream stages that define the branch map are required in full. The selection is kept also when it is given before the task's branch map exists. Over several eras the numbers carry the era, see below.
--test	-1	AnaTuple production only: keep just the first input file of every dataset and process at most N events of it (-1 = everything). See the warning below.
--process	""	Restrict to some processes: a comma-separated list of names or ^regex patterns (e.g. custom_CI_Signal). A meta-process name expands to its members.
--dataset	""	Restrict to some datasets: a comma-separated list of names or ^regex patterns. Over several eras the selection has to match something in every era of the run.
--model	""	Override the physics model for this run.
--customisations	""	Ad-hoc key=value;key=value overrides of existing config entries (see below).
--user-custom	""	Path to an extra user_custom-style YAML, loaded last (see below).

--test writes to the same paths as a full run

Nothing in the output paths records --test, so a truncated test output looks complete to a later full run of the same --version, which then reuses it. Give test runs their own --version. (Test jobs are also not used to calibrate the AnaTuple job-cost model.)

Several eras: --era and --branches

A run over several eras is one workflow per stage. Over one era --branches takes the numbers of that era's own branch map, as always. Over several eras a branch is numbered era id × 1 000 000 + its number in that era's own map (Run3_2022 is 5 … Run3_2026 is 11; the table and python -m FLAF.Common.Setup --era-branch <era> <number> are in Eras):

# every stage up to HistMergerTask, for both eras
law run FLAF.Analysis.tasks.HistMergerTask --version v1 --era Run3_2022,Run3_2022EE --workflow htcondor
# branches 0 and 1 of Run3_2022 and branch 0 of Run3_2022EE
law run FLAF.Analysis.tasks.HistMergerTask --version v1 --era Run3_2022,Run3_2022EE \
  --workflow htcondor --branches 5000000,5000001,6000000

• Every era of --era needs at least one number in --branches; a range has a start and an end and stays inside one era. Once an era's branch map exists, at least one of its numbers must be a branch of that map: a run over one era would take a selection naming none of its branches as the whole map, which a run over several eras refuses.
• --branch is refused over several eras for the stages with era branches: use --era <era> --branch <its number in that era>. A StatInference card workflow takes --branch <card number> over several eras.
• A run over one era refuses the numbers of a run over several eras (1 000 000 and above): with the decoded form when the number belongs to a configured era, as beyond the numbers of one era otherwise. See Troubleshooting.
• --parallel-jobs is one cap for the whole run, all eras together (HTCondor, CRAB); --tasks-per-job, --max-runtime, --n-cpus and --crab-memory apply to the jobs of every era as they would to a run over that era alone.

HTCondor options (on every workflow task)

Option	Default	Meaning
--transfer-logs	on	Keep each job's log (HTCondor and CRAB). With a remote fs_default the log is uploaded to <version>/logs/<Task>/<era>/ there (plus the producer name for the analysis-cache tasks); with a local fs_default the log is part of the job's output sandbox, which HTCondor copies back to the task's local directory under data/ only with --htcondor-spool False (with the default -spool it stays on the schedd until condor_transfer_data is run, which law does not do). On CRAB the file is stdall_<first branch>_crab<tag>.<job number>.txt (see CRAB). Switch off with --transfer-logs False.
--parallel-jobs	unbounded (HTCondor) / 2000 (AnaTupleFileTask on HTCondor, anaTuple_scheduling.parallel_jobs) / 5000 (CRAB, crab.parallel_jobs)	Cap concurrent jobs. On CRAB this is also the max size of each CRAB task. Over several eras it is one cap for the whole run, all eras together. The bare form, and the launched task's prefixed form, are copied to every task it requires; --<Task>-parallel-jobs for another task caps that task alone.
--tasks-per-job	1 (10 for HistTupleProducerTask)	Branches per job. Applies to the launched task only; set it for an upstream task with --<Task>-tasks-per-job. On AnaTupleFileTask, jobs are normally composed by estimated cost instead; passing this option explicitly restores fixed-size chunking.
--max-runtime	(task default)	Per-job wall-clock limit in hours: 12 unless the task sets its own (e.g. 40 for AnaTupleFileTask, 48 for AnaTupleMergeTask). On HTCondor, a resubmitted AnaTupleFileTask job gets a longer limit, unless --tasks-per-job is given.
--n-cpus	1 (4 for AnaTupleFileTask, AnaTupleCostProbeTask and HistTupleProducerTask; 2 for AnaTupleMergeTask, HistFromNtupleProducerTask and HistMergerTask)	CPUs requested per job. AnalysisCacheTask always takes n_cpus and max_runtime (and, on CRAB, crab_memory) from its producer's entry in payload_producers.
--priority	0	Job priority among your HTCondor jobs, from -20 to 20.
--bundle	off	Ship a code/environment tarball to the worker. See HTCondor → bundles. Always on for --workflow crab.
--htcondor-spool	on	Pass -spool to condor_submit, so the input files (including the proxy) are sent to the schedd instead of being read from a shared filesystem. Switch off with --htcondor-spool False.

The per-task defaults of --n-cpus and --max-runtime apply to the task you launch. They are not handed from a requiring task to the tasks it requires: each dependency keeps its own default, or the value given for it with --<Task>-n-cpus / --<Task>-max-runtime — unless the requiring task pins it, as most FLAF production tasks do for what they require (they pass the dependency's own default explicitly, which wins over the prefixed option).

Code that derives a task of its own family (X.req(task) with task an X) has to say which of the two it means, or FLAF's Task.req refuses the call: _skip_task_excludes=True for the same task, which keeps the values given to task (--tasks-per-job, --max-runtime, --n-cpus, --crab-memory; law's req_branch, req_workflow, as_branch and as_workflow pass it already), _skip_task_excludes=False for a separate task that takes its own defaults.

CRAB options (on every workflow task)

Option	Default	Meaning
--workflow crab	—	Submit branches via CMS CRAB (WLCG). See CRAB.
--crab-memory	0 (= the most CRAB grants for the job's cores)	CRAB maxMemoryMB per job, in MB. A per-task value, like --n-cpus: set it for one task with --<Task>-crab-memory <MB>, or as crab_memory in a [luigi_<Task>] section of law.cfg; for AnalysisCacheTask as payload_producers.<producer>.crab_memory, which overrides the command line. A request above what the cores allow buys cores; one that no core count can hold (above 20000 MB), or below 1000, is refused at submission. See CRAB → Resources.

Optional site white/black lists go in global.yaml under crab: (not CLI flags). Unset whitelist ⇒ all T1/T2/T3 sites; blacklisted (or auto-quarantined) sites are cut out of the whitelist itself — see CRAB. Default --parallel-jobs on CRAB is 5000 (crab.parallel_jobs); jobs are aggregated into CRAB tasks of at least crab.refill_fraction * parallel_jobs jobs while such a wave is still achievable (the tail is released immediately, and a retry waits at most crab.retry_release_minutes). Site.storageSite / Data.outLFNDirBase are a fixed placeholder, independent of fs_default (see CRAB). Cores are --n-cpus rounded up to 1, 2, 4 or 8 (--n-cpus above 8 is refused); memory defaults to max(3000, 2500 * cores) MB (3000, 5000, 10000 and 20000 MB for 1, 2, 4 and 8 cores), the most CRAB grants, because maxMemoryMB is a kill threshold. crab.memory_mb_per_cpu is retired and raises an error if present. --transfer-logs uses FLAF's own log upload to fs_default; CRAB's log transfer stays off.

--parallel-jobs and --poll-interval for one task

--<Task>-parallel-jobs and --<Task>-poll-interval reach the task they name, on CRAB and HTCondor alike, and are not overridden by the CRAB defaults or crab.* values. The bare form — and, the same for luigi, the prefixed form of the task that is launched — is copied to every task of the graph.

Status & cleanup (LAW built-ins)

Option	Meaning
--print-status N,K	Show the dependency tree status to task depth N, target-collection depth K. Also prints output paths. --print-status 3,1 is a good default.
--print-deps N	Print the dependency tree to depth N without checking outputs.
--remove-output N,a,y	Remove outputs to task depth N (0 = only the launched task). The mode a removes everything without asking (i asks per task, d is a dry run); the final y runs the task afterwards, so the outputs are recomputed (leave it out to only remove). Deletes real files — check the version first.

--customisations

Pass quick overrides of entries of the merged global configuration as a semicolon-separated list, quoted so that the shell does not split it:

--customisations "key1=value1;key2=value2"

• Each item is exactly one key=value. Commas stay inside the value (channels=eTau,muTau), so key1=a,key2=b is read as one item and fails with len of substring is not 2!.
• A dotted key (section.key=value) addresses a nested entry.
• The key must already exist in the configuration, and the value is converted to the type of the value it replaces. Booleans are handled inconsistently: the configuration converts them with bool(), so "False" becomes True, while a few task-side checks compare the raw text with "True". A list would be split into single characters. For booleans, lists and new keys use --user-custom.
• A customised run writes to the same output paths as a run without it: use a new --version.

AnaTuple production sees only three keys

AnaTupleFileTask does not pass the string on to anaTupleProducer.py: of its keys, only channels, store_noncentral and compute_unc_variations reach AnaTuple production (as options of the producer). Any other key changes only the configuration seen by the law tasks themselves and by the histTuple and histogram-filling scripts, which receive the string. For a setting the AnaTuple producer reads — such as HH→bb̄ττ's deepTauVersion — use --user-custom, which is forwarded to it.

HH→bb̄ττ: restrict the channels

HH_bbtautau declares a top-level channels key, so --customisations "channels=eTau,muTau" limits every stage that takes a channel list (AnaTuple, analysis-cache, histTuple, histogram and plot tasks) to those channels. The other analyses do not declare it, and there the same option fails with Key "channels" not found in global configuration.

--user-custom: per-run config overlay

--user-custom <path> appends an extra YAML after every other configuration file — including config/user_custom.yaml and config/<era>/global.yaml — so its values win (see How values combine). A missing file is an error. Use an absolute path or one relative to $ANALYSIS_PATH. The file is also passed to the producer scripts and shipped with remote jobs. It is the cleanest way to change settings for a single run without touching your (git-ignored) config/user_custom.yaml:

law run FLAF.Analysis.tasks.HistPlotTask \
  --version test --era Run3_2022 --workflow local --branches 0 --test 1000 \
  --user-custom config/user_custom_test.yaml

See user_custom.yaml.

Per-task version overrides

Every task carries its own --version, so you can make one run read an existing upstream production while writing its downstream outputs under a new version. Override an upstream task's version with --<TaskClassName>-version:

law run FLAF.Analysis.tasks.HistTupleProducerTask \
  --version my_dev \
  --AnaTupleMergeTask-version v2605 \
  --AnaTupleFileListTask-version v2605 \
  --era Run3_2022EE --workflow local

Here the anaTuples are reused from the central v2605 production (provided your fs_anaTuple points at the central storage, see the warning below), while the histTuples are written under my_dev. This is the key to fast, parallel development: many people can share one upstream production without recomputing it.

First use of a production in a checkout: its merge plans are copied into data/

The branch maps of AnaTupleMergeTask and of every analysis task after it are built from a copy of the production's merge plans in your checkout's data area, $ANALYSIS_DATA_PATH/<anaTuple version>/AnaTupleFileListTask/<era>/ (see AnaTupleFileListTask). A checkout that has not used the production yet has no such copy. If the production's own plans are all on fs_anaTuple (its AnaTupleFileListBuilderTask is complete), the first command that builds one of these tasks copies them there, --print-status and --print-deps included: it prints Copying <N> AnaTuple merge plans of version <v> .... The plans are copied 16 at a time; the time depends on the storage and its protocol: for the 315 plans of an HH_bbWW era read from FNAL on lxplus it took about 70 s over davs (the T3_US_FNALLPC:/store/... form) and about 110 s over xrootd (root://cmseos.fnal.gov//...). Later commands read the copy. Batch jobs never copy: they find the plans in the AnaTupleFileList bundle or in the shared data area of the submission.

If the plans are not all on fs_anaTuple (for example because it does not point at the production's storage), nothing is copied and the run requires the whole upstream chain, as for a new production. Until the plans exist, --print-status cannot judge the upstream tasks (it shows TargetCollection(len=1), law's placeholder), and --branches does not narrow what is required upstream.

Shortcuts for the whole upstream

Listing every --<Task>-version is tedious. The base task exposes three shortcuts that set the version of all upstream tasks at once:

Flag	Sets the version of
--anaTuple-version <v>	every AnaTuple/AnaProd task (InputFileTask, AnaTupleFileList*, AnaTupleMergeTask, …)
--anaCache-version <v>	AnalysisCacheTask and AnalysisCacheAggregationTask
--ana-version <v>	both of the above — a single flag for the entire upstream production

So the multi-flag example above (and any deeper fork) collapses to one flag:

law run FLAF.Analysis.tasks.HistTupleProducerTask \
  --version my_dev --ana-version v2605 \
  --era Run3_2022EE --workflow local

Use --anaTuple-version / --anaCache-version when you want to fork only one of the two upstream stages.

A version override does not change where the outputs are looked up

These flags replace only the version label. The upstream outputs are still looked up on your fs_anaTuple / fs_anaCacheTuple (each falling back to fs_default). To reuse a central production, point those at the central storage as well (for example in a --user-custom overlay); otherwise LAW finds nothing there and re-runs the upstream chain under the central label in your own area.

--<AnyTaskInTree>-<param> works for parameters the requiring task does not pass on

LAW lets you set a parameter of any task in the dependency tree by prefixing it with the task's class name, e.g. --HistFromNtupleProducerTask-n-files-per-job 10. A parameter that the requiring task also declares (--test, --workflow, …), or sets explicitly, is handed down by it and wins over the prefixed value — except for the ones FLAF prefers from the command line: version, the three version shortcuts, tasks_per_job, parallel_jobs and poll_interval. The per-task resources --n-cpus, --max-runtime and --crab-memory are never handed down implicitly, so their prefixed form reaches its task unless the requiring task pins the value explicitly (FLAF's production tasks pin --n-cpus and --max-runtime of what they require to its defaults; --crab-memory is never pinned).