Tasks & LAW

DSProd is built on LAW (the Luigi Analysis Workflow). You declare what output you want by running a task; LAW resolves the dependency graph and runs only the steps that are not already done. Every production task takes the same --setup <path> parameter (the production setup YAML) and a --workflow backend.

The tasks live in dsprod/tasks.py.

Task reference

InstallCMSSW

Installs, once on the shared AFS area, the CMSSW releases an era's production needs (all production steps plus every requested NanoAOD version). It is a local workflow with one branch per selected era, and is idempotent — env.sh guards each release with a .installed flag. The batch tasks require it so the releases exist before jobs run.

ImportGridpack and MakeGridpack

Both provide the same product — <output>/gridpacks/<gridpack-name>/gridpack.tar.xz, relative to fs_default — and both branch over distinct gridpacks, not points, since several points can share one (the 2B2JLNu, 2B2L2Nu and 2B2Tau final states of a mass point use the same gridpack, and one branch per point would make them race on the same output). Which of the two does the work depends only on where the gridpack already is:

Two setups do not share the copy, even for the same gridpack: the product path above starts at the setup's output:, so XHHbbWW/gridpacks/... and XHHbbtautau/gridpacks/... are different targets and each production imports its own. What is shared is the content and the store it comes from — for the 80 gridpacks of the X_HH grid that is 2.3 GB written once per setup.

state	what happens
already on fs_default	nothing — both tasks are complete
in the DSProdGridpacks store	ImportGridpack copies it to fs_default
nowhere	MakeGridpack generates it, and stages the result to fs_default

The setup never names a gridpack: its location in the store is derived from the process's gridpack_rel_path, which mirrors the model's own layout level for level (<process>/<generator>/<comEnergy>/<production_mode>/<gridpack-name>/), and presence there alone decides import vs. generate. The gridpack name carries the production mode (GluGlutoRadiontoHH_M-800), because a production stores its gridpacks flat under <output>/gridpacks/.

ImportGridpack is always local, and MakeGridpack requires it, so the import has happened by the time MakeGridpack decides what to submit. That matters: reading the store needs the git checkout and the Git-LFS server, which a grid worker has neither of — without this split, a batch job would regenerate a gridpack the repository already holds. Under the sparse checkout the tarball is not even in the working tree, so ImportGridpack streams the object from the LFS server straight to fs_default and verifies it against the pointer's size and sha256.

MakeGridpack renders the point's cards and runs genproductions_scripts/bin/<generator>/gridpack_generation.sh. It is the only one of the two that is worth a batch backend (--workflow htcondor|crab).

Clean environment for generation

gridpack_generation.sh sets up its own CMSSW and aborts if one is already active. DSProd strips the CMSSW_*/SCRAM/PYTHON* variables from the generation subprocess, so gridpack generation works even on a CRAB worker (where env.sh sets up a cvmfs CMSSW for LAW itself).

Never generated inside a production job

RunProd requires MakeGridpack, so a worker that finds the gridpack missing — often only because it cannot reach fs_default — would schedule it and spend the production slot on MadGraph, about 1.5 h, before failing to upload the result from that same worker.

A MakeGridpack branch running on a batch node therefore checks what was submitted: if this job was launched for MakeGridpack itself, it generates normally — producing gridpacks on the grid is a supported thing to do, law run MakeGridpack --workflow htcondor|crab. If the job was launched for something else, it refuses and says so at once. That error means either the gridpack really is missing, in which case submit MakeGridpack for it, or fs_default is unreachable from the worker.

PremixFileList

Resolves an era's premix pileup dataset to a plain file list on fs_default (<output>/premix/<era>.txt), once, and RunProd passes it to cmsDriver as --pileup_input filelist:....

Without it, --pileup_input dbs:<dataset> makes every job resolve that dataset — ~38 000 files — with its own DAS query. At a few thousand concurrent jobs the queries start returning nothing, cmsDriver then writes a config with no secondary input, and cmsRun dies on

NoSecondaryFiles: RootEmbeddedFileSequence no input files specified for secondary input source

after the job has already produced its GEN-SIM. A 5000-job production lost 10 % of its jobs that way. The list is identical to what a successful DAS query returns, so nothing about the physics changes; it is stored in the production area (not <output>_test) because it depends only on the era, and a batch node refuses to build one — that query belongs on the submitting machine.

Seeing premix/ while listing the production directory only means that directory exists. It is not a listing of the files inside it: a job that has just looked for its own, not yet written, nano record must still find <era>.txt there. Treating every name that listing did not mention as absent made every CRAB job try to rebuild the list and die on the guard above.

RunProd

The core production task: a fused GEN→…→MiniAOD→NanoAOD chain for one (era, point, seed), run via cmsDriver steps (dsprod/run_step.py). Branches are enumerated by runprod_branches(eras, points) — the single source of truth for branch numbering, shared with NanoMergeTask. For each requested NanoAOD version it stages one file and records that it did:

<output>/staging/nanoAOD_<version>/<era>/<point>/nano_<version>_<seed>.root
<output>/produced/nanoAOD_<version>/<era>/<point>/nano_<version>_<seed>.json

The record, not the staged file, is what RunProd declares as its output. NanoMergeTask deletes each staged file once it has merged it, so the file's presence cannot say whether a seed ran: a resumed workflow would find the outputs of every already-merged seed missing and regenerate the whole era (law marks such jobs initially missing task outputs). Nothing deletes the records, so completeness survives the merge. To redo a seed deliberately, delete its record along with its nano file.

RunProd carries its own failure budget (retries: 9, tolerance: 0.05, acceptance: 1.0) rather than law's, so one dead branch cannot end a multi-day production while a short sample still fails the workflow.

The failure budget

A production is not allowed to die because one of its branches did.

law counts attempts per job and failed jobs per workflow, and its defaults are made for short workflows: retries: 5 and tolerance: 0.0, i.e. the first branch to run out of attempts raises tolerance exceeded and ends the run — 4799 finished jobs and all. RunProd therefore sets its own budget:

	value	meaning
retries	9	10 attempts per branch: law submits a job once, then resubmits it retries times
tolerance	0.05	up to 5 % of the branches may run out of attempts without ending the run
acceptance	1.0	law's default, kept: every branch must finish for the workflow to succeed

Ten attempts is what it takes to survive a run of broken sites, which is how a budget is really spent: of the 301 branches of the 2026-09 Run3_2023BPix production that failed more than once, 295 failed at two or more different sites, ~30 % of all attempts failed, and three sites failed 93–98 % of everything sent to them. A budget of four made it a coin flip whether a healthy branch escaped them before running out — 34 of 16 000 branches were written off with nothing wrong with any of them. A larger budget costs a successful branch nothing; only a branch that is genuinely dead pays, in wall clock nobody is waiting for, since tolerance keeps the production running past it and the quarantine holds the site that spent the budget out for longer each time. (law offers a branch that has just spent its last attempt to the submission step once more and then ignores whatever that job reports, so the number of CRAB jobs a hopeless branch produces is one higher than its budget. Only the budgeted attempts decide when it counts as failed.)

tolerance is a fraction of the branches, as law reads any value at or below 1, so a production of fewer than 20 branches — a single point, or a --test run — has a budget below one job and still ends on its first dead branch. law reads a value above 1 as an absolute number of jobs instead, so --RunProd-tolerance 2 is how a small production gets the same slack.

It is acceptance, not tolerance, that forbids a silently short sample: the run carries on past a failed branch, but once nothing is left to finish, law reports

acceptance of 4800 not reached, total jobs: 4800, failed jobs: 7

and the workflow fails. law keeps its retry counts in memory only, so running the same command again hands each failed branch a fresh budget and resubmits it after one polling iteration. In other words: tolerance decides how much of a production may fail before law stops driving the rest, and a rerun is what finishes it.

That last point cuts both ways: a branch that is broken for good — a bad gridpack, a seed that crashes the generator — makes every run end the same way, and every rerun hands it a fresh budget, so repeated reruns retry it forever. It shows up as runs of the same length failing on the same branch numbers; the fix is to fix or drop the branch, not to raise the budget.

Both are ordinary law parameters, so a run can override them — --RunProd-retries 3, --RunProd-tolerance 0.1. Address RunProd by name: like --max-runtime and --n-cpus, neither is passed on to or taken from the tasks around it, which is what keeps NanoMergeTask from handing its RunProd requirement law's tolerance: 0.0 back.

Every step asserts that its output holds exactly the -n events it was asked for, counted in the release that wrote it. -n is only a request: a step that returns fewer produces a valid file, each later step carries the shortfall forward, and the merge only ever compared a group with the sum of its own inputs — so of the 166 merged Run3_2023 v12 files, 40 were delivered holding 49 998 or 49 999 events while advertising 50 000. Checking per step is what says where the events went; entering the release to count costs seconds against a step measured in hours. A generator that filtered events would legitimately return fewer, and no DSProd process does that today.

Those 40 files were a generator defect rather than a framework one, which is also what the assertion is for: a fragment that lets Pythia sample an off-shell W/Z below the mass at which it can be decayed loses the events it cannot decay — about 5 in 10^6, so one job in two hundred — and an LHE record, readable only once, cannot be replaced, so the job ends one event short. The X_HH fragments keep mMin above that floor, so the count a job is asked for is one it can deliver.

A shortfall like that is not reproducible per seed, which is why a failing branch is worth retrying: the seed fixes the LHE file, but the resonance masses are drawn by Pythia, and with concurrent hadronizer streams the draw a given record sees depends on which stream took it and what that stream did before. So one attempt in a hundred failing is bad luck and the retry succeeds; a branch that comes up short on every attempt is a generator setting to fix.

RunProd requires the VOMS proxy, InstallCMSSW (for its era), and MakeGridpack (for its point). Its steps run cmsDriver with --nThreads <n_cpus> (4 by default, and a setup's resources: may set another), so the job's core allocation is what cmsRun actually uses; a nThreads in the conditions overrides it per step. The proxy requirement is satisfied by the batch-delegated proxy inside a job (see Grid proxy). The number of seeds per point and era follows from events_total[era] / events_per_job — events_total is per era, so one setup covers all of them.

Never generated inside a merge job

NanoMergeTask requires single RunProd branches, so a merge branch whose seed has no record would make luigi run this 7 h chain inside the merge job — a 3 h slot with one core, which it loses on walltime. RunProd therefore makes the same check MakeGridpack does: on a batch node, a job launched for anything other than RunProd refuses to generate and names the seed it was asked for. Either that seed really is missing — produce it first — or fs_default is unreachable from the worker, which is the other way a record looks missing.

NanoMergeTask

Merges a group of per-seed nanos (files_per_merge per group) into one output with haddnano, verifies that the merged event count equals the sum of the inputs, and — only then — removes the staged per-seed inputs. Those counts are taken in one invocation of the nano release for the whole group: entering it (scram runtime, in a container when the worker OS differs) costs far more than the counting, so a 50-input group counted file by file spent about 10 minutes of its 3 h slot before anything had been verified. The counts come back in a JSON file keyed by path — stdout is where ROOT prints its own warnings, so nothing may be read off it by position.

Before merging, the group's job size is checked against the events_requested field of the seeds' produced/ records, so a group mixing seeds produced at different sizes is refused rather than delivered as a file of an unadvertised size; afterwards the merged file is checked against the size those seeds were produced for, which — since every step delivers its -n exactly — can only fail if the merge itself lost events. It takes the input paths from the grouping itself, not from what RunProd declares (that is the record), and refuses to run if a staged file of its group is gone although its seed is recorded as produced. Output is the final, FLAF-facing file:

<output>/nanoAOD_<version>/<era>/<point>/nano_<version>_<group>.root

A group waits for its own seeds only, not for the generation stage as a whole: the workflow requires exactly the RunProd branches of the groups it is running, taken from its branch map after --branches has been applied. A group whose 50 seeds are on storage can therefore be merged while the other 4750 of the era are still being produced. The requirement used to be the entire RunProd workflow, and that is how the Run3_2023BPix production reached 169 of its 192 groups complete with not one merged.

Three consequences to know about:

• Asking for the whole merge (no --branches) requires every RunProd branch, which law collapses back to "all branches" — so a production driven through NanoMergeTask polls the same job data as law run RunProd itself, data/RunProd/<store>/<backend>_jobs_0To<n>.json, and merges only once the last seed of the selection is done. A narrowed merge run instead gets its own job-data file, named after the branch ranges it requires (crab_jobs_0To51_100To251.json). It lands in the same data/RunProd/<store>/ directory only when it is given the same --eras / --points / --test as the driver: store_parts() appends a slug and hash of those, so a merge run narrowed by --eras while the production is driven without it keeps a separate set of job ids for the same seeds. Either way, two processes that can submit RunProd must not run in one area — they would submit the same seeds twice under two sets of job ids. The product paths themselves do not depend on any selection.
• --branches on the merge selects merge groups and asks for the seeds behind them. It used to hand the merge's own branch numbers to RunProd as if they were seeds, so --branches 5 waited on RunProd branch 5 rather than on the 50 seeds of group 5.
• A seed selection stops at RunProd. Its own requirements branch over gridpacks and eras, so law copying branches one level further meant --branches 10:20 asked MakeGridpack for gridpacks 10–19 — while seed 10 needs gridpack 0, whose absence the requirement then never noticed — and dropped the premix list of every era outside the range.

ProductionTask

One command for a whole production: it produces, repairs what the storage lost, and merges.

law run ProductionTask --setup <setup> --eras Run3_2022EE --points "<pattern>"

It yields RunProd, then runs the PruneProducedRecords check, then yields NanoMergeTask — the three steps an operator otherwise runs by hand, in the order that makes the middle one worth having. A produced/ record whose staged nano file has been lost is invisible until the merge of its group runs, and then that group fails on every attempt; checking before the merge turns each one into a seed that is simply produced again. If the check deletes anything, the pruned seeds go back through RunProd (only those, since only those are now incomplete) before the merge is submitted.

--backend (crab by default, also htcondor or local) reaches both stages; the usual --eras, --points, --nano-versions and --test narrow the run as everywhere else. Completeness is the merged files themselves, so a finished production is a no-op and an interrupted one resumes.

The repair here runs in delete mode, unlike the standalone task, which reports by default — there is no operator to read a report mid-run. --max-repairs (2) bounds it: each round produces the pruned seeds again, so a storage that keeps losing them would otherwise be answered with submission after submission, and the run stops with a message naming the count instead. --max-repairs 0 skips the repair entirely. The count lives on the driver task, which luigi keeps as one instance only while it runs the task in the same process — so while a repair is allowed, --workers above one and --worker-timeout are refused with a message rather than left to degrade the bound silently. On crab and htcondor that costs nothing, since the batch system is the parallelism. On --backend local it does: there --workers is what runs branches in parallel (backends), so a large local production wants --max-repairs 0 --workers n and a law run PruneProducedRecords of its own before and after.

When the bound does stop a run, the records that sweep deleted are already gone, so those seeds are simply unproduced: fix the storage and run the command again. And because the sweep follows the production stage immediately, a staged file that is written but not yet visible — remote writes can lag by seconds — would be read as lost and its seed produced once more. RunProd writes the staged file before its record, which makes that unlikely, and the cost if it happens is one job.

Why the repair runs before the merge, not after a failure

luigi never re-runs a task whose dynamic dependency failed, so there is no point at which "merge, and prune if it failed" could do the pruning — the generator simply never regains control. A file lost after this has swept therefore still fails its merge branch; running the same command again repairs it and finishes the production. The driver prints the payload's own error for a failed job, so a merge that fails for some other reason says so directly.

BackfillProducedRecords

A migration for productions that ran before the produced/ records existed, whose staged files have already been merged away: it reconstructs the records from what is on storage — a merged file accounts for the whole group of seeds behind it, a surviving staged file for its own seed — and leaves existing records alone, so it is safe to re-run (delete its backfill.done flag to make it list storage again).

law run BackfillProducedRecords --setup <setup> --eras Run3_2023 --workflow local --workers 8

One branch per (era, point, nano version). Each branch reads the storage with three directory listings rather than a stat per seed, and uploads its records --upload-threads at a time (16 by default): a full era is 16000-28000 seeds per nano version, and at one remote round trip per seed the migration takes hours instead of minutes. Raise --workers and --upload-threads on a slow endpoint — the work is all latency, not CPU.

PruneProducedRecords

The repair for the opposite drift: a produced/ record whose staged nano file is gone and unaccounted for. A record is what the merge trusts — it is why a merge may delete the files it consumed without the next run reading a finished era as unproduced — so a staged file that disappears anyway (storage lost it, or a hand removed it) leaves a record no one can honour, and the only symptom is a merge group that fails on every attempt with N of 50 staged nano files of this merge group are gone. Three groups of the Run3_2022EE production sat like that on 2026-09-15, each missing exactly one file of fifty.

law run PruneProducedRecords --setup <setup> --eras Run3_2022EE
law run PruneProducedRecords --setup <setup> --eras Run3_2022EE --prune

Without --prune it only reports, because the cost of a wrong deletion is a seed produced again from scratch; with it, the stale records are deleted and those seeds are produced again on the next RunProd. A record is stale only when nothing accounts for its seed:

• a merged file covering the seed's group accounts for it — the normal state after a merge, when every record of the group has no staged file and all of them are healthy;
• a staged nano file accounts for its own seed.

Two rails, because the blast radius of a mistake here is an era. A directory listing that fails stops the prune instead of reading as "nothing is staged", which would condemn every record of the point — listdir() and not exists(), because exists() answers an unknown file from a cached listing of its directory and so can answer from evidence older than the question (a failed listing itself is no longer a "no" anywhere: gfal_ls_checked raises). Absence is established instead from the first successful listing of an ancestor that does not carry the branch below it, climbing when it has to: before a production's first merge there is no merged tree at all, so the point's directory and the era's above it are both missing and the answer comes from higher up. And a point whose stale share exceeds --max-stale-fraction (0.5) is reported as a storage or configuration fault rather than pruned.

The task never reports itself complete — records can go stale again tomorrow — and it is a plain task rather than a workflow for that reason. A law local workflow yields its branches as dynamic dependencies and luigi re-runs the workflow once they finish, re-checking each branch, so a branch that is never complete is scheduled again on every pass and the run never ends; a live production hit exactly that. A plain task is run once and marked done whatever its completeness says. The work is remote listings, so the (era, point, version) units are checked --threads at a time (8) instead of by luigi workers, and every unit is reported before an unreadable one stops the run.

Not while merges of the same era are in flight

A merge uploads its merged file and only then deletes the staged inputs it consumed, and this task reads the staging tree before the merged one, so a group that merges underneath it is still accounted for. What that does not cover is storage that has not yet made the new merged file visible — FNAL EOS has shown read-after-write lag — which would read as "neither staged nor merged". Run it when the era's merges are not running, and read the report before passing --prune.

CollectGridpacks

The way back: it collects the gridpacks a setup produced into the local DSProdGridpacks checkout, so they can be committed and reused (and so the next production imports them instead of regenerating them).

For each distinct gridpack of the setup it reports one of three states — already in store, collected (it was on fs_default, so it is downloaded into the store checkout), or not produced yet — and writes the same list to data/CollectGridpacks/<setup>/collected.json. Every collected gridpack gets a README.md next to it recording how it was made: the setup, the cards it came from, and the exact commits of DSProd, DSProdModels and genproductions_scripts — the provenance the store requires.

It commits nothing. Adding ~30 MB of Git-LFS content per gridpack stays a deliberate act, so the task ends by printing the commands to run:

law run CollectGridpacks --setup <setup>
# ... then, as printed:
git -C gridpacks add --sparse '<process>/<generator>/<comEnergy>/<production-mode>/<gridpack-name>'
git -C gridpacks commit -m "add ..."
git -C gridpacks push

--sparse is required: the store is checked out sparsely, and a plain git add skips a *.tar.xz path with only a hint.

The task has no output to be "done": it re-runs every time it is asked for.

MakeManifest (planned)

A future task that writes the dataset manifest FLAF uses to enumerate the produced samples. Not yet implemented.

Useful LAW options

• --print-status -1 — show what LAW considers done vs. pending for the full graph, without running anything.
• --print-deps -1 — print the dependency tree (with the backend each task would use).
• --eras '<glob>' / --points '<glob>' — produce only the matching eras / points of the setup (see production setups).
• --test <n> — produce <n> events per point and era in one job, into a separate <output>_test area.
• --branch <n> / --branches <a,b> — run only selected branches of a workflow.
• --workers <n> — run several branches in parallel locally.
• --<TaskName>-<param> <value> — override a parameter of an upstream task (e.g. --MakeGridpack-workflow local while RunProd runs on HTCondor).