Backends

Every production task accepts --workflow <backend>. The three backends differ only in where the jobs run — the task logic, and all remote EOS I/O, are identical. DSProd always writes its products to the production's storage area (see settings) itself, so the batch systems are used purely for compute.

--workflow	Runs on	Use for
local	the current machine	tests, small productions, debugging
htcondor	CERN HTCondor batch	medium productions
crab	the WLCG grid (CRAB)	large productions needing many sites

local

Runs branches in the current shell. Combine with --workers <n> to run several branches in parallel. Needs a valid VOMS proxy for the EOS writes. This is the backend used by the first-production walkthrough.

htcondor

Submits to the CERN HTCondor pool. Jobs bootstrap from the shared AFS checkout (bootstrap.sh) and see the installed CMSSW releases under soft/ directly. Relevant knobs (all significant=False, so they do not change task identity):

• --max-runtime <hours> (DSProd's per-task defaults: MakeGridpack 12 h, RunProd 24 h, NanoMergeTask 3 h);
• --n-cpus <n> (RunProd defaults to 4); cmsDriver runs its steps with that many threads;
• --krenew <hours> — how often to renew the Kerberos ticket while polling.

--max-runtime and --n-cpus are per task: each task keeps its own value and neither is passed on to the tasks it requires. Running NanoMergeTask therefore still gives its RunProd requirement the runtime and CPUs of RunProd, not the merge task's 3 h and 1 CPU. To change one, address the task by name — --RunProd-max-runtime 36.

The defaults above are DSProd's own. What a production asks for is written in its setup, which outranks them and is outranked in turn by the command line — see resources:. The setup this repository ships for the HH production, for instance, gives RunProd 16 h.

Jobs request AlmaLinux9 workers and write their HTCondor logs under data/logs/.

Resuming a production (both batch backends)

When a run picks up an existing submission, law re-checks the outputs of every job it had recorded as finished and retries the ones whose outputs are missing (initially missing task outputs). For a few files deleted by hand that is exactly what you want. For a whole production it is not, so a check that condemns more than 10 % of the workflow stops the run instead, without submitting anything:

8299 of 8300 jobs recorded as finished no longer have their outputs, so this run would
regenerate most of the sample. Nothing was submitted.

What to look at, in order:

1. Were the outputs consumed downstream? NanoMergeTask deletes each nano file it merges, and the produced/ records are what marks those seeds done. Check that the records exist — for a production that predates them, run BackfillProducedRecords.
2. Was the storage reachable? A listing that fails no longer reads as "the file is not there" — gfal_ls_checked retries and then raises, and only gfal's own no such file or directory counts as absence — so an unreachable endpoint now stops the run with its own error instead of condemning the sample. If that is what you got, run again once it is back.
3. Do you actually want the work redone? Delete the workflow's submission file under data/jobs/ and start again — a run with nothing to resume never performs this check.

crab

Submits to the WLCG grid via CRAB, built on law.contrib.cms.CrabWorkflow. This is the backend for large-scale private production, where CERN HTCondor alone does not provide enough resources.

Because WLCG workers have no AFS, the DSProd code (plus genproductions_scripts and the vendored law/luigi) is shipped as a CRAB inputFiles tarball, built at submit time and unpacked by bootstrap.sh; CMSSW is set up from cvmfs on the worker. DSProd owns all output and log I/O (products go to the production's storage area via the gfal-CLI interface), so CRAB's own stageout and log transfer are forced off.

The tarball is built once per law run and checked before it is used: every top-level entry must be present and the archive readable. GNU tar's "file changed as we read it" warning — routine when the production area sits on EOS and harmless, since the entry is still archived in full — is tolerated rather than allowed to abort a submission of thousands of jobs.

Gridpacks are not part of that tarball — the input sandbox is size-limited, and a ~30 MB gridpack per job would be wasteful anyway. A production job downloads the gridpack it needs from fs_default, where ImportGridpack or MakeGridpack put it. The gridpacks/ store is likewise never shipped: importing from it is local by construction.

Requirements

• a shell with env.sh sourced — it puts DSProd's crab wrapper and python shim on PATH, both of which law needs to drive CRAB (see the note below);
• a VOMS proxy and a MyProxy credential valid for at least 5 days, the latter delegated with crab createmyproxy and not with myproxy-init (see Installation);
• optionally, a crab: block in the global / user config — not in the production setup, so the same setup runs on any backend:

crab:
  max_cores: 4          # ceiling on a job's cores (1, 2, 4 or 8); caps each task's own n_cpus
  # memory_mb: 0        # default per-job request in MB; 0 = CRAB's max(3000, 2500 * numCores)
  # parallel_jobs: 5000     # jobs per CRAB task / in flight; `auto` = scale with the run
  # refill_fraction: 0.2    # min wave size / free slots, as a fraction of parallel_jobs
  # retry_release_minutes: 45  # release a parked retry after this long, whatever the wave size

No CRAB output location

The crab: block holds compute settings only. Products go to fs_default — the same location as with any other backend. CRAB does demand a storageSite/outLFNDirBase even when it transfers nothing; DSProd fills those in automatically as a submit-time formality, so there is nothing to configure and no second storage area to keep in sync.

Job waves

A production of tens of thousands of branches cannot be one CRAB task (a task holds at most a few thousand jobs). DSProd therefore keeps at most crab.parallel_jobs jobs in flight — 5000 by default — and submits the rest in waves. So a 43 000-branch production is simply launched as one law run; there is no need to chunk the branch range by hand. --parallel-jobs <n> on the command line overrides both settings.

crab.parallel_jobs: auto derives the ceiling from the run instead: the larger of 5000 and the number of branches this run submits, capped at 8000. It is worth setting only for a production too large to go out in one wave. Nothing queues on this backend — 4798 of the 4800 branches of Run3_2023BPix started within half an hour of being submitted — so the makespan is the ramp plus one job length per wave, and the number of waves is branches / parallel_jobs. At the 87 600 branches of the 2024 setup that is 17.5 waves at 5000 against 11.0 at 8000, about -37 % of the makespan; at BPix scale auto resolves to the same 5000 as the default and changes nothing at all, which is why it is opt-in. The cap is below the number everyone quotes: "a CRAB task holds ~10 000 jobs" is folklore — the limit is enforced on the server and invisible from the client — and the largest task submitted from here so far held 4090 jobs.

A wave becomes its own CRAB task only when it is worth one. crab.refill_fraction (default 0.2) sets that bar as a fraction of parallel_jobs: with the defaults a wave needs 1000 jobs waiting and 1000 free slots. Jobs below the bar are held back — but only for as long as reaching it is still possible. Once the work left in the whole production (running + waiting) can no longer fill a wave, waiting could only delay it, so whatever is waiting goes out at once, however little that is. parallel_jobs set to unlimited (--parallel-jobs 0) bypasses all of this and restores law's own behaviour.

auto raises that bar with it

The bar is refill_fraction of parallel_jobs, so at 8000 a wave needs 1600 waiting jobs instead of 1000, and the tail rule above releases the last 1600 jobs in whatever small tasks the polls make of them. If that tail matters, set refill_fraction: 0.125 next to auto to put the bar back at ~1000. The default deliberately stays where it is, so auto moves one number only.

What is measured against the bar is the waiting backlog — never-submitted branches plus the retries already held back. A generation of retries offered by the current poll is parked first and counted on the next one, so a fresh generation never opens the gate on size before it has waited for anything (it still counts towards the tail rule above, which is about what the production can still fill, not about what has waited).

Three consequences worth knowing:

• Small productions are never batched. A 12-job production can never fill a wave, so a job that fails there is resubmitted on the next poll, exactly as before.
• Large productions have a short tail, not a serialised one. Retries are held only while the production is still busy; they are released as soon as fewer than one wave of work remains, not when the last job finishes. A 3270-job production that loses 226 jobs early runs as two CRAB tasks, with the retries going out around three quarters of the way through.
• A parked retry is released on a timer after crab.retry_release_minutes (default 45), whatever the wave size. Waiting for a wave that a handful of retries cannot fill costs a full job length per retry generation: over the 4800-branch Run3_2023BPix production the parked retries waited 11.35 h at the median, ~10.5 h of the 68.4 h it took to reach 99.4 %. The window is per parked set, not per job: a release restarts it for whatever it could not take, so a production losing jobs in a trickle creates at most one extra CRAB task per window (226 failures over those 68.4 h would be ~1 task per 45 min holding a couple of jobs each). Set retry_release_minutes: 0 to let every retry out on the next poll, or a value larger than any production takes (e.g. 100000) to switch the timer off and go back to the size bar alone — which is worth knowing if refill_fraction: 1.0 was set to insist on full waves.

Only the waiting time lives in the driver's memory; the parked jobs themselves are in the submission file, next to their attempt counts. A restarted driver therefore finds them, starts a fresh window on its first polling iteration and releases them one window later — at worst one extra window of waiting, never a lost job.

Once a wave does go out, law fills it up to parallel_jobs from the backlog whatever opened the gate — so a release on the timer takes as many never-submitted branches with it as there are free slots. The CRAB task is being created either way, and the parked retries are at the front of the queue, so they are in it. A never-submitted branch never starts a window of its own: the backlog alone still waits for a full-sized wave.

Without the size bar, law creates a fresh CRAB task the moment a single job finishes or fails — that 3270-job production produced a second, 226-job CRAB task ten minutes in.

Site selection

DSProd jobs carry no real input dataset (they generate their own events, and the CRAB config sets ignoreLocality), so nothing ties them to a particular site. Because ignoreLocality is set, the CRAB client requires a Site.whitelist; DSProd therefore defaults it to every tier (T1_*, T2_*, T3_*), which is the widest pool the client accepts. Configuring a whitelist can only narrow it, so do not add one just to "get more sites".

Restricting is worth it in two situations:

• Sites that cannot reach your storage. Unlike a normal CRAB task, DSProd does its own stageout: the job writes its product over the WAN to fs_default (e.g. CERNBox via davs://) instead of to local site storage. A worker without outbound access to that endpoint will run the full payload and only then fail on the copy. Gridpack generation likewise needs outbound network to fetch generator tarballs. Put such sites in blacklist as you find them.
• Keeping jobs near the storage, e.g. whitelist: [ T2_CH_CERN ] for short tests against CERNBox — convenient for debugging, but it throttles throughput, so avoid it for real production.

CMS's global blacklist of known-broken sites stays in force; ignore_global_blacklist: true waives it, which is not recommended with an open site pool.

whitelist entries must be processing sites

If you do set a whitelist, every entry must be a genuine CMS processing site: do not put a storage-only site (e.g. T3_CH_CERNBOX) there, or CRAB refuses the submission with "not in the list of known CMS Processing Site Names".

The same applies to the default: the tier globs are expanded against the list of processing site names CRAB itself validates against — CRIC's ?json&preset=site-names, the rows of type psn, which is what WMCore.Services.CRIC.getAllPSNs returns — cached for a day in data/cms_psn_sites.json. An earlier version asked CRIC for every site with computeunits instead, on the reasoning that those are the ones that run jobs. They are not the same set: it admitted three names that are not processing sites, one of which (T3_CH_CERN_HelixNebula_REHA, an active site with a real compute element) refused a 36 000-branch production, while omitting 39 names that are.

A submission the server refuses

crab submit returning successfully does not mean the task was accepted: the TaskWorker validates it afterwards and can set SUBMITREFUSED, which is terminal — the task never reaches a scheduler, and crab resubmit and even crab kill refuse it.

Such a task never publishes per-job information, which law reports as an unreadable status, so it used to be retried as though it were merely slow — four attempts a poll, 15 s apart, for as many polls as max_unreadable_polls allows, and then the driver died with the real reason buried a thousand repetitions up the log. DSProd now recognises the server status: the refused task's jobs are reported failed, so law submits them again as a new task, and the reason the server gave is printed once, with the project directory and the site list it was computed from:

the CRAB server refused this submission (SUBMITREFUSED). It will never run, so its jobs are
reported failed and law will submit them as a new task.
  project:  .../crab_RunProd_Run3_XHHbbWW_a741224d
  server:   A site name T3_CH_CERN_HelixNebula_REHA that user specified is not in the list of
            known CMS Processing Site Names
  sites:    computed from .../data/cms_psn_sites.json, now dropped so the next submission
            re-reads CRIC

The cached site list is deleted at that point, so a refusal caused by a stale list heals itself on the next wave. A second refused submission of this run stops the run — counted per submission, so polling one refused task ten times is still one refusal, and a task refused by an earlier run does not count at all: a resumed production re-polls those, and they say nothing about the configuration it is running with. Their jobs are still failed, which is exactly how their branches come back on a corrected run. A refusal is a verdict on what was sent rather than on the grid, and once the site list has been re-read a second one cannot be the cache. Retrying instead would spend every branch's attempts on the same verdict and end in "acceptance not reached" with the cause long out of sight. The jobs of that last task are still reported failed, so law's own bookkeeping stays consistent whichever way the run ends.

WAITING is not a failure

Every task enters the CRAB database as WAITING — crab status prints WAITING on command SUBMIT — and is promoted to NEW by a separate scheduler. That is most visible when the TaskWorker is backlogged, i.e. when several productions submit at once. law reports the jobs of such a task pending and polls on: nothing is retried and nothing counts towards max_unreadable_polls. But law sets no limit on that wait, so DSProd still bounds it — the wait is repeated in the log while it lasts and stops the run after max_unscheduled_polls (60, five hours at the default interval), because a task that never reaches a scheduler would otherwise stall the production in silence. Only WAITING is bounded; law's other pending states (NEW, HOLDING, QUEUED on command SUBMIT, and SUBMITTED) are left to it.

A submission the server fails

SUBMITFAILED is the TaskWorker or the schedd failing to submit a task, rather than refusing it. law reports every job of such a task failed, with the server's Failure message from server as the error (or submission failed when there is none), and submits them again as a new task — the right recovery for a TaskWorker that had a bad moment. A cause that stays, such as a MyProxy credential the TaskWorker cannot retrieve, fails every new task the same way, and each one spends a retry of every branch in it. So DSProd prints the server's message once per task and stops the run at the second failed submission of this run, quoting what the server said for each. As for refusals, a task is counted once however often it is polled, a task left by an earlier run never counts, and the jobs stay failed either way.

Failing sites

One broken worker node fails jobs in seconds, frees its slot and takes the next one, so it can eat a large share of a production before anyone notices — on 2026-08-27 a single host at one T2 failed 258 jobs with /usr/bin/base64: Input/output error, before any physics ran.

DSProd therefore keeps its own record of how jobs fare per site in data/crab_site_stats.json and quarantines a site that is clearly misbehaving. CRAB reports where each job ran, and since every wave is a new CRAB task, the next wave — retries included — is submitted without the quarantined sites. It is on by default; the thresholds live under crab.auto_blacklist:

key	default	meaning
enabled	true	auto_blacklist: false keeps only the static blacklist
min_failures	5	failures needed before a site can be quarantined at all
min_failure_rate	0.5	... and the fraction of the jobs sent there that failed (of the jobs that ended in the window, for the burst below)
relative_factor	2.0	... and how many times worse than the other sites it must be
min_baseline_jobs	20	... judged against at least this many jobs elsewhere
burst_failures	20	failures inside burst_minutes that quarantine a site at once (a very large value leaves only the rate test)
burst_minutes	15	the window a burst is measured over
quarantine_hours	24	how long its first quarantine lasts; each further one doubles
max_quarantine_hours	768	the ceiling the doubling stops at — 32 days
window_hours	24	outcomes older than this stop counting
max_sites	10	never quarantine more than this many sites at once

Two tests can quarantine a site, and either is enough. The standing one is the rate over window_hours; the other is a burst — burst_failures failures inside burst_minutes. The rate test is slowest against exactly the site that costs most: a black hole fails in seconds, so it cycles through slots faster than any healthy site can finish a job, while its own successes from earlier in the day hold its 24 h ratio under min_failure_rate until they age out. On 2026-09-13 one site's failures were visible from 07:00 — 77 ended failed within that hour, 287 by 08:00 — and it first appears in a submitted blacklist at 09:52. 77 in an hour is just under 20 per quarter of an hour, so the burst test would have fired around the end of that first hour and comfortably inside the second: one to two hours and several hundred jobs earlier.

It keeps the same relative checks — min_failure_rate, relative_factor and min_baseline_jobs, measured over the same short window — so a fault of your own, which fails everywhere at once and therefore looks like a burst everywhere, still blacklists nothing. min_failure_rate is read against a different denominator here, though: the rate test counts every job sent to a site (ended plus in flight), while the burst counts only what ended inside the window, because a job still running carries no timestamp that could place it in a quarter of an hour. That is what lets a large site whose thousand running jobs are healthy still be caught on the 25 failures one bad node produced in ten minutes — and it is worth knowing when tuning the key, since the two tests share it. The burst's one blind spot is a grid on which nothing else has ended a job inside the window: with no baseline to compare against it declines to fire, and only the rate test can.

A site that is still broken when its quarantine runs out earns a longer one: the bans double, 24 h → 48 h → 96 h → … → 32 days, because the count of quarantines served is kept when a ban is lifted. A fixed 6-hour ban that also wiped the site's record was no defence against a site that stays broken — it returned to the whitelist with a clean sheet, had to earn min_failures again, and bought itself another wave every six hours. Over 2026-09-08..10 three sites did exactly that three times each, and 93 % of that production's 4197 job failures came from them.

The evidence still ages out with window_hours, and a lifted ban is never re-armed on the failures it was served for — the site is judged on the outcomes recorded after its ban, so one clean wave clears the quarantine (though never the count) and one bad one doubles it. quarantines in data/crab_site_stats.json is the count; delete the file to forget a site's history entirely.

Only what CRAB says about a job enters the record, and only from the status response itself: a job that finished, or that failed with a job-level error code. Everything else is law's own bookkeeping and says nothing about a site — a resumed run flips jobs whose outputs are gone to retry, and a killed task reports all of its jobs as failed. Counting those once turned a single poll into 8285 failures spread over every site of the production, and the ~100 % baseline that resulted left the quarantine unable to fire for a node that was failing two thirds of its jobs. If a record ever looks like that, delete data/crab_site_stats.json — it is advisory and rebuilds within a poll or two. For the same reason, entries in it that cannot be read back (a file written by an older version, a half-written one) are dropped when it is loaded: an advisory file must not be able to stop a submission.

A site's failure rate counts every job sent there — the ones that already ended plus the ones still in flight. That distinction matters more than it looks: a job fails in seconds and succeeds in hours, so a rate computed over finished jobs alone reads as ~100 % at every site early in a production, no site stands out, and nothing is ever quarantined. Counting jobs in flight, the site that swallowed 335 of its 391 jobs sits at 0.86 while everyone else is between 0.007 and 0.08.

These defaults are what keep this from making things worse. A site is only quarantined for being worse than the others, judged against a real baseline, so a bug of your own — which fails everywhere — blacklists nothing; a lone site is never quarantined, because there would be nowhere left to run; at most max_sites are held out at once; and every quarantine expires, after which the site is judged again on the outcomes recorded after its ban rather than staying condemned on the ones that earned it — what is kept is the count of bans served, which sets the length of the next one. Jobs already submitted keep going to the site they were assigned — CRAB cannot re-target a running task.

A site you know is bad belongs in the static blacklist instead: that one is never lifted.

An unreadable status response does not stop the production

crab status occasionally returns output with no Status on the CRAB server line, and law treats that as a query error — one per job of the task, because a group query maps a single failure onto every job in it (4763 in one production poll). law then skips the rest of that poll entirely: no status line, no resubmission, and any other task's good data discarded with it.

DSProd retries such a response three times, 15 s apart. If it still cannot be read, the task's jobs are reported as pending — what law itself does for a freshly submitted task with no per-job information yet — and the fact is printed once for the task instead of once per job. A task whose status stays unreadable for ten consecutive polls does raise: a production that quietly stalls is worse than one that stops.

A job that has run is done polling — which is not the same as having succeeded

DSProd disables CRAB's stageout, so a job CRAB reports as transferring/transferred has done all it will ever do and there is nothing left to wait for. law normally decides that per poll by reading disableAutomaticOutputCollection out of the project's crab.log; a log that is missing or was rewritten without that line reads as "transfers expected", and those jobs are then polled as running until the workflow gives up on them. DSProd pins the setting instead, so it does not depend on a log file surviving.

What that state does not say is that the job worked. CRAB parks a payload that exited non-zero in transferring too, in the window between the payload exiting and the post-job classifying it as failed, and law maps the state to finished either way. A poll landing inside that window therefore books a failed job as finished, stamps it dummy_job_id and never queries it again. On 2026-09-18 one poll harvested 113 of them and the status line read finished: 113 for a production that had written nothing at all. Since then the products decide: see A finished job is one with products.

CRAB does not write to your AFS home

CRAB rewrites its task cache ~/.crab3 on every command, status queries included. With $HOME on AFS that makes a long production depend on an AFS token: when the token lapses, every status query fails at once with PermissionError: [Errno 13] Permission denied: '/afs/.../.crab3.<pid>' and law reports it as a status-query failure for all jobs. The crab wrapper env.sh installs therefore points HOME at $DSPROD_CRAB_HOME (default: a per-user directory under $TMPDIR), so nothing in a production run needs AFS. law passes --proxy to submit, status and kill, and that alone makes CRAB skip everything that reads ~/.globus, so a production run never needs the real home either. crab createmyproxy is the exception — you run it yourself, it does read ~/.globus, and that scratch home is node-local and has none — so the wrapper symlinks the real .globus into it, refreshing the link on every call so it self-heals per node (see Installation). A symlink and not export X509_USER_CERT=…: those variables outrank the default proxy in the GSI credential search order, and setting them makes every grid client in the process reach for the encrypted user key instead.

DSProd still renews Kerberos and the AFS token (kinit -R + aklog, hourly, from crab_poll_callback) — but renewal can only extend a ticket that is still valid, so it is not a substitute for keeping the production off AFS.

Why env.sh matters for CRAB

law runs crab inside a CMSSW sandbox of its own, and dumps that sandbox's environment with bare python — which modern CMSSW no longer ships, and for which the DSProd venv's python is not a working substitute under a cmsenv. env.sh therefore writes a crab wrapper and a python → python3 shim into soft/bin and prepends it to PATH. Submitting from a shell that never sourced env.sh fails the sandbox, and law reports it only indirectly, as every job carrying dummy_job_id and being retried with error: unknown job id. DSProd now checks the sandbox before submitting and reports that case directly.

A finished job is one with products

law asks CRAB what happened to a job, and CRAB's answer alone is not enough to retire a branch: a payload that exited non-zero passes through transferring, which law reads as finished (see the note above). DSProd therefore has law verify the branch's outputs before it accepts that status (crab_check_job_completeness). A job whose products are not on fs_default is put back as

branch task(s) incomplete due to missing outputs

and retried like any other failure, instead of being retired with dummy_job_id and dropped from the poll.

The check is not a per-poll storage sweep. law keeps an accepted job in finished_jobs and skips it at the top of every later iteration, so each job is verified once for the whole run.

What it does cost is one directory listing per poll per output directory it consults, because the cached listings are dropped at the start of each check (expire_path_caches, called from crab_check_job_completeness — the one moment law offers between the status query and the verdicts). That is deliberate and it is the difference between a check and a guess: exists() answers an unknown file from a cached listing of its directory, valid for 600 s against a 5 min poll, so a record written since the last listing would read as missing and demote a job that really did finish — costing it one of its 9 retries, a wait at the wave gate, and a retry whose own check reads the same stale entry. Confirming each negative with a stat instead would be sound too, but a gfal-stat against the production endpoint measures ~0.9 s, so it would put ~10 minutes into every driver start where a listing per directory costs seconds.

It is only meaningful because a listing that fails now raises instead of reading as "not there" (gfal_ls_checked). With the old behaviour one blinked gfal-ls would have demoted a job that really did finish and thrown away hours of grid time — which is why the two arrived together.

The htcondor backend keeps law's default and retires a branch on the batch status alone. That is deliberate: HTCondor has no state that means "the payload has stopped but nobody has classified it yet", which is the state this guards against.

When law's own tree briefly disappears

DSProd's software is installed under soft/ in the checkout that drives the production, so a submission can hit a moment when law's installed tree cannot be read. That is expensive without a guard because the job file is built inside law's submit() and the error raised there is caught nowhere between it and luigi: it leaves the poll loop, the whole workflow is marked failed and the driver ends — which it did to a 16 000-branch production on two consecutive days.

The sources law needs are therefore probed before law is handed control. If they cannot be read, the submission round is abandoned: nothing is submitted, no job leaves the backlog, a message names the path, and the next poll — minutes away — submits normally. Only a tree that vanishes inside one submission still raises, from create().

Nothing is lost by the crash either way: the jobs are still recorded as unsubmitted on disk, which is why restarting the driver resumes where it left off. The guard removes the interruption, not a data loss.

If it persists rather than blips

The message carries the errno the storage answered with, because os.path.isfile() returns False for every cause alike. ENOENT points at the tree or its mount not being there — each /eos home letter is a separate automounted FUSE instance, so a mount can be reaped and re-triggered, and findmnt -T <path> says whether it is there now. EACCES/EPERM points at the credential that storage is reached with: an /eos/home-* or /eos/user/* path is authenticated per access against the caller's Kerberos ticket (getfattr --only-values -n eos.identity <path> names the credential cache in use), and a production only ever renews a ticket, never creates one. Read klist -f for two things: expires (24 h) and renew until, which is fixed at the original kinit and cannot be pushed out by renewals — a production expected to run past it needs a fresh kinit. tokens and aklog matter only where the checkout is installed on AFS.

Neither errno was captured for the two incidents — recovering it is exactly why the message now carries one — so treat the split as a starting point, not a diagnosis.

A tree that stays unreadable does not leave the production skipping rounds forever. law dumps its job data before it submits within one poll iteration, and in the production both that data ($ANALYSIS_PATH/data) and the software live under the same mount: so one skip is published and the next iteration's dump ends the run. A repeating message means the dump kept succeeding in between — either the two sit on different storage, or the failure is intermittent.

Stalled jobs (the watchdog)

CRAB reports a job as running for as long as the batch system says its slot is held, which is not the same as the payload still doing anything. A 600-branch production ended with two jobs whose whole status record — wall duration, memory, CPU time — was identical across eight consecutive polls: they had started promptly, run for ~2.7 h, then stopped reporting, and nothing would have reclaimed them until CRAB's 24 h wall-clock rule fired 21 hours later. 598 of 600 branches were done and the production simply waited.

So each running CRAB job refreshes a flag file in one flat directory on fs_watchdog (<output>/heartbeat/<task>_<run>/<branch>), and the driver lists that directory once per interval — one remote call however many jobs are in flight, about 1.6 s at 3000 entries. A job whose flag has not moved for missed_checks intervals is rewritten as failed on that poll, which puts it through law's ordinary retry path: the attempt is counted and the branches go back to the wave gate like any other failure. There is no separate resubmission mechanism.

fs_watchdog: davs://eoshome-k.cern.ch:8444/eos/user/k/kandroso/DSProd_watchdog/  # unset = fs_default
crab:
  watchdog: false          # switch it off entirely
  # or tune it:
  watchdog:
    interval_minutes: 30   # refresh/check period
    missed_checks: 2       # -> declared dead after 60 min of silence

fs_watchdog is optional and defaults to fs_default. There are two reasons to separate them: the flags are one small overwrite per job per interval on a single directory — a metadata load quite unlike the products' few large writes — and a heartbeat is most informative when the endpoint it is written to is not the one the products go to, so that a products-storage outage does not look like a fleet of dead jobs. (It would not be read as one anyway: see the refusal rules below.)

The watchdog is CRAB-only: the failure is a batch system holding a slot it cannot account for, and a local run has no slot to hold. Jobs detect this themselves — the heartbeat is written only when law's CRAB wrapper is what launched the job — so an HTCondor or local run writes no flags and the driver watches nothing.

It abandons the job, it does not kill it

crab kill has no per-job form, and law's cancel ignores the ids it is given and kills the whole task — so condemning one job would take every healthy sibling with it. The branch is resubmitted immediately and the stalled slot is left to CRAB's wall-clock limit.

Refusing to act is the safe direction, and it does refuse

Every verdict spends one of a branch's attempts, and ~30 exhausted branches of a 600-job run end the whole workflow — so a watchdog that condemns healthy jobs is far worse than one that misses a stall. It issues nothing when the listing cannot be read (no listing is no evidence, and an outage must not accumulate staleness); nothing for a job too young to have written its first flag; nothing beyond max_per_interval (5) in one pass; and nothing at all when more than max_stale_fraction (0.5) of running jobs look stale, because writing to the storage can break while reading it still works, and then every flag goes stale while every job is healthy. A branch is rescued at most max_per_branch (1) times: one that stalls wherever it runs is the branch's problem, not the slot's. dry_run: true logs the verdicts it would issue and issues none.

A flag that was there and is gone also produces no verdict: removing it is what a job does on its way out, and CRAB keeps reporting the job as running for minutes afterwards, so failing it then would resubmit a branch that had just been written. A worker that dies without exiting cleanly leaves its flag behind instead, and that is caught as a stale one — which is the shape the incident this exists for actually had.

Why a job failed

CRAB's exit code is a label rather than a diagnosis: every one of the 4197 failures of one production carried exit 5, Error while running CMSSW, and law repeats exactly that next to the job id. What says what actually happened is the exception the payload raised, which lands in the job's stdout on the scheduler — and DSProd's payload errors are written to be read, naming the file, the seed and usually the repair.

The driver therefore fetches that stdout for each newly failed job and prints its last exception:

crab job 66 of crab_NanoMergeTask_Run3_XHHbbWW_4fa198da failed at T2_CH_CERN with exit code 5:
RuntimeError: 1 of 50 staged nano files of this merge group are gone -- seeds 45, e.g. ...

It is a diagnostic, so it stays out of the way: one fetch per failed attempt (a poll that repeats prints nothing new), at most max_failure_reports (5) per poll with the remainder counted, and a stdout that cannot be read or carries no exception says so rather than going quiet. Failures without a job-level exit code are skipped — those are law's own bookkeeping or a killed task, and have no payload output to read, the same rule the site record uses.

The fetch uses law's own log_file URL for the job, which the scheduler serves over HTTPS with client-certificate authentication; the run's VOMS proxy is that certificate.

To look at a job by hand — for one that did not fail, or for the full log — fetch its stdout from the task's web directory with your proxy, remembering --capath /etc/grid-security/certificates or curl returns HTTP 000. crab status/crab getlog re-delegate a MyProxy interactively when run without --proxy (DSProd always passes it, which is why a production run never renews the credential for you). The CRAB backend module documents the details.