cat / goq
clone

turns a linux box into a private steam streaming appliance: plug in, tap your key, play from another computer

ci: write Cargo caches only from main and share redundant entries

Dami committed 1af9ebe
A cache entry belongs to the branch that wrote it, so saving from pull
requests gave every open pull request its own full set. The first run of the
split gate wrote 9.4GB across eight entries against GitHub's 10GB per-repo
ceiling, which meant two or three concurrent pull requests would evict each
other's entries continuously and quietly send the legs back to cold builds --
undoing the speedup the split just bought.

Legs now restore with actions/cache/restore and save with actions/cache/save
gated to `github.event_name == 'push'`, both at the digest already pinned for
actions/cache, so no new action enters the trust set. Pull requests read
main's entries through restore-keys and write nothing, so the repository holds
one set instead of one per branch.

Loopback cases 1-3 are all debug-profile builds of the same binaries and were
storing three near-identical 767MB copies. They now share one entry, with case
1 nominated as its writer and case 4 owning the release entry, so no two
concurrent matrix legs race to reserve the same key.

Together that takes main's steady-state footprint from 9.4GB to roughly 6.3GB.

The policy checker gains the matching invariants: a step-level `if` is now
permitted only on a cache save and only with an approved push-gated condition,
an unconditional save is rejected outright, a restore must be paired with a
save under the same key and paths, the restore must carry the id the save
condition reads, and the combined read-write cache action may only ever cache
the one-megabyte cargo-zigbuild binary. Suite grows from 51 tests to 59.
commit
1af9ebe47dcdd84c5f79a1bfd6d7960fcdb2f83e
parent
8941f0b20bbeaecf6b7b87dfd1a84624325e3f99
author
Dami <dami.etoile@gmail.com>
3 files changed, +337 -12
M.github/workflows/ci.yml114+++++++++++++++++++++++--
Mscripts/tests/test_ci_cross_build_policy.py140++++++++++++++++++++++++++++++
Mscripts/verify_ci_cross_build_policy.py95++++++++++++++++++++-
M .github/workflows/ci.yml +105 -9
@@ -17,6 +17,12 @@ concurrency:
1717 # with `demo-gate` as the single required status check: it fails unless every1818 # leg either succeeded or was a leg the audited docs-only scope may skip.1919 #20+# Caches are restored everywhere but written only on a push to main. A cache21+# entry is scoped to the branch that wrote it, so saving on pull requests gave22+# every open pull request its own ~9GB set and the repository blew through the23+# 10GB ceiling, evicting the entries the legs depend on. Pull requests read24+# main's entries through restore-keys instead, so the repository holds one set.25+#2026 # scripts/verify_ci_cross_build_policy.py enforces that structure, and2127 # scripts/tests/demo-gate-stages.sh proves the legs still spend every check the2228 # single job used to run. Adding a leg means adding it to `demo-gate.needs`;
@@ -108,7 +114,8 @@ jobs:
108114 run: ./scripts/install-linux-build-deps.sh --profile full109115 110116 - name: Restore the Cargo build cache111- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4117+ id: cargo-cache118+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4112119 with:113120 path: |114121 ~/.cargo/registry/index
@@ -122,6 +129,17 @@ jobs:
122129 - name: Run the repository test stage123130 run: ./scripts/verify-demo-build.sh --stage repo-tests124131 132+ - name: Save the Cargo build cache133+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'134+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4135+ with:136+ path: |137+ ~/.cargo/registry/index138+ ~/.cargo/registry/cache139+ ~/.cargo/git/db140+ target141+ key: cargo-repo-tests-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}142+125143 native-tests:126144 name: Workspace tests and Clippy127145 runs-on: ubuntu-24.04
@@ -139,7 +157,8 @@ jobs:
139157 run: ./scripts/install-linux-build-deps.sh --profile full140158 141159 - name: Restore the Cargo build cache142- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4160+ id: cargo-cache161+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4143162 with:144163 path: |145164 ~/.cargo/registry/index
@@ -153,6 +172,17 @@ jobs:
153172 - name: Run the native gate stage154173 run: ./scripts/verify-demo-build.sh --stage native155174 175+ - name: Save the Cargo build cache176+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'177+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4178+ with:179+ path: |180+ ~/.cargo/registry/index181+ ~/.cargo/registry/cache182+ ~/.cargo/git/db183+ target184+ key: cargo-native-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}185+156186 cross-build:157187 name: Linux glibc cross build158188 runs-on: ubuntu-24.04
@@ -169,6 +199,8 @@ jobs:
169199 - name: Install Linux build dependencies170200 run: ./scripts/install-linux-build-deps.sh --profile full171201 202+ # A megabyte, and worth writing from every branch so a pull request never203+ # pays the cargo-zigbuild compile.172204 - name: Restore pinned cargo-zigbuild173205 uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4174206 with:
@@ -176,7 +208,8 @@ jobs:
176208 key: cargo-zigbuild-${{ runner.os }}-${{ runner.arch }}-0.23.0177209 178210 - name: Restore the Cargo build cache179- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4211+ id: cargo-cache212+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4180213 with:181214 path: |182215 ~/.cargo/registry/index
@@ -220,6 +253,17 @@ jobs:
220253 GOQ_REQUIRE_LINUX_CROSS_BUILD: "1"221254 run: ./scripts/verify-demo-build.sh --stage cross222255 256+ - name: Save the Cargo build cache257+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'258+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4259+ with:260+ path: |261+ ~/.cargo/registry/index262+ ~/.cargo/registry/cache263+ ~/.cargo/git/db264+ target265+ key: cargo-cross-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}266+223267 gstreamer-gate:224268 name: In-process GStreamer gate225269 runs-on: ubuntu-24.04
@@ -237,7 +281,8 @@ jobs:
237281 run: ./scripts/install-linux-build-deps.sh --profile full238282 239283 - name: Restore the Cargo build cache240- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4284+ id: cargo-cache285+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4241286 with:242287 path: |243288 ~/.cargo/registry/index
@@ -253,6 +298,17 @@ jobs:
253298 GOQ_VERIFY_IN_PROCESS_GSTREAMER: "1"254299 run: ./scripts/verify-demo-build.sh --stage gstreamer255300 301+ - name: Save the Cargo build cache302+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'303+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4304+ with:305+ path: |306+ ~/.cargo/registry/index307+ ~/.cargo/registry/cache308+ ~/.cargo/git/db309+ target310+ key: cargo-gstreamer-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}311+256312 loopback:257313 name: Loopback proof ${{ matrix.case }}258314 runs-on: ubuntu-24.04
@@ -261,8 +317,24 @@ jobs:
261317 if: needs.scope.outputs.code == 'true'262318 strategy:263319 fail-fast: false320+ # Cases 1-3 are all debug-profile builds of the same binaries, so they321+ # share one cache entry instead of storing three near-identical copies.322+ # One designated case per entry writes it, which also means no two323+ # concurrent matrix legs race to reserve the same key.264324 matrix:265- case: [1, 2, 3, 4]325+ include:326+ - case: 1327+ cache: debug328+ save: "true"329+ - case: 2330+ cache: debug331+ save: "false"332+ - case: 3333+ cache: debug334+ save: "false"335+ - case: 4336+ cache: release337+ save: "true"266338 env:267339 CARGO_TERM_COLOR: always268340 RUST_BACKTRACE: "1"
@@ -274,20 +346,32 @@ jobs:
274346 run: ./scripts/install-linux-build-deps.sh --profile full275347 276348 - name: Restore the Cargo build cache277- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4349+ id: cargo-cache350+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4278351 with:279352 path: |280353 ~/.cargo/registry/index281354 ~/.cargo/registry/cache282355 ~/.cargo/git/db283356 target284- key: cargo-loopback-${{ matrix.case }}-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}357+ key: cargo-loopback-${{ matrix.cache }}-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}285358 restore-keys: |286- cargo-loopback-${{ matrix.case }}-${{ runner.os }}-359+ cargo-loopback-${{ matrix.cache }}-${{ runner.os }}-287360 288361 - name: Run the loopback proof289362 run: ./scripts/verify-demo-build.sh --stage loopback --loopback-case ${{ matrix.case }}290363 364+ - name: Save the Cargo build cache365+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true' && matrix.save == 'true'366+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4367+ with:368+ path: |369+ ~/.cargo/registry/index370+ ~/.cargo/registry/cache371+ ~/.cargo/git/db372+ target373+ key: cargo-loopback-${{ matrix.cache }}-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}374+291375 release-containment:292376 name: Release-profile bypass containment293377 runs-on: ubuntu-24.04
@@ -305,7 +389,8 @@ jobs:
305389 run: ./scripts/install-linux-build-deps.sh --profile full306390 307391 - name: Restore the Cargo build cache308- uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4392+ id: cargo-cache393+ uses: actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4309394 with:310395 path: |311396 ~/.cargo/registry/index
@@ -319,6 +404,17 @@ jobs:
319404 - name: Run the release containment stage320405 run: ./scripts/verify-demo-build.sh --stage containment321406 407+ - name: Save the Cargo build cache408+ if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'409+ uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4410+ with:411+ path: |412+ ~/.cargo/registry/index413+ ~/.cargo/registry/cache414+ ~/.cargo/git/db415+ target416+ key: cargo-containment-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}417+322418 decky-gate:323419 name: Decky plugin gate324420 runs-on: ubuntu-24.04
M scripts/tests/test_ci_cross_build_policy.py +140 -0
@@ -502,5 +502,145 @@ class GateScriptTests(unittest.TestCase):
502502 verify_gate_script(mutated)503503 504504 505+class CacheBudgetTests(unittest.TestCase):506+ """A cache entry belongs to the branch that wrote it.507+508+ Saving from pull requests gave every open pull request its own509+ multi-gigabyte set, which exhausted the repository's cache ceiling and510+ evicted the entries the legs depend on. These assert that saves stay gated511+ to pushes and that a leg cannot smuggle in an unpaired or mismatched entry.512+ """513+514+ def test_rejects_unconditional_cache_save(self) -> None:515+ mutated = replace_once(516+ WORKFLOW,517+ " - name: Save the Cargo build cache\n"518+ " if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'\n"519+ " uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4\n"520+ " with:\n"521+ " path: |\n"522+ " ~/.cargo/registry/index\n"523+ " ~/.cargo/registry/cache\n"524+ " ~/.cargo/git/db\n"525+ " target\n"526+ " key: cargo-repo-tests-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}\n",527+ " - name: Save the Cargo build cache\n"528+ " uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4\n"529+ " with:\n"530+ " path: |\n"531+ " ~/.cargo/registry/index\n"532+ " ~/.cargo/registry/cache\n"533+ " ~/.cargo/git/db\n"534+ " target\n"535+ " key: cargo-repo-tests-${{ runner.os }}-${{ hashFiles('Cargo.lock', 'rust-toolchain.toml') }}\n",536+ )537+ with self.assertRaisesRegex(PolicyError, "unconditionally"):538+ verify(mutated)539+540+ def test_rejects_cache_save_on_pull_requests(self) -> None:541+ for condition in (542+ "if: always()",543+ "if: github.event_name == 'pull_request'",544+ "if: steps.cargo-cache.outputs.cache-hit != 'true'",545+ ):546+ with self.subTest(condition=condition):547+ mutated = in_job(548+ WORKFLOW,549+ "native-tests",550+ "if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'",551+ condition,552+ )553+ with self.assertRaisesRegex(PolicyError, "unapproved condition"):554+ verify(mutated)555+556+ def test_rejects_conditional_step_that_is_not_a_cache_save(self) -> None:557+ mutated = in_job(558+ WORKFLOW,559+ "native-tests",560+ " - name: Run the native gate stage\n",561+ " - name: Run the native gate stage\n"562+ " if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'\n",563+ )564+ with self.assertRaisesRegex(PolicyError, "conditionally disabled step"):565+ verify(mutated)566+567+ def test_rejects_restore_without_a_paired_save(self) -> None:568+ start = WORKFLOW.index(" - name: Save the Cargo build cache\n")569+ end = WORKFLOW.index(" native-tests:\n")570+ mutated = WORKFLOW[:start] + WORKFLOW[end:]571+ with self.assertRaisesRegex(PolicyError, "pair its Cargo cache restore"):572+ verify(mutated)573+574+ def test_rejects_save_under_a_different_key(self) -> None:575+ mutated = in_job(576+ WORKFLOW,577+ "release-containment",578+ " - name: Save the Cargo build cache\n"579+ " if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'\n"580+ " uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4\n"581+ " with:\n"582+ " path: |\n"583+ " ~/.cargo/registry/index\n"584+ " ~/.cargo/registry/cache\n"585+ " ~/.cargo/git/db\n"586+ " target\n"587+ " key: cargo-containment-",588+ " - name: Save the Cargo build cache\n"589+ " if: github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'\n"590+ " uses: actions/cache/save@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4\n"591+ " with:\n"592+ " path: |\n"593+ " ~/.cargo/registry/index\n"594+ " ~/.cargo/registry/cache\n"595+ " ~/.cargo/git/db\n"596+ " target\n"597+ " key: cargo-elsewhere-",598+ )599+ with self.assertRaisesRegex(PolicyError, "different key"):600+ verify(mutated)601+602+ def test_rejects_missing_cargo_cache_id(self) -> None:603+ mutated = in_job(604+ WORKFLOW,605+ "gstreamer-gate",606+ " - name: Restore the Cargo build cache\n id: cargo-cache\n",607+ " - name: Restore the Cargo build cache\n",608+ )609+ with self.assertRaisesRegex(PolicyError, "cargo-cache id"):610+ verify(mutated)611+612+ def test_rejects_read_write_cache_beyond_the_zigbuild_binary(self) -> None:613+ mutated = in_job(614+ WORKFLOW,615+ "cross-build",616+ " path: ~/.cargo/bin/cargo-zigbuild\n",617+ " path: target\n",618+ )619+ with self.assertRaisesRegex(PolicyError, "cargo-zigbuild binary"):620+ verify(mutated)621+622+ def test_shared_matrix_cache_nominates_exactly_one_writer(self) -> None:623+ """Two concurrent legs must not race to reserve the same key.624+625+ Cases 1-3 share one debug entry and case 4 owns the release entry, so626+ exactly one leg per distinct cache key may carry save: "true".627+ """628+ legs = WORKFLOW[WORKFLOW.index(job_marker("loopback")) :]629+ legs = legs[: legs.index(job_marker("release-containment"))]630+ writers: dict[str, int] = {}631+ cache = None632+ for line in legs.splitlines():633+ stripped = line.strip()634+ if stripped.startswith("- case:") or stripped.startswith("cache:"):635+ if stripped.startswith("cache:"):636+ cache = stripped.split(":", 1)[1].strip()637+ writers.setdefault(cache, 0)638+ elif stripped.startswith("save:"):639+ if stripped.split(":", 1)[1].strip().strip('"') == "true":640+ assert cache is not None641+ writers[cache] += 1642+ self.assertEqual(writers, {"debug": 1, "release": 1})643+644+505645 if __name__ == "__main__":506646 unittest.main()
M scripts/verify_ci_cross_build_policy.py +92 -3
@@ -36,6 +36,27 @@ from pathlib import Path
3636 3737 3838 CHECKOUT_ACTION = "actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8"39+CACHE_DIGEST = "0400d5f644dc74513175e3cd8d07132dd4860809"40+CACHE_RESTORE_ACTION = f"actions/cache/restore@{CACHE_DIGEST}"41+CACHE_SAVE_ACTION = f"actions/cache/save@{CACHE_DIGEST}"42+COMBINED_CACHE_ACTION = f"actions/cache@{CACHE_DIGEST}"43+COMBINED_CACHE_ALLOWED_PATHS = frozenset({"~/.cargo/bin/cargo-zigbuild"})44+45+# A cache entry is scoped to the branch that writes it, so writing from pull46+# requests gave every open pull request its own multi-gigabyte set and evicted47+# the entries the legs depend on. Saves are therefore gated to pushes. These are48+# the only step-level conditions the policy permits at all: a conditional step49+# is otherwise a way to silently stop running a check.50+PUSH_ONLY_SAVE = "github.event_name == 'push' && steps.cargo-cache.outputs.cache-hit != 'true'"51+ALLOWED_STEP_CONDITIONS = frozenset(52+ {53+ PUSH_ONLY_SAVE,54+ # Matrix legs that share one cache entry nominate a single writer, so no55+ # two concurrent legs race to reserve the same key.56+ f"{PUSH_ONLY_SAVE} && matrix.save == 'true'",57+ }58+)59+3960 GATE_SCRIPT = "./scripts/verify-demo-build.sh"4061 LINUX_CROSS_BUILD_GATE = "./scripts/run-linux-cross-build-gate.sh"4162 DEPENDENCY_SCRIPT = "./scripts/install-linux-build-deps.sh"
@@ -309,9 +330,18 @@ def _check_no_failure_suppression(job_name: str, job: dict) -> None:
309330 if "continue-on-error" in step:310331 raise PolicyError(f"jobs.{job_name} has a step that suppresses failures")311332 if "if" in step:312- raise PolicyError(313- f"jobs.{job_name} has a conditionally disabled step; gate a whole leg instead"314- )333+ # Only a push-gated cache save may be conditional. Anything else334+ # would be a way to stop running a check without saying so.335+ if step.get("uses") != CACHE_SAVE_ACTION:336+ raise PolicyError(337+ f"jobs.{job_name} has a conditionally disabled step; "338+ "gate a whole leg instead"339+ )340+ if step["if"] not in ALLOWED_STEP_CONDITIONS:341+ raise PolicyError(342+ f"jobs.{job_name} saves its cache under an unapproved "343+ f"condition: {step['if']!r}"344+ )315345 body = _run_body(step)316346 for token in FORBIDDEN_RUN_TOKENS:317347 if token in body:
@@ -334,6 +364,64 @@ def _check_no_failure_suppression(job_name: str, job: dict) -> None:
334364 raise PolicyError(f"jobs.{job_name} overrides the checked-out ref")335365 336366 367+def _validate_caches(job_name: str, job: dict) -> None:368+ steps = _steps(job, job_name)369+ restores = [step for step in steps if step.get("uses") == CACHE_RESTORE_ACTION]370+ saves = [step for step in steps if step.get("uses") == CACHE_SAVE_ACTION]371+ combined = [step for step in steps if step.get("uses") == COMBINED_CACHE_ACTION]372+373+ # The combined action reads and writes from every branch, so it stays374+ # available only for the one entry small enough not to matter.375+ for step in combined:376+ with_fields = step.get("with")377+ path = with_fields.get("path", "") if isinstance(with_fields, dict) else ""378+ if path.strip() not in COMBINED_CACHE_ALLOWED_PATHS:379+ raise PolicyError(380+ f"jobs.{job_name} may only read-write cache the pinned "381+ f"cargo-zigbuild binary, not {path.strip()!r}"382+ )383+384+ # Without this an unconditional save would quietly restore the per-pull-385+ # request cache sets that exhausted the repository's cache ceiling.386+ for step in saves:387+ if "if" not in step:388+ raise PolicyError(389+ f"jobs.{job_name} saves its Cargo cache unconditionally; "390+ "saves must be gated to pushes"391+ )392+393+ if len(restores) > 1 or len(saves) > 1:394+ raise PolicyError(395+ f"jobs.{job_name} must restore and save at most one Cargo cache"396+ )397+ if len(restores) != len(saves):398+ raise PolicyError(399+ f"jobs.{job_name} must pair its Cargo cache restore with a save so "400+ "main keeps a fresh entry"401+ )402+ if not restores:403+ return404+405+ restore_with = restores[0].get("with")406+ save_with = saves[0].get("with")407+ if not isinstance(restore_with, dict) or not isinstance(save_with, dict):408+ raise PolicyError(f"jobs.{job_name} Cargo cache steps need a with mapping")409+ if restores[0].get("id") != "cargo-cache":410+ raise PolicyError(411+ f"jobs.{job_name} must give its Cargo cache restore the cargo-cache id "412+ "that the save condition reads"413+ )414+ if restore_with.get("key") != save_with.get("key"):415+ raise PolicyError(416+ f"jobs.{job_name} saves its Cargo cache under a different key than it "417+ "restores"418+ )419+ if restore_with.get("path") != save_with.get("path"):420+ raise PolicyError(421+ f"jobs.{job_name} saves a different set of paths than it restores"422+ )423+424+337425 def _gate_invocation(job_name: str, job: dict) -> dict:338426 matches = [step for step in _steps(job, job_name) if GATE_SCRIPT in _run_body(step)]339427 if len(matches) != 1:
@@ -510,6 +598,7 @@ def verify(workflow_text: str) -> None:
510598 511599 for name, job in jobs.items():512600 _check_no_failure_suppression(name, _require_mapping(job, f"jobs.{name}"))601+ _validate_caches(name, jobs[name])513602 514603 for name in sorted(ALWAYS_RUN_JOBS):515604 if "if" in jobs[name]: