Add the jobs accessor to the runtime handle
Job run-fns can now read upstream outputs via (jobs name) — direct or transitive ancestors only, and only :quire/push has actual outputs (sha, ref, pushed-at) for now. Names outside the calling job's transitive-input set raise a Lua error so typos and undeclared dependencies fail loudly. Assisted-by: Claude Opus 4.7 via Claude Code
diff --git a/docs/CI-FENNEL.md b/docs/CI-FENNEL.md
index 4876d53..5ea0036 100644
--- a/docs/CI-FENNEL.md
+++ b/docs/CI-FENNEL.md
@@ -47,7 +47,7 @@ The dependency graph is *derived* from the inputs list. No separate `:needs` fie
### Accessing inputs
-The function receives the **runtime handle** as its sole argument — a table with `sh`, `secret`, and (planned) `jobs` bound on it. Destructure the primitives the body needs:
+The function receives the **runtime handle** as its sole argument — a table with `sh`, `secret`, and `jobs` bound on it. Destructure the primitives the body needs:
```
(fn [{: sh : jobs}]
@@ -59,7 +59,7 @@ The function receives the **runtime handle** as its sole argument — a table wi
The function-call form sidesteps the awkward dot-access on `/`-containing keys. The `:as` rebinding sugar planned for cron and cherry-picked outputs (see "Future: input args" below) layers on top of the same accessor.
-> **v0 status:** the runtime handle currently carries `sh` and `secret`. The `jobs` accessor is the next addition (tracked separately); until it lands, jobs cannot read upstream outputs and `:quire/push` data is not exposed to the body.
+> **v0 status:** `(jobs :quire/push)` is wired. Job-to-job outputs (where `(jobs :build)` returns a job's `run-fn` return value) are not — there's no writer API yet, and a reachable name with no recorded outputs returns `nil`.
### Sources
@@ -192,7 +192,7 @@ Bound on the runtime handle passed into each `run` function. Destructure what yo
Each of these blocks the Fennel function until it returns. Multi-container parallelism inside one job is a v2 want; the v1 model is "the function runs sequentially, calling primitives that block."
-> **v0 status:** `sh` and `secret` are bound today. `jobs`, `container`, `read-file`/`read-json`/`write-file`, `log`, and `env` are planned and tracked separately.
+> **v0 status:** `sh`, `secret`, and `jobs` are bound today. `container`, `read-file`/`read-json`/`write-file`, `log`, and `env` are planned and tracked separately.
Eval is unsandboxed by default (see CI.md). A `run` function that loops forever or allocates without bound will hang or OOM `quire serve`. The mitigation is the same as for any Fennel hang: write `ci.fnl` thoughtfully. The bwrap opt-in (also see CI.md) covers eval and primitive calls together when it lands.
diff --git a/src/ci/lua.rs b/src/ci/lua.rs
index 9cf4659..43fd403 100644
--- a/src/ci/lua.rs
+++ b/src/ci/lua.rs
@@ -8,7 +8,7 @@
//! each `run-fn` at execute time.
use std::cell::RefCell;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use std::rc::Rc;
use mlua::{IntoLua, Lua, LuaSerdeExt};
@@ -94,7 +94,9 @@ fn register_job(
}
/// Per-execution runtime: holds the secrets exposed to the job, the
-/// current-job cursor, and the per-job captured `sh` outputs.
+/// inputs available via `(jobs name)`, the per-job transitive-input
+/// reachability sets, the current-job cursor, and the per-job
+/// captured `sh` outputs.
///
/// Wrap an `Rc<Runtime>` in [`RuntimeHandle`] and convert it via
/// [`IntoLua`] to install it on the Lua VM (sets app data, returns
@@ -104,27 +106,44 @@ fn register_job(
///
/// Outside a run, no runtime is installed; in that case `(sh …)`
/// runs the command but doesn't record (the cursor lookup misses).
-/// `(secret …)` requires a runtime — without one, calls error.
+/// `(secret …)` and `(jobs …)` require a runtime — without one, calls
+/// error.
#[derive(Debug)]
pub(super) struct Runtime {
secrets: HashMap<String, SecretString>,
+ /// Inputs readable via `(jobs name)`. Source outputs (e.g.
+ /// `quire/push`) are populated by the executor before the loop;
+ /// job-to-job outputs are not yet wired up.
+ inputs: RefCell<HashMap<String, mlua::Value>>,
+ /// For each job, the set of names it may legally read via
+ /// `(jobs name)` — its transitive ancestors in the input graph,
+ /// including source refs. Self is never present.
+ transitive_inputs: HashMap<String, HashSet<String>>,
current_job: RefCell<Option<String>>,
outputs: RefCell<HashMap<String, Vec<ShOutput>>>,
}
impl Runtime {
- /// Build a fresh runtime with the given secrets. Cursor and
- /// outputs start empty.
- pub(super) fn new(secrets: HashMap<String, SecretString>) -> Self {
+ /// Build a fresh runtime. `inputs` is the map of source outputs
+ /// already prepared as Lua values; `transitive_inputs` is the
+ /// per-job reachability set from [`Pipeline::transitive_inputs`].
+ pub(super) fn new(
+ secrets: HashMap<String, SecretString>,
+ inputs: HashMap<String, mlua::Value>,
+ transitive_inputs: HashMap<String, HashSet<String>>,
+ ) -> Self {
Self {
secrets,
+ inputs: RefCell::new(inputs),
+ transitive_inputs,
current_job: RefCell::new(None),
outputs: RefCell::new(HashMap::new()),
}
}
/// Mark `id` as the currently executing job. `(sh …)` invocations
- /// from this job's `run_fn` will record output under `id`.
+ /// from this job's `run_fn` will record output under `id`, and
+ /// `(jobs …)` lookups will validate against `id`'s reachability set.
pub(super) fn enter_job(&self, id: &str) {
*self.current_job.borrow_mut() = Some(id.to_string());
}
@@ -143,7 +162,7 @@ impl Runtime {
}
/// `IntoLua` carrier for an `Rc<Runtime>`. Stows the Rc on the VM as
-/// app data and returns the handle table — `{sh, secret}` for now.
+/// app data and returns the handle table — `{sh, secret, jobs}`.
pub(super) struct RuntimeHandle(pub Rc<Runtime>);
impl IntoLua for RuntimeHandle {
@@ -152,10 +171,48 @@ impl IntoLua for RuntimeHandle {
let table = lua.create_table()?;
table.set("sh", lua.create_function(run_sh)?)?;
table.set("secret", lua.create_function(lookup_secret)?)?;
+ table.set("jobs", lua.create_function(lookup_input)?)?;
table.into_lua(lua)
}
}
+/// Body of `(jobs name)`. Returns the outputs registered for `name`
+/// if `name` is a transitive ancestor of the calling job (or a source
+/// ref reachable through one). Errors loudly if `name` isn't reachable
+/// or no runtime is installed.
+fn lookup_input(lua: &Lua, name: String) -> mlua::Result<mlua::Value> {
+ let rt = lua
+ .app_data_ref::<Rc<Runtime>>()
+ .ok_or_else(|| mlua::Error::external("runtime not installed on Lua VM"))?;
+ let calling = rt.current_job.borrow();
+ let calling = calling
+ .as_ref()
+ .ok_or_else(|| mlua::Error::external("(jobs ...) called outside a job's run-fn"))?;
+ let reachable = rt.transitive_inputs.get(calling).ok_or_else(|| {
+ mlua::Error::external(format!(
+ "no transitive-input set for calling job '{calling}'"
+ ))
+ })?;
+ if !reachable.contains(&name) {
+ if name == *calling {
+ return Err(mlua::Error::external(format!(
+ "Job '{calling}' cannot read its own outputs"
+ )));
+ }
+ return Err(mlua::Error::external(format!(
+ "Job '{calling}' cannot read outputs from '{name}' — not in transitive inputs"
+ )));
+ }
+ // Reachable but no outputs recorded yet: nil. Job-to-job outputs
+ // aren't wired up, so this is the common case for non-source names.
+ Ok(rt
+ .inputs
+ .borrow()
+ .get(&name)
+ .cloned()
+ .unwrap_or(mlua::Value::Nil))
+}
+
/// Body of `(secret name)`. Errors as a Lua error if the runtime
/// isn't installed, the name is undeclared, or the file form fails to
/// read.
@@ -315,9 +372,13 @@ mod tests {
/// return the runtime handle. Mirrors what `Run::execute` does so
/// tests can drive a `run_fn` directly.
fn rt(pipeline: &Pipeline, secrets: HashMap<String, SecretString>) -> mlua::Value {
- RuntimeHandle(Rc::new(Runtime::new(secrets)))
- .into_lua(pipeline.fennel().lua())
- .expect("install runtime")
+ RuntimeHandle(Rc::new(Runtime::new(
+ secrets,
+ HashMap::new(),
+ HashMap::new(),
+ )))
+ .into_lua(pipeline.fennel().lua())
+ .expect("install runtime")
}
#[test]
diff --git a/src/ci/pipeline.rs b/src/ci/pipeline.rs
index df01835..e8b77d0 100644
--- a/src/ci/pipeline.rs
+++ b/src/ci/pipeline.rs
@@ -4,11 +4,12 @@
//! Lua/Fennel evaluation lives in the sibling [`super::lua`] module;
//! this module owns the domain types and the structural rules.
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use miette::{NamedSource, SourceSpan};
use petgraph::Graph;
use petgraph::graph::NodeIndex;
+use petgraph::visit::{Bfs, Reversed};
use super::lua;
use crate::Result;
@@ -117,6 +118,40 @@ impl Pipeline {
.collect()
}
+ /// For each job, the set of input names — direct and transitive,
+ /// including source refs — reachable through the input graph. The
+ /// job's own id is not included.
+ ///
+ /// Used by the executor to validate `(jobs name)` lookups: the
+ /// calling job may only read outputs from names in its set.
+ ///
+ /// Walks the existing dependency graph in reverse (ancestors of
+ /// the job) via petgraph's BFS. Source refs aren't graph nodes,
+ /// so they're scooped up from the inputs lists of every visited
+ /// job.
+ pub(crate) fn transitive_inputs(&self) -> HashMap<String, HashSet<String>> {
+ let reversed = Reversed(&self.graph);
+ let mut result: HashMap<String, HashSet<String>> = HashMap::new();
+ for job in &self.jobs {
+ let start = self.node_index[&job.id];
+ let mut reachable = HashSet::new();
+ let mut bfs = Bfs::new(reversed, start);
+ while let Some(idx) = bfs.next(reversed) {
+ let visited = &self.jobs[self.graph[idx]];
+ if idx != start {
+ reachable.insert(visited.id.clone());
+ }
+ for input in &visited.inputs {
+ if !self.node_index.contains_key(input) {
+ reachable.insert(input.clone());
+ }
+ }
+ }
+ result.insert(job.id.clone(), reachable);
+ }
+ result
+ }
+
/// Parse and validate a ci.fnl source string into a `Pipeline`.
///
/// Delegates evaluation to [`lua::parse`] for the Fennel-side work,
@@ -549,4 +584,50 @@ mod tests {
"should report unreachable job 'orphan': {errs:?}"
);
}
+
+ #[test]
+ fn transitive_inputs_collects_direct_and_indirect() {
+ let pipeline = Pipeline::load(
+ r#"(local ci (require :quire.ci))
+(ci.job :setup [:quire/push] (fn [_] nil))
+(ci.job :build [:setup] (fn [_] nil))
+(ci.job :test [:build :setup] (fn [_] nil))"#,
+ "ci.fnl",
+ )
+ .expect("load should succeed");
+
+ let map = pipeline.transitive_inputs();
+
+ assert_eq!(
+ map["setup"],
+ ["quire/push"].iter().map(|s| s.to_string()).collect()
+ );
+ assert_eq!(
+ map["build"],
+ ["setup", "quire/push"]
+ .iter()
+ .map(|s| s.to_string())
+ .collect()
+ );
+ assert_eq!(
+ map["test"],
+ ["build", "setup", "quire/push"]
+ .iter()
+ .map(|s| s.to_string())
+ .collect()
+ );
+ }
+
+ #[test]
+ fn transitive_inputs_excludes_self() {
+ let pipeline = Pipeline::load(
+ r#"(local ci (require :quire.ci))
+(ci.job :only [:quire/push] (fn [_] nil))"#,
+ "ci.fnl",
+ )
+ .expect("load should succeed");
+
+ let map = pipeline.transitive_inputs();
+ assert!(!map["only"].contains("only"), "self should not be in set");
+ }
}
diff --git a/src/ci/run.rs b/src/ci/run.rs
index 8c8d6d7..4a654d0 100644
--- a/src/ci/run.rs
+++ b/src/ci/run.rs
@@ -250,7 +250,7 @@ impl Run {
base,
state,
id,
- runtime: Rc::new(Runtime::new(HashMap::new())),
+ runtime: Rc::new(Runtime::new(HashMap::new(), HashMap::new(), HashMap::new())),
};
run.read_meta()?;
run.read_times()?;
@@ -259,14 +259,16 @@ impl Run {
/// Drive `pipeline` to completion through this run.
///
- /// Constructs a fresh [`Runtime`] with `secrets`, installs it on
- /// the pipeline's Lua VM, topo-sorts the jobs, transitions
- /// Pending → Active, then invokes each `run_fn` in dependency
- /// order with the runtime handle as its sole argument. `(sh …)`
- /// calls record their captured output under the current job —
- /// readable via [`Run::outputs`] after `execute` returns. The
- /// run finishes in `Complete` if every job's `run_fn` returned
- /// without error, otherwise `Failed`.
+ /// Constructs a fresh [`Runtime`] with `secrets`, the source
+ /// outputs (`:quire/push` from `meta.yml`), and the per-job
+ /// transitive-input sets; installs it on the pipeline's Lua VM,
+ /// topo-sorts the jobs, transitions Pending → Active, then
+ /// invokes each `run_fn` in dependency order with the runtime
+ /// handle as its sole argument. `(sh …)` calls record their
+ /// captured output under the current job — readable via
+ /// [`Run::outputs`] after `execute` returns. The run finishes
+ /// in `Complete` if every job's `run_fn` returned without error,
+ /// otherwise `Failed`.
///
/// Source-ref filtering (e.g. running only `quire/push`-reachable
/// jobs) is not yet implemented; for now every validated job runs.
@@ -275,9 +277,13 @@ impl Run {
pipeline: &Pipeline,
secrets: HashMap<String, SecretString>,
) -> Result<()> {
- self.runtime = Rc::new(Runtime::new(secrets));
+ let lua = pipeline.fennel().lua();
+ let inputs =
+ source_outputs(lua, &self.read_meta()?).expect("build source outputs on Lua VM");
+ let transitive_inputs = pipeline.transitive_inputs();
+ self.runtime = Rc::new(Runtime::new(secrets, inputs, transitive_inputs));
let rt_value = RuntimeHandle(self.runtime.clone())
- .into_lua(pipeline.fennel().lua())
+ .into_lua(lua)
.expect("install runtime on Lua VM");
let order: Vec<String> = pipeline
@@ -381,6 +387,20 @@ impl Run {
}
}
+/// Build the source-ref outputs table read by `(jobs name)` for source
+/// names. For v1, only `:quire/push` is exposed, derived from the run
+/// meta — `:sha`, `:ref`, `:pushed-at`. Branch/tag/etc are deferred.
+fn source_outputs(lua: &mlua::Lua, meta: &RunMeta) -> mlua::Result<HashMap<String, mlua::Value>> {
+ let push = lua.create_table()?;
+ push.set("sha", meta.sha.clone())?;
+ push.set("ref", meta.r#ref.clone())?;
+ push.set("pushed-at", meta.pushed_at.to_string())?;
+
+ let mut inputs = HashMap::new();
+ inputs.insert("quire/push".to_string(), mlua::Value::Table(push));
+ Ok(inputs)
+}
+
/// Write a serializable value to a YAML file atomically (temp file + rename).
pub(crate) fn write_yaml<T: serde::Serialize>(path: &Path, value: &T) -> Result<()> {
let tmp_path = path.with_extension("yml.tmp");
@@ -563,7 +583,7 @@ mod tests {
base: PathBuf::from("/tmp/quire-test-runs/test.git"),
state: RunState::Pending,
id: uuid::Uuid::now_v7().to_string(),
- runtime: Rc::new(Runtime::new(HashMap::new())),
+ runtime: Rc::new(Runtime::new(HashMap::new(), HashMap::new(), HashMap::new())),
};
let result = run.transition(RunState::Active);
@@ -728,4 +748,130 @@ mod tests {
"b should not have run after a failed"
);
}
+
+ #[test]
+ fn jobs_returns_quire_push_outputs_for_direct_input() {
+ let (_dir, quire) = tmp_quire();
+ let runs = test_runs(&quire);
+ let mut run = runs.create(&test_meta()).expect("create");
+
+ let pipeline = load(
+ r#"(local ci (require :quire.ci))
+(ci.job :grab [:quire/push]
+ (fn [{: sh : jobs}]
+ (let [push (jobs :quire/push)]
+ (sh ["echo" push.sha push.ref]))))"#,
+ );
+
+ run.execute(&pipeline, HashMap::new()).expect("execute");
+
+ let outputs = run.outputs("grab");
+ assert_eq!(outputs.len(), 1);
+ assert_eq!(outputs[0].stdout, "abc123 refs/heads/main\n");
+ }
+
+ #[test]
+ fn jobs_returns_quire_push_outputs_through_transitive_input() {
+ // `b` depends on `a` which depends on `:quire/push`; `b` reads
+ // `:quire/push` directly even though it's not a direct input.
+ let (_dir, quire) = tmp_quire();
+ let runs = test_runs(&quire);
+ let mut run = runs.create(&test_meta()).expect("create");
+
+ let pipeline = load(
+ r#"(local ci (require :quire.ci))
+(ci.job :a [:quire/push] (fn [_] nil))
+(ci.job :b [:a]
+ (fn [{: sh : jobs}]
+ (let [push (jobs :quire/push)]
+ (sh ["echo" push.sha]))))"#,
+ );
+
+ run.execute(&pipeline, HashMap::new()).expect("execute");
+
+ let outputs = run.outputs("b");
+ assert_eq!(outputs.len(), 1);
+ assert_eq!(outputs[0].stdout, "abc123\n");
+ }
+
+ #[test]
+ fn jobs_errors_on_unknown_name() {
+ let (_dir, quire) = tmp_quire();
+ let runs = test_runs(&quire);
+ let mut run = runs.create(&test_meta()).expect("create");
+
+ let pipeline = load(
+ r#"(local ci (require :quire.ci))
+(ci.job :grab [:quire/push] (fn [{: jobs}] (jobs :nope)))"#,
+ );
+
+ let err = run
+ .execute(&pipeline, HashMap::new())
+ .expect_err("expected failure");
+ match err {
+ Error::JobFailed { job, source } => {
+ assert_eq!(job, "grab");
+ let msg = source.to_string();
+ assert!(
+ msg.contains("not in transitive inputs") && msg.contains("nope"),
+ "expected 'not in transitive inputs' error, got: {msg}"
+ );
+ }
+ other => panic!("expected JobFailed, got: {other:?}"),
+ }
+ }
+
+ #[test]
+ fn jobs_errors_on_non_ancestor_job() {
+ // `peer` exists as a job but isn't an ancestor of `grab`.
+ let (_dir, quire) = tmp_quire();
+ let runs = test_runs(&quire);
+ let mut run = runs.create(&test_meta()).expect("create");
+
+ let pipeline = load(
+ r#"(local ci (require :quire.ci))
+(ci.job :peer [:quire/push] (fn [_] nil))
+(ci.job :grab [:quire/push] (fn [{: jobs}] (jobs :peer)))"#,
+ );
+
+ let err = run
+ .execute(&pipeline, HashMap::new())
+ .expect_err("expected failure");
+ match err {
+ Error::JobFailed { source, .. } => {
+ let msg = source.to_string();
+ assert!(
+ msg.contains("not in transitive inputs") && msg.contains("peer"),
+ "expected non-ancestor error, got: {msg}"
+ );
+ }
+ other => panic!("expected JobFailed, got: {other:?}"),
+ }
+ }
+
+ #[test]
+ fn jobs_errors_on_self_lookup() {
+ let (_dir, quire) = tmp_quire();
+ let runs = test_runs(&quire);
+ let mut run = runs.create(&test_meta()).expect("create");
+
+ let pipeline = load(
+ r#"(local ci (require :quire.ci))
+(ci.job :grab [:quire/push] (fn [{: jobs}] (jobs :grab)))"#,
+ );
+
+ let err = run
+ .execute(&pipeline, HashMap::new())
+ .expect_err("expected failure");
+ match err {
+ Error::JobFailed { source, .. } => {
+ let msg = source.to_string();
+ assert!(
+ msg.contains("cannot read its own outputs"),
+ "expected self-lookup error, got: {msg}"
+ );
+ }
+ other => panic!("expected JobFailed, got: {other:?}"),
+ }
+ }
}