diff --git a/.changeset/vale-rule-engine.md b/.changeset/vale-rule-engine.md index 11f1e297..9079c3ce 100644 --- a/.changeset/vale-rule-engine.md +++ b/.changeset/vale-rule-engine.md @@ -2,16 +2,39 @@ "@taskless/cli": minor --- -Add Vale as a second static-tier rule engine. +Add Vale as a second static-tier rule engine, and rename the agent-facing command. -`check` now dispatches by engine directory and runs ast-grep, Vale, and runtime -rules concurrently, merging their findings into one result set. Vale rules live -in `.taskless/vale/` and execute against the committed `.vale.ini`; an -unavailable Vale reports itself and the other engines still return, while a Vale -that times out or rejects its config fails the check rather than passing as a -clean run. Vale rules are verified from `rule-tests//pass|fail` fixtures -against a generated per-rule config. +`check` now dispatches by engine and runs ast-grep, Vale, and runtime rules +concurrently, merging their findings into one result set. An unavailable Vale +reports itself and the other engines still return. A Vale that times out or +rejects its config fails the check rather than passing as a clean run. -Adds the `engine-selection` knowledge topic — which engine enforces a given -rule, and why — available from `taskless help engine-selection` and exported -through `@taskless/cli/prompts`. +**BREAKING: `taskless help ` is now `taskless agent `.** The +command is named for who reads it. Agents fetching a procedure are not asking +for help, and the old name is gone rather than aliased. + +**BREAKING: topics are addressed by a single token.** `taskless help rule +create` becomes `taskless agent create-sg-rule`; multiple positionals are no +longer joined into a topic key. A topic name is now a literal string an agent +copies rather than a phrase it can reorder. The renames: + +| Was | Now | +| -------------------- | --------------------------------------- | +| `rule create` | `create-sg-rule` / `create-remote-rule` | +| `rule improve` | `improve-rule` | +| `rule delete` | `delete-rule` | +| `rule verify` | `verify-rule` | +| `rule meta` | `rule-meta` | +| `static`, `existing` | `route` | +| `engine-selection` | `route` | + +`route` now applies the engine reasoning itself and names a concrete +`create-*-rule` topic, so `engine-selection`, `static`, and `existing` are +removed rather than renamed. Their criterion is stated once, in `route`. + +**BREAKING for `@taskless/cli/prompts` consumers.** `engine-selection` is no +longer exported. `TOPICS` is now `create-sg-rule`, `create-vale-rule`, and +`create-runtime-rule`, so a consumer that decides an engine can reach the +procedure for each destination. Because the export is a string union, a +consumer passing the removed name dynamically breaks on upgrade rather than at +build time. diff --git a/README.md b/README.md index 26c6748d..56c5dc18 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Starting in v0.7, Taskless ships a **single consolidated skill** (`taskless`) pl | | | delete, check, auth, CI). Fetches the canonical recipe | | | | for the user's intent and follows it. | -Available `taskless help` topics: `rule create`, `rule improve`, `rule delete`, `check`, `auth`, `ci`, `info`, `init`, `update`. Append `--anonymous` for the local-only flow on rule create/improve. +Available `taskless agent` topics: `route`, `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, `create-remote-rule`, `improve-rule`, `delete-rule`, `check`, `auth`, `ci`, `info`, `init`, `update`. Run `taskless agent` with no topic for the index. Append `--anonymous` for the local-only flow on improve. ## CLI diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/.openspec.yaml b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/.openspec.yaml new file mode 100644 index 00000000..b6b2d1f6 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-13 diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/design.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/design.md new file mode 100644 index 00000000..9e37534f --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/design.md @@ -0,0 +1,147 @@ +## Context + +The CLI's `help/*.txt` recipes are the knowledge surface agents read. They are addressed longform (`positionals.join("-")` in `commands/help.ts`), exported in part through `@taskless/cli/prompts` for the platform generator, and cross-referenced from each other by literal command string — ~306 occurrences of `taskless help` across 77 files. + +`add-vale-rule-engine` added `engine-selection`, which decides between `sg`, `vale`, and `runtime`. Two of those three answers have no authoring procedure. That change recorded Vale authoring as an explicit non-goal, which was defensible when nothing chose Vale; the chooser makes it reachable. + +Separately, the scaffolded `.vale.ini` opens an unscoped `[*]`. A whole-project check under it lints build output and, until #100, `.taskless/` itself. + +## Goals / Non-Goals + +**Goals:** + +- Every answer `route` can produce leads to a procedure that exists. +- One fetch from request to destination, returning a command an agent can run verbatim. +- A topic vocabulary that reads as literal tokens rather than paraphrasable phrases. +- A scaffold that lints nothing until someone scopes it deliberately. + +**Non-Goals:** + +- **A `.vale.ini` writer.** The agent authors the section, exactly as it authors `sgconfig.yml` rule entries today. Construction moves to the downstream generator, consistent with `add-vale-rule-engine/design.md:107`. +- Rule _generation_ for Vale via the service. `create-remote-rule` dispatches to the service as it does today; this change gives the local paths destinations and renames the remote one. +- Renaming the `cli-help` capability file. The command renames; the spec keeps its name, so this change does not also move spec files (see D5). +- Restricting Vale's feature set, or deciding build-output exclusion (tracked separately in #101). + +## Decisions + +### D1 — `route` and `engine-selection` merge into one topic + +There is one decision, made once. `route` absorbs the engine reasoning and returns one of the five `create-*-rule` topics. `engine-selection` ceases to exist as a separate topic. + +This reverses the scoping `engine-selection` asserts today — that route decides destination, the topic decides engine, and locally the two compose. The separation is clean on paper and expensive in practice: it costs an agent two fetches and a correct handoff between them to answer one question, and the handoff is where an agent drops context. Worse, the two decisions are not independent in the direction the split assumes; "author this locally" and "which engine can express it" are answered from the same evidence, so splitting them means reading the same signals twice. + +**Where the reasoning goes for consumers outside the CLI.** The platform generator consumes `engine-selection` through `TOPICS`, so merging cannot simply delete what it reads. `route` is expected to be exported in a later change, at which point it carries the criterion to the service directly; it is not exported here because it still contains local mechanics (`taskless detect --json`, on-device authoring) that a Worker cannot run, and untangling those is its own piece of work. + +The criterion therefore lives **once, in `route`'s destination table**, which is where the comparison is actually made. Restating it in each destination would be the drift risk the merge was meant to remove, one level down. + +Each destination instead opens with a short orientation line naming what it is for and what to do if that is wrong — see D9. That is deliberately less than the full criterion: enough for a reader who arrived at the wrong recipe to notice and go back, not a second copy of the test. + +**What this costs the exported surface, and for how long.** A consumer reading only `create-vale-rule` gets its scope ("prose and markup") but not the boundary cases that settle hard calls — prose-about-code, per-document versus cross-document. Sufficient for picking between destinations; not for adjudicating a genuinely ambiguous rule. + +That gap closes when `route` is exported. The service will hold the route prompt, which states when each engine applies, and needs no escalation path of its own — it is the escalation. It can then supply its own runtime prompt for its own agentic flow. + +Worth being clear about why the prompts are exported at all, because it changes what "enough" means: the goal is **consistency between the local and remote paths**, not transferring a capability the service lacks. The service can classify without us. What it should not do is classify _differently_ — a rule routed to `vale` locally and to `sg` server-side is the same request answered two ways, and that is the failure the shared surface exists to prevent. + +_Alternative rejected:_ keep `engine-selection` as a third exported topic that `route` also applies. Two statements of the same criterion, guaranteed to drift, and it preserves the second fetch for exactly the consumer we were trying to simplify. + +_Alternative rejected:_ restate the full criterion in every destination. Five copies of one test, and the first edit to any of them is a divergence nobody notices. + +_Deferred, not rejected:_ export `route`. Its local mechanics need separating from its reasoning first, and doing that inside a change that already renames a command and five topics is how a rename becomes unreviewable. + +### D2 — Verb-noun names, single token, no aliases + +`create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, `create-legacy-rule`, `create-remote-rule`. Not everything needs the noun — `route` stays `route`. + +Hyphenated single tokens are the point rather than a side effect. A multi-word phrase invites an agent to paraphrase or reorder; a hyphenated token reads as a literal string to copy. This is the same reason the resolution stops joining positionals: with one token there is no order to get wrong. + +`static` → `create-sg-rule` also removes a leak. "Static" is a trust tier, and `engine-selection` is explicit that tier and engine are different axes; naming the ast-grep authoring topic after the tier taught the confusion the other topic exists to correct. + +_Alternative rejected:_ `static-sg` / `static-vale`. Preserves the tier leak and does not match what `route` decides. + +### D3 — Break `TOPICS`, no deprecation window + +`TOPICS` becomes `["create-sg-rule", "create-vale-rule", "create-runtime-rule"]`. `engine-selection` leaves the export because it stops existing (D1); the criterion it carried is now stated by the destinations themselves. + +The package is pre-1.0, so this ships **MINOR**. That is what the leading zero means, and it holds for every backwards-incompatible item in this change. An alias would have to be carried by the type union (`PromptTopic`), the `PROMPTS` map, and the disjointness test, and would be dead the moment the generator updates. + +The real exposure is not the rename but the **deploy skew**: the generator is a separate deploy consuming a published package, so it breaks on upgrade rather than at our build time. Mitigated by the changeset, not by code. + +_Alternative rejected:_ export both names for one release. Doubles the exported surface to protect a single known consumer that we control. + +### D4 — Section-less scaffold, and stderr notices as its precondition + +The scaffolded `.vale.ini` carries `StylesPath` and `MinAlertLevel` and no section. Measured: Vale runs clean and reports `{}`. + +This is only safe **with** stderr surfacing, and the two ship together. With no section to copy, the natural first edit is `rules. = YES` at top level, which Vale reports as `W101 '' isn't a core option; Vale is ignoring it` — on stderr, with exit 0 and valid `{}` on stdout. `runVale` reads stderr only on a non-zero exit, so today that diagnostic is discarded and the user gets: rule authored, `verify` passes, `check` silent. That is the exact failure the Vale work exists to eliminate, and shipping the scaffold change alone would reintroduce it one level up. + +_Alternative rejected:_ scaffold `[*.md]`. A user's first rule works immediately, but the default silently decides scope for them, and markdown is a guess about what a repo's prose is. + +### D5 — The `cli-help` capability keeps its name + +The command becomes `agent`; the spec file stays `openspec/specs/cli-help/spec.md`. + +Renaming a capability means moving a spec directory and rewriting every cross-reference to it in the same change that already renames a command and four topics. The capability's subject — the CLI's agent-knowledge surface — is unchanged; only its command name moves. Worth doing later on its own, and worth not doing here. + +## Risks / Trade-offs + +- **Generator breaks on CLI upgrade, not at build time** → `TOPICS` is consumed across a deploy boundary. The changeset must name the rename explicitly, and the generator's update is a coordinated follow-up rather than an assumption. +- **306 mechanical edits invite a missed one** → a stale `taskless help X` in a recipe is invisible until an agent runs it and gets nothing. Mitigated by an assertion that no shipped recipe contains the string `taskless help`, which is cheap and total. +- **Absorbing engine choice into `route` makes `route` longer** → it now carries the reasoning that justified a separate topic. If it grows past being readable in one pass, the split was load-bearing after all and should come back as a fetch. +- **A section-less scaffold means a fresh project's first Vale rule does nothing until scoped** → intended, and the reason `create-vale-rule` must teach the section rather than assume it. The stderr notice is what makes the failure legible instead of silent. +- **Telemetry vocabulary changes** → `cli_help` topic values change wholesale; anything keyed on `static` goes quiet rather than erroring. Worth naming before it is diagnosed as a traffic drop. + +## Migration Plan + +No user data or on-disk state migrates. Existing projects keep whatever `.vale.ini` they have — the scaffold change affects new projects only, and `0004` is unreleased, so no project has the old scaffold in the field. + +The rename is a hard cutover in one PR: recipes cross-reference each other by literal command, so a partial rename produces recipes pointing at commands that do not exist. + +### D6 — The logged-out gate is explained once, by `create-runtime-rule` + +"Remote" describes who generates rather than which engine, so the two are not peers in kind — but the place that matters is the same place `create-runtime-rule` has to speak anyway: the user is logged out. + +`create-runtime-rule` therefore owns the gated story — why executing code requires login, reconciliation, and signing, and how to get there. A logged-out user meets one topic explaining one gate, rather than being handed between a topic about remoteness and one about authentication. + +**What this leaves unresolved, deliberately.** `remote.txt` and `rule-create.txt` today serve a real and different flow: the service generating an _ast-grep_ rule when local authoring cannot. That is escalation, not a destination — `route` is already specified as biased local with the service as last resort — so it survives as a fallback inside `route` rather than as a peer of the four. Whether those two recipes keep their names, merge, or fold into `create-sg-rule`'s failure path is not settled here. + +_Alternative rejected:_ a fifth `create-remote-rule` destination. Puts a non-engine on an engine-shaped list, and splits the logged-out explanation across two topics. + +### D7 — Login state is read early; remote is offered only where it is a choice + +`route` reads login state near the top, before dispatching. It changes which destinations exist, so discovering it late means classifying against a set that may be wrong. + +It does **not** follow that the agent should open by asking "do you want remote generation?". At that point it does not know whether the rule is a two-line `sg` pattern or something local authoring cannot express, and neither does the user — the question costs a turn and cannot be answered well. Remote is offered when it is genuinely a choice: the rule is locally expressible **and** the user is logged in. Not logged in, or not locally expressible, are not choices and are not posed as one. + +This narrows the existing "biased to stay local" requirement rather than reversing it. The bias survives for the case it was written about — local authoring that _works_ is not abandoned for the service — while a logged-in user stops being steered away from a path they have already paid for. + +**No topic delegates to another.** A logged-in runtime request routes to `create-remote-rule` directly; `create-runtime-rule` is the logged-out path. Routing runtime through a topic that then forwards would reintroduce the second fetch D1 exists to remove, and would split the login explanation across two files. + +_Alternative rejected:_ offer remote unconditionally as step one. Reverses the local bias outright, and asks a question before the information exists to answer it. + +_Alternative rejected:_ `create-runtime-rule` checks login and forwards. Two fetches, and the reader meets the gate explanation only on one branch. + +### D8 — `remote` and `rule-create` merge into `create-remote-rule` + +`remote.txt` states the client-side boundary; `rule-create.txt` is the procedure that enriches a description, calls the API, and reports. Split across two topics, an agent fetches one to learn it needs the other. + +This is a content merge, not a rename: both texts have material that survives, and the result has to read as one procedure rather than two concatenated. + +_Alternative rejected:_ keep `remote` as a boundary statement and `create-remote-rule` as the procedure. Preserves the second fetch under new names. + +### D9 — Destinations orient, they do not re-decide + +Each `create-*-rule` recipe opens with a fixed-shape line: what topic the reader is in, what kinds of rule it helps write, and an instruction to revisit the routing decision if that is not what they need. + +Its job is self-correction, not classification. An agent that arrived at the wrong recipe — because it guessed, because a user named a topic directly, or because `route` was wrong — should discover that in the first line rather than after authoring the wrong artifact. Recovery is cheap there and expensive later. + +Keeping it to orientation is what stops it becoming a second criterion. The comparison between engines happens in one place; a destination only has to answer "am I the right place", which needs its own scope and nothing about the others. + +### D10 — `create-runtime-rule` defers to `auth` + +It explains why the runtime tier is gated — executing code requires reconciliation and signing — and points at `auth` for obtaining access, rather than restating the login procedure. + +An extra CLI turn is not a cost worth avoiding when each turn delivers something concrete: `auth` is maintained as the authority on login, and a copy inside a rule-authoring recipe is a copy that goes stale the first time login changes. + +## Open Questions + +- None outstanding. diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/iteration-log.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/iteration-log.md new file mode 100644 index 00000000..cfe1308b --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/iteration-log.md @@ -0,0 +1,319 @@ +# 2b iteration log — executing the authoring recipes + +Task 2b.7: the record of what failed, what changed, and what finally held. The recipes are +the deliverable, and a recipe that reads well to its author while producing the wrong +artifact is exactly what reviewing the prose cannot catch — so they get executed. + +Delete this file when the change is archived. + +## Harness + +- `pnpm --filter @taskless/cli build:dev`, so `__TASKLESS_CLI__` is an absolute path to + `dist-dev/index.js` and the rendered recipe carries a command that runs from anywhere. +- Sandbox is a real `init --no-interactive` scaffold, not a hand-made approximation. +- Each run gets a **fresh, non-forked** subagent handed exactly three things: the rendered + recipe, the sandbox path, and a rule intent in plain words. **No repository access** — + with it, the agent finds `no-simply.yml` and the mixed-engine fixture and copies them, + and the loop tests our fixtures rather than our writing. +- Intents chosen to exercise different extension points. Everything in this repo today is + `existence`, so a recipe drafted from our own examples would teach token blocklists and + nothing else. + +## Round 1 + +| Run | Intent | Extension point | Converged | +| --- | ------ | --------------- | --------- | +| A | flag hedging phrases in docs | `existence` | **yes, first try, zero retries** | +| B | "sign in" is the verb, "login" the noun | `substitution` | **yes, first try, zero retries** | +| C | it is `GitHub`, not `Github`/`github` | ended up `substitution` | **one retry** | + +### Run A — converged, and that is the problem + +The agent produced `no-hedging.yml`, added a scoped `[*.md]` section with +`BasedOnStyles =` and `rules.no-hedging = YES`, wrote both fixture buckets, and got six +findings on `fail/` and zero on `pass/` on the first execution. No `W101` notice, so the +assignment landed inside the section. + +That is the 2b.6 bar met for one extension point. It also means **the step-5 debug ladder +was never exercised**, and the agent's critique is mostly about the rungs that ladder is +missing. Taking a first-try pass as evidence the prose is finished would be reading the +result backwards. + +Findings, triaged. Verified against the source rather than taken on the agent's word: + +| # | Finding | Verdict | +| - | ------- | ------- | +| a | `--json` reports `success: true` and exit 0 on **both** buckets; only `results.length` distinguishes them. The recipe hands you `--json` and describes no field of it | **Real.** An agent checking `$?` or grepping `success` concludes a working rule failed, and starts debugging it | +| b | A whole-project `check` reports nothing from the fixtures, contradicting step 3's walk model and step 6's suggestion | **Real.** `run.ts:137` excludes `.taskless/**` from the whole-project walk (the #100 fix). Fixtures live under `.taskless/`, so they are invisible to a bare `check` — correct behavior, undocumented, and the recipe's own model mispredicts its own suggested command | +| g | Step 4 asserts the CLI rejects a nested fixture directory and requires both buckets — neither is reachable | **Real, and the worst of these.** That validation lives in `verifyValeRule`, which has **zero CLI callers**. The recipe states an invariant nothing enforces, so an author who omits `pass/` sails through step 5 and never learns | +| d | `MinAlertLevel` is absent from the debug ladder, though `MinAlertLevel = error` against a `level: warning` rule produces exactly the silent failure the ladder exists for, with every listed item checking out | **Real** | +| c | `tokens` are regexes; the recipe presents them as a word list and never mentions `raw` or `nonword`, or what happens to a phrase containing regex metacharacters | **Real.** Live for hedging phrases specifically ("maybe?") | +| i | `%s` count is a property of the extension point — one for `existence`, two for `substitution` — stated nowhere, so copying the field table into a substitution rule yields `%!s(MISSING)` | **Real, cheap to fix** | +| f | `StylesPath = .` is never said to resolve relative to the directory holding `.vale.ini`, and the recipe uses two path roots (`.taskless/vale/rules/…` vs `vale/rules/…`) without reconciling them | **Real** | +| h | Three names must agree — filename, `id`, fixture directory — and are explained as if they were one | **Real** | +| e | Step 5 gives a runnable `node … check` while step 6 and See Also give `taskless agent check`; the agent could not tell whether `agent check` is a different subcommand | **Real, and corpus-wide.** Every recipe uses the bare `taskless agent ` form for a fetch and `npx @taskless/cli ` for an invocation. Obvious to us, not to a first-time reader | +| j | Fenced code blocks inside markdown: does a rule fire inside them? Undefined here, and live for a docs rule | **Real, needs measuring** before writing an answer | +| k | Step 1 sends the reader to `docs.vale.sh` with no offline fallback for the eight extension points it declines to describe | Accepted. Embedding the full Vale reference is out of scope | + +Nothing in this list is a defect in the agent. Every one is a defect in the prose, which +is what 2b.6 says to treat them as. + +### Run B — converged, and independently confirms A's three worst findings + +The agent produced `login-as-verb.yml` using regex `swap` keys, scoped it to `[*.md]`, +wrote both buckets, and got four findings on `fail/` and zero on `pass/` first try. It +also volunteered an extra probe — appending a sentence to `pass/` to test a false positive +it suspected in the `to login` key — and re-ran the bucket. That is the behavior the +recipe wants and does not currently ask for. + +Two runs, no shared context, and both independently reported **a**, **g**, and the +fetch-versus-invoke ambiguity. Those are not taste. + +New findings on top of run A: + +| # | Finding | Verdict | +| - | ------- | ------- | +| l | `swap` keys are **Go RE2** regexes — `(?:…)` works, lookahead and lookbehind do not. The only example is two literal strings, so the agent guessed. A wrong guess fails as a **silent non-match**, the exact failure the recipe spends a section warning about | **Real, and the most valuable finding of the round.** Generalizes A's finding (c): both `tokens` and `swap` keys are patterns presented as literals | +| m | `%s` **ordering** in a substitution message is asserted only by an example that reads correctly under either interpretation. First `%s` is the swap value, second is the matched text — confirmed only by running the tool | **Real.** Pairs with A's finding (i) on `%s` count | +| n | Overlapping `swap` keys have undefined precedence. `can login` won over `login with`; `to logout` over `logout of`. First-alternative-wins is fine, but an author enumerating alternatives cannot tell how many findings a sentence yields, or which message | **Real** | +| o | Step 1 says "eleven in total" and the table lists eight; the other three are named in the following paragraph, behind a URL. Eleven only via arithmetic across two paragraphs | **Real, trivial.** List all eleven | +| p | `check ` lints everything under the path against the **whole config** — it is not scoped to the rule under test. With a second rule whose glob matches, the fail bucket reports both, and the recipe gives no vocabulary for that. Step 3 hints at the real isolation mechanism and never connects it to step 5 | **Real.** "Run the rule against each bucket" overstates what the command does | +| q | `BasedOnStyles =` with an empty right-hand side: the recipe insists on it without saying whether an empty value is valid INI to Vale | Minor. Measured: valid, no warning. Worth one clause | + +**One finding rejected as a harness artifact, recorded so nobody "fixes" it:** run B +objected that step 5 hardcodes an absolute path into someone's checkout +(`node /Users/…/dist-dev/index.js check …`). That is `build:dev` doing its job — it +rewrites `npx @taskless/cli` to an absolute path precisely so the harness command runs +from any directory. The shipped recipe says `npx @taskless/cli`. No change. + +Run B's second half of that objection is **not** an artifact and stands: the document +uses `npx @taskless/cli ` for invocations and `taskless agent ` for fetches +without ever explaining the relationship. That is finding **e**. + +### Run C — the only run that failed, and it found the worst defect + +The one intent that did not converge first try, and the one that earned the round. It +also declined the extension point the recipe recommended, which is why it succeeded. + +**C1 — the field table was factually wrong, and it fails silently.** The table said +"`%s` interpolates the match". For `substitution` that is false. Measured directly: + +``` +swap: {Github: GitHub}, message: "Use GitHub not %s", document text: "Github" +→ "Use GitHub not GitHub" (matchedText: "Github") +``` + +A single `%s` interpolates the **replacement**. The correct form takes two, filling +`(replacement, match)`. The recipe's own example block had it right while the normative +table one paragraph above said the opposite — so an author who reads the table rather than +copy-pasting the example ships a nonsense message. This is the worst defect found in the +round because it **passes every check the recipe tells you to run**: the rule fires, both +fixtures behave, the exit code is right, and only a human reading the message notices. + +**C2 — the extension-point table pointed at a check that cannot do the job.** The row read +"how something is capitalized (headings, **product names**)" → `capitalization`, and a +later line endorsed a literal `match` "for a product name". Measured, `match: GitHub`: + +``` +findings: 2 + 'Working with Github should be GitHub' matched: 'Working with Github' + 'We host on Github and it is fine. should be GitHub' matched: 'We host on Github and it is fine.' +``` + +`capitalization` applies `match` to a whole **scope** — a heading, a sentence — so it +flags entire sentences and cannot express "this word, wherever it appears". Product-name +spelling is a `substitution`. The recipe named the one use case in that row the check +cannot serve, then reinforced it two lines later. The agent only avoided the trap because +it already knew the check was scope-shaped. + +**C3 — the `pass/` bucket framing under-tests.** Step 4 justified the pass bucket as "the +half that catches an over-broad pattern" but described it as prose that is *correct*. +Correct prose proves nothing; the rule was never going to fire on it. The agent had to +build a throwaway probe file — URLs, code spans, `GITHUB_TOKEN` — because the fixture +model had no place for near-misses. + +**C4 — the debug ladder is one-sided.** Every rung addresses `fail/` reporting nothing. +Nothing addresses `pass/` firing, which is the over-broad case the pass bucket exists for. + +**One of run C's recommendations was rejected on measurement.** It proposed warning that +`ignorecase: true` would make the key `Github` also flag the correct `GitHub`. Measured: + +``` +ignorecase: true, swap {Github: GitHub}, text "Wrong github and Github here. Correct GitHub here." +→ 2 findings: 'github', 'Github' ('GitHub' NOT flagged) +``` + +Vale skips a match that already equals its replacement, so `ignorecase: false` is not +needed to protect the correct spelling. Writing that warning in would have taught +something false. The recipe states the measured behavior instead. Worth noting as the +round's reminder that an agent's diagnosis is a lead, not a finding. + +## Round 1 outcome + +Two of three converged first try; the third took one retry and produced the two findings +that mattered most. Against 2b.6 — "an intent the recipe never names, first try, +uncorrected" — the recipe **passed for `existence` and `substitution` and failed for the +product-name case**, which is the honest reading. + +Sixteen findings, fifteen accepted, one rejected on measurement. Every accepted finding is +a defect in the prose, which is what 2b.6 says to treat them as. + +## Revision applied + +`create-vale-rule.txt` was rewritten against all three reports (200 → 286 lines). What +changed, beyond the wording items: + +- **All eleven extension points** are in the table, with `capitalization` explicitly + scoped to "a whole heading or sentence" and product names routed to `substitution`. + The trap parenthetical and the literal-`match` endorsement are gone. +- **A `%s` table**, per extension point, with the measured substitution behavior quoted. +- **A new step 3, "tokens and swap keys are patterns, not literals"**: Go RE2, no + lookaround, implicit word boundaries, live metacharacters, first-wins on overlap, the + measured `ignorecase` behavior, `raw`/`nonword`, and the markdown scoping that spares + URLs and code spans. This one step carries findings c, l, n and C1's neighbours. +- **Step 6 says to read `results[].ruleId`** and states plainly that `success` and the + exit code answer a different question — with the note that exit 1 on `fail/` is correct + for a `level: error` rule and exit 0 is correct for a `warning` one. +- **`check ` is described as not scoped to the rule under test.** +- **`MinAlertLevel` joins the debug ladder**, and the ladder gains a `pass/`-fires branch. +- **The `pass/` bucket is now specified as near-misses**, not correct prose. +- **The three names that must agree** are stated once, together, with the fact that a + mismatch is silent. +- **Claims of CLI enforcement are removed.** "Nothing checks that you wrote all three" is + the honest statement of today's behavior, and step 7 says a whole-project `check` will + not report the fixtures because `.taskless/` is excluded from the walk. + +## Round 2 + +Re-run against the revised recipe with fresh agents and intents round 1 never used. + +| Run | Intent | Extension point | Converged | +| --- | ------ | --------------- | --------- | +| D | flag "click here" / "read more" as link text | `existence` + `scope: link` | **yes, first try, zero retries** | +| E | headings in sentence case, with exceptions | `capitalization` (its actual use case) | **yes, first try, zero retries** | + +**2b.6 is met.** Three extension points, five runs, and the only failure in the set was +round 1's product-name case — whose cause was corrected and whose check (`capitalization`) +now converges first try on the use case it can actually serve. Both round-2 agents +independently reported that step 6's "read `results`, not `success`" saved them from +misreading a clean `fail/` run, which is the round-1 fix working. + +Round 2 still found nine gaps, three of them defects in prose written *during* round 1's +revision. Measured before acting on them, as before. + +| # | Finding | Verdict | +| - | ------- | ------- | +| r | The field table lists five common fields under "Every rule carries" and **omits every field a rule actually needs** — `tokens`, `swap`, `match`, `exceptions`. Run E's rule depended entirely on `exceptions`, which appeared only as an undocumented line in an example | **Real, and the round's biggest gap.** Fixed with a per-extension-point field table | +| s | `scope` was documented as "e.g. `heading`, `paragraph`" — an example, not a list — for the field that decides where a rule looks. Run D's rule rested on `scope: link` existing; it guessed | **Real.** All sixteen markdown scopes now listed | +| t | `match: $sentence` was never defined. "The difference between 'first letter capitalized, everything else lowercase' and 'proper nouns permitted' decides whether `Getting started with Kubernetes` fires" | **Real, and the answer matters.** Measured below | +| u | Is `[click here](url)` prose? The recipe answered the inverse — what Vale *excludes* — and never said what link text is | **Real.** Measured below | +| v | Word boundaries were stated for a single-word key only; multi-word behaviour left to assumption | **Real.** Measured: whole-phrase. `click here` does not fire inside `Clicking here` | +| w | The `%%s` table covers 2 of 11 extension points while being billed as the authority on "the one mistake that passes every check" | **Real.** `capitalization` added (measured); the other eight now carry an explicit "don't guess, read it back off the finding" | +| x | Step 3 (RE2, boundaries, `ignorecase`) is irrelevant to `capitalization`/`occurrence`/`metric` and had no skip marker, so run E read all of it looking for `exceptions` semantics | **Real.** Skip line added | +| y | The pass-fixture advice was written for token rules only — "the word inside a longer word" is not a near-miss for a whole-scope check | **Real.** Now branches by rule shape, including "the same phrase *outside* the scope", which is the only thing that proves a `scope` works | +| z | "a fixture in a nested subdirectory is linted but **never counted against either bucket**" references counting machinery the reader has never been shown — a leftover from when `verifyValeRule` was assumed reachable | **Real, my error.** Rewritten to describe what `check` actually does | +| aa | The `BasedOnStyles =` rationale — "it stops a later edit from switching a whole style on by accident" — is not a real mechanism | **Real, my error.** Run D is right: an empty assignment prevents nothing. Replaced with the honest reason (it makes intent readable without knowing the default) | + +Run D repeated run B's objection to the absolute path in step 5. Same answer: `build:dev` +artifact, already recorded as rejected. + +### Measured for round 2 + +**`$sentence` is stricter than "sentence case".** First word capitalized, everything else +lowercase — proper nouns included, unless listed in `exceptions`: + +| Heading | Result | +| ------- | ------ | +| `Getting started with the API` | quiet | +| `Getting started with APIs` | quiet — an exception covers its plural | +| `Taskless and the API` | quiet — an exception may lead the scope | +| `Getting started with Kubernetes` | **fires** — a proper noun not listed | +| `getting started lowercase` | **fires** — the first word must be capitalized | + +So `exceptions` is load-bearing: every proper noun the docs use must be listed or the rule +flags correct headings. That is now in the recipe as a table, and it answers the question +run E said it could not resolve (`APIs`, the plural of an exception, is covered). + +**Link text is prose; the URL is not.** With no `scope`, a `click here` token fired on both +`[click here](https://example.com/x)` and the same phrase in an ordinary sentence. With +`scope: link`, only the link. Both facts are now stated. + +## Round 2 revision applied + +`create-vale-rule.txt`, 286 → 356 lines. Per-extension-point field table; full `scope` +list; `$sentence` semantics as a measured table; link-text scoping; whole-phrase +boundaries; `capitalization` in the `%%s` table with honest guidance for the rest; a skip +marker on step 3; pass-fixture advice branched by rule shape; and the two sentences of my +own that run D correctly called out as describing machinery and mechanisms that do not +exist. + +## Round 3 — worked examples + +Both rounds converged, but every run had to *invent* its rule shape from three examples, +and the near-misses each one flagged as "I guessed and happened to be right" were the +recurring theme. The recipe explained the mechanics well and showed almost nothing. + +Added a **Worked rules** section: nine rules, one per extension point the recipe covers, +each paired with the near-miss that fails and why. 356 → 545 lines. + +Every example is measured, and — the part that matters — the **YAML blocks were extracted +from the rendered recipe and executed verbatim**, so what ships is what was tested rather +than something adjacent to it: + +| # | Extension point | Result | +| - | --------------- | ------ | +| 1 | `existence` (hedging) | fires ×3 — `"Avoid hedging: 'We think'"` | +| 2 | `substitution` (login → sign in) | fires ×2 — `"Use 'sign in to' instead of 'login to'"` | +| 3 | `substitution` (GitHub) | fires ×2 on `Github`/`github`, **not** on the correct `GitHub` | +| 4 | `capitalization` (sentence case) | fires ×1 on a Title Case heading | +| 5 | `existence` + `scope: link` | fires ×1 — the link only, not the same phrase in prose | +| 6 | `occurrence` (max 1 per paragraph) | fires ×1 | +| 7 | `repetition` (doubled word) | fires ×1 — `"'is' is repeated"` | +| 8 | `consistency` (-ize/-ise) | fires ×1 | +| 9 | `conditional` (define the acronym) | fires ×1 on the undefined `XYZ`, not the defined `API` | + +**One near-miss was found by writing the examples, not by an agent.** An unquoted +`[^\s]+` in a `repetition` rule's `tokens` matches nothing — zero findings, no error, no +diagnostic. It is a YAML escaping failure that presents exactly like a Vale scoping +failure, and it is now example 7's "goes wrong". Worth noting that this is the failure +mode the recipe warns about most, arriving through a layer the recipe had not covered. + +Three other "goes wrong" entries are behaviors an agent would not guess: `consistency` +enforces internal consistency rather than picking a winner (use `substitution` for house +style); `occurrence` counts per `scope`, so omitting it caps the document; and +`conditional`'s `first`/`second` invert the rule if swapped. + +## Round 1 planned revision (applied — kept for the record) + +## Planned revision (applied — kept for the record) + +Batched until B and C report, so the recipe is revised once against all three rather than +three times against one. + +1. **Step 5 — say how to read the output.** `results` non-empty on `fail/`, empty on + `pass/`; `success` and the exit code are the same in both and are not the signal. +2. **Step 5/6 — fixtures are excluded from a whole-project `check`.** Say so, and stop + implying a bare `check` is a way to see the rule fire on its own fixtures. +3. **Step 4 — stop asserting validation nothing performs.** Either the invariants become + reachable (wire `rule verify ` to dispatch by engine — see `resume.md`, needs the + user's call) or the recipe states them as author discipline rather than as something + the CLI enforces. Do not leave the current wording; it is false. +4. **Step 5 — add `MinAlertLevel` to the debug ladder**, above the pattern. +5. **Step 2 — patterns, not literals.** Both `tokens` and `swap` keys are **Go RE2** + regexes: `(?:…)` works, lookahead and lookbehind do not, and metacharacters in a real + phrase are live. Name `raw` and `nonword`. State whether word boundaries are applied, + per extension point. Say that overlapping alternatives resolve first-wins. This one + item now carries findings c, l and n, and is the round's biggest single change. +6. **Step 2 — `%s` count *and* order follow the extension point.** One for `existence`, + two for `substitution`, and in a substitution the first is the swap value, the second + the matched text. +6a. **Step 1 — list all eleven extension points in the table**, rather than eight plus + three in prose behind a URL. +6b. **Step 5 — `check ` is not scoped to the rule under test.** It lints everything + under the path against the whole config. Say so, rather than "run the rule against + each bucket". +7. **Step 3 — `StylesPath` resolves relative to the config file**, and use one path root. +8. **Step 2/3 — the three names that must agree**, said once as one fact. +9. **Measure the fenced-code-block question**, then answer it in a line. + +Item 3 is the one that needs a decision rather than a wording pass. diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/proposal.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/proposal.md new file mode 100644 index 00000000..f64115ba --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/proposal.md @@ -0,0 +1,51 @@ +## Why + +`add-vale-rule-engine` shipped a chooser without a destination. `engine-selection` teaches an agent to decide a rule belongs to `vale`, and then there is nowhere to go: `static.txt` is 76 lines of ast-grep authoring, and its only mention of Vale is a See Also line telling the reader to _confirm_ `sg` was right. An agent that follows the procedure correctly and lands on `vale` dead-ends, and so does one that lands on `runtime`. That was recorded as a deliberate non-goal at the time (`add-vale-rule-engine/design.md:16` excludes "generating Vale rules and authoring the committed `.vale.ini`"), but the chooser is what makes the gap reachable, and it is now shipped. + +The same work exposed that the surface an agent reads is shaped for a human. `taskless help` names the command after a human's reason for typing it; agents are not asking for help, they are fetching a procedure. And the surface is addressed longform — `taskless help rule create` resolves by joining positionals — so an agent must know both the words and their order. Hyphenated single tokens read as literal strings an agent copies rather than a phrase it might paraphrase, which is the failure this surface cannot afford. + +## What Changes + +- **BREAKING** — `taskless help ` becomes `taskless agent `. The command is named for who reads it. +- **BREAKING** — topic addressing flattens to a single token. `taskless help rule create` becomes `taskless agent create-rule`; the `positionals.join("-")` resolution is removed rather than generalized. +- **BREAKING** — authoring topics are renamed to verb-noun, matching what `route` decides: + - `static` → `create-sg-rule` + - `existing` → `create-legacy-rule` + - new `create-vale-rule` + - new `create-runtime-rule` + - `remote` + `rule-create` merge into `create-remote-rule` +- **BREAKING** — `TOPICS` in `@taskless/cli/prompts` renames with them. `["static", "engine-selection"]` becomes `["create-sg-rule", "create-vale-rule", "create-runtime-rule"]`. No alias is kept. Pre-1.0, a backwards-incompatible change is a **MINOR** bump. +- `route` becomes the single front door, returning a concrete next command rather than a category. Its decision set is the five `create-*-rule` topics. It reads login state early, because that changes which destinations exist, and offers service generation **only where it is a real choice** — when the rule is locally expressible and the user is logged in. A user who is not logged in, or a rule local authoring cannot express, is not being offered anything. +- **BREAKING** — `engine-selection` merges into `route` and stops existing as a topic. Its criterion distributes: `route` applies it to dispatch, and each `create-*-rule` recipe states the evidence that makes its own engine right. That is what keeps it exportable — a consumer outside the CLI has no `route` step and cannot run `taskless detect --json`, so a chooser topic was unusable to it anyway. +- `create-runtime-rule` becomes the logged-**out** path: what a runtime rule is, why executing code requires login, reconciliation, and signing, and how to get there. A logged-in runtime request goes straight to `create-remote-rule` from `route`, so no topic delegates to another. +- `create-vale-rule` covers what no topic covers today: authoring a Vale style file under `vale/rules/`, scoping it with a `.vale.ini` section, and writing `pass/`/`fail` fixtures. Consistent with `create-sg-rule`, the agent writes these files; no CLI writer is introduced. +- The scaffolded `.vale.ini` ships **no section**, so a fresh project lints nothing until a user scopes something deliberately. `create-vale-rule` teaches writing that first section. +- Vale's stderr diagnostics on a successful run surface as notices. This is required by the change above, not incidental: with no section to copy, the likely first mistake is a rule assignment at top level, which Vale reports as `W101 ... is ignoring it` on stderr and which today is discarded — reproducing the silent-disable class the Vale work exists to eliminate. + +## Capabilities + +### New Capabilities + +- `cli-agent-authoring`: the four `create-*-rule` procedures — what each engine's authored artifacts are, where they live, and what makes one complete. Covers the Vale authoring path that has no home today. + +### Modified Capabilities + +- `cli-help`: the command renames to `agent` and topic addressing flattens to a single token. Longform resolution is removed. +- `cli-rule-routing`: `route` dispatches to a concrete `create-*-rule` topic rather than a category, absorbing the engine decision it previously deferred to `engine-selection`. +- `cli-knowledge-prompts`: `TOPICS` renames, gains the Vale and runtime authoring topics, and loses `engine-selection`; pre-1.0 breaking changes are restated as MINOR. +- `cli-vale-rule-engine`: the scaffolded config carries no section, and Vale's stderr diagnostics on a zero-exit run become notices. + +## Impact + +- **`packages/cli/src/commands/help.ts`** — renamed, positional-join resolution removed. +- **`packages/cli/src/help/*.txt`** — two renames, two new files, and ~306 cross-references across 77 files that name `taskless help`. +- **`packages/cli/src/prompts/index.ts`** — `TOPICS`/`INTERNAL_TOPICS` membership and the `PromptTopic` union. +- **`@taskless/cli/prompts`** — published, typed export. The platform generator consumes `TOPICS` and deploys separately from the CLI, so it breaks on upgrade rather than at build time. Flagged in the changeset. +- **`packages/cli/src/filesystem/migrations/0004-vale-engine.ts`** — `VALE_CONFIG_CONTENT` drops its `[*]` section. +- **`packages/cli/src/rules/vale/run.ts`** — stderr captured on a zero-exit run and returned as a notice. +- **`skills/taskless/SKILL.md`** — the one skill naming `taskless help`. +- **Telemetry** — `cli_help` events carry a `topic` whose vocabulary changes; dashboards keyed on `static` go quiet. + +## Delivery Shape + +**Single PR**, stacked on #100. The rename is mechanical but total: a half-renamed command surface is not a shippable intermediate state, and splitting the topic renames from the command rename would leave cross-references pointing at commands that do not exist yet. Reviewable because the diff is overwhelmingly one substitution repeated, with four files of genuinely new prose. diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/resume.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/resume.md new file mode 100644 index 00000000..e3920023 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/resume.md @@ -0,0 +1,190 @@ +# Resume notes — `agent-command-and-vale-authoring` + +Handoff for picking this up after a context reset, or on another machine. Not part of +the OpenSpec artifact set; delete it when the change is archived. + +## Where you are + +- **PR #102** (draft), branch `openspec/agent-command-and-vale-authoring`, stacked on **#100**. +- Stack below you, all green and already reviewed: **#71 → #93 → #94 → #95 → #100**. + The user drives the merge-down; do not merge anything without being asked. +- **Groups 1, 2, 5.1 and 5.2 are done** (`093aca1`, `b0a0ee0`). 572 tests pass, typecheck + and lint clean. `tasks.md` is the authority — read it first and trust its checkboxes + over this file. +- Work happens in the worktree at `worktrees/impl-102`, not the main checkout. On a fresh + machine: `git worktree add worktrees/impl-102 openspec/agent-command-and-vale-authoring` + then **`pnpm install` inside it** — a worktree gets its own empty `node_modules`, and + skipping the install breaks `git commit` (lint-staged) and every `pnpm` script. + +## Read these before doing anything + +1. `tasks.md` — the plan, including group **2b** (the recipe test harness) +2. `design.md` — decisions D1–D10, each with its rejected alternatives. No open questions. +3. `specs/*/spec.md` — what the recipes and the scaffold must do +4. `proposal.md` — the why, and the delivery shape (single PR, stacked on #100) + +## Environment traps that will cost you an hour each + +- **`NODE_OPTIONS` was broken in the previous session's shell** — a `--require` preload + pointing at a deleted temp file, so every `node`, `pnpm`, and `git commit` died with + `MODULE_NOT_FOUND`. Every command in that session was prefixed with + `NODE_OPTIONS="--max-old-space-size=4096"`. **On a fresh machine, check whether you + still need this** (`echo $NODE_OPTIONS`) rather than cargo-culting it. +- **Run `pnpm --filter @taskless/cli build` before `pnpm --filter @taskless/cli test`.** + Many suites spawn the built CLI. A stale `dist/` produces failures that read exactly + like real regressions. +- **`commit.gpgsign` must be true locally.** An earlier restack silently stripped + signatures from 34 commits because `git rebase` does not re-sign without it, and CI does + not catch unsigned commits. Audit with `git log --format='%G?'` after any rebase. +- **A recipe containing a literal `%` must escape it as `%%`.** Recipes render through + sprintf-js named args, so a bare `%s` in prose (Vale's `message:` examples are full of + them) fails at render with "mixing positional and named placeholders is not supported". + It is caught by rendering the topic, not by the build or by typecheck. +- **`zsh` mangles `perl -0pi -e` one-liners containing `@`.** Use `python3 - <<'PY'` for + multi-file text surgery; two attempts were lost to quoting before switching. + +## What group 2 actually produced + +Topic map after the rename, so you do not have to reconstruct it from the diff: + +| Before | After | +| --------------------------- | ------------------------- | +| `static.txt` | `create-sg-rule.txt` | +| `existing.txt` | `create-legacy-rule.txt` | +| `remote.txt` + `rule-create.txt` | `create-remote-rule.txt` (content merge) | +| `engine-selection.txt` | merged into `route.txt`, deleted | +| `rule-create.anonymous.txt` | **deleted** (see below) | +| `rule-improve*.txt` | `improve-rule*.txt` | +| `rule-delete.txt` | `delete-rule.txt` | +| `rule-verify.txt` | `verify-rule.txt` | +| — | `create-vale-rule.txt` (new) | +| — | `create-runtime-rule.txt` (new) | +| `rule.txt`, `rule-meta.txt` | unchanged names | + +**The one deviation from the task text**: `rule-create.anonymous.txt` was deleted rather +than renamed to `create-remote-rule.anonymous.txt`. It duplicated `static.txt` outright, +and "the local-only variant of the remote recipe" is the contradiction `route` exists to +resolve. Its unique material (upstream-schema pointer, optional fields, the per-layer +verify error table) moved into `create-sg-rule.txt`, and `rule create --anonymous` now +points at `taskless agent create-sg-rule`. Recorded in `tasks.md` 2.3. + +**5.1 and 5.2 were pulled forward**, out of group order and deliberately: 2b tests +`create-vale-rule` against a scaffolded project, and the recipe's central claim is that +the scaffold ships section-less. Testing against a scaffold that still wrote `[*]` would +have exercised a recipe nobody will receive. + +## Next up: finish 2b (the harness), then 5.3/5.4, then group 3 + +### 2b is done except its control run — read `iteration-log.md` for the evidence + +Five sandboxed runs across two rounds, three extension points, twenty-five findings. +**2b.1–2b.7 are complete and 2b.6 is met**: an agent given an intent the recipe never +names now produces a working rule first try, uncorrected, for `existence`, +`substitution` and `capitalization`. `create-vale-rule` went 200 → 356 lines across two +revisions. + +**Only 2b.8 remains** — run the same harness over `create-sg-rule` as a control. A failure +there means the harness is wrong rather than the recipe. The procedure is below and the +scratchpad sandboxes are machine-local, so re-create them. + +**One task is checked off with a caveat you should read: 2b.4.** "`verify` passes" cannot +be satisfied — see "A real gap" below. It is the one open decision in this group. Their sandboxes are under +the scratchpad at `vale-harness/sandbox-{a,b,c}`, each a real `init` scaffold, alongside +`create-vale-rule.rendered.txt` (the dev-build render they were given). **That scratchpad +is machine-local — on another machine, re-run the harness from scratch rather than looking +for it.** + +To re-run it: + +1. `pnpm --filter @taskless/cli build:dev` — `TASKLESS_BUILD_TARGET=dev` bakes + `__TASKLESS_CLI__` as an **absolute** path to `dist-dev/index.js`, so the rendered + recipe carries a command that runs from any directory. Confirmed working: the render's + step-5 commands come out as `node /abs/path/dist-dev/index.js check …`. +2. Scaffold a throwaway project with that binary (`init --no-interactive -d `), + so the sandbox is a real scaffold and not a hand-made approximation. +3. Render the recipe to a file (`agent create-vale-rule > …rendered.txt`). +4. Hand a **fresh, non-forked** subagent only: the rendered recipe path, the sandbox path, + and a rule intent in plain words. It must **not** have repository access — with it, it + finds `no-simply.yml` and the mixed-engine fixture and copies them, and the loop tests + our fixtures rather than our writing. Ask explicitly for a blunt critique of the prose; + that is the deliverable, not the rule. +5. The three intents used, chosen to exercise different extension points (everything in + this repo today is `existence`, so a recipe drafted from our own examples teaches token + blocklists and nothing else): hedging phrases (`existence`), "sign in" vs "login" + (`substitution`), and GitHub's capitalization (`capitalization`, literal-match form — + the variant the recipe covers in one line). +6. Every failure is a defect in the prose. Fix the recipe, re-run with a fresh agent. + Converged when an agent produces a rule that fires on `fail/` and stays quiet on + `pass/`, first try, uncorrected. Keep the iteration log (2b.7) — it is the only part a + reviewer can check without rerunning the loop. +7. 2b.8: run the same harness over `create-sg-rule` as a control. A failure there means + the harness is wrong rather than the recipe. + +### Facts measured this session — do not re-derive + +- Vale field reference for the three taught extension points is in the recipe and came + from `https://docs.vale.sh/llms-full.txt`. **`https://docs.vale.sh/styles` is fine (200)** + — the earlier note that it 404s was wrong; it needs `curl -L`. +- `StylesPath = .` makes `rules/` the StyleName, so `vale/rules/no-simply.yml` is the + check `rules.no-simply`. `StylesPath = rules` resolves nothing. +- **`BasedOnStyles =` is not required for a rule to fire, and omitting it added no noise** + in the bundled Vale version. The recipe still tells authors to write it, on the honest + grounds that it is explicit and matches what `verify` generates — not on the claim that + omitting it produces spurious findings, which measurement did not support. +- A rule assignment outside any section: `W101 '' isn't a core option; Vale is + ignoring it` on **stderr**, exit 0, valid `{}` on stdout. With 5.2 this now surfaces as + `Notice: Vale reported while running: …` and the check still exits 0. Verified end to + end against the built CLI. + +### A real gap found, not yet decided + +**`verifyValeRule` / `verifyValeRules` have no CLI caller.** They are exported from +`src/rules/vale/verify.ts` and exercised only by `test/vale-verify.test.ts`; +`taskless rule verify ` routes to `src/rules/verify.ts`, which is ast-grep only. So +there is currently **no way for an agent to verify a Vale rule from the CLI**. + +`create-vale-rule` works around this by validating with `check` over each fixture bucket, +which does work today and is what the harness exercises. But it means task **2b.4's** +"`verify` passes" cannot be satisfied as written, and it is the same class of dead end +this whole change exists to remove — a capability that exists but is unreachable. + +Wiring it looks small: dispatch `rule verify ` by which engine owns the id +(`.taskless/vale/rules/.yml` vs `.taskless/sg/rules/.yml`), then call +`verifyValeRule`. **Raise this with the user before doing it** — it is scope not in +`tasks.md`, and the alternative (a follow-up issue, like #99 and #101) is defensible. + +## Group 3 note, when you get there + +`rule.txt` documents a table of multi-token forms (`taskless help rule create`, +`… rule meta`). After group 1 these are **actively broken**, not merely stale — they hit +the "Too many arguments" path. Lead group 3 with it. + +Group 2 deliberately left the wider cross-reference sweep alone (~306 `taskless help` +occurrences across 77 files). It did update See Also blocks in the six recipes it +rewrote, so 3.1 is the remaining files plus `skills/taskless/SKILL.md`, both READMEs, and +the TS sources. Leave `CHANGELOG.md` alone. + +## Decisions you should not silently revisit + +All are argued in `design.md` with rejected alternatives. The two most likely to be +re-litigated by accident: + +- **D1** — `route` and `engine-selection` merge; the engine criterion is stated **once**, + in `route`'s destination table. Destinations carry a short orientation line (D9), never + a second copy of the criterion. `test/help-extensions.test.ts` now guards both halves. +- **D4** — the section-less scaffold ships **paired** with surfacing Vale's stderr on a + zero-exit run. Shipping the scaffold alone reintroduces the silent-disable failure the + whole Vale stack exists to eliminate. They are one requirement, not two. Both are now in. + +Also settled: pre-1.0, every backwards-incompatible change here is a **MINOR** bump — +never MAJOR. The telemetry event stays `cli_help` (agent-call volume stays visible under +the existing event). + +## Outside this PR + +- **#99** — migrate subprocess handling to execa (inventory and sequencing already written up) +- **#101** — whether a whole-project Vale check should skip build output; `.taskless/` is + already excluded as of #100 +- Stale worktrees under `worktrees/`; `git worktree list` to review +- `openspec validate --all --strict` fails on `spec/cli-rules` and `spec/cli-update-engine` + on `main` already — pre-existing, unrelated, do not chase it diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-agent-authoring/spec.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-agent-authoring/spec.md new file mode 100644 index 00000000..1b6340f0 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-agent-authoring/spec.md @@ -0,0 +1,100 @@ +## ADDED Requirements + +### Requirement: Every engine a rule can be routed to has an authoring recipe + +The CLI SHALL provide an authoring recipe for each engine `route` can name: `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule`, alongside `create-legacy-rule` for a linter the repository already uses. + +A decision procedure that can produce an answer with no destination is incomplete. Engine selection can conclude `vale` or `runtime`, and before this change neither had a procedure, so an agent that reasoned correctly arrived nowhere. + +#### Scenario: Each engine choice reaches a procedure + +- **WHEN** engine selection concludes `sg`, `vale`, or `runtime` +- **THEN** a recipe exists that authors a rule for that engine + +#### Scenario: A legacy destination exists for repositories with their own linter + +- **WHEN** the repository already runs a linter that can express the rule +- **THEN** `create-legacy-rule` SHALL author it in that tool's own dialect + +### Requirement: The Vale authoring recipe covers rule, scope, and fixtures + +The `create-vale-rule` recipe SHALL instruct the agent to produce three artifacts, and SHALL state that a rule is incomplete without all three: + +1. A Vale style file under `.taskless/vale/rules/.yml`. +2. A section in the committed `.taskless/vale/.vale.ini` scoping which files the rule applies to, enabling it as `rules. = YES`. +3. `pass/` and `fail/` fixture documents under `.taskless/vale/rule-tests//`. + +The recipe SHALL state that the scaffolded config carries no section, so the first rule authored in a project also authors the first scope. + +#### Scenario: Authoring produces all three artifacts + +- **WHEN** the agent follows `create-vale-rule` +- **THEN** it writes the style file, a scoping section enabling the rule, and both fixture buckets + +#### Scenario: The recipe teaches the first section + +- **WHEN** a project's `.vale.ini` has no section yet +- **THEN** the recipe SHALL direct the agent to add one scoped to the files the rule is about, rather than assuming a section exists + +#### Scenario: An unscoped rule is not silently accepted + +- **WHEN** the agent enables a rule without placing it inside a section +- **THEN** the recipe SHALL identify this as incomplete, because Vale ignores a rule assignment outside a section + +### Requirement: Authoring recipes write files rather than invoking a writer + +The `create-*-rule` recipes SHALL instruct the agent to write the rule, its configuration, and its fixtures directly. The CLI SHALL NOT provide a command that generates a Vale style file or edits `.vale.ini` on the agent's behalf. + +This matches how ast-grep rules are authored today: the agent writes the rule and its config entry, and construction belongs to the downstream generator rather than to the CLI. + +#### Scenario: No CLI writer for Vale configuration + +- **WHEN** an agent authors a Vale rule +- **THEN** it edits `.vale.ini` itself +- **AND** the CLI SHALL NOT offer a subcommand that performs that edit + +### Requirement: The runtime authoring recipe is the logged-out path + +The `create-runtime-rule` recipe SHALL explain that runtime rules execute code and therefore require login, reconciliation, and signing, and SHALL state this as a property of executing code rather than of the engine's capability. It SHALL point at `auth` for obtaining access rather than restating the login procedure, which `auth` owns. + +It SHALL NOT forward the agent to another authoring recipe. A logged-in runtime request is routed to `create-remote-rule` by `route`, so this recipe is reached only when the gate is closed and exists to explain that one gate once. + +#### Scenario: The gate is explained where it is encountered + +- **WHEN** an agent follows `create-runtime-rule` +- **THEN** the recipe SHALL state why the runtime tier is gated when the static tiers are not +- **AND** it SHALL refer the reader to `auth` rather than restating how to log in + +#### Scenario: The recipe does not delegate + +- **WHEN** an agent follows `create-runtime-rule` +- **THEN** it SHALL NOT be directed to fetch another authoring recipe to proceed + +### Requirement: Service generation is one recipe + +The CLI SHALL provide a single `create-remote-rule` recipe covering both the client-side boundary of service generation and the procedure itself — enriching the user's description, dispatching to the Taskless service, and reporting the result. + +Split across a boundary statement and a procedure, an agent fetches one only to learn it needs the other, which is the second fetch this change exists to remove. + +#### Scenario: One fetch reaches the whole procedure + +- **WHEN** an agent follows `create-remote-rule` +- **THEN** the recipe SHALL carry both the boundary and the dispatch procedure +- **AND** it SHALL NOT require fetching a second topic to complete the request + +### Requirement: Every authoring recipe opens by orienting the reader + +Each `create-*-rule` recipe SHALL open with a line naming the topic the reader is in, the kinds of rule it helps write, and an instruction to revisit the routing decision if that is not what they need. + +The line SHALL orient, not classify: it states this recipe's own scope and SHALL NOT restate the criterion distinguishing the engines from each other, which `route` holds in one place. An agent that arrived at the wrong recipe — by guessing, by a user naming a topic directly, or because `route` was wrong — should discover it in the first line, where recovery is cheap, rather than after authoring the wrong artifact. + +#### Scenario: A misrouted reader is told how to recover + +- **WHEN** an agent opens any `create-*-rule` recipe +- **THEN** the first lines SHALL name what that recipe helps write +- **AND** SHALL instruct the agent to revisit its routing decision if it needs a different kind of check + +#### Scenario: The orientation is not a second criterion + +- **WHEN** the orientation line is read +- **THEN** it SHALL describe only this recipe's scope, not the comparison between engines diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-help/spec.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-help/spec.md new file mode 100644 index 00000000..3da108d5 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-help/spec.md @@ -0,0 +1,183 @@ +## MODIFIED Requirements + +### Requirement: Help subcommand displays rich help text for commands + +The CLI SHALL support an `agent` subcommand that accepts at most one positional argument identifying a topic AND an optional `--anonymous` boolean flag. Topics SHALL be addressed by a single token; the subcommand SHALL NOT join multiple positionals into a topic key. When a topic is provided, the subcommand SHALL look up a matching help text file embedded at build time using the following resolution order: + +1. If `--anonymous` is set AND `.anonymous.txt` exists in the embedded map, return that file. +2. Otherwise, return `.txt`. +3. If neither exists, exit with code 1 and an error message suggesting `taskless agent` for the topic index. + +When no positional argument is provided, the subcommand SHALL print a topic index containing a one-paragraph human slug followed by a topic disambiguation table mapping topic names to their summaries. + +The subcommand is named for its reader. It serves agents fetching a procedure, not humans asking for help, and single-token addressing exists so a topic name is a literal string an agent copies rather than a phrase it can reorder or paraphrase. + +#### Scenario: Agent subcommand for a topic returns the recipe + +- **WHEN** a user runs `taskless agent check` +- **THEN** the CLI SHALL print the contents of `check.txt` to stdout + +#### Scenario: Multi-word topic paths are not resolved + +- **WHEN** a user runs `taskless agent rule create` +- **THEN** the CLI SHALL NOT look up `rule-create.txt` by joining the positionals +- **AND** it SHALL exit non-zero rather than guessing a topic + +#### Scenario: Formerly nested topics are addressed by one token + +- **WHEN** a user runs `taskless agent improve-rule` +- **THEN** the CLI SHALL look up `improve-rule.txt` and print its contents + +#### Scenario: The former command name is gone + +- **WHEN** a user runs `taskless help check` +- **THEN** the CLI SHALL NOT print recipe text for `check` + +### Requirement: onboard topic is registered in the help index + +A help topic `onboard` SHALL be registered. The CLI SHALL embed `packages/cli/src/help/onboard.txt` at build time via the existing `import.meta.glob` mechanism. `taskless agent onboard` SHALL print the contents of `onboard.txt`. The topic SHALL appear in the output of `taskless agent` (the index) with a one-line summary describing it as the post-install rule-discovery flow. + +#### Scenario: The onboard topic returns the recipe + +- **WHEN** a user runs `taskless agent onboard` +- **THEN** the CLI SHALL print the contents of `onboard.txt` to stdout +- **AND** SHALL exit with code 0 + +#### Scenario: Topic index includes onboard + +- **WHEN** a user runs `taskless agent` (no args) +- **THEN** the topic index SHALL include a row for `onboard` +- **AND** the row SHALL describe it as the post-install rule-discovery flow + +### Requirement: help_onboard intent telemetry + +Fetching the `onboard` topic SHALL emit the command's single intent event, `cli_help`, carrying `onboard` as its `topic` property. + +Per-topic event names (`help_onboard` and siblings) are not emitted. One event with a topic property is filterable the same way and does not grow the event vocabulary every time a topic is added or renamed, which this change would otherwise have to do for every rename below. + +#### Scenario: Fetching onboard captures its topic + +- **WHEN** an agent runs `taskless agent onboard` +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `onboard` + +### Requirement: Routing topics are registered in the help system + +The help system SHALL register `route` and each `create-*-rule` recipe as embedded topics, retrievable via `taskless agent ` and listed in the topic index, consistent with the existing topic embedding and format requirements. + +`existing`, `static`, and `remote` are no longer topics. `route` applies the criterion they carried and names a concrete destination, so an agent reaches an authoring recipe in one fetch. + +#### Scenario: Routing topics resolve + +- **WHEN** `taskless agent route` or any `taskless agent create-*-rule` is run +- **THEN** the corresponding recipe text SHALL be returned +- **AND** an unknown-topic error SHALL NOT be raised + +#### Scenario: Removed routing topics do not resolve + +- **WHEN** `taskless agent existing`, `taskless agent static`, or `taskless agent remote` is run +- **THEN** the CLI SHALL exit non-zero +- **AND** it SHALL NOT print recipe text + +#### Scenario: Routing topics appear in the index + +- **WHEN** `taskless agent` (no arguments) is run +- **THEN** the topic index SHALL include `route` and every `create-*-rule` topic + +### Requirement: Routing topics emit intent telemetry + +Fetching a routing recipe SHALL emit the command's single intent event, `cli_help`, carrying the served topic as its `topic` property. + +#### Scenario: Intent is captured for routing recipes + +- **WHEN** the agent fetches `route` or any `create-*-rule` topic +- **THEN** the command SHALL capture a `cli_help` event whose `topic` property is that topic name + +### Requirement: Anonymous variant lookup uses a compile-time map + +The help command SHALL construct, at build time, a Set of topic names that have a corresponding `.anonymous.txt` file. Lookup at runtime SHALL be O(1). The Set SHALL be derived from `import.meta.glob` matching `*.anonymous.txt` in the help directory. + +#### Scenario: Topics with variants are detected at build time + +- **WHEN** the CLI bundle is built +- **AND** a file `improve-rule.anonymous.txt` exists +- **THEN** the embedded variants set SHALL contain `improve-rule` + +#### Scenario: Topics without variants are absent from the map + +- **WHEN** the CLI bundle is built +- **AND** no `check.anonymous.txt` file exists +- **THEN** the embedded variants set SHALL NOT contain `check` +- **AND** `taskless agent check --anonymous` SHALL fall back to `check.txt` + +### Requirement: Embedded JSON schemas are generated via zod-to-json-schema + +For every recipe topic that documents a CLI command accepting `--from `, the corresponding Zod input schema in `packages/cli/src/schemas/` SHALL be converted to JSON Schema and embedded in the recipe's `## Input schema` section as a fenced code block. Generation MAY happen at runtime (small dep, fast) or at build time; runtime is acceptable. + +#### Scenario: The remote authoring recipe embeds its input schema + +- **WHEN** a user runs `taskless agent create-remote-rule` +- **THEN** the output SHALL contain an `## Input schema` section +- **AND** the section SHALL contain a code-fenced JSON Schema block derived from the `rules-create` Zod schema + +#### Scenario: The improve recipe embeds its input schema + +- **WHEN** a user runs `taskless agent improve-rule` +- **THEN** the output SHALL contain an `## Input schema` section with the rule-improve JSON Schema + +### Requirement: Help command emits intent telemetry + +The `agent` command SHALL emit one PostHog event, `cli_help`, on every invocation, carrying a `topic` property: + +- the served topic when a positional resolves to a known topic +- the attempted topic string when it resolves to none +- `(index)` when called with no positional arguments +- the joined positionals when more than one is supplied + +#### Scenario: Topic fetch captures the topic + +- **WHEN** an agent runs `taskless agent create-sg-rule` +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `create-sg-rule` + +#### Scenario: Index fetch captures the index + +- **WHEN** an agent runs `taskless agent` (no args) +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `(index)` + +## ADDED Requirements + +### Requirement: Routing recipes name a destination, not a second decision + +The `route` recipe SHALL apply the engine reasoning directly and name a concrete `create-*-rule` topic, rather than referring the reader onward to a topic that selects an engine. No shipped recipe SHALL refer to `engine-selection`, which no longer exists. + +Each `create-*-rule` recipe SHALL instead point back at `route` for a reader who arrived at the wrong one, so recovery costs a re-decision rather than a second copy of the criterion (see "Every authoring recipe opens by orienting the reader"). + +#### Scenario: Route names a destination without a second fetch + +- **WHEN** an agent follows `route` +- **THEN** the recipe SHALL name one `create-*-rule` topic +- **AND** it SHALL NOT require fetching a separate engine-selection topic first to do so + +#### Scenario: Authoring recipes point back rather than re-deciding + +- **WHEN** an agent reads any `create-*-rule` recipe +- **THEN** the recipe SHALL name `route` as where to go if this is the wrong destination +- **AND** it SHALL NOT reference `engine-selection` + +### Requirement: Shipped recipes name only commands that exist + +No embedded recipe SHALL contain the string `taskless help`. Recipes cross-reference each other by literal command string, so a stale reference is invisible until an agent runs it and receives nothing. + +#### Scenario: No recipe references the removed command + +- **WHEN** the embedded recipe set is inspected +- **THEN** no recipe SHALL contain `taskless help` + +## REMOVED Requirements + +### Requirement: The engine-selection topic is registered in the help system + +**Reason**: The topic no longer exists. Its criterion moved into `route`, which now applies the engine reasoning itself and names a concrete destination, so there is nothing left to register or to fetch. + +### Requirement: Routing recipes reference engine selection + +**Reason**: Replaced by "Routing recipes name a destination, not a second decision". The requirement named `route` and `static` and obliged them to forward to a separate engine-selection topic; `static` is gone, and forwarding is the behavior this change removes. diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-knowledge-prompts/spec.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-knowledge-prompts/spec.md new file mode 100644 index 00000000..4722b089 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-knowledge-prompts/spec.md @@ -0,0 +1,41 @@ +## MODIFIED Requirements + +### Requirement: Topic names and accessor shape are stable public API + +The set of `PromptTopic` names, the `getPrompt`/`PROMPTS` shape, and the existing fields of `PromptOptions` SHALL be treated as public API; recipe _text_ MAY change freely. + +The package is pre-1.0, so a backwards-incompatible change to that surface SHALL be released as a **MINOR** bump. This is what the leading zero means, and it applies to renaming a topic, removing one, or changing the accessor signature. + +What the requirement actually protects is not the version number but the notice. `TOPICS` is consumed across a deploy boundary, so a downstream consumer breaks when it upgrades rather than when this package builds, and the version alone cannot warn anyone. A breaking change SHALL therefore name the removed or renamed topics explicitly in its changeset. + +#### Scenario: Renaming or removing a topic + +- **WHEN** a topic is removed or renamed, or the accessor signature changes +- **THEN** it SHALL be released as a MINOR bump +- **AND** the changeset SHALL name the removed or renamed topics +- **AND** a recipe text edit SHALL require neither + +#### Scenario: Adding an option + +- **WHEN** a new optional field is added to `PromptOptions` +- **THEN** it SHALL NOT require more than a PATCH bump, since existing call sites keep their behavior + +## ADDED Requirements + +### Requirement: Exported topics cover every engine a rule can be routed to + +`TOPICS` SHALL export the authoring recipe for each engine — `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule`. + +A consumer that can decide a rule belongs to an engine must be able to reach the procedure for authoring one. Exporting a chooser without its destinations reproduces, for the platform generator, the dead end this change removes from the CLI. + +`engine-selection` leaves the export because it stops existing: the criterion it carried now lives in `route`, stated once. `route` is not exported here — it still contains local mechanics a Worker cannot run — so until it is, a consumer gets each destination's own scope from these three and adjudicates a genuinely ambiguous call itself. + +#### Scenario: Every engine's authoring path is reachable from the export + +- **WHEN** a consumer imports `TOPICS` +- **THEN** it SHALL contain `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule` + +#### Scenario: The exported set follows the rename + +- **WHEN** a consumer imports `TOPICS` +- **THEN** it SHALL NOT contain `static` or `engine-selection`, neither of which names a recipe any more diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-rule-routing/spec.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-rule-routing/spec.md new file mode 100644 index 00000000..cdfa94f1 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-rule-routing/spec.md @@ -0,0 +1,153 @@ +## MODIFIED Requirements + +### Requirement: Route is the local authoring classifier + +The CLI SHALL provide a `route` help recipe that instructs the agent to classify a rule-authoring request into one of five destinations — `create-legacy-rule`, `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, or `create-remote-rule` — using `taskless detect --json` signals plus the user's intent. The `route` recipe SHALL read the user's login state before dispatching, since it determines which destinations are reachable. It SHALL remain biased to stay local: local authoring that works SHALL NOT be abandoned for the service. + +`route` SHALL decide the engine as part of this classification rather than deferring it to a separate topic. There is one decision, made from one reading of the evidence: whether a rule is expressible locally and which engine can express it are answered from the same signals, so splitting them costs a second fetch and a handoff without adding information. + +Each destination SHALL be a topic an agent can fetch by name, so classifying produces a command to run rather than a category to interpret. + +#### Scenario: Route fetches detection before classifying + +- **WHEN** the agent fetches the `route` recipe to author a rule +- **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and + use its signals as input to the classification + +#### Scenario: Route classifies into one of five destinations + +- **WHEN** the agent follows `route` +- **THEN** it SHALL select exactly one of `create-legacy-rule`, `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, or `create-remote-rule` +- **AND** it SHALL fetch the corresponding recipe to perform the authoring + +#### Scenario: Every destination resolves to a recipe + +- **WHEN** any destination `route` can name is fetched +- **THEN** a recipe of that exact name SHALL exist + +#### Scenario: Service generation is offered only where it is a choice + +- **WHEN** the rule is expressible locally AND the user is logged in +- **THEN** `route` MAY offer `create-remote-rule` as an alternative and ask the user +- **AND WHEN** the user is not logged in, or the rule is not expressible locally +- **THEN** `route` SHALL NOT pose service generation as a choice, because it is not one + +#### Scenario: A logged-in runtime request routes straight to the service + +- **WHEN** the rule requires the runtime engine AND the user is logged in +- **THEN** `route` SHALL name `create-remote-rule` +- **AND** no recipe SHALL forward the agent from one destination to another + +#### Scenario: A logged-out runtime request reaches the explanation + +- **WHEN** the rule requires the runtime engine AND the user is not logged in +- **THEN** `route` SHALL name `create-runtime-rule` + +#### Scenario: The engine is decided without a second fetch + +- **WHEN** the agent follows `route` +- **THEN** it SHALL arrive at an engine-specific recipe without fetching a separate engine-selection topic + +### Requirement: Static recipe authors a verified local ast-grep rule + +The CLI SHALL provide a `create-sg-rule` help recipe that instructs the agent to author a +local ast-grep rule on-device, without calling the Taskless service, and to +verify it against the user's success and failure cases before reporting success. +The recipe SHALL produce the canonical on-disk rule shape and paths used by remote +generation so that `check`, `improve`, and `verify` see a single dialect. + +The recipe SHALL be named for the artifact it produces rather than for a trust tier. "Static" describes when a rule runs, which is a different axis from which engine enforces it, and naming the ast-grep authoring path after the tier taught the conflation that engine selection exists to correct. + +#### Scenario: Local authoring without the service + +- **WHEN** the agent follows `create-sg-rule` +- **THEN** it SHALL write the rule on-device without requiring login or the + Taskless API + +### Requirement: Available code context outranks the phrasing of the request + +Where code or diff context is available, `route` SHALL weigh the concrete syntactic form present in the repository above the wording of the request, since the same request routes differently depending on the form the code actually takes. + +This bound the standalone engine-selection topic. That topic is gone, but the reasoning is not — it now binds the place the decision is actually made. + +#### Scenario: Concrete form changes the engine + +- **WHEN** a rule is statically correlatable in the form the repository actually contains +- **THEN** `route` selects `create-sg-rule` +- **AND WHEN** the equivalent rule requires normalizing a captured value to match a declaration elsewhere +- **THEN** it selects a runtime destination, despite an identically phrased request + +### Requirement: Ambiguity resolves to an engine known to be available + +When no engine is clearly indicated, `route` SHALL direct the reader to choose an engine whose availability can be asserted in the situation at hand, and to give that availability as the reason for the call. It SHALL NOT name a fixed fallback engine. Both `sg` and `vale` ship as platform binaries, so either can be the missing one on an unsupported architecture or where an install was blocked; server-side the constraint is different again, `sg` being the only ungated route. A named default is wrong in whichever of those situations it failed to anticipate, which is why the requirement is stated as a property rather than as a fact about any one engine. + +#### Scenario: Ambiguous request resolves to an assertably available engine + +- **WHEN** the available context does not disambiguate which engine can enforce a rule +- **THEN** `route` selects an engine whose availability it can assert, and states that availability as the reasoning that made the call close + +#### Scenario: The default is never an unavailable engine + +- **WHEN** an engine is unavailable in the current environment, such as the Vale binary being absent +- **THEN** the ambiguity default SHALL NOT name it + +### Requirement: Existing recipe authors in the detected linter's dialect + +The CLI SHALL provide a `create-legacy-rule` help recipe that instructs the agent to author a rule in a linter already detected in the repository, expressed in that tool's own dialect. The recipe SHALL direct the agent to source authoring knowledge first from the repository's own existing rules and only then from the agent's own web research. The recipe SHALL NOT embed or rely on a Taskless-maintained catalog of linter rules. + +The recipe is named for the artifact it produces. "Existing" described the repository's state rather than the rule being written, which is not something an agent can address by name. + +#### Scenario: Repo-first knowledge sourcing + +- **WHEN** the agent follows `create-legacy-rule` +- **THEN** it SHALL read the repository's own rules for that linter before consulting any external source + +### Requirement: Remote recipe collects inputs and delegates to the service + +The CLI SHALL provide a `create-remote-rule` help recipe that instructs the agent to gather the inputs required to call the Taskless service and to invoke the existing rule generation backend, which runs the service-side classifier and returns either a static or a runtime rule. The recipe SHALL require authentication and SHALL NOT itself decide static versus runtime. + +#### Scenario: The remote recipe requires authentication + +- **WHEN** the agent follows `create-remote-rule` while logged out +- **THEN** the recipe SHALL direct the agent to `auth` rather than calling the service + +## ADDED Requirements + +### Requirement: Trust tier is not an engine-selection input + +Engine reasoning SHALL NOT treat login, reconciliation, or signing as inputs to the engine choice: `sg` and `vale` are both static-tier, and only `runtime` carries those concerns, so trust tier is a distinct axis from which engine can express a rule. + +#### Scenario: Trust tier is not an engine-selection input + +- **WHEN** the reasoning distinguishes `sg` from `vale` +- **THEN** it does so on the prose-versus-structure axis, not on any auth, reconcile, or signing property, since both are static-tier + +### Requirement: Engine reasoning lives in route and in each destination + +The engine criterion SHALL be stated once, in `route`'s destination table, which is where the comparison between engines is made. It SHALL NOT be stated in a separate chooser topic, and SHALL NOT be restated in the destination recipes. + +One statement is the point. A criterion copied into each destination is five copies of one test, and the first edit to any of them is a divergence nobody notices — the drift this merge exists to remove, reappearing one level down. Destinations orient the reader to their own scope instead, which needs nothing about the other engines. + +#### Scenario: The comparison lives in one place + +- **WHEN** the embedded recipe set is inspected +- **THEN** exactly one recipe SHALL state the criterion distinguishing the engines from each other + +#### Scenario: No separate chooser topic exists + +- **WHEN** the embedded recipe set is inspected +- **THEN** there SHALL be no topic whose only purpose is selecting among engines + +## REMOVED Requirements + +### Requirement: Engine selection is a separate axis from authoring destination + +**Reason**: The separation cost an agent two fetches and a handoff to answer one question. Whether a rule is expressible locally and which engine can express it are answered from the same evidence, so reading it twice added a failure point without adding information. + +**Migration**: The engine criterion moves into `route` and into each `create-*-rule` recipe (see "Engine reasoning lives in route and in each destination"). The one part of this requirement that was not about the split — that trust tier is a distinct axis — is retained as its own requirement above. Consumers that fetched `engine-selection` read the destination recipes instead, which is the surface `TOPICS` now exports. + +### Requirement: An engine-selection topic states which engine can enforce a rule + +**Reason**: The topic it required no longer exists as a separate recipe. + +**Migration**: Its content — the three engine definitions, evidence-before-answer, and the boundary cases — moves into `route` and the `create-*-rule` recipes. The requirements that constrained the reasoning itself ("Available code context outranks the phrasing of the request", "Ambiguity resolves to an engine known to be available") remain in force and now bind `route` and the destination recipes rather than a standalone topic. diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-vale-rule-engine/spec.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-vale-rule-engine/spec.md new file mode 100644 index 00000000..91816527 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/specs/cli-vale-rule-engine/spec.md @@ -0,0 +1,39 @@ +## ADDED Requirements + +### Requirement: The scaffolded Vale config carries no section + +The `.vale.ini` written when a project is scaffolded SHALL contain `StylesPath` and `MinAlertLevel` and no section. A project therefore lints nothing with Vale until someone scopes something deliberately. + +An unscoped `[*]` applies every enabled rule to every file the walk reaches, which makes the default the most aggressive scope available rather than the narrowest. Scope is the author's decision, and the scaffold SHALL NOT make it on their behalf. + +#### Scenario: A freshly scaffolded project reports nothing + +- **WHEN** `check` runs against a scaffolded project with a rule file present and no section added +- **THEN** Vale SHALL report no findings +- **AND** the run SHALL NOT be reported as an engine failure + +#### Scenario: Scope is added by the author + +- **WHEN** an author scopes a rule by adding a section +- **THEN** only files matching that section SHALL be subject to it + +### Requirement: Vale diagnostics on a successful run are surfaced as notices + +When Vale exits zero and writes to stderr, the CLI SHALL surface that output as a notice on the check result. A notice SHALL NOT affect the exit code. + +This is a precondition of the section-less scaffold rather than an independent improvement. With no section to copy, the likely first edit is a rule assignment at the top level of the file, which Vale reports as ignoring — on stderr, with a zero exit and a well-formed empty result. Discarding that output leaves the author with a rule that verifies, runs, and reports nothing, which is the silent-disable failure this engine's design exists to prevent. + +#### Scenario: An ignored rule assignment reaches the user + +- **WHEN** `.vale.ini` enables a rule outside any section and `check` runs +- **THEN** the CLI SHALL surface Vale's diagnostic that the assignment was ignored + +#### Scenario: A diagnostic does not fail the check + +- **WHEN** Vale exits zero, writes a diagnostic to stderr, and reports no findings +- **THEN** the check SHALL exit zero + +#### Scenario: Silence stays silent + +- **WHEN** Vale exits zero and writes nothing to stderr +- **THEN** the CLI SHALL add no notice diff --git a/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/tasks.md b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/tasks.md new file mode 100644 index 00000000..2aabb874 --- /dev/null +++ b/openspec/changes/archive/2026-08-15-agent-command-and-vale-authoring/tasks.md @@ -0,0 +1,70 @@ +# Tasks + +## 1. Rename the command + +- [x] 1.1 Rename `packages/cli/src/commands/help.ts` to `agent.ts` and the exported command to `agent`. Register it in `src/index.ts` +- [x] 1.2 Remove the positional-join resolution (`positionals.join("-")`). Accept at most one positional; more than one is an error rather than a joined key. The unknown-topic message points at `taskless agent` +- [x] 1.3 Keep the telemetry event name `cli_help` or rename it deliberately — decide once and record it, since dashboards key on it. If renamed, note it in the changeset alongside the `TOPICS` break + - **Decision: keep `cli_help`.** Renaming it in the same change that breaks the `TOPICS` export would take the dashboards dark for a reason unrelated to this change, and agent-call volume needs to stay visible under the existing event. The event name is not part of any agent-facing contract, so it can be renamed later on its own. Recorded as a comment at the capture site in `agent.ts`; nothing to add to the changeset +- [x] 1.4 Update `help-extensions.test.ts`, `help-routing-telemetry.test.ts`, `anonymous-flag.test.ts`, `onboard.test.ts`, and `cli.test.ts` to invoke `agent` + - Also required, not listed: `help-telemetry.test.ts` (imports `createHelpCommand` directly), `prompts.test.ts` (spawns `binPath help ` for the parity test), and `cli-run.test.ts` (its `resolveCommandName` case named `help`) + +## 2. Rename and add the authoring topics + +- [x] 2.1 `git mv` `help/static.txt` → `help/create-sg-rule.txt`; retitle its header and rewrite its Goal to name the artifact rather than the tier + - Also folded in the material from `rule-create.anonymous.txt` that `static.txt` lacked: the upstream-schema pointer, the optional-field list, and the per-layer verify error table +- [x] 2.2 `git mv` `help/existing.txt` → `help/create-legacy-rule.txt`; retitle and update its header +- [x] 2.3 Flatten the `rule-*` topics to verb-noun single tokens (`rule-create` → `create-rule`, `rule-improve` → `improve-rule`, `rule-delete` → `delete-rule`, `rule-verify` → `verify-rule`), including their `.anonymous` variants. Decide `rule-meta` and `rule` deliberately — they are not creation verbs and may keep their names + - **`rule-create` does not become `create-rule`.** 2.5a consumes it: the API-backed text *is* the service procedure, so it became `create-remote-rule`. A `create-rule` topic would have been a third name for the same thing + - **`rule-create.anonymous.txt` is deleted, not renamed.** It duplicated `static.txt` — both are "author an ast-grep rule locally, no service call" — and carrying it forward as `create-remote-rule.anonymous.txt` would have meant "the local variant of the remote recipe", which is the contradiction `route` exists to resolve. Its unique material moved into `create-sg-rule.txt` (2.1), and `rule create --anonymous` now points there. This is the one place group 2 deviates from the task text as written + - **`rule-meta` and `rule` keep their names.** Neither is a creation verb, both are already single tokens, and `rule`'s broken table is group 3's opening item +- [x] 2.4 Author `help/create-vale-rule.txt`: the three artifacts (style file, `.vale.ini` section, `pass/`/`fail` fixtures), that the scaffold ships section-less so the first rule writes the first scope, and that a rule enabled outside a section is ignored by Vale. State the evidence that makes `vale` the right engine for a rule, since no chooser topic states it any more. Cross-reference `verify-rule` +- [x] 2.5 Author `help/create-runtime-rule.txt` as the logged-**out** path: what a runtime rule is, where its `check.ts` lives, and why executing code requires login, reconciliation, and signing when the static tiers do not. Point at `auth` for obtaining access rather than restating it. It must not forward to another authoring recipe +- [x] 2.5a Merge `help/remote.txt` and `help/rule-create.txt` into `help/create-remote-rule.txt` (no `.anonymous` variant — see 2.3). A content merge, not a rename: both texts have material that survives, and the result must read as one procedure rather than two concatenated +- [x] 2.6 Rewrite `help/route.txt` to read login state early and classify into the five `create-*-rule` destinations, applying the engine reasoning inline rather than deferring to a second fetch. Offer `create-remote-rule` only where it is a genuine choice — locally expressible AND logged in. A logged-in runtime request routes straight to `create-remote-rule`; a logged-out one to `create-runtime-rule`. It must name a command the agent can run verbatim +- [x] 2.7 Merge `help/engine-selection.txt` into `help/route.txt` and delete it. Its three engine definitions, evidence-before-answer procedure, and boundary cases move into `route`'s destination table — stated once, not copied into the destinations +- [x] 2.7a Give every `create-*-rule` recipe the same opening orientation line: what topic this is, what it helps you write, and revisit routing if that is not what you need. Fixed shape across all five so an agent recognises it; scope only, never the comparison between engines +- [x] 2.8 Re-home the engine-reasoning requirements that survive the merge — "Available code context outranks the phrasing of the request" and "Ambiguity resolves to an engine known to be available" now bind `route` and the destinations. Update `help-extensions.test.ts`, which asserts against the standalone topic + +## 2b. Prove the authoring recipes by executing them + +The recipes are the deliverable, and a recipe that reads well to its author while producing the wrong artifact is exactly what reviewing the prose cannot catch. Execute them instead. + +- [x] 2b.1 `pnpm --filter @taskless/cli build:dev`. This target exists for this: `TASKLESS_BUILD_TARGET=dev` bakes `__TASKLESS_CLI__` as an **absolute path** to `dist-dev/index.js`, so recipe text carries a command that actually runs from any directory. Testing against `dist/` instead would exercise a recipe no reader ever receives, since theirs says `npx @taskless/cli` +- [x] 2b.2 Build the harness: scaffold a throwaway project in a temp directory using the built CLI, so the sandbox is a real `taskless init` scaffold — section-less `.vale.ini`, empty `vale/rules/` — and not a hand-made approximation of one +- [x] 2b.3 Hand a **fresh, non-forked** subagent only three things: the recipe text, the sandbox path, and a rule intent stated in plain words. It must NOT have repository access. With it, the agent finds the existing `no-simply.yml` and the mixed-engine fixture and copies them, and the loop tests our fixtures rather than our writing +- [x] 2b.4 Check the artifacts mechanically: a style file at `.taskless/vale/rules/.yml` with valid `extends`/`message`/`level`; a **scoped section** in `.vale.ini` enabling `rules.`; fixtures in both `pass/` and `fail/`. Then the assertion that matters — `check` reports the finding, and `verify` passes + - **Done except "`verify` passes", which cannot be satisfied as written.** `verifyValeRule`/`verifyValeRules` have no CLI caller: `taskless rule verify ` routes to `src/rules/verify.ts`, which is ast-grep only. There is no way for an agent to verify a Vale rule from the CLI today. The harness asserts on `check` over each bucket instead, which is what `create-vale-rule` now tells authors to do. **Open decision for the user** — wire `rule verify ` to dispatch by owning engine, or file it alongside #99/#101. Until then the recipe says plainly that nothing validates the fixture layout, rather than claiming a check that does not run +- [x] 2b.5 Iterate across three intents that exercise different extension points — one `existence`, one `substitution` (prefer X over Y), one `capitalization` (headings, product names). Everything in this repo today is `existence`, so a recipe drafted from our own examples teaches token blocklists and nothing else. Vale has eleven extension points and most real prose rules are not blocklists +- [x] 2b.6 Every failure is a defect in the prose, not in the agent. Fix the recipe and re-run with a fresh agent. Converged when an agent, given an intent the recipe never names, produces a rule that fires on its `fail` fixture and stays quiet on its `pass` fixture, first try, uncorrected +- [x] 2b.7 Keep the iteration log — what failed, what changed, what finally held. It is the evidence the prose works, and the only part of this a reviewer can check without rerunning the loop +- [x] 2b.8 Run the same harness over `create-sg-rule` as a control. It documents a flow that already works, so a failure there means the harness is wrong rather than the recipe + +## 3. Sweep the cross-references + +- [x] 3.1 Replace every `taskless help ` occurrence with `taskless agent ` across `src/help/*.txt`, `src/**/*.ts`, `skills/taskless/SKILL.md`, `README.md`, and `packages/cli/README.md` (~306 occurrences, 77 files). Leave `CHANGELOG.md` alone — it is a historical record +- [x] 3.2 Update every reference to a renamed topic (`static`, `existing`, `rule create`, …) to its new single-token name +- [x] 3.3 Add a test asserting no shipped recipe contains the string `taskless help`, and that every topic named in a recipe's See Also resolves to an embedded file. A stale cross-reference is otherwise invisible until an agent runs it + +## 4. Update the export surface + +- [x] 4.1 `TOPICS` becomes `["create-sg-rule", "create-vale-rule", "create-runtime-rule"]` and no longer exports `engine-selection`; move the renamed authoring topics through `INTERNAL_TOPICS` as their membership requires, keeping the two lists disjoint and jointly exhaustive over the recipe files +- [x] 4.2 Update `prompts.test.ts` — the membership test compares against the files on disk, so it fails until the rename is complete in both places +- [x] 4.3 Write the changeset as **MINOR** — pre-1.0, backwards-incompatible is MINOR — naming the removed topic names explicitly and stating that `@taskless/cli/prompts` consumers break on upgrade rather than at build time + +## 5. Scaffold and diagnostics (ships together) + +- [x] 5.1 `VALE_CONFIG_CONTENT` in `0004-vale-engine.ts` drops its `[*]` section, leaving `StylesPath` and `MinAlertLevel` +- [x] 5.2 `runVale` captures stderr on a zero-exit run and returns it as a notice on the `ok` outcome; `runValeEngine` forwards it to `DispatchResult.notices`. A notice must not touch the exit code + - **Pulled forward out of order, deliberately.** 2b tests `create-vale-rule` against a scaffolded project, and the recipe's central claim is that the scaffold ships section-less. Running the harness against a scaffold that still wrote `[*]` would have tested a recipe nobody will receive. Measured end to end afterwards: section-less scaffold + a top-level `rules. = YES` now prints `Notice: Vale reported while running: W101 'rules.no-simply' isn't a core option; Vale is ignoring it.` and exits 0 +- [x] 5.3 Test that a rule enabled outside a section produces a notice containing Vale's `W101` text and exits zero — this is the pairing that keeps 5.1 from reintroducing a silent disable +- [x] 5.4 Extend the mixed-engine integration test: a scaffolded project with a rule file and no section reports nothing and does not fail; adding a section makes the same rule fire + - **Superseded immediately above this change in the stack.** `self-contained-rules` gives every Vale rule its own config, so "a rule file and no section" stops being a quiet no-op and becomes a `verify` error: nothing scopes it, so it can never run, and saying so is better than reporting nothing. This test is correct for the layout this change ships and is replaced there rather than carried forward. The W101 pairing in 5.3 does survive, because an assignment can still be written above its own matcher. + +## 6. Verify + +- [x] 6.1 `pnpm typecheck`, `pnpm lint`, `pnpm --filter @taskless/cli build`, `pnpm --filter @taskless/cli test` +- [x] 6.2 **Rehearse `route` against a fresh agent.** Hand it the text with no prior context and a request to author a rule, then check which destination it names and why. Unlike the authoring recipes (2b), `route` produces a decision rather than artifacts, so the plan it describes is the only thing there is to check. Cover one case per destination, including a logged-out runtime request +- [x] 6.2a Run `taskless agent` with no argument, with each renamed topic, and with a removed name, confirming the index lists the new vocabulary and a removed name exits non-zero +- [x] 6.3 `pnpm openspec validate --all --strict` (note: `cli-rules` and `cli-update-engine` fail on `main` already and are unrelated) +- [x] 6.4 Archive the change diff --git a/openspec/specs/cli-agent-authoring/spec.md b/openspec/specs/cli-agent-authoring/spec.md new file mode 100644 index 00000000..ad09bd03 --- /dev/null +++ b/openspec/specs/cli-agent-authoring/spec.md @@ -0,0 +1,104 @@ +# cli-agent-authoring Specification + +## Purpose +TBD - created by archiving change agent-command-and-vale-authoring. Update Purpose after archive. +## Requirements +### Requirement: Every engine a rule can be routed to has an authoring recipe + +The CLI SHALL provide an authoring recipe for each engine `route` can name: `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule`, alongside `create-legacy-rule` for a linter the repository already uses. + +A decision procedure that can produce an answer with no destination is incomplete. Engine selection can conclude `vale` or `runtime`, and before this change neither had a procedure, so an agent that reasoned correctly arrived nowhere. + +#### Scenario: Each engine choice reaches a procedure + +- **WHEN** engine selection concludes `sg`, `vale`, or `runtime` +- **THEN** a recipe exists that authors a rule for that engine + +#### Scenario: A legacy destination exists for repositories with their own linter + +- **WHEN** the repository already runs a linter that can express the rule +- **THEN** `create-legacy-rule` SHALL author it in that tool's own dialect + +### Requirement: The Vale authoring recipe covers rule, scope, and fixtures + +The `create-vale-rule` recipe SHALL instruct the agent to produce three artifacts, and SHALL state that a rule is incomplete without all three: + +1. A Vale style file under `.taskless/vale/rules/.yml`. +2. A section in the committed `.taskless/vale/.vale.ini` scoping which files the rule applies to, enabling it as `rules. = YES`. +3. `pass/` and `fail/` fixture documents under `.taskless/vale/rule-tests//`. + +The recipe SHALL state that the scaffolded config carries no section, so the first rule authored in a project also authors the first scope. + +#### Scenario: Authoring produces all three artifacts + +- **WHEN** the agent follows `create-vale-rule` +- **THEN** it writes the style file, a scoping section enabling the rule, and both fixture buckets + +#### Scenario: The recipe teaches the first section + +- **WHEN** a project's `.vale.ini` has no section yet +- **THEN** the recipe SHALL direct the agent to add one scoped to the files the rule is about, rather than assuming a section exists + +#### Scenario: An unscoped rule is not silently accepted + +- **WHEN** the agent enables a rule without placing it inside a section +- **THEN** the recipe SHALL identify this as incomplete, because Vale ignores a rule assignment outside a section + +### Requirement: Authoring recipes write files rather than invoking a writer + +The `create-*-rule` recipes SHALL instruct the agent to write the rule, its configuration, and its fixtures directly. The CLI SHALL NOT provide a command that generates a Vale style file or edits `.vale.ini` on the agent's behalf. + +This matches how ast-grep rules are authored today: the agent writes the rule and its config entry, and construction belongs to the downstream generator rather than to the CLI. + +#### Scenario: No CLI writer for Vale configuration + +- **WHEN** an agent authors a Vale rule +- **THEN** it edits `.vale.ini` itself +- **AND** the CLI SHALL NOT offer a subcommand that performs that edit + +### Requirement: The runtime authoring recipe is the logged-out path + +The `create-runtime-rule` recipe SHALL explain that runtime rules execute code and therefore require login, reconciliation, and signing, and SHALL state this as a property of executing code rather than of the engine's capability. It SHALL point at `auth` for obtaining access rather than restating the login procedure, which `auth` owns. + +It SHALL NOT forward the agent to another authoring recipe. A logged-in runtime request is routed to `create-remote-rule` by `route`, so this recipe is reached only when the gate is closed and exists to explain that one gate once. + +#### Scenario: The gate is explained where it is encountered + +- **WHEN** an agent follows `create-runtime-rule` +- **THEN** the recipe SHALL state why the runtime tier is gated when the static tiers are not +- **AND** it SHALL refer the reader to `auth` rather than restating how to log in + +#### Scenario: The recipe does not delegate + +- **WHEN** an agent follows `create-runtime-rule` +- **THEN** it SHALL NOT be directed to fetch another authoring recipe to proceed + +### Requirement: Service generation is one recipe + +The CLI SHALL provide a single `create-remote-rule` recipe covering both the client-side boundary of service generation and the procedure itself — enriching the user's description, dispatching to the Taskless service, and reporting the result. + +Split across a boundary statement and a procedure, an agent fetches one only to learn it needs the other, which is the second fetch this change exists to remove. + +#### Scenario: One fetch reaches the whole procedure + +- **WHEN** an agent follows `create-remote-rule` +- **THEN** the recipe SHALL carry both the boundary and the dispatch procedure +- **AND** it SHALL NOT require fetching a second topic to complete the request + +### Requirement: Every authoring recipe opens by orienting the reader + +Each `create-*-rule` recipe SHALL open with a line naming the topic the reader is in, the kinds of rule it helps write, and an instruction to revisit the routing decision if that is not what they need. + +The line SHALL orient, not classify: it states this recipe's own scope and SHALL NOT restate the criterion distinguishing the engines from each other, which `route` holds in one place. An agent that arrived at the wrong recipe — by guessing, by a user naming a topic directly, or because `route` was wrong — should discover it in the first line, where recovery is cheap, rather than after authoring the wrong artifact. + +#### Scenario: A misrouted reader is told how to recover + +- **WHEN** an agent opens any `create-*-rule` recipe +- **THEN** the first lines SHALL name what that recipe helps write +- **AND** SHALL instruct the agent to revisit its routing decision if it needs a different kind of check + +#### Scenario: The orientation is not a second criterion + +- **WHEN** the orientation line is read +- **THEN** it SHALL describe only this recipe's scope, not the comparison between engines + diff --git a/openspec/specs/cli-help/spec.md b/openspec/specs/cli-help/spec.md index ff82a10b..46c464c3 100644 --- a/openspec/specs/cli-help/spec.md +++ b/openspec/specs/cli-help/spec.md @@ -3,53 +3,39 @@ ## Purpose TBD — Defines the help subcommand for the `@taskless/cli` package, including help text display, embedding, and formatting. - ## Requirements - ### Requirement: Help subcommand displays rich help text for commands -The CLI SHALL support a `help` subcommand that accepts zero or more positional arguments identifying a topic path AND an optional `--anonymous` boolean flag. When positional arguments are provided, the help subcommand SHALL look up a matching help text file embedded at build time using the following resolution order: +The CLI SHALL support an `agent` subcommand that accepts at most one positional argument identifying a topic AND an optional `--anonymous` boolean flag. Topics SHALL be addressed by a single token; the subcommand SHALL NOT join multiple positionals into a topic key. When a topic is provided, the subcommand SHALL look up a matching help text file embedded at build time using the following resolution order: 1. If `--anonymous` is set AND `.anonymous.txt` exists in the embedded map, return that file. 2. Otherwise, return `.txt`. -3. If neither exists, exit with code 1 and an error message suggesting `taskless help` for the topic index. - -When no positional arguments are provided, the help subcommand SHALL print a topic index containing a one-paragraph human slug followed by a topic disambiguation table mapping topic names to their summaries. +3. If neither exists, exit with code 1 and an error message suggesting `taskless agent` for the topic index. -#### Scenario: Help for a topic returns the recipe +When no positional argument is provided, the subcommand SHALL print a topic index containing a one-paragraph human slug followed by a topic disambiguation table mapping topic names to their summaries. -- **WHEN** a user runs `taskless help check` -- **THEN** the CLI SHALL print the contents of `check.txt` to stdout +The subcommand is named for its reader. It serves agents fetching a procedure, not humans asking for help, and single-token addressing exists so a topic name is a literal string an agent copies rather than a phrase it can reorder or paraphrase. -#### Scenario: Help for a nested topic joins with hyphens +#### Scenario: Agent subcommand for a topic returns the recipe -- **WHEN** a user runs `taskless help rule create` -- **THEN** the CLI SHALL look up `rule-create.txt` and print its contents - -#### Scenario: Help with --anonymous returns the variant when present - -- **WHEN** a user runs `taskless help rule create --anonymous` -- **AND** `rule-create.anonymous.txt` exists in the embedded help map -- **THEN** the CLI SHALL print the contents of `rule-create.anonymous.txt` +- **WHEN** a user runs `taskless agent check` +- **THEN** the CLI SHALL print the contents of `check.txt` to stdout -#### Scenario: Help with --anonymous falls back when no variant exists +#### Scenario: Multi-word topic paths are not resolved -- **WHEN** a user runs `taskless help check --anonymous` -- **AND** no `check.anonymous.txt` exists -- **THEN** the CLI SHALL print the contents of `check.txt` (no error, no warning — anonymous is a no-op for this topic) +- **WHEN** a user runs `taskless agent rule create` +- **THEN** the CLI SHALL NOT look up `rule-create.txt` by joining the positionals +- **AND** it SHALL exit non-zero rather than guessing a topic -#### Scenario: Help with no arguments shows index with human slug and disambiguation table +#### Scenario: Formerly nested topics are addressed by one token -- **WHEN** a user runs `taskless help` -- **THEN** the CLI SHALL print a one-paragraph human-facing slug explaining what the help command does for human vs. agent audiences -- **AND** SHALL print a topic table mapping each topic name to its one-line summary -- **AND** SHALL include a note about the `--anonymous` flag +- **WHEN** a user runs `taskless agent improve-rule` +- **THEN** the CLI SHALL look up `improve-rule.txt` and print its contents -#### Scenario: Help for an unknown topic exits with error +#### Scenario: The former command name is gone -- **WHEN** a user runs `taskless help nonexistent` -- **THEN** the CLI SHALL print an error message indicating the topic is not recognized -- **AND** exit with code 1 +- **WHEN** a user runs `taskless help check` +- **THEN** the CLI SHALL NOT print recipe text for `check` ### Requirement: Help text files are embedded at build time @@ -120,140 +106,62 @@ Recipe authors SHALL escape any literal `%` character in recipe content as `%%` ### Requirement: onboard topic is registered in the help index -A new help topic `onboard` SHALL be registered. The CLI SHALL embed `packages/cli/src/help/onboard.txt` at build time via the existing `import.meta.glob` mechanism. `taskless help onboard` SHALL print the contents of `onboard.txt`. The topic SHALL appear in the output of `taskless help` (the index) with a one-line summary describing it as the post-install rule-discovery flow. +A help topic `onboard` SHALL be registered. The CLI SHALL embed `packages/cli/src/help/onboard.txt` at build time via the existing `import.meta.glob` mechanism. `taskless agent onboard` SHALL print the contents of `onboard.txt`. The topic SHALL appear in the output of `taskless agent` (the index) with a one-line summary describing it as the post-install rule-discovery flow. -#### Scenario: Help for onboard returns the recipe +#### Scenario: The onboard topic returns the recipe -- **WHEN** a user runs `taskless help onboard` +- **WHEN** a user runs `taskless agent onboard` - **THEN** the CLI SHALL print the contents of `onboard.txt` to stdout - **AND** SHALL exit with code 0 -#### Scenario: Help index includes onboard +#### Scenario: Topic index includes onboard -- **WHEN** a user runs `taskless help` (no args) -- **THEN** the topic table SHALL include a row for `onboard` +- **WHEN** a user runs `taskless agent` (no args) +- **THEN** the topic index SHALL include a row for `onboard` - **AND** the row SHALL describe it as the post-install rule-discovery flow -#### Scenario: Onboard recipe is embedded at build time - -- **WHEN** the CLI bundle is built -- **THEN** `import.meta.glob` matching the help directory SHALL include `onboard.txt` -- **AND** the recipe SHALL be available at runtime without filesystem access - -#### Scenario: Help onboard with --anonymous falls back - -- **WHEN** a user runs `taskless help onboard --anonymous` -- **AND** no `onboard.anonymous.txt` exists -- **THEN** the CLI SHALL print the contents of `onboard.txt` (anonymous is a no-op for this topic) - ### Requirement: help_onboard intent telemetry -The help command's existing intent-telemetry requirement SHALL extend naturally to the new topic: invocations of `taskless help onboard` SHALL emit a `help_onboard` PostHog event, consistent with the `help_` pattern. +Fetching the `onboard` topic SHALL emit the command's single intent event, `cli_help`, carrying `onboard` as its `topic` property. + +Per-topic event names (`help_onboard` and siblings) are not emitted. One event with a topic property is filterable the same way and does not grow the event vocabulary every time a topic is added or renamed, which this change would otherwise have to do for every rename below. -#### Scenario: Help onboard emits help_onboard +#### Scenario: Fetching onboard captures its topic -- **WHEN** an agent runs `taskless help onboard` -- **THEN** PostHog SHALL receive a `help_onboard` event +- **WHEN** an agent runs `taskless agent onboard` +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `onboard` ### Requirement: Routing topics are registered in the help system -The help system SHALL register the routing recipes `route`, `existing`, `static`, -and `remote` as embedded help topics, retrievable via `taskless help ` and -listed in the help index, consistent with the existing topic embedding and format -requirements. +The help system SHALL register `route` and each `create-*-rule` recipe as embedded topics, retrievable via `taskless agent ` and listed in the topic index, consistent with the existing topic embedding and format requirements. + +`existing`, `static`, and `remote` are no longer topics. `route` applies the criterion they carried and names a concrete destination, so an agent reaches an authoring recipe in one fetch. #### Scenario: Routing topics resolve -- **WHEN** `taskless help route`, `taskless help existing`, - `taskless help static`, or `taskless help remote` is run +- **WHEN** `taskless agent route` or any `taskless agent create-*-rule` is run - **THEN** the corresponding recipe text SHALL be returned -- **AND** an unknown-topic error SHALL NOT be raised for any of the four - -#### Scenario: Routing topics appear in the index - -- **WHEN** `taskless help` (no arguments) is run -- **THEN** the topic index SHALL include the routing topics so an agent can - discover the authoring front door - -### Requirement: Routing topics emit intent telemetry - -Fetching a routing recipe SHALL emit a per-topic intent telemetry event, -consistent with the existing `help_` telemetry convention. - -#### Scenario: Help topic intent is captured for routing recipes - -- **WHEN** the agent fetches `route`, `existing`, `static`, or `remote` -- **THEN** the help command SHALL capture the corresponding `help_` intent - event with the topic name - -### Requirement: The engine-selection topic is registered in the help system - -The help system SHALL register the engine-selection recipe as an embedded help topic, retrievable via `taskless help ` and listed in the help index, consistent with the existing topic embedding and format requirements. - -#### Scenario: Engine-selection topic resolves - -- **WHEN** `taskless help` is run for the engine-selection topic -- **THEN** the recipe text SHALL be returned and an unknown-topic error SHALL NOT be raised - -#### Scenario: Engine-selection topic appears in the index - -- **WHEN** `taskless help` is run with no arguments -- **THEN** the topic index SHALL include the engine-selection topic so an agent can discover it - -### Requirement: Routing recipes reference engine selection - -The `route` and `static` recipes SHALL reference the engine-selection topic so an agent following the local authoring flow applies the same engine test the service applies, rather than assuming ast-grep. - -#### Scenario: Local flow reaches engine selection +- **AND** an unknown-topic error SHALL NOT be raised -- **WHEN** an agent follows `route` to a destination that authors a Taskless rule -- **THEN** the recipe directs it to the engine-selection topic before the rule is authored +#### Scenario: Removed routing topics do not resolve -## Goal +- **WHEN** `taskless agent existing`, `taskless agent static`, or `taskless agent remote` is run +- **THEN** the CLI SHALL exit non-zero +- **AND** it SHALL NOT print recipe text - - -## Preconditions - - - -## Steps - - - -## Input schema - - - -## Errors - - - -## See Also - - -``` - -The header line SHALL include the CLI version (interpolated at build time) and a topic version integer maintained by the recipe author and bumped when the recipe changes meaningfully. - -#### Scenario: Recipe contains all template sections +#### Scenario: Routing topics appear in the index -- **WHEN** any `.txt` file is read -- **THEN** it SHALL begin with the `# Topic: (CLI v / topic v)` header -- **AND** SHALL contain `## Goal`, `## Preconditions`, `## Steps`, `## Errors`, and `## See Also` sections in that order +- **WHEN** `taskless agent` (no arguments) is run +- **THEN** the topic index SHALL include `route` and every `create-*-rule` topic -#### Scenario: Recipe with --from input includes JSON schema +### Requirement: Routing topics emit intent telemetry -- **WHEN** a topic recipe documents a CLI invocation that uses `--from ` -- **THEN** the recipe SHALL contain an `## Input schema` section with a code-fenced JSON Schema block -- **AND** the JSON Schema SHALL be derived from the corresponding Zod schema in `packages/cli/src/schemas/` +Fetching a routing recipe SHALL emit the command's single intent event, `cli_help`, carrying the served topic as its `topic` property. -#### Scenario: Header version reflects build-time CLI version +#### Scenario: Intent is captured for routing recipes -- **WHEN** the CLI bundle is built -- **THEN** the recipe header's CLI version SHALL be interpolated at build time from `packages/cli/package.json` -- **AND** SHALL match the version reported by `taskless info` +- **WHEN** the agent fetches `route` or any `create-*-rule` topic +- **THEN** the command SHALL capture a `cli_help` event whose `topic` property is that topic name ### Requirement: Anonymous variant lookup uses a compile-time map @@ -262,47 +170,74 @@ The help command SHALL construct, at build time, a Set of topic names that have #### Scenario: Topics with variants are detected at build time - **WHEN** the CLI bundle is built -- **AND** files `rule-create.anonymous.txt` and `rule-improve.anonymous.txt` exist -- **THEN** the embedded variants set SHALL contain `rule-create` and `rule-improve` +- **AND** a file `improve-rule.anonymous.txt` exists +- **THEN** the embedded variants set SHALL contain `improve-rule` #### Scenario: Topics without variants are absent from the map - **WHEN** the CLI bundle is built - **AND** no `check.anonymous.txt` file exists - **THEN** the embedded variants set SHALL NOT contain `check` -- **AND** `taskless help check --anonymous` SHALL fall back to `check.txt` +- **AND** `taskless agent check --anonymous` SHALL fall back to `check.txt` ### Requirement: Embedded JSON schemas are generated via zod-to-json-schema -For every recipe topic that documents a CLI command accepting `--from `, the corresponding Zod input schema in `packages/cli/src/schemas/` SHALL be converted to JSON Schema via `zod-to-json-schema` and embedded in the recipe's `## Input schema` section as a fenced code block. Generation MAY happen at runtime (small dep, fast) or at build time; runtime is acceptable. +For every recipe topic that documents a CLI command accepting `--from `, the corresponding Zod input schema in `packages/cli/src/schemas/` SHALL be converted to JSON Schema and embedded in the recipe's `## Input schema` section as a fenced code block. Generation MAY happen at runtime (small dep, fast) or at build time; runtime is acceptable. -#### Scenario: rule create recipe embeds input schema +#### Scenario: The remote authoring recipe embeds its input schema -- **WHEN** a user runs `taskless help rule create` +- **WHEN** a user runs `taskless agent create-remote-rule` - **THEN** the output SHALL contain an `## Input schema` section -- **AND** the section SHALL contain a code-fenced JSON Schema block derived from the `rules-create` Zod schema (or the renamed `rule-create` schema) +- **AND** the section SHALL contain a code-fenced JSON Schema block derived from the `rules-create` Zod schema -#### Scenario: rule improve recipe embeds input schema +#### Scenario: The improve recipe embeds its input schema -- **WHEN** a user runs `taskless help rule improve` +- **WHEN** a user runs `taskless agent improve-rule` - **THEN** the output SHALL contain an `## Input schema` section with the rule-improve JSON Schema ### Requirement: Help command emits intent telemetry -The help command SHALL emit a PostHog event on every invocation: +The `agent` command SHALL emit one PostHog event, `cli_help`, on every invocation, carrying a `topic` property: + +- the served topic when a positional resolves to a known topic +- the attempted topic string when it resolves to none +- `(index)` when called with no positional arguments +- the joined positionals when more than one is supplied + +#### Scenario: Topic fetch captures the topic + +- **WHEN** an agent runs `taskless agent create-sg-rule` +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `create-sg-rule` + +#### Scenario: Index fetch captures the index + +- **WHEN** an agent runs `taskless agent` (no args) +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` property is `(index)` + +### Requirement: Routing recipes name a destination, not a second decision + +The `route` recipe SHALL apply the engine reasoning directly and name a concrete `create-*-rule` topic, rather than referring the reader onward to a topic that selects an engine. No shipped recipe SHALL refer to `engine-selection`, which no longer exists. + +Each `create-*-rule` recipe SHALL instead point back at `route` for a reader who arrived at the wrong one, so recovery costs a re-decision rather than a second copy of the criterion (see "Every authoring recipe opens by orienting the reader"). + +#### Scenario: Route names a destination without a second fetch + +- **WHEN** an agent follows `route` +- **THEN** the recipe SHALL name one `create-*-rule` topic +- **AND** it SHALL NOT require fetching a separate engine-selection topic first to do so + +#### Scenario: Authoring recipes point back rather than re-deciding -- `help_` (e.g. `help_rule_create`, `help_check`, `help_auth`) when called with positional arguments resolving to a known topic -- `help_index` when called with no positional arguments -- `help_unknown` (with the attempted topic as a property) when called with positional arguments resolving to no topic +- **WHEN** an agent reads any `create-*-rule` recipe +- **THEN** the recipe SHALL name `route` as where to go if this is the wrong destination +- **AND** it SHALL NOT reference `engine-selection` -These events SHALL replace the previous `cli_help_` events in a single hard rename. +### Requirement: Shipped recipes name only commands that exist -#### Scenario: Topic fetch emits intent event +No embedded recipe SHALL contain the string `taskless help`. Recipes cross-reference each other by literal command string, so a stale reference is invisible until an agent runs it and receives nothing. -- **WHEN** an agent runs `taskless help rule create` -- **THEN** PostHog SHALL receive a `help_rule_create` event +#### Scenario: No recipe references the removed command -#### Scenario: Index fetch emits help_index +- **WHEN** the embedded recipe set is inspected +- **THEN** no recipe SHALL contain `taskless help` -- **WHEN** an agent runs `taskless help` (no args) -- **THEN** PostHog SHALL receive a `help_index` event diff --git a/openspec/specs/cli-knowledge-prompts/spec.md b/openspec/specs/cli-knowledge-prompts/spec.md index 4e967642..335cf450 100644 --- a/openspec/specs/cli-knowledge-prompts/spec.md +++ b/openspec/specs/cli-knowledge-prompts/spec.md @@ -15,9 +15,7 @@ cannot drift into giving different guidance. The export carries no CLI runtime, so a Worker can import it without dragging in the command tree, and topic membership is an explicit hand-maintained list so a new recipe file cannot silently become public API. - ## Requirements - ### Requirement: The package exposes knowledge prompts via a dedicated import The package SHALL expose its knowledge prompts (the `help/*.txt` recipes) through a subpath export `@taskless/cli/prompts`, built into `dist` and listed in `files`, so consumers can import them without invoking the CLI. @@ -108,17 +106,23 @@ Where a `.anonymous.txt` variant exists, the export SHALL make it retriev ### Requirement: Topic names and accessor shape are stable public API -The set of `PromptTopic` names, the `getPrompt`/`PROMPTS` shape, and the existing fields of `PromptOptions` SHALL be treated as public API under semver; recipe _text_ MAY change within a major version. +The set of `PromptTopic` names, the `getPrompt`/`PROMPTS` shape, and the existing fields of `PromptOptions` SHALL be treated as public API; recipe _text_ MAY change freely. + +The package is pre-1.0, so a backwards-incompatible change to that surface SHALL be released as a **MINOR** bump. This is what the leading zero means, and it applies to renaming a topic, removing one, or changing the accessor signature. + +What the requirement actually protects is not the version number but the notice. `TOPICS` is consumed across a deploy boundary, so a downstream consumer breaks when it upgrades rather than when this package builds, and the version alone cannot warn anyone. A breaking change SHALL therefore name the removed or renamed topics explicitly in its changeset. -#### Scenario: Removing a topic is a breaking change +#### Scenario: Renaming or removing a topic - **WHEN** a topic is removed or renamed, or the accessor signature changes -- **THEN** it SHALL be released as a major version bump; a text edit SHALL NOT +- **THEN** it SHALL be released as a MINOR bump +- **AND** the changeset SHALL name the removed or renamed topics +- **AND** a recipe text edit SHALL require neither -#### Scenario: Adding an option is not a breaking change +#### Scenario: Adding an option - **WHEN** a new optional field is added to `PromptOptions` -- **THEN** it SHALL NOT require a major version bump, since existing call sites keep their behavior +- **THEN** it SHALL NOT require more than a PATCH bump, since existing call sites keep their behavior ### Requirement: Topic membership is explicit and verified against the recipe files @@ -140,3 +144,22 @@ An automated check SHALL assert that the set of canonical `help/*.txt` topics on - **WHEN** a recipe file is listed as internal - **THEN** the check SHALL pass and the topic SHALL NOT be a member of `PromptTopic` + +### Requirement: Exported topics cover every engine a rule can be routed to + +`TOPICS` SHALL export the authoring recipe for each engine — `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule`. + +A consumer that can decide a rule belongs to an engine must be able to reach the procedure for authoring one. Exporting a chooser without its destinations reproduces, for the platform generator, the dead end this change removes from the CLI. + +`engine-selection` leaves the export because it stops existing: the criterion it carried now lives in `route`, stated once. `route` is not exported here — it still contains local mechanics a Worker cannot run — so until it is, a consumer gets each destination's own scope from these three and adjudicates a genuinely ambiguous call itself. + +#### Scenario: Every engine's authoring path is reachable from the export + +- **WHEN** a consumer imports `TOPICS` +- **THEN** it SHALL contain `create-sg-rule`, `create-vale-rule`, and `create-runtime-rule` + +#### Scenario: The exported set follows the rename + +- **WHEN** a consumer imports `TOPICS` +- **THEN** it SHALL NOT contain `static` or `engine-selection`, neither of which names a recipe any more + diff --git a/openspec/specs/cli-rule-routing/spec.md b/openspec/specs/cli-rule-routing/spec.md index cebcbd9f..bb23c2c7 100644 --- a/openspec/specs/cli-rule-routing/spec.md +++ b/openspec/specs/cli-rule-routing/spec.md @@ -3,16 +3,14 @@ ## Purpose TBD - created by archiving change local-rule-routing. Update Purpose after archive. - ## Requirements - ### Requirement: Route is the local authoring classifier -The CLI SHALL provide a `route` help recipe that instructs the agent to classify -a rule-authoring request into one of three destinations — `existing`, `static`, -or `remote` — using `taskless detect --json` signals plus the user's intent. The -`route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or -`static` and SHALL treat `remote` as the escalation of last resort. +The CLI SHALL provide a `route` help recipe that instructs the agent to classify a rule-authoring request into one of five destinations — `create-legacy-rule`, `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, or `create-remote-rule` — using `taskless detect --json` signals plus the user's intent. The `route` recipe SHALL read the user's login state before dispatching, since it determines which destinations are reachable. It SHALL remain biased to stay local: local authoring that works SHALL NOT be abandoned for the service. + +`route` SHALL decide the engine as part of this classification rather than deferring it to a separate topic. There is one decision, made from one reading of the evidence: whether a rule is expressible locally and which engine can express it are answered from the same signals, so splitting them costs a second fetch and a handoff without adding information. + +Each destination SHALL be a topic an agent can fetch by name, so classifying produces a command to run rather than a category to interpret. #### Scenario: Route fetches detection before classifying @@ -20,12 +18,40 @@ or `remote` — using `taskless detect --json` signals plus the user's intent. T - **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and use its signals as input to the classification -#### Scenario: Route classifies into one of three destinations +#### Scenario: Route classifies into one of five destinations - **WHEN** the agent follows `route` -- **THEN** it SHALL select exactly one of `existing`, `static`, or `remote` +- **THEN** it SHALL select exactly one of `create-legacy-rule`, `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, or `create-remote-rule` - **AND** it SHALL fetch the corresponding recipe to perform the authoring +#### Scenario: Every destination resolves to a recipe + +- **WHEN** any destination `route` can name is fetched +- **THEN** a recipe of that exact name SHALL exist + +#### Scenario: Service generation is offered only where it is a choice + +- **WHEN** the rule is expressible locally AND the user is logged in +- **THEN** `route` MAY offer `create-remote-rule` as an alternative and ask the user +- **AND WHEN** the user is not logged in, or the rule is not expressible locally +- **THEN** `route` SHALL NOT pose service generation as a choice, because it is not one + +#### Scenario: A logged-in runtime request routes straight to the service + +- **WHEN** the rule requires the runtime engine AND the user is logged in +- **THEN** `route` SHALL name `create-remote-rule` +- **AND** no recipe SHALL forward the agent from one destination to another + +#### Scenario: A logged-out runtime request reaches the explanation + +- **WHEN** the rule requires the runtime engine AND the user is not logged in +- **THEN** `route` SHALL name `create-runtime-rule` + +#### Scenario: The engine is decided without a second fetch + +- **WHEN** the agent follows `route` +- **THEN** it SHALL arrive at an engine-specific recipe without fetching a separate engine-selection topic + ### Requirement: Route states reasoning before naming a destination The `route` recipe SHALL require the agent to write an explicit rationale before @@ -135,141 +161,89 @@ a default. ### Requirement: Existing recipe authors in the detected linter's dialect -The CLI SHALL provide an `existing` help recipe that instructs the agent to -author a rule in a linter already detected in the repository, expressed in that -tool's own dialect. The recipe SHALL direct the agent to source authoring -knowledge first from the repository's own existing rules and only then from the -agent's own web research. The recipe SHALL NOT embed or rely on a Taskless- -maintained catalog of linter rules. - -#### Scenario: Repo-first knowledge sourcing +The CLI SHALL provide a `create-legacy-rule` help recipe that instructs the agent to author a rule in a linter already detected in the repository, expressed in that tool's own dialect. The recipe SHALL direct the agent to source authoring knowledge first from the repository's own existing rules and only then from the agent's own web research. The recipe SHALL NOT embed or rely on a Taskless-maintained catalog of linter rules. -- **WHEN** the agent follows `existing` for a detected linter -- **THEN** it SHALL first mine the repository's existing rules of that kind for - house style -- **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the - repository signal is insufficient +The recipe is named for the artifact it produces. "Existing" described the repository's state rather than the rule being written, which is not something an agent can address by name. -#### Scenario: Existing path is author-only +#### Scenario: Repo-first knowledge sourcing -- **WHEN** the agent authors a rule via `existing` -- **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and - that `taskless check` does not execute the external linter +- **WHEN** the agent follows `create-legacy-rule` +- **THEN** it SHALL read the repository's own rules for that linter before consulting any external source ### Requirement: Static recipe authors a verified local ast-grep rule -The CLI SHALL provide a `static` help recipe that instructs the agent to author a +The CLI SHALL provide a `create-sg-rule` help recipe that instructs the agent to author a local ast-grep rule on-device, without calling the Taskless service, and to verify it against the user's success and failure cases before reporting success. The recipe SHALL produce the canonical on-disk rule shape and paths used by remote generation so that `check`, `improve`, and `verify` see a single dialect. +The recipe SHALL be named for the artifact it produces rather than for a trust tier. "Static" describes when a rule runs, which is a different axis from which engine enforces it, and naming the ast-grep authoring path after the tier taught the conflation that engine selection exists to correct. + #### Scenario: Local authoring without the service -- **WHEN** the agent follows `static` +- **WHEN** the agent follows `create-sg-rule` - **THEN** it SHALL write the rule on-device without requiring login or the Taskless API -#### Scenario: Verification gates success - -- **WHEN** the agent authors a static rule -- **THEN** it SHALL verify the rule against the provided success/failure cases - before reporting the rule as complete - -#### Scenario: Canonical output shape - -- **WHEN** the agent writes a static rule to disk -- **THEN** the files, paths, and shape SHALL match those produced by remote - generation - ### Requirement: Remote recipe collects inputs and delegates to the service -The CLI SHALL provide a `remote` help recipe that instructs the agent to gather -the inputs required to call the Taskless service and to invoke the existing rule -generation backend, which runs the service-side classifier and returns either a -static or a runtime rule. The `remote` recipe SHALL require authentication and -SHALL NOT itself decide static versus runtime. - -#### Scenario: Remote requires authentication - -- **WHEN** the agent follows `remote` while logged out -- **THEN** the recipe SHALL direct the agent to the authentication flow before - submitting the request - -#### Scenario: Static-versus-runtime is decided by the service +The CLI SHALL provide a `create-remote-rule` help recipe that instructs the agent to gather the inputs required to call the Taskless service and to invoke the existing rule generation backend, which runs the service-side classifier and returns either a static or a runtime rule. The recipe SHALL require authentication and SHALL NOT itself decide static versus runtime. -- **WHEN** the agent submits an authored request via `remote` -- **THEN** the recipe SHALL rely on the service to classify static versus runtime -- **AND** SHALL NOT make that determination locally +#### Scenario: The remote recipe requires authentication -#### Scenario: Remote output matches local on-disk shape +- **WHEN** the agent follows `create-remote-rule` while logged out +- **THEN** the recipe SHALL direct the agent to `auth` rather than calling the service -- **WHEN** the service returns a generated rule via `remote` -- **THEN** the written files and paths SHALL match the shape produced by the - local `static` path - -### Requirement: An engine-selection topic states which engine can enforce a rule - -The CLI SHALL provide a knowledge topic that decides, for a requested rule, **which engine can enforce it** — `sg`, `vale`, or `runtime` — valued as the engine's on-disk directory name. The topic SHALL define each engine by the information a rule fundamentally needs: - -- **`sg`** — expressible as a pattern over a single file's syntax tree, including correlation between constructs within that same file via relational operators. -- **`vale`** — the target is prose or markup content rather than code structure. -- **`runtime`** — needs information no single file's syntax tree contains: cross-file consistency, import or call graph, comparison against a non-code file, file metadata, or values requiring normalization a static pattern cannot express. +### Requirement: Available code context outranks the phrasing of the request -The topic SHALL instruct that the decision follow from what the rule fundamentally needs rather than how the request was phrased, and that the reasoning be stated before the engine is named. +Where code or diff context is available, `route` SHALL weigh the concrete syntactic form present in the repository above the wording of the request, since the same request routes differently depending on the form the code actually takes. -#### Scenario: Engine named for a single-file structural rule +This bound the standalone engine-selection topic. That topic is gone, but the reasoning is not — it now binds the place the decision is actually made. -- **WHEN** the topic is applied to a request expressible as a pattern over one file's syntax tree -- **THEN** it selects `sg` +#### Scenario: Concrete form changes the engine -#### Scenario: Engine named for a prose rule +- **WHEN** a rule is statically correlatable in the form the repository actually contains +- **THEN** `route` selects `create-sg-rule` +- **AND WHEN** the equivalent rule requires normalizing a captured value to match a declaration elsewhere +- **THEN** it selects a runtime destination, despite an identically phrased request -- **WHEN** the topic is applied to a request targeting prose or markup content -- **THEN** it selects `vale` +### Requirement: Ambiguity resolves to an engine known to be available -#### Scenario: Engine named for a cross-file rule +When no engine is clearly indicated, `route` SHALL direct the reader to choose an engine whose availability can be asserted in the situation at hand, and to give that availability as the reason for the call. It SHALL NOT name a fixed fallback engine. Both `sg` and `vale` ship as platform binaries, so either can be the missing one on an unsupported architecture or where an install was blocked; server-side the constraint is different again, `sg` being the only ungated route. A named default is wrong in whichever of those situations it failed to anticipate, which is why the requirement is stated as a property rather than as a fact about any one engine. -- **WHEN** the topic is applied to a request requiring information beyond a single file's syntax tree -- **THEN** it selects `runtime` +#### Scenario: Ambiguous request resolves to an assertably available engine -### Requirement: Engine selection is a separate axis from authoring destination +- **WHEN** the available context does not disambiguate which engine can enforce a rule +- **THEN** `route` selects an engine whose availability it can assert, and states that availability as the reasoning that made the call close -The engine-selection topic SHALL decide only which engine enforces a rule, and SHALL NOT decide where the rule is authored — that remains the `route` topic's concern. Locally the two compose, `route` first and engine selection second. +#### Scenario: The default is never an unavailable engine -The topic SHALL NOT describe login, reconciliation, or signing as inputs to the engine choice: `sg` and `vale` are both static-tier, and only `runtime` carries those concerns, so trust tier is a distinct axis from engine selection. +- **WHEN** an engine is unavailable in the current environment, such as the Vale binary being absent +- **THEN** the ambiguity default SHALL NOT name it -#### Scenario: Topic stays clear of authoring destination +### Requirement: Trust tier is not an engine-selection input -- **WHEN** the engine-selection topic is applied -- **THEN** it names an engine and does not select among `existing`, `static`, or `remote` authoring destinations +Engine reasoning SHALL NOT treat login, reconciliation, or signing as inputs to the engine choice: `sg` and `vale` are both static-tier, and only `runtime` carries those concerns, so trust tier is a distinct axis from which engine can express a rule. #### Scenario: Trust tier is not an engine-selection input -- **WHEN** the topic distinguishes `sg` from `vale` +- **WHEN** the reasoning distinguishes `sg` from `vale` - **THEN** it does so on the prose-versus-structure axis, not on any auth, reconcile, or signing property, since both are static-tier -### Requirement: Available code context outranks the phrasing of the request - -Where code or diff context is available, the engine-selection topic SHALL weigh the concrete syntactic form present in the repository above the wording of the request, since the same request routes differently depending on the form the code actually takes. - -#### Scenario: Concrete form changes the engine +### Requirement: Engine reasoning lives in route and in each destination -- **WHEN** a rule is statically correlatable in the form the repository actually contains -- **THEN** the topic selects `sg` -- **AND WHEN** the equivalent rule requires normalizing a captured value to match a declaration elsewhere -- **THEN** it selects `runtime`, despite an identically phrased request +The engine criterion SHALL be stated once, in `route`'s destination table, which is where the comparison between engines is made. It SHALL NOT be stated in a separate chooser topic, and SHALL NOT be restated in the destination recipes. -### Requirement: Ambiguity resolves to an engine known to be available +One statement is the point. A criterion copied into each destination is five copies of one test, and the first edit to any of them is a divergence nobody notices — the drift this merge exists to remove, reappearing one level down. Destinations orient the reader to their own scope instead, which needs nothing about the other engines. -When no engine is clearly indicated, the engine-selection topic SHALL direct the reader to choose an engine whose availability can be asserted in the situation at hand, and to give that availability as the reason for the call. The topic SHALL NOT name a fixed fallback engine. Both `sg` and `vale` ship as platform binaries, so either can be the missing one on an unsupported architecture or where an install was blocked; server-side the constraint is different again, `sg` being the only ungated route. A named default is wrong in whichever of those situations it failed to anticipate, which is why the requirement is stated as a property rather than as a fact about any one engine. +#### Scenario: The comparison lives in one place -#### Scenario: Ambiguous request resolves to an assertably available engine +- **WHEN** the embedded recipe set is inspected +- **THEN** exactly one recipe SHALL state the criterion distinguishing the engines from each other -- **WHEN** the available context does not disambiguate which engine can enforce a rule -- **THEN** the topic selects an engine whose availability it can assert, and states that availability as the reasoning that made the call close +#### Scenario: No separate chooser topic exists -#### Scenario: The default is never an unavailable engine +- **WHEN** the embedded recipe set is inspected +- **THEN** there SHALL be no topic whose only purpose is selecting among engines -- **WHEN** an engine is unavailable in the current environment, such as the Vale binary being absent -- **THEN** the ambiguity default SHALL NOT name it diff --git a/openspec/specs/cli-vale-rule-engine/spec.md b/openspec/specs/cli-vale-rule-engine/spec.md index f87026c1..0c67b282 100644 --- a/openspec/specs/cli-vale-rule-engine/spec.md +++ b/openspec/specs/cli-vale-rule-engine/spec.md @@ -3,9 +3,7 @@ ## Purpose TBD - created by archiving change add-vale-rule-engine. Update Purpose after archive. - ## Requirements - ### Requirement: Vale runs in the static tier without reconciliation or signing The system SHALL treat Vale as a static-tier engine — always run, with no server reconciliation or signature verification. Vale's `script` checks execute in a sandbox that exposes only pure-computation modules (`text`/`math`/`fmt`) with no host access, so a Vale rule is inert data equivalent in trust to a static ast-grep rule. @@ -122,3 +120,42 @@ Any Taskless-owned breadcrumb the system records in `.vale.ini` SHALL use a `tsk - **WHEN** a rule's scoping spans several matchers each tagged `tskl) rule = no-simply` - **THEN** tooling can find every matcher owned by `no-simply` by its `tskl) rule` id rather than by glob + +### Requirement: The scaffolded Vale config carries no section + +The `.vale.ini` written when a project is scaffolded SHALL contain `StylesPath` and `MinAlertLevel` and no section. A project therefore lints nothing with Vale until someone scopes something deliberately. + +An unscoped `[*]` applies every enabled rule to every file the walk reaches, which makes the default the most aggressive scope available rather than the narrowest. Scope is the author's decision, and the scaffold SHALL NOT make it on their behalf. + +#### Scenario: A freshly scaffolded project reports nothing + +- **WHEN** `check` runs against a scaffolded project with a rule file present and no section added +- **THEN** Vale SHALL report no findings +- **AND** the run SHALL NOT be reported as an engine failure + +#### Scenario: Scope is added by the author + +- **WHEN** an author scopes a rule by adding a section +- **THEN** only files matching that section SHALL be subject to it + +### Requirement: Vale diagnostics on a successful run are surfaced as notices + +When Vale exits zero and writes to stderr, the CLI SHALL surface that output as a notice on the check result. A notice SHALL NOT affect the exit code. + +This is a precondition of the section-less scaffold rather than an independent improvement. With no section to copy, the likely first edit is a rule assignment at the top level of the file, which Vale reports as ignoring — on stderr, with a zero exit and a well-formed empty result. Discarding that output leaves the author with a rule that verifies, runs, and reports nothing, which is the silent-disable failure this engine's design exists to prevent. + +#### Scenario: An ignored rule assignment reaches the user + +- **WHEN** `.vale.ini` enables a rule outside any section and `check` runs +- **THEN** the CLI SHALL surface Vale's diagnostic that the assignment was ignored + +#### Scenario: A diagnostic does not fail the check + +- **WHEN** Vale exits zero, writes a diagnostic to stderr, and reports no findings +- **THEN** the check SHALL exit zero + +#### Scenario: Silence stays silent + +- **WHEN** Vale exits zero and writes nothing to stderr +- **THEN** the CLI SHALL add no notice + diff --git a/packages/cli/README.md b/packages/cli/README.md index eb27e363..af0595f5 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -36,7 +36,7 @@ current project (`.claude/`, `.opencode/`, `.cursor/`, `.agents/`), asks which tools to enable Taskless for, and walks through the auth tradeoff before writing anything. Running `taskless` with no subcommand in a TTY also launches this wizard. Without a TTY, bare `taskless` prints a short context preamble -followed by the topic index from `taskless help`. +followed by the topic index from `taskless agent`. In v0.7+, there is exactly one skill (`taskless`) and one command (`tskl`) — no opt-in selection needed. @@ -142,10 +142,10 @@ taskless rule delete no-console-log Lists available subcommands. -### `taskless help [topic]` +### `taskless agent [topic]` Returns agent-facing recipes. With no args, prints the topic index. With a -topic (e.g. `taskless help rule create`), prints the full step-by-step recipe +topic (e.g. `taskless agent route`), prints the full step-by-step recipe for that operation, including an embedded JSON Schema for any `--from` input and a table of stable error codes. Append `--anonymous` to fetch the local-only variant where one exists (currently `rule create`/`rule improve`). @@ -159,7 +159,7 @@ relevant recipe on demand. Recognized on every command. Behavior matrix: - `rule create` / `rule improve` — exits with a pointer to - `taskless help --anonymous`. The local-only flow runs in the agent + `taskless agent --anonymous`. The local-only flow runs in the agent per the recipe variant. - `info` — skips the API/auth probe; reports local state only. - `auth login` — rejected (auth commands cannot be anonymous). diff --git a/packages/cli/src/commands/help.ts b/packages/cli/src/commands/agent.ts similarity index 65% rename from packages/cli/src/commands/help.ts rename to packages/cli/src/commands/agent.ts index d89e4c9c..da79afac 100644 --- a/packages/cli/src/commands/help.ts +++ b/packages/cli/src/commands/agent.ts @@ -10,15 +10,16 @@ import { import { getTelemetry } from "../telemetry"; import { getRecipe } from "../prompts/recipes"; -// Help-only recipe topics (no backing subcommand) that should still be -// discoverable from the `taskless help` index. The rule-authoring front +// Recipe-only topics (no backing subcommand) that should still be +// discoverable from the `taskless agent` index. The rule-authoring front // door (`route`) and its destinations live here so an agent can find them. const RECIPE_TOPICS: ReadonlyArray<[string, string]> = [ - ["route", "Decide where to author a rule (existing/static/remote)"], - ["existing", "Author a rule in a linter the repo already uses"], - ["static", "Author a local ast-grep rule on this machine (no login)"], - ["remote", "Generate a rule via the Taskless service (login)"], - ["engine-selection", "Decide which engine enforces a rule (sg/vale/runtime)"], + ["route", "Decide which recipe authors a rule (start here)"], + ["create-legacy-rule", "Author a rule in a linter the repo already uses"], + ["create-sg-rule", "Author a local ast-grep rule over code (no login)"], + ["create-vale-rule", "Author a local Vale rule over prose (no login)"], + ["create-runtime-rule", "The runtime tier, and why it needs an account"], + ["create-remote-rule", "Generate a rule via the Taskless service (login)"], ]; async function unwrap(resolvable: Resolvable): Promise { @@ -36,11 +37,11 @@ async function resolveDescription( return meta?.description ?? ""; } -export function createHelpCommand(subCommands: SubCommandsDef) { +export function createAgentCommand(subCommands: SubCommandsDef) { return defineCommand({ meta: { - name: "help", - description: "Show help for a command", + name: "agent", + description: "Return a recipe for an AI coding agent to follow", }, args: { dir: { @@ -67,14 +68,19 @@ export function createHelpCommand(subCommands: SubCommandsDef) { if (!argument.includes("=") && valueFlagSet.has(argument)) index++; continue; } - if (argument !== "help") positionals.push(argument); + if (argument !== "agent") positionals.push(argument); } const cwd = resolve(args.dir); const telemetry = await getTelemetry(cwd); if (positionals.length === 0) { - // cli_help with the index marker: agent fetched the topic list + // cli_help with the index marker: agent fetched the topic list. + // The event name stays `cli_help` even though the command is now + // `agent`: dashboards key on it, it is not part of any agent-facing + // contract, and renaming it in the same change that breaks the + // `TOPICS` export would take those dashboards dark for a reason + // unrelated to this change. telemetry.capture("cli_help", { topic: "(index)" }); console.log("Taskless CLI\n"); @@ -89,7 +95,7 @@ export function createHelpCommand(subCommands: SubCommandsDef) { const entries: Array<[string, string]> = []; for (const [name, cmd] of Object.entries(subCommands)) { - if (name === "help") continue; + if (name === "agent") continue; const description = await resolveDescription(cmd); entries.push([name, description]); } @@ -114,30 +120,44 @@ export function createHelpCommand(subCommands: SubCommandsDef) { ); console.log("and use local-only behavior."); console.log( - "\nRun `taskless help ` for the full recipe (e.g. `taskless help rule create`)." + "\nRun `taskless agent ` for the full recipe (e.g. `taskless agent create-sg-rule`)." ); return; } - // Join positional args to form the lookup key - const key = positionals.join("-"); + // Topics are addressed by exactly one token. Joining positionals into a + // key used to make `rule create` resolve `rule-create.txt`, which invited + // an agent to reorder or paraphrase a topic name and still get a hit. + // A single hyphenated token is a literal string to copy, so extra + // positionals are an error rather than something to guess at. + if (positionals.length > 1) { + telemetry.capture("cli_help", { topic: positionals.join(" ") }); + console.error(`Too many arguments: ${positionals.join(" ")}`); + console.error( + "A topic is a single token. Run `taskless agent` for the topic index." + ); + process.exitCode = 1; + return; + } + + const key = positionals[0]!; // Anonymous variant lookup: prefer .anonymous.txt when // --anonymous is set, fall back to the canonical recipe. The lookup and - // the render both live in the shared prompts module, so `help` and the + // the render both live in the shared prompts module, so `agent` and the // `@taskless/cli/prompts` export emit the same text. const recipe = getRecipe(key, { anonymous: args.anonymous }); if (recipe) { // cli_help: agent fetched a specific recipe (intent signal). The topic // is the served topic; filtering on it replaces the old per-topic events. - telemetry.capture("cli_help", { topic: positionals.join(" ") }); + telemetry.capture("cli_help", { topic: key }); console.log(recipe.trimEnd()); } else { // cli_help for an unknown topic — still the attempted topic string. - telemetry.capture("cli_help", { topic: positionals.join(" ") }); - console.error(`Unknown command: ${positionals.join(" ")}`); - console.error("Run `taskless help` for available commands."); + telemetry.capture("cli_help", { topic: key }); + console.error(`Unknown command: ${key}`); + console.error("Run `taskless agent` for available topics."); process.exitCode = 1; } }, diff --git a/packages/cli/src/commands/rules.ts b/packages/cli/src/commands/rules.ts index 17250bb3..b1d8b477 100644 --- a/packages/cli/src/commands/rules.ts +++ b/packages/cli/src/commands/rules.ts @@ -6,6 +6,8 @@ import { ZodError } from "zod"; import { resolveIdentity } from "../auth/identity"; import { verifyRule } from "../rules/verify"; +import { verifyValeRule } from "../rules/vale/verify"; +import { rulefileOwners, ruleFileLocation } from "../rules/owner"; import { submitRule, pollRuleStatus, iterateRule } from "../api/rules"; import { writeRuleFile, @@ -24,7 +26,10 @@ import { outputSchema as improveOutputSchema, } from "../schemas/rules-improve"; import { outputSchema as metaOutputSchema } from "../schemas/rules-meta"; -import { verifyOutputSchema } from "../schemas/rules-verify"; +import { + verifyOutputSchema, + valeVerifyOutputSchema, +} from "../schemas/rules-verify"; import { getTelemetry } from "../telemetry"; import { CLIError } from "../util/cli-error"; import { type CLIErrorCode, makeErrorEnvelope } from "../types/errors"; @@ -103,9 +108,12 @@ const createCommand = defineCommand({ if (args.anonymous) { // Anonymous rule creation runs in the agent, not the CLI. Point the - // agent at the local-only recipe and exit cleanly. + // agent at the local-only recipe and exit cleanly. That recipe is + // `create-sg-rule`: authoring an ast-grep rule on-device with no service + // call is exactly what anonymous mode asks for, so it is the destination + // rather than an `--anonymous` variant of the service recipe. const message = - "Anonymous rule generation runs in the agent. Run `taskless help rule create --anonymous` to fetch the local-only recipe."; + "Anonymous rule generation runs in the agent. Run `taskless agent create-sg-rule` to fetch the local-only recipe."; if (args.json) { console.log( JSON.stringify(makeErrorEnvelope("INVALID_INPUT", message)) @@ -348,7 +356,7 @@ const improveCommand = defineCommand({ if (args.anonymous) { const message = - "Anonymous rule improvement runs in the agent. Run `taskless help rule improve --anonymous` to fetch the local-only recipe."; + "Anonymous rule improvement runs in the agent. Run `taskless agent improve-rule --anonymous` to fetch the local-only recipe."; if (args.json) { console.log( JSON.stringify(makeErrorEnvelope("INVALID_INPUT", message)) @@ -680,10 +688,112 @@ const deleteCommand = defineCommand({ }, }); +/** + * Verify a Vale rule against its fixture buckets. + * + * Split out because the two engines answer different questions and their + * reports share no fields beyond `ruleId`/`success`. Vale's failure modes are + * about the *fixtures*: a bucket nobody populated proves nothing, and reporting + * that as a pass is how an unverified rule ships looking verified. + */ +async function verifyValeRuleCommand( + cwd: string, + ruleId: string, + json: boolean +): Promise { + let result; + try { + result = await verifyValeRule(cwd, ruleId); + } catch (error) { + // A nested fixture directory throws rather than being skipped: Vale lints + // the rule's whole tree, so a nested document is linted but never checked + // against a bucket, and silently ignoring it would let half a rule's + // fixtures go unverified while it reported a pass. + const message = error instanceof Error ? error.message : String(error); + if (json) { + console.log(JSON.stringify(makeErrorEnvelope("INVALID_INPUT", message))); + } else { + console.error(`Error: ${message}`); + } + process.exitCode = 1; + return; + } + + if ("outcome" in result) { + // Vale never ran. Not a verification result, and not a pass. + const { outcome } = result; + if (json) { + console.log( + JSON.stringify( + makeErrorEnvelope( + outcome.status === "unavailable" + ? "ENGINE_UNAVAILABLE" + : "SCAN_FAILED", + outcome.message + ) + ) + ); + } else { + console.error(`Error: ${outcome.message}`); + } + process.exitCode = 1; + return; + } + + if (json) { + console.log( + JSON.stringify( + valeVerifyOutputSchema.parse({ + engine: "vale", + success: result.passed, + ruleId: result.ruleId, + fixtures: result.fixtures, + missingFailures: result.missingFailures, + unexpectedFindings: result.unexpectedFindings, + }) + ) + ); + } else { + console.log(`Verifying Vale rule: ${result.ruleId}\n`); + + const coverage: Record = { + both: "✓ pass/ and fail/ both have documents", + "pass-only": "✗ no fail/ fixtures — the rule is never shown to fire", + "fail-only": + "✗ no pass/ fixtures — the rule is never shown to stay quiet", + none: "✗ no fixtures at all", + }; + console.log(`Fixtures: ${coverage[result.fixtures]}`); + + console.log( + `Fires: ${result.missingFailures.length === 0 ? "✓ every fail/ document was flagged" : "✗ some fail/ documents were not flagged"}` + ); + for (const file of result.missingFailures) { + console.log(` - ${file}`); + } + + console.log( + `Stays quiet: ${result.unexpectedFindings.length === 0 ? "✓ no pass/ document was flagged" : "✗ some pass/ documents were flagged"}` + ); + for (const file of result.unexpectedFindings) { + console.log(` - ${file}`); + } + + console.log( + `\nResult: ${result.passed ? "✓ All checks passed" : "✗ Verification failed"}` + ); + } + + if (!result.passed) { + process.exitCode = 1; + } +} + const verifyCommand = defineCommand({ meta: { name: "verify", - description: "Validate a rule against the ast-grep schema and run tests", + description: + "Validate a rule and run its tests (ast-grep) or its fixtures (Vale)", }, args: { dir: { @@ -726,10 +836,41 @@ const verifyCommand = defineCommand({ return; } - const result = await verifyRule(cwd, args.id); + // Which engine owns the rule is decided by where its file sits, the same + // way `dispatch` decides it, so a rule cannot be verified by one engine and + // run by another. + const ruleId = args.id; + const owners = await rulefileOwners(cwd, ruleId); + + if (owners.length > 1) { + // Both engines hold this id. Verifying one silently would report on a + // file the user may not have meant, so name both and let them say which. + const message = + `Rule "${ruleId}" exists for more than one engine: ` + + owners.map((engine) => ruleFileLocation(engine, ruleId)).join(", ") + + ". Rename one so the id identifies a single rule."; + if (args.json) { + console.log( + JSON.stringify(makeErrorEnvelope("INVALID_INPUT", message)) + ); + } else { + console.error(`Error: ${message}`); + } + process.exitCode = 1; + return; + } + + if (owners[0] === "vale") { + await verifyValeRuleCommand(cwd, ruleId, args.json); + return; + } + + // No owner falls through to the ast-grep verifier, which already reports a + // missing rule file across its layers — including the legacy location. + const result = await verifyRule(cwd, ruleId); if (args.json) { - console.log(JSON.stringify(verifyOutputSchema.parse(result))); + console.log(JSON.stringify(verifyOutputSchema.parse({ engine: "sg", ...result }))); } else { console.log(`Verifying rule: ${result.ruleId}\n`); diff --git a/packages/cli/src/filesystem/migrations/0004-vale-engine.ts b/packages/cli/src/filesystem/migrations/0004-vale-engine.ts index 6d7baabf..fe3c9b88 100644 --- a/packages/cli/src/filesystem/migrations/0004-vale-engine.ts +++ b/packages/cli/src/filesystem/migrations/0004-vale-engine.ts @@ -34,8 +34,18 @@ const SG_CONFIG_CONTENT = `ruleDirs:\n - rules\ntestConfigs:\n - testDir: rule * nothing, Vale reports `{}`, and a prose check passes clean with every rule * silently disabled. Measured against the real binary, which is the only way * this is visible — the layout is identical either way. + * + * It carries **no section**, so a scaffolded project lints nothing until an + * author scopes something deliberately. An unscoped `[*]` would apply every + * enabled rule to every file the walk reaches, making the default the widest + * scope available rather than the narrowest — and scope is the author's + * decision to make. `create-vale-rule` teaches writing the first section; the + * mistake that invites (a `rules. = YES` above the first `[…]` line, which + * Vale ignores with a `W101` on stderr and exit 0) is why `runVale` surfaces a + * zero-exit stderr as a notice. The two ship together: without the notice, this + * scaffold would trade a too-wide default for a silent one. */ -const VALE_CONFIG_CONTENT = `StylesPath = .\nMinAlertLevel = suggestion\n\n[*]\n`; +const VALE_CONFIG_CONTENT = `StylesPath = .\nMinAlertLevel = suggestion\n`; /** Directories that must exist after the migration, tracked when empty. */ const SCAFFOLD_DIRECTORIES = [ diff --git a/packages/cli/src/help/auth.txt b/packages/cli/src/help/auth.txt index 7c69bc89..7376aa41 100644 --- a/packages/cli/src/help/auth.txt +++ b/packages/cli/src/help/auth.txt @@ -73,5 +73,5 @@ report. ## See Also -- `taskless help info` — see auth state and skill versions -- `taskless help rule create` — first action that requires auth +- `taskless agent info` — see auth state and skill versions +- `taskless agent route` — first action that requires auth diff --git a/packages/cli/src/help/check.txt b/packages/cli/src/help/check.txt index 9523aa32..fe9e01d0 100644 --- a/packages/cli/src/help/check.txt +++ b/packages/cli/src/help/check.txt @@ -40,7 +40,7 @@ only: they never change the exit code. Under `--json` they do NOT appear as warnings; instead an additive optional `skipped: [{ rule, reason }]` array is included alongside the unchanged `{ success, results }`. The authoritative allow-list is the server's; the CI backstop -(`taskless help ci`) is the enforcement point for runtime rules. +(`taskless agent ci`) is the enforcement point for runtime rules. ## Flags - `--json` — machine output (`{ success, results, skipped? }`). @@ -125,5 +125,5 @@ When `--json` is set, failures emit `{ ok: false, code, message }`: ## See Also -- `taskless help rule create` — add a rule if none exist -- `taskless help ci` — wire `check` into a CI pipeline +- `taskless agent route` — add a rule if none exist +- `taskless agent ci` — wire `check` into a CI pipeline diff --git a/packages/cli/src/help/ci.txt b/packages/cli/src/help/ci.txt index 64f5d1cd..3164cd9e 100644 --- a/packages/cli/src/help/ci.txt +++ b/packages/cli/src/help/ci.txt @@ -12,7 +12,7 @@ you recognize one not on the list, apply the same patterns. ## Preconditions - `.taskless/` directory exists and contains at least one rule. (If no rules exist, instruct the user to fetch - `taskless help rule create` first — wiring CI with zero rules + `taskless agent route` first — wiring CI with zero rules produces an always-green check that gives false confidence.) - A local `taskless check` succeeds (or fails with real findings the user is OK with seeing in CI's first run). @@ -66,7 +66,7 @@ main. Run `npx @taskless/cli check`: - Clean pass → proceed. -- "No rules configured" → stop. Fetch `taskless help rule create`. +- "No rules configured" → stop. Fetch `taskless agent route`. - Findings → tell the user CI will fail; ask whether to fix, suppress, or proceed knowing the first CI run will be red. @@ -207,7 +207,7 @@ Show: ## Errors -- **No rules** → fetch `taskless help rule create`. Don't write CI +- **No rules** → fetch `taskless agent route`. Don't write CI config. - **Unrecognized CI** → produce a generic `.taskless/ci/check.sh` script implementing the six universal steps. Be upfront it's a @@ -217,5 +217,5 @@ Show: ## See Also -- `taskless help check` — the command being wired into CI -- `taskless help rule create` — required if no rules exist yet +- `taskless agent check` — the command being wired into CI +- `taskless agent route` — required if no rules exist yet diff --git a/packages/cli/src/help/existing.txt b/packages/cli/src/help/create-legacy-rule.txt similarity index 78% rename from packages/cli/src/help/existing.txt rename to packages/cli/src/help/create-legacy-rule.txt index efc00e53..6920cc8c 100644 --- a/packages/cli/src/help/existing.txt +++ b/packages/cli/src/help/create-legacy-rule.txt @@ -1,4 +1,11 @@ -# Topic: existing (CLI v%(CLI_VERSION)s / topic v1) +# Topic: create-legacy-rule (CLI v%(CLI_VERSION)s / topic v1) + +## You are here +This is `create-legacy-rule`. It helps you write a rule for a linter the +repository already runs — ESLint, Ruff, RuboCop, Stylelint — in that +tool's own dialect, so that tool enforces it. +If that is not the kind of check you need, re-run `taskless agent route` +and follow its decision rather than adapting this recipe. ## Goal Author a rule in a linter the repository ALREADY uses, expressed in that @@ -52,6 +59,6 @@ then write the rule where that tool expects it. ## See Also -- `taskless help route` — re-decide the destination if this no longer fits -- `taskless help static` — author a local ast-grep rule instead -- `taskless help remote` — generate via the Taskless service (login) +- `taskless agent route` — re-decide the destination if this no longer fits +- `taskless agent create-sg-rule` — author a local ast-grep rule instead +- `taskless agent create-remote-rule` — generate via the service (login) diff --git a/packages/cli/src/help/create-remote-rule.txt b/packages/cli/src/help/create-remote-rule.txt new file mode 100644 index 00000000..24c6fdbd --- /dev/null +++ b/packages/cli/src/help/create-remote-rule.txt @@ -0,0 +1,141 @@ +# Topic: create-remote-rule (CLI v%(CLI_VERSION)s / topic v1) + +## You are here +This is `create-remote-rule`. It helps you have the Taskless service +write a rule, when the rule is beyond what you can express on-device or +the user has chosen to spend a generation on it. +If that is not the kind of check you need, re-run `taskless agent route` +and follow its decision rather than adapting this recipe. + +## Goal +Enrich the user's description into a request the service can act on, +submit it, and report what came back. The service generates the rule and +writes the rule files; your job is everything on either side of that +call. + +This recipe is the whole path — the boundary and the procedure. Do not +go looking for a second topic to perform the submission. + +## Preconditions +- `.taskless/` directory exists. +- The user is logged in. This path requires auth. +- The repository has a GitHub origin remote. +- The request reached here through `route`, or through a local attempt + that failed and a user who confirmed the escalation. + +## Cost, and when this path is right + +A generation costs the user something and requires an account. That is +why `route` prefers a local rule it can write and verify, and why this +recipe is not a default. + +Two ways to legitimately be here: + +- **`route` sent you.** Either the rule is not expressible on-device, or + it is expressible and the user was offered the choice and took it. +- **A local attempt failed and the user confirmed.** If you came from + `create-sg-rule`'s failure path, that recipe has already deleted the + broken candidate and asked. If it has not been asked, ask now — a + silent fall-through from a failed local attempt to a paid service call + is not acceptable, even when the service would obviously do better. + +## Steps + +1. **Confirm auth.** Run: + ``` + npx @taskless/cli info --json + ``` + Check `loggedIn`. If false, fetch `taskless agent auth` and follow the + login recipe before continuing. Do not build a request you cannot + submit. + +2. **Check for a rule that already covers this.** Scan + `.taskless/sg/rules/` and read each rule's `message`, `note`, and + `rule` fields. If one overlaps, show the user and ask whether they + would rather iterate on it — `taskless agent improve-rule` refines an + existing rule and is usually the better answer than a second rule + that half-overlaps the first. + +3. **Gather the request.** Even when the user supplied a description, + ask: + - What exact pattern should be flagged? Get concrete examples. + - In what language? + - Where is the pattern acceptable — are there contexts to exclude? + + If you arrived from a failed local attempt, reuse the cases you + already gathered rather than asking again. + +4. **Enrich it.** This is the part that decides the quality of what + comes back: + - Search the codebase for real instances and show the user what you + found. Real matches beat invented ones. + - Ask for further success and failure cases, especially near-misses. + - Read `.gitignore`, the linter configs, and `tsconfig.json`'s + `exclude` for exclusion patterns the repo already believes in. + Present them as proposed defaults rather than applying them. + - Infer the primary language from the codebase and confirm it. Put it + in the `prompt` field. + +5. **Confirm before spending the generation.** Summarize the full + prompt, the success cases, the failure cases, and the exclusions. + This is the last point at which a misunderstanding is free. + +6. **Write the payload** to `.taskless/.tmp-rule-request.json`, matching + the input schema below. + +7. **Submit.** Run: + ``` + npx @taskless/cli rule create --from .taskless/.tmp-rule-request.json --json + ``` + This may take 30–60 seconds while the service generates the rule. + +8. **Clean up.** Delete `.taskless/.tmp-rule-request.json` whether the + call succeeded or failed. + +9. **Report.** The service writes the rule to `.taskless/sg/rules/.yml`, + its tests to `.taskless/sg/rule-tests/-YYYYMMDD-test.yml`, and + metadata to `.taskless/rule-metadata/.yml`. These are the same + paths and the same shape a locally authored rule uses, so `check`, + `improve-rule`, and `verify-rule` treat them identically. Show the + user the paths and suggest `taskless agent check`. + +## Input schema + +The `--from` JSON file conforms to: + +```json +%(INPUT_SCHEMA)s +``` + +Each entry in `successCases` and `failureCases` is a separate string. +Multi-line code goes in one string with literal newlines. + +## Important Notes + +- Do NOT pre-build the rule yourself here. Submit the request and let + the service generate it; a hand-written rule attached to a generation + request is neither reviewed nor used. +- The service owns rule-type selection. Today it generates ast-grep + rules under `.taskless/sg/rules/` — the same shape `create-sg-rule` + produces locally. + +## Errors + +With `--json`, failures emit `{ ok: false, code, message }`: + +| code | meaning | fix | +|--------------------------|------------------------------------|----------------------------------------------| +| `AUTH_REQUIRED` | not logged in | fetch `taskless agent auth` | +| `NO_GITHUB_REMOTE` | no GitHub origin remote | tell the user; we cannot proceed | +| `INVALID_INPUT` | `--from` JSON failed validation | re-read the input schema, fix, retry | +| `NETWORK_ERROR` | submit/poll failed | report and suggest retry | +| `RULE_GENERATION_FAILED` | the service failed to generate | report the message; suggest enriching prompt | +| `RULE_UNSUPPORTED` | plan lacks this generation type | tell the user to enable it; do not retry | + +## See Also + +- `taskless agent route` — the routing decision that leads here +- `taskless agent auth` — log in before generating +- `taskless agent create-sg-rule` — author a rule locally instead +- `taskless agent improve-rule` — iterate on a rule that already exists +- `taskless agent check` — validate the generated rule diff --git a/packages/cli/src/help/create-runtime-rule.txt b/packages/cli/src/help/create-runtime-rule.txt new file mode 100644 index 00000000..2445dd2b --- /dev/null +++ b/packages/cli/src/help/create-runtime-rule.txt @@ -0,0 +1,118 @@ +# Topic: create-runtime-rule (CLI v%(CLI_VERSION)s / topic v1) + +## You are here +This is `create-runtime-rule`. It helps you write a runtime rule: a +check that runs your own code, because answering it needs more than one +file — the repository graph, git metadata, build output, a resolved +config chain. +If that is not the kind of check you need, re-run `taskless agent route` +and follow its decision rather than adapting this recipe. + +You are reading this recipe rather than `create-remote-rule` because the +user is **not logged in**. That is the whole difference, and this recipe +explains it once. + +## Goal +Explain what a runtime rule is, why this tier is gated when the static +tiers are not, and what the user has to do before one can run. + +## Preconditions +- `.taskless/` directory exists. +- The user is **not** logged in. If `taskless info --json` reports + `loggedIn: true`, you are in the wrong recipe — re-run + `taskless agent route`. + +## What a runtime rule is + +A rule directory under `.taskless/runtime/rules//` holding two +kinds of file: + +| File | Role | +|---------------|------------------------------------------------------------------| +| `*.yml` | ast-grep capture rules that narrow which files the check looks at | +| `check.ts` | a module whose default export receives those matches and returns findings | + +The capture rules do the cheap work — if nothing matches, `check.ts` is +never invoked. `check.ts` does the part no static engine can: hold two +files at once, read git, resolve an alias, look outside the repository. + +That is also exactly why it is gated. + +## Why this tier is gated + +**Because it executes code, not because of what it can express.** + +`sg` and `vale` rules are inert data. Whatever is in them, the worst a +malicious rule achieves is a wrong finding — `taskless check` runs them +with no login, no network, and nothing to verify, because there is +nothing to verify. + +A runtime rule's `check.ts` is a program that runs on the developer's +machine with the developer's permissions. A rule file that arrives in a +pull request is code that arrives in a pull request. So `check`: + +1. **Signs** each `check.ts` — a signature over the exact bytes on disk. +2. **Reconciles** those signatures with the Taskless service, which + answers with the set it will vouch for. +3. **Runs only what came back blessed.** Anything else is reported as + skipped, with the reason. + +Reconciliation is what makes the signature mean something: it is the +service saying "this rule, these exact bytes, for this repository." +Without a login there is nobody to ask, so there is no answer, so +nothing runs. + +This is a property of executing code. It says nothing about whether a +runtime rule is more or less capable, and it is not a quality tier — +`sg` and `vale` rules are not less trusted, they are unexecuted. + +## What happens today, logged out + +Nothing breaks. `taskless check` still runs every static rule; each +runtime rule it finds is listed as skipped with the reason +`not authenticated — runtime rules were not verified and did not run`. + +So a runtime rule you write now is inert until the user logs in. Say +that plainly rather than letting them discover it from a silent check. + +## Steps + +1. **Tell the user what their rule needs, and why it is gated.** Name + the evidence — "this has to compare two files", "this reads git + history" — and then the consequence: it has to run code, so it needs + an account. Do not present this as a limitation of the rule. + +2. **Point them at login.** Obtaining access is `auth`'s job, and this + recipe deliberately does not restate it: + + ``` + npx @taskless/cli agent auth + ``` + + Follow that recipe with the user. If they do not want an account, + stop here — say the rule cannot run without one, and offer to + reconsider whether a narrower version of the request could be + answered by a static rule instead. That is a new routing decision, + not a fallback you take on their behalf. + +3. **Once they are logged in, re-route.** Run + `npx @taskless/cli info --json` to confirm `loggedIn: true`, then + re-run `taskless agent route` with the original request. The + destination changes now that the gate is open. + +## Important Notes + +- `--dangerously-run-scripts` makes `check` execute runtime rules + without server verification. It exists for local iteration on a rule + you wrote yourself and just read. It is not a way to ship a rule to a + team, and suggesting it to work around a login turns a deliberate gate + into an unreviewed code-execution path on someone else's machine. +- Do not author a `check.ts` and leave it in the repository unmentioned. + A skipped runtime rule reports nothing, which reads exactly like a + passing one. + +## See Also + +- `taskless agent auth` — log in, log out, check status +- `taskless agent route` — re-decide once the login state changes +- `taskless agent check` — see which rules ran and which were skipped diff --git a/packages/cli/src/help/create-sg-rule.txt b/packages/cli/src/help/create-sg-rule.txt new file mode 100644 index 00000000..784c68f9 --- /dev/null +++ b/packages/cli/src/help/create-sg-rule.txt @@ -0,0 +1,113 @@ +# Topic: create-sg-rule (CLI v%(CLI_VERSION)s / topic v1) + +## You are here +This is `create-sg-rule`. It helps you write an ast-grep rule: a check +over the structure of a single source file, authored on this machine. +If that is not the kind of check you need, re-run `taskless agent route` +and follow its decision rather than adapting this recipe. + +## Goal +Produce a verified ast-grep rule and its test file, locally, without +contacting the Taskless service. You derive the rule yourself, write it +in the canonical on-disk shape, and validate it with `verify-rule` in a +feedback loop. The files you produce match exactly what the service +writes, so `check`, `improve-rule`, and `verify-rule` treat them +identically. + +## Preconditions +- `.taskless/` directory exists. +- The agent can read/write files and run shell commands. +- No auth required. + +## Artifacts + +A rule is complete when both of these exist: + +1. The rule at `.taskless/sg/rules/.yml`. +2. Its tests at `.taskless/sg/rule-tests/-YYYYMMDD-test.yml`. + +## Steps + +1. **Learn the ast-grep rule format.** Consult the ast-grep rule + reference at https://ast-grep.github.io/guide/rule-config.html for + valid fields and operators (`pattern`, `kind`, `regex`, + `any`/`all`/`has`/`inside`/`not`) and meta-variable syntax. This + recipe does not embed the schema — read it from upstream rather than + writing a rule from memory. + +2. **Gather and confirm the pattern.** Make sure you have concrete + success cases (code that should pass) and failure cases (code that + should be flagged), the target language, and any exceptions. Search + the codebase for real instances and confirm exclusions with the user. + +3. **Check for an existing rule that already covers this.** Scan + `.taskless/sg/rules/` and read each rule's `message`, `note`, and + `rule` fields. If one overlaps, show the user and ask whether they + would rather improve it — `taskless agent improve-rule --anonymous` + iterates a rule locally. + +4. **Author the rule in the canonical shape.** Write the rule to + `.taskless/sg/rules/.yml` with at minimum: + - `id` — kebab-case, matching the filename (e.g. `no-eval`) + - `language` — the target language + - `severity` — `error`, `warning`, `info`, or `hint` + - `message` — a concise single-line explanation + - `rule` — the ast-grep rule object + + Optional but useful: `note` (multi-line guidance, supports markdown), + `fix` (auto-fix pattern), `ignores` (file patterns to skip). + +5. **Write the tests.** Write + `.taskless/sg/rule-tests/-YYYYMMDD-test.yml` with the matching + `id` field plus `valid` and `invalid` arrays — at least two of each, + drawn from real code where you can. The `id` must match the rule's + `id` so ast-grep test filtering pairs them. These paths and this + shape are the same ones the service writes; do not invent a different + layout. + +6. **Run the verify feedback loop.** Run: + ``` + npx @taskless/cli rule verify --json + ``` + - `success: true` → the rule passes. Go to step 7. + - `success: false` → read the per-layer errors and fix. Repeat up to + 3 times. + + | layer | what a failure means | fix | + |----------------|-----------------------------------------|------------------------------------------------| + | `schema` | YAML doesn't match the ast-grep schema | check field types against the upstream schema | + | `requirements` | a Taskless-required field is missing | add `id`/`language`/`severity`/`message`/`rule`| + | `tests` | a case didn't behave as expected | fix the rule pattern OR the test case | + + A `regex` without an accompanying `kind` fails `requirements` — the + two always travel together. + +7. **On success, report.** Show the rule and test file paths and a + one-line summary of what the rule detects. Suggest + `taskless agent check` to validate against the broader codebase. + +8. **On failure, escalate — with confirmation.** If after the feedback + loop the rule still cannot capture the user's cases: + - Delete the candidate `.taskless/sg/rules/.yml` and its test file + so the repo is not left with a broken rule. + - Tell the user the local rule could not capture the cases, and that + generating via the Taskless service uses a generation and requires + login. + - Only after the user confirms, fetch `taskless agent + create-remote-rule` and follow it. Do not call the service silently. + +## Important Notes + +- Do NOT make any HTTP requests to taskless.io on this path. +- Do NOT write to `.taskless/rule-metadata/` — locally authored rules + have no metadata sidecar; they iterate via file edits. +- The verify loop is the quality gate. A clean failure is a legitimate + reason to escalate, but only with the user's confirmation (step 8). + +## See Also + +- `taskless agent route` — re-decide the destination +- `taskless agent verify-rule` — the validation primitive step 6 calls +- `taskless agent improve-rule` — iterate on a rule that already exists +- `taskless agent create-remote-rule` — generate via the service (login) +- `taskless agent check` — validate the new rule against the codebase diff --git a/packages/cli/src/help/create-vale-rule.txt b/packages/cli/src/help/create-vale-rule.txt new file mode 100644 index 00000000..796f7943 --- /dev/null +++ b/packages/cli/src/help/create-vale-rule.txt @@ -0,0 +1,570 @@ +# Topic: create-vale-rule (CLI v%(CLI_VERSION)s / topic v1) + +## You are here +This is `create-vale-rule`. It helps you write a Vale rule: a check over +the words of a document — prose, markup, and the prose parts of code. +If that is not the kind of check you need, re-run `taskless agent route` +and follow its decision rather than adapting this recipe. + +## Goal +Produce a Vale rule that fires on the prose it should flag and stays +quiet on the prose it should not, scoped to the files it is about. + +## Preconditions +- `.taskless/` directory exists (it contains `vale/.vale.ini`). +- The agent can read/write files and run shell commands. +- No auth required. + +## The three artifacts + +A Vale rule is three files, not one. It is **incomplete until all three +exist**, and the failure mode of skipping one is silence rather than an +error — the rule runs, the check passes, and nothing is ever reported. + +| # | Artifact | Path | +|---|-----------|-------------------------------------------------------------| +| 1 | The style | `.taskless/vale/rules/.yml` | +| 2 | The scope | a section in `.taskless/vale/.vale.ini` enabling `rules.`| +| 3 | Fixtures | `.taskless/vale/rule-tests//pass/` and `.../fail/` | + +`verify-rule` checks all three, and refuses a rule that has only one +fixture bucket — a rule with no `fail/` was never shown to fire, and one +with no `pass/` was never shown to stay quiet. Either alone is half a +claim, so neither passes. + +**Three names must agree:** + +``` +.taskless/vale/rules/no-simply.yml <- the filename +.taskless/vale/rule-tests/no-simply/ <- the fixture directory +rules.no-simply = YES <- the .vale.ini assignment +``` + +`verify-rule` catches the first two: it builds its own config from the +id you give it, so a misnamed rule file or fixture directory shows up as +fixtures that never fired, or as no fixtures at all. + +**It cannot catch the third.** Verification runs in an isolated config +of its own making, so a rule can verify perfectly and still be absent +from the committed `.vale.ini` that `check` reads — verified, and never +run. That assignment is the one thing you have to get right by looking. + +## Steps + +1. **Say what the rule reads, then pick an extension point.** Write one + sentence: "this fires when a document contains ___." Vale rules are + built by extending one of its eleven checks, and the sentence tells + you which: + + | If the rule is about… | extends | + |-------------------------------------------------------|------------------| + | words or phrases that should not appear | `existence` | + | preferring one term over another — **including the correct spelling of a product name** | `substitution` | + | the case of a whole heading or sentence | `capitalization` | + | how many times something may appear | `occurrence` | + | a word repeated back to back | `repetition` | + | picking one of two acceptable spellings, consistently | `consistency` | + | "if X appears, Y must also appear" | `conditional` | + | readability or length thresholds | `metric` | + | a misspelling, against a dictionary | `spelling` | + | phrases that must appear in a fixed order | `sequence` | + | anything the above cannot express (Tengo script) | `script` | + + **`capitalization` is about a whole scope, not a word.** It asks + whether an entire heading or sentence matches a case pattern. It + cannot express "the word GitHub, wherever it appears, is spelled + thus" — that is a `substitution`, because you are swapping a wrong + spelling for a right one. Reaching for `capitalization` on a product + name produces a rule that flags whole sentences: measured, a rule with + `match: GitHub` reports `We host on Github and it is fine. should be + GitHub`. + + For the two this recipe does not describe, read + https://docs.vale.sh/styles before inventing something — Vale has no + facility for a rule that does not extend one of these eleven. + + **Nine of them have a worked rule at the end of this recipe**, each + with the near-miss that fails and why. Read the one closest to your + intent before writing anything — the mistakes documented there are + observed, and most of them fail silently. + +2. **Write the style file** to `.taskless/vale/rules/.yml`, where + `` is kebab-case and matches the filename. Every rule carries: + + | Field | Required | Notes | + |-----------|----------|---------------------------------------------------------| + | `extends` | yes | one of the eleven above | + | `message` | yes | shown to the user; see the `%%s` table below | + | `level` | no | `suggestion` (default), `warning`, or `error` | + | `scope` | no | narrow to part of a document — see below | + | `link` | no | a URL the reader can follow for the reasoning | + + **`scope` decides where the rule looks**, so getting it wrong is a + silent over- or under-fire rather than an error. The useful values for + markdown: `heading` (and `heading.h1`…`heading.h6`), `paragraph`, + `sentence`, `list`, `blockquote`, `link`, `code`, `alt`, `summary`, + `table.header`, `table.cell`, `table.caption`, `figure.caption`, + `strong`, `emphasis`, `raw`. Omit it and the rule sees all prose. + + **Then the fields the extension point adds** — this is where the rule + actually lives, and each check reads only its own: + + | extends | its fields | + |------------------|-------------------------------------------------------------------| + | `existence` | `tokens` (a list) or `raw`; `ignorecase`, `nonword`, `exceptions`, `append` | + | `substitution` | `swap` (a map of observed → expected); `ignorecase`, `nonword`, `exceptions` | + | `capitalization` | `match`; `exceptions`, `style` (with `$title`), `threshold`, `indicators`, `prefix` | + | `occurrence` | `token`, `max` and/or `min` | + | `repetition` | `tokens`; `alpha`, `ignorecase` | + | `consistency` | `either` (a map of the two acceptable forms) | + | `conditional` | `first`, `second`; `exceptions` | + | `metric` | `formula`, `condition` | + + **What `%%s` fills with depends on the extension point.** Getting this + wrong is the one mistake in this recipe that passes every check below + — the rule fires, the fixtures are green, and only a human reading the + message sees that it is nonsense. + + | extends | `%%s` count | fills with, left to right | + |------------------|------------|---------------------------| + | `existence` | one | the matched text | + | `substitution` | **two** | the **replacement**, then the matched text | + | `capitalization` | one | the scope that failed (the whole heading or sentence) | + + Measured: a `substitution` message with a single `%%s` interpolates the + *replacement*, not the match, so `"Use GitHub not %%s"` against the text + `Github` renders `Use GitHub not GitHub`. + + For the other eight, do not guess. Write the message, run step 6, and + read it back off the finding — no test you can write catches a wrong + `%%s`, so your own eyes on the rendered message are the check. + + ```yaml + # existence — flag these tokens wherever they appear + extends: existence + message: "Avoid '%%s' — it hides the work from the reader" + level: warning + ignorecase: true + tokens: + - simply + - just + ``` + + ```yaml + # substitution — first %%s is the replacement, second is what was found + extends: substitution + message: "Use '%%s' instead of '%%s'" + level: warning + ignorecase: true + swap: + utilize: use + "in order to": to + ``` + + ```yaml + # capitalization — a whole heading must be in sentence case + extends: capitalization + message: "'%%s' should be in sentence case" + level: warning + scope: heading + match: $sentence + exceptions: + - Taskless + - API + ``` + + `match` takes `$sentence`, `$title`, `$lower`, or `$upper`. A literal + string is legal but means "this whole scope must read exactly that", + which is almost never what anyone wants — see step 1. + + **`$sentence` means first word capitalized, everything else lowercase + — proper nouns included.** It is not "sentence case allowing proper + nouns". Measured with `exceptions: [Taskless, API]` on headings: + + | Heading | Result | + |------------------------------------|--------| + | `Getting started with the API` | quiet | + | `Getting started with APIs` | quiet — an exception covers its plural | + | `Taskless and the API` | quiet — an exception may lead the scope | + | `Getting started with Kubernetes` | **fires** — a proper noun you did not list | + | `getting started lowercase` | **fires** — the first word must be capitalized | + | `Getting Started With Title Case` | **fires** | + + So `exceptions` is not decoration: every proper noun, product name and + acronym the docs use has to be listed, or the rule flags correct + headings. Collect them from the docs before writing the rule, and + expect to add to the list. + +3. **Know what you are writing: `tokens` and `swap` keys are patterns, + not literals.** They compile as **Go RE2** regular expressions. + + *This step is about `tokens` and `swap` only. A `capitalization`, + `occurrence` or `metric` rule has neither — skip to step 4.* + + - `(?:…)`, `[…]`, `|`, `+`, `?` all work. + - **Lookahead and lookbehind do not exist in RE2.** A rule that needs + "X but not when followed by Y" cannot be written as a single + `substitution`; split it or narrow with `scope`. + - **Word boundaries are applied for you, around the whole pattern.** + Measured: `Github` does not fire inside `GithubToken`, and the + multi-word `click here` does not fire inside `Clicking here`. + - **Regex metacharacters in a real phrase are live.** "maybe?" is a + pattern meaning "mayb" followed by an optional "e". Escape it. + - **Overlapping alternatives resolve first-wins**, one finding per + match. If `can login` and `login with` both match a sentence, you + get whichever is written first, once — not both. + - `ignorecase: true` matches any casing **and still skips text that + already equals the replacement.** Measured with `Github: GitHub`: + `github` and `Github` are flagged, `GitHub` is not. You do not need + `ignorecase: false` to protect the correct spelling. + - `raw` takes a full regex when `tokens` is too restrictive; + `nonword` removes the implicit boundaries. + + Vale also understands the markup, which decides what counts as text + before your pattern ever runs. Measured in markdown: + + - **URLs and code spans are not prose.** A `Github` key fires on + `Plain Github here` and not on `https://Github.com/x` or + `` `Github/docs` ``. + - **Link text *is* prose.** In `[click here](https://example.com)`, + `click here` is matched — the URL is not. With no `scope`, a rule + fires on both link text and ordinary prose; `scope: link` narrows it + to link text alone. Measured: without a scope the token hit both the + link and the sentence; with `scope: link`, only the link. + +4. **Scope the rule by adding a section to `.taskless/vale/.vale.ini`.** + This is the step that is easy to skip and impossible to notice + skipping. + + A scaffolded project ships **no section at all** — only `StylesPath` + and `MinAlertLevel` — because scope is the author's decision and the + scaffold will not guess it. So the first rule in a project also + writes the first section: + + ```ini + StylesPath = . + MinAlertLevel = suggestion + + [*.md] + BasedOnStyles = + rules.no-simply = YES + ``` + + - `StylesPath = .` resolves **relative to the directory holding + `.vale.ini`**, which is `.taskless/vale/`. That makes the `rules/` + directory beside it a style named `rules`, which is where the + `rules.` prefix comes from. + - `[*.md]` is a glob over paths. Match it to the files the rule is + actually about — `[*.{md,mdx}]`, `[docs/**/*.md]`. Every file the + walk reaches that matches the glob is subject to every rule enabled + under it. + - `BasedOnStyles =` with an empty value is valid and says explicitly + that no bundled style loads. The default is already empty, so this + changes nothing today; write it because it makes the section's + intent readable to the next person, who would otherwise have to know + the default to know whether a style is missing or deliberately off. + - `MinAlertLevel` is a floor. A rule whose `level` sits below it is + enabled and silent. + + If a section already covers the right files, add the rule to it + rather than opening a second one — a later section matching the same + files wins outright. + + **A rule enabled outside a section does nothing.** Put + `rules.no-simply = YES` above the first `[…]` line and Vale does not + error: it prints `W101 'rules.no-simply' isn't a core option` to + stderr, exits zero, and reports no findings. The CLI surfaces that + diagnostic as a notice — if you see it, the assignment is in the + wrong place, not the rule. + +5. **Write the fixtures.** Two directories, both flat. Vale lints the + rule's whole tree, so a document in a nested subdirectory would be + linted but never checked against either bucket — `verify-rule` rejects + one by name rather than skipping it. Keep both buckets one level deep: + + ``` + .taskless/vale/rule-tests//pass/ok.md # rule must stay quiet + .taskless/vale/rule-tests//fail/bad.md # rule must fire + ``` + + Give the fixtures an extension your section's glob matches. A `.txt` + fixture under a `[*.md]` section is never linted, so the `fail/` + document silently passes. + + **The `pass/` bucket is not "correct prose".** Correct prose proves + nothing — the rule was never going to fire on it. Fill it with the + near-misses that would catch an over-broad pattern. What counts as a + near-miss depends on the rule's shape: + + - **`tokens`/`swap` rules** — the noun form you are not flagging, the + word inside a longer word, the term in a URL or a code span, the + correct spelling itself. + - **`scope`d rules** — the same phrase *outside* the scope. A rule + with `scope: link` needs the phrase in ordinary prose; a rule with + `scope: heading` needs it in body text. Without that, nothing proves + the scope is doing anything. + - **`capitalization` rules** — a scope that is entirely exceptions, a + scope whose exception word comes first, and the plural of an + exception. + + That is the half of the fixture set that has to work for you. + +6. **Verify it.** One command, run from the project root: + + ``` + npx @taskless/cli rule verify --json + ``` + + It runs your rule against both buckets in an isolated config — only + your rule enabled, no bundled styles — and answers the question the + fixtures were written to ask: + + ```json + {"engine":"vale","success":true,"ruleId":"no-simply", + "fixtures":"both","missingFailures":[],"unexpectedFindings":[]} + ``` + + - `success: true` and `fixtures: "both"` is the only passing shape. + - `fixtures` other than `"both"` fails without running Vale: a rule + with one bucket has proved half of what it needs to. + - `missingFailures` names `fail/` documents that were not flagged. + - `unexpectedFindings` names `pass/` documents that were. + + Exit code is 0 on success and 1 on any failure, so this one *is* safe + to script. + + If you would rather see the raw findings — the message text, the line + numbers — run `check` against a bucket instead: + + ``` + npx @taskless/cli check .taskless/vale/rule-tests//fail --json + ``` + + Read `results` there, not `success` and not the exit code. `check` + lints the path against the whole config rather than your rule alone, + `success` says the run worked rather than that the fixture behaved, + and the exit code follows severity — a `level: error` rule exits 1 on + `fail/` and a `warning` rule exits 0, both correct. `verify` is the + command that answers pass-or-fail; `check` is the one that shows you + the finding. + + If `fail/` documents show up in `missingFailures`, work down this list + before touching the pattern — the cause is usually further up: + - Is the assignment inside a section? + - Does the section's glob match the fixture's extension? + - Is the rule's `level` at or above `MinAlertLevel`? + - Do the filename, the `rules.` assignment, and the fixture + directory all agree? + - Only then: does the pattern actually match the text? + + Anything in `unexpectedFindings` means the pattern is too broad. Look + for a missing word boundary, an unescaped metacharacter, or a swap key + that also matches the form you meant to allow. + +7. **Report.** Show the three paths you wrote, a one-line summary of + what the rule flags, and which files the section scopes it to — the + scope is a decision the user should see, not one buried in a config. + Note that a whole-project `taskless check` will **not** report your + fixtures: `.taskless/` is excluded from the project walk, by design. + `verify-rule` is what exercises them. + +## Worked rules + +Nine rules that work, each paired with the near-miss that fails. Every +one was run against the bundled Vale; the "what goes wrong" lines are +observed behavior, not warnings in principle. Find the entry closest to +your intent and start there. + +### 1. Ban a word or phrase — `existence` + +> "Our docs shouldn't hedge." + +```yaml +extends: existence +message: "Avoid hedging: '%%s'" +level: warning +ignorecase: true +tokens: + - we think + - it seems + - sort of +``` + +**Goes wrong:** dropping `ignorecase: true` when you meant any casing — +`We think` at the start of a sentence then sails through. And a phrase +with punctuation is a *pattern*: `maybe?` means "mayb" plus an optional +"e", so it matches `mayb`. Escape it: `maybe\?`. + +### 2. Prefer one term over another — `substitution` + +> "Say 'sign in', not 'login', when it's a verb." + +```yaml +extends: substitution +message: "Use '%%s' instead of '%%s'" +level: warning +ignorecase: true +swap: + 'login (?:to|into)': sign in to + 'to login': to sign in +``` + +**Goes wrong:** one `%%s` instead of two. Measured, `"Use sign in not +%%s"` against `login to` renders **"Use sign in not sign in to"** — the +replacement, twice. The rule fires, both fixtures pass, and only a human +reading the message sees it. Two `%%s`, always, in that order. + +### 3. Enforce a product's spelling — `substitution`, not `capitalization` + +> "It's 'GitHub', never 'Github' or 'github'." + +```yaml +extends: substitution +message: "Use '%%s' instead of '%%s'" +level: error +ignorecase: true +swap: + github: GitHub +``` + +**Goes wrong:** reaching for `capitalization` because the complaint is +about capitals. Measured, `match: GitHub` flags whole sentences — +`'We host on Github and it is fine. should be GitHub'` — because that +check tests a *scope*, not a word. Note also that `ignorecase: true` is +safe here: Vale skips text already equal to the replacement, so the +correct `GitHub` is not flagged. + +### 4. Sentence-case headings — `capitalization` + +> "Headings are sentence case; our product names keep their capitals." + +```yaml +extends: capitalization +message: "'%%s' should be in sentence case" +level: warning +scope: heading +match: $sentence +exceptions: + - Taskless + - API + - Kubernetes +``` + +**Goes wrong:** a short `exceptions` list. `$sentence` lowercases +everything after the first word, proper nouns included, so every product +name and acronym in the docs must be listed or correct headings get +flagged. Collect them from the docs first; expect to add more. + +### 5. Restrict a rule to link text — any check, plus `scope` + +> "'click here' is useless link text." + +```yaml +extends: existence +message: "Link text '%%s' says nothing — name the destination" +level: warning +scope: link +ignorecase: true +tokens: + - click here + - read more +``` + +**Goes wrong:** omitting `scope: link`. Measured, the token then fires +on `[click here](…)` **and** on "click here to focus the search box" in +ordinary prose, which is a false positive on a sentence that is fine. +Whenever a rule is about a *place* in the document, the `pass/` fixture +must contain the same phrase outside that place — otherwise nothing +proves the scope works. + +### 6. Cap how often something appears — `occurrence` + +> "At most one exclamation mark per paragraph." + +```yaml +extends: occurrence +message: "Too many exclamation marks" +level: warning +scope: paragraph +token: "!" +max: 1 +``` + +**Goes wrong:** forgetting `scope`. The count is per scope, so with no +scope you are capping the whole document rather than the paragraph. +Note `token` here is singular — this check takes one, not a `tokens` list. + +### 7. Catch a doubled word — `repetition` + +> "'the the' keeps slipping through review." + +```yaml +extends: repetition +message: "'%%s' is repeated" +level: warning +alpha: true +tokens: + - '[^\s]+' +``` + +**Goes wrong:** leaving the pattern unquoted. Measured, an unquoted +`[^\s]+` in YAML silently matches nothing — zero findings, no error, no +diagnostic. Quote any pattern containing a backslash. This is the +failure mode this recipe warns about most, arriving through YAML rather +than through Vale. + +### 8. One spelling or the other, consistently — `consistency` + +> "Pick -ize or -ise and stick to it." + +```yaml +extends: consistency +message: "Use '%%s' consistently" +level: warning +nonword: true +either: + organize: organise +``` + +**Goes wrong:** expecting it to pick a winner. It flags the *second* +form once both appear in a document — it enforces internal consistency, +not house style. If you want one specific spelling, that is a +`substitution`. + +### 9. Require a definition — `conditional` + +> "An acronym must be spelled out before it's used." + +```yaml +extends: conditional +message: "'%%s' has no definition" +level: warning +scope: text +ignorecase: false +first: '\b([A-Z]{3,5})\b' +second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)' +``` + +`first` is what must be justified; `second` is what justifies it. +Measured: `Application Programming Interface (API)` licenses every later +`API`, while an undefined `XYZ` is flagged. + +**Goes wrong:** swapping the two, which inverts the rule into "flag the +definition when the acronym is missing". + +## Important Notes + +- Vale reads one document at a time and has no cross-document view. A + rule about consistency *between* documents cannot be written here. +- Prose inside code is still prose: comments and docstrings are Vale's + subject, and a rule about them belongs in a section whose glob covers + the source files. +- Do NOT add a `[*]` section to widen a rule that isn't firing. It + applies every enabled rule to every file the walk reaches, which turns + one rule's scoping bug into every rule's false positives. + +## See Also + +- `taskless agent route` — re-decide the destination +- `taskless agent check` — run every engine over the repo +- `taskless agent create-sg-rule` — author a rule over code structure diff --git a/packages/cli/src/help/rule-delete.txt b/packages/cli/src/help/delete-rule.txt similarity index 89% rename from packages/cli/src/help/rule-delete.txt rename to packages/cli/src/help/delete-rule.txt index 1bdec306..108ad16c 100644 --- a/packages/cli/src/help/rule-delete.txt +++ b/packages/cli/src/help/delete-rule.txt @@ -1,4 +1,4 @@ -# Topic: rule delete (CLI v%(CLI_VERSION)s / topic v1) +# Topic: delete-rule (CLI v%(CLI_VERSION)s / topic v2) ## Goal Remove a rule and its associated test files from `.taskless/`. Does @@ -44,5 +44,5 @@ emitted from this command. ## See Also -- `taskless help rule create` — make a new rule -- `taskless help check` — run remaining rules to confirm nothing broke +- `taskless agent route` — make a new rule +- `taskless agent check` — run remaining rules to confirm nothing broke diff --git a/packages/cli/src/help/detect.txt b/packages/cli/src/help/detect.txt index a8036054..fdc399dc 100644 --- a/packages/cli/src/help/detect.txt +++ b/packages/cli/src/help/detect.txt @@ -45,9 +45,9 @@ routing flow reads `detect` to decide where a new rule should live. 3. **Use the signals to route.** Feed the output into rule authoring: - A detected linter the repo already uses → author the rule there - (`taskless help existing`). - - No suitable linter, local-only → `taskless help static`. - - See `taskless help route` for the full decision. + (`taskless agent route`). + - No suitable linter, local-only → `taskless agent create-sg-rule`. + - See `taskless agent route` for the full decision. ## Errors @@ -59,6 +59,5 @@ When `--json` is set, failures emit `{ ok: false, code, message }`: ## See Also -- `taskless help route` — decide where to author a rule from these signals -- `taskless help existing` — author a rule in a linter the repo already uses -- `taskless help check` — run rules against the codebase +- `taskless agent route` — decide where to author a rule from these signals +- `taskless agent check` — run rules against the codebase diff --git a/packages/cli/src/help/engine-selection.txt b/packages/cli/src/help/engine-selection.txt deleted file mode 100644 index 15ffed6d..00000000 --- a/packages/cli/src/help/engine-selection.txt +++ /dev/null @@ -1,113 +0,0 @@ -# Topic: engine-selection (CLI v%(CLI_VERSION)s / topic v1) - -## Goal -Decide WHICH ENGINE enforces a rule: `sg` (ast-grep), `vale`, or -`runtime`. This is a question about the shape of the evidence the rule -needs, not about where the rule gets authored and not about how much -the rule is trusted. - -## Preconditions -- A rule intent stated in words ("no `eval`", "don't say 'simply'", - "every exported symbol must appear in the changelog"). -- No auth required. Engine choice is a property of the rule, not of the - user's session. - -## The three engines - -**`sg` — one file's syntax tree.** -Code structure, decided by parsing a single file: a call, an import, a -JSX attribute, a type annotation. Relational correlation counts as -in-file — "a `useEffect` whose dependency array omits a value used in -its body" is still one file's tree, and `sg` handles it. If the rule -can be settled by looking at exactly one file's structure, it is `sg`. - -**`vale` — prose and markup.** -Natural language in documents and in the prose parts of code: docs, -README, comments, commit-message bodies. Word choice, phrasing, -terminology, capitalization, banned or preferred vocabulary. Vale reads -a document, not a syntax tree. - -**`runtime` — everything that needs more than one file, or needs the -world.** -Cross-file and graph questions ("this exported symbol is unused across -the repo"), metadata not in any file's text (git history, ownership, -build outputs), and normalization a parser cannot express (resolving -aliases, following a config chain). If answering the rule requires -holding two files at once, or looking outside the files, it is -`runtime`. - -## Steps - -1. **State the evidence the rule needs, before naming an engine.** - Write one sentence: "to decide this, you must look at ___." Do this - first, every time. Naming an engine and then justifying it is how - prose-about-code ends up in `sg` and cross-file questions end up in - `vale`. - -2. **Match the evidence to an engine.** - - one file's structure → `sg` - - a document's words → `vale` - - more than one file, or anything outside the files → `runtime` - -3. **Check the boundary cases below.** Most wrong answers are one of - the three. - -4. **If it is still ambiguous, choose an engine you know is - available.** State that as the reason. Both `sg` and `vale` ship as - platform binaries with the CLI, so both are normally present; on an - unsupported architecture, or where an install was blocked, the - fallback may find nothing. Server-side the same conclusion follows - from a different cause: `sg` is the only ungated route. Prefer the - engine whose availability you can actually assert in the situation - you are in, rather than defaulting to a name. - -## Boundary cases - -**Prose about code is still prose.** -"Comments must not say 'obviously'" is `vale` — the evidence is the -words. "Every exported function has a doc comment" is `sg` — the -evidence is whether a node exists above a declaration. Ask what you -would have to read to decide, not what the subject matter is. - -**Vale sees one document at a time.** -Vale has no cross-document view. "This term is spelled consistently -ACROSS the docs directory" is `runtime`, not `vale`, even though it is -entirely about prose. Per-document prose is `vale`; prose consistency -between documents is a graph question. - -**Engine is not trust tier.** -`sg` and `vale` are both static-tier: inert data, always run, no login, -no reconcile, no signing. Only `runtime` involves verification, because -only `runtime` executes code. "Static vs runtime" is a different axis -from "which engine", and conflating them is what makes a prose rule -look like it needs login. - -## Worked examples - -| Rule intent | Evidence needed | Engine | -| -------------------------------------------------------- | ---------------------------------------- | --------- | -| No `eval(...)` anywhere | one file's call expressions | `sg` | -| `useEffect` deps must include what the body reads | one file's tree, correlated within it | `sg` | -| Don't write "simply" or "just" in docs | a document's words | `vale` | -| Comments must not say "obviously" | a document's words (comments are prose) | `vale` | -| Headings use sentence case | a document's markup | `vale` | -| Exported symbols must be used somewhere in the repo | every file, correlated | `runtime` | -| Product name spelled the same across all docs | many documents, compared | `runtime` | -| Files changed in the last release need a changelog entry | git metadata, not file contents | `runtime` | -| Imports must resolve through the tsconfig path aliases | config chain resolution, outside the file| `runtime` | - -## Important Notes - -- This topic decides the engine and nothing else. Where a rule is - authored — in a linter the repo already uses, locally, or via the - service — is `route`, and it is a separate decision made earlier. - Locally the two compose: route first, then engine. -- A rule that seems to need two engines is usually two rules. Split it - and let each half pick its own engine, rather than forcing one engine - to cover evidence it cannot see. - -## See Also - -- `taskless help route` — decide WHERE to author a rule (a different question) -- `taskless help static` — author a local ast-grep rule (no login) -- `taskless help check` — run every engine over the repo diff --git a/packages/cli/src/help/rule-improve.anonymous.txt b/packages/cli/src/help/improve-rule.anonymous.txt similarity index 91% rename from packages/cli/src/help/rule-improve.anonymous.txt rename to packages/cli/src/help/improve-rule.anonymous.txt index d2160cbf..e719f4cd 100644 --- a/packages/cli/src/help/rule-improve.anonymous.txt +++ b/packages/cli/src/help/improve-rule.anonymous.txt @@ -1,4 +1,4 @@ -# Topic: rule improve (anonymous) (CLI v%(CLI_VERSION)s / topic v1) +# Topic: improve-rule (anonymous) (CLI v%(CLI_VERSION)s / topic v2) ## Goal Iterate on an existing ast-grep rule **locally** without contacting @@ -59,7 +59,7 @@ validate with `rule verify` in a feedback loop. - The updated rule file path - The updated test file path - A diff-style summary of what changed - Suggest fetching `taskless help check` to validate against the + Suggest fetching `taskless agent check` to validate against the broader codebase. ## Important Notes @@ -83,6 +83,6 @@ The verify primitive returns structured errors per layer: ## See Also -- `taskless help rule improve` — API-backed flow (auth required) -- `taskless help rule create --anonymous` — make a new rule locally -- `taskless help check` — validate the updated rule +- `taskless agent improve-rule` — API-backed flow (auth required) +- `taskless agent create-sg-rule` — make a new rule locally +- `taskless agent check` — validate the updated rule diff --git a/packages/cli/src/help/rule-improve.txt b/packages/cli/src/help/improve-rule.txt similarity index 88% rename from packages/cli/src/help/rule-improve.txt rename to packages/cli/src/help/improve-rule.txt index 5c12f0a9..d727eea7 100644 --- a/packages/cli/src/help/rule-improve.txt +++ b/packages/cli/src/help/improve-rule.txt @@ -1,4 +1,4 @@ -# Topic: rule improve (CLI v%(CLI_VERSION)s / topic v1) +# Topic: improve-rule (CLI v%(CLI_VERSION)s / topic v2) ## Goal Iterate on an existing Taskless rule. The CLI submits the user's @@ -8,7 +8,7 @@ is to gather the right ruleId + guidance + supporting references and to report the result. If the user wants the local-only flow (no API call), fetch -`taskless help rule improve --anonymous` instead. +`taskless agent improve-rule --anonymous` instead. ## Preconditions - User is logged in. @@ -22,7 +22,7 @@ If the user wants the local-only flow (no API call), fetch ## Steps 1. **Confirm auth.** Run `npx @taskless/cli info --json` and check - `loggedIn`. If false, fetch `taskless help auth`. + `loggedIn`. If false, fetch `taskless agent auth`. 2. **Identify the rule to improve.** If the user named one, use it. Otherwise, list rules in `.taskless/sg/rules/` and ask which one. @@ -34,7 +34,7 @@ If the user wants the local-only flow (no API call), fetch ``` This returns the `ticketId` needed for the iterate request. If the metadata is missing, the rule cannot be iterated via API — fetch - `taskless help rule improve --anonymous` instead. + `taskless agent improve-rule --anonymous` instead. 4. **Gather improvement guidance.** Ask the user what should change: - Are there false positives we need to exclude? @@ -69,7 +69,7 @@ If the user wants the local-only flow (no API call), fetch 10. **Report results.** The CLI overwrites the rule file (and its test file) with the updated version. Show the file paths and a - summary of what changed. Suggest fetching `taskless help check` + summary of what changed. Suggest fetching `taskless agent check` to validate. ## Input schema @@ -89,7 +89,7 @@ When `--json` is set, failures emit `{ ok: false, code, message }`: | code | meaning | fix | |--------------------------|----------------------------------------|----------------------------------------------| -| `AUTH_REQUIRED` | not logged in | fetch `taskless help auth` | +| `AUTH_REQUIRED` | not logged in | fetch `taskless agent auth` | | `NO_GITHUB_REMOTE` | no GitHub origin remote | tell the user; we cannot proceed | | `INVALID_INPUT` | `--from` JSON failed validation | re-read input schema, fix, retry | | `RULE_NOT_FOUND` | metadata missing for the given rule | use anonymous variant or recreate via create | @@ -99,6 +99,6 @@ When `--json` is set, failures emit `{ ok: false, code, message }`: ## See Also -- `taskless help rule improve --anonymous` — local-only flow -- `taskless help rule create` — make a new rule from scratch -- `taskless help check` — validate the updated rule +- `taskless agent improve-rule --anonymous` — local-only flow +- `taskless agent route` — make a new rule from scratch +- `taskless agent check` — validate the updated rule diff --git a/packages/cli/src/help/info.txt b/packages/cli/src/help/info.txt index 6870a247..0544de93 100644 --- a/packages/cli/src/help/info.txt +++ b/packages/cli/src/help/info.txt @@ -63,5 +63,5 @@ failing.) ## See Also -- `taskless help auth` — log in / log out / status detail -- `taskless help check` — run rules against the codebase +- `taskless agent auth` — log in / log out / status detail +- `taskless agent check` — run rules against the codebase diff --git a/packages/cli/src/help/init.txt b/packages/cli/src/help/init.txt index a755fb07..04c05a68 100644 --- a/packages/cli/src/help/init.txt +++ b/packages/cli/src/help/init.txt @@ -50,5 +50,5 @@ what was removed. ## See Also -- `taskless help info` — verify what's installed and check staleness -- `taskless help auth` — authenticate after installing +- `taskless agent info` — verify what's installed and check staleness +- `taskless agent auth` — authenticate after installing diff --git a/packages/cli/src/help/onboard.txt b/packages/cli/src/help/onboard.txt index 58105cbf..b1ddc288 100644 --- a/packages/cli/src/help/onboard.txt +++ b/packages/cli/src/help/onboard.txt @@ -15,7 +15,7 @@ rules as a bullet list the user can choose to materialize via the is automatically satisfied. - A working repository the agent can read. - No auth required to surface candidates. (Materializing a rule via - `rule create` may require auth — fetch `taskless help auth` if + `rule create` may require auth — fetch `taskless agent auth` if needed at that point.) ## Steps @@ -91,7 +91,7 @@ rules as a bullet list the user can choose to materialize via the 6. **Offer materialization per bullet.** For each bullet, ask whether the user wants to turn it into a real Taskless rule. On - yes, fetch `taskless help rule create` and follow that recipe — + yes, fetch `taskless agent route` and follow that recipe — the user's accepted bullet becomes the rule description input. 7. **Ask before marking onboarding complete.** When the user signals @@ -117,6 +117,6 @@ rules as a bullet list the user can choose to materialize via the ## See Also -- `taskless help rule create` — materialize an accepted bullet into a real rule -- `taskless help check` — validate newly created rules against the codebase -- `taskless help info` — inspect the current `.taskless/taskless.json` state +- `taskless agent route` — materialize an accepted bullet into a real rule +- `taskless agent check` — validate newly created rules against the codebase +- `taskless agent info` — inspect the current `.taskless/taskless.json` state diff --git a/packages/cli/src/help/remote.txt b/packages/cli/src/help/remote.txt deleted file mode 100644 index 4188ad21..00000000 --- a/packages/cli/src/help/remote.txt +++ /dev/null @@ -1,60 +0,0 @@ -# Topic: remote (CLI v%(CLI_VERSION)s / topic v1) - -## Goal -Generate a rule using the Taskless service. Your job on this path is to -gather the inputs and hand off to `rule create`; the service generates -the rule and writes the standard rule files. This path consumes a -generation and requires login. Reach it when a rule is not reasonably -solvable locally, or when a believed-local attempt failed and the user -confirmed. - -## Preconditions -- `.taskless/` directory exists. -- The user is logged in (this path requires auth). -- The request has been routed here (see `taskless help route`), not - reached by skipping the local-first decision. - -## Steps - -1. **Confirm this is the right path.** You should be here because the - request is not reasonably solvable locally, OR a local `static` - attempt failed and the user confirmed spending a generation. If the - user has not confirmed an escalation from a failed local attempt, get - that confirmation first — the service costs a generation and login. - -2. **Confirm auth.** Run: - ``` - npx @taskless/cli info --json - ``` - Check `loggedIn`. If false, fetch `taskless help auth` and follow the - login recipe before continuing. - -3. **Gather the request.** Collect the rule description plus concrete - success and failure cases. If you arrived here from `static`, reuse - the cases you already gathered. The service uses these to generate the - rule. - -4. **Delegate to the generation backend.** Fetch `taskless help rule - create` and follow it. That recipe builds the request payload and - invokes `rule create`, which submits to the service and writes the - result. Do not re-implement the submission here. - -5. **Report results.** The service writes the generated rule to the same - on-disk paths and shape as a locally authored rule, so `check`, - `improve`, and `verify` treat it identically. Show the user the file - paths and suggest `taskless help check` to validate. - -## Important Notes - -- Do NOT pre-build the rule yourself on this path — submit the request - and let the service generate it. -- The service owns rule-type selection. Today it generates ast-grep - rules written under `.taskless/sg/rules/`, the same shape the local - `static` path produces. - -## See Also - -- `taskless help route` — the local-first routing decision -- `taskless help rule create` — the generation backend this path uses -- `taskless help auth` — log in before generating -- `taskless help check` — validate the generated rule diff --git a/packages/cli/src/help/route.txt b/packages/cli/src/help/route.txt index b39ac71d..aaf7fca6 100644 --- a/packages/cli/src/help/route.txt +++ b/packages/cli/src/help/route.txt @@ -1,23 +1,23 @@ # Topic: route (CLI v%(CLI_VERSION)s / topic v1) ## Goal -Decide where a rule-authoring request should be built: in a linter the -repo ALREADY uses (`existing`), as a local ast-grep rule on this machine -(`static`), or by the Taskless service (`remote`). This is the front -door for "author/write/create a rule" requests. You stay local whenever -you can reasonably build the rule on-device; you only send the user to -the login-gated service when you are not reasonably confident the rule -is locally solvable — or when a believed-local attempt has genuinely -failed and the user confirms. +Turn "write me a rule that…" into one command to run. This is the front +door for every rule-authoring request, and it makes one decision — which +of five recipes authors this rule — from one reading of the evidence. + +That single decision covers both halves that used to be asked +separately: whether the rule can be built here, and which engine can +express it. They are answered from the same signals, so they are +answered together. ## Preconditions - A working repository the agent can read. -- No auth required to route. (The `remote` destination requires login; - the `existing` and `static` destinations do not.) +- No auth required to route. Login state is an *input* to the decision, + not a requirement of making it. ## Steps -1. **Scan the repo.** Run: +1. **Read the repository.** Run: ``` npx @taskless/cli detect --json ``` @@ -36,62 +36,151 @@ failed and the user confirms. } ``` -2. **Write your reasoning BEFORE naming a destination.** Do not pick a - route first and justify it after. Write a short rationale covering: - - what the `detect` signals show (linters, languages, repo rule styles); - - whether an existing linter plausibly already covers this request; - - whether the pattern is expressible as a simple static ast-grep rule; - - your resulting confidence that the request is locally solvable. - The destination you choose in step 3 MUST follow from this rationale. - -3. **Choose the destination as a conclusion of the rationale:** - - - **Both an existing linter AND a local static rule fit** → do NOT - silently pick one. Present both to the user with trade-offs and let - them choose. Note that `remote` consumes a Taskless generation and - requires login, so it is for what cannot be solved locally — not a - default. - - **An existing linter clearly fits** (the repo uses it and it can - express this) → fetch `taskless help existing`. - - **You are reasonably confident it is a simple static ast-grep - pattern** → fetch `taskless help static`. Reasonable confidence is - enough here; you do not need certainty, because step 4 backstops a - wrong-but-reasonable bet. - - **You reasonably believe it CANNOT be solved locally** (a positive - judgment — e.g. it needs cross-file or semantic checks ast-grep can't - express) → fetch `taskless help remote`. Do not manufacture a - deliberate local failure to reach this; route here directly. - - These are three distinct states. If you genuinely cannot tell whether - the request is local or remote — mere uncertainty, not a belief that it - needs the service — ASK the user a clarifying question. Uncertainty is a - reason to ask, never a reason to spend a generation on `remote`. - -4. **Failure fallback (try-verify-escalate).** If you committed to a - local `static` rule on reasonable confidence and it then FAILS - verification against the user's success/failure cases: - - Tell the user the local rule could not capture the cases. - - State that generating via the Taskless service uses a generation - and requires login. - - Call the service only after the user confirms. On yes, fetch - `taskless help remote`. - Never silently fall through from a failed local attempt to a service - call — the confirmation step is required. +2. **Read the login state.** Run: + ``` + npx @taskless/cli info --json + ``` + and note `loggedIn`. Do this now, not later: it changes which + destinations exist, so classifying first means classifying against a + set that may be wrong. + + Reading it is not the same as asking about it. Do **not** open by + offering service generation — at this point neither you nor the user + knows whether this is a two-line pattern or something local authoring + cannot express, so the question costs a turn and cannot be answered + well. + +3. **State the evidence before you name a destination.** Write one + sentence: *"to decide this, you must look at ___."* Then a short + rationale covering what `detect` showed, whether a linter the repo + already runs could express this, and what the rule needs to read. + + Do this first, every time. Naming a destination and justifying it + afterwards is how prose-about-code ends up in `sg` and cross-file + questions end up in `vale`. + +4. **Match the evidence to a destination.** The comparison is made here + and only here — the destination recipes describe their own scope and + deliberately do not restate this table. + + | The rule is decided by… | Destination | Login | + |-----------------------------------------------------------------|----------------------|-------| + | a tool the repo already runs, in that tool's dialect | `create-legacy-rule` | no | + | **one file's syntax tree** — a call, an import, a JSX attribute, a type annotation | `create-sg-rule` | no | + | **a document's words** — docs, README, comments, commit bodies | `create-vale-rule` | no | + | **more than one file, or something outside the files** — the repo graph, git metadata, build output, a resolved config chain | `create-runtime-rule` / `create-remote-rule` | see step 5 | + + Sharpening the three engine rows, because most wrong answers are one + of these: + + - **Relational correlation inside one file is still `sg`.** "A + `useEffect` whose dependency array omits a value used in its body" + is one file's tree. If exactly one file settles it, it is `sg`. + - **Prose about code is still prose.** "Comments must not say + 'obviously'" is `vale` — the evidence is the words. "Every exported + function has a doc comment" is `sg` — the evidence is whether a node + exists above a declaration. Ask what you would have to *read* to + decide, not what the subject matter is. + - **Vale sees one document at a time.** "This term is spelled + consistently ACROSS the docs directory" is a graph question, so it + is runtime — even though it is entirely about prose. + - **Trust tier is not a destination.** `sg` and `vale` are both + static-tier: inert data, always run, no login, no reconcile, no + signing. Only runtime executes code. "Static vs runtime" is a + different axis from "which engine", and conflating them is what + makes a prose rule look like it needs an account. + + Worked examples: + + | Rule intent | Evidence needed | Destination | + |-----------------------------------------------------------|-------------------------------------------|----------------------| + | No `eval(...)` anywhere | one file's call expressions | `create-sg-rule` | + | `useEffect` deps must include what the body reads | one file's tree, correlated within it | `create-sg-rule` | + | Don't write "simply" or "just" in docs | a document's words | `create-vale-rule` | + | Comments must not say "obviously" | a document's words (comments are prose) | `create-vale-rule` | + | Headings use sentence case | a document's markup | `create-vale-rule` | + | Exported symbols must be used somewhere in the repo | every file, correlated | runtime | + | Product name spelled the same across all docs | many documents, compared | runtime | + | Files changed in the last release need a changelog entry | git metadata, not file contents | runtime | + | Imports must resolve through the tsconfig path aliases | config chain resolution, outside the file | runtime | + +5. **Split the runtime row on login state.** Runtime rules execute code, + so they run only against a server-verified signature: + - **logged in** → `create-remote-rule`. Go there directly; no recipe + forwards to another. + - **logged out** → `create-runtime-rule`, which explains the gate. + +6. **Offer the service only where it is a genuine choice.** When the + rule is expressible locally **and** the user is logged in, present + both and let them pick: a local rule you write and verify now, or a + generation that costs against their plan. Say which you would choose + and why. + + Not logged in, or not locally expressible, are not choices — do not + pose them as one. + +7. **Name the command.** Finish by telling the user, or running, the + exact fetch for the destination you chose: + ``` + npx @taskless/cli agent create-vale-rule + ``` + A destination that is not a runnable command is a category, and a + category is not an answer. + +## When two destinations both fit + +- **A linter the repo already runs AND a local Taskless rule both fit** → + do not silently pick. Present both with trade-offs and let the user + choose. +- **A rule that seems to need two engines is usually two rules.** Split + it and route each half, rather than forcing one engine to cover + evidence it cannot see. +- **You genuinely cannot tell** → ask the user a clarifying question. + Uncertainty is a reason to ask; it is never a reason to spend a + generation. + +## Failure fallback (try-verify-escalate) + +If you committed to a local rule on reasonable confidence and it then +fails against the user's cases: + +- Tell the user the local rule could not capture them. +- State that generating via the service uses a generation and requires + login. +- Call the service only after the user confirms. On yes, fetch + `taskless agent create-remote-rule`. + +Never fall through silently from a failed local attempt to a service +call. A developer who watches a local attempt fail reads it as a +Taskless failure, so only attempt local when you reasonably believe it +will work. ## Important Notes -- Reason first, route second. Do not name a destination before the - rationale is written. -- Stay local when you reasonably can. `remote` is the path for what - cannot be solved on-device, and it costs a generation plus login. -- A developer who watches a local attempt fail reads it as a Taskless - failure. Only attempt local when you reasonably believe it will work; - otherwise route `remote` upfront. +- **Reason first, route second.** Do not name a destination before the + rationale in step 3 is written. +- **What the code actually looks like outranks how the request was + phrased.** Where you can see the code or the diff, weigh the concrete + syntactic form present in the repository above the wording. The same + sentence routes to `create-sg-rule` when the repository contains a + form that correlates statically, and to runtime when the equivalent + check would have to normalize a captured value against a declaration + elsewhere. +- **When it stays ambiguous, choose an engine whose availability you can + assert, and say that is why.** There is no fixed fallback. `sg` and + `vale` both ship as platform binaries, so either can be the missing one + on an unsupported architecture or where an install was blocked; + server-side the constraint is different again. A named default is + wrong in whichever situation it failed to anticipate. +- **Stay local when you reasonably can.** Reasonable confidence is + enough to commit to a local rule — the failure fallback backstops a + wrong-but-reasonable bet. ## See Also -- `taskless help existing` — author in a linter the repo already uses -- `taskless help static` — author a local ast-grep rule (no login) -- `taskless help remote` — generate via the Taskless service (login) -- `taskless help engine-selection` — decide WHICH engine enforces the rule - (the next decision once the destination is settled) +- `taskless agent create-legacy-rule` — author in a linter the repo already uses +- `taskless agent create-sg-rule` — author a local ast-grep rule (no login) +- `taskless agent create-vale-rule` — author a local Vale rule (no login) +- `taskless agent create-runtime-rule` — the runtime tier, logged out +- `taskless agent create-remote-rule` — generate via the service (login) +- `taskless agent check` — run every engine over the repo diff --git a/packages/cli/src/help/rule-create.anonymous.txt b/packages/cli/src/help/rule-create.anonymous.txt deleted file mode 100644 index 74c88de1..00000000 --- a/packages/cli/src/help/rule-create.anonymous.txt +++ /dev/null @@ -1,112 +0,0 @@ -# Topic: rule create (anonymous) (CLI v%(CLI_VERSION)s / topic v1) - -## Goal -Create a new ast-grep rule **locally** without contacting the Taskless -API. You (the agent) derive the rule yourself using the ast-grep -schema as a guide, write the rule and test files, then validate them -using `rule verify` in a feedback loop. - -## Preconditions -- `.taskless/` directory exists. -- The agent can read/write files and run shell commands. -- No auth required. - -## Steps - -1. **Learn the ast-grep rule format.** Before writing a rule, consult - the ast-grep rule reference at https://ast-grep.github.io/guide/rule-config.html - for valid fields, operators (`pattern`, `kind`, `regex`, - `any`/`all`/`has`/`inside`/`not`), and meta-variable syntax. The - Taskless-required fields you'll add on top are listed in step 5. - -2. **Gather the rule description.** Even if the user already provided - one, ask clarifying questions: - - What specific code pattern should be flagged? (concrete examples) - - What language is it in? - - Are there exceptions or edge cases where the pattern is OK? - - Can they show valid and invalid examples? - -3. **Check for similar existing rules.** Scan `.taskless/sg/rules/`. If - anything overlaps, point it out and ask whether the user wants to - improve an existing rule via `taskless help rule improve --anonymous`. - -4. **Search the codebase for real instances.** Show the user what you - found and confirm any exclusions. - -5. **Derive the rule.** Write a YAML file at `.taskless/sg/rules/.yml` - with at minimum: - - `id`: kebab-case identifier (e.g. `no-eval`, `prefer-const`) - - `language`: target language - - `severity`: `error`, `warning`, `info`, or `hint` - - `message`: concise single-line explanation - - `rule`: the ast-grep rule object - - Optional but useful: `note` (multi-line guidance, supports markdown), - `fix` (auto-fix pattern), `ignores` (file patterns to skip). - -6. **Write test cases.** Create `.taskless/sg/rule-tests/-YYYYMMDD-test.yml` - with `id`, `valid: [...]`, and `invalid: [...]` arrays. Include at - least 2 valid and 2 invalid cases. Use real patterns from the - codebase where possible. - -7. **Run the verify feedback loop.** Run: - ``` - npx @taskless/cli rule verify --json - ``` - - If `success: true`: the rule passes. Report success to the user. - - If `success: false`: read the per-layer errors (`schema`, - `requirements`, `tests`). Fix the rule or tests. Re-run verify. - Repeat up to 3 times. If still failing after 3 iterations, report - to the user with the latest errors. - - Common fixes: - - Schema errors → check field types against the ast-grep schema. - - Missing required fields → add what the requirements list demands. - - Regex without kind → add a `kind` field alongside any `regex`. - - Test failures → adjust pattern or test cases so valid cases pass - and invalid cases trigger. - -8. **Report results.** Show the rule file path, test file path, and a - one-line summary of what the rule detects. Suggest fetching - `taskless help check` to validate against the broader codebase. - -## Important Notes - -- Do NOT write files to `.taskless/rule-metadata/`. Anonymous rules - have no metadata sidecar (the API path uses metadata for - iteration; anonymous rules iterate directly via file edits). -- Do NOT make any HTTP requests to taskless.io. -- The verify feedback loop is the quality gate — always run it before - reporting success. - -## ast-grep schema - -This recipe does not embed the full ast-grep schema. Read it from the -upstream docs at https://ast-grep.github.io/guide/rule-config.html -before authoring a rule. The Taskless-specific required fields layered -on top of the ast-grep rule are listed in step 5. - -## Errors - -`rule verify --json` has two output shapes depending on whether the -`` argument is supplied: - -- When `` is missing, the command emits the standardized - `{ ok:false, code:"INVALID_INPUT", message }` envelope on stdout - and exits non-zero. -- When `` is supplied, the command emits the layered result - `{ success, schema, requirements, tests, ... }` on stdout (success - or failure exit code depending on the layers). - -| where | code / shape | meaning | fix | -|---------------------|-----------------|------------------------------------------|--------------------------------------------| -| envelope (no ``)| `INVALID_INPUT` | Rule ID missing | Pass a valid rule ID positionally | -| layered: schema | (schema layer) | YAML didn't match ast-grep schema | Fix the rule structure | -| layered: req | (req layer) | Missing Taskless-required field | Add `id`/`language`/`severity`/etc. | -| layered: tests | (tests layer) | A test case didn't behave as expected | Fix the rule pattern OR the test case | - -## See Also - -- `taskless help rule create` — API-backed flow (auth required) -- `taskless help rule improve --anonymous` — iterate locally -- `taskless help check` — validate the new rule against the codebase diff --git a/packages/cli/src/help/rule-create.txt b/packages/cli/src/help/rule-create.txt deleted file mode 100644 index 6cbc60e9..00000000 --- a/packages/cli/src/help/rule-create.txt +++ /dev/null @@ -1,101 +0,0 @@ -# Topic: rule create (CLI v%(CLI_VERSION)s / topic v1) - -## Goal -Generate a new ast-grep rule from a description and write the rule and -its tests to the user's `.taskless/` directory. The CLI calls the -Taskless API to do the generation; the agent's job is to enrich the -user's description before submitting and to report the result. - -If the user wants the local-only flow (no API call), fetch -`taskless help rule create --anonymous` instead. - -## Preconditions -- User is logged in (`taskless info --json` reports `loggedIn: true`). - If not, fetch `taskless help auth` first. -- Repository has a GitHub origin remote. -- `.taskless/` directory exists (otherwise the user has not run - `npx @taskless/cli` to install Taskless). - -## Steps - -1. **Confirm auth.** Run `npx @taskless/cli info --json` and check the - `loggedIn` field. If false, fetch `taskless help auth` and follow the - login recipe before continuing. If the user explicitly wants - anonymous mode, fetch `taskless help rule create --anonymous` instead. - -2. **Gather the rule description.** Even if the user already provided - one, ask clarifying questions: - - What specific code pattern should be flagged? (concrete examples) - - In what language? - - Are there contexts where the pattern is acceptable? - -3. **Check for similar existing rules.** Scan `.taskless/sg/rules/` for - rule files. Read each rule's `message`, `note`, and `rule` fields. - If any overlap with the user's request, show the user and ask: - "It looks like you already have a rule that covers something - similar. Would you like to improve the existing rule instead?" - If yes, fetch `taskless help rule improve` instead. - -4. **Enrich the request.** - - Search the codebase for real instances of the pattern. Show the - user what you found. - - Ask for additional success and failure cases. - - Read `.gitignore`, ESLint configs, and `tsconfig.json` `exclude` - for default exclusion patterns. Present them as defaults. - - Ask about additional exclusions (e.g. `.d.ts` files, tests). - - Infer the primary language from the codebase. Confirm with the - user. Include it in the `prompt` field. - -5. **Confirm the enriched request.** Before submitting, summarize: - - The full prompt (including language and exclusion notes) - - The success case(s) - - The failure case(s) - -6. **Write the JSON payload.** Build a JSON object matching the input - schema below and write it to `.taskless/.tmp-rule-request.json`. - -7. **Invoke the CLI.** Run: - ``` - npx @taskless/cli rule create --from .taskless/.tmp-rule-request.json --json - ``` - This may take 30–60 seconds while the API generates the rule. - -8. **Clean up.** Delete `.taskless/.tmp-rule-request.json` regardless - of success or failure. - -9. **Report results.** The CLI writes the generated rule to - `.taskless/sg/rules/.yml`, tests to - `.taskless/sg/rule-tests/-YYYYMMDD-test.yml` (timestamped per - generation), and metadata to `.taskless/rule-metadata/.yml`. - Show the user the file paths. Suggest fetching `taskless help check` - to validate. - -## Input schema - -The `--from` JSON file conforms to: - -```json -%(INPUT_SCHEMA)s -``` - -Each example in `successCases` and `failureCases` is a separate -string. Multi-line code goes in a single string with literal newlines. - -## Errors - -When `--json` is set, failures emit `{ ok: false, code, message }`: - -| code | meaning | fix | -|--------------------------|------------------------------------|---------------------------------------------| -| `AUTH_REQUIRED` | not logged in | fetch `taskless help auth` | -| `NO_GITHUB_REMOTE` | no GitHub origin remote | tell the user; we cannot proceed | -| `INVALID_INPUT` | `--from` JSON failed validation | re-read the input schema, fix, retry | -| `NETWORK_ERROR` | API submit/poll failed | report and suggest retry | -| `RULE_GENERATION_FAILED` | API returned a generation failure | report the message; suggest enriching prompt | -| `RULE_UNSUPPORTED` | plan lacks this generation type | tell the user to enable it; do not retry | - -## See Also - -- `taskless help rule create --anonymous` — local-only flow (no API) -- `taskless help rule improve` — iterate on an existing rule -- `taskless help check` — validate the new rule against the codebase diff --git a/packages/cli/src/help/rule-meta.txt b/packages/cli/src/help/rule-meta.txt index d9361ef0..023c0c10 100644 --- a/packages/cli/src/help/rule-meta.txt +++ b/packages/cli/src/help/rule-meta.txt @@ -1,4 +1,4 @@ -# Topic: rule meta (CLI v%(CLI_VERSION)s / topic v1) +# Topic: rule-meta (CLI v%(CLI_VERSION)s / topic v2) ## Goal Read sidecar metadata for an API-generated rule. Used internally by @@ -27,4 +27,4 @@ version, etc. ## See Also -- `taskless help rule improve` — the primary consumer of this command +- `taskless agent improve-rule` — the primary consumer of this command diff --git a/packages/cli/src/help/rule.txt b/packages/cli/src/help/rule.txt index 05499fa8..26acc655 100644 --- a/packages/cli/src/help/rule.txt +++ b/packages/cli/src/help/rule.txt @@ -1,22 +1,24 @@ # Topic: rule (CLI v%(CLI_VERSION)s / topic v1) ## Goal -Umbrella for rule operations. Use the specific subcommand for the -action you want. +Umbrella for rule operations. Fetch the topic for the action you want. -## Subcommands +## Topics -| Subcommand | Recipe | -|---------------------|----------------------------------------------| -| Create a rule | `taskless help rule create` | -| Improve a rule | `taskless help rule improve` | -| Delete a rule | `taskless help rule delete` | -| Verify a rule | `taskless help rule verify` (agent-internal) | -| Read rule metadata | `taskless help rule meta` (agent-internal) | +| Action | Recipe | +|---------------------|-----------------------------------------------| +| Create a rule | `taskless agent route` | +| Improve a rule | `taskless agent improve-rule` | +| Delete a rule | `taskless agent delete-rule` | +| Verify a rule | `taskless agent verify-rule` (agent-internal) | +| Read rule metadata | `taskless agent rule-meta` (agent-internal) | -For local-only flows on create/improve, append `--anonymous` to the -help fetch. +`route` is the entry point for authoring: it reads the request and +names the `create-*-rule` topic that fits, so you do not pick an engine +yourself. + +For the local-only flow on improve, append `--anonymous`. ## See Also -- `taskless help check` — run all configured rules +- `taskless agent check` — run all configured rules diff --git a/packages/cli/src/help/static.txt b/packages/cli/src/help/static.txt deleted file mode 100644 index a67f6b26..00000000 --- a/packages/cli/src/help/static.txt +++ /dev/null @@ -1,76 +0,0 @@ -# Topic: static (CLI v%(CLI_VERSION)s / topic v1) - -## Goal -Author a Taskless ast-grep rule **locally**, on this machine, without -contacting the Taskless service. You derive the rule yourself, write it -in the canonical on-disk shape, and validate it with `rule verify` in a -feedback loop. The files you produce match exactly what the service -writes, so `check`, `improve`, and `verify` treat them identically. - -## Preconditions -- `.taskless/` directory exists. -- The agent can read/write files and run shell commands. -- No auth required. - -## Steps - -1. **Learn the ast-grep rule format.** Consult the ast-grep rule - reference at https://ast-grep.github.io/guide/rule-config.html for - valid fields and operators (`pattern`, `kind`, `regex`, - `any`/`all`/`has`/`inside`/`not`) and meta-variable syntax. - -2. **Gather and confirm the pattern.** Make sure you have concrete - success cases (code that should pass) and failure cases (code that - should be flagged), the target language, and any exceptions. Search - the codebase for real instances and confirm exclusions with the user. - -3. **Author the rule in the canonical shape.** Write the rule to - `.taskless/sg/rules/.yml` with at minimum `id` (kebab-case), - `language`, `severity` (`error`/`warning`/`info`/`hint`), `message`, - and the `rule` object. Write tests to - `.taskless/sg/rule-tests/-YYYYMMDD-test.yml` with the matching `id` - field plus `valid` and `invalid` arrays (at least two of each). The - `id` must match the rule's `id` so ast-grep test filtering pairs them. - These paths and shape are the same ones the service writes — do not - invent a different layout. - -4. **Run the verify feedback loop.** Run: - ``` - npx @taskless/cli rule verify --json - ``` - - `success: true` → the rule passes. Go to step 5. - - `success: false` → read the per-layer errors (`schema`, - `requirements`, `tests`), fix the rule or tests, and re-run. Repeat - up to 3 times. - -5. **On success, report.** Show the rule and test file paths and a - one-line summary of what the rule detects. Suggest `taskless help - check` to validate against the broader codebase. - -6. **On failure, escalate via the route fallback — with confirmation.** - If after the feedback loop the rule still cannot capture the user's - cases, this is the try-verify-escalate fallback: - - Delete the candidate `.taskless/sg/rules/.yml` and its test file so - the repo is not left with a broken rule (guaranteed cleanup of the - abandoned candidate). - - Tell the user the local rule could not capture the cases, and that - generating via the Taskless service uses a generation and requires - login. - - Only after the user confirms, fetch `taskless help remote` and - follow it. Do not call the service silently. - -## Important Notes - -- Do NOT make any HTTP requests to taskless.io on this path. -- Do NOT write to `.taskless/rule-metadata/` — local rules have no - metadata sidecar; they iterate via file edits. -- The verify loop is the quality gate. A clean failure is a legitimate - reason to escalate, but only with the user's confirmation (step 6). - -## See Also - -- `taskless help route` — re-decide the destination -- `taskless help remote` — generate via the Taskless service (login) -- `taskless help engine-selection` — confirm `sg` is the right engine for - this rule, rather than `vale` or `runtime` -- `taskless help check` — validate the new rule against the codebase diff --git a/packages/cli/src/help/update.txt b/packages/cli/src/help/update.txt index 21886634..4bf75b3c 100644 --- a/packages/cli/src/help/update.txt +++ b/packages/cli/src/help/update.txt @@ -44,5 +44,5 @@ non-zero with the error message on stderr. ## See Also -- `taskless help init` — interactive variant (wizard with prompts) -- `taskless help info` — verify what's installed and check staleness +- `taskless agent init` — interactive variant (wizard with prompts) +- `taskless agent info` — verify what's installed and check staleness diff --git a/packages/cli/src/help/rule-verify.txt b/packages/cli/src/help/verify-rule.txt similarity index 89% rename from packages/cli/src/help/rule-verify.txt rename to packages/cli/src/help/verify-rule.txt index 4cd7aa8b..04b143e2 100644 --- a/packages/cli/src/help/rule-verify.txt +++ b/packages/cli/src/help/verify-rule.txt @@ -1,4 +1,4 @@ -# Topic: rule verify (CLI v%(CLI_VERSION)s / topic v1) +# Topic: verify-rule (CLI v%(CLI_VERSION)s / topic v2) ## Goal Validate a rule against the ast-grep schema and run its test cases. @@ -54,5 +54,5 @@ attempts and report to the user if still failing. ## See Also -- `taskless help rule create --anonymous` — primary consumer (loop) -- `taskless help rule improve --anonymous` — primary consumer (loop) +- `taskless agent create-sg-rule` — primary consumer (loop) +- `taskless agent improve-rule --anonymous` — primary consumer (loop) diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 0ed1ba36..154bbe4e 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -1,11 +1,11 @@ import { defineCommand, runCommand, showUsage } from "citty"; +import { createAgentCommand } from "./commands/agent"; import { authCommand } from "./commands/auth"; import { checkCommand } from "./commands/check"; import { detectCommand } from "./commands/detect"; import { initCommand, updateCommand } from "./commands/init"; import { infoCommand } from "./commands/info"; -import { createHelpCommand } from "./commands/help"; import { onboardCommand } from "./commands/onboard"; import { ruleCommand } from "./commands/rules"; import { @@ -27,7 +27,7 @@ const subCommands = { rule: ruleCommand, }; -const helpCommand = createHelpCommand(subCommands); +const agentCommand = createAgentCommand(subCommands); const main = defineCommand({ meta: { @@ -55,7 +55,7 @@ const main = defineCommand({ }, subCommands: { ...subCommands, - help: helpCommand, + agent: agentCommand, }, async run({ rawArgs, cmd }) { // citty always calls the parent's run handler, even after a subcommand. @@ -90,7 +90,7 @@ const main = defineCommand({ } // TTY → run the interactive wizard. Non-TTY → print a short preamble - // explaining the context and then delegate to `help` so agents and + // explaining the context and then delegate to `agent` so agents and // pipes see the topic index. if (process.stdout.isTTY === true && process.stdin.isTTY === true) { await runCommand(initCommand, { rawArgs }); @@ -101,11 +101,11 @@ const main = defineCommand({ "Taskless CLI — non-interactive context detected.\n" + " For interactive install, run from a terminal.\n" + " For scripted install, run `taskless init --no-interactive`.\n" + - " For agent recipes, run `taskless help` (no args) for the topic index.\n" + " For agent recipes, run `taskless agent` (no args) for the topic index.\n" ); - // Forward the parent's rawArgs (e.g. `-d `) so the help command + // Forward the parent's rawArgs (e.g. `-d `) so the agent command // doesn't mis-parse them as positional topic names. - await runCommand(helpCommand, { rawArgs: ["help", ...rawArgs] }); + await runCommand(agentCommand, { rawArgs: ["agent", ...rawArgs] }); }, }); diff --git a/packages/cli/src/prompts/index.ts b/packages/cli/src/prompts/index.ts index f7e10175..4a2506c0 100644 --- a/packages/cli/src/prompts/index.ts +++ b/packages/cli/src/prompts/index.ts @@ -11,7 +11,7 @@ import { getRecipe, type RecipeOptions } from "./recipes.js"; * Public entry for `@taskless/cli/prompts`. * * Everything here renders through the same embedded recipe text and the same - * render path `taskless help ` serves, so the two surfaces cannot emit + * render path `taskless agent ` serves, so the two surfaces cannot emit * different guidance. Nothing in this graph reaches the CLI runtime: no citty * command tree, no telemetry, no filesystem or network, so a Worker can import * it without dragging the CLI in behind it. @@ -25,18 +25,24 @@ import { getRecipe, type RecipeOptions } from "./recipes.js"; * {@link INTERNAL_TOPICS} accounts for every canonical recipe on disk. * * The list starts at what a consumer has actually asked for and grows on - * demand. `static` is the canonical on-disk rule shape, the one topic the - * generator's decision router can use server-side. + * demand. It is the authoring recipe for each engine a rule can be routed to, + * so a consumer that can decide a rule belongs to an engine can also reach the + * procedure for writing one. Exporting a chooser without its destinations + * reproduces, for the platform generator, the dead end this surface exists to + * remove. * - * `engine-selection` is exported for the same consumer and the same reason - * (D7 of `add-vale-rule-engine`). Choosing an engine is a decision the service - * already makes — today as a binary `static | runtime` classifier that predates - * Vale, and is therefore now wrong. Exporting the topic is what lets the - * generator render the same guidance `taskless help` serves instead of keeping - * a second copy free to drift, which is the divergence this module exists to - * prevent. + * `engine-selection` used to be exported alongside them. It no longer exists: + * the criterion it carried now lives in `route`, stated once. `route` is not + * exported yet because it still contains local mechanics (`taskless detect`, + * on-device authoring) a Worker cannot run; until it is, a consumer gets each + * destination's own scope from these three and adjudicates genuinely ambiguous + * calls itself. */ -export const TOPICS = ["static", "engine-selection"] as const; +export const TOPICS = [ + "create-sg-rule", + "create-vale-rule", + "create-runtime-rule", +] as const; /** * Recipes deliberately withheld from the export, recorded so they stay visible @@ -46,29 +52,28 @@ export const TOPICS = ["static", "engine-selection"] as const; * subcommand on a developer's machine. There is no caller for them outside * the CLI that hosts those commands. * - Authoring recipes are unreachable server-side: `route` picks an authoring - * destination before the service is involved, `remote` states the boundary - * from the client's side, `detect` documents a CLI subprocess a Worker - * cannot spawn, `existing` targets a local toolchain, and `rule-meta` reads - * a `rule improve` sidecar file. + * destination before the service is involved, `create-remote-rule` states + * the boundary from the client's side, `detect` documents a CLI subprocess a + * Worker cannot spawn, `create-legacy-rule` targets a local toolchain, and + * `rule-meta` reads an `improve-rule` sidecar file. */ export const INTERNAL_TOPICS = [ "auth", "check", "ci", + "create-legacy-rule", + "create-remote-rule", + "delete-rule", "detect", - "existing", + "improve-rule", "info", "init", "onboard", - "remote", "route", "rule", - "rule-create", - "rule-delete", - "rule-improve", "rule-meta", - "rule-verify", "update", + "verify-rule", ] as const; /** A topic name the package exports. Unknown names fail to type-check. */ diff --git a/packages/cli/src/prompts/recipes.ts b/packages/cli/src/prompts/recipes.ts index 423e8417..0815c54a 100644 --- a/packages/cli/src/prompts/recipes.ts +++ b/packages/cli/src/prompts/recipes.ts @@ -58,8 +58,8 @@ export function canonicalRecipeTopics(): string[] { // placeholder, the renderer substitutes the JSON Schema rendered from // this Zod source. const TOPIC_INPUT_SCHEMAS: Record = { - "rule-create": ruleCreateInputSchema, - "rule-improve": ruleImproveInputSchema, + "create-remote-rule": ruleCreateInputSchema, + "improve-rule": ruleImproveInputSchema, }; /** Agent-fill marker used when the caller does not supply a real value. */ diff --git a/packages/cli/src/rules/dispatch.ts b/packages/cli/src/rules/dispatch.ts index e51d1060..ec6eb6c7 100644 --- a/packages/cli/src/rules/dispatch.ts +++ b/packages/cli/src/rules/dispatch.ts @@ -156,7 +156,15 @@ async function runValeEngine(options: DispatchOptions): Promise { }); if (outcome.status === "ok") { - return { engine: "vale", results: outcome.results }; + // A zero-exit run that still wrote to stderr carries a diagnostic — most + // often a rule assignment Vale ignored for sitting outside a section. It + // rides through as a notice and never as a failure: the run succeeded, and + // letting it touch the exit code would fail checks over a warning. + return { + engine: "vale", + results: outcome.results, + ...(outcome.notice === undefined ? {} : { notice: outcome.notice }), + }; } return outcome.blocking ? { engine: "vale", results: [], failure: outcome.message } diff --git a/packages/cli/src/rules/owner.ts b/packages/cli/src/rules/owner.ts new file mode 100644 index 00000000..68509ae8 --- /dev/null +++ b/packages/cli/src/rules/owner.ts @@ -0,0 +1,73 @@ +import { stat } from "node:fs/promises"; +import { join } from "node:path"; + +import { + astGrepRuleFileCandidates, + ENGINE_LAYOUTS, + type EngineName, +} from "./engines"; + +/** + * Which engines have a rule file for `ruleId`. + * + * Ownership is decided by **where the file is**, never by parsing it — the same + * rule `dispatch` follows, so a rule cannot be verified by one engine and run + * by another. `.taskless/vale/rules/.yml` is Vale's; the ast-grep + * candidates (including the pre-`0004` location) are `sg`'s. + * + * Returns every match rather than a single winner. Two engines holding the same + * id is a real state — a project that authored `no-simply` under both — and + * picking one silently would verify a file the user was not asking about. The + * caller reports the ambiguity and names both paths. + * + * A missing directory or a path whose ancestor is a file both read as "not + * here"; anything else is a real IO problem and propagates. + */ +export async function rulefileOwners( + cwd: string, + ruleId: string +): Promise { + const candidates: Array<[EngineName, string[]]> = [ + ["sg", astGrepRuleFileCandidates(cwd, ruleId)], + [ + "vale", + [ + join( + cwd, + ".taskless", + ENGINE_LAYOUTS.vale.rulesDirectory, + `${ruleId}.yml` + ), + ], + ], + ]; + + const owners: EngineName[] = []; + for (const [engine, paths] of candidates) { + for (const path of paths) { + if (await isFile(path)) { + owners.push(engine); + break; + } + } + } + return owners; +} + +/** Whether `path` is an existing regular file. */ +async function isFile(path: string): Promise { + try { + const stats = await stat(path); + return stats.isFile(); + } catch (error) { + const { code } = error as NodeJS.ErrnoException; + if (code === "ENOENT" || code === "ENOTDIR") return false; + throw error; + } +} + +/** Where a rule of each engine lives, for an error message that can be acted on. */ +export function ruleFileLocation(engine: EngineName, ruleId: string): string { + const layout = ENGINE_LAYOUTS[engine]; + return `.taskless/${layout.rulesDirectory}/${ruleId}.yml`; +} diff --git a/packages/cli/src/rules/vale/run.ts b/packages/cli/src/rules/vale/run.ts index a252e5a8..7c1fb093 100644 --- a/packages/cli/src/rules/vale/run.ts +++ b/packages/cli/src/rules/vale/run.ts @@ -55,9 +55,23 @@ export const VALE_TIMEOUT_MS = 60_000; * * `ok` is non-blocking even when it carries findings: severity decides the exit * code there, the same as for every other engine. + * + * `ok` also carries an optional `notice`: whatever Vale wrote to stderr while + * still exiting zero. That combination is not noise. Vale reports a rule + * assignment placed outside any section as `W101 … is ignoring it` — on stderr, + * with exit 0 and a well-formed empty result on stdout — so discarding it + * leaves an author with a rule that verifies, runs, and reports nothing. + * Surfacing it is what makes the section-less scaffold safe: the mistake it + * invites becomes legible instead of silent. */ export type ValeRunOutcome = - | { status: "ok"; blocking: false; results: CheckResult[] } + | { + status: "ok"; + blocking: false; + results: CheckResult[]; + /** Vale's stderr on a zero-exit run, when it wrote any. */ + notice?: string; + } | { status: "unavailable"; blocking: false; message: string } | { status: "timeout"; blocking: true; message: string } | { status: "failed"; blocking: true; message: string }; @@ -214,13 +228,23 @@ export async function runVale( return; } + // Exit was zero, so anything on stderr is a diagnostic about a run that + // otherwise succeeded — the `W101` ignored-assignment warning above all. + // Attached to every `ok` path so a diagnostic cannot be dropped by which + // branch happened to produce the (empty) results. + const diagnostic = stderrChunks.join("").trim(); + const notice = + diagnostic === "" + ? {} + : { notice: `Vale reported while running: ${diagnostic}` }; + const stdout = stdoutChunks.join("").trim(); if (stdout === "") { // Measured: Vale prints `{}` when it finds nothing, which parses and // maps to [] below. This branch is for a Vale that says nothing at all // — cheap insurance against JSON.parse("") reporting a clean run as a // failure. - settle({ status: "ok", blocking: false, results: [] }); + settle({ status: "ok", blocking: false, results: [], ...notice }); return; } @@ -250,6 +274,7 @@ export async function runVale( status: "ok", blocking: false, results: toValeCheckResults(parsed as ValeOutput), + ...notice, }); } catch (error) { settle({ diff --git a/packages/cli/src/schemas/rules-verify.ts b/packages/cli/src/schemas/rules-verify.ts index 39187aab..fa2f9ea1 100644 --- a/packages/cli/src/schemas/rules-verify.ts +++ b/packages/cli/src/schemas/rules-verify.ts @@ -42,6 +42,9 @@ const testLayerResultSchema = layerResultSchema.extend({ }); export const verifyOutputSchema = z.object({ + engine: z + .literal("sg") + .describe("The engine that owns this rule, decided by where its file lives"), success: z.boolean().describe("True if all layers passed"), ruleId: z.string(), schema: layerResultSchema.describe("Layer 1: Zod schema validation"), @@ -51,6 +54,39 @@ export const verifyOutputSchema = z.object({ tests: testLayerResultSchema.describe("Layer 3: sg test execution"), }); +/** + * Vale verification output. + * + * Deliberately not squeezed into the ast-grep shape. `sg` verification is three + * layers over a rule file and its test cases; Vale verification is one question + * asked of two fixture buckets — did every `fail/` document fire, did every + * `pass/` document stay quiet. Mapping the second onto `schema`/`requirements`/ + * `tests` would invent two empty layers and lose the fixture coverage, which is + * the part that catches a rule that was never really verified. + * + * `engine` is the discriminant. A consumer branches on it before reading + * anything else, and adding an engine cannot silently change the meaning of a + * field another engine already emits. + */ +export const valeVerifyOutputSchema = z.object({ + engine: z.literal("vale"), + success: z + .boolean() + .describe("True only when both buckets are populated and both behaved"), + ruleId: z.string(), + fixtures: z + .enum(["both", "pass-only", "fail-only", "none"]) + .describe( + "Which fixture buckets held documents. Only 'both' can succeed: one bucket alone is half a claim" + ), + missingFailures: z + .array(z.string()) + .describe("fail/ fixtures the rule should have flagged and did not"), + unexpectedFindings: z + .array(z.string()) + .describe("pass/ fixtures the rule flagged and should not have"), +}); + export const verifyErrorSchema = z.object({ success: z.literal(false), error: z.string().describe("Error message"), diff --git a/packages/cli/src/telemetry-run.ts b/packages/cli/src/telemetry-run.ts index 3421b929..26607bfb 100644 --- a/packages/cli/src/telemetry-run.ts +++ b/packages/cli/src/telemetry-run.ts @@ -7,8 +7,8 @@ import { CLIError } from "./util/cli-error"; * Derive the cli_run `command` property from the raw argv. Flags (and the * value after `-d`/`--dir`) are skipped; the first positional is the command, * and `rule` keeps its subcommand (e.g. `rule create`) since that distinction - * is meaningful. `help`'s topic is recorded separately on cli_help, so the - * command for a help invocation is just `help`. + * is meaningful. `agent`'s topic is recorded separately on cli_help, so the + * command for an agent invocation is just `agent`. */ export function resolveCommandName(rawArguments: string[]): string { const valueFlags = new Set(["-d", "--dir"]); diff --git a/packages/cli/src/types/errors.ts b/packages/cli/src/types/errors.ts index 9711e6df..efc25b89 100644 --- a/packages/cli/src/types/errors.ts +++ b/packages/cli/src/types/errors.ts @@ -15,6 +15,11 @@ export type CLIErrorCode = | "INVALID_INPUT" | "NETWORK_ERROR" | "SCAN_FAILED" + // An engine binary is not present on this host — an ordinary state on an + // unsupported architecture, and distinct from the engine running and + // failing. Kept apart from SCAN_FAILED because the two ask the caller for + // different things: install something, versus fix something. + | "ENGINE_UNAVAILABLE" | "RECONCILE_FAILED" | "SCAFFOLD_VERSION_MISMATCH" | "SCAFFOLD_CONFLICT" diff --git a/packages/cli/test/anonymous-flag.test.ts b/packages/cli/test/anonymous-flag.test.ts index c763d4f5..4ee35d8b 100644 --- a/packages/cli/test/anonymous-flag.test.ts +++ b/packages/cli/test/anonymous-flag.test.ts @@ -86,7 +86,7 @@ describe("--anonymous flag (per-command behavior matrix)", () => { // file validation. const result = await runCli(["rule", "create", "--anonymous", "-d", cwd]); expect(result.exitCode).not.toBe(0); - expect(result.stderr).toContain("taskless help rule create --anonymous"); + expect(result.stderr).toContain("taskless agent create-sg-rule"); }); it("with --json, emits the standardized envelope", async () => { @@ -115,7 +115,7 @@ describe("--anonymous flag (per-command behavior matrix)", () => { cwd, ]); expect(result.exitCode).not.toBe(0); - expect(result.stderr).toContain("taskless help rule improve --anonymous"); + expect(result.stderr).toContain("taskless agent improve-rule --anonymous"); }); }); diff --git a/packages/cli/test/cli-run.test.ts b/packages/cli/test/cli-run.test.ts index 40aa46bb..c6a1ebe2 100644 --- a/packages/cli/test/cli-run.test.ts +++ b/packages/cli/test/cli-run.test.ts @@ -9,7 +9,7 @@ describe("resolveCommandName", () => { [["check", "--json"], "check"], [["rule", "create"], "rule create"], [["rule"], "rule"], - [["help", "route"], "help"], + [["agent", "route"], "agent"], [["-d", "/tmp", "check"], "check"], [["--dir", "/tmp", "info"], "info"], [[], "(default)"], diff --git a/packages/cli/test/cli.test.ts b/packages/cli/test/cli.test.ts index 92915892..444c3824 100644 --- a/packages/cli/test/cli.test.ts +++ b/packages/cli/test/cli.test.ts @@ -34,7 +34,7 @@ describe("cli", () => { }); describe("no args", () => { - it("shows help text and exits with code 0", async () => { + it("shows the agent topic index and exits with code 0", async () => { const { stdout } = await execFileAsync("node", [binPath]); expect(stdout).toContain("taskless"); expect(stdout).toContain("info"); diff --git a/packages/cli/test/help-extensions.test.ts b/packages/cli/test/help-extensions.test.ts index eca96b91..8ea2d88b 100644 --- a/packages/cli/test/help-extensions.test.ts +++ b/packages/cli/test/help-extensions.test.ts @@ -28,11 +28,11 @@ async function runCli( } } -describe("taskless help (no args)", () => { +describe("taskless agent (no args)", () => { let cwd: string; beforeEach(async () => { - cwd = await mkdtemp(join(tmpdir(), "taskless-help-")); + cwd = await mkdtemp(join(tmpdir(), "taskless-agent-")); }); afterEach(async () => { @@ -40,14 +40,14 @@ describe("taskless help (no args)", () => { }); it("prints the human slug", async () => { - const result = await runCli(["help", "-d", cwd]); + const result = await runCli(["agent", "-d", cwd]); expect(result.exitCode).toBe(0); expect(result.stdout).toContain("For agents:"); expect(result.stdout).toContain("For humans:"); }); it("prints the topic table including all expected topics", async () => { - const result = await runCli(["help", "-d", cwd]); + const result = await runCli(["agent", "-d", cwd]); expect(result.stdout).toContain("Topics:"); for (const topic of ["init", "info", "check", "auth", "rule"]) { expect(result.stdout).toContain(topic); @@ -55,53 +55,96 @@ describe("taskless help (no args)", () => { }); it("mentions the --anonymous flag", async () => { - const result = await runCli(["help", "-d", cwd]); + const result = await runCli(["agent", "-d", cwd]); expect(result.stdout).toContain("--anonymous"); }); it("lists the routing recipe topics under Authoring recipes", async () => { - const result = await runCli(["help", "-d", cwd]); + const result = await runCli(["agent", "-d", cwd]); expect(result.stdout).toContain("Authoring recipes:"); for (const topic of [ "route", - "existing", - "static", - "remote", - "engine-selection", + "create-legacy-rule", + "create-sg-rule", + "create-vale-rule", + "create-runtime-rule", + "create-remote-rule", ]) { expect(result.stdout).toContain(topic); } }); + + it("no longer advertises the removed topic names", async () => { + const result = await runCli(["agent", "-d", cwd]); + // `static`/`existing`/`remote` were renamed and `engine-selection` was + // merged into `route`. Listing a name that resolves to nothing is worse + // than not listing it — the agent spends a fetch to learn it is gone. + for (const removed of ["engine-selection", " existing", " static"]) { + expect(result.stdout).not.toContain(removed); + } + }); }); -describe("taskless help ", () => { +describe("taskless agent ", () => { let cwd: string; beforeEach(async () => { - cwd = await mkdtemp(join(tmpdir(), "taskless-help-routing-")); + cwd = await mkdtemp(join(tmpdir(), "taskless-agent-routing-")); }); afterEach(async () => { await rm(cwd, { recursive: true, force: true }); }); - it.each(["route", "existing", "static", "remote", "engine-selection"])( + it.each([ + "route", + "create-legacy-rule", + "create-sg-rule", + "create-vale-rule", + "create-runtime-rule", + "create-remote-rule", + ])( "resolves the %s recipe without an unknown-topic error", async (topic) => { - const result = await runCli(["help", topic, "-d", cwd]); + const result = await runCli(["agent", topic, "-d", cwd]); expect(result.exitCode).toBe(0); expect(result.stdout).toContain(`# Topic: ${topic}`); expect(result.stdout).toContain("## Goal"); expect(result.stderr).not.toContain("Unknown command"); } ); + + // D9: a reader who arrived at the wrong recipe should find that out in the + // first line, where recovery is a re-decision, rather than after authoring + // the wrong artifact. Fixed shape across all five so it is recognisable. + it.each([ + "create-legacy-rule", + "create-sg-rule", + "create-vale-rule", + "create-runtime-rule", + "create-remote-rule", + ])("opens %s with the orientation banner", async (topic) => { + const result = await runCli(["agent", topic, "-d", cwd]); + expect(result.stdout).toContain("## You are here"); + expect(result.stdout).toContain(`This is \`${topic}\`.`); + expect(result.stdout).toContain("`taskless agent route`"); + }); + + it.each(["existing", "static", "remote", "engine-selection", "rule-create"])( + "no longer resolves the removed topic %s", + async (topic) => { + const result = await runCli(["agent", topic, "-d", cwd]); + expect(result.exitCode).not.toBe(0); + expect(result.stderr).toContain("Unknown command"); + } + ); }); -describe("taskless help ", () => { +describe("taskless agent ", () => { let cwd: string; beforeEach(async () => { - cwd = await mkdtemp(join(tmpdir(), "taskless-help-topic-")); + cwd = await mkdtemp(join(tmpdir(), "taskless-agent-topic-")); }); afterEach(async () => { @@ -109,15 +152,15 @@ describe("taskless help ", () => { }); it("returns the canonical recipe for a known topic", async () => { - const result = await runCli(["help", "rule", "create", "-d", cwd]); + const result = await runCli(["agent", "create-remote-rule", "-d", cwd]); expect(result.exitCode).toBe(0); - expect(result.stdout).toContain("# Topic: rule create"); + expect(result.stdout).toContain("# Topic: create-remote-rule"); expect(result.stdout).toContain("## Goal"); expect(result.stdout).toContain("## Steps"); }); it("interpolates %(CLI_VERSION)s in the recipe header", async () => { - const result = await runCli(["help", "rule", "create", "-d", cwd]); + const result = await runCli(["agent", "create-remote-rule", "-d", cwd]); // Should contain a version pattern, not the literal placeholder. // Guard against both the legacy mustache syntax and the current // sprintf-js named-arg syntax leaking through. @@ -127,7 +170,7 @@ describe("taskless help ", () => { }); it("interpolates %(INPUT_SCHEMA)s for topics with a Zod input", async () => { - const result = await runCli(["help", "rule", "create", "-d", cwd]); + const result = await runCli(["agent", "create-remote-rule", "-d", cwd]); expect(result.stdout).not.toContain("{{INPUT_SCHEMA}}"); expect(result.stdout).not.toContain("%(INPUT_SCHEMA)s"); // Embedded schema includes the JSON Schema $schema URI @@ -137,7 +180,7 @@ describe("taskless help ", () => { }); it("renders %(PACKAGE_MANAGER_DLX)s as the agent-fill marker", async () => { - const result = await runCli(["help", "ci", "-d", cwd]); + const result = await runCli(["agent", "ci", "-d", cwd]); expect(result.exitCode).toBe(0); // Sprintf substitutes the placeholder; the agent-fill marker should // appear in its rendered form, never as the @@ -148,69 +191,91 @@ describe("taskless help ", () => { }); it("exits 1 for an unknown topic", async () => { - const result = await runCli(["help", "totally-unknown", "-d", cwd]); + const result = await runCli(["agent", "totally-unknown", "-d", cwd]); expect(result.exitCode).not.toBe(0); expect(result.stderr).toContain("Unknown command"); }); + + it("points an unknown topic at `taskless agent`, not the removed command", async () => { + const result = await runCli(["agent", "totally-unknown", "-d", cwd]); + expect(result.stderr).toContain("Run `taskless agent` for available"); + }); + + // A topic is one token. The old resolver joined positionals, so + // `rule create` and `create rule` both reached `rule-create.txt` — which + // invited an agent to paraphrase a topic name and still get a hit. Extra + // positionals are now an error rather than something to guess at. + it("rejects a multi-token topic instead of joining the positionals", async () => { + const result = await runCli(["agent", "rule", "create", "-d", cwd]); + expect(result.exitCode).not.toBe(0); + expect(result.stdout).not.toContain("# Topic:"); + expect(result.stderr).toContain("Too many arguments"); + expect(result.stderr).toContain("A topic is a single token"); + }); }); -describe("taskless help --anonymous (variant lookup)", () => { +describe("taskless agent --anonymous (variant lookup)", () => { let cwd: string; beforeEach(async () => { - cwd = await mkdtemp(join(tmpdir(), "taskless-help-anon-")); + cwd = await mkdtemp(join(tmpdir(), "taskless-agent-anon-")); }); afterEach(async () => { await rm(cwd, { recursive: true, force: true }); }); - it("returns the .anonymous variant when one exists (rule create)", async () => { + it("returns the .anonymous variant when one exists (improve-rule)", async () => { const result = await runCli([ - "help", - "rule", - "create", + "agent", + "improve-rule", "--anonymous", "-d", cwd, ]); expect(result.exitCode).toBe(0); // The anonymous recipe declares "(anonymous)" in its header - expect(result.stdout).toContain("# Topic: rule create (anonymous)"); + expect(result.stdout).toContain("(anonymous)"); }); - it("returns the .anonymous variant for rule improve", async () => { - const result = await runCli([ - "help", - "rule", - "improve", + it("falls back to the canonical recipe when no variant exists (check)", async () => { + const canonical = await runCli(["agent", "check", "-d", cwd]); + const anonymous = await runCli(["agent", "check", "--anonymous", "-d", cwd]); + expect(anonymous.exitCode).toBe(0); + // Same body — falls back to check.txt since no check.anonymous.txt + expect(anonymous.stdout).toBe(canonical.stdout); + }); + + // `rule-create.anonymous.txt` used to be the local-only variant of the + // service recipe, which duplicated `static.txt` outright. `create-sg-rule` + // now *is* that path, so an `--anonymous` variant of the remote recipe would + // be "the local version of the remote one" — the contradiction `route` + // resolves. It falls back to the canonical text instead. + it("has no anonymous variant of the service recipe", async () => { + const canonical = await runCli(["agent", "create-remote-rule", "-d", cwd]); + const anonymous = await runCli([ + "agent", + "create-remote-rule", "--anonymous", "-d", cwd, ]); - expect(result.exitCode).toBe(0); - expect(result.stdout).toContain("# Topic: rule improve (anonymous)"); - }); - - it("falls back to the canonical recipe when no variant exists (check)", async () => { - const canonical = await runCli(["help", "check", "-d", cwd]); - const anonymous = await runCli(["help", "check", "--anonymous", "-d", cwd]); expect(anonymous.exitCode).toBe(0); - // Same body — falls back to check.txt since no check.anonymous.txt expect(anonymous.stdout).toBe(canonical.stdout); + expect(anonymous.stdout).not.toContain("(anonymous)"); }); it("returns the canonical recipe when --anonymous is omitted", async () => { - const result = await runCli(["help", "rule", "create", "-d", cwd]); - expect(result.stdout).toContain("# Topic: rule create"); + const result = await runCli(["agent", "improve-rule", "-d", cwd]); + expect(result.stdout).toContain("# Topic: improve-rule"); expect(result.stdout).not.toContain("(anonymous)"); }); }); -describe("bare taskless (non-TTY) routes to help index", () => { +describe("bare taskless (non-TTY) routes to the agent index", () => { it("prints the non-interactive preamble + topic index", async () => { // execFile gives no TTY, which triggers the routing. No flags so - // citty doesn't try to forward them to the help subcommand. + // citty doesn't try to forward them to the agent subcommand. const result = await runCli([]); expect(result.exitCode).toBe(0); expect(result.stderr).toContain("non-interactive context detected"); @@ -218,52 +283,77 @@ describe("bare taskless (non-TTY) routes to help index", () => { }); }); -describe("taskless help engine-selection", () => { +// D1: `engine-selection` was merged into `route`, so the engine reasoning it +// carried has to still be reachable — from one fetch instead of two. These are +// the assertions that used to guard the standalone topic, re-pointed at the +// recipe that now owns them. +describe("taskless agent route (absorbed engine reasoning)", () => { let cwd: string; beforeEach(async () => { - cwd = await mkdtemp(join(tmpdir(), "taskless-help-engine-")); + cwd = await mkdtemp(join(tmpdir(), "taskless-agent-route-")); }); afterEach(async () => { await rm(cwd, { recursive: true, force: true }); }); - it("names all three engines and puts evidence before the answer", async () => { - const result = await runCli(["help", "engine-selection", "-d", cwd]); + it("puts the evidence before the destination", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); expect(result.exitCode).toBe(0); - for (const engine of ["`sg`", "`vale`", "`runtime`"]) { - expect(result.stdout).toContain(engine); - } - expect(result.stdout).toContain("before naming an engine"); + expect(result.stdout).toContain("before you name a destination"); }); - it("carries the three boundary cases", async () => { - const result = await runCli(["help", "engine-selection", "-d", cwd]); + it("carries the boundary cases", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); // Each is a wrong answer someone actually reaches for. expect(result.stdout).toContain("Prose about code is still prose"); expect(result.stdout).toContain("one document at a time"); - expect(result.stdout).toContain("Engine is not trust tier"); + expect(result.stdout).toContain("Trust tier is not a destination"); }); it("states the ambiguity default as a property, not as `sg`", async () => { - const result = await runCli(["help", "engine-selection", "-d", cwd]); - // D7: "choose an engine you know is available" stays correct on an + const result = await runCli(["agent", "route", "-d", cwd]); + // "choose an engine whose availability you can assert" stays correct on an // unsupported arch and server-side alike; naming `sg` outright would be // false on the host where `sg` is the missing one. - expect(result.stdout).toContain("choose an engine you know is"); - expect(result.stdout).toContain("available"); + expect(result.stdout).toContain("whose availability you can"); + expect(result.stdout).toContain("There is no fixed fallback"); + }); + + it("weighs the concrete form above the wording of the request", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); + expect(result.stdout).toContain("outranks how the request was"); + }); + + it("names all five destinations", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); + for (const destination of [ + "create-legacy-rule", + "create-sg-rule", + "create-vale-rule", + "create-runtime-rule", + "create-remote-rule", + ]) { + expect(result.stdout).toContain(destination); + } + }); + + it("splits the runtime destination on login state", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); + // D7: reading login state early is what makes the destination set correct. + expect(result.stdout).toContain("info --json"); + expect(result.stdout).toContain("logged out"); }); - it("stays out of the authoring-destination decision", async () => { - const result = await runCli(["help", "engine-selection", "-d", cwd]); - // Scope guard (3.4): it may point at `route`, never re-decide it. - expect(result.stdout).toContain("is `route`, and it is a separate"); + it("sends the reader to a command rather than a category", async () => { + const result = await runCli(["agent", "route", "-d", cwd]); + expect(result.stdout).toMatch(/agent create-vale-rule/); }); it("follows the recipe header and section convention", async () => { - const result = await runCli(["help", "engine-selection", "-d", cwd]); - expect(result.stdout).toContain("# Topic: engine-selection"); + const result = await runCli(["agent", "route", "-d", cwd]); + expect(result.stdout).toContain("# Topic: route"); for (const section of [ "## Goal", "## Preconditions", diff --git a/packages/cli/test/help-routing-telemetry.test.ts b/packages/cli/test/help-routing-telemetry.test.ts index c3925b4b..a391ae01 100644 --- a/packages/cli/test/help-routing-telemetry.test.ts +++ b/packages/cli/test/help-routing-telemetry.test.ts @@ -1,6 +1,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -// Spy on the telemetry capture by mocking the telemetry module the help +// Spy on the telemetry capture by mocking the telemetry module the agent // command imports. The factory is invoked lazily at import time, so the // closure over `capture` resolves after initialization (same pattern as // telemetry.test.ts mocking posthog-node). @@ -15,7 +15,7 @@ vi.mock("../src/telemetry", () => ({ shutdownTelemetry: () => Promise.resolve(), })); -const { createHelpCommand } = await import("../src/commands/help"); +const { createAgentCommand } = await import("../src/commands/agent"); interface RunnableCommand { run: (context: { @@ -24,7 +24,7 @@ interface RunnableCommand { }) => Promise; } -describe("help routing topics emit cli_help intent telemetry", () => { +describe("agent routing topics emit cli_help intent telemetry", () => { let logSpy: ReturnType; beforeEach(() => { @@ -40,10 +40,10 @@ describe("help routing topics emit cli_help intent telemetry", () => { it.each(["route", "existing", "static", "remote", "engine-selection"])( "captures cli_help for %s", async (topic) => { - const command = createHelpCommand({}) as unknown as RunnableCommand; + const command = createAgentCommand({}) as unknown as RunnableCommand; await command.run({ args: { dir: process.cwd(), anonymous: false }, - rawArgs: ["help", topic], + rawArgs: ["agent", topic], }); expect(capture).toHaveBeenCalledWith( diff --git a/packages/cli/test/help-telemetry.test.ts b/packages/cli/test/help-telemetry.test.ts index 5e11937e..b103c238 100644 --- a/packages/cli/test/help-telemetry.test.ts +++ b/packages/cli/test/help-telemetry.test.ts @@ -3,7 +3,7 @@ import { join, resolve } from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -// Spy on telemetry by mocking the module the help command imports. The factory +// Spy on telemetry by mocking the module the agent command imports. The factory // is invoked lazily at import time (same pattern as telemetry.test.ts). const capture = vi.fn(); vi.mock("../src/telemetry", () => ({ @@ -13,7 +13,7 @@ vi.mock("../src/telemetry", () => ({ shutdownTelemetry: () => Promise.resolve(), })); -const { createHelpCommand } = await import("../src/commands/help"); +const { createAgentCommand } = await import("../src/commands/agent"); interface RunnableCommand { run: (context: { @@ -22,15 +22,15 @@ interface RunnableCommand { }) => Promise; } -async function runHelp(rawArguments: string[]): Promise { - const command = createHelpCommand({}) as unknown as RunnableCommand; +async function runAgent(rawArguments: string[]): Promise { + const command = createAgentCommand({}) as unknown as RunnableCommand; await command.run({ args: { dir: process.cwd(), anonymous: false }, rawArgs: rawArguments, }); } -describe("help emits cli_help { topic }", () => { +describe("agent emits cli_help { topic }", () => { let logSpy: ReturnType; let errorSpy: ReturnType; @@ -46,19 +46,27 @@ describe("help emits cli_help { topic }", () => { }); it("captures the served topic", async () => { - await runHelp(["help", "rule", "create"]); - expect(capture).toHaveBeenCalledWith("cli_help", { topic: "rule create" }); + await runAgent(["agent", "rule-create"]); + expect(capture).toHaveBeenCalledWith("cli_help", { topic: "rule-create" }); }); it("captures the index marker for no topic", async () => { - await runHelp(["help"]); + await runAgent(["agent"]); expect(capture).toHaveBeenCalledWith("cli_help", { topic: "(index)" }); }); it("captures the attempted topic for an unknown topic", async () => { - await runHelp(["help", "nope"]); + await runAgent(["agent", "nope"]); expect(capture).toHaveBeenCalledWith("cli_help", { topic: "nope" }); }); + + // The rejected invocation is still an intent signal — it says an agent + // reached for a topic by phrase rather than by token — so it is captured + // under the same event with the attempted words joined for readability. + it("captures the attempted words when too many positionals are given", async () => { + await runAgent(["agent", "rule", "create"]); + expect(capture).toHaveBeenCalledWith("cli_help", { topic: "rule create" }); + }); }); // Rather than asserting "no help_* event" inside every behavioral test above, diff --git a/packages/cli/test/mixed-engine-check.test.ts b/packages/cli/test/mixed-engine-check.test.ts index c60995f2..14a51d22 100644 --- a/packages/cli/test/mixed-engine-check.test.ts +++ b/packages/cli/test/mixed-engine-check.test.ts @@ -271,6 +271,92 @@ describe("check over a project with both engines", () => { await rm(scaffold, { recursive: true, force: true }); } }); + + // The pairing that keeps the section-less scaffold safe. Dropping `[*]` + // from the scaffold means scope is the author's decision, which is right, + // and it also means the first rule someone writes can land its assignment + // above every matcher. Vale does not error on that: it warns on stderr, + // exits zero, and returns a well-formed empty result, which is + // indistinguishable from a clean run. Surfacing the warning is the only + // thing standing between that mistake and a silently disabled rule. + it("surfaces Vale's W101 when an assignment sits outside every matcher", async () => { + const scaffold = await mkdtemp(join(tmpdir(), "taskless-w101-")); + try { + const init = await runCli(["init", "--no-interactive", "-d", scaffold]); + expect(init.exitCode).toBe(0); + + const valeDirectory = join(scaffold, ".taskless", "vale"); + await writeFile( + join(valeDirectory, "rules", "no-simply.yml"), + "extends: existence\nmessage: \"Avoid 'simply'\"\nlevel: warning\ntokens:\n - simply\n" + ); + // The mistake: enabled, but above the first `[…]` line, so it belongs + // to no matcher. + const config = await readFile(join(valeDirectory, ".vale.ini"), "utf8"); + await writeFile( + join(valeDirectory, ".vale.ini"), + `${config}\nrules.no-simply = YES\n` + ); + await writeFile(join(scaffold, "doc.md"), "Just simply do it.\n"); + + const { stdout, stderr, exitCode } = await runCli([ + "check", + "-d", + scaffold, + ]); + + // Advisory, so the run still succeeds: nothing is broken, something is + // misplaced. Failing here would make an ignorable warning block a check. + expect(exitCode).toBe(0); + expect(`${stdout}${stderr}`).toContain("W101"); + expect(`${stdout}${stderr}`).toContain("no-simply"); + } finally { + await rm(scaffold, { recursive: true, force: true }); + } + }); + + // The other half: a rule file with no matcher enabling it at all. This is + // the state a scaffold plus a half-finished rule leaves behind, and `check` + // must neither report it nor fail on it — reporting would be a lie and + // failing would break a project mid-authoring. + it("reports nothing and does not fail when no matcher enables the rule", async () => { + const scaffold = await mkdtemp(join(tmpdir(), "taskless-unscoped-")); + try { + const init = await runCli(["init", "--no-interactive", "-d", scaffold]); + expect(init.exitCode).toBe(0); + await writeFile( + join(scaffold, ".taskless", "vale", "rules", "no-simply.yml"), + "extends: existence\nmessage: \"Avoid 'simply'\"\nlevel: warning\ntokens:\n - simply\n" + ); + await writeFile(join(scaffold, "doc.md"), "Just simply do it.\n"); + + const quiet = await runCli(["check", "-d", scaffold, "--json"]); + expect(quiet.exitCode).toBe(0); + expect( + (JSON.parse(quiet.stdout.trim()) as CheckOutput).results.filter( + (finding) => finding.source === "vale" + ) + ).toEqual([]); + + // Adding the matcher is the whole difference. Same rule, same + // document, now scoped. + const configPath = join(scaffold, ".taskless", "vale", ".vale.ini"); + await writeFile( + configPath, + `${await readFile(configPath, "utf8")}\n[*.md]\nBasedOnStyles =\nrules.no-simply = YES\n` + ); + + const loud = await runCli(["check", "-d", scaffold, "--json"]); + expect( + (JSON.parse(loud.stdout.trim()) as CheckOutput).results.some( + (finding) => + finding.source === "vale" && finding.ruleId === "no-simply" + ) + ).toBe(true); + } finally { + await rm(scaffold, { recursive: true, force: true }); + } + }); }); describe("whatever the host provides", () => { diff --git a/packages/cli/test/onboard.test.ts b/packages/cli/test/onboard.test.ts index 298547da..c7afc258 100644 --- a/packages/cli/test/onboard.test.ts +++ b/packages/cli/test/onboard.test.ts @@ -182,7 +182,7 @@ describe("taskless onboard", () => { expect(stderr).toContain("--mark-complete"); }); - it("`taskless help onboard` matches the recipe printed by `taskless onboard --force`", async () => { + it("`taskless agent onboard` matches the recipe printed by `taskless onboard --force`", async () => { // Pre-mark onboarded so the `onboard` path also prints the recipe via // --force, ensuring we compare recipe-vs-recipe rather than gate-vs-recipe. await mkdir(join(cwd, ".taskless"), { recursive: true }); @@ -192,11 +192,11 @@ describe("taskless onboard", () => { "utf8" ); - const help = await runCli(["help", "onboard", "-d", cwd], cwd); + const agentRecipe = await runCli(["agent", "onboard", "-d", cwd], cwd); const onboard = await runCli(["onboard", "--force", "-d", cwd], cwd); - expect(help.exitCode).toBe(0); + expect(agentRecipe.exitCode).toBe(0); expect(onboard.exitCode).toBe(0); - expect(onboard.stdout.trim()).toBe(help.stdout.trim()); + expect(onboard.stdout.trim()).toBe(agentRecipe.stdout.trim()); }); }); diff --git a/packages/cli/test/prompts.test.ts b/packages/cli/test/prompts.test.ts index 8c35a769..9f365cd7 100644 --- a/packages/cli/test/prompts.test.ts +++ b/packages/cli/test/prompts.test.ts @@ -30,12 +30,12 @@ const UNRESOLVED_PLACEHOLDER = /%\([A-Z_]+\)s/; * something the bundler or the type-checker resolves back to source. */ async function importBuiltPrompts(): Promise<{ - getPrompt: (topic: "static", options?: PromptOptions) => string; + getPrompt: (topic: "create-sg-rule", options?: PromptOptions) => string; TOPICS: readonly string[]; }> { const url = pathToFileURL(distributionPromptsPath).href; return (await import(/* @vite-ignore */ url)) as { - getPrompt: (topic: "static", options?: PromptOptions) => string; + getPrompt: (topic: "create-sg-rule", options?: PromptOptions) => string; TOPICS: readonly string[]; }; } @@ -62,12 +62,12 @@ describe("prompt rendering", () => { }); it("renders the CLI version into the header", () => { - expect(getPrompt("static")).toContain(`CLI v${__VERSION__}`); + expect(getPrompt("create-sg-rule")).toContain(`CLI v${__VERSION__}`); }); it.each([ - ["rule-create", "prompt"], - ["rule-improve", "ruleId"], + ["create-remote-rule", "prompt"], + ["improve-rule", "ruleId"], ])("renders the JSON Schema for %s", (topic, property) => { const rendered = getRecipe(topic) ?? ""; expect(rendered, `${topic} lost its schema`).not.toContain( @@ -95,10 +95,10 @@ describe("prompt rendering", () => { describe("header suppression", () => { it("drops the header line and the blank line after it, leaving the body intact", () => { - const withHeader = getPrompt("static"); - const withoutHeader = getPrompt("static", { header: false }); + const withHeader = getPrompt("create-sg-rule"); + const withoutHeader = getPrompt("create-sg-rule", { header: false }); - expect(withHeader.startsWith("# Topic: static")).toBe(true); + expect(withHeader.startsWith("# Topic: create-sg-rule")).toBe(true); expect(withoutHeader.startsWith("# Topic:")).toBe(false); // The body is the same string, minus the header line and its blank line. expect(withoutHeader).toBe(withHeader.split("\n").slice(2).join("\n")); @@ -117,22 +117,26 @@ describe("header suppression", () => { }); it("keeps the header by default", () => { - expect(getPrompt("static")).toBe(getPrompt("static", { header: true })); - expect(getPrompt("static")).toBe(getPrompt("static", {})); + expect(getPrompt("create-sg-rule")).toBe( + getPrompt("create-sg-rule", { header: true }) + ); + expect(getPrompt("create-sg-rule")).toBe(getPrompt("create-sg-rule", {})); }); }); describe("anonymous variants", () => { it("returns the variant text for a topic that has one", () => { - const canonical = getRecipe("rule-create"); - const anonymous = getRecipe("rule-create", { anonymous: true }); + const canonical = getRecipe("improve-rule"); + const anonymous = getRecipe("improve-rule", { anonymous: true }); expect(anonymous).toBeDefined(); expect(anonymous).not.toBe(canonical); expect(anonymous).toContain("(anonymous)"); }); it("falls back to the canonical recipe for a topic without one", () => { - expect(getRecipe("static", { anonymous: true })).toBe(getRecipe("static")); + expect(getRecipe("create-sg-rule", { anonymous: true })).toBe( + getRecipe("create-sg-rule") + ); }); }); @@ -145,8 +149,8 @@ describe("typed accessor", () => { }); it("passes options through the map", () => { - expect(PROMPTS.static({ header: false })).toBe( - getPrompt("static", { header: false }) + expect(PROMPTS["create-sg-rule"]({ header: false })).toBe( + getPrompt("create-sg-rule", { header: false }) ); }); @@ -156,11 +160,11 @@ describe("typed accessor", () => { }); }); -describe("help command parity", () => { +describe("agent command parity", () => { it.each([...TOPICS])( - "matches `taskless help %s` byte for byte", + "matches `taskless agent %s` byte for byte", async (topic) => { - const { stdout } = await execFileAsync("node", [binPath, "help", topic]); + const { stdout } = await execFileAsync("node", [binPath, "agent", topic]); // The command trims trailing whitespace before printing; console.log then // adds the single newline that stdout carries. expect(stdout.trimEnd()).toBe(getPrompt(topic).trimEnd()); @@ -220,7 +224,7 @@ describe("built prompts entry", () => { it("inlines every build define", async () => { const { getPrompt: getBuiltPrompt } = await importBuiltPrompts(); - const rendered = getBuiltPrompt("static"); + const rendered = getBuiltPrompt("create-sg-rule"); expect(rendered).toContain(`CLI v${__VERSION__}`); expect(rendered).not.toMatch(/__[A-Z_]+__/); expect(rendered).not.toMatch(UNRESOLVED_PLACEHOLDER); @@ -228,9 +232,9 @@ describe("built prompts entry", () => { it("renders identically from the built artifact and from source", async () => { const { getPrompt: getBuiltPrompt } = await importBuiltPrompts(); - expect(getBuiltPrompt("static")).toBe(getPrompt("static")); - expect(getBuiltPrompt("static", { header: false })).toBe( - getPrompt("static", { header: false }) + expect(getBuiltPrompt("create-sg-rule")).toBe(getPrompt("create-sg-rule")); + expect(getBuiltPrompt("create-sg-rule", { header: false })).toBe( + getPrompt("create-sg-rule", { header: false }) ); }); }); diff --git a/packages/cli/test/recipe-cross-references.test.ts b/packages/cli/test/recipe-cross-references.test.ts new file mode 100644 index 00000000..8311da9d --- /dev/null +++ b/packages/cli/test/recipe-cross-references.test.ts @@ -0,0 +1,85 @@ +import { readdir, readFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { describe, expect, it } from "vitest"; + +/** + * Recipes cite each other by topic name, in prose. Nothing resolves those + * citations at build time — they are strings an agent reads and then types — + * so a renamed topic leaves a reference that stays valid-looking and fails + * only when an agent runs it and gets a non-zero exit and no recipe. + * + * That is the failure mode the `taskless help` → `taskless agent` rename + * created 90 opportunities for. These read the recipe sources, which are the + * authority for what ships, rather than a built bundle: the bundler resolves + * imports, and a prose cross-reference is not one, so there is no structured + * answer to ask it for. + */ +const helpDirectory = resolve(import.meta.dirname, "../src/help"); + +async function recipeFiles(): Promise { + const entries = await readdir(helpDirectory); + return entries.filter((entry) => entry.endsWith(".txt")).toSorted(); +} + +/** Topic names that resolve, i.e. `.txt` exists in the help directory. */ +async function embeddedTopics(): Promise> { + const files = await recipeFiles(); + return new Set( + files + .map((file) => file.replace(/\.txt$/, "")) + .map((name) => name.replace(/\.anonymous$/, "")) + ); +} + +describe("shipped recipes name only commands that exist", () => { + it("contains no reference to the former `taskless help` command", async () => { + const offenders: string[] = []; + for (const file of await recipeFiles()) { + const source = await readFile(join(helpDirectory, file), "utf8"); + for (const [index, line] of source.split("\n").entries()) { + if (line.includes("taskless help")) { + offenders.push(`${file}:${String(index + 1)}: ${line.trim()}`); + } + } + } + expect(offenders).toEqual([]); + }); + + // A topic that does not resolve is the expensive case: the agent has already + // committed to a plan that routes through it before finding out. + it("cites only topics that resolve to an embedded recipe", async () => { + const topics = await embeddedTopics(); + const dangling: string[] = []; + + for (const file of await recipeFiles()) { + const source = await readFile(join(helpDirectory, file), "utf8"); + for (const [index, line] of source.split("\n").entries()) { + // `taskless agent `, however it is punctuated around. + for (const match of line.matchAll(/taskless agent ([a-z][a-z-]*)/g)) { + const topic = match[1]; + if (topic !== undefined && !topics.has(topic)) { + dangling.push(`${file}:${String(index + 1)} cites '${topic}'`); + } + } + } + } + + expect(dangling).toEqual([]); + }); + + // Every recipe is reachable by the name its filename implies. A header that + // disagrees with the filename means `taskless agent ` prints a recipe + // announcing itself as something else. + it("opens each recipe with a header naming its own topic", async () => { + const mismatched: string[] = []; + for (const file of await recipeFiles()) { + const source = await readFile(join(helpDirectory, file), "utf8"); + const topic = file.replace(/(\.anonymous)?\.txt$/, ""); + const firstLine = source.split("\n")[0] ?? ""; + if (!firstLine.startsWith(`# Topic: ${topic}`)) { + mismatched.push(`${file}: ${firstLine}`); + } + } + expect(mismatched).toEqual([]); + }); +}); diff --git a/packages/cli/test/rule-verify-dispatch.test.ts b/packages/cli/test/rule-verify-dispatch.test.ts new file mode 100644 index 00000000..926abccc --- /dev/null +++ b/packages/cli/test/rule-verify-dispatch.test.ts @@ -0,0 +1,277 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { findValeBinary } from "../src/rules/vale/binary"; + +const execFileAsync = promisify(execFile); +const binPath = resolve(import.meta.dirname, "../dist/index.js"); + +/** + * Vale ships as a platform binary, so a host without one (an unsupported arch, + * a blocked install) cannot run these. Skipping is right where the alternative + * is a suite that fails for a reason unrelated to the code under test. + */ +const withVale = findValeBinary().path === undefined ? describe.skip : describe; + +async function runCli( + args: string[] +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args]); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const execError = error as { stdout: string; stderr: string; code: number }; + return { + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + exitCode: execError.code, + }; + } +} + +const VALE_RULE = `extends: existence +message: "Avoid %s" +level: warning +tokens: + - simply +`; + +const VALE_CONFIG = `StylesPath = . +MinAlertLevel = suggestion + +[*.md] +BasedOnStyles = +rules.no-simply = YES +`; + +/** Scaffold a project with a Vale rule and both fixture buckets populated. */ +async function scaffoldValeRule(cwd: string): Promise { + const vale = join(cwd, ".taskless", "vale"); + await mkdir(join(vale, "rules"), { recursive: true }); + await mkdir(join(vale, "rule-tests", "no-simply", "pass"), { + recursive: true, + }); + await mkdir(join(vale, "rule-tests", "no-simply", "fail"), { + recursive: true, + }); + await writeFile(join(vale, "rules", "no-simply.yml"), VALE_RULE); + await writeFile(join(vale, ".vale.ini"), VALE_CONFIG); + await writeFile( + join(vale, "rule-tests", "no-simply", "fail", "bad.md"), + "You simply do it.\n" + ); + await writeFile( + join(vale, "rule-tests", "no-simply", "pass", "ok.md"), + "You do it.\n" + ); +} + +interface ValeVerifyJson { + engine: string; + success: boolean; + ruleId: string; + fixtures: string; + missingFailures: string[]; + unexpectedFindings: string[]; +} + +describe("rule verify dispatches by owning engine", () => { + let cwd: string; + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-verify-dispatch-")); + }); + + afterEach(async () => { + await rm(cwd, { recursive: true, force: true }); + }); + + // The id lives under both engines. Verifying one silently would report on a + // file the caller may not have meant, so this is an error naming both. + it("refuses an id that exists under two engines", async () => { + await scaffoldValeRule(cwd); + await mkdir(join(cwd, ".taskless", "sg", "rules"), { recursive: true }); + await writeFile( + join(cwd, ".taskless", "sg", "rules", "no-simply.yml"), + "id: no-simply\nlanguage: TypeScript\nseverity: error\nmessage: x\nrule:\n pattern: eval($A)\n" + ); + + const result = await runCli(["rule", "verify", "no-simply", "-d", cwd]); + expect(result.exitCode).not.toBe(0); + expect(result.stderr).toContain("more than one engine"); + expect(result.stderr).toContain(".taskless/sg/rules/no-simply.yml"); + expect(result.stderr).toContain(".taskless/vale/rules/no-simply.yml"); + }); + + it("tags the ast-grep result with its engine", async () => { + await mkdir(join(cwd, ".taskless", "sg", "rules"), { recursive: true }); + await writeFile( + join(cwd, ".taskless", "sg", "rules", "no-eval.yml"), + "id: no-eval\nlanguage: TypeScript\nseverity: error\nmessage: no eval\nrule:\n pattern: eval($$$A)\n" + ); + + const result = await runCli([ + "rule", + "verify", + "no-eval", + "-d", + cwd, + "--json", + ]); + const parsed = JSON.parse(result.stdout) as { engine: string }; + expect(parsed.engine).toBe("sg"); + }); +}); + +withVale("rule verify enforces the Vale fixture layout", () => { + let cwd: string; + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-vale-verify-")); + await scaffoldValeRule(cwd); + }); + + afterEach(async () => { + await rm(cwd, { recursive: true, force: true }); + }); + + it("passes a rule that fires on fail/ and stays quiet on pass/", async () => { + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).toBe(0); + const parsed = JSON.parse(result.stdout) as ValeVerifyJson; + expect(parsed.engine).toBe("vale"); + expect(parsed.success).toBe(true); + expect(parsed.fixtures).toBe("both"); + }); + + // A one-sided rule is the failure this enforcement exists for. With only + // `fail/` fixtures the rule was never shown not to over-fire, and reporting + // that as a pass is how an unverified rule ships looking verified. + it("fails a rule with only fail/ fixtures", async () => { + await rm(join(cwd, ".taskless", "vale", "rule-tests", "no-simply", "pass"), { + recursive: true, + force: true, + }); + + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).not.toBe(0); + const parsed = JSON.parse(result.stdout) as ValeVerifyJson; + expect(parsed.success).toBe(false); + expect(parsed.fixtures).toBe("fail-only"); + }); + + it("fails a rule with only pass/ fixtures", async () => { + await rm(join(cwd, ".taskless", "vale", "rule-tests", "no-simply", "fail"), { + recursive: true, + force: true, + }); + + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).not.toBe(0); + const parsed = JSON.parse(result.stdout) as ValeVerifyJson; + expect(parsed.success).toBe(false); + expect(parsed.fixtures).toBe("pass-only"); + }); + + // Vale lints the rule's whole tree, so a nested document is linted but never + // checked against a bucket. Rejecting names the path at the moment someone + // creates it, rather than letting half a rule's fixtures go unverified. + it("rejects a nested fixture directory by name", async () => { + const nested = join( + cwd, + ".taskless", + "vale", + "rule-tests", + "no-simply", + "pass", + "nested" + ); + await mkdir(nested, { recursive: true }); + await writeFile(join(nested, "x.md"), "fine\n"); + + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).not.toBe(0); + const parsed = JSON.parse(result.stdout) as { + ok: boolean; + code: string; + message: string; + }; + expect(parsed.ok).toBe(false); + expect(parsed.message).toContain("flat"); + expect(parsed.message).toContain("nested"); + }); + + it("reports which fail/ fixture did not fire", async () => { + await writeFile( + join(cwd, ".taskless", "vale", "rule-tests", "no-simply", "fail", "b.md"), + "This text does not contain the token.\n" + ); + + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).not.toBe(0); + const parsed = JSON.parse(result.stdout) as ValeVerifyJson; + expect(parsed.success).toBe(false); + expect(parsed.missingFailures).toHaveLength(1); + expect(parsed.missingFailures[0]).toContain("b.md"); + }); + + it("reports which pass/ fixture wrongly fired", async () => { + await writeFile( + join(cwd, ".taskless", "vale", "rule-tests", "no-simply", "pass", "b.md"), + "You simply cannot.\n" + ); + + const result = await runCli([ + "rule", + "verify", + "no-simply", + "-d", + cwd, + "--json", + ]); + expect(result.exitCode).not.toBe(0); + const parsed = JSON.parse(result.stdout) as ValeVerifyJson; + expect(parsed.success).toBe(false); + expect(parsed.unexpectedFindings).toHaveLength(1); + expect(parsed.unexpectedFindings[0]).toContain("b.md"); + }); +}); diff --git a/skills/taskless/SKILL.md b/skills/taskless/SKILL.md index 1a32ef12..6801a975 100644 --- a/skills/taskless/SKILL.md +++ b/skills/taskless/SKILL.md @@ -17,7 +17,7 @@ description: | Also trigger on any request to add/write/create a lint or code rule, including ones that name a specific tool (eslint, ruff, biome, stylelint, ast-grep). Naming a tool ENGAGES this skill's routing flow via - `taskless help route`; it does NOT suppress the skill. + `taskless agent route`; it does NOT suppress the skill. metadata: author: taskless version: 0.10.2