Version 0.3.0: resolver/merge/cache refactor, signing design notes
- Split the monolith into package/provider/fetch/merge/cache modules with the Resolver deciding install/adopt/update/merge/conflict outcomes - Three-way merges via git merge-file/diff3 against a content-addressed merge-base cache; conflicts go to a .picopackage-merge sibling - Tests for package, provider, resolver, merge, and cache - notes.md: update UX and signing design — diff-by-default, SSH signature identity pinning (TOFU), key changes as a hard stop, exit-code contract - Add CLAUDE.md; remove the pre-refactor exe/pppkg monolith and scratch files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EiyJC73Mz8xZyCTvCEY8qn
This commit is contained in:
Dan Milne
co-authored by
Claude Fable 5
Claude Fable 5
parent
e0cd0f0d7a
commit
6dd57e84f1
@@ -1,25 +1,195 @@
|
||||
## Package Installation Flow
|
||||
## Resolution rules
|
||||
|
||||
1. **Fetch URL**
|
||||
└─> Download content
|
||||
└─> Build Package Instance (Metadata & Payload)
|
||||
└─> Check local file status
|
||||
Two principles decide every case below:
|
||||
|
||||
2. **Local File Check**
|
||||
├─> If file doesn't exist:
|
||||
│ └─> Save file
|
||||
│
|
||||
└─> If file exists:
|
||||
└─> Compare versions
|
||||
├─> If older/same:
|
||||
│ └─> "Package already installed"
|
||||
│
|
||||
└─> If newer:
|
||||
└─> Check local modifications
|
||||
├─> If modified:
|
||||
│ └─> "Local modifications detected"
|
||||
│ └─> "Use 'update <file_path>'"
|
||||
│
|
||||
└─> If unmodified:
|
||||
└─> "Update available"
|
||||
└─> "Use 'update <file_path> -f' to force update"
|
||||
1. **The checksum decides, never the timestamp.** mtime is unreliable (a raw
|
||||
file re-uploaded unchanged gets a new mtime; a gist mirrored elsewhere gets
|
||||
an arbitrary one) and it is not evidence about content. If the payload
|
||||
checksums agree, the payload is settled — there is nothing to resolve no
|
||||
matter what the timestamps say. This is why every `????` in the old table
|
||||
collapsed: all of those rows assumed equal checksums, so all of them are
|
||||
metadata questions, not content questions.
|
||||
|
||||
2. **Merge the payload, regenerate the metadata.** The metadata block is
|
||||
derived, not authored, so it never takes part in a merge — merging it would
|
||||
conflict on every single update. Three-way merges run on the payload alone
|
||||
and the block is rewritten afterwards.
|
||||
|
||||
### Terms
|
||||
|
||||
- **payload** — the file with its metadata block removed, then normalised.
|
||||
- **bare** — a file with no metadata block. A hand-copied file, or an upstream
|
||||
that has never been packaged.
|
||||
- **`payload_checksum`** — checksum of the payload as it sits in this file.
|
||||
Local edits break it, which is how modification is detected.
|
||||
- **`base_checksum`** — checksum of the upstream payload this file was last
|
||||
reconciled with. Equal to `payload_checksum` on a clean install; they diverge
|
||||
after a merge or a local edit. This is the merge base pointer and the cache
|
||||
key.
|
||||
|
||||
### Same payload checksum
|
||||
|
||||
The content is identical, so the only question is which metadata to keep.
|
||||
|
||||
| Remote | Local | Action | State |
|
||||
|---------|---------|-------------------------------------------|-------------|
|
||||
| Bare | missing | Write payload + generated metadata | `installed` |
|
||||
| Package | missing | Write payload + upstream metadata | `installed` |
|
||||
| Bare | Bare | Adopt: attach generated metadata | `adopted` |
|
||||
| Package | Bare | Adopt: attach upstream metadata | `adopted` |
|
||||
| Bare | Package | Nothing to do | `current` |
|
||||
| Package | Package | Rewrite only if the metadata itself changed | `current` |
|
||||
|
||||
mtime appears nowhere in this table. That is the point of principle 1.
|
||||
|
||||
### Different payload checksum
|
||||
|
||||
| Local | Modified | Base available | Action | State |
|
||||
|--------------------|----------|----------------|-------------------------------------|-------------|
|
||||
| missing | — | — | Write it | `installed` |
|
||||
| Package | no | — | Fast-forward to upstream | `updated` |
|
||||
| Package | yes | yes | Three-way merge, clean | `merged` |
|
||||
| Package | yes | yes | Three-way merge, conflicted | `conflict` |
|
||||
| Package | yes | no | Refuse — no ancestor to merge from | `conflict` |
|
||||
| Bare | n/a | never | Refuse — unrelated file, same name | `conflict` |
|
||||
| any | any | any | `--force` overwrites unconditionally | `updated` |
|
||||
|
||||
A bare local file whose payload differs is not a modified picopackage — it is an
|
||||
unrelated file that happens to share a name. There is no ancestor and no claim
|
||||
that the two share a history, so merging would be guesswork. Refuse and say so.
|
||||
|
||||
### Conflicts are non-destructive
|
||||
|
||||
`git`/`diff3` conflict markers are written to `<file>.picopackage-merge`, beside
|
||||
the original. The original is never touched. A package manager with no `reset`
|
||||
and no index has no business writing conflict markers into a file the user may
|
||||
not have committed.
|
||||
|
||||
## Merge base cache
|
||||
|
||||
Three-way merge needs the common ancestor — the payload exactly as upstream last
|
||||
sent it. It is content-addressed under `$XDG_CACHE_HOME/picopackage/sha256/ab/cdef…`
|
||||
and written on every save, so a file installed by `ppkg` carries the pointer to
|
||||
its own ancestor in `base_checksum`.
|
||||
|
||||
Cache misses are expected and survivable: a file installed on another machine, a
|
||||
cleared cache, a package predating this feature. The fallback is to refuse the
|
||||
merge and say why, which is honest. A future `source_url` pinned to an immutable
|
||||
revision would let us refetch the ancestor instead of refusing — see below.
|
||||
|
||||
## Update UX and signing (designed 2026-08-13, not implemented)
|
||||
|
||||
The security question splits in three, and today's answers are: **first
|
||||
install** — pure trust in the URL, same as copy+paste, mitigated by "it's one
|
||||
file and you can read it"; **update** — worse than copy+paste, because `ppkg
|
||||
update` fetches whatever the mutable url serves and writes it with less
|
||||
friction than a human re-pasting; **transport** — plain `http` is currently
|
||||
allowed. The checksum contributes nothing adversarial: it lives inside the
|
||||
file it hashes and is regenerated on every write, so it is integrity against
|
||||
yourself, not against an attacker. The design below closes the update gap.
|
||||
|
||||
### Diff by default
|
||||
|
||||
`ppkg update` shows the diff and asks before writing. The diff shown is
|
||||
**ancestor → upstream** — the untrusted delta. Local edits are not news to the
|
||||
user, and the Resolver already holds all three payloads at that point, so this
|
||||
is nearly free. We are not a diff tool: shell out (`git diff --no-index`, else
|
||||
`diff -u`), respect `$PAGER` — the same philosophy as merge.rb. `-y`/`--yes`
|
||||
skips the prompt for scripts; it never covers an identity change (below).
|
||||
|
||||
### Identity: SSH signatures, TOFU-pinned
|
||||
|
||||
SSH rather than Sigstore. A full Sigstore bundle is 4–10KB of JSON — routinely
|
||||
larger than the payload it certifies — and drags a dependency tree into a
|
||||
zero-dependency gem. An armored ed25519 ssh signature is ~0.5KB, embeds in the
|
||||
metadata block, and the authors this format serves (gists, blogs) already hold
|
||||
ssh keys, with `github.com/<user>.keys` as a verification channel separate
|
||||
from the file itself. What Sigstore would add — identity that survives key
|
||||
loss, a transparency log — is real but not a different security class at this
|
||||
scale. It stays a possible opt-in upgrade, not the price of entry.
|
||||
|
||||
- The signature covers the **normalized payload + the claims** (url, filename,
|
||||
payload_version). Signing the payload alone leaves `url` unsigned, so a
|
||||
tampered first install could silently redirect every future update. The
|
||||
claims/records split in provider.rb is the seam: sign the claims, keep
|
||||
deriving the records.
|
||||
- TOFU: the key fingerprint is pinned at install, or the first time a
|
||||
signature appears on an already-installed file. Unsigned packages keep
|
||||
working; signing is opt-in per file.
|
||||
|
||||
### Key changes are a hard stop
|
||||
|
||||
The known_hosts model: banner, refuse, distinct exit code — not a y/n prompt
|
||||
someone can fat-finger through. The only door is `--accept-key <fingerprint>`;
|
||||
making the human transcribe the new fingerprint is what proves they looked.
|
||||
|
||||
Routine updates and identity changes are different signals and get different
|
||||
ceremonies. "Upstream changed, here's the diff" is routine: default-proceed,
|
||||
informational. "The signing identity changed" is how account compromise
|
||||
presents: default-refuse. Flattening both into one always-maximum-caution
|
||||
ceremony is how alarm fatigue happens.
|
||||
|
||||
### No rotation chains
|
||||
|
||||
Old-key-signs-new-key succession (signify/TUF style) was considered and
|
||||
dropped. If every key change is loud and manual anyway, a valid rotation proof
|
||||
can only *downgrade* the alarm — and a stolen key holds everything needed to
|
||||
mint that proof, so chains sharpen theft (a thief can rotate the author out of
|
||||
their own package) while buying convenience only for planned rotation. At this
|
||||
scale, "key changed, human reads the diff carefully, re-pins" *is* the
|
||||
rotation ceremony, and it covers loss and theft with the same motion.
|
||||
|
||||
### Exit codes are the contract
|
||||
|
||||
Callers — scripts and reviewing agents — build policy on exit codes, not on
|
||||
prose. An agent's policy may auto-accept a clean diff under a continuous key;
|
||||
an identity change must be unrepresentable in that path — no flag an agent
|
||||
routinely passes gets through it.
|
||||
|
||||
| Code | Meaning |
|
||||
|------|------------------------------------------------------------|
|
||||
| 0 | written (installed/adopted/updated/merged) or already current |
|
||||
| 1 | conflict — original untouched, as ever |
|
||||
| 2 | declined — user said no to the diff |
|
||||
| 3 | identity change — signing key differs from the pin |
|
||||
|
||||
### Review is the content check; the signature is the identity check
|
||||
|
||||
A signature answers "same author?" and says nothing about whether the code is
|
||||
safe. Review answers "does this change look sane?" and says nothing about who
|
||||
made it. They are complements, and single files invert the economics that make
|
||||
library review hopeless: the whole unit — old file, new file, three-way diff,
|
||||
the project it lives in — fits in one reading (or one context window), and
|
||||
with no install hooks, build steps, or dependencies, what you review is
|
||||
exactly what runs.
|
||||
|
||||
The gem's job is to surface facts, not to judge: emit the diff plus the
|
||||
machine-readable facts (`--json` — key changed, size delta, e.g. "adds
|
||||
net/http") and exit accordingly. Any LLM stays out of the gem. Note for
|
||||
whoever builds the reviewing agent: the payload is adversarial input —
|
||||
prompt injection in comments, aimed at the reviewer, is an expected attack.
|
||||
|
||||
### Order of work
|
||||
|
||||
1. Refuse plain `http`, including via redirect — cheap, orthogonal.
|
||||
2. `source_url` pinning (see open question below) — makes the update diff a
|
||||
pointer change and recovers merge bases on cache miss.
|
||||
3. Diff-by-default + the exit-code contract.
|
||||
4. SSH signing and pinning.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Immutable source pinning.** `url` is mutable ("whatever is at this address
|
||||
today"), which means builds aren't reproducible and "upstream improved" is
|
||||
indistinguishable from "upstream was compromised". Wants splitting into a
|
||||
pinned `source_url` (gist revision SHA, commit SHA) and a mutable `update_url`
|
||||
used only for checking. This also recovers merge bases on a cache miss.
|
||||
- **Multi-file packages.** `Jmap::Client` + `Jmap::Http` are one unit of reuse
|
||||
across two files. Either the spec grows a flat file-set, or single-file
|
||||
packages accept that they define several classes in one file and lose Zeitwerk
|
||||
file-per-class layout. Unresolved, and the first real consumer hits it.
|
||||
- **Comment styles.** `#` covers Ruby/Python/shell/YAML/Perl. Needs a table for
|
||||
`//`, `--`, `<!-- -->`, `/* */`, `;` before the format is honestly
|
||||
language-agnostic.
|
||||
- **Unknown metadata keys** are preserved verbatim on rewrite (see
|
||||
`Metadata#extra`), so optional spec fields survive a tool that predates them.
|
||||
|
||||
Reference in New Issue
Block a user