Externalization plan

In short: docs/intro.md splits the extension into a UI layer, an API layer and a client layer. Most of the UI layer can be shared with the internal build. The API and client layers need external implementations, built on the jj CLI where Google uses gRPC-web servers. At this pin, the commit graph and the conflict highlighter have landed. Several pieces of the lower layers are in the tree but nothing calls them yet: a Client interface with a SubprocessClient that reads the commit graph through jj, a @parcel/watcher-based FileSystemWatcher, and bare interfaces for an op-head watcher, a network-reconnect watcher and telemetry. Everything else is planned. The M1–M4 labels on this page come from the four-milestone timeline intro.md carried until this pin; it now points to a GitHub project board instead (§2).

1. The three layers

The plan's architecture drawing (architecture.svg) colours each component green if it will be shared between the internal and external builds, and red if it needs a separate external implementation (source). The diagram below redraws those layers with their status at this pin. Solid boxes have code in the tree. Dashed boxes are only planned, or have code that nothing uses yet. Each box links to the page that covers it, or to the plan.

UI layer

The UI layer is "mostly reusable code" (source).

API layer

The API layer sits between the UI and the client. Each component needs an external implementation (source).

Client layer

The client layer fetches information about jj (source). Internally that is gRPC-web calls to Google's jj servers, which are built directly on jj-lib. Externally it will be the jj CLI with templates, possibly replaced by a daemon client in the long run. The plan warns that a CLI client is harder than it looks. The internal server exposes things the CLI doesn't, such as file IDs, and answers in one request what may take several CLI commands. The plan says the internal server and the CLI may need to converge as part of open-sourcing. At this pin the client reads the commit graph through the CLI, but nothing calls it; see §3.

2. Milestones

At this pin the Milestones section of intro.md is a single link to the jj-vcs project board (source). The board lives outside the repository and can't be pinned, so this page doesn't track it. The table below is the timeline intro.md listed up to the previous commit, kept for the M1–M4 labels used above; its dates and statuses are no longer in the repository, but the last column is checked against the code at this pin.

3. The subprocess client so far

client.ts (88 lines), subprocess_client.ts (437 lines), subprocess_util.ts (90 lines), their tests, src/api/api.ts and BUILD.bazel

The Client interface has four methods. getWorkspaceState snapshots the working copy without publishing an operation and returns a RepoState: a list of commits and the opHead they were read at, which may be that unpublished operation (source). Its doc comment explains the optional lastKnownUnpublishedOperation argument. Because an unpublished snapshot is never remembered, jj keeps snapshotting the same files again, which caused "resource exhaustion issues internally inside Google". The internal extension passes the last unpublished operation to its server as the base to snapshot on top of. The CLI can't do that, so the CLI implementation ignores the argument (source). workspaceAdd is "the equivalent of jj workspace add <workspaceRoot>" and returns the new WorkingCopy (source). The other two read file contents as bytes. readFileAtCommit reads a path at a commit, or at its combined parents when showParent is set, and returns undefined if the file isn't there. It takes an opHead because on the Git backend jj file show only finds commits reachable from the operation it runs at; the comment says Google's commit cloud may ignore it. readFileById reads a file by its FileId (source).

The types it returns live in the //src/api package. A Commit carries:

• its ID and change ID, parentIds and childIds;
• whether it is immutable: the root commit or an ancestor of immutable_heads();
• files, the files it modifies, each a CommitFile with a path, a modification type and its metadata as a Merge that may be conflicted (source);
• whether it is the working copy (active) or a parent of it (isWorkingCopyParentCommit);
• its full description and a descriptionTitle, "usually" the first line;
• author and committer times in milliseconds, and conflict and divergence flags;
• the display ID with how much of it to highlight, and the change offset when the change is divergent;
• its bookmarks, each with a name, an optional remote, needsUpload (jj's *) and isConflicted (source);
• ancestors and descendants, read-only sets of commit IDs that include the commit itself;
• workingCopies: the workspaces whose working copy it is, each with a jj workspace name, an optional root path and a display name (source).

CommitId, ChangeId and OperationId are small classes that wrap a hex string and implement HashMapKey, as does the new FileId, so they can be HashMap keys (source). A MutableCommit mapped type drops the readonlys and turns the read-only sets into HashSets, so the client can fill them in (source). These are not the commit graph's own protocol types; nothing converts one into the other yet.

SubprocessClient answers getWorkspaceState with two jj calls:

1. snapshotWithoutIntegrating runs jj --no-integrate-operation op log -n=1 --no-graph -T=id. Running any command snapshots the working copy, and --no-integrate-operation keeps the resulting operation out of the op log, so this returns the ID of an unpublished operation (see operations). A TODO notes that --no-graph drops jj's elided edges, which the commit graph can't draw either.
2. getCommits runs jj --at-op <op> log --no-graph with a template that prints one JSON object per commit, including its bookmarks and working copies (source). The template is built field by field by toJsonObjectTemplate. No revset is passed, so it shows whatever the user's default log revset selects. It then:
   • makes every display ID as long as the longest shortest unique prefix, and at least 4 characters (source);
   • builds each commit's child list from the parent lists, and marks the working copy's parents (source);
   • gives the working-copy commit a default workspace rooted at the client's directory when jj reports none, which jj does when there is only one workspace. A comment admits this is "technically not correct" for a renamed default workspace (source);
   • fills in every commit's ancestors and descendants with commit_topology (source).
   rawCommitToApiCommit appends /<offset> to a divergent commit's display ID, takes the first line as the title, and sets needsUpload for a local bookmark that is neither synced nor conflicted. isImmutable comes straight from the template. files is always empty: a TODO waits for a jj release that includes jj PR 10430 (source).

workspaceAdd runs jj workspace add --name=<basename> -- <root>. It names the workspace after the last path component itself, because jj workspace add prints nothing machine-readable, then finds that name in jj workspace list --ignore-working-copy and returns it. The -- keeps a path that starts with - from being read as a flag. If the new workspace isn't listed, it throws through logAndShowInternalError (see activation), which is why the target now depends on //src/logging.

readFileAtCommit calls fileShow, which runs jj --at-op=<op> --color=never file show -r=<id> -- root:"<path>", adding - after the ID for showParent. The quoted root: pattern makes the path match literally instead of as a glob. jj exits with 1 for every user error, so a missing file is recognised by the text Error: No such path: and turned into undefined. A TODO notes that - fails on a merge commit until jj PR 10464 makes parents_of() usable. readFileById runs jj debug object file -- <path> <id>; its comment says there is no stable API for this yet, a jj file show --id was rejected upstream, and git cat-file would only work on the Git backend.

Every call goes through subprocess, Node's execFile promisified with a 10-second timeout, or its twin subprocessBuffer, which returns stdout as bytes for the file reads. Both run the command directly, with no shell, and raise execFile's 1 MiB output limit to 10 MiB. The comment explains the cap: diffs still go through VS Code's FileSystemProvider, which loads whole files, and jj doesn't snapshot files over 1 MiB by default anyway (source). The helper's only dependency is @types/node, so under the deps scheme it belongs on the external-only side. The interface target client depends only on //src/api.

The 28 SubprocessClient tests run the pinned jj (see the jj rule) in fresh repositories from newRepo. Eleven cover a new repository, the returned opHead, descriptions (including binary data and a multi-line title), divergent commits with their change offsets, a conflict, ancestors and descendants across several commits, a second workspace, and a workspace path that starts with -. The snapshot test reads the op log the same unpublished way. A clean working copy adds no operation. A dirty file shows one "snapshot working copy" operation on top, and once the file is removed, the op log is back to what it was. Nothing was ever published (source). Five more run against a local repository with a remote and check local, conflicted, synced, unsynced and diverged remote bookmarks (source). Eight read files at a commit: text, binary data, a missing file, glob characters taken literally, showParent on a normal commit, and files just under and just over the buffer limit. On a merge commit the showParent reads are expected to fail until the TODO above is fixed (source). Four read files by ID, which the test gets from jj debug tree (source). The subprocess_util test still runs jj version (source). No file outside src/client/ imports the client, and //src's deps don't include it (source).

4. Interfaces without implementations

Three new directories hold one interface each. None has an implementation, a test or its own BUILD.bazel, so //src's source glob picks them up. Nothing imports them.

5. The file-system watcher

file_system_watcher.ts (58 lines), local_file_system_watcher.ts (204 lines), local_file_system_watcher_test.ts (20 tests) and BUILD.bazel

The package follows the interface/implementation split from the plan's design notes. The file_system_watcher target holds only the interface and depends only on @types/vscode. local_file_system_watcher adds @parcel/watcher, @types/node and the logging and error packages (source). The interface's doc comment says the abstraction exists so the extension can work with virtual file systems, and that internally it corresponds to a citc pubsub (source).

The interface

LocalFileSystemWatcher

Its doc comment explains why it uses @parcel/watcher rather than VS Code's createFileSystemWatcher: parcel supports exclusions, which are needed to ignore changes to gitignored files, and VS Code has no plans to add them (source).

1. Construction takes a directory, which must be an absolute path, or it throws an internal error. Parcel doesn't follow symlinks, so it resolves the directory with realpath first, which also drops a trailing slash; a path that can't be resolved is reported as a user error. It then subscribes to the directory recursively. Tests can inject a replacement subscribe. Once the subscription is ready, it logs the path and every subscriber gets undefined, so nothing that happened during setup is missed. If subscribing fails, the error is logged and shown to the user, and not rethrown (source).
2. Each batch of parcel events becomes one SnapshotDelta with the next counter value. Each path is made relative to the watched directory. stateChanged is set when an event's path is exactly <dir>/.jj/working_copy/checkout. A parcel error is logged and sent to subscribers as undefined (source, source).
3. Fan-out calls every callback and logs any exception it throws, so one failing subscriber doesn't starve the others (source).
4. Dispose clears the callbacks, swaps in a no-op subscription so repeated calls are harmless, and then unsubscribes, logging any failure (source).

The 20 tests run against the real @parcel/watcher on a temporary directory. They wait up to 30 seconds for each expected delta and use the fake VS Code API from //src/testing (fake API). Besides the three change types, they check that stateChanged fires for the workspace's own checkout file but not for a nested workspace's or for foo.jj/working_copy/checkout (source). The others cover subscriber isolation, disposal in every order, the error path, and the new path handling: a relative path is rejected, a trailing slash is ignored, and a directory opened through a symlink still produces events (source).

//src lists //src/file_system_watcher in its deps, which is the interface target, but nothing in the extension creates a watcher yet (source).

6. Design considerations

The plan ends with five rules for the work (source).

• Use Bazel for builds. The internal extension runs on Cider, a VS Code Web fork that can't import packages like node. So code is split into interface files and implementation files, and Bazel deps keep external-only packages out of the interfaces. See build and test for how far this is enforced at this pin.
• Avoid dependencies. Fewer dependencies mean less supply-chain risk and less work importing TypeScript libraries into Google. The extension's runtime dependencies at this pin are @parcel/watcher, @vscode-elements/elements, @vscode/codicons, jsonc-parser, lit and safevalues (source).
• Reads should stay reads. Read commands will pass --no-integrate-operation and --ignore-working-copy, so a read neither causes divergence nor adds entries to the user's op log (see operations and working copy). SubprocessClient passes --no-integrate-operation and reads at that operation with --at-op, but not --ignore-working-copy: it snapshots on purpose, so the graph reflects unsaved file changes without publishing an operation.
• Different defaults internally and externally. Google developers are used to settings such as jj.includeWorkingCopyInParentDiffs and threeWayMerge.enabled being on. The plan says to check whether those defaults suit external users. At this pin contributes.configuration is empty, so neither setting exists.
• Missing features in parallel. Features the internal extension still lacks, such as blame and squashing part of a file, are open for outside contributors while externalization goes on.

7. Loose ends at this pin

• Exclusions are the reason for parcel, but none are passed. The watcher picks @parcel/watcher for its exclusion support, yet calls subscribe with no ignore option (source). As written, gitignored build output and every write under .jj/ produce events.
• A native module in a package that ships no dependencies. @parcel/watcher is a native npm package, and the .vsix is built with --no-dependencies. A TODO already notes this has to change before the watcher is used (build and test).
• The subprocess helper's limits. Output is capped at 10 MiB and every call has a fixed 10-second timeout (source). getCommits prints the whole log in one call, so a very large repository would still make it reject, and a file over the cap can't be read at all.
• Two commit types. The client returns //src/api commits, while the graph takes its own protocol types. Something still has to convert one into the other before the graph can show real data.
• Nothing new is reachable from activate. The client, the watcher and the three interfaces are built, and the first two are tested, on every CI run, but none affects the shipped extension. They move from "in progress" to implemented once a state manager uses them.