jj-commit-cloud-poc

Commit Cloud architecture

A detailed, source-linked map of how jj-commit-cloud-poc works inside: a gRPC server that stores a jj repository's commits, trees, files, operations, views and op heads, the client-side jj-lib store implementations that talk to it, and the custom jj binary that ties them together.

On this page
  1. The whole system on one page
  2. The components
  3. The custom jj binary
  4. cc-lib, the client stores
  5. cc-common, the wire
  6. jj-cc-server
  7. Crate map
  8. Status and limitations
  9. Pages

Upstream calls this a proof of concept, and these pages say plainly what it doesn't do yet. For jj's own storage model (commits, operations, views and backends), see the jj architecture reference. That reference describes its own pinned jj version, which can differ from the jj-lib 0.43.0 this project builds against.

The whole system on one page

Top: what runs on your machine, where the custom jj binary plugs commit-cloud stores into an otherwise stock jj. Middle: the gRPC boundary. Bottom: the server. Every box links to the page that explains it, or to pinned source while that page is still being written. jj-lib boxes link to the jj reference.

jj-commit-cloud-poc architecture On the user's machine, the custom jj binary runs stock jj-cli commands plus jj cc init. jj-lib's ReadonlyRepo, Transaction and Store call three cc-lib implementations: CommitCloudBackend, CommitCloudOpStore and CommitCloudOpHeadsStore. The working copy, the commit index and the config.toml holding the server URL and repo ID stay local. The cc-lib stores call two gRPC services over tonic, BackendService and OpStoreService. On the server, jj-cc-server implements them with CommitCloudBackendService and CommitCloudOpStoreService, which compute every object ID with hash_utils and keep everything in a Store. The only storage engine is the in-memory MemoryStore. A SQLite engine is declared, but selecting it exits at startup. Your machine jj (the cli crate) stock jj-cli commands + jj cc init ReadonlyRepo / Transaction jj-lib Store jj-lib CommitCloud Backend CommitCloud OpStore CommitCloud OpHeadsStore Stays local working copy .jj/working_copy commit index default local index store config.toml .jj/repo/store: server_url, repo_id network: gRPC over HTTP/2 (tonic 0.12) BackendService 9 RPCs, backend.proto OpStoreService 6 RPCs, op_store.proto jj-cc-server CommitCloud BackendService CommitCloud OpStoreService hash_utils computes every object ID Store trait repos, commits, trees, files, operations, views, op heads tonic-health reports SERVING MemoryStore the only engine SQLite declared, exits at startup

The components

The custom jj binary

The cli crate builds a binary named jj. Its main is jj-cli's own CliRunner with two additions: store factories that know the commit_cloud backend, op store and op heads store, and a cc subcommand whose only child is init. The factories are built from StoreFactories::empty() (source), but jj-cli merges them into its defaults (source), so this binary opens ordinary jj repositories as well as commit-cloud ones. jj cc init --server <url> registers a new repository on the server and initialises a local workspace whose stores all point at it. The --create flag is accepted and ignored. See the custom jj binary.

cc-lib, the client stores

cc-lib implements three jj-lib traits over gRPC: Backend (commits, trees and files), OpStore (operations and views) and OpHeadsStore. Each loads server_url and repo_id from .jj/repo/store/config.toml. Every trait method calls run_async, which spawns an OS thread, builds a single-threaded tokio runtime on it, connects a new tonic client, makes one RPC and joins the thread. See the client Backend and the client OpStore and OpHeadsStore.

cc-common, the wire

cc-common compiles backend.proto and op_store.proto with tonic-build, and holds the shared ID lengths and root IDs. See every schema.

jj-cc-server

The server crate (jj-commit-cloud-server, binary jj-cc-server) listens on --host (default 127.0.0.1) and --port (default 8080), serves both services plus the standard gRPC health service, and keeps every repository in one shared Store (source). It computes the ID of every object it is sent, using Git's object format for commits, trees and files. See the server, storage engines and object IDs.

Crate map

DirectoryCrateBuildsDepends on
cli/clibinary jjjj-cli and jj-lib 0.43.0, cc-lib, clap, tokio, serde, toml, uuid. Tests add testutils, cc-common, tonic, assert_cmd, predicates, tempfile.
lib/cc-liblibraryjj-lib 0.43.0, cc-common, tonic 0.12.3, tokio, async-trait, futures, serde, toml, uuid
common/cc-commonlibrary (generated gRPC code)tonic 0.12.3, prost 0.13.3. Build: tonic-build 0.12.3.
server/jj-commit-cloud-serverbinary jj-cc-servercc-common, tonic, tonic-health, tokio, tokio-stream, gix 0.68.0, hex, clap, tracing, uuid, async-trait. It doesn't depend on jj-lib.
testutils/testutilstest helper librarytokio, assert_cmd, tempfile

The workspace is defined in the root Cargo.toml (resolver 2). The server never links jj-lib: it works only with the generated protobuf types and gix, which is why it must re-derive Git-compatible IDs itself.

Status and limitations

Upstream calls this a proof of concept, "beginning life as an intern project", with "no guarantee of completeness, performance, or API stability". At the pinned commit:

  • Storage is in memory only. Restarting the server loses every repository. --store-type sqlite prints "SQLite storage backend is not yet implemented" and exits (source).
  • No symlinks, copies or submodules. The symlink RPCs exist but are unimplemented on both sides, and copy tracking and submodules return Unsupported on the client.
  • Files travel whole. WriteFile is a single message, so large files can hit gRPC's message size limit (a TODO in both the schema and the server).
  • The custom binary only opens commit-cloud repos, and jj cc init --create is ignored.
  • Signatures, tags and Git refs don't round-trip. See what the wire doesn't carry.
  • No authentication. Any client that knows a repository's UUID can read and write it, and the server takes a client-supplied commit ID without checking it.

Pages

  • Every schema

    Both .proto files: every service, RPC, message and field, the jj-lib type each converts to and from, and how the Rust code is generated.

  • The client Backend

    The commit-cloud implementation of jj-lib's Backend trait, method by method: which RPC each calls, IDs, the root commit, and the sync/async bridge.

  • The client OpStore and OpHeadsStore

    How operations, views and op heads are read and written over gRPC, whether op-head updates are atomic, and what is unimplemented.

  • The custom jj binary

    How the jj binary is customised with commit-cloud stores and a cc subcommand, what jj cc init does, and how stock commands run on top.

  • The server

    jj-cc-server's arguments, storage selection and tonic setup, and its backend and op-store services RPC by RPC.

  • Server storage engines

    The server's Store trait, the in-memory engine, and which other engines are declared but not implemented.

  • Object IDs and hashing

    Which IDs exist and who computes them, the exact bytes hashed for commits, trees, files, operations and views, and how they compare with jj's Git backend.

  • Tests

    The test utilities, every integration test and what it asserts, and which tests are expected to fail.