jj-commit-cloud-poc / The client OpStore and OpHeadsStore

The client OpStore and OpHeadsStore

CommitCloudOpStore and CommitCloudOpHeadsStore put jj's operation log on the server. Operations and views are converted to protobuf and stored under IDs the server computes. The set of op heads lives on the server too, behind two RPCs and no lock.

On this page
  1. Loading
  2. OpStore methods
  3. The root operation and root view
  4. Conversions
  5. Operations
  6. Views
  7. OpHeadsStore methods
  8. What the server does with an update
  9. Concurrent operations
  10. Errors
  11. What these stores don't do

In short: four OpStore methods map one-to-one onto ReadView, WriteView, ReadOperation and WriteOperation, using the same thread-per-call bridge as the client Backend. Op-ID prefix lookup only knows the root operation, and gc does nothing. The op heads store calls GetOpHeads and UpdateOpHeads, and its lock is a no-op. The views it stores leave out tags, Git refs and Git HEAD. Operation times lose their time zone. And when two clients publish on top of the same operation, the server keeps the first and silently drops the second.

The modules are lib/src/cc_op_store.rs and lib/src/cc_op_heads_store.rs. Both are registered under the name commit_cloud by add_commit_cloud, which turns load errors into BackendLoadError and ignores the root-operation data jj passes in. The messages are on protobufs, and the operation and view hashes on object-ids. jj's own model is on /jj/operations and /jj/view. Those pages describe the jj version /jj is pinned to. This project uses jj-lib 0.43.0.

Loading

Neither store has an init. Both have only load(store_path) (op store, op heads store), which reads server_url and repo_id through CommitCloudConfig::load_from_store. Their store paths are .jj/repo/op_store and .jj/repo/op_heads, which have no config.toml, so the loader falls back to ../store/config.toml, the file the backend's init wrote. All three stores therefore talk to the same server and repository. No network call is made until the first read or write.

FieldTypeValue
server_urlStringFrom the shared config.toml.
repo_idStringSent with every request.
root_operation_id (op store only)OperationId16 zero bytes (ROOT_OPERATION_ID_BYTES).

CommitCloudOpStore · CommitCloudOpHeadsStore · constants

OpStore methods

Every method of jj-lib's OpStore trait is implemented (impl). The four RPC methods each clone the server URL and repo ID, convert their argument, and run OpStoreServiceClient::connect plus the call inside run_async: a new thread, a new tokio runtime and a new connection per call, as described on client-backend.

MethodBehaviourRPCSource
name"commit_cloud"—L250
root_operation_idThe 16 zero bytes. No call.—L254
read_viewSends the view ID and converts the returned view. A response with no view is an error.ReadViewL258
write_viewConverts the view and returns the server's view ID.WriteViewL280
read_operationSends the operation ID and converts the returned operation. A response with no operation is an error.ReadOperationL299
write_operationConverts the operation and returns the server's operation ID.WriteOperationL321
resolve_operation_id_prefixNo RPC exists for it. A prefix of the root ID is a SingleMatch on the root operation. Anything else is NoMatch. A TODO asks for a server RPC.—L340
gcDoes nothing.—L355

The prefix gap reaches further than jj op show. jj-lib resolves any operation written as hex, even a full ID, through resolve_operation_id_prefix (resolve_single_op_from_store). So with these stores, an explicit operation ID other than the root's gives "no such operation", in any command that accepts one. Symbolic forms such as @ start from the op heads and don't use it.

The root operation and root view

The client never builds the root operation itself. It reads it from the server like any other. When the store has nothing under the root ID, ReadOperation answers with a fixed operation: view ID of 16 zero bytes, no parents, description "root()", zero times, empty host and user names, and commit_predecessors_set: true with no entries. ReadView answers the zero view ID with a view whose only head is the root commit. Any other unknown ID is a gRPC NotFound, as is any request for an unregistered repo ID (ensure_repo_registered_error).

Conversions

Operations

jj-lib fieldOn the wireRead back
view_id, parentsRaw bytes.Unchanged.
metadata.time.start, .endMilliseconds only.tz_offset is always 0, so the local time zone is lost.
description, hostname, username, is_snapshotCopied.Unchanged.
workspace_nameOptional string.Unchanged, including None.
attributesA map.Collected back into a BTreeMap.
commit_predecessorsA list of CommitPredecessors entries plus commit_predecessors_set, so None and an empty map stay distinct.Unchanged.

operation_to_proto · operation_from_proto · jj-lib Timestamp

A missing metadata message reads back as all defaults. The operation ID is computed by the server from the converted message (WriteOperation), so the dropped time zone isn't part of it.

Views

jj-lib View fieldOn the wireRead back
head_idsList of bytes.A HashSet.
wc_commit_idsMap from workspace name.Unchanged.
local_bookmarksMap of RefTargets (removes and adds).Rebuilt with RefTarget::from_legacy_form.
remote_views[r].bookmarks[b]One flat map, keyed "b@r", with the target and is_tracked.Split at the last @. Tracked or New. A missing target becomes absent.
remote_views[r].tagsNot sent.Empty.
local_tags, git_refsNot sent.Empty.
git_headNot sent.Absent.

ref target helpers · view_to_proto · view_from_proto · jj-lib View, RemoteView

Three consequences follow from the code:

  • Tags, Git-tracking refs and Git HEAD written by any command are gone the next time the view is read.
  • A remote whose name contains @ is split wrongly on read. Bookmark main on remote a@b is sent as main@a@b and comes back as bookmark main@a on remote b. An @ in a bookmark name is fine, since the split is at the last @.
  • A remote with no bookmarks has no entries in the flat map, so it doesn't come back at all.

OpHeadsStore methods

MethodBehaviourRPCSource
name"commit_cloud"—L37
get_op_headsReturns the server's list. Errors become OpHeadsStoreError::Read.GetOpHeadsL71
update_op_heads(old_ids, new_id)Sends both and ignores the current_op_head_ids the server returns. Errors become OpHeadsStoreError::Write. A TODO asks for locking.UpdateOpHeadsL41
lockReturns CommitCloudOpHeadsStoreNoLock, an empty type. No call.—L89

jj-lib's OpHeadsStore docs say the lock "is not needed for correctness" and only stops two processes from merging the same divergent heads. What correctness does rest on is update_op_heads: jj calls it when a transaction is published, with the new operation's parents as old_ids (Transaction::publish; see /jj/transactions).

What the server does with an update

GetOpHeads returns the stored heads, or just the root operation if the repository has never had an update. UpdateOpHeads passes the request to the store. The in-memory engine (MemoryStore::update_op_heads) does the whole update under one mutex, so each update is atomic on its own:

  1. Start from the stored heads, or [root] if there are none.
  2. If old_ids is empty, add new_id unless it's already there.
  3. Otherwise remove every head in old_ids. Add new_id only if at least one head was removed.
  4. Return the resulting heads.

A code comment says sibling heads are kept so that clients can later reconcile divergent heads. Step 3 works against that. jj-lib's local SimpleOpHeadsStore always adds the new head first, then removes the old ones. So when two processes publish on the same parent, both operations remain as heads and the next command merges them (resolve_op_heads).

Concurrent operations

With the commit-cloud stores, the second of two concurrent publishes is dropped, as the diagram shows. Both clients start from head H. Client A publishes A and the heads become [A]. Client B then publishes B with old_ids = [H]. No current head matches, so nothing is removed and B isn't added. The server answers with [A], B's client ignores the answer, and B's command succeeds. B's operation and view stay stored on the server, but nothing points to them, so the next command anywhere starts from A. No divergence is reported. This follows from the code and was checked against a reimplementation of the update rule. It hasn't been reproduced against a live server: it needs two commands to interleave between reading and publishing op heads.

Two clients publishing on the same operation Client A and client B both load op head H. Client A writes operation A, then calls UpdateOpHeads with old H and new A. The server's heads go from H to A. Client B writes operation B, then calls UpdateOpHeads with old H and new B. H is no longer a head, so nothing is removed and B is not added. The heads stay A, the server returns A, and client B ignores the response and reports success. Operation B is stored but unreachable. client A server op heads client B heads = [H] UpdateOpHeads(old [H], new A) H removed → heads = [A] UpdateOpHeads(old [H], new B) nothing removed → B not added returns [A], ignored; command succeeds operation B stored, unreachable

Concurrent operations from one client behave the same way: two jj processes on one machine share these stores, and lock doesn't stop them either.

Errors

Every OpStore failure, including a gRPC NotFound, becomes OpStoreError::Other. jj-lib's ObjectNotFound, ReadObject and WriteObject variants are never produced (OpStoreError). The op heads store uses OpHeadsStoreError::Read for get_op_heads, and Write with the new operation's ID for update_op_heads. It never returns Lock (OpHeadsStoreError).

What these stores don't do

  • A concurrent publish on the same parent operation is lost without any error, and divergent operations never appear.
  • lock takes no lock, and the update response is ignored.
  • Operation IDs given as hex, other than the root's, can't be resolved.
  • gc does nothing, so unreachable operations and views stay on the server.
  • Views drop local tags, remote tags, Git refs and Git HEAD, and remote names containing @ are split wrongly.
  • Operation times come back in UTC, with the time zone offset lost.
  • Each call opens a new thread, runtime and connection, with no timeouts or authentication.