jj-commit-cloud-poc / Server storage engines

Server storage engines

The server reaches its data only through the Store trait. At the pinned commit there is one implementation, MemoryStore: seven mutex-guarded hash maps, kept for the life of the process. A SQLite engine is declared on the command line and in a test, but it doesn't exist.

On this page
  1. The Store trait
  2. MemoryStore
  3. Reads and writes
  4. Locking
  5. Op heads
  6. SQLite: declared, not implemented
  7. What storage doesn't do

In short: fourteen async methods, a register/check pair plus a get/put pair for each object kind and for op heads. MemoryStore keys every map by repo ID, then by object ID, and holds one std::sync::Mutex per map, only for the duration of one call. Every single call is atomic, but nothing spans two calls. Op-head updates are atomic on their own, and their rule drops a concurrent publish (below).

The engine is chosen in StoreType, and the services share it as an Arc<dyn Store> (see server).

The Store trait

Store is an async_trait that is Send + Sync. A comment says its methods are async so that later engines don't block server threads. None of them return errors: reads return Option and writes return nothing. A storage failure has no way to reach the client as a status.

MethodReturnsCalled byMemoryStore
is_repo_registered(repo_id)boolevery RPC except registrationL35
register_repo(repo_id)—RegisterRepositoryL39
get_commit / put_commitOption<Commit> / —ReadCommit / WriteCommitL43
get_tree / put_treeOption<Vec<TreeEntry>> / —ReadTree / WriteTreeL53
get_file / put_fileOption<Vec<u8>> / —ReadFile / WriteFileL63
get_operation / put_operationOption<Operation> / —ReadOperation / WriteOperationL73
get_view / put_viewOption<View> / —ReadView / WriteViewL83
get_op_heads(repo_id)Option<Vec<OpId>>GetOpHeadsL93
update_op_heads(repo_id, old_ids, new_id)the resulting headsUpdateOpHeadsL98

The values are the generated protobuf types themselves (Commit, TreeEntry, Operation, View), so the server stores exactly what was on the wire. The ID types are aliases: RepoId is a String, and every object ID is a Vec<u8> (L9–14).

MemoryStore

The layout of MemoryStore MemoryStore has seven fields, each behind its own mutex. repos is a set of repo IDs. commits, trees, files, ops and views each map a repo ID to a map from object ID to the stored object. op_heads maps a repo ID to a list of operation IDs. MemoryStore repos: Mutex<HashSet<RepoId>> commits: Mutex<HashMap<RepoId, HashMap<CommitId, Commit>>> trees: Mutex<HashMap<RepoId, HashMap<TreeId, Vec<TreeEntry>>>> files: Mutex<HashMap<RepoId, HashMap<FileId, Vec<u8>>>> ops: Mutex<HashMap<RepoId, HashMap<OpId, Operation>>> views: Mutex<HashMap<RepoId, HashMap<ViewId, View>>> op_heads: Mutex<HashMap<RepoId, Vec<OpId>>>

Reads and writes

Every getter locks its map, looks up the repo and then the ID, and returns a clone. A whole file's bytes or a whole tree's entries are copied on each read. Every putter locks its map, creates the repo's inner map if it is missing, and inserts, replacing whatever was under that ID. Putters don't check that the repo is registered; the services do that first. Nothing is ever removed.

Locking

The seven mutexes are std::sync::Mutex, not tokio's. Each method takes exactly one lock and never holds it across an .await, so there is no lock ordering and no deadlock. A contended lock blocks the tokio worker thread for the length of a hash-map operation and a clone. Every lock is lock().unwrap(), so a panic while a lock is held poisons that map and every later call to it panics.

Atomicity stops at one call. Writing an operation, writing its view and moving the op heads are three calls on three maps, and other requests can run between them. That is harmless for objects, because they are content-addressed and only added. Op heads are the one place where it matters.

Op heads

The heads for a repository are a Vec, not a set, and a repository that has never been updated has no entry. get_op_heads then returns None, and the service substitutes the root operation. update_op_heads does all its work under the op_heads lock, so two updates never interleave. Its comment says it preserves "concurrent sibling heads" for client-side reconciliation. The rule:

Heads beforeold_idsnew_idHeads afterWhy
none stored[root]A[A]The entry starts as [root], root is removed, A is added.
[H][H]A[A]The ordinary case.
[A][H]B[A]Nothing removed, so B isn't added.
[A, C][A]D[C, D]A is removed, the sibling C stays, D is added.
[A]emptyB[A, B]Empty old_ids only adds.
[A][A]A[A]Removed, then re-added.

The third row is the lost publish described on client-op-store. A sibling head survives only if it is already in the list, and nothing in this store ever puts two heads there: the only way to add a head without removing one is an empty old_ids. jj-lib sends that when resolving op heads finds a single head with no ancestors to prune (op_heads_store.rs), and the head it names is already in the list. So the "sibling heads" the comment mentions don't arise from two clients racing. The rows above follow from reading the code and from a reimplementation of the rule, not from a running server.

SQLite: declared, not implemented

--store-type sqlite and --sqlite-path exist (main.rs L21–46), but choosing SQLite prints "SQLite storage backend is not yet implemented" and exits with status 1 (L73–76). There is no SQLite module under server/src/store/, and the server crate has no SQLite dependency (Cargo.toml). The test harness can start a server this way (spawn_sqlite_server), and test_sqlite_store_init_and_snapshot_succeeds is marked #[should_panic], so it passes because the engine is missing.

What storage doesn't do

  • Nothing persists: a restart loses every repository and object.
  • Memory grows without bound. There is no deletion, eviction or garbage collection.
  • Reads clone whole objects, including whole files.
  • Calls have no errors to report, and a poisoned lock panics instead.
  • No multi-call transactions.
  • The op-head rule drops concurrent publishes instead of keeping divergent heads.