Storage backends

In short: a backend is a content-addressed object store with a few jj-specific extras: change IDs, a virtual root commit, conflicted trees, and copy records. The Git backend keeps real Git objects and stores what Git cannot hold in a side table, in commit headers, or in specially named tree entries. Which backend a repo uses is the name written in .jj/repo/store/type.

1. The layers

The trait is object-safe, so a program built on jj-lib can plug in a backend of its own; the docs for concurrency have a cloud-backed one in mind. Code that needs a specific implementation downcasts: Store::backend_impl::<GitBackend>() (store.rs, via downcast_ref) returns None for any other backend.

2. The Backend trait

pub trait Backend: Any + Send + Sync + Debug is an #[async_trait]. The file, symlink and tree methods also take the RepoPath the object lives at, even though IDs are content hashes; the Secret backend (§8) uses it to deny access. Errors are a BackendError: object not found, read failed, read access denied, invalid hash length, invalid UTF-8, write failed, Unsupported, or Other.

The root commit is never stored. Every backend answers read_commit(root_commit_id) with make_root_commit: no parents, the empty tree, empty names and emails, timestamp zero. Writing a commit with no parents is an error in every built-in backend. More on IDs and the root on Commits.

3. Store: caching in front of the backend

Store "wraps the low-level backend and makes it return more convenient types. Also adds caching." It holds two LRU caches: 100 commits and 1000 trees, because "there are more tree objects than commits, and trees are often shared across commits". Trees are keyed by (directory, TreeId), matching the backend's read_tree(path, id).

Both caches are a CachedLoader, so "concurrent cache misses for the same object share a single backend read". The first caller for a key starts the read; callers that arrive while it is in progress wait for its result instead of starting their own. Any of them may poll the shared read, which is why the backend's read_commit and read_tree must not depend on the caller's task-locals or async runtime. If the read fails, only the caller that started it gets the error; the others each read the object themselves.

clear_caches is "mainly intended for testing". Nothing invalidates entries otherwise, and nothing needs to: objects are immutable, so a cached commit or tree can never go stale.

4. Choosing a backend

ReadonlyRepo::init creates store/, runs the backend initializer, and writes the backend's name() into store/type. On load, StoreFactories::load_backend reads that file and looks the name up in a map of factories; an unknown name fails with "Unsupported commit backend type '…'". The default factories register Simple always, Git when the git feature is on, and Secret only under the testing feature. A program embedding jj-lib can merge its own factories in; registering a name twice panics. The same scheme picks the op store, op heads store, index and submodule store; see type files and factory dispatch.

5. The Git backend

GitBackend stores every commit, tree and blob as a real Git object, so Git tools see ordinary history. Where the Git repo lives (inside store/git, colocated, or external) is recorded in store/git_target; that and the extra/ directory are covered under store/, and colocation under Git colocation.

IDs

• Commit, tree and file IDs are Git object IDs, so commit_id_length is the repo's hash length (SHA-1 or SHA-256).
• The root commit ID is the all-zero "null" hash and the empty tree is Git's own empty tree (new). Since Git has no root commit, a jj commit whose only parent is the root is written as a Git root commit, and a merge that includes the root is written without it, since the root is an ancestor of every commit. Reading reverses this: a Git commit with no parents gets the root as its parent (read_commit).
• Change IDs are 16 bytes. A commit's change ID comes from its change-id header if it has one (stored as reverse hex); jj writes that header unless git.write-change-id-header is turned off. A commit made by plain Git has none, so jj derives one from the commit ID: the last 16 bytes, reversed, with each byte's bits reversed. The docs warn that "the exact algorithm … should not be relied upon".

Objects

read_tree re-sorts entries when needed, because Git sorts directory names "as if they had trailing '/'" and jj sorts plain names. An entry name that isn't a valid path component (Git never writes one like a/b, but a hand-made tree can hold it) is a read error, not a panic (source). write_file reads the whole stream into memory and hashes it first, skipping the write if Git already has the object. Tracked copies (read_copy and friends) are "not supported yet"; instead, get_copy_records runs gitoxide's rename tracking over the diff, with copies detected from modified files, a 50% similarity threshold and a limit of 1000.

What Git cannot hold

A jj commit has fields a Git commit does not. They go to three places:

• Commit headers. change-id as above. For a conflicted commit, jj:trees lists every term's tree ID and jj:conflict-labels the labels (write_commit); the Git commit's own tree is a readable stand-in with .jjconflict-* entries. Details on Conflicts.
• The extras table in store/extra/, keyed by commit ID. Each value is a git_store.Commit message holding the change ID, the predecessors, and a flag saying the commit uses the tree-level conflict format (serialize_extras). Predecessors are stored only here.
• Signature placeholders. Git rejects empty names and emails, so jj writes JJ_EMPTY_STRING in their place and maps it back on read.

A Git commit that jj never wrote or imported has no extras row. Rather than fail, read_commit calls import_head_commits on it, which walks its ancestors and adds a row for each one that lacks one; the source calls this a hack kept for repos imported by jj older than 0.8.0.

Writing a commit

1. Write the tree: the resolved tree ID, or a conflict stand-in tree.
2. Drop the root from the parents; refuse a merge with the root.
3. Add the extra headers, and write Git's empty tree if any term uses it.
4. Take the extras table lock, then loop: build the Git commit, sign it if asked (the signature goes in Git's gpgsig header, gpgsig-sha256 in a SHA-256 repo), write it, and look its ID up in the table. If the same ID already has different extras, which happens when two changes would otherwise be byte-identical Git commits (issue #694), move the committer time back one second and try again (source).
5. Add a refs/jj/keep/<id> ref so Git's GC cannot delete the commit (to_no_gc_ref_update), set the returned committer timestamp to the whole seconds Git stored, and save the extras row.

The table lock is held across the whole loop because, as the comment notes, two local writers producing the same ID with different metadata would otherwise lose one entry. It "is still racy if multiple machines are involved and the repository is rsync-ed."

concurrency is 1: reads go to a local object store through one mutex-guarded gix::Repository.

6. Garbage collection

Because every commit jj writes is pinned by a refs/jj/keep/* ref, Git's own git gc keeps them all, including abandoned ones. jj util gc (cmd_util_gc) first prunes old operations from the op store (see Operations) and then calls the backend's gc with the index's heads. The Git backend's:

1. recreates the keep refs so that only the commits the index still reports as heads are pinned (the source: "All commits found in the index won't be removed");
2. garbage-collects the extras table;
3. runs git gc --prune=<time> with the cutoff, so objects newer than it survive even if unreferenced, which protects commits another process is writing at the same time.

The cutoff defaults to 14 days ago; --expire now is the only other value accepted.

7. The Simple backend

SimpleBackend is jj's own format, with no Git underneath. init makes commits/, trees/, files/, symlinks/ and conflicts/ under store/ and writes the empty tree. Each object is one file named by the hex of its ID.

• IDs are BLAKE2b-512 hashes, so commit IDs are 64 bytes and change IDs 16. The root commit ID and root change ID are all zeros (load).
• write_file streams to a temp file while hashing, then moves it into place under its hash. read_file reads the whole file into memory.
• Trees and commits are simple_store.proto messages (schemas). write_commit signs the encoded message bytes, so a signature covers the exact stored bytes.
• Copy tracking is unsupported and get_copy_records returns an empty stream, so the Simple backend never reports renames.

8. The Secret backend

"Provides a backend for testing ACLs." SecretBackend wraps a GitBackend and forwards everything to it, except that read_file and read_symlink return ReadAccessDenied when the path contains "secret" or the content is exactly "secret" or "secret\n". Tests create it from a Git repo with adopt_git_repo, which overwrites store/type (marked "TODO: Avoid this hack"). It is only registered when jj-lib is built with the testing feature.

9. Side by side

Backend trait · Store · GitBackend · SimpleBackend · SecretBackend · default factories