Issue Tracker Migration

2024-10-31 by Pere Lev

Data Migration

Before we begin: As always, my task board is available.

Data migration, in this context, is my ability as the user, to copy or move my data from one server to another server on the network. Often in web applications this service isn't offered, and causes what's often called vendor lock-in: I'm stuck with the server I chose, and whoever is managing and hosting it.

What if my current server is shutting down? Or what if the terms of use are changing? It's really a fundamental part of data freedom, the ability to download my data and continue using it in a new place. I don't start over; I rather continue, with the very same data, just on a new server.

Migrating repositories is the easier part: You can always git clone a repo, and git push it somewhere else. But how to do the same with the other kinds of project data?

Vocabulary, Format, Process

I decided to use the F3 format for representing project data. At the time of writing, it's slightly incomplete, so I added a little extension:

An IssueTracker object
The Issue object has a comments property
The Comment object has a children property
I picked a content type, application/f3+json

This way the entire data of the tracker can be represented as a single JSON object (another option might be to download each object separately, or provide a .tar.gz containing the directory tree of issues and comments).

Run this command to see some example data:

curl 'https://grape.fr33domlover.site/decks/OZLdZ' -H 'Accept: application/f3+json'

On the ForgeFed side, I picked the AS2 origin property, to represent the source from which migration is or was done. So, when creating a new project/tracker/etc., specifying the origin is the way to request migration.

The process is quite simple:

(Currently) every tracker servers a F3 version of its data
The newly created tracker downloads the data and creates local copies, and links pointing back to the originals

I added a switch in Vervis settings, to make migrations opt-in, since this feature could probably be abused easily.

Other Project Data

So, issue migration is now available in Vervis! But what about the other kinds of project data:

Repository
Pull Request
Project (i.e. a collection of such components)

With the foundations in place, these will come in the near future :) I suspect the least trivial one is PRs, which is why I'm adding it to my 2025 roadmap. Repo and project are easier, and I hope they come in 2025 as well.

Implementation

Here's a summary of the changes I made in Vervis:

Added a switch in the settings, as mentioned above
In the DB, in UI and in the ForgeFed vocabulary: Tracker, issue and comment optionally specify their origin object
When a Factory is asked to create a new tracker with an origin, it creates the tracker, and then sends it a message requesting it to download the data from that origin and create copies
When a tracker is created, it optionally downloads the origin data and makes copies

And here's a list of commits:

Funding

I really want to thank NLnet for funding this work! The extended grant is allowing me to continue backend work, and allowing André to work on the Anvil frontend.

See It in Action

I recorded a little demo of all this! Watch it on my PeerTube instance.

If you want to play with things yourself, you can create account(s) on the demo instances - fig, grape, walnut - and try the things I've mentioned and done in the video.

If you encounter any bugs, let me know! Or open an issue

Comments

Come chat with us on Matrix!

And we have an account for ForgeFed on the Fediverse: https://floss.social/@forgefed

Right after publishing this post, I'll make a toot there to announce the post, and you can comment there :)