Updating the production branch of the KCIDB mirrors

Background

The following repositories are mirrored on gitlab.com:

These repositories are set up in deployment-all via gitlab-repo-config to

• use GitLab pull mirroring linking to the corresponding project on GitHub
• have their issue tracker disabled

Branching workflow and production tagging

For kcidb-io and kcidb, a local branching workflow is set up to allow the management of a CKI production environment.

Both repositories have a downstream main-cki default branch that is independent of the upstream branches. In this branch, a .gitlab-ci.yml CI/CD configuration allows to run deployments into a production branch. This branch is then used for Python installations via something like

pip install git+https://gitlab.com/cki-project/mirror/kcidb-io.git@production

Updating the production branches

In a first step, update kcidb-io, which is used ubiquitously all over CKI. In a second step, update kcidb, which is only used in the datawarehouse-forwarder.

To update the production branch in one of the projects:

1. fork the project
2. in your fork, create a new branch based on the upstream main branch
3. propose a new merge request against the main-cki branch
4. in the merge request, child pipelines are triggered in the following projects:
   • kcidb-io: a child pipeline will trigger in cki-lib, which in turn triggers child pipelines all over CKI
   • kcidb: a child pipeline will trigger in cki-tools
5. an automatic production deployment is triggered when the merge request is merged

Updating KCIDB schema on consumers and producers

If you intend to support new fields in the latest schema, first you need to make sure all consumers are ready!

1. Start by upgrading CONSUMER_KCIDB_SCHEMA in datawarehouse/settings.py and add the new fields to the models and deserializers as desired. Make sure the produced data has not changed in this stage.
2. Upgrade cki-lib’s CONSUMER_KCIDB_SCHEMA in cki_lib/kcidb/validate.py, this will reflect in cki-tools, in upt, and in kernel-qe-tools, which will affect CKI pipelines and QE pipelines respectively.
3. Go back to datawarehouse/settings.py and upgrade PRODUCER_KCIDB_SCHEMA, taking this opportunity to update the serializers, changing the produced data.
4. Now, go back to cki_lib/kcidb/validate.py to upgrade PRODUCER_KCIDB_SCHEMA.
5. Finally, go to pipeline-definition and add the desired new fields.

To remove fields, make sure they aren’t being referenced anymore following the inverse logic, start in pipeline-definition, then datawarehouse serializers, followed by deserializers, and finally models. Make sure to split the work in DW in multiple MRs, otherwise you can face exceptions while each deployed instance is updated.