A Copier Template for LinkML Projects

This template uses the code-scaffolding tool copier to create a LinkML project. Copier supports code lifecycle management, allowing you to seamlessly incorporate updates into your project when the template is enhanced.

• Starting from 0.2.x we give up compatibility with the directory layout from linkml-project-cookiecutter that we followed initially. This is required to introduce new features and to realise our idea of a cleaner, easier to update linkml project template.
• Early releases (0.1.x) are backwards compatibility with linkml-project-cookiecutter (same directory layout and commands). This facilitates experimenting with the migration of existing cruft/cookiecutter-based projects. Over time the migration is expected to become more difficult as the cookiecutter template evolves. We don't plan to maintain compatible releases beyond v0.1.7.

The generated project uses just as preferred command runner, even in the 0.1.x releases.

The starting point of this template was linkml-project-cookiecutter (commit 1094cf2, date 2025-01-10). The code from the just command runner PR was also included (up to commit 3eb2522).

Prerequisites

The following are required and recommended tools for using this copier template and the LinkML project that it generates. This is all one-time setup, so if you have already done it skip to the next section! We assume that you have full internet access.

• git / GitHub account

  Git is the version management system with which this template and your repository are managed. The template also assumes that you use GitHub for hosting your project which implies that you have a GitHub account.

• Python >= 3.9

  LinkML tools are mainly written in Python, so you will need a recent Python interpreter to run this generator and to use the generated project.

• pipx

  pipx is a tool for managing isolated Python-based applications. It is the recommended way to install the required tools. To install pipx follow their instructions.

• Poetry

  Poetry is a Python project management tool. You will use it in your generated project to manage dependencies and build distribution files. Install Poetry by running:

  pipx install poetry

  This will install Poetry 2.x as required by this project. If you also need Poetry 1.x for other projects, you can have both Poetry 2.x and Poetry 1.x installed at the same time. pipx has the option to install another version with a suffix-modified name, here "poetry1",

  pipx install --suffix=1 "poetry<2.0"

• Poetry Dynamic Versioning Plugin:

  This plugin automatically updates certain version strings in your generated project when you publish it. Install it by running:

  pipx inject poetry "poetry-dynamic-versioning[plugin]"

• copier

  Copier is a tool for generating projects based on a template (like this one!). It also allows re-configuring the projects and to keep them updated when the original template changes. To insert dates into the template, copier requires jinja2_time in the copier environment. Install both with pipx by running:

  pipx install copier
  pipx inject copier jinja2_time

• just

  The project contains a justfile with pre-defined complex commands. To execute these commands you need just as command runner. Install it by running:

  pipx install rust-just

Creating a new project

Step 1: Generate the project files

To generate a new LinkML project first create a new empty directory for the project and then run the following:

cd path/to/new/directory
copier copy --trust https://github.com/dalito/linkml-project-copier .

The --trust option is needed because the template uses the jinja_extension jinja2_time.

You will be prompted a few questions. The defaults are fine for most projects, but pick the name for your project carefully as it will also be used as project name on GitHub.

It is also possible to use non-default branches or specific tags via --vcs-ref which is useful when developing the template:

copier copy --trust --vcs-ref branch-name gh:dalito/linkml-project-copier ./path/to/destination

Step 2: Set up the LinkML project

Change to the folder your generated project is in.

Optionally customize your project if needed:

• pass arguments to linkml generators via 'config.yaml' configuration file;
• pass supported environment variables via '.env.public' configuration file;

Setup your project

cd my-awesome-schema  # using the folder example above
just setup

Step 3: Edit the schema

Edit the schema (the .yaml file) in the src/my_awesome_schema/schema folder with an editor of your choice.

For developing consistent, well-formatted schemas, the project provides a pre-commit configuration for some helpful tools. Included are yamllint for consistent formatting of the schema-yaml file, ruff for formatting and linting Python code and the spell checkers codespell and typos. To use this

• install pre-commit with: pipx install pre-commit
• activate pre-commit in the project by running (at the root of the project): pre-commit install

Once installed pre-commit will perform the checks on every commit and reject a commit if errors are found; it will even auto-correct several types of errors. You can also run the pre-configured checks manually with pre-commit run -a.

Step 4: Validate the schema

just test

This commands generates the project artefacts from the schema, runs pytest for the Python datamodel and tests loading all valid & invalid data examples.

Another important command to check your schema is

just lint

which runs the linkML linter on your schema.

Step 5: Generate documentation locally

LinkML generates schema documentation automatically. The template includes the configuration for generating and publishing the documentation with GitHub whenever you push schema changes to GitHub. The published documentation can be found at a URL like this one: https://{github-user-or-organization}.github.io/{project-name}/

You can also preview the documentation locally before pushing to GitHub by running:

just testdoc

Step 6: Create a GitHub project

1. Open GitHub new project and follow the instructions, being sure to NOT add a README or .gitignore file (this copier template will add those files for you)

2. Add the remote to your local git repository:

   git remote add origin https://github.com/{github-user-or-organization}/{project-name}.git
   git branch -M main
   git push -u origin main

3. Configure your repository for deploying the documentation as GitHub pages

• Under Settings > Actions > General in section "Workflow Permissions" mark "Read repository and packages permission".
• Go to "Actions" tab, select on the left under Actions "Deploy docs", and click the "Run workflow" button on the right. Run from main-branch as suggested and verify successful completion.
• Now go back to Settings > Pages. In section "Build and Deployment" select
  • Under "Source": "Deploy from a branch"
  • Under "Branch": "gh-pages" and "/ (root)"
    • Hint: The "gh-pages" branch is created automatically in the first successful run of the "deploy docs" workflow.

Step 7: Register the schema

See How to register a schema

Making releases

See How to Manage Releases of your LinkML Schema

Migrating an existing project to use this template

This is a rough guide on the required steps. Feedback and suggestions for improvement based on your experiences are very welcome. The commands are written to be run at the root of your project.

• Start with a clean state of the existing project (check with git status).

• Create a new branch and activate it:

  git switch -c migrate-to-copier

• Adapt your project and create a copier answers file (.copier-answers) by running (this updates to the latest released version):

  copier copy --trust gh:dalito/linkml-project-copier .

  • Starting from a linkml-project-cookiecutter based project: You may want to migrate in two steps to reduces the number of changes to review. For the first step, a migration to the 0.1.x-series is suggested, which still has the same directory layout as linkml-project-cookiecutter. The command for migrating to a specific tag/release is:

    copier copy --trust --vcs-ref v0.1.7 gh:dalito/linkml-project-copier .

    Look into the .cruft.json file to find out which values you chose when you created your original project. Be sure to enter the same values when answering the copier questions.

• Carefully review the changes that copier made to your project.

• If you used a cruft/cookiecutter template before, you may delete the cruft file .cruft.json.

• If you are happy, commit all changes to the migrate-to-copier branch.

• To finalise the migration, merge the migrate-to-copier branch to your main branch.

Keeping your project up to date or changing its configuration

Copier allows you to update your project with changes from the template. You can also change the project by providing different answers to the questions than the last time.

To update your project with changes from the template and to reconfigure your project options, run:

copier update --trust

To do a pure update without re-configuration run:

copier update --trust --skip-answered

If you initialized the project from a non-default branch, you must add the git branch name (or tag name) also to the update commands:

copier update --trust --vcs-ref branch-or-tag-name --skip-answered

When updating, Copier will do its best to respect the project evolution by using the answers provided last time. However, sometimes this is impossible and conflicts occur. They will be inlined into the conflicting files and can be resolved just like any other git conflict.

For more on updating see copier's documentation.

Notes on specific updates

From 0.1.x to 0.2.x

The directory layout has changed a lot with the v0.2.0 release. Series 0.1.x had still the directory layout from linkml-project-cookiecutter. For the update some directories have to be cleaned up. Since copier can't do this automatically, you have to do the following steps manually:

• Before running the update: Run just clean to remove the folder docs which was git-ignored in 0.1.x (and linkml-cookiecutter) but will become the main version-managed docs folder in 0.2.x.
• After running the update (before committing): Remove the folder project/docs which was also gitignored before v0.2.0 and is of no use for 0.2.0 and later releases.

Contributors

A big thanks to all contributors.

Main author:

• David Linke (ORCID: 0000-0002-5898-1820) - Idea, initial setup of repository and current maintainer.

License

The code in this repository is distributed under MIT license.

Acknowledgement

linkml-project-cookiecutter provided a great basis for starting this new linkml project template.