Skip to content

Commit 4241eac

Browse files
fsteegfsteeg
authored
Merge pull request #43 from metafacture/28-use-jekyll-theme
Use jekyll theme for documentation
2 parents a3f0a22 + 035c82f commit 4241eac

20 files changed

+393
-95
lines changed

‎Gemfile

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
source 'https://rubygems.org'
2+
3+
gem "jekyll", "~> 4.3.3" # installed by `gem jekyll`
4+
# gem "webrick" # required when using Ruby >= 3 and Jekyll <= 4.2.2
5+
6+
gem "just-the-docs", "0.8.1" # pinned to the current release
7+
# gem "just-the-docs" # always download the latest release

‎README.md

+5-60
Original file line numberDiff line numberDiff line change
@@ -2,71 +2,16 @@
22

33
# Metafacture Documentation
44

5-
Metafacture is a toolkit for processing semi-structured data with a focus on library metadata. It provides a versatile set of tools for reading, writing and transforming data. Metafacture can be used as a stand-alone application via CLI or as a Java library in other applications. There is also a playground where you can test workflows.
5+
Metafacture is a toolkit for processing semi-structured data with a focus on library metadata
6+
For the documentation see: http://metafacture.github.io/metafacture-documentation/
67

7-
[Have a look here for a quick intro into metafacture.](./MF-in-5-min.md)
8+
This repo is the central place for the documentation about Metafacture that we render using the jekyll template [Just the docs](https://github.com/just-the-docs/just-the-docs).
89

9-
This is the central place for the documentation about Metafacture.
10-
11-
Metafacture comprises three main parts: **Framework**, **Flux** and one of the **Transformation-Modules Fix and Morph**. It can be extended with modules.
10+
The content pages can be found [here](/docs/)
1211

1312
> [!NOTE]
1413
> With regard to the Transformation-Modules this documentation focusses on Fix instead of MORPH. If you want to find out more about MORPH. Have a look at [the old documentation](https://github.com/metafacture/metafacture-core/wiki/Metamorph-User-Guide) and the german cookbook by [Swissbib](https://swissbib.gitlab.io/metamorph-doku/).
1514
1615

17-
Our goal with this repo is to collaboratively create comprehensive documentation on Metafacture in the [issue tracker](https://github.com/culturegraph/metafacture-documentation/issues?q=). Feel free to open issues not only for bugs or enhancements, but also questions about Metafacture usage, or to share your experiences. We hope that over time, in that way we can create useful tutorials, how-tos, and collect good practices for using Metafacture.
18-
19-
__________________
20-
21-
Deciding which parts are relevant to you depends on the way you are using Metafacture:
22-
23-
## Using Metafacture via playground or CLI
24-
25-
> [!NOTE]
26-
> No Java-Code is necessary!!!
27-
28-
While working with the playground or the command line you only need [Flux](#flux) and the transformation module [Fix](#fix).
29-
Have a look here for [Getting started](/Getting-Started.md).
30-
31-
## Framework for Java integration/development
32-
33-
If you plan to use Metafacture as a Java library or if you wish to add commands to Flux. You should get familar with the [Framework](#framework).
34-
35-
__________________
36-
37-
## FLUX
38-
39-
Flux is a scripting language to easily build and run processing pipelines. No Java programming is necessary - it's used as a command line. To use Flux you may download the binary distribution of Metafacture.
40-
41-
For more information on how to use Flux, see the [Flux User Guide](/Flux-User-Guide.md).
42-
43-
See [here a list for all available FLUX-Commands](/flux-commands.md).
44-
45-
__________________
46-
47-
## FIX
48-
49-
Metafix is a domain specific language for metadata transformation based on Catmandu FIX. The FIX object performing the transformation is used as part of a processing pipeline.
50-
51-
If you are using **Metafacture with CLI or Playground** and therefore the Flux scripting language to build and run pipelines, use the `fix` command in your FLUX-Pipeline.
52-
53-
If you are using **Metafacture as a Java library**, just create a Metafix object and add it to your pipeline (see also the [Framework User Guide](#framework)).
54-
55-
The transformation itself is declared in a fix-object which can be a file. For more information on how to declare transformations see [Metafix User Guide](/Fix-User-Guide.md).
56-
57-
See [here a list for all available FIX functions and a cookbook for using fix](/Fix-function-and-Cookbook.md).
58-
59-
> [!NOTE]
60-
> PS: There is also the transformation modul MORPH. Have a look at[ the old documentation](https://github.com/metafacture/metafacture-core/wiki/Metamorph-User-Guide) and the german cookbook by [Swissbib](https://swissbib.gitlab.io/metamorph-doku/).
61-
62-
__________________
63-
64-
## Framework
65-
66-
> [!NOTE]
67-
>Relevant for developers
68-
69-
The framework includes the interfaces and abstract classes which form the foundation of the data processing pipelines. This part of Metafacture is only relevant for you if you plan to use Metafacture as a Java library or if you wish to add pipe elements to Flux.
70-
71-
For more information see the [Framework User Guide](/Framework-User-Guide.md).
16+
Our goal with this repo is to collaboratively create comprehensive documentation on Metafacture in the [issue tracker](https://github.com/metafacture/metafacture-documentation/issues?q=). Feel free to open issues not only for bugs or enhancements, but also questions about Metafacture usage, or to share your experiences. We hope that over time, in that way we can create useful tutorials, how-tos, and collect good practices for using Metafacture.
7217

‎_config.yml

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
title: Metafacture Documentation
2+
description: This is the central place for the documentation about Metafacture.
3+
theme: just-the-docs
4+
5+
url: https://metafacture.github.io/metafacture-documentation
6+
7+
aux_links:
8+
Metafacture Documentation on Github: https://github.com/metafacture/metafacture-documentation
9+
10+
# External navigation links
11+
nav_external_links:
12+
- title: LICENSE
13+
url: https://github.com/metafacture/metafacture-core/blob/master/LICENSE
14+
15+
# Set a path/url to a logo that will be displayed instead of the title
16+
logo: "/assets/images/metafacture_small.png"
17+
favicon_ico: "/assets/images/favicon.ico"
18+
19+
# Footer content
20+
# appears at the bottom of every page's main content
21+
22+
# Back to top link
23+
back_to_top: true
24+
back_to_top_text: "Back to top"
25+
26+
footer_content: 'Metafacture Documentation is maintained by the <a href="https://lobid.org/team-en/">Open infrastructure team of hbz.</a>'
27+
28+
# Footer last edited timestamp
29+
last_edit_timestamp: true # show or hide edit time - page must have `last_modified_date` defined in the frontmatter
30+
last_edit_time_format: "%b %e %Y at %I:%M %p" # uses ruby's time format: https://ruby-doc.org/stdlib-2.7.0/libdoc/time/rdoc/Time.html
31+
32+
# Custom colors, see _sass/color_schemes/metafacture.scss
33+
color_scheme: metafacture
34+
35+
# see https://just-the-docs.github.io/just-the-docs/docs/search/#search-granularity
36+
search:
37+
heading_level: 6
38+
previews: 6
39+
40+
# Callouts, see https://just-the-docs.com/docs/configuration/#callouts
41+
callouts_level: quiet # or loud
42+
callouts:
43+
highlight:
44+
color: yellow
45+
important:
46+
title: Important
47+
color: blue
48+
new:
49+
title: New
50+
color: green
51+
note:
52+
title: Note
53+
color: purple
54+
warning:
55+
title: Warning
56+
color: red

‎_sass/color_schemes/metafacture.scss

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
$link-color: #4183c4;

‎_sass/custom/custom.scss

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
h4 {font-size: 1rem !important;}
2+
h5 {
3+
font-size: 1.5rem !important;
4+
}

‎assets/images/favicon.ico

1.54 KB
Binary file not shown.

‎assets/images/metafacture_small.png

24.2 KB
Loading

‎Approaching a transformation with metafacture.md renamed to ‎docs/Approaching a transformation with metafacture.md

+6
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
---
2+
layout: default
3+
title: Approaching a transformation with metafacture
4+
nav_order: 4
5+
---
6+
17
Every approach to transform metadata with metafacture is quite similiar:
28

39
- You need to know the type and source of the input and the type and destination of the output:

‎Documentation-Maintainer-Guide.md renamed to ‎docs/Documentation-Maintainer-Guide.md

+12-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1+
---
2+
layout: default
3+
title: Maintainer Guide
4+
nav_order: 8
5+
---
16

2-
## how to change flux-commands.md
7+
# Maintainer Guide
38

4-
The entries in flux-commands.md describe the usage of commands used by flux.
9+
## how to change [docs/flux/flux-commands.md](https://github.com/metafacture/metafacture-documentation/blob/28-use-jekyll-theme/docs/flux/flux-commands.md)
10+
11+
The entries in [docs/flux/flux-commands.md](https://github.com/metafacture/metafacture-documentation/blob/28-use-jekyll-theme/docs/flux/flux-commands.md) describe the usage of commands used by flux.
512
flux-commands.md is fully automatically generated. To make this happen one has to
613
fill in the proper annotations in the correponding java classes. E.g.
714

@@ -30,7 +37,7 @@ The option's name is produced by cutting away the "set" from the methods name, l
3037
"BatchSize" which is then lowercased. The parameter of this option is generated from the
3138
parameter type of the method - here an "int"eger.
3239

33-
## how to publish flux-commands.md
40+
## how to publish [docs/flux/flux-commands.md](https://github.com/metafacture/metafacture-documentation/blob/28-use-jekyll-theme/docs/flux/flux-commands.md)
3441

3542
If you have updated some of these annotations, say "description", and these changes are
3643
merged into the master branch, generate a new flux-commands.md like this:
@@ -43,6 +50,7 @@ $ flux.sh > flux-commands.md
4350
```
4451

4552
Open the generated flux-commands.md and remove some boilerplate at the beginning of the
46-
file manually. Save it, copy it here, commit and push.
53+
file manually. Add the naviagtion part of the page, save it, copy it [here](https://github.com/metafacture/metafacture-documentation/blob/28-use-jekyll-theme/docs/flux/flux-commands.md), commit and push.
54+
4755

4856
The [publishing process will be automated with an github action](https://github.com/metafacture/metafacture-core/issues/368).

‎Getting-Started.md renamed to ‎docs/Getting-Started.md

+11-8
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,20 @@
1-
![logo](https://github.com/culturegraph/metafacture-core/wiki/img/metafacture_small.png)
2-
1+
---
2+
layout: default
3+
title: Getting Started
4+
nav_order: 2
5+
---
36

47
# Getting started!
58

69
## Playground
710

811
The easiest way to get started with Metafacture is the Playground. Take a look at the [first example](https://metafacture.org/playground/?example=encode-xml) and run it by pressing the !["Process"](https://metafacture.org/img/process.png) button. Check out the other examples (first button, !["Load Examples"](https://metafacture.org/img/load-exmples.png)) for different input sources, transformations, and output formats.
912

10-
For commands available in the Flux, see [the Flux commands documentation](/flux-commands.md).
13+
For commands available in the Flux, see [the Flux commands documentation](flux/flux-commands.html).
1114

12-
For functions and usage of the Fix, see [the Fix functions and cookbook](/Fix-functions-and-cookbook).
15+
For functions and usage of the Fix, see [the Fix functions and cookbook](fix/Fix-functions-and-Cookbook.html).
1316

14-
For next steps get familar with [FLUX](/Flux-User-Guide.md) and [FIX](/Fix-User-Guide.md). And try out some Metafacture workflows.
17+
For next steps get familar with [FLUX](flux/Flux-User-Guide.html) and [FIX](fix/Fix-User-Guide.html). And try out some Metafacture workflows.
1518

1619
## Command line
1720

@@ -22,9 +25,9 @@ To use Metafacture as a command-line tool, download the latest metafix-runner fr
2225

2326
To get started, you can export a workflow from the Playground (last button, !["Export Workflow"](https://metafacture.org/img/export.png)).
2427

25-
To set up IDE support for editing your Flux and Fix files, see [the IDE extensions page](https://metafacture.org/ide-extensions/index.html).
28+
To set up IDE support for editing your Flux and Fix files, see [the IDE extensions page](https://metafacture.org/ide-extensions.html).
2629

27-
For next steps get familar with [FLUX](/Flux-User-Guide.md) and [FIX](/Fix-User-Guide.md). And try out some Metafacture workflows.
30+
For next steps get familar with [FLUX](flux/Flux-User-Guide.html) and [FIX](fix/Fix-User-Guide.html). And try out some Metafacture workflows.
2831

2932
## Using Metafacture as a Java library
3033

@@ -55,4 +58,4 @@ To use Fix you would declare `metafix` instead of `metafacture-io` as in the exa
5558
Occasionally, we publish snapshot builds on [Sonatype OSS Repository](https://oss.sonatype.org/index.html#nexus-search;gav~org.metafacture~~~~~kw,versionexpand). The version number is derived from the branch name. Snapshot builds from the master branch always have the version `master-SNAPSHOT`. We also provide sometimes pre releases as github packages.
5659

5760

58-
If you plan to use Metafacture as a Java library or if you wish to add commands to Flux you should get familar with the [Framework](/Framework-User-Guide.md).
61+
If you plan to use Metafacture as a Java library or if you wish to add commands to Flux you should get familar with the [Framework](java-integration/Framework-User-Guide.html).

‎MF-in-5-min.md renamed to ‎docs/MF-in-5-min.md

+9-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
---
2+
layout: default
3+
title: 5 min Intro into MF
4+
nav_order: 3
5+
---
6+
17
# Introduction - A quick 5 minute introduction to Metafacture
28

39
## HELLO WORLD
@@ -87,7 +93,7 @@ $ cat yaml2json.flux
8793

8894
## FIX LANGUAGE
8995

90-
Many data conversions need a mapping from one field to another field plus optional conversions of the data inside these fields. Metafacture provides the transformation module `fix` that uses the Catmandu-inspired Fix language to assist in these mappings. A full list Fix functions is available at https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions.
96+
Many data conversions need a mapping from one field to another field plus optional conversions of the data inside these fields. Metafacture provides the transformation module `fix` that uses the Catmandu-inspired Fix language to assist in these mappings. See the [full list of Fix functions](fix/Fix-functions-and-Cookbook.html#functions).
9197

9298
Fixes can be provided inline as text argument of the fix module in the Flux script, or as a pointer to a Fix script. A Fix script groups one or more fixes in a file.
9399

@@ -173,6 +179,6 @@ The 245 field with its subfields of each MARC record is mapped to the title fiel
173179
The `retain` Fix function keeps only the title field in the output. (In contrast to Catmandu there are no special marc or pica fixes since the internal handling of records and elements is more generic. Also the internal serialization of MARC is not as complex as in Catmandu.)
174180

175181

176-
Now you should be ready to [get started](./Getting-Started.md).
182+
Now you should be ready to [get started](Getting-Started.html).
177183

178-
(Note: This mini introduction to Metafacture is inspired by the mini introduction to Catmandu here: https://metacpan.org/dist/Catmandu/view/lib/Catmandu/Introduction.pod)
184+
(Note: This mini introduction to Metafacture is inspired by the mini [introduction to Catmandu](https://metacpan.org/dist/Catmandu/view/lib/Catmandu/Introduction.pod))

‎Fix-User-Guide.md renamed to ‎docs/fix/Fix-User-Guide.md

+8-3
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,16 @@
1-
![logo](https://github.com/culturegraph/metafacture-core/wiki/img/metafacture_small.png)
1+
---
2+
layout: default
3+
title: Fix User Guide
4+
parent: Fix
5+
nav_order: 1
6+
---
27

38
# Fix User Guide
49

510
This document provides an introduction to the Metafacture Fix language (short: Metafix or Fix). The Fix language for Metafacture is introduced as an alternative to configuring data transformations with Metamorph. Inspired by Catmandu Fix, Metafix processes metadata not as a continuous data stream but as discrete records.
611

712
## Part of a metafacture worflow
8-
Metafacture Fix is a transformation module that can be used in a [Flux Workflow](/Flux-User-Guide.md), for this you have to use this in your pipeline:
13+
Metafacture Fix is a transformation module that can be used in a [Flux Workflow](../flux/Flux-User-Guide.html), for this you have to use this in your pipeline:
914

1015
Flux-Example:
1116
```PERL
@@ -85,7 +90,7 @@ do Bind(params,…)
8590
end
8691
```
8792

88-
Find here a [list of all functions, selectors, binds and conditionals](/Fix-function-and-Cookbook.md).
93+
Find here a [list of all functions, selectors, binds and conditionals](Fix-functions-and-Cookbook.html).
8994

9095

9196
## Addressing Pieces of Data: FIX-Path and the record structure in FIX

0 commit comments

Comments
 (0)