Skip to content

Commit 3ea21ee

Browse files
line-oline-o
authored
Merge pull request #57 from eeditiones/fix/github-file-download
improve GitHub API usage
2 parents c5f503e + e5b6497 commit 3ea21ee

File tree

3 files changed

+248
-104
lines changed

3 files changed

+248
-104
lines changed

‎README.md

+99-52
Original file line numberDiff line numberDiff line change
@@ -31,77 +31,120 @@ To initialize the project and load dependencies run
3131
3232
| Run | Description |
3333
|---------|-------------|
34-
|```gulp build```|to just build Tuttle. |
35-
|```gulp install```|To build and install Tuttle in one go|
34+
|```npm run build```|to just build Tuttle. |
35+
|```npm run deploy```|To build and install Tuttle in one go|
3636

3737
The resulting xar(s) are found in the root of the project.
3838

3939
## Testing
4040

41-
To run the local test suite you need an instance of eXist running on `localhost:8080` and `npm` to be available in your path.
42-
Run tests with ```npm test```
41+
To run the local test suite you need an
42+
* instance of eXist running on `localhost:8080` and
43+
* `npm` to be available in your path
44+
* a GitHub personal access token with read access to public repositories
45+
* a gitlab personal access token with read access to public repositories
46+
47+
In CI these access tokens are read from environment variables.
48+
You can do the same with
49+
```bash
50+
export tuttle_token_tuttle_sample_data=<GITHUB_PAT>; \
51+
export tuttle_token_gitlab_sample_data=<GITLAB_PAT>; \
52+
path/to/startup.sh
53+
```
54+
55+
Alternatively, you can modify `/db/apps/tuttle/data/tuttle.xml` _and_ `test/fixtures/alt-tuttle.xml` to include your tokens. But remember to never commit them!
56+
57+
Run tests with ```npm test```
4358

4459
## Configuration
4560

4661
Tuttle is configured in `data/tuttle.xml`.
4762

48-
### Gitservice configuration
49-
@name is always the name of the destination collection. It will be configured in `data/tuttle.xml`
63+
New with version 2.0.0:
64+
65+
A commented example configuration is available `data/tuttle-example-config.xml`.
66+
If you want to update tuttle your modified configuration file will be backed up to
67+
`/db/tuttle-backup/tuttle.xml` and restored on installation of the new version.
68+
69+
Otherwise, when no back up of an existing config-file is found, the example configuration is copied to `data/tuttle.xml`.
70+
71+
> [!TIP]
72+
> When migrating from an earlier version you can copy your existing configuration to the backup location:
73+
> `xmldb:copy-resource('/db/apps/tuttle/data', 'tuttle.xml', '/db/tuttle-backup', 'tuttle.xml')`
74+
75+
### Repository configuration
76+
77+
The repositories to keep in sync with a gitservice are all listed under the repos-element.
78+
79+
The name-attribute refers to the **destination collection** also known as the **target collection**.
80+
81+
#### Collection
82+
83+
An example: `<collection name="tuttle-sample-data">`
84+
The collection `/db/apps/tuttle-sample-data` is now considered to be kept in sync with a git repository.
5085

51-
An example:
5286
```xml
53-
<repos>
54-
<collection name="tuttle-sample-data">
55-
<default>true</default>
56-
<type>github</type>
57-
<baseurl>https://api.github.com/</baseurl>
58-
<repo>tuttle-sample-data</repo>
59-
<owner>tuttle-sample-data</owner>
60-
<token></token>
61-
<ref>master</ref>
62-
<hookuser>admin</hookuser>
63-
<hookpasswd></hookpasswd>
64-
</collection>
65-
66-
<collection name="tuttle-sample-gitlab">
67-
<type>gitlab</type>
68-
<baseurl>https://gitlab.com/api/v4/</baseurl>
69-
<project-id>tuttle-sample-data</project-id>
70-
<token>XXX</token>
71-
<ref>master</ref>
72-
<hookuser>admin</hookuser>
73-
<hookpasswd></hookpasswd>
74-
</collection>
75-
</repos>
87+
<collection name="tuttle-sample-data">
88+
<default>true</default>
89+
90+
<type>github</type>
91+
<baseurl>https://api.github.com/</baseurl>
92+
93+
<repo>tuttle-sample-data</repo>
94+
<owner>tuttle-sample-data</owner>
95+
96+
<token>a-personal-access-token</token>
97+
98+
<ref>a-branch</ref>
99+
100+
<hookuser>a-exist-user</hookuser>
101+
<hookpasswd>that-users-password</hookpasswd>
102+
</collection>
76103
```
77104

78105
#### type
79-
Gitserver type: 'github' or 'gitlab'
80106

81-
#### baseurl
82-
* For github the baseurl is always api.github.com
83-
* For gitlab the url can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'
107+
```xml
108+
<type>gitlab</type>
109+
```
110+
111+
There are two supported git services at the moment `github` and `gitlab`
112+
113+
#### baseurl
114+
115+
```xml
116+
<baseurl>https://api.server/</baseurl>
117+
```
118+
119+
* For github the baseurl is `https://api.github.com/` or your github-enterprise API endpoint
120+
* For gitlab the baseurl is `https://gitlab.com/api/v4/` but can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'
121+
122+
#### repo, owner and project-id
123+
124+
* For github you **have to** specify the owner and the repo
125+
* For gitlab you **have to** specify the project-id of the repository
84126

85-
#### repo, owner and project-id
86-
* For github you have to specify the owner and the repo
87-
* For gitlab you have to specify the project-id of the repository
88127

89-
#### ref
90-
Define the working branch of the git repository
128+
#### ref
129+
130+
```xml
131+
<ref>main</ref>
132+
```
133+
134+
Defines the branch you want to track.
91135

92136
#### hookuser & hookpasswd
93137

94138
#### token
95139

96140
If a token is specified Tuttle authenticates against GitHub or GitLab. When a token is not defined, Tuttle assumes a public repository without any authentication.
97141

98-
It is also possible to pass the token via an environment variable. The name of the variable have to be `tuttle_token_ + collection` (all dashes must be replaces by underscore). Example: `tuttle_token_tuttle_sample_data`
99-
100-
Be aware of the rate limits
101-
* GitHub:
102-
* 60 unauthenticated requests per hour
103-
* 5,000 authenticated requests per hour
142+
> [!NOTE]
143+
> Be aware of the rate limits for unauthenticated requests
144+
> GitHub allows 60 unauthenticated requests per hour but 5,000 for authenticated requests
104145
146+
> [!TIP]
147+
> It is also possible to pass the token via an environment variable. The name of the variable have to be `tuttle_token_ + collection` (all dashes must be replaces by underscore). Example: `tuttle_token_tuttle_sample_data`
105148
106149
##### Create API-Keys for Github / Gitlab
107150

@@ -123,17 +166,22 @@ The dashboard can trigger a full deployment or an incremental update.
123166
Full deployment clones the repository from git and install it as a `.xar` file.
124167
With incremental update only the changes to the database collection are applied.
125168

169+
> [!NOTE]
170+
> Tuttle is built to keep track of **data collections**
171+
172+
> [!NOTE]
173+
> Tuttle is does not run pre- or post install scripts nor change the index configuration on incremental updates!
174+
126175
### Lets start
127176

128-
1) customize the configuration (`modules/config.xql`)
177+
1) customize the configuration (`data/tuttle.xml`)
129178
2) click on 'full' to trigger a full deployment from git to existdb
130179
3) now you can update your collection with a click on 'incremental'
131180

132181
Repositories from which a valid XAR (existing `expath-pkg.xml` and `repo.xml`) package can be generated are installed as a package, all others are created purely on the DB.
133182

134-
**REMARK: Note that there may be index problems if a collection is not installed as a package.**
135-
136-
**REMARK: Only use it with data collections**
183+
> [!NOTE]
184+
> Note that there may be index problems if a collection is not installed as a package.
137185
138186
## API
139187

@@ -243,10 +291,9 @@ be obtained from the respective service.
243291

244292
The key for the gitservice must be configured in Gitservice configuration as shown above.
245293

294+
## Roadmap
246295

247-
## DB to Git
248-
249-
Will be implemented in release 2.0.0
296+
- [ ] DB to Git
250297

251298
## Honorable mentions:
252299

‎src/modules/api.xql

+1-1
Original file line numberDiff line numberDiff line change
@@ -277,7 +277,7 @@ declare function api:git-deploy($request as map(*)) as map(*) {
277277

278278
let $remove-staging := collection:remove($staging, true())
279279
let $remove-lock := app:lock-remove($destination)
280-
280+
let $reindex := xmldb:reindex($destination)
281281
return map {
282282
"hash": config:deployed-sha($destination),
283283
"message": "success"

0 commit comments

Comments
 (0)