Skip to content

Automatic Metrics API #407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 134 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
4926f78
Create automate-metrics.yaml
jukent Mar 11, 2024
7b0acfa
outline for automate-metrics.yaml
jukent Mar 11, 2024
b8511a2
Create get-metrics.py
jukent Mar 11, 2024
a9ed412
Update automate-metrics.yaml
jukent Mar 11, 2024
37cd664
Update automate-metrics.yaml
jukent Mar 11, 2024
8914529
Create metrics.md
jukent Mar 11, 2024
a19462d
Update footer-menu.html with metrics
jukent Mar 11, 2024
cdc525c
Update footer-menu.html
jukent Mar 11, 2024
aa0463b
Update footer-menu.html with metrics
jukent Mar 11, 2024
1918bae
Delete portal/metrics.md
jukent Mar 11, 2024
8b91ca4
Create metrics.md
jukent Mar 11, 2024
9f664fa
add json_extract to extensions
jukent Mar 11, 2024
2dafc21
Update get-metrics.py to also write file
jukent Mar 11, 2024
47fad36
Update get-metrics.py with returned condition
jukent Mar 11, 2024
9a4a657
Update automate-metrics.yaml
jukent Mar 11, 2024
f6dbf6d
pre-commit
jukent Mar 11, 2024
238edf2
use os to access passed in keys
jukent Mar 11, 2024
314b93d
fix double quoted strings
jukent Mar 11, 2024
ce7f60e
add jsonextract to environment.yml
jukent Mar 11, 2024
49ba11b
write metrics.md in a python file
jukent Mar 11, 2024
53c472c
rm from config
jukent Mar 11, 2024
99143ec
use env files
jukent Mar 11, 2024
7da6027
update comment
jukent Mar 11, 2024
e15d349
state with a table
jukent Mar 11, 2024
1d7c268
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 11, 2024
6e55f19
rm table because it won't format and add to toc tree
jukent Mar 11, 2024
73ad6f1
Merge branch 'metrics-api' of https://github.com/jukent/projectpythia…
jukent Mar 11, 2024
d5c040c
Merge pull request #119 from jukent/metrics-api
jukent Mar 11, 2024
99f8acf
add dispatch
jukent Mar 11, 2024
3e3f9ed
Merge pull request #120 from jukent/metrics-api
jukent Mar 11, 2024
ac5d69e
starting metrics file, and - vs _
jukent Mar 11, 2024
47da077
Merge pull request #121 from jukent/metrics-api
jukent Mar 11, 2024
2345bb0
pre-commit
jukent Mar 11, 2024
fc92d94
fix write-metrics
jukent Mar 11, 2024
5ac1362
Metrics API test (#408)
jukent Mar 11, 2024
4a18be6
make a metrics folder for holding future graphs
jukent Mar 12, 2024
5c449f0
missed a path herr
jukent Mar 12, 2024
ed3a388
add metrics automation as a step before site building
jukent Mar 12, 2024
820b976
rm automate-metrics schedule
jukent Mar 12, 2024
546fec2
Add last updated time to write-metrics
jukent Mar 12, 2024
411e54d
pre-commit
jukent Mar 13, 2024
402188c
Merge pull request #122 from jukent/metrics-api
jukent Mar 13, 2024
de6db3e
Update automate-metrics.yaml to use wget
jukent Mar 13, 2024
57b494f
Merge branch 'ProjectPythia:main' into main
jukent Mar 13, 2024
949ca30
Update automate-metrics.yaml
jukent Mar 13, 2024
a907dff
use curl
jukent Mar 15, 2024
7a3c583
Merge branch 'metrics-gh-action' into metrics-api
jukent Mar 15, 2024
f36f4b2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 15, 2024
b9335c2
Merge pull request #123 from jukent/metrics-api
jukent Mar 15, 2024
bb435b5
Update automate-metrics.yaml
jukent Mar 15, 2024
efbcb3e
Update automate-metrics.yaml fix file name
jukent Mar 15, 2024
5975536
Update automate-metrics.yaml
jukent Mar 15, 2024
5000655
Update automate-metrics.yaml
jukent Mar 15, 2024
6e3d0d5
Update automate-metrics.yaml
jukent Mar 15, 2024
7158ed6
Update get-metrics.py
jukent Mar 15, 2024
2d60b08
Update get-metrics.py
jukent Mar 15, 2024
3896a5e
Update automate-metrics.yaml
jukent Mar 15, 2024
6536a35
Update get-metrics.py
jukent Mar 15, 2024
9aeb2ac
Update automate-metrics.yaml
jukent Mar 15, 2024
5055a20
Update get-metrics.py
jukent Mar 15, 2024
5e51a43
Update get-metrics.py
jukent Mar 15, 2024
c3109d7
Update get-metrics.py
jukent Mar 15, 2024
f87d309
Update get-metrics.py
jukent Mar 15, 2024
c35f622
Update get-metrics.py
jukent Mar 15, 2024
3259172
Update get-metrics.py
jukent Mar 15, 2024
03d4c64
Update get-metrics.py
jukent Mar 15, 2024
b8a47fc
Update get-metrics.py
jukent Mar 15, 2024
6d0defc
Update get-metrics.py
jukent Mar 15, 2024
98d72b1
Update get-metrics.py
jukent Mar 15, 2024
1bd42b6
Update get-metrics.py
jukent Mar 15, 2024
3c4333b
Update get-metrics.py
jukent Mar 15, 2024
6778a5f
Update get-metrics.py to build dict
jukent Mar 15, 2024
83cc193
Update automate-metrics.yaml
jukent Mar 15, 2024
e601526
Update get-metrics.py
jukent Mar 15, 2024
49ee717
Update get-metrics.py
jukent Mar 15, 2024
7a4a66d
Update automate-metrics.yaml
jukent Mar 15, 2024
fe4f03b
Update get-metrics.py
jukent Mar 15, 2024
3779b3a
Update get-metrics.py
jukent Mar 15, 2024
b0e94ec
Update get-metrics.py
jukent Mar 15, 2024
7dc9faa
Update get-metrics.py
jukent Mar 15, 2024
c1cf304
Update automate-metrics.yaml
jukent Mar 15, 2024
8d65de2
Update get-metrics.py
jukent Mar 15, 2024
2742484
Update get-metrics.py
jukent Mar 15, 2024
def65be
Update get-metrics.py
jukent Mar 15, 2024
4d738c7
Update get-metrics.py
jukent Mar 15, 2024
151f56c
Update get-metrics.py
jukent Mar 15, 2024
9252577
Update get-metrics.py
jukent Mar 15, 2024
41b9f35
Update get-metrics.py
jukent Mar 15, 2024
32763a4
Update get-metrics.py
jukent Mar 15, 2024
983e830
Update get-metrics.py
jukent Mar 15, 2024
ababfc3
Update get-metrics.py
jukent Mar 15, 2024
d451527
Update get-metrics.py
jukent Mar 15, 2024
c1516f6
Update get-metrics.py
jukent Mar 15, 2024
6958ce4
Update automate-metrics.yaml
jukent Mar 15, 2024
39743c4
Update write-metrics-md.py
jukent Mar 15, 2024
404ff7f
Update get-metrics.py
jukent Mar 15, 2024
f419437
Update automate-metrics.yaml
jukent Mar 15, 2024
80331d0
Update nightly-build.yaml
jukent Mar 15, 2024
0b03712
Update publish-site.yaml
jukent Mar 15, 2024
715a3c4
Update trigger-preview.yaml
jukent Mar 15, 2024
eed44c5
Update trigger-site-build.yaml
jukent Mar 15, 2024
c000323
change from workflow_dispatch to workflow_call
jukent Mar 15, 2024
cbbf91b
Merge pull request #124 from jukent/main
jukent Mar 15, 2024
2157372
Update get-metrics.py
jukent Mar 15, 2024
6e93d32
Update get-metrics.py
jukent Mar 15, 2024
2ffd0eb
add files to gitignore
jukent Mar 16, 2024
c3865f7
add scripts for more plotting and nest automate-metrics better
jukent Mar 18, 2024
afa69bd
Merge pull request #125 from jukent/metrics-api
jukent Mar 18, 2024
2af7aec
print small portion of key
jukent Mar 18, 2024
4ca8cb3
use pip
jukent Mar 18, 2024
e536908
testing
jukent Mar 18, 2024
cdbb7f4
testing
jukent Mar 18, 2024
88f4063
print stmts
jukent Mar 18, 2024
8484848
more prints
jukent Mar 18, 2024
ef8942c
don't save
jukent Mar 18, 2024
943f82a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 18, 2024
2f95574
try -u before python call
jukent Mar 18, 2024
ce9561e
Merge branch 'metrics-api' of https://github.com/jukent/projectpythia…
jukent Mar 18, 2024
0ea9659
Try no quotes no $, move env files, combine pip
jukent Mar 18, 2024
0cfdc51
try printing in main
jukent Mar 18, 2024
c3616a0
Merge branch 'main' into metrics-api
jukent Mar 18, 2024
060eece
Merge pull request #126 from jukent/metrics-api
jukent Mar 18, 2024
5086626
Update get-metrics.py
jukent Mar 18, 2024
4dc512a
Update get-metrics.py
jukent Mar 18, 2024
7f97b4a
Update get-metrics.py
jukent Mar 18, 2024
e7f0a17
duplicate needs
jukent Mar 18, 2024
2282d73
try with linebreaks
jukent Mar 18, 2024
06592cc
try except
jukent Mar 18, 2024
3d728f3
valueerror
jukent Mar 18, 2024
6603f69
change the error
jukent Mar 18, 2024
45f38ce
import google
jukent Mar 18, 2024
b4393aa
change print move imports
jukent Mar 18, 2024
b9cbf02
isolate to part that fails
jukent Mar 18, 2024
b1f3d98
Merge branch 'main' into metrics-api
jukent Mar 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/get-metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import logging
import os

import google
from google.analytics.data_v1beta import BetaAnalyticsDataClient

PRIVATE_KEY_ID = os.environ.get('PRIVATE_KEY_ID')
PRIVATE_KEY = os.environ.get('PRIVATE_KEY')

credentials_dict = {
'type': 'service_account',
'project_id': 'cisl-vast-pythia',
'private_key_id': PRIVATE_KEY_ID,
'private_key': PRIVATE_KEY,
'client_email': '[email protected]',
'client_id': '113402578114110723940',
'auth_uri': 'https://accounts.google.com/o/oauth2/auth',
'token_uri': 'https://oauth2.googleapis.com/token',
'auth_provider_x509_cert_url': 'https://www.googleapis.com/oauth2/v1/certs',
'client_x509_cert_url': 'https://www.googleapis.com/robot/v1/metadata/x509/pythia-metrics-api%40cisl-vast-pythia.iam.gserviceaccount.com',
'universe_domain': 'googleapis.com',
}

try:
client = BetaAnalyticsDataClient.from_service_account_info(credentials_dict)
except google.auth.exceptions.MalformedError as e:
print('Malformed Error:', repr(e))
logging.error('Malformed Error:', repr(e))
logging.error('Credentials dict:', credentials_dict)

print(credentials_dict)
30 changes: 30 additions & 0 deletions .github/workflows/nightly-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,30 @@ on:
- cron: '0 0 * * *' # Daily “At 00:00”

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PORTAL_ID: ${{ secrets.PORTAL_ID }}
FOUNDATIONS_ID: ${{ secrets.FOUNDATIONS_ID }}
COOKBOOKS_ID: ${{ secrets.COOKBOOKS_ID }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/get-metrics.py
curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/write-metrics-md.py

python get-metrics.py
python write-metrics-md.py

build:
needs: automate-metrics
if: ${{ github.repository_owner == 'ProjectPythia' }}
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
Expand All @@ -20,3 +43,10 @@ jobs:
uses: ./.github/workflows/sphinx-link-checker.yaml
with:
path_to_source: 'portal'

deploy:
needs: build
uses: ProjectPythia/cookbook-actions/.github/workflows/deploy-book.yaml@main
with:
cname: projectpythia.org
publish_dir: 'portal/_build/html'
23 changes: 23 additions & 0 deletions .github/workflows/publish-site.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,30 @@ on:
workflow_dispatch:

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PORTAL_ID: ${{ secrets.PORTAL_ID }}
FOUNDATIONS_ID: ${{ secrets.FOUNDATIONS_ID }}
COOKBOOKS_ID: ${{ secrets.COOKBOOKS_ID }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/get-metrics.py
curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/write-metrics-md.py

python get-metrics.py
python write-metrics-md.py

build:
needs: automate-metrics
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
environment_file: 'environment.yml'
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/trigger-preview.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,30 @@ on:
- completed

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PORTAL_ID: ${{ secrets.PORTAL_ID }}
FOUNDATIONS_ID: ${{ secrets.FOUNDATIONS_ID }}
COOKBOOKS_ID: ${{ secrets.COOKBOOKS_ID }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/get-metrics.py
curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/write-metrics-md.py

python get-metrics.py
python write-metrics-md.py

find-pull-request:
needs: automate-metrics
uses: ProjectPythia/cookbook-actions/.github/workflows/find-pull-request.yaml@main
deploy-preview:
needs: find-pull-request
Expand Down
22 changes: 22 additions & 0 deletions .github/workflows/trigger-site-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,28 @@ on:
pull_request:

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PORTAL_ID: ${{ secrets.PORTAL_ID }}
FOUNDATIONS_ID: ${{ secrets.FOUNDATIONS_ID }}
COOKBOOKS_ID: ${{ secrets.COOKBOOKS_ID }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/get-metrics.py
curl -O https://raw.githubusercontent.com/jukent/projectpythia.github.io/main/.github/workflows/write-metrics-md.py

python get-metrics.py
python write-metrics-md.py

build:
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
Expand Down
38 changes: 38 additions & 0 deletions .github/workflows/write-metrics-md.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import datetime
import json


def process_user_data(user_data_file, bar_plot_file, map_plot_file, markdown_file):
"""
Reads user data from a JSON file and writes it to a markdown file.

Args:
user_data_file: Path to the JSON file containing user data.
markdown_file: Path to the output markdown file.
"""
now = datetime.datetime.now()

with open(user_data_file, 'r') as f:
user_data = json.load(f)

# Write processed data to markdown file
with open(markdown_file, 'w') as f:
f.write('# Metrics \n\n')
f.write(f'Last Updated: {now}\n\n')
f.write('Total Users:\n\n')
for key in user_data:
f.write(f'{key}: {(user_data[key])}\n')
f.write('\n')
f.write('![Top Pages](bar_plot_file)')
f.write('\n\n')
f.write('![Users by Country](../.github/workflows/map_plot_file)')
f.write('\n')
f.close()


if __name__ == '__main__':
user_data_file = 'user_metrics.json'
bar_plot_file = '../.github/workflows/bypage.png'
map_plot_file = '../.github/workflows/bycountry.png'
markdown_file = 'portal/metrics.md'
process_user_data(user_data_file, bar_plot_file, map_plot_file, markdown_file)
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,8 @@ portal/code_of_conduct.md
portal/resource-gallery.md
portal/resource-gallery/*.md
resource-gallery-submission-input.json

# Analytics
.github/workflows/analytics-api/
.github/workflows/*.json
.github/workflows/*.png
1 change: 1 addition & 0 deletions portal/_templates/footer-menu.html
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ <h1>{{ _("About") }}</h1>
<li><a href="https://projectpythia.org/about.html">{{ _("About Project Pythia") }}</a></li>
<li><a href="https://foundations.projectpythia.org/preamble/how-to-use.html">{{ _("How to use Pythia Foundations") }}</a></li>
<li><a href="https://projectpythia.org/#how-to-cite">{{ _("How to Cite") }}</a></li>
<li><a href="https://projectpythia.org/metrics.html">{{ _("Metrics") }}</a></li>
</ul>
</div>
<div class="col-9 col-sm-4 col-md-4 col-lg-3 footer-menu-col">
Expand Down
1 change: 1 addition & 0 deletions portal/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,4 +227,5 @@ contributing.md
code_of_conduct.md
cookbook-gallery.md
resource-gallery.md
metrics.md
```
1 change: 1 addition & 0 deletions portal/metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Metrics
Loading