Feature: Enhance Analaytics module to fetch Ontology visits, User vis…

…its, and the Most visited pages in the month (#17) * remove forgot variables * fix for #61 - create contact instance if it doesn't exist - changed --from-api to --from-apikey - minor linting * Restore branch specifier to develop * Optimization - remove repeated query * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * Gemfile had references to develop branch * implemented #64 - ability to generate labels independently of RDF processing (and vise versa) * Gemfile.lock update * fixed a bug in #64 * Relocate docker-compose file and update default configs * Add GH workflow for publishing docker images * use ruby native method for listing files instead of a git function Resolves warning messages when we exclude .git directory from docker image * remove comment * capitalize argument in order to be consistent with other scripts * add arm/64 platform * additional error handling for SPAM deletion script, #60 * additional error handling for SPAM deletion script, #60 * implemented #67 - improved corrupt data and error handling * Gemfile.lock update * exclude test/data/dictionary.txt from git commits * update version of solr-ut * Gemfile.lock update * Restore branch specifier to master * fixed configuration for the analytics module * Gemfile.lock update * implemented #69 - scheduled annotator dictionary file generation should be a configurable option instead of the default * Gemfile.lock update * gem update * create new rake taks for updating purls for all ontologies moved from ontologies_api/fix_purls.rb * initial implementation of #70 - Google Analytics v4 Update Compatibility Issue * added the /data folder to ignore * update gems * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * use patched version of agraph v7.3.1 * unpin faraday gem * A chnage to reference Analytics Redis from LinkedData block * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * use assert_operator instead of assert minitest style guide adherence. encountered an intermittent unit test failure so assert_operator will provide better failure feedback than assert * use local solr to pass the tests * fixed ncbo_ontology_archive_old_submissions error output * Gemfile.lock update * Gemfile.lock update * Gemfile update * Gemfile update * fixes to the analytics script and a new script to generate UA analytics for documentation * Gemfile.lock update * Gemfile.lock update * implemented the first pass at bmir-radx/radx-project#37 * implemented the first pass at bmir-radx/radx-project#37 * set bundler version to be comptatible with ruby 2.7 + AG v8 * refactor ontologies analytics job to handle the new google analytics migration * add user analytics fetching the monthly user visits count * add page visits analytics fetching last month most visited pages * extract google analytics UA import code to a script to make current code clean of it * add option to force submission archiving even if already archived --------- Co-authored-by: Alex Skrenchuk <[email protected]> Co-authored-by: mdorf <[email protected]> Co-authored-by: Jennifer Vendetti <[email protected]>
ontoportal-lirmm · Dec 28, 2023 · 8b9edab · 8b9edab
1 parent 987fba3
commit 8b9edab
Show file tree

Hide file tree

Showing 36 changed files with 1,543 additions and 519 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -1,5 +1,6 @@
 # Git
-#.git
+.git
+.github
 .gitignore
 # Logs
 log/*
@@ -8,3 +9,9 @@ tmp/*
 # Editor temp files
 *.swp
 *.swo
+coverage
+create_permissions.log
+# Ignore generated test data
+test/data/dictionary.txt
+test/data/ontology_files/repo/**/*
+test/data/tmp/*
diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml
@@ -0,0 +1,42 @@
+name: Docker Image CI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  push_to_registry:
+    name: Push Docker image to Docker Hub
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: bioportal/ncbo_cron
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          build-args: |
+            RUBY_VERSION=2.7
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
diff --git a/.github/workflows/ruby-unit-tests.yml b/.github/workflows/ruby-unit-tests.yml
@@ -7,18 +7,17 @@ on:
 jobs:
   test:
     strategy:
+      fail-fast: false
       matrix:
-        backend: ['ruby'] # ruby runs tests with 4store backend and ruby-agraph runs with AllegroGraph backend
+        backend: ['ncbo_cron'] # ruby runs tests with 4store backend and ruby-agraph runs with AllegroGraph backend
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
     - name: copy config.rb file from template
       run:  cp config/config.rb.sample config/config.rb
     - name: Build docker-compose
-      working-directory: ./test
       run: docker-compose build
     - name: Run unit tests
-      working-directory: ./test
       run: |
         ci_env=`bash <(curl -s https://codecov.io/env)`
         docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} bundle exec rake test TESTOPTS='-v'

diff --git a/.gitignore b/.gitignore
@@ -5,6 +5,8 @@ config/config.rb
 config/appliance.rb
 config/config_*.rb
 config/*.p12
+config/*.json
+data/
 projectFilesBackup/
 .ruby-version
 repo*

diff --git a/Dockerfile b/Dockerfile
@@ -14,10 +14,16 @@ COPY Gemfile* *.gemspec /srv/ontoportal/ncbo_cron/
 
 WORKDIR /srv/ontoportal/ncbo_cron
 
+# set rubygem and bundler to the last version supported by ruby 2.7
+# remove version after ruby v3 upgrade
+RUN gem update --system '3.4.22'
+RUN gem install bundler -v '2.4.22'
 RUN gem update --system
 RUN gem install bundler
 ENV BUNDLE_PATH=/srv/ontoportal/bundle
 RUN bundle install
 
 COPY . /srv/ontoportal/ncbo_cron
+RUN cp /srv/ontoportal/ncbo_cron/config/config.rb.sample /srv/ontoportal/ncbo_cron/config/config.rb
+
 CMD ["/bin/bash"]
diff --git a/Gemfile b/Gemfile
@@ -2,13 +2,17 @@ source 'https://rubygems.org'
 
 gemspec
 
-gem 'faraday', '~> 1.9'
 gem 'ffi'
+
+# This is needed temporarily to pull the Google Universal Analytics (UA)
+# data and store it in a file. See (bin/import_google_ua_analytics_data)
+# The ability to pull this data from Google will cease on July 1, 2024
 gem "google-apis-analytics_v3"
+
+gem 'google-analytics-data'
 gem 'mail', '2.6.6'
-gem 'minitest', '< 5.0'
 gem 'multi_json'
-gem 'oj', '~> 2.0'
+gem 'oj', '~> 3.0'
 gem 'parseconfig'
 gem 'pony'
 gem 'pry'
@@ -21,13 +25,14 @@ gem 'request_store'
 # Monitoring
 gem 'cube-ruby', require: 'cube'
 
-gem 'goo', github: 'ontoportal-lirmm/goo', branch: 'development'
+gem 'goo', github: 'ontoportal-lirmm/goo', branch: 'master'
 gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
-gem 'ontologies_linked_data', github: 'ontoportal-lirmm/ontologies_linked_data', branch: 'development'
+gem 'ontologies_linked_data', github: 'ontoportal-lirmm/ontologies_linked_data', branch: 'master'
 gem 'ncbo_annotator', github: 'ontoportal-lirmm/ncbo_annotator', branch: 'master'
 # Testing
 group :test do
   gem 'email_spec'
+  gem 'minitest', '< 5.0'
   gem 'simplecov'
   gem 'simplecov-cobertura' # for codecov.io
   gem 'test-unit-minitest'

diff --git a/Gemfile.lock b/Gemfile.lock
@@ -1,7 +1,7 @@
 GIT
   remote: https://github.com/ontoportal-lirmm/goo.git
-  revision: 9aa0ccacc9d76bff096218e6d48edc5b0bffd54f
-  branch: development
+  revision: 03da25b671d2ffa515b5dce51c6bd35980ae60c7
+  branch: master
   specs:
     goo (0.0.2)
       addressable (~> 2.8)
@@ -26,8 +26,8 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/ontologies_linked_data.git
-  revision: 1cfaf4482b7bf8c9001f0ff309f2a0ff06f683b5
-  branch: development
+  revision: e98b884999e5ce917a8be5fdc37f7b4797a1559e
+  branch: master
   specs:
     ontologies_linked_data (0.0.1)
       activesupport
@@ -60,7 +60,7 @@ PATH
     ncbo_cron (0.0.1)
       dante
       goo
-      google-apis-analytics_v3
+      google-analytics-data
       mlanett-redis-lock
       multi_json
       ncbo_annotator
@@ -76,7 +76,9 @@ GEM
       multi_json (~> 1.0)
     addressable (2.8.6)
       public_suffix (>= 2.0.2, < 6.0)
+    base64 (0.2.0)
     bcrypt (3.1.20)
+    bigdecimal (3.1.5)
     binding_of_caller (1.0.0)
       debug_inspector (>= 0.0.1)
     builder (3.2.4)
@@ -85,38 +87,36 @@ GEM
     connection_pool (2.4.1)
     cube-ruby (0.0.3)
     dante (0.2.0)
-    debug_inspector (1.1.0)
+    debug_inspector (1.2.0)
     declarative (0.0.20)
     docile (1.4.0)
     domain_name (0.6.20231109)
     email_spec (2.1.1)
       htmlentities (~> 4.3.3)
       launchy (~> 2.1)
       mail (~> 2.6)
-    faraday (1.10.3)
-      faraday-em_http (~> 1.0)
-      faraday-em_synchrony (~> 1.0)
-      faraday-excon (~> 1.1)
-      faraday-httpclient (~> 1.0)
-      faraday-multipart (~> 1.0)
-      faraday-net_http (~> 1.0)
-      faraday-net_http_persistent (~> 1.0)
-      faraday-patron (~> 1.0)
-      faraday-rack (~> 1.0)
-      faraday-retry (~> 1.0)
+    faraday (2.8.1)
+      base64
+      faraday-net_http (>= 2.0, < 3.1)
       ruby2_keywords (>= 0.0.4)
-    faraday-em_http (1.0.0)
-    faraday-em_synchrony (1.0.0)
-    faraday-excon (1.1.0)
-    faraday-httpclient (1.0.1)
-    faraday-multipart (1.0.4)
-      multipart-post (~> 2)
-    faraday-net_http (1.0.1)
-    faraday-net_http_persistent (1.2.0)
-    faraday-patron (1.0.0)
-    faraday-rack (1.0.0)
-    faraday-retry (1.0.3)
+    faraday-net_http (3.0.2)
+    faraday-retry (2.2.0)
+      faraday (~> 2.0)
     ffi (1.16.3)
+    gapic-common (0.21.1)
+      faraday (>= 1.9, < 3.a)
+      faraday-retry (>= 1.0, < 3.a)
+      google-protobuf (~> 3.18)
+      googleapis-common-protos (>= 1.4.0, < 2.a)
+      googleapis-common-protos-types (>= 1.11.0, < 2.a)
+      googleauth (~> 1.9)
+      grpc (~> 1.59)
+    google-analytics-data (0.4.0)
+      google-analytics-data-v1beta (>= 0.7, < 2.a)
+      google-cloud-core (~> 1.6)
+    google-analytics-data-v1beta (0.10.0)
+      gapic-common (>= 0.20.0, < 2.a)
+      google-cloud-errors (~> 1.0)
     google-apis-analytics_v3 (0.13.0)
       google-apis-core (>= 0.11.0, < 2.a)
     google-apis-core (0.11.2)
@@ -128,15 +128,33 @@ GEM
       retriable (>= 2.0, < 4.a)
       rexml
       webrick
-    google-cloud-env (2.0.1)
+    google-cloud-core (1.6.1)
+      google-cloud-env (>= 1.0, < 3.a)
+      google-cloud-errors (~> 1.0)
+    google-cloud-env (2.1.0)
       faraday (>= 1.0, < 3.a)
-    googleauth (1.9.0)
+    google-cloud-errors (1.3.1)
+    google-protobuf (3.25.1-x86_64-darwin)
+    google-protobuf (3.25.1-x86_64-linux)
+    googleapis-common-protos (1.4.0)
+      google-protobuf (~> 3.14)
+      googleapis-common-protos-types (~> 1.2)
+      grpc (~> 1.27)
+    googleapis-common-protos-types (1.11.0)
+      google-protobuf (~> 3.18)
+    googleauth (1.9.1)
       faraday (>= 1.0, < 3.a)
-      google-cloud-env (~> 2.0, >= 2.0.1)
+      google-cloud-env (~> 2.1)
       jwt (>= 1.4, < 3.0)
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
       signet (>= 0.16, < 2.a)
+    grpc (1.60.0-x86_64-darwin)
+      google-protobuf (~> 3.25)
+      googleapis-common-protos-types (~> 1.0)
+    grpc (1.60.0-x86_64-linux)
+      google-protobuf (~> 3.25)
+      googleapis-common-protos-types (~> 1.0)
     htmlentities (4.3.4)
     http-accept (1.7.0)
     http-cookie (1.0.5)
@@ -164,10 +182,10 @@ GEM
     mlanett-redis-lock (0.2.7)
       redis
     multi_json (1.15.0)
-    multipart-post (2.3.0)
     net-http-persistent (2.9.4)
     netrc (0.11.0)
-    oj (2.18.5)
+    oj (3.16.3)
+      bigdecimal (>= 3.0)
     omni_logger (0.1.4)
       logger
     os (1.1.4)
@@ -186,7 +204,7 @@ GEM
       addressable (>= 2.2)
     redis (5.0.8)
       redis-client (>= 0.17.0)
-    redis-client (0.19.0)
+    redis-client (0.19.1)
       connection_pool
     representable (3.2.0)
       declarative (< 0.1.0)
@@ -244,16 +262,16 @@ DEPENDENCIES
   binding_of_caller (~> 1.0)
   cube-ruby
   email_spec
-  faraday (~> 1.9)
   ffi
   goo!
+  google-analytics-data
   google-apis-analytics_v3
   mail (= 2.6.6)
   minitest (< 5.0)
   multi_json
   ncbo_annotator!
   ncbo_cron!
-  oj (~> 2.0)
+  oj (~> 3.0)
   ontologies_linked_data!
   parseconfig
   pony