Reinstate go-ontology and go-annotation summary emails #1530

kltm · 2020-07-02T23:59:27Z

We are going to reinstate the go-ontology and go-annotation summary emails, which had been left behind when the old pipeline system finally went offline. The jobs were: go-github-annotation-tracker-summary-email and go-github-ontology-requests-summary-email.

For this, we'd like to keep it out of the main pipeline, as they are solely a function of GitHub, and are looking to implement them with GH actions. As well, we'd like to take this chance to migrate from the crufty old JS to standard and no-frills python.

Originally geneontology/helpdesk#259

Tagging @dustine32 @dougli1sqrd

kltm · 2020-07-03T00:06:46Z

Operationally:

go-annotation

Cron:
      <spec>H H(20-23) * * *</spec>

Commands:
npm install
node ./query-github-for-new-tickets.js -p go-annotation > ../../body.txt

Email profile:
          <email>
            <recipientList>[email protected],go-curator-tracker@l
ists.stanford.edu</recipientList>
            <subject>Github Annotation Tracker Update</subject>
            <body>${FILE,path=&quot;body.txt&quot;}</body>
            <recipientProviders/>
            <attachmentsPattern></attachmentsPattern>
            <attachBuildLog>false</attachBuildLog>
            <compressBuildLog>false</compressBuildLog>
            <replyTo>$PROJECT_DEFAULT_REPLYTO</replyTo>
            <contentType>project</contentType>
          </email>

go-ontology

Cron:
      <spec>H H(20-23) * * *</spec>

Commands:
npm install
node ./query-github-for-new-tickets.js > ../../body.txt

Email profile:
          <email>
            <recipientList>[email protected],[email protected]</recipientList>
            <subject>Github Ontology Tracker Update</subject>
            <body>${FILE,path=&quot;body.txt&quot;}</body>
            <recipientProviders/>
            <attachmentsPattern></attachmentsPattern>
            <attachBuildLog>false</attachBuildLog>
            <compressBuildLog>false</compressBuildLog>
            <replyTo>$PROJECT_DEFAULT_REPLYTO</replyTo>
            <contentType>project</contentType>
          </email>

kltm · 2020-07-03T00:07:56Z

See https://github.com/geneontology/go-site/blob/master/scripts/jenkins/query-github-for-new-tickets.js

pgaudet · 2020-09-02T11:05:33Z

@kltm Can you assign someone to do this ? It would be nice to have this for the GOC meeting in Oct.

dustine32 · 2020-09-02T14:25:25Z

@pgaudet @kltm I got pretty far with using a github action for this in this repo: https://github.com/dustine32/daily-tracker-email

I'll move the repo over to https://github.com/geneontology soon.

pgaudet · 2020-09-02T14:26:54Z

Thanks !

kltm · 2020-10-16T00:46:03Z

Talking to @dustine32, we'll be needing to move on to some testing to try working with an SMTP server (either GH or LBL, depending).

kltm · 2020-10-21T23:18:31Z

Thank you @dustine32 .

@vanaukenk Here is some example output from the "new" iteration of the script:

sjcarbon@moiraine:~/local/src/git/go-site[master]$:( python3 scripts/github_issue_summary.py geneontology/go-annotation 1

Summary for tickets from 2020-10-20 to 2020-10-21

New Tickets

There are 42 new tickets.

3372 PTN000275634 not mRNA cis splicing, via spliceosome |
3371 PTN002768818
3370 PTN000129548, PTN000567445, PTN000567445, PTN000566947, PTN000397856, PTN000567445, PTN000328844, PTN000298794
3369 PTN000403280
3368 PTN000263578
3367 PTN000423706 not mitochondrion organization
3366 PTN000246311 not mitochondrial respiratory chain complex III assembly/ PTN000312316
3365 PTN000277108 NOT mitochondrial outer membrane translocase complex assembly
3364 IPR039131
3363 PTN000395816 mitochondrion morphogenesis (challenge: does not exist)
3362 PTN000170085 / dynamin family protein polymerization involved in mitochondrial fission
3361 PTN000902162, PTN000326738 not All mitochondrial fission
3360 PTN002166039 not spindle assembly
3359 PTN001997934
3358 PTN000195421
3357 PTN000447085
3356 stg1 / SPAC4F8.10c PTN005319067
3355 PTN000549861 pls remove from SPAC29A4.05
3354 PTN001052565 NOT establishment of mitotic spindle orientation
3353 PTN000286904 NOT meiotic sister chromatid cohesion
3352 PTN001854091NOT meiotic sister chromatid cohesion
3351 PTN000286904 NOT meiotic sister chromatid cohesion
3350 PTN000835262
3349 PTN000150497
3348 PTN001000243
3347 PTN000207233
3346 PTN000012880 / PTN000030137 NOT mitochondrial ATP synthesis coupled proton transport
3345 PTN000127027
3344 PTN002001446
3343 PTN000373411 rmn1, not nucleocytoplasmic export
3342 PTN001114632
3341 ribosome biogenesis and ribosomal subunits
3340 PTN000184007/ PTN000391312 /PTN000242048
3339 PTN000204884 / PTN000337444/ PTN000428162
3338 PTN005171867 SPAC1B3.06c
3337 InterPro2GO: Add ATPase activity to SMC subunit entries?
3336 InterPro2GO: Heat shock protein Hsp90 family (IPR001404)
3335 InterPro2GO: Heat shock protein 70 family (IPR013126)
3334 InterPro2Go - GroES chaperonin (IPR020818)
3333 InterPro2GO: Add ATPase activity to TRiC/CCT complex subunit entries?
3332 InterPro2GO: Adenylate kinase isoenzyme 6 (IPR020618)
3330 new mapping: Glycosyl transferase, family 1 InterPro IPR001296 -> GO:0016757 glycosyltransferase

Updated Tickets

There are 7 updated tickets.

3331 PTN001582398 EKC/KEOPS complex
3324 PTN000164255 lap2 lipid metabolism
3303 InterPro2GO - Myosin tail (IPR002928)
3255 Review annotations to GO:0001300 chronological cell aging and children
3192 CoA pyrophosphatase activity (GO:0010945) annotations
2788 Remove GO:0008375 from Glycosyl transferase, family 14 (IPR003406)
2378 PANTHER:PTN000156123 Serpins without activity

I believe that this is a straight port of the old script. Also note that it is sending in HTML (which I think is weird, but I so don't want to rock the boat on this one--I guess everybody else allows that in their incoming email and that's apparently what we had before).

If this looks okay, we can start playing around with getting this sent out.

pgaudet · 2020-10-22T05:17:27Z

Looks good ! Although I thought people were mostly interested in the go-ontology repo. In any case, can you change the header so that it states which repo is the report from ?

ie Summary for tickets from 2020-10-20 to 2020-10-21
-> change to Summary for tickets in go-annotation from 2020-10-20 to 2020-10-21

(& likewise for go-ontology).

Thanks, Pascale

vanaukenk · 2020-10-22T13:59:13Z

Thanks @dustine32 and @kltm

This looks good.

Let's start with go-annotation and go-ontology and take it from there.

kltm · 2021-02-02T01:56:15Z

Summary to date:

What is missing from the ticket are some discussions that I had with @dustine32 on the side. Essentially, what we had been hoping to do was take advantage of GitHub Actions (now that Travis is going away) to send the email periodically and easily. While Dustin was able to make it mostly work, creating the scripts and using the GH API, the actual email sending part turned out to be a difficult prospect using GHA. After a few tries, we decided it might, in fact, be better handled by the pipeline, as the email environment was a little better understood at the BBOP site.

The core issue that has been run into here is that most outgoing email servers tend to be locked to individual identities and fixed or paid services, so getting the email out with parts-off-the-shelf approach did work out as we had hoped; related to that is that sending sorta-anonymous email to mailing lists gets flagged as spam.

With these issues, the ball was in my court to try and get this done through the mail servers that BBOP has access to, or find another way forward. Unfortunately, I have been otherwise occupied with various issues and have not been able to commit enough time to make progress. That said, we've had a little more experience as a group with GHA now, so I've just asked internally if there is anybody who could help out with this.

kltm · 2021-02-02T02:22:17Z

I think there are three ways forward here. I would propose the following path:

Try again to make the GHA method work. While possibly the "hardest", this is also the easiest for us to work on and change as a group, and would truly be a group resource (as opposed to tying it to a pipeline or a cron on a machine). I believe that this is our ideal and it would be worth taking a final crack at. Perhaps we could start with https://github.com/marketplace/actions/send-email or something similar. Failing another attempt at that, we can...
Add this as a cron job to a standing server at BBOP (although another site would be fine). This would be along the lines of 1) letting site security know what we are doing (see recent excitement) and 2) attaching the job to run periodically in a very straightforward manner. This is almost guaranteed to work, but also is the most tied to a specific context and person at a specific site and is completely hidden. This is the worst long-term thing to do, but maybe we could do this at the end of the week until we have exhausted ourselves to our satisfaction that 1 isn't going to work as a temporary measure.
If we believe that 1 is off the table, to guarantee a group resource, I would then suggest that we bolt this into a pipeline (or the pipeline,TBD). While this is a little clumsy, it at least is a good centralized and public place to keep things until we come up with something better.

@dustine32 Would you be willing and available to take another crack at this? Ping me and we can chat.

kltm · 2021-02-16T20:10:23Z

Status update on this.
@dustine32 and @kltm are still chipping away on the best way to deliver the email, and whether to use LBL (GMail) or USC (MS) as the backend. We're still hoping for a neutral solution--@dustine32 looking at GHA runners to get it on on the correct side of the firewall and @kltm looking at the migration what LBL is doing for SMTP and GMail.

kltm · 2021-04-29T22:45:14Z

Pinging @dustine32 for any further news for the GHA effort.
After some experiments on AWS EC2 with SES and talking to @dustine32, we are going to go ahead and pilot/test this within the pipeline framework, which should allow us to restart the emails while we work.

dustine32 · 2021-04-30T00:15:32Z

Thanks @kltm!

For the GHA route, I just need to finish coding the OAuth2 login step for accessing the USC mail server in order to send out the email from GitHub's servers.

pgaudet · 2021-04-30T07:57:51Z

Will we be ready to mention this at the GOC meeting next week? @suzialeksander this could go in the 'Communication' presentation.

kltm · 2021-05-03T23:40:08Z

@pgaudet @suzialeksander You should be able to mention it in some capacity; I'm not sure it will be finalized by then, but things are quicker now that we're putting a temp hold on the GHA route. We still need to cycle owner @vanaukenk , who may be a little busy at this point :)

dustine32 · 2021-05-04T20:10:43Z

@pgaudet Is the full geneontology/go-ontology repo name OK in the header? As in:

Summary for tickets in geneontology/go-ontology from 2021-05-03 to 2021-05-04

Or we can get surgical if we don't want the "geneontology/" part.

suzialeksander · 2021-05-04T20:49:15Z

I have a slight personal preference for just go-ontology, but if it's too much bother I think the full header is fine.

…eader Change str formatting for compatibility and update header for #1530

dustine32 · 2021-05-04T22:53:27Z

@suzialeksander No prob! Just dropped the org part. This change should be safe.

…eader GH issue search now full 24hr, for #1530

dustine32 · 2021-05-06T13:29:16Z

From @pgaudet: Filter out pull requests in email.

dustine32 · 2021-05-06T16:37:40Z

Easy to do, actually. Thanks GH API!
https://docs.github.com/en/github/searching-for-information-on-github/searching-issues-and-pull-requests#search-only-issues-or-pull-requests

…eader Filter for only issues (not PRs) for #1530

kltm added the enhancement label Jul 2, 2020

kltm mentioned this issue Jul 2, 2020

tracker update to mailing list geneontology/helpdesk#259

Closed

dustine32 self-assigned this Sep 2, 2020

dustine32 added a commit that referenced this issue Oct 20, 2020

GitHub API caller script for daily issue summaries - #1530

8a77255

dustine32 mentioned this issue Oct 20, 2020

GitHub API caller script for daily issue summaries #1563

Merged

kltm added a commit to geneontology/pipeline that referenced this issue May 3, 2021

work on geneontology/go-site#1530

966a3c2

kltm added a commit to geneontology/pipeline that referenced this issue May 3, 2021

first untested draft for geneontology/go-site#1530

d73a735

kltm added a commit to geneontology/pipeline that referenced this issue May 3, 2021

script block; for geneontology/go-site#1530

f66044b

kltm added a commit to geneontology/pipeline that referenced this issue May 3, 2021

less chatty; for geneontology/go-site#1530

6cede52

kltm added a commit to geneontology/pipeline that referenced this issue May 4, 2021

switch to python 3.6; work on geneontology/go-site#1530

9e042d3

kltm added a commit to geneontology/pipeline that referenced this issue May 4, 2021

add in both targets; work on geneontology/go-site#1530

64e6f93

kltm added a commit to geneontology/pipeline that referenced this issue May 4, 2021

try html mime-type; work on geneontology/go-site#1530

2774cb0

kltm added a commit to geneontology/pipeline that referenced this issue May 4, 2021

add @vanaukenk; work on geneontology/go-site#1530

9a9a712

dustine32 added a commit that referenced this issue May 4, 2021

Change str formatting for compatability and update header for #1530

a6df633

dustine32 added a commit that referenced this issue May 4, 2021

Truncate repo name to remove org, for #1530

557c11a

kltm added a commit that referenced this issue May 4, 2021

Merge pull request #1677 from geneontology/issue-1530-summary-email-h…

28a2173

…eader Change str formatting for compatibility and update header for #1530

dustine32 added a commit that referenced this issue May 5, 2021

GH issue search now full 24hr, for #1530

e9abe81

kltm added a commit that referenced this issue May 5, 2021

Merge pull request #1679 from geneontology/issue-1530-summary-email-h…

526b556

…eader GH issue search now full 24hr, for #1530

kltm added a commit to geneontology/pipeline that referenced this issue May 5, 2021

wait 15 minutes before triggering; work on geneontology/go-site#1530

a263bea

kltm added a commit to geneontology/pipeline that referenced this issue May 5, 2021

use go-consortium address; work on geneontology/go-site#1530

0f7a53d

dustine32 added a commit that referenced this issue May 6, 2021

Filter for only issues (not PRs) for #1530

779ca5b

kltm added a commit that referenced this issue May 6, 2021

Merge pull request #1680 from geneontology/issue-1530-summary-email-h…

963988e

…eader Filter for only issues (not PRs) for #1530

kltm closed this as completed Oct 26, 2021

cmungall mentioned this issue Aug 17, 2022

Add an action that emails daily summaries of changes to a mailing list INCATools/ontology-development-kit#660

Closed

kltm added this to Pipeline and Imports Aug 22, 2024

kltm moved this to Done in Pipeline and Imports Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reinstate go-ontology and go-annotation summary emails #1530

Reinstate go-ontology and go-annotation summary emails #1530

kltm commented Jul 2, 2020

kltm commented Jul 3, 2020

kltm commented Jul 3, 2020

pgaudet commented Sep 2, 2020

dustine32 commented Sep 2, 2020

pgaudet commented Sep 2, 2020

kltm commented Oct 16, 2020

kltm commented Oct 21, 2020

pgaudet commented Oct 22, 2020

vanaukenk commented Oct 22, 2020

kltm commented Feb 2, 2021

kltm commented Feb 2, 2021

kltm commented Feb 16, 2021

kltm commented Apr 29, 2021 •

edited

Loading

dustine32 commented Apr 30, 2021

pgaudet commented Apr 30, 2021

kltm commented May 3, 2021

dustine32 commented May 4, 2021

suzialeksander commented May 4, 2021

dustine32 commented May 4, 2021

dustine32 commented May 6, 2021

dustine32 commented May 6, 2021

Reinstate go-ontology and go-annotation summary emails #1530

Reinstate go-ontology and go-annotation summary emails #1530

Comments

kltm commented Jul 2, 2020

kltm commented Jul 3, 2020

go-annotation

go-ontology

kltm commented Jul 3, 2020

pgaudet commented Sep 2, 2020

dustine32 commented Sep 2, 2020

pgaudet commented Sep 2, 2020

kltm commented Oct 16, 2020

kltm commented Oct 21, 2020

Summary for tickets from 2020-10-20 to 2020-10-21

New Tickets

Updated Tickets

pgaudet commented Oct 22, 2020

vanaukenk commented Oct 22, 2020

kltm commented Feb 2, 2021

kltm commented Feb 2, 2021

kltm commented Feb 16, 2021

kltm commented Apr 29, 2021 • edited Loading

dustine32 commented Apr 30, 2021

pgaudet commented Apr 30, 2021

kltm commented May 3, 2021

dustine32 commented May 4, 2021

suzialeksander commented May 4, 2021

dustine32 commented May 4, 2021

dustine32 commented May 6, 2021

dustine32 commented May 6, 2021

kltm commented Apr 29, 2021 •

edited

Loading