Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Synda Certificate Renew Fails #121

Open
ericnienhouse opened this issue Aug 5, 2019 · 21 comments
Open

Synda Certificate Renew Fails #121

ericnienhouse opened this issue Aug 5, 2019 · 21 comments

Comments

@ericnienhouse
Copy link

ericnienhouse commented Aug 5, 2019 •
edited
Loading

We've recently had our synda certificate renew fail, always:

$ synda certificate --debug renew
Error occurs while renewing certificate ([('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')])

It is unclear what may have changed in our local (or remote server) configuration to affect this. It seems related to protocol support by the myproxy server?

Note we're using: esgf-node.llnl.gov:7512

Note: Adding --debug does not seem to provide additional information about the failure.
@senesis
Copy link

senesis commented Aug 8, 2019

The same occurs for me, for a fresh install, at first trial of a 'synda install' command, and on 'synda certificate renew'

@AtefBN
Copy link
Contributor

AtefBN commented Aug 19, 2019

The error message is pretty cryptic and not a good indication of what went wrong in my experience with synda. But in my previous encounters, it has always been a wrong version of a dependency (openssl or myproxyclient mostly). Can you verify they're in order?

@senesis
Copy link

senesis commented Aug 20, 2019

Updating a synda 3.9 DEB install with a synda 3.10 conda install did, in my case, update openssl, and led to a successful 'synda certificate renew'

@hot007
Copy link

hot007 commented Jan 13, 2020

I had the same problem using the centos71 RPM on a centos7.6 system, which is synda-3.8-1.x86_64.
Removed the RPM and tried instead with conda. This was successful and synda seems to be fundamentally working (can install files).

@hot007
Copy link

hot007 commented Jan 13, 2020

I say "fundamentally" because I've found that in working through the synda intro examples, one that used to work now fails:

> synda search -f cmip5.output1.MPI-M.MPI-ESM-LR.decadal1995.mon.land.Lmon.r2i1p1.v20120529 baresoilFrac
WARNING: 'baresoilFrac' value has been associated with 'variable_id' facet.
WARNING: 'baresoilFrac' value has been associated with 'variable_id' facet.
File not found

Has anyone else found this?
As far as I can tell things are otherwise working, I've downloaded a few test files and I'm about to try a selection file and some other options. Just noting something seems to have changed in synda search in the conda version that's causing something odd to happen in that example.

@oloapinivad
Copy link

Hi all,

I am starting to use this tool and I found extremely powerful, so first of all thanks to the developers!

I just wanted to mention that I installed synda 3.10 via conda and I was facing the same issue as above. I had to manual downgrade the openssl package from 1.1.1e to 1.1.1d in order to overcome it. I suspect that the current conda installation is not working as it is.

@YanchunHe YanchunHe mentioned this issue Apr 1, 2020
Closed
@francocatalano
Copy link

Hi all,
I am experiencing a similar error. If I do:
synda certificate --debug renew
I get:
Error occurs while renewing certificate (exit)
without any additional information about the error.
I did not have problems untill january 2021, then I started getting this issue.
version of synda: 3.14
version of conda: 4.7.12
Any idea about what's wrong?
Thanks a lot in advance for your help!

@francocatalano
Copy link

I have tried downgrading openssl and still same problem with certificate.
Then I tried removing synda and conda and doing a clean conda/synda reinstall. After that I have:
conda version 4.8.3
synda version 3.20
but when I launch synda I got the following error:
synda.source.process.subcommand.exceptions.InvalidRequest: Not found Invalid request

Then, I tried to downgrade synda to version 3.14 and got again the certificate error:
Error occurs while renewing certificate (exit)

Is there anyone currently able to use synda?
If yes, could you please share your configuration?
Any information would be very appreciated. Thanks!

@painter1
Copy link
Contributor

painter1 commented Mar 1, 2021 via email

@painter1
Copy link
Contributor

painter1 commented Mar 2, 2021

Actually I had submitted this patch last year, pull request #145. Nobody paid attention, and since then the master branch was changed so as to make it incompatible. I will submit a new pull request with the same patch, but compatible with the present master branch. I will not revise it again.

@painter1
Copy link
Contributor

painter1 commented Mar 3, 2021

A new pull request, $164, does the same thing as #145. To enable it, edit the configuration file:
add a line "continue_on_cert_errors=true" in the download section. Then, if Synda tries to renew a certificate while starting transfers, and if the renewal fails, Synda will continue to run. If the certificate hadn't expired yet (which is likely), Synda will transfer data exactly as if the renewal had succeeded.

I run with this feature turned on, 24 hours a day with about 4-8 data nodes simultaneously, 6 files per node. Although it's a tiny and simple change, it has a tremendous effect on reliability.

@francocatalano
Copy link

Thanks Jeffrey.
How do I get your fix in synda version 3.14?
Unfortunately, as I wrote, it seems the new version 3.20 is not working on my server.

@painter1
Copy link
Contributor

painter1 commented Mar 3, 2021

The clean way to get it into your version is with a git merge from the continue_on_cert_errors branch to your git-controlled copy of Synda. If you have a separate working copy as I do, you will have to copy the changed files to there. That's four files, the four files containing the string "continue_on_cert_errors".

A dirtier (slightly more dangerous from the long-term maintenance perspective) way is to directly edit these four files in the working copy, in order to match.
In my case and probably yours, the working copy of Synda is in /usr/share/python/synda/, in various subdirectories.
Three of the four changed files are about getting user settings from sdt.conf and are named contants.py, models.py, and models.py (again!) IIRC. The one which directly affects the logic of what Synda does is sddmdefault.py, typically in /usr/share/python/synda/synda/sdt/.

The super-dirty way to incorporate this change is to bypass the user settings in sdt.conf and instead directly edit /usr/share/python/synda/synda/sdt/sddmdefault.py. You can make it match the sddmdefault.py in the continue_on_cert_errors branch, but replace "preferences.is_download_continue_on_cert_errors" with True. I can't really recommend hacks like that, except for code development purposes.

However you do it, you will have to restart the daemon afterwards.

This is quite a short simple patch, and is intended to keep things going through brief outages of the identity (myproxy) server. If it is down for an extended period of time, you really will need to renew some certificates. Then the only solution is to find another server for renewing certificates.

@francocatalano
Copy link

Thanks again Jeffrey, I really appreciate your help!
Since, because of this issue, I have not been able to download files with synda in the last month (at least) I fear that I will need to renew my certificate in any case. How do I check which identity (myproxy) server I am using and how can I switch to a different one?

@painter1
Copy link
Contributor

painter1 commented Mar 3, 2021

I believe that the server name is extracted from your OpenID which is set in credentials.conf.

@francocatalano
Copy link

Thanks Jeffrey.
Then I believe the problem is not related to my identity server (esgf-node.llnl.gov) because I am able to download CMIP files using esgf-generated wget scripts without any certificate issues. Therefore, I think the problem must be related to synda.
Indeed, I am not even able to start synda daemon. If I do:
synda daemon start
I see the following error in transfer log:
2021-03-04 08:01:18,320 INFO SDMYPROX-002 Renew certificate..
2021-03-04 08:01:18,489 ERROR SYDLOGON-012 Error occured while retrieving certificate from myproxy server (exit)
As I wrote, I have tried with fresh reinstall of different synda versions (3.12, 3.13, 3.14) and got the same error while with latest v3.20 I got errors even when initialising synda environment.
It's a pity because synda is a very useful software and getting all the data we need in our lab for CMIP6 analysis without synda would be a nightmare.
Thanks again for your effort.

@painter1
Copy link
Contributor

painter1 commented Mar 4, 2021

You are right. I also use esgf-node.llnl.gov. It only has an occasional brief failure, possibly due to overloading. My little patch deals with this successfully but couldn't possibly help your situation.
Can you run myproxy-logon manually, using the OpenID and password in your credentials.conf?

@francocatalano
Copy link

Unfortunately, myproxy-logon is not installed on our server. When I use esgf-generated wget scripts, certificate is obtained with java (getcert.jar) and stored in my $HOME/.esg/credentials.pem. The OpenID and password I pass to the wget script are exactly the same as those specified in synda credentials.conf.

This is the output:
Retrieving Federation Certificates...--2021-03-05 13:44:25-- https://github.com/ESGF/esgf-dist/raw/master/installer/certs/esg-truststore.ts
Risoluzione di github.com (github.com)... 140.82.121.4
Connessione a github.com (github.com)|140.82.121.4|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 302 Found
Posizione: https://raw.githubusercontent.com/ESGF/esgf-dist/master/installer/certs/esg-truststore.ts [segue]
--2021-03-05 13:44:26-- https://raw.githubusercontent.com/ESGF/esgf-dist/master/installer/certs/esg-truststore.ts
Risoluzione di raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.110.133, ...
Connessione a raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connesso.
Richiesta HTTP inviata, in attesa di risposta... 200 OK

Then it asks for my openid and password
Retrieving Credentials...mar 05, 2021 1:45:32 PM esg.security.myproxy.CredentialConnection getCredential
INFO: done!
done!

and the download starts.
I don't know why synda is failing to get certificate.
Thanks again for your help.

@plesager
Copy link

plesager commented Aug 4, 2021

@francocatalano were you able to create a certificate finally? If you did, which version of synda are you using?

@francocatalano
Copy link

@plesager Hi Philippe. After many attempts, I managed to get it working with synda v3.32.
So far, it seems to work...
good luck!

@plesager
Copy link

plesager commented Aug 6, 2021

Thanks Franco. I was trying my openID created by CEDA (it adds an extra username between enduser and ESGF) which gave the same certificate problem as you had. I've created another openID on esgf-node.llnl.gov and that fixes it (synda 3.35 installed through conda).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@painter1 @ericnienhouse @senesis @plesager @AtefBN @oloapinivad @hot007 @francocatalano