Skip to content

SysAdmin

Jump to bottom Edit New page
Thomas Sibley edited this page Jan 11, 2014 · 59 revisions

SysAdmin FAQ

Manual maintenance issues

How to reindex a missing module?

cd api.metacpan.org
bin/metacpan release http://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/MetaCPAN-API-0.33.tar.gz --latest

How to index all unindexed dists uploaded over the last 24 hours?

bin/metacpan release --skip --age 24 --latest ~/CPAN/authors/id/

How to index the latest Perl release

bin/metacpan release http://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.16.0.tar.bz2
bin/metacpan release --status latest http://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.16.1.tar.bz2

The above syntax will force the status bit to "latest", which we need to do manually only for new, latest Perl releases. Make sure you reindex the predecessor as well.

Restarting services

The following services are set up:

metacpan-www
metacpan-api
metacpan-rrr
metacpan-watcher
elasticsearch

Each of those services can be restarted by calling service $name restart (as superuser).

How to deploy a new version of MetaCPAN-API and MetaCPAN-Web?

Log in as the metacpan user (which loads perlbrew automatically), go to the appropriate folder (~/metacpan.org or ~/api.metacpan.org), pull from github and restart the service as root (rcmetacpan-ww restart or rcmetacpan-api restart).

For now prereqs can be installed manually with sudo /home/metacpan/bin/install_modules Foo::Bar.

How to increase storage space for ElasticSearch and the CPAN mirror?

The CPAN mirror and the ElasticSearch data are stored in /var/cpan and /var/elasticsearch, respectively. Those are filesystems on top of the LVM LVs /dev/mapper/vg0-cpan and /dev/mapper/vg0-elasticsearch.

To increase the space available on one of them, change the following example, which adds an additional 100 MB for the CPAN mirror. There's no need to unmount anything.

# Show current usage (and what is free, see note below)
pvscan
# Grow the LVM volume
lvextend -L +100M /dev/mapper/vg0-cpan
# Extend the filesystem to the fit the new LV size
resize2fs /dev/mapper/vg0-cpan

Do NOT allocate all the unused space to logical volumes. We need some free space to use by LVM snapshots during the backup process. We haven't actually checked how much spare space we need for that, so let's play it safe and say that at least 1.5 GiB should be left alone.

Need to clean out /var/tmp/metacpan/ ?

find /var/tmp/metacpan/source/ -maxdepth 2 -type d -mtime +215 | head -5000 | xargs sudo rm -rf

This doesn't solve it as such - but cleans up files that haven't been modified/extracted in a long while

Network/high level issues

Managing DNS

We are currently sponsored by Dyn with a DynECT Managed DNS Lite account.

URL: https://manage.dynect.net

When delegating your domain names, please use the following nameservers:

ns1.p24.dynect.net

ns2.p24.dynect.net

ns3.p24.dynect.net

ns4.p24.dynect.net

The best place to get started is look at the DynECT Managed DNS Lite User Manual located at: https://manage.dynect.net/help

Our contact at Dyn is Chris Gonyea [email protected]. Also, alh in #metacpan works for Dyn and can help with technical issues.

ByteMark mirror

Specs:

What is the procedure if the server is unreachable?

ByteMark

We should be able to fix most stuff because we have console access (see above)

Booking.com

Contact the booking staff either by email ([email protected]) or in emergencies by phone (+31207153409). Most problems are better solved on IRC. Our contact on irc.perl.org is Seveas.

Where are log files stored?

ElasticSearch logs can be found in /opt/elasticsearch-0.20.2/logs

Where are system monitoring reports stored?

The ElasticSearch status can be queried from within the box:

$ curl localhost:9200/cpan/_status?pretty

$ curl localhost:9200/_cluster/health/cpan_v1?level=shards

http://munin.bm-n2.metacpan.org/metacpan.org/bm-n2.metacpan.org/ http://nagios.omega.pqpq.de/

web api watcher

Where are backups stored

SSL certificates

Certificates are currently minted by StartSSL using their free Class 1 level process. The Class 1 level certs are good for one year, cover a top-level domain as well as a subdomain, and only require minimal personal identity information (name, email, physical address, phone). Best of all, they're free. The only caveat is that you cannot mint a cert if an existing cert exists for the same DNS name and the existing cert's expiry is more than two weeks out. To do so would first require a 25$ revocation fee for the original cert before being able to mint a new one.

Where are certs stored?

Certificates are stored in /etc/puppet/private/bm-n2/ssl/<dns-name>. Each DNS name directory contains:

  • server.key - Copy of the 2048-bit RSA private key
  • server.csr - Certificate Signing Request sent to StartSSL
  • server.pub - Public cert provided by StartSSL
  • server.crt - Combined public cert + intermediate cert + StartSSL CA root

The combined server.crt file is generated using the /etc/puppet/private/bm-n2/ssl/chain-to-startssl script and files in /etc/puppet/private/bm-n2/ssl/startssl-ca.

The server.key for api.metacpan.org, cpan.metacpan.org, and metacpan.org is a copy of 2014-01-09.key. Sharing a private key makes renewing certs easier. Note that currently www.metacpan.org has a different key.

The containing directory, ssl/, is a local git repository. This aids in not losing our keys or certs, which may be hard or impossible to replace easily if overwritten during a botched update.

Current certificates

Only four vhosts currently use SSL. You can check which vhosts expect SSL with a grep like:

cd /etc/puppet
git grep -P '(nginx::vhost|ssl)' modules/metacpan/manifests/web

api.metacpan.org

Good for api.metacpan.org and metacpan.org. Only used by api. Minted by trs.

cpan.metacpan.org

Good for cpan.metacpan.org and metacpan.org. Used by both DNS names. Minted by trs.

metacpan.org

Copy of cpan.metacpan.org files. See above.

www.metacpan.org

Good for www.metacpan.org and metacpan.org. Only used by www. before redirection. Minted by Olaf.

Renewing certificates

Renewing the certs from StartSSL requires:

  1. Re-validate control over metacpan.org via their process. Email to hostmaster@ or [email protected] will both go to [email protected] where you can see it.

  2. Skip their private key generation step and supply a CSR directly. You can and should reuse the existing CSRs to avoid problems. If you regenerate a CSR, make sure it matches the key! Compare the output of:

    openssl rsa -noout -modulus < server.key | sha1sum openssl req -noout -modulus < new.csr | sha1sum

  3. Save the new public cert to <dns-name>/server.pub

  4. Run chain-to-startssl <dns-name> to generate a server.crt with the appropriate certificate chain.

  5. Install new certs and restart services by running puppet: /etc/puppet/run.sh

  6. Manually verify that everything worked!

  7. Commit your changes to the local git repo:

    cd /etc/puppet/private/bm-n2/ssl sudo git add -A sudo git commit --author='Your Name [email protected]'

Add a custom footer

Meta

Clone this wiki locally