Skip to content

InstallCharts

Jump to bottom
Jason Fesler edited this page Jun 19, 2016 · 8 revisions

Setting up the charts is probably the least automated part of setting up the mirror. Thankfully, these get touched the least; it will be rare that I ask you to update the chart scripts; or warn that your charting might break.

Updating from a previous version?

Many of you are looking at this in response to a recent email (June 9, 2016).

  • rsync -a [email protected]:stable/extras/. falling-sky-extras
  • Update your crontab
    • Remove falling-sky-chart.pl
    • Replace with falling-sky-dyngraphs-csv.pl
    • Optionally, add either --days 750 or --since 2010/05/10

Mine looks like this:

% crontab -l
# m h  dom mon dow   command
[email protected]
*/15 * * * * rsync -avq "[email protected]:stable/content/."  /var/www/. --delete --exclude site 
0 * * * * /home/jfesler/falling-sky-extras/falling-sky-dyngraphs-csv.pl --config /var/www/site/config.js --since 2010/05/01

First time setup:

Prerequisites

  • perl . I've tested 5.10 myself; nothing is fancy here though.
  • perl modules: JSON, DBI, DBD::mysql, Socket6
  • patience, and perhaps a willingness to improve these docs for others
  • librrdtool with developer headers RRD is no longer used, period.

Downloading the extras package

Files

After extracting the files from the archive, you'll see (at least) these:

  • falling-sky-dyngraphs-csv.pl is the database updater and CSV file generator.
  • falling-sky-chart.sql is the SQL setup for the database
  • falling-sky-chart.a29.pl is the PII removal script (needed just once, see bottom of page)

Obsolete:

  • falling-sky-chart.pl is the charting script.
    • Updates the database
    • Produces RRD files and images
    • Now obsolete: We no longer refer to the output from this.

Database setup

Do this only if you have not set up the database yet.

Database setup is required for both storing end user results, as well as generating graphs.

These instructions assume you're at least a bit familiar with mysql. If not, seek competent help. Or consider skipping this feature of the mirror site entirely.

Install and start mysql per your OS instructions.

Create a mysql database; call it testipv6.

Create a unique username and password in mysql

Grant access to the testipv6 database to this username and password.

mysql -u USERNAME -p testipv6 < falling-sky-chart.sql

  • This is the same create.sql as found in the charts directory.
  • You will be asked for the password.
  • This command will drop any previous tables you had in this database; only run this once.

private.js setup

In your site's private.js you'll need to configure the database host, database name, username, and password. This should be in your site's /site/private.js file. On my server, this is in /usr/local/www/data/virt/beta.test-ipv6.com/site/private.js (as an example).

PrivateConfig =
{
...
  "db": {
    "host": "localhost",
    "db": "testipv6",
    "username": "sampleusername",
    "password": "samplepassword",
  },
...
}

config.js setup

Your config.js will also need updating. Make sure that in the options section, that "survey" is enabled and configured like this:

MirrorConfig =
{
...
   "options": {
    "show_stats": "/stats.html",
    "survey": "/survey.php",
    "comment": "/comment.php",
    "ip": "/ip/",
  },
...

Testing survey posting

Visit your site. Make sure the survey runs. You'll know by the message "(Updated server side IPv6 readiness stats)". If you have errors, fix them, before worrying too much about charts.

CSV File Generation

Testing

 ./falling-sky-dyngraphs-csv.pl --config /path/to/site/config.js

This should produce /path/to/site/graphdata.csv and /path/to/site/graphdata_100.csv

Cron

If you like what you see (warning: You may not have much data yet..), then you can set up a cron job. You can also relocate the falling-sky-chart.pl script to the location of your choice.

Use crontab -e if you're using per-user crontabs; or edit /etc/crontab . The recommended frequency for updating graphs is hourly.

# Example of a per-user crontab
MAILTO: [email protected]
0 * * * * /path/to/falling-sky-dyngraphs-csv.pl  --config /path/to/config.js --days 750

If you're using /etc/crontab it might need to look more like this:

# Example for /etc/crontab entry
MAILTO: [email protected]
0 * * * * username /path/to/falling-sky-dyngraphs-csv.pl  --config /path/to/config.js --days 750

You can specify a start date instead of number of days. The following example will capture from May 1, 2010. The format is important (sorry, non-US folks!). Be sure to use YYYY-MM-DD here.

# Example of a per-user crontab
MAILTO: [email protected]
0 * * * * /path/to/falling-sky-dyngraphs-csv.pl  --config /path/to/config.js --since 2010/05/01

Recording of IP addresses vs European Law

By default, we stopped recording IP addresses in October 2013. This is due to the number of mirrors operating in jurisdictions where there are strict operating requirements when it comes to personally identifiable information. Short of having everyone consult local legal advice, we have just simply stopped collecting the data. We also stopped using cookies (previously used to detect when one visitor visited more than once in 24 hours).

This .. has had an impact on research capabilities.

Removing IP addresses

If you have an older mirror, and you want to remove IP address data from the database (without resetting the stats), do the following to remove all potential PII.

 mysql -h databasehost -u databaseuser -p  databasename
 [password]
 mysql> update survey set ip="a29" where ip != '';
 mysql> update survey set ip6="a29" where ip6 != '';
 mysql> update survey set ip4="a29" where ip4 != ''; 
 mysql> update survey set survey.cookie = rand() where survey.cookie != '';

Easier alternate

If you upgrade your "extras" directory, you can run

  falling-sky-chart.a29.pl --config /path/to/site/config.js

... which will do the same SQL statements documented above.

Clone this wiki locally