Skip to content

Commit 7da4de0

Browse files
aidanheerdegenaidanheerdegen
committed
Added conda packaging. Required rearrangement of directory structure.
Changed testing to directly call new main() routine. Now using console_scripts in setup.py (via setup.cfg and pbr) to make mdssdiff entry point. Requires explicit injection of sys.argv via call to main_argv()
1 parent 0c136e8 commit 7da4de0

22 files changed

+937
-115
lines changed

‎.circleci/config.yml

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
version: 2.0
2+
jobs:
3+
py27:
4+
working_directory: /circleci
5+
docker:
6+
- image: scottwales/conda-build
7+
environment:
8+
PYTHON_VER: 2.7
9+
10+
steps:
11+
- checkout
12+
13+
- run: |
14+
conda build conda --python=${PYTHON_VER}
15+
16+
- run: |
17+
mkdir /artefacts
18+
cp $(conda build conda --python=${PYTHON_VER} --output) /artefacts
19+
20+
- persist_to_workspace:
21+
root: /artefacts
22+
paths: '*'
23+
24+
py36:
25+
working_directory: /circleci
26+
docker:
27+
- image: scottwales/conda-build
28+
environment:
29+
PYTHON_VER: 3.6
30+
31+
steps:
32+
- checkout
33+
34+
- run: |
35+
conda build conda --python=${PYTHON_VER}
36+
37+
- run: |
38+
mkdir /artefacts
39+
cp $(conda build conda --python=${PYTHON_VER} --output) /artefacts
40+
41+
- persist_to_workspace:
42+
root: /artefacts
43+
paths: '*'
44+
45+
publish:
46+
working_directory: /circleci
47+
docker:
48+
- image: scottwales/conda-build
49+
steps:
50+
- attach_workspace:
51+
at: /artefacts
52+
53+
- run:
54+
anaconda --token "${ANACONDA_TOKEN}" upload --user "${ANACONDA_USER}" /artefacts/*.tar.bz2
55+
56+
workflows:
57+
version: 2
58+
build_and_publsh:
59+
jobs:
60+
- py27:
61+
filters:
62+
tags:
63+
only: /.*/
64+
65+
- py36:
66+
filters:
67+
tags:
68+
only: /.*/
69+
70+
- publish:
71+
requires:
72+
- py27
73+
- py36
74+
filters:
75+
tags:
76+
only: /.*/
77+
branches:
78+
ignore: /.*/

‎.codeclimate.yml

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
exclude_paths:
2+
- docs/_build/**/*

‎.gitignore

+8
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,11 @@ docs/_build/
5555

5656
# PyBuilder
5757
target/
58+
59+
# Editor files
60+
*.swp
61+
62+
# Databases
63+
*.db
64+
65+
/README.cookiecutter.md

‎.landscape.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
python-targets:
2+
- 2
3+
- 3
4+
ignore-paths:
5+
- docs/_build

‎.travis.yml

+18-7
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,27 @@
11
language: python
22
python:
33
- '2.7'
4-
before_install:
5-
- pip install codecov
4+
- '3.5'
65
install:
7-
- pip install --upgrade pytest
8-
env:
9-
global:
10-
- PYTHONPATH=$PYTHONPATH:..:.
6+
- if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
7+
wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh;
8+
else
9+
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
10+
fi
11+
- bash miniconda.sh -b -p $HOME/miniconda
12+
- export PATH="$HOME/miniconda/bin:$PATH"
13+
14+
- conda config --set always_yes yes --set changeps1 no
15+
- conda create -n travis python=$TRAVIS_PYTHON_VERSION
16+
- conda env update -n travis -f conda/dev-environment.yml
17+
- if [ -f conda/dev-environment-py${TRAVIS_PYTHON_VERSION}.yml ]; then conda env update -n travis -f conda/dev-environment-py${TRAVIS_PYTHON_VERSION}.yml; fi
18+
- source activate travis
19+
20+
- pip install --upgrade pytest coverage codecov codeclimate-test-reporter
21+
- pip install .
1122
script:
1223
- py.test test/test_mdsspath.py
1324
after_success:
1425
- coverage run --source=mdssdiff.py,mdsspath.py,test/test_mdsspath.py -m py.test
1526
- codecov
16-
27+
- codeclimate-test-reporter

‎README.md

-92
This file was deleted.

‎README.rst

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
=============================
2+
mdssdiff
3+
=============================
4+
5+
Report difference between directory tree on a local filesystem and on a remote mass data store. Some rudimentary synching supported.
6+
7+
.. |Build Status| image:: https://travis-ci.org/coecms/mdssdiff.svg?branch=master
8+
:target: https://travis-ci.org/coecms/mdssdiff
9+
.. |codecov.io| image:: https://codecov.io/github/coecms/mdssdiff/coverage.svg?branch=master
10+
:target: https://codecov.io/github/coecms/mdssdiff?branch=master
11+
.. |Code Health| image:: https://landscape.io/github/coecms/mdssdiff/master/landscape.svg?style=flat
12+
:target: https://landscape.io/github/coecms/mdssdiff/master
13+
.. |Conda Status| image:: https://circleci.com/gh/coecms/mdssdiff.svg?style=shield
14+
:target: https://circleci.com/gh/coecms/mdssdiff
15+
16+
Look for differences between local filesystem and its copy on a mass
17+
data store system
18+
19+
The latest version uses the `conda environment
20+
https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda`:
21+
22+
::
23+
24+
module use /g/data3/hh5/public/modules
25+
module use /g/data3/hh5/public/modules
26+
module load python/analysis27
27+
28+
To use the older standalone python version:
29+
30+
::
31+
32+
module purge
33+
module use ~access/modules
34+
module load mdssdiff
35+
36+
Basic usage message:
37+
38+
::
39+
40+
mdssdiff -h
41+
usage: mdssdiff [-h] [-v] [-P PROJECT] [-p PATHPREFIX] [-r] [-cr | -cl] [-f]
42+
inputs [inputs ...]
43+
44+
Compare local directories and those on mdss. Report differences
45+
46+
positional arguments:
47+
inputs directories (-r must be specified to recursively descend
48+
into sub-directories)
49+
50+
optional arguments:
51+
-h, --help show this help message and exit
52+
-v, --verbose Increase verbosity
53+
-P PROJECT, --project PROJECT
54+
Project code for mdss (default to $PROJECT)
55+
-p PATHPREFIX, --pathprefix PATHPREFIX
56+
Prefix for mdss path
57+
-r, --recursive Recursively descend directories (default False)
58+
-cr, --copyremote Copy over files that are missing on remote (False)
59+
-cl, --copylocal Copy over files that are missing on local (False)
60+
-f, --force Force copying of different sized files, following --cr
61+
or --cl (False)
62+
63+
For example, say you have a personal directory on mdss:
64+
65+
::
66+
67+
mdss ls -ld personal/me
68+
drwxrws--- 4 abc123 a00 92 Dec 14 2014 personal/me
69+
70+
And you have used mdss to put another directory there
71+
72+
::
73+
74+
mdss put -r data personal/me/
75+
mdss ls -ld personal/me/data
76+
-rw-r--r-- 1 abc123 a00 1219 Nov 9 12:40 personal/me/data
77+
78+
To check if all the files have been correctly copied:
79+
80+
::
81+
82+
mdssdiff -p personal/me data
83+
84+
This will show a list of which files are present/absent on the local or
85+
remote (mdss) filesystem. It will also notify show any files which
86+
differ in size.
87+
88+
To recursively descend directories to check for differences use the
89+
``-r`` flag
90+
91+
::
92+
93+
mdssdiff -p personal/me -r data
94+
95+
This will also work the other way, and tell you if there are files on
96+
the remote system that are not present locally.
97+
98+
If there are files in your local directory that are not on the mdss, say
99+
you made some new ones or your last mdss copy command failed to complete
100+
cleanly, and you wish to copy them to mdss you can use the
101+
``--copyremote/-cr`` flag:
102+
103+
::
104+
105+
mdssdiff -p personal/me -r -cr data
106+
107+
Equally, if you have deleted some files in your local directory that you
108+
wish to copy back from mdss you can use the ``--copylocal/-cl`` flag:
109+
110+
::
111+
112+
mdssdiff -p personal/me -r -cl data
113+
114+
If there are files of unequal size you must specify ``-f/--force`` to
115+
force copying the files, and in this case the decision of which
116+
way the copy will go (from or to mdss) depends on specifying either of
117+
the ``-cr`` or ``-cl`` options. e.g. to copy files of different size from the
118+
local directory to mdss
119+
120+
::
121+
122+
mdssdiff -p personal/me -r -cr -f data

‎conda/dev-environment.yml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
name: mdssdiff-dev
2+
3+
channels:
4+
- conda-forge
5+
- defaults
6+
7+
dependencies:
8+
- python
9+
- pbr

‎conda/meta.yaml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package:
2+
name: mdssdiff
3+
version: {{ GIT_DESCRIBE_TAG }}
4+
5+
build:
6+
number: {{ GIT_DESCRIBE_NUMBER }}
7+
string: {{ GIT_BUILD_STR }}
8+
9+
source:
10+
git_url: ../
11+
12+
build:
13+
script: python setup.py install
14+
15+
requirements:
16+
build:
17+
- python
18+
- pbr
19+
run:
20+
- python
21+
- six
22+
23+
test:
24+
source_files:
25+
- setup.cfg
26+
- conftest.py
27+
- test
28+
requires:
29+
- pytest
30+
commands:
31+
- py.test
32+
33+
about:
34+
home: https://github.com/coecms/mdssdiff
35+
license: Apache 2.0
36+
license_file: LICENSE-2.0.txt
37+
summary: Report difference between directory tree on a local filesystem and on a remote mass data store. Some rudimentary synching supported.

0 commit comments

Comments
 (0)