|
| 1 | +# Sky |
| 2 | + |
| 3 | + |
| 4 | + |
| 5 | +Sky is a tool to run any workload seamlessly across different cloud providers through a unified interface. No knowledge of cloud offerings is required or expected – you simply define the workload and its resource requirements, and Sky will automatically execute it on AWS, Google Cloud Platform or Microsoft Azure. |
| 6 | + |
| 7 | +<!-- TODO: We need a logo here --> |
| 8 | +## A Quick Example |
| 9 | +The following command can automatically spin up a cluster on the cheapest available cloud fulfilled the required resources, setup and run the commands in the `hello_sky.yaml` |
| 10 | +```bash |
| 11 | +sky launch -c mycluster hello_sky.yaml |
| 12 | +``` |
| 13 | + |
| 14 | +```yaml |
| 15 | +# hello_sky.yaml |
| 16 | +resources: |
| 17 | + accelerators: |
| 18 | + K80:4 |
| 19 | + |
| 20 | +setup: | |
| 21 | + # Typical use: pip install -r requirements.txt |
| 22 | + echo "running setup" |
| 23 | +
|
| 24 | +run: | |
| 25 | + # Typical use: make use of resources, such as running training. |
| 26 | + echo "hello sky!" |
| 27 | + conda env list |
| 28 | +``` |
| 29 | +
|
| 30 | +## Getting Started |
| 31 | +Please refer to our [documentation](https://sky-proj-sky.readthedocs-hosted.com/en/latest/). |
| 32 | +- [Installation](https://sky-proj-sky.readthedocs-hosted.com/en/latest/getting-started/installation.html) |
| 33 | +- [Quickstart](https://sky-proj-sky.readthedocs-hosted.com/en/latest/getting-started/quickstart.html) |
| 34 | +- [Sky CLI](https://sky-proj-sky.readthedocs-hosted.com/en/latest/reference/cli.html) |
| 35 | +
|
| 36 | +### Installation |
| 37 | +
|
| 38 | +```bash |
| 39 | +# Clone the sky codebase |
| 40 | +git clone ssh://[email protected]/sky-proj/sky.git |
| 41 | +cd sky |
| 42 | +# Sky requires python >= 3.6. |
| 43 | +pip install ".[all]" |
| 44 | +``` |
| 45 | + |
| 46 | +If you only want the dependencies for certain clouds, you can also use |
| 47 | +`".[aws,azure,gcp]"`. |
| 48 | + |
| 49 | +### Cloud Account Setup |
| 50 | + |
| 51 | +Sky currently supports three major cloud providers: AWS, GCP, and Azure. To run |
| 52 | +tasks in the clouds, configure access to at least one cloud: |
| 53 | + |
| 54 | +**AWS**: |
| 55 | + |
| 56 | +```bash |
| 57 | +# Install boto |
| 58 | +pip install boto3 |
| 59 | + |
| 60 | +# Configure your AWS credentials |
| 61 | +aws configure |
| 62 | +``` |
| 63 | + |
| 64 | +To get the **AWS Access Key** required by the `aws configure`, please refer to the [AWS manual](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). The **Default region name [None]:** and **Default output format [None]:** are optional. |
| 65 | + |
| 66 | +**GCP**: |
| 67 | + |
| 68 | +```bash |
| 69 | +pip install google-api-python-client |
| 70 | +# Install `gcloud`; see https://cloud.google.com/sdk/docs/quickstart |
| 71 | +conda install -c conda-forge google-cloud-sdk |
| 72 | + |
| 73 | +# Init. |
| 74 | +gcloud init |
| 75 | + |
| 76 | +# Run this if you don't have a credentials file. |
| 77 | +# This will generate ~/.config/gcloud/application_default_credentials.json. |
| 78 | +gcloud auth application-default login |
| 79 | +``` |
| 80 | + |
| 81 | +**Azure**: |
| 82 | + |
| 83 | +```bash |
| 84 | +# Install the Azure CLI |
| 85 | +pip install azure-cli==2.30.0 |
| 86 | +# Login azure |
| 87 | +az login |
| 88 | +# Set the subscription to use |
| 89 | +az account set -s <subscription_id> |
| 90 | +``` |
| 91 | + |
| 92 | +**Verifying cloud setup** |
| 93 | + |
| 94 | +Sky allows you to verify that cloud credentials are correctly configured using |
| 95 | +the CLI: |
| 96 | + |
| 97 | +```bash |
| 98 | +# Verify cloud account setup |
| 99 | +sky check |
| 100 | +``` |
| 101 | + |
| 102 | +This will produce output verifying the correct setup of each supported cloud. |
| 103 | + |
| 104 | +``` |
| 105 | +Checking credentials to enable clouds for Sky. |
| 106 | + AWS: enabled |
| 107 | + GCP: enabled |
| 108 | + Azure: enabled |
| 109 | +
|
| 110 | +Sky will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check. |
| 111 | +``` |
| 112 | + |
| 113 | +## Developer Guide |
| 114 | +### Setup |
| 115 | + |
| 116 | +```bash |
| 117 | +# Sky requires python version >= 3.6 |
| 118 | + |
| 119 | +# You can just install the dependencies for |
| 120 | +# certain clouds, e.g., ".[aws,azure,gcp]" |
| 121 | +pip install -e ".[all]" |
| 122 | +``` |
| 123 | + |
| 124 | +<!-- TODO (gautam): Removed since we have reversed it --> |
| 125 | +<!-- ## SSH Access |
| 126 | +The system currently supports SSH access for launched VMs by modifying your local `~/.ssh/config`. For git credentials to forward seamlessly, users must start their SSH agent and add their GitHub SSH key to it: |
| 127 | +``` |
| 128 | +eval "$(ssh-agent -s)" |
| 129 | +ssh-add -K /path/to/key # e.g. ~/.ssh/id_ed25519 |
| 130 | +``` |
| 131 | +For more information on GitHub authentication and keys, see their [setup tutorial](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). --> |
| 132 | + |
| 133 | +### Some general engineering practice suggestions |
| 134 | + |
| 135 | +These are suggestions, not strict rules to follow. For general coding style, follow [google style guide](https://google.github.io/styleguide/pyguide.html). |
| 136 | + |
| 137 | +* Use `TODO(author_name)`/`FIXME(author_name)` instead of blank `TODO/FIXME`. This is critical for tracking down issues. You can write TODOs with your name and assign it to others (on github) if it is someone else's issue. |
| 138 | +* Delete your branch after merging it. This keeps the repo clean and faster to sync. |
| 139 | +* Use an exception if this is an error. Only use `assert` for debugging or proof-checking purpose. This is because exception messages usually contain more information. |
| 140 | +* Use modern python features and styles that increases code quality. |
| 141 | + * Use f-string instead of `.format()` for short expressions to increase readability. |
| 142 | + * Use `class MyClass:` instead of `class MyClass(object):`. The later one was a workaround for python2.x. |
| 143 | + * Use `abc` module for abstract classes to ensure all abstract methods are implemented. |
| 144 | + * Use python typing. But you should not import external objects just for typing. Instead, import typing-only external objects under `if typing.TYPE_CHECKING:`. |
0 commit comments