Merge pull request #138 from github/v2

Release licensed v2

.gitignore

@@ -24,6 +24,7 @@ test/fixtures/cabal/*
 test/fixtures/git_submodule/*
 !test/fixtures/git_submodule/README
 test/fixtures/pip/venv
+!test/fixtures/migrations/**/*
 
 vendor/licenses
 .licenses

CHANGELOG.md

@@ -6,6 +6,27 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 2.0.0 - 2019-02-09
+
+**This is a major release and includes breaking changes to the configuration and cached record file formats**
+
+### Added
+- New `migrate` command to automatically update configuration and cached record file formats
+- New extensible reporting infrastructure
+- New base command and source classes to abstract away implementation details
+
+### Changes
+- Cached dependency metadata files are now stored entirely as YAML, with `.dep.yml` extension
+- The Bundler dependency source is now identified in configuration files and output as `bundler` instead of `rubygem`
+- Refactored sources for better consistency between classes
+- Refactored commands for better consistency between classes
+- Command outputs have changed for better consistency
+- Updated Dependency classes for better integration with `licensee`
+
+### Fixed
+- Licensed no longer exits on errors when evaluating dependency sources or finding dependencies
+- The Bundler dependency source correctly finds the `bundler` gem as a dependency in more cases
+
 ## 1.5.2 - 2018-12-27
 
 ### Changes
@@ -112,4 +133,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/1.5.2...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/2.0.0...HEAD

README.md

@@ -12,6 +12,13 @@ Licensed is **not** a complete open source license compliance solution. Please u
 
 Licensed is in active development and currently used at GitHub.  See the [open issues](https://github.com/github/licensed/issues) for a list of potential work.
 
+## Licensed v2
+
+Licensed v2 includes many internal changes intended to make licensed more extensible and easier to update in the future.  While not too much has changed externally, v2 is incompatible with configuration files and cached records from previous versions.  Fortunately, migrating is easy using the `licensed migrate` command.
+
+See [CHANGELOG.md](./CHANGELOG.md) for more details on whats changed.
+See the [migration documentation](./docs/migrating_to_newer_versions.md) for more info on migrating to v2, or run `licensed help migrate`.
+
 ## Installation
 
 ### With a Gemfile
@@ -50,32 +57,12 @@ For example, on macOS with Homebrew: `brew install cmake pkg-config` and on Ubun
 ## Usage
 
 - `licensed list`: Output enumerated dependencies only.
-
 - `licensed cache`: Cache licenses and metadata.
-
 - `licensed status`: Check status of dependencies' cached licenses. For example:
-
-```
-$ bundle exec licensed status
-Checking licenses for 3 dependencies
-
-Warnings:
-
-.licenses/rubygem/bundler.txt:
-  - license needs review: mit.
-
-.licenses/rubygem/licensee.txt:
-  - cached license data missing
-
-.licenses/bower/jquery.txt:
-  - license needs review: mit.
-  - cached license data out of date
-
-3 dependencies checked, 3 warnings found.
-```
-
 - `licensed version`: Show current installed version of Licensed. Aliases: `-v|--version`
 
+See the [commands documentation](./docs/commands.md) for additional documentation, or run `licensed -h` to see all of the current available commands.
+
 ### Configuration
 
 All commands, except `version`, accept a `-c|--config` option to specify a path to a configuration file or directory.
@@ -93,7 +80,7 @@ See the [configuration file documentation](./docs/configuration.md) for more det
 
 Dependencies will be automatically detected for all of the following sources by default.
 1. [Bower (bower)](./docs/sources/bower.md)
-2. [Bundler (rubygem)](./docs/sources/bundler.md)
+2. [Bundler](./docs/sources/bundler.md)
 3. [Cabal (cabal)](./docs/sources/cabal.md)
 4. [Go (go)](./docs/sources/go.md)
 5. [Go Dep (dep)](./docs/sources/dep.md)
@@ -106,7 +93,7 @@ You can disable any of them in the configuration file:
 
 ```yml
 sources:
-  rubygem: false
+  bundler: false
   npm: false
   bower: false
   cabal: false
@@ -138,6 +125,12 @@ if Licensed::Shell.tool_available?('bundle')
 end
 ```
 
+See the [documentation on adding new sources](./docs/adding_a_new_source.md) for more information.
+
+#### Adding Commands
+
+See the [documentation on commands](./docs/commands.md) for information about adding a new CLI command.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/github/licensed. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org/) code of conduct.  See [CONTRIBUTING](CONTRIBUTING.md) for more details.

Rakefile

@@ -27,7 +27,7 @@ task :setup, [:arguments] do |task, args|
   end
 end
 
-sources_search = File.expand_path("lib/licensed/source/*.rb", __dir__)
+sources_search = File.expand_path("lib/licensed/sources/*.rb", __dir__)
 sources = Dir[sources_search].map { |f| File.basename(f, ".*") }
 
 namespace :test do
@@ -47,7 +47,7 @@ namespace :test do
 
       # use negative lookahead to exclude all source tests except
       # the tests for `source`
-      t.test_files = FileList["test/**/*_test.rb"].exclude(/test\/source\/(?!#{source}).*?_test.rb/,
+      t.test_files = FileList["test/**/*_test.rb"].exclude(/test\/sources\/(?!#{source}).*?_test.rb/,
                                                            "test/fixtures/**/*_test.rb")
     end
   end

docs/adding_a_new_source.md

@@ -0,0 +1,93 @@
+# Adding a new source dependency enumerator
+
+## Implement new `Source` class
+
+Dependency enumerators inherit and override the [`Licensed::Sources::Source`](../lib/licensed/sources/source.rb) class.
+
+#### Required method overrides
+1. `Licensed::Sources::Source#enabled?`
+   - Returns whether dependencies can be enumerated in the current environment.
+2. `Licensed::Sources::Source#enumerate_dependencies`
+   - Returns an enumeration of `Licensed::Dependency` objects found which map to the dependencies of the current project.
+
+#### Optional method overrides
+1. `Licensed::Sources::Source.type`
+   - Returns the name of the current dependency enumerator as it is found in a licensed configuration file.
+
+## Determining if dependencies should be enumerated
+
+This section covers the `Licensed::Sources::Source#enabled?` method.  This method should return a truthy/falsey value indicating
+whether `Licensed::Source::Sources#enumerate_dependencies` should be called on the current dependency source object.
+
+Determining whether dependencies should be enumerated depends on whether all the tools or files needed to find dependencies are present.
+For example, to enumerate `npm` dependencies the `npm` CLI tool must be found with `Licensed::Shell.tool_available?` and a `package.json` file needs to exist in the licensed app's configured [`source_path`](./configuration.md#configuration-paths).
+
+#### Gating functionality when required tools are not available.
+
+When adding new dependency sources, ensure that `script/bootstrap` scripting and tests are only run if the required tooling is available on the development machine.
+
+* See `script/bootstrap` for examples of gating scripting based on whether tooling executables are found.
+* Use `Licensed::Shell.tool_available?` when writing test files to gate running a test suite when tooling executables aren't available.
+```ruby
+if Licensed::Shell.tool_available?('bundle')
+  describe Licensed::Source::Bundler do
+    ...
+  end
+end
+```
+
+## Enumerating dependencies
+
+This section covers the `Licensed::Sources::Source#enumerate_dependencies` method.  This method should return an enumeration of
+`Licensed::Dependency` objects.
+
+Enumerating dependencies will require some knowledge of the package manager, language or framework that manages the dependencies.
+
+Relying on external tools always has a risk that the tool could change.  It's generally preferred to not rely on package manager files
+or other implementation details as these could change over time.  CLI tools that provides the necessary information are generally preferred
+as they will more likely have requirements for backwards compatibility.
+
+#### Creating dependency objects
+
+Creating a new `Licensed::Dependency` object requires name, version, and path arguments.  Dependency objects optionally accept a path to use as search root when finding licenses along with any other metadata that is useful to identify the dependency.
+
+##### `Licensed::Dependency` arguments
+
+1. name (required)
+   - The name of the dependency. Together with the version, this should uniquely identify the dependency.
+2. version (required)
+   - The current version of the dependency, used to determine when a dependency has changed. Together with the name, this should uniquely identify the dependency.
+3. path (required)
+   - A path used by [`Licensee`](https://github.com/benbalter/licensee) to find dependency license content.  Can be either a folder or a file.
+4. search_root (optional)
+   - The root of the directory hierarchy to search for a license file.
+5. metadata (optional)
+   - Any additional metadata that would be useful in identifying the dependency.
+   - suggested metadata
+      1. summary
+         - A short description of the dependencies purpose.
+      2. homepage
+         - The dependency's homepage.
+6. errors (optional)
+  - Any errors found when loading dependency information.
+
+#### Finding licenses
+
+In some cases, license content will be in a parent directory of the specified location.  For instance, this can happen with Golang packages
+that share a license file, e.g. `github.com/go/pkg/1` and `github.com/go/pkg/2` might share a license at `github.com/go/pkg`.  In this case, create a `Licensed::Dependency` with the optional `search_root` property, which denotes the root of the directory hierarchy that should be searched.  Directories will be examined in order from the given license location to the `search_root` location to prefer license files with more specificity, i.e. `github.com/go/pkg/1` will be searched before `github.com/go/pkg`.
+
+#### Handling errors when enumerating dependencies
+
+External tools have their own error handling which, if left unhandled, can cause dependency enumeration as a whole to fail either for an individual dependency source or for licensed as a whole.  These errors should be gracefully handled to allow for the best possible user experience.
+
+##### Handling errors related to a specific dependency
+
+`Licensed::Dependency#initialize` will already set errors related to `nil` or empty `path:` arguments, as well as paths that don't exist.  Additional errors can be set to a dependency using the `errors:` argument, e.g. `Licensed::Dependency.new(errors: ["error"])`.
+
+When a dependency contains errors, all errors will be reported to the user and `Licensed::Command::Command#evaluate_dependency` will be not be called.
+
+##### Handling errors related to source evaluation
+
+When an error occurs related to a specific source, raise a `Licensed::Sources::Source::Error` with an informative message.  The error will be caught and reported to the user, and further evaluation of the source will be halted.
+
+As an example, this could be useful if a source is enabled but incorrectly configured.

docs/commands.md

@@ -0,0 +1,81 @@
+# Commands
+
+Run `licensed -h` to see help content for running licensed commands.
+
+## `list`
+
+Running the list command finds the dependencies for all sources in all configured applications.  No additional actions are taken on each dependency.
+
+## `cache`
+
+The cache command finds all dependencies and ensures that each dependency has an up-to-date cached record.
+
+Dependency records will be saved if:
+1. The `force` option is set
+2. No cached record is found
+3. The cached record's version is different than the current dependency's version
+   - If the cached record's license text contents matches the current dependency's license text then the `license` metadata from the cached record is retained for the new saved record.
+
+After the cache command is run, any cached records that don't match up to a current application dependency will be deleted.
+
+## `status`
+
+The status command finds all dependencies and checks whether each dependency has a valid cached record.
+
+A dependency will fail the status checks if:
+1. No cached record is found
+2. The cached record's version is different than the current dependency's version
+3. The cached record doesn't contain any license text
+4. The cached record's `license` metadata doesn't match an `allowed` license from the dependency's application configuration.
+
+## `version`
+
+Displays the current licensed version.
+
+# Adding a new command
+
+## Implement new `Command` class
+
+Licensed commands inherit and override the [`Licensed::Sources::Command`](../lib/licensed/commands/command.rb) class.
+
+#### Required method overrides
+1. `Licensed::Commands::Command#evaluate_dependency`
+   - Runs a command execution on an application dependency.
+
+The `evaluate_dependency` method should contain the specific command logic.  This method has access to the application configuration, dependency source enumerator and dependency currently being evaluated as well as a reporting hash to contain information about the command execution.
+
+#### Optional method overrides
+
+The following methods break apart the different levels of command execution.  Each method wraps lower levels of command execution in a corresponding reporter method.
+
+1. `Licensed::Commands::Command#run`
+   - Runs `run_app` for each application configuration found.  Wraps the execution of all applications in `Reporter#report_run`.
+2. `Licensed::Commands::Command#run_app`
+   - Runs `run_source` for each dependency source enumerator enabled for the application configuration.  Wraps the execution of all sources in `Reporter#report_app`.
+3. `Licensed::Commands::Command#run_source`
+   - Runs `run_dependency` for each dependency found in the source.  Wraps the execution of all dependencies in `Reporter#report_source`.
+4. `Licensed::Commands::Command#run_dependency`
+   - Runs `evaluate_dependency` for the dependency.  Wraps the execution of all dependencies in `Reporter#report_dependency`.
+
+As an example, `Licensed::Commands::Command#run_app` calls `Reporter#report_app` to wrap every call to `Licensed::Commands::Command#run_source`.
+
+##### Overriding optional methods
+
+The `run` methods can be overridden to provide additional reporting data or functionality.  Overriding a method should call the original method with a block for the additional logic.
+
+```ruby
+def run_app(app)
+  super do |report|
+    result = yield report
+
+    # do other thing
+    call_additional_functionality(app)
+
+    # add reporting information
+    report["result"] = result
+
+    # return the result
+    result
+  end
+end
+```

docs/configuration.md

@@ -32,7 +32,7 @@ it may still determine that it can't enumerate dependencies for a project.
 ```yml
 sources:
   bower: true
-  rubygem: false
+  bundler: false
 ```
 
 `licensed` determines which sources will try to enumerate dependencies based on the following rules:
@@ -76,7 +76,7 @@ source_path: 'relative/path/to/source'
 # Sources of metadata
 sources:
   bower: true
-  rubygem: false
+  bundler: false
 
 # Dependencies with these licenses are allowed by default.
 allowed:
@@ -88,15 +88,15 @@ allowed:
 
 # These dependencies are explicitly ignored.
 ignored:
-  rubygem:
+  bundler:
     - some-internal-gem
 
   bower:
     - some-internal-package
 
 # These dependencies have been reviewed.
 reviewed:
-  rubygem:
+  bundler:
     - bcrypt-ruby
 
   bower:
@@ -122,22 +122,22 @@ Here are some examples:
 ```yml
 sources:
   go: true
-  rubygem: false
+  bundler: false
 
 ignored:
-  rubygem:
+  bundler:
     - some-internal-gem
 
 reviewed:
-  rubygem:
+  bundler:
     - bcrypt-ruby
 
 cache_path: 'path/to/cache'
 apps:
   - source_path: 'path/to/app1'
   - source_path: 'path/to/app2'
     sources:
-      rubygem: true
+      bundler: true
       go: false
 ```
 

docs/migrating_to_newer_versions.md

@@ -0,0 +1,3 @@
+# Migrating your licensed configuration and cached records to the latest version of licensed
+
+Licensed v2+ ships with an additional executable, `licensed-migrator`, that can be used to update your licensed files to the format expected by the currently installed version.  To run, execute `licensed migrate --from v1 -c <path to licensed configuration file>`, replacing `v1` with the major version of licensed to migrate from.