Merge branch 'release/3.0.1'

Author: To-om

CHANGELOG.md

@@ -1,6 +1,26 @@
 # Change Log
 
-## [3.0.0](https://github.com/TheHive-Project/Cortex/tree/HEAD) (2019-09-05)
+## [3.0.1](https://github.com/TheHive-Project/Cortex/tree/3.0.1) (2020-01-20)
+
+[Full Changelog](https://github.com/TheHive-Project/Cortex/compare/3.0.0...HEAD)
+
+**Implemented enhancements:**
+
+- Cortex logs the Play secret key at startup. [\#244](https://github.com/TheHive-Project/Cortex/issues/244)
+- Analyzer reports "no output" when it fails [\#241](https://github.com/TheHive-Project/Cortex/issues/241)
+- Docker image has many CVE's open against it [\#238](https://github.com/TheHive-Project/Cortex/issues/238)
+- Remove Elasticsearch cluster configuration option [\#230](https://github.com/TheHive-Project/Cortex/pull/230) ([adl1995](https://github.com/adl1995))
+- Handle second/minute-rates limits on Flavors and Analyzers [\#164](https://github.com/TheHive-Project/Cortex/issues/164)
+
+**Fixed bugs:**
+
+- Fix error message display for failed analyzers/responders [\#243](https://github.com/TheHive-Project/Cortex/issues/243)
+- Remove reference to google fonts [\#242](https://github.com/TheHive-Project/Cortex/issues/242)
+- Encoding issue causes invalid format for catalog file [\#240](https://github.com/TheHive-Project/Cortex/issues/240)
+- Missing dependency for cluster [\#239](https://github.com/TheHive-Project/Cortex/issues/239)
+- Old non-existent analysers showing in Cortex after an upgrade [\#234](https://github.com/TheHive-Project/Cortex/issues/234)
+
+## [3.0.0](https://github.com/TheHive-Project/Cortex/tree/3.0.0) (2019-09-05)
 
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/3.0.0-RC4...3.0.0)
 
@@ -9,6 +29,7 @@
 - cortex 3.0.0-RC4 container : StreamSrv error popup spamming the setup page [\#210](https://github.com/TheHive-Project/Cortex/issues/210)
 
 ## [3.0.0-RC4](https://github.com/TheHive-Project/Cortex/tree/3.0.0-RC4) (2019-07-10)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/3.0.0-RC3...3.0.0-RC4)
 
 **Fixed bugs:**
@@ -41,6 +62,7 @@
 - Get user detials via API is available to non-admin users [\#194](https://github.com/TheHive-Project/Cortex/issues/194)
 
 ## [3.0.0-RC2](https://github.com/TheHive-Project/Cortex/tree/3.0.0-RC2) (2019-05-03)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/3.0.0-RC1...3.0.0-RC2)
 
 **Fixed bugs:**
@@ -50,6 +72,7 @@
 - Docker container exposes tcp/9000 instead of tcp/9001 [\#166](https://github.com/TheHive-Project/Cortex/issues/166)
 
 ## [3.0.0-RC1](https://github.com/TheHive-Project/Cortex/tree/3.0.0-RC1) (2019-04-05)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.1.3...3.0.0-RC1)
 
 **Implemented enhancements:**
@@ -60,7 +83,7 @@
 
 **Fixed bugs:**
 
-- SSO: Authentication module not found  [\#181](https://github.com/TheHive-Project/Cortex/issues/181)
+- SSO: Authentication module not found [\#181](https://github.com/TheHive-Project/Cortex/issues/181)
 - Akka Dispatcher Blocked [\#170](https://github.com/TheHive-Project/Cortex/issues/170)
 
 **Closed issues:**
@@ -71,6 +94,7 @@
 - File extraction [\#120](https://github.com/TheHive-Project/Cortex/issues/120)
 
 ## [2.1.3](https://github.com/TheHive-Project/Cortex/tree/2.1.3) (2018-12-20)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.1.2...2.1.3)
 
 **Implemented enhancements:**
@@ -88,17 +112,19 @@
 
 **Closed issues:**
 
-- Systemd: cortex.service: Failed with result 'exit-code'.  [\#155](https://github.com/TheHive-Project/Cortex/issues/155)
+- Systemd: cortex.service: Failed with result 'exit-code'. [\#155](https://github.com/TheHive-Project/Cortex/issues/155)
 - conf/logback.xml: Rotate logs [\#62](https://github.com/TheHive-Project/Cortex/issues/62)
 
 ## [2.1.2](https://github.com/TheHive-Project/Cortex/tree/2.1.2) (2018-10-12)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.1.1...2.1.2)
 
 **Fixed bugs:**
 
 - findSimilarJob function broken [\#144](https://github.com/TheHive-Project/Cortex/issues/144)
 
 ## [2.1.1](https://github.com/TheHive-Project/Cortex/tree/2.1.1) (2018-10-09)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.1.0...2.1.1)
 
 **Implemented enhancements:**
@@ -114,6 +140,7 @@
 - Console output should not be logged in syslog [\#136](https://github.com/TheHive-Project/Cortex/issues/136)
 
 ## [2.1.0](https://github.com/TheHive-Project/Cortex/tree/2.1.0) (2018-09-25)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.1.0-RC1...2.1.0)
 
 **Implemented enhancements:**
@@ -124,7 +151,7 @@
 **Fixed bugs:**
 
 - MISP API fails [\#109](https://github.com/TheHive-Project/Cortex/issues/109)
-- File\_Info issue [\#53](https://github.com/TheHive-Project/Cortex/issues/53)
+- File_Info issue [\#53](https://github.com/TheHive-Project/Cortex/issues/53)
 - Temporary files are not removed at the end of job [\#129](https://github.com/TheHive-Project/Cortex/issues/129)
 - MISP fails to run analyzers [\#128](https://github.com/TheHive-Project/Cortex/issues/128)
 
@@ -133,6 +160,7 @@
 - Update resolvers in build.sbt to contain Maven as a dependency [\#130](https://github.com/TheHive-Project/Cortex/pull/130) ([adl1995](https://github.com/adl1995))
 
 ## [2.1.0-RC1](https://github.com/TheHive-Project/Cortex/tree/2.1.0-RC1) (2018-07-31)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.0.4...2.1.0-RC1)
 
 **Implemented enhancements:**
@@ -163,6 +191,7 @@
 - Update GitHub path [\#100](https://github.com/TheHive-Project/Cortex/pull/100) ([saadkadhi](https://github.com/saadkadhi))
 
 ## [2.0.4](https://github.com/TheHive-Project/Cortex/tree/2.0.4) (2018-04-13)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.0.3...2.0.4)
 
 **Implemented enhancements:**
@@ -184,6 +213,7 @@
 - Insufficient Rights To Perform This Action [\#87](https://github.com/TheHive-Project/Cortex/issues/87)
 
 ## [2.0.3](https://github.com/TheHive-Project/Cortex/tree/2.0.3) (2018-04-09)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.0.2...2.0.3)
 
 **Implemented enhancements:**
@@ -203,6 +233,7 @@
 - Allow configuring auto artifacts extraction per analyzer [\#80](https://github.com/TheHive-Project/Cortex/issues/80)
 
 ## [2.0.2](https://github.com/TheHive-Project/Cortex/tree/2.0.2) (2018-04-04)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.0.1...2.0.2)
 
 **Fixed bugs:**
@@ -214,6 +245,7 @@
 - Session collision when TheHive & Cortex 2 share the same URL [\#70](https://github.com/TheHive-Project/Cortex/issues/70)
 
 ## [2.0.1](https://github.com/TheHive-Project/Cortex/tree/2.0.1) (2018-03-30)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/2.0.0...2.0.1)
 
 **Fixed bugs:**
@@ -223,6 +255,7 @@
 - User can't change his password [\#67](https://github.com/TheHive-Project/Cortex/issues/67)
 
 ## [2.0.0](https://github.com/TheHive-Project/Cortex/tree/2.0.0) (2018-03-30)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.1.4...2.0.0)
 
 **Implemented enhancements:**
@@ -245,7 +278,7 @@
 - Local, LDAP, AD and API Key Authentication [\#7](https://github.com/TheHive-Project/Cortex/issues/7)
 - Limit Rates and Respect Quotas [\#6](https://github.com/TheHive-Project/Cortex/issues/6)
 - Persistence and Report Caching [\#5](https://github.com/TheHive-Project/Cortex/issues/5)
-- Provide alternative paths for analyzers in addition to standard path.  [\#4](https://github.com/TheHive-Project/Cortex/issues/4)
+- Provide alternative paths for analyzers in addition to standard path. [\#4](https://github.com/TheHive-Project/Cortex/issues/4)
 - Provide way to reload conf file for new API keys without shutdown. [\#3](https://github.com/TheHive-Project/Cortex/issues/3)
 - Provide Secret Key auth to upstream service [\#2](https://github.com/TheHive-Project/Cortex/issues/2)
 
@@ -256,6 +289,7 @@
 - Adding WOT config sample [\#43](https://github.com/TheHive-Project/Cortex/pull/43) ([mthlvt](https://github.com/mthlvt))
 
 ## [1.1.4](https://github.com/TheHive-Project/Cortex/tree/1.1.4) (2017-09-15)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.1.3...1.1.4)
 
 **Fixed bugs:**
@@ -269,6 +303,7 @@
 - Disable analyzer in configuration file [\#32](https://github.com/TheHive-Project/Cortex/issues/32)
 
 ## [1.1.3](https://github.com/TheHive-Project/Cortex/tree/1.1.3) (2017-06-14)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/debian/1.1.2-2...1.1.3)
 
 **Fixed bugs:**
@@ -277,9 +312,11 @@
 - Error when parsing analyzer failure report [\#33](https://github.com/TheHive-Project/Cortex/issues/33)
 
 ## [debian/1.1.2-2](https://github.com/TheHive-Project/Cortex/tree/debian/1.1.2-2) (2017-05-24)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.1.2...debian/1.1.2-2)
 
 ## [1.1.2](https://github.com/TheHive-Project/Cortex/tree/1.1.2) (2017-05-24)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/debian/1.1.1-2...1.1.2)
 
 **Implemented enhancements:**
@@ -294,16 +331,19 @@
 - Error 500 in TheHive when a job is submited to Cortex [\#27](https://github.com/TheHive-Project/Cortex/issues/27)
 
 ## [debian/1.1.1-2](https://github.com/TheHive-Project/Cortex/tree/debian/1.1.1-2) (2017-05-19)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/rpm/1.1.1-2...debian/1.1.1-2)
 
 ## [rpm/1.1.1-2](https://github.com/TheHive-Project/Cortex/tree/rpm/1.1.1-2) (2017-05-19)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.1.1...rpm/1.1.1-2)
 
 **Fixed bugs:**
 
 - After Upgrade from Cortex 1.0.2 to 1.1.1 system does not come up [\#26](https://github.com/TheHive-Project/Cortex/issues/26)
 
 ## [1.1.1](https://github.com/TheHive-Project/Cortex/tree/1.1.1) (2017-05-17)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.1.0...1.1.1)
 
 **Fixed bugs:**
@@ -316,6 +356,7 @@
 - MISP integration [\#21](https://github.com/TheHive-Project/Cortex/issues/21)
 
 ## [1.1.0](https://github.com/TheHive-Project/Cortex/tree/1.1.0) (2017-05-12)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.0.2...1.1.0)
 
 **Implemented enhancements:**
@@ -330,6 +371,7 @@
 - Use new logo and favicon [\#22](https://github.com/TheHive-Project/Cortex/issues/22)
 
 ## [1.0.2](https://github.com/TheHive-Project/Cortex/tree/1.0.2) (2017-04-19)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.0.1...1.0.2)
 
 **Fixed bugs:**
@@ -345,6 +387,7 @@
 - Documentation on 'How to create an analyzer' [\#10](https://github.com/TheHive-Project/Cortex/issues/10)
 
 ## [1.0.1](https://github.com/TheHive-Project/Cortex/tree/1.0.1) (2017-03-08)
+
 [Full Changelog](https://github.com/TheHive-Project/Cortex/compare/1.0.0...1.0.1)
 
 **Fixed bugs:**
@@ -357,5 +400,4 @@
 
 ## [1.0.0](https://github.com/TheHive-Project/Cortex/tree/1.0.0) (2017-02-01)
 
-
-\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
+\* _This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)_

app/org/thp/cortex/models/Worker.scala

@@ -1,22 +1,22 @@
 package org.thp.cortex.models
 
 import javax.inject.{Inject, Singleton}
-
-import scala.concurrent.Future
-import scala.util.Try
-
-import play.api.libs.json.{JsObject, JsString, Json}
-
 import org.elastic4play.models.JsonFormat.enumFormat
 import org.elastic4play.models.{AttributeDef, BaseEntity, ChildModelDef, EntityDef, HiveEnumeration, AttributeFormat ⇒ F, AttributeOption ⇒ O}
 import org.elastic4play.utils.Hasher
-
 import org.thp.cortex.models.JsonFormat.workerTypeFormat
+import play.api.libs.json.{JsObject, JsString, Json}
+
+import scala.concurrent.Future
+import scala.util.Try
 
 object RateUnit extends Enumeration with HiveEnumeration {
   type Type = Value
-  val Day            = Value(1)
-  val Month          = Value(30)
+  val Second         = Value(1)
+  val Minute         = Value(60)
+  val Hour           = Value(60 * 60)
+  val Day            = Value(60 * 60 * 24)
+  val Month          = Value(60 * 60 * 24 * 30)
   implicit val reads = enumFormat(this)
 }
 

app/org/thp/cortex/services/DockerJobRunnerSrv.scala

@@ -92,10 +92,12 @@ class DockerJobRunnerSrv(client: DockerClient, autoUpdate: Boolean, implicit val
       ()
     }.andThen {
       case r ⇒
-        if (!Files.exists(jobDirectory.resolve("output").resolve("output.json"))) {
-          val message = r.fold(e ⇒ s"Docker creation error: ${e.getMessage}\n", _ ⇒ "") +
-            Try(client.logs(containerCreation.id(), LogsParam.stdout(), LogsParam.stderr()).readFully())
-              .recover { case e ⇒ s"Container logs can't be read (${e.getMessage}" }
+        val outputFile = jobDirectory.resolve("output").resolve("output.json")
+        if (!Files.exists(outputFile) || Files.size(outputFile) == 0) {
+          val output = Try(client.logs(containerCreation.id(), LogsParam.stdout(), LogsParam.stderr()).readFully())
+            .fold(e ⇒ s"Container logs can't be read (${e.getMessage})", identity)
+          val message = r.fold(e ⇒ s"Docker creation error: ${e.getMessage}\n$output", _ ⇒ output)
+
           val report = Json.obj("success" → false, "errorMessage" → message)
           Files.write(jobDirectory.resolve("output").resolve("output.json"), report.toString.getBytes(StandardCharsets.UTF_8))
         }

app/org/thp/cortex/services/JobSrv.scala

@@ -327,7 +327,7 @@ class JobSrv(
       import org.elastic4play.services.QueryDSL._
       val now = new Date().getTime
       logger.info(s"Checking rate limit on worker ${worker.name()} from ${new Date(now - rateUnit.id.toLong * 24 * 60 * 60 * 1000)}")
-      stats(and("createdAt" ~>= (now - rateUnit.id.toLong * 24 * 60 * 60 * 1000), "workerId" ~= worker.id), Seq(selectCount)).map { s ⇒
+      stats(and("createdAt" ~>= (now - rateUnit.id.toLong * 1000), "workerId" ~= worker.id), Seq(selectCount)).map { s ⇒
         val count = (s \ "count").as[Long]
         logger.info(s"$count analysis found (limit is $rate)")
         count < rate

app/org/thp/cortex/services/ProcessJobRunnerSrv.scala

@@ -3,22 +3,18 @@ package org.thp.cortex.services
 import java.nio.charset.StandardCharsets
 import java.nio.file.{Files, Path, Paths}
 
-import scala.concurrent.duration.FiniteDuration
-import scala.concurrent.{ExecutionContext, Future}
-import scala.sys.process.{Process, ProcessLogger}
-
-import play.api.Logger
-
 import akka.actor.ActorSystem
 import javax.inject.{Inject, Singleton}
+import org.elastic4play.utils.RichFuture
 import org.thp.cortex.models._
+import play.api.Logger
+import play.api.libs.json.Json
 
-import org.elastic4play.utils.RichFuture
-import scala.sys.process._
+import scala.concurrent.duration.FiniteDuration
+import scala.concurrent.{ExecutionContext, Future}
+import scala.sys.process.{Process, ProcessLogger, _}
 import scala.util.Try
 
-import play.api.libs.json.Json
-
 @Singleton
 class ProcessJobRunnerSrv @Inject()(implicit val system: ActorSystem) {
 
@@ -37,19 +33,31 @@ class ProcessJobRunnerSrv @Inject()(implicit val system: ActorSystem) {
 
   def run(jobDirectory: Path, command: String, job: Job, timeout: Option[FiniteDuration])(implicit ec: ExecutionContext): Future[Unit] = {
     val baseDirectory = Paths.get(command).getParent.getParent
+    val output        = StringBuilder.newBuilder
     logger.info(s"Execute $command in $baseDirectory, timeout is ${timeout.fold("none")(_.toString)}")
     val process = Process(Seq(command, jobDirectory.toString), baseDirectory.toFile)
-      .run(ProcessLogger(s ⇒ logger.info(s"  Job ${job.id}: $s")))
+      .run(ProcessLogger { s ⇒
+        logger.info(s"  Job ${job.id}: $s")
+        output ++= s
+      })
     val execution = Future
       .apply {
         process.exitValue()
         ()
       }
+      .map { _ ⇒
+        val outputFile = jobDirectory.resolve("output").resolve("output.json")
+        if (!Files.exists(outputFile) || Files.size(outputFile) == 0) {
+          val report = Json.obj("success" → false, "errorMessage" → output.toString)
+          Files.write(outputFile, report.toString.getBytes(StandardCharsets.UTF_8))
+        }
+        ()
+      }
       .recoverWith {
         case error ⇒
           logger.error(s"Execution of command $command failed", error)
           Future.apply {
-            val report = Json.obj("success" → false, "errorMessage" → error.getMessage)
+            val report = Json.obj("success" → false, "errorMessage" → s"${error.getMessage}\n$output")
             Files.write(jobDirectory.resolve("output").resolve("output.json"), report.toString.getBytes(StandardCharsets.UTF_8))
             ()
           }