Initial commit

Author: nadouani

.gitattributes

@@ -0,0 +1 @@
+ui/app/styles/vendors/* linguist-vendored

.github/issue_template.md

@@ -0,0 +1,30 @@
+# EDIT THIS TITLE BEFORE POSTING. Use this template for bug reports. If you'd like to request a feature, please be as descriptive as possible and delete the template except the first section (Request Type)
+
+### Request Type
+(select Bug or Feature Request and **remove this line**)
+Bug / Feature Request
+
+### Work Environment
+
+| Question              | Answer
+|---------------------------|--------------------
+| OS version (server)       | Debian, Ubuntu, CentOS, RedHat, ...
+| OS version (client)       | XP, Seven, 10, Ubuntu, ...
+| Cortex version / git hash   | 1.x, hash of the commit
+| Package Type              | Docker, Binary, From source
+| Browser type & version    | If applicable
+
+
+### Problem Description
+Describe the problem/bug as clearly as possible.
+
+### Steps to Reproduce
+1. step 1
+1. step 2
+1. step 3...
+
+### Possible Solutions
+(keep this section if you have suggestions on how to solve the problem. **Otherwise delete it**)
+
+### Complementary information
+(add anything that can help identifying the problem such as **logs**, **screenshots**, **configuration dumps** etc.)

.gitignore

@@ -0,0 +1,18 @@
+logs
+target
+/.idea
+/.idea_modules
+/.classpath
+/.project
+/.settings
+/RUNNING_PID
+.cache-main
+.cache-tests
+*.py[cod]
+/report-templates/*.zip
+
+/bin/
+!/bin/activator
+!/bin/activator.bat
+
+conf/application.conf

AUTHORS

@@ -0,0 +1,16 @@
+Authors
+-------
+
+* Thomas Franco <[email protected]> (lead developer, back-end)
+* Saâd Kadhi <[email protected]> (project leader, product management & design)
+* Jérôme Leonard <[email protected]> (developer, analyzers)
+
+Contributors
+------------
+
+* Nabil Adouani
+* CERT Banque de France (CERT-BDF)
+
+Copyright (C) 2016-2017 Thomas Franco
+Copyright (C) 2016-2017 Saâd Kadhi
+Copyright (C) 2016-2017 Jérôme Leonard

LICENSE

@@ -0,0 +1,59 @@
+
+**Cortex** tries to solve a common problem frequently encountered by SOCs, CSIRTs and security researchers in the course of threat intelligence, digital forensics and incident response: how to **analyze observables** they have collected, **at scale**, **by querying a single tool** instead of several?
+
+Cortex, an open source and free software, has been created by [TheHive Project](https://thehive-project.org) for this very purpose. Observables, such as IP and email addresses, URLs, domain names, files or hashes, can be analyzed one by one or in bulk mode using a Web interface. Analysts can also **automate** these operations thanks to the Cortex REST API. 
+![](images/cortex-analyzers.png)
+
+By using Cortex, you won't need to rewrite the wheel every time you'd like to use a service or a tool to analyze an observable and help you investigate the case at hand. Leverage one of the several analyzers it contains and if you are missing a tool or a service, create a suitable program easily and make it available for the whole team (or better, [for the whole community](https://github.com/CERT-BDF/cortex-analyzers/)) thanks to Cortex.
+
+# Cortex and TheHive
+Along with [MISP](http://www.misp-project.org/), Cortex is the perfect companion for [TheHive](https://thehive-project.org). Starting from Buckfast (TheHive version 2.10), you can analyze tens or hundreds of observables in a few clicks using one or several Cortex instances depending on your OPSEC needs and security requirements. Moreover, TheHive comes with a report template engine that allows you to adjust the output of Cortex analyzers to your taste instead of having to create your own JSON parsers for Cortex output.
+
+# Architecture
+Cortex is written in Scala. The front-end uses AngularJS with Bootstrap. Its REST API is stateless which allows it to be horizontally scalable. The provided analyzers are written in Python. Additional analyzers may be written using the same language or any other language supported by Linux.
+
+<p align="center">
+  <img src="images/cortex-architecture.png" alt="Cortex Architecture" width="400">
+</p>
+
+
+# Analyzers
+Cortex 1.0.0 is provided with 13 analyzers.
+
++ Abuse Finder: use CERT-SG's [Abuse Finder](https://github.com/certsocietegenerale/abuse_finder) to find the abuse contact associated with domain names, URLs, IP and email addresses.
++ DNSDB\*: leverage Farsight's [DNSDB](https://www.dnsdb.info/) for pDNS.
++ DomainTools\*: look up domain names, IP addresses, WHOIS records, etc. using the popular [DomainTools](http://domaintools.com/) service API.
++ File Info: parse files in several formats such as OLE and OpenXML to detect VBA macros, extract their source code, generate useful information on PE, PDF files and much more.
++ Hippocampe: query threat feeds through [Hippocampe](https://github.com/CERT-BDF/Hippocampe), a FOSS tool that centralizes feeds and allows you to associate a confidence level to each one of them (that can be changed over time) and get a score indicating the data quality.
++ MaxMind: geolocation.
++ Outlook MsgParser: parse Outlook message files automatically and show the key information it contains such as headers, attachments etc.
++ OTXQuery\*: query AlienVault [Open Threat Exchange](https://otx.alienvault.com/) for IPs, domains, URLs, or file hashes.
++ PassiveTotal\*: leverage [RiskIQ's PassiveTotal](https://www.passivetotal.org/) service to gain invaluable insight on observables, identify overlapping infrastructure using Passive DNS, WHOIS, SSL certificates and more.
++ URLCategory: checks the Fortinet categories of URLs.
++ Phishing Initiative\*: queries [Phishing Initiative](https://phishing-initiative.fr/contrib/) to assess whether a URL has been flagged a phishing site.
++ PhishTank\*: queries [PhishTank](https://www.phishtank.com/) to assess whether a URL has been flagged a phishing site.
++ VirusTotal\*: look up files, URLs and hashes through [VirusTotal](https://www.virustotal.com/).
+
+The star (\*) indicates that the analyzer needs an API key to work correctly. **We do not provide API keys**. You have to use your own.
+
+# License
+Cortex is an open source and free software released under the [AGPL](https://github.com/CERT-BDF/Cortex/blob/master/LICENSE) (Affero General Public License). We, TheHive Project, are committed to ensure that Cortex will remain a free and open source project on the long-run.
+
+# Updates
+Information, news and updates are regularly posted on [TheHive Project Twitter account](https://twitter.com/thehive_project) and on [the blog](https://blog.thehive-project.org/).
+
+# Contributing
+We welcome your contributions, **[particularly new analyzers](https://github.com/CERT-BDF/cortex-analyzers/)** that can take away the load off overworked fellow analysts. Please feel free to fork the code, play with it, make some patches and send us pull requests.
+
+# Support
+Please [open an issue on GitHub](https://github.com/CERT-BDF/Cortex/issues) if you'd like to report a bug or request a feature.
+
+**Important Note**: if you encounter an issue with an analyzer or would like to request a new one or an improvement to an existing analyzer, please open an issue on the [analyzers' dedicated GitHub repository](https://github.com/CERT-BDF/cortex-analyzers/issues/new). If you have problems with TheHive or would like to request a TheHive-related feature, please [open an issue on its dedicated GitHub repository](https://github.com/CERT-BDF/TheHive/issues/new).
+
+Alternatively, if you need to contact the project team, send an email to <[email protected]>.
+
+# Community Discussions
+We have set up a Google forum at <https://groups.google.com/a/thehive-project.org/d/forum/users>. To request access, you need a Google account. You may create one [using a Gmail address](https://accounts.google.com/SignUp?hl=en) or [without one](https://accounts.google.com/SignUpWithoutGmail?hl=en).
+
+# Website
+<https://thehive-project.org/>

README.md

@@ -0,0 +1,21 @@
+import play.api.{ Configuration, Environment, Mode }
+import play.api.libs.concurrent.AkkaGuiceSupport
+
+import com.google.inject.AbstractModule
+
+import net.codingwell.scalaguice.ScalaModule
+
+import controllers.{ AssetCtrl, AssetCtrlDev, AssetCtrlProd }
+import services.JobActor
+
+class Module(environment: Environment, configuration: Configuration) extends AbstractModule with ScalaModule with AkkaGuiceSupport {
+
+  override def configure() = {
+    bindActor[JobActor]("JobActor")
+
+    if (environment.mode == Mode.Prod)
+      bind[AssetCtrl].to[AssetCtrlProd]
+    else
+      bind[AssetCtrl].to[AssetCtrlDev]
+  }
+}

app/Module.scala

@@ -0,0 +1,63 @@
+package controllers
+
+import javax.inject.Inject
+
+import scala.annotation.implicitNotFound
+import scala.concurrent.{ ExecutionContext, Future }
+
+import play.api.libs.json.{ JsObject, JsString, Json }
+import play.api.mvc.{ Action, AnyContent, Controller, Request }
+
+import models.{ DataArtifact, FileArtifact }
+import models.JsonFormat.{ analyzerWrites, dataActifactReads, jobWrites }
+import services.{ AnalyzerSrv, JobSrv }
+
+class AnalyzerCtrl @Inject() (
+    analyzerSrv: AnalyzerSrv,
+    jobSrv: JobSrv,
+    implicit val ec: ExecutionContext) extends Controller {
+
+  def list = Action { request ⇒
+    Ok(Json.toJson(analyzerSrv.list))
+  }
+
+  def get(analyzerId: String) = Action { request ⇒
+    analyzerSrv.get(analyzerId) match {
+      case Some(analyzer) ⇒ Ok(Json.toJson(analyzer))
+      case None           ⇒ NotFound
+    }
+  }
+
+  private[controllers] def readDataArtifact(request: Request[AnyContent]) = {
+    for {
+      json ← request.body.asJson
+      artifact ← json.asOpt[DataArtifact]
+    } yield artifact
+  }
+
+  private[controllers] def readFileArtifact(request: Request[AnyContent]) = {
+    for {
+      parts ← request.body.asMultipartFormData
+      filePart ← parts.file("data").headOption
+      attrList ← parts.dataParts.get("_json")
+      attrStr ← attrList.headOption
+      attr ← Json.parse(attrStr).asOpt[JsObject]
+    } yield FileArtifact(filePart.ref.file, attr +
+      ("content-type" → JsString(filePart.contentType.getOrElse("application/octet-stream"))) +
+      ("filename" → JsString(filePart.filename)))
+  }
+
+  def analyze(analyzerId: String) = Action.async { request ⇒
+    readDataArtifact(request)
+      .orElse(readFileArtifact(request))
+      .map { artifact ⇒
+        jobSrv.create(artifact, analyzerId)
+          .map(j ⇒ Ok(Json.toJson(j)))
+      }
+      .getOrElse(Future.successful(BadRequest("???")))
+  }
+
+  def listForType(dataType: String) = Action { request ⇒
+    Ok(Json.toJson(analyzerSrv.listForType(dataType)))
+  }
+}

app/controllers/Analyzer.scala

@@ -0,0 +1,28 @@
+package controllers
+
+import javax.inject.{ Inject, Singleton }
+
+import play.api.Environment
+import play.api.http.HttpErrorHandler
+import play.api.mvc.{ Action, AnyContent, Controller }
+
+trait AssetCtrl {
+  def get(file: String): Action[AnyContent]
+}
+
+@Singleton
+class AssetCtrlProd @Inject() (errorHandler: HttpErrorHandler) extends Assets(errorHandler) with AssetCtrl {
+  def get(file: String) = at("/ui", file)
+}
+
+@Singleton
+class AssetCtrlDev @Inject() (environment: Environment) extends ExternalAssets(environment) with AssetCtrl {
+  def get(file: String) = {
+    if (file.startsWith("bower_components/")) {
+      at("ui", file)
+    }
+    else {
+      at("ui/app", file)
+    }
+  }
+}

app/controllers/Asset.scala

@@ -0,0 +1,58 @@
+package controllers
+
+import javax.inject.Inject
+
+import scala.annotation.implicitNotFound
+import scala.concurrent.ExecutionContext
+import scala.concurrent.duration.Duration
+import scala.util.{ Failure, Success }
+
+import play.api.libs.json.{ JsString, Json }
+import play.api.mvc.{ Action, Controller }
+
+import models.JsonFormat.{ jobStatusWrites, jobWrites }
+import services.JobSrv
+
+class JobCtrl @Inject() (
+    jobSrv: JobSrv,
+    implicit val ec: ExecutionContext) extends Controller {
+  def list(dataTypeFilter: Option[String], dataFilter: Option[String], analyzerFilter: Option[String], start: Int, limit: Int) = Action.async { request ⇒
+    jobSrv.list(dataTypeFilter, dataFilter, analyzerFilter, start, limit).map {
+      case (total, jobs) ⇒ Ok(Json.toJson(jobs)).withHeaders("X-Total" → total.toString)
+    }
+  }
+
+  def get(jobId: String) = Action.async { request ⇒
+    jobSrv.get(jobId).map { job ⇒
+      Ok(Json.toJson(job))
+    }
+  }
+
+  def remove(jobId: String) = Action.async { request ⇒
+    jobSrv.remove(jobId).map(_ ⇒ Ok(""))
+  }
+
+  def report(jobId: String) = Action.async { request ⇒
+    jobSrv
+      .get(jobId)
+      .map { job ⇒
+        val report = job.report.value match {
+          case Some(Success(report)) ⇒ report
+          case Some(Failure(error))  ⇒ JsString(error.getMessage)
+          case None                  ⇒ JsString("Running")
+        }
+        Ok(jobWrites.writes(job) +
+          ("status" → jobStatusWrites.writes(job.status)) +
+          ("report" → report))
+      }
+  }
+
+  def waitReport(jobId: String, atMost: String) = Action.async { request ⇒
+    for {
+      job ← jobSrv.get(jobId)
+      (status, report) ← jobSrv.waitReport(jobId, Duration(atMost))
+    } yield Ok(jobWrites.writes(job) +
+      ("status" → jobStatusWrites.writes(job.status)) +
+      ("report" → report))
+  }
+}

app/controllers/Job.scala

@@ -0,0 +1,13 @@
+package models
+
+import scala.concurrent.Future
+import play.api.libs.json.JsObject
+
+abstract class Analyzer {
+  def analyze(artifact: Artifact): Future[JsObject]
+  val name: String
+  val version: String
+  val description: String
+  val dataTypeList: Seq[String]
+  val id = (name + "_" + version).replaceAll("\\.", "_")
+}

app/models/Analyzer.scala

@@ -0,0 +1,25 @@
+package models
+
+import play.api.libs.json.JsObject
+import java.io.File
+
+abstract class Artifact(attributes: JsObject) {
+  def dataTypeFilter(filter: String): Boolean = (attributes \ "dataType").asOpt[String].fold(false)(_.toLowerCase.contains(filter.toLowerCase))
+  def dataFilter(filter: String): Boolean = false
+}
+class FileArtifact(val data: File, val attributes: JsObject) extends Artifact(attributes) {
+  override def finalize {
+    data.delete()
+  }
+}
+object FileArtifact {
+  def apply(data: File, attributes: JsObject) = {
+    val tempFile = File.createTempFile("cortex-", "-datafile")
+    data.renameTo(tempFile)
+    new FileArtifact(tempFile, attributes)
+  }
+  def unapply(fileArtifact: FileArtifact) = Some(fileArtifact.data → fileArtifact.attributes)
+}
+case class DataArtifact(data: String, attributes: JsObject) extends Artifact(attributes) {
+  override def dataFilter(filter: String): Boolean = data.toLowerCase.contains(filter.toLowerCase)
+}