Merge pull request #36 from MislavReversingLabs/main

v2.4.2

Author: MislavReversingLabs

CHANGELOG.md

@@ -157,9 +157,11 @@ v2.3.0 (2023-09-29)
 ---
 
 
-v2.4.0 (2023-12-29)
+v2.4.0 (2023-12-29) - [YANKED]
 -------------------
 
+**Note:** Contains breaking changes in the `ExpressionSearch` class. We recommend using **v2.4.2**
+
 #### Improvements
 
 - **ticloud** module:
@@ -173,15 +175,35 @@ v2.4.0 (2023-12-29)
 ---
 
 
-v2.4.1 (2024-01-11)
+v2.4.1 (2024-01-11) - [YANKED]
 -------------------
 
+**Note:** Contains breaking changes in the `ExpressionSearch` class. We recommend using **v2.4.2**
+
 #### Improvements
 
 - **ticloud** module:
   - The `get_dynamic_analysis_results` method of the `DynamicAnalysis` class now also supports using a URL-s SHA-1 hash for fetching the URL dynamic analysis results. 
 
 - Error handling: Custom error classes now also carry the original response object. Users can now reach the original status code, error message and all other response properties using the caught error's `response_object` property. 
+---
+
+
+v2.4.2 (2024-01-22)
+-------------------
+
+All changes are calculated agains **v2.3.0**
+
+### Improvements
+- **ticloud** module:
+  - Added the `NewFilesFirstScan`, `NewFilesFirstAndRescan`, `FilesWithDetectionChanges`, `CvesExploitedInTheWild`, `NewExploitOrCveSamplesFoundInWildHourly`, `NewExploitAndCveSamplesFoundInWildDaily`, `NewWhitelistedFiles`, `ChangesWhitelistedFiles`, `MalwareFamilyDetection`, `ExpressionSearch`, `VerticalFeedStatistics` and `VerticalFeedSearch` classes.
+  - The following changes were made to the `DynamicAnalysis` class:
+    - Added `windows11` and `linux` to available Dynamic Analysis platforms.
+    - Added the `detonate_url` method.
+    - The `get_dynamic_analysis_results` method now supports fetching the URL dynamic analysis results using the URL string or its SHA-1 hash as a parameter.
+
+- Added TitaniumCloud API codes to the README for better correspondence and orientation.
+- Error handling: Custom error classes now also carry the original response object. Users can now reach the original status code, error message and all other response properties using the caught error's `response_object` property. 
 
 
 

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2023 ReversingLabs
+Copyright (c) 2024 ReversingLabs
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -311,20 +311,23 @@ _TCA-0320_
     - Accepts a search query string and performs advanced search on the API
     - Returns a list of results aggregated through multiple paginated requests
 
-
 #### Class:
 ```python
 class ExpressionSearch(TiCloudAPI)
-````
+```
 _TCA-0306_
 #### Methods:
 - `search`
-    - Accepts a list containing the search query and performs expression search on the API
-    - Returns only one defined page of results using one request
+    - Provides samples first seen on a particular date, filtered by search criteria.
 - `search_aggregated`
-    - Accepts a list containing the search query and performs expression search on the API
-    - Returns a list of results aggregated through multiple paginated requests
-    
+    - Provides samples first seen on a particular date, filtered by search criteria.
+    - This method performs the paging automatically.
+- `get_latest_expression`
+    - Provdes samples for yesterday’s date tha match the requested criteria.
+- `statistics_search`
+    - Returns statistics about new samples in ReversingLabs TitaniumCloud on the requested date that match the used search criteria.
+- `get_latest_statistics`
+    - Returns statistics about new samples in ReversingLabs TitaniumCloud from yesterday's date.
 
 #### Class:
 ```python
@@ -786,21 +789,6 @@ _TCA-0305_
 - `get_malware_family`
     - Returns all malware families to which sample belongs based on the detections from the latest AV scan
 
-#### Class:
-```python
-class ExpressionSearch(TiCloudAPI)
-```
-_TCA-0306_
-#### Methods:
-- `search`
-    - Returns a list of aggregated results from Titanium Cloud system that matches requested criteria
-- `get_latest_expression`
-    - Returns a list of  latest aggregated results from Titanuium Cloud system that matches requested criteria
-- `statistics_search`
-    - Returns aggregated statistics about new samples in Titainum Cloud system that matches requested criteria
-- `get_latest_statistics`
-    - Returns aggregated latest statistics about new samples in Titanium Cloud system that matches requested criteria
-
 #### Class:
 ```python
 class VerticalFeedsStatistics(TiCloudAPI)

ReversingLabs/SDK/ticloud.py

@@ -1215,7 +1215,7 @@ def search_aggregated(self, query_string, sorting_criteria=None, sorting_order="
 class ExpressionSearch(TiCloudAPI):
     """TCA-0306 - Expression Search with Statistics (Sample Search)"""
 
-    __EXPRESSION_QUERY_ENDPOINT = "/api/sample/search/download/v1/query/{time_format}/{time_value}"
+    __EXPRESSION_QUERY_ENDPOINT = "/api/sample/search/download/v1/query/date/{str_date}"
     __LATEST_EXPRESSION_ENDPOINT = "/api/sample/search/download/v1/query/latest"
     __STATISTICS_QUERY_ENDPOINT = "/api/sample/search/download/v1/statistics/{time_format}/{time_value}"
     __LATEST_STATISTICS_QUERY_ENDPOINT = "/api/sample/search/download/v1/statistics/latest"
@@ -1227,7 +1227,7 @@ def __init__(self, host, username, password, verify=True, proxies=None, user_age
 
         self._url = "{host}{{endpoint}}".format(host=self._host)
 
-    def search(self, query, time_format, time_value, page_number=1):
+    def search(self, query, date=None, page_number=1):
         """Sends the query to the Expression Search API.
         The query must be a list of at least 2 strings. Each string is in the form of a key and a value with
         an equals sign between them and no spaces.
@@ -1247,39 +1247,13 @@ def search(self, query, time_format, time_value, page_number=1):
 
             :param query: search query
             :type query: list
-            :param time_format: possibles values 'utc' or 'timestamp', 'date'
-            :type time_format: str
-            :param time_value: results will be retrieved from the specified date
-            :type time_value: str
+            :param date: return results from this date forward. the accepted date format is YYYY-mm-dd
+            :type date: str or any
             :param page_number: page number
             :type page_number: int
             :return: response
             :rtype: requests.Response
         """
-        if time_format == "timestamp":
-            try:
-                int(time_value)
-
-            except ValueError:
-                raise WrongInputError("if timestamp is used, time_value needs to be a unix timestamp")
-
-        elif time_format == "utc":
-            try:
-                datetime.datetime.strptime(time_value, "%Y-%m-%dT%H:%M:%S")
-
-            except ValueError:
-                raise WrongInputError("if utc is used, time_value needs to be in format 'YYYY-MM-DDThh:mm:ss'")
-
-        elif time_format == "date":
-            try:
-                datetime.datetime.strptime(time_value, "%Y-%m-%d")
-
-            except ValueError:
-                raise WrongInputError("if the date format is used, time_value must be provided as 'YYYY-MM-DD'")
-
-        else:
-            raise WrongInputError("time_format parameter must be one of the following: 'timestamp', 'utc', 'date'")
-
         if not isinstance(query, list):
             raise WrongInputError("query parameter must be a list of strings.")
 
@@ -1294,29 +1268,85 @@ def search(self, query, time_format, time_value, page_number=1):
         if not isinstance(page_number, int):
             raise WrongInputError("page_number parameter must be integer.")
 
-        base = self.__EXPRESSION_QUERY_ENDPOINT.format(
-            time_format=time_format,
-            time_value=time_value
+        if not date:
+            date = datetime.date.today() - datetime.timedelta(days=1)
+        if isinstance(date, str):
+            date = datetime.datetime.fromisoformat(date)
+
+        str_date = date.strftime("%Y-%m-%d")
+
+        endpoint_base = self.__EXPRESSION_QUERY_ENDPOINT.format(
+            str_date=str_date
         )
 
-        endpoint = "{base}?{query_expression}".format(
-            base=base,
+        parameters = "?format=json&page={page}&{query_expression}".format(
+            page=page_number,
             query_expression=query_expression
         )
 
-        query_params = {
-            "page": page_number,
-            "format": "json"
-        } 
+        endpoint = "{base}{params}".format(
+            base=endpoint_base,
+            params=parameters
+        )
 
         url = self._url.format(endpoint=endpoint)
 
-        response = self._get_request(url=url, params=query_params)
-
+        response = self._get_request(url=url)
         self._raise_on_error(response)
 
         return response
 
+    def search_aggregated(self, query, date=None, max_results=5000):
+        """Sends the query to the Expression Search API.
+        The query must be a list of at least 2 strings. Each string is in the form of a key and a value with
+        an equals sign between them and no spaces.
+        The value can have multiple options separated with a pipe symbol.
+        This method returns a list of aggregated results with a maximum length defined in the 'max_results' parameter.
+            Query examples:
+
+            ['status=MALICIOUS',
+            'sample_type=MicrosoftWord|MicrosoftExcel|MicrosoftPowerPoint']
+
+            or
+
+            ['threat_level>=3',
+            'status=malicious',
+            'malware_family=CVE-2017-11882']
+
+            :param query: search query
+            :type query: list
+            :param date: return results from this date forward
+            :type date: str or any
+            :param max_results: maximum results to be returned in the list; default value is 5000
+            :type max_results: int
+            :return: list of results
+            :rtype: list
+        """
+        if not isinstance(max_results, int):
+            raise WrongInputError("max_results parameter must be integer.")
+
+        results = []
+        next_page = 1
+
+        while next_page:
+            response = self.search(
+                query=query,
+                date=date,
+                page_number=next_page
+            )
+
+            response_json = response.json()
+
+            entries = response_json.get("rl").get("web_sample_search_download").get("entries", [])
+            results.extend(entries)
+
+            next_page = response_json.get("rl").get("web_sample_search_download").get("next_page", None)
+
+            if len(results) >= max_results:
+                break
+
+        return results[:max_results]
+
     def get_latest_expression(self, query):
         """Service returns only new samples from the last 24 hours.
         The query must be a list of at least 2 strings. Each string is in the form of a key and a value with 

setup.py

@@ -11,7 +11,7 @@
 
 setup(
     name="reversinglabs-sdk-py3",
-    version="2.4.1",
+    version="2.4.2",
     description="Python SDK for using ReversingLabs services - Python 3 version.",
     long_description=long_description,
     long_description_content_type="text/markdown",