Skip to content

Files

Latest commit

profnandaaprofnandaa
logmonitor: docs: add security advisory for config file (#196)
Oct 14, 2024
fb712c3 · Oct 14, 2024

History

History
496 lines (408 loc) · 20.6 KB

README.md

File metadata and controls

496 lines (408 loc) · 20.6 KB

Log Monitor Documentation

Contents:

Sample Config File

A sample Log Monitor Config file would be structured as follows:

{
  "LogConfig": {
    "sources": [
      {
        "type": "EventLog",
        "startAtOldestRecord": true,
        "eventFormatMultiLine": false,
        "channels": [
          {
            "name": "system",
            "level": "Information"
          },
          {
            "name": "application",
            "level": "Error"
          }
        ]
      },
      {
        "type": "File",
        "directory": "c:\\inetpub\\logs",
        "filter": "*.log",
        "includeSubdirectories": true,
        "includeFileNames": false
      },
      {
        "type": "ETW",
        "eventFormatMultiLine": false,
        "providers": [
          {
            "providerName": "IIS: WWW Server",
            "providerGuid": "3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83",
            "level": "Information"
          },
          {
            "providerName": "Microsoft-Windows-IIS-Logging",
            "providerGuid": "7E8AD27F-B271-4EA2-A783-A47BDE29143B",
            "level": "Information"
          }
        ]
      }
    ]
  }
}

Please see below for how to customize your Config file for Log Monitor to pull from.

ETW Monitoring

Description

ETW (Event Tracing for Windows) is a system for routing events. ETW is primarily intended for diagnostic purposes and is optimized to minimize impact on the overall system performance. Please note that ETW is different from the Windows Event Log.

To monitor ETW logs using Log Monitor you require a valid provider GUID (Globally Unique Identifier).

The provider GUID that you pass in the JSON config file must be registered with ETW. Log Monitor currently allows you to monitor events generated by MOF (classic) providers, manifest-based providers and Trace Logging providers. Note: we do not monitor log events from WPP providers.

You can see all the providers on your system using the logman query providers command.

To confirm if you have a valid provider name or valid provider GUID you can type the command below in command prompt or PowerShell.

logman query providers | findstr "<GUID or Provider Name>"

Configuration

  • type (required): This indicates the type of log you want to monitor for. It should be ETW.
  • eventFormatMultiLine (optional): This is Boolean to indicate whether you want the logs displayed with or without new lines. It is initially set to True and you can set it to False depending on how you want to view the logs on the console.
  • providers (required): Providers are components that generate events. This field is a list that shows the event providers you are monitoring for.
    • providerName (optional): This represents the name of the provider. It is what shows up when you use logman.
    • providerGuid (required): This is a globally unique identifier that uniquely identifies the provider you specified in the ProviderName field.
    • level (optional): This string field specifies the verboseness of the events collected. These include Critical, Error, Warning, Information and Verbose. If the level is not specified, level will be set to Error.
    • keywords (optional): This string field is a bitmask that specifies what events to collect. Only events with keywords matching the bitmask are collected This is an optional parameter. Default is 0 and all the events will be collected.

Examples

Using only the provider's GUID in your configuration:

{
  "LogConfig": {
    "sources": [
      {
        "type": "ETW",
        "eventFormatMultiLine": false,
        "providers": [
          {
            "providerGuid": "DAA6A96B-F3E7-4D4D-A0D6-31A350E6A445",
            "level": "Information"
          }
        ]
      }
    ]
  }
}

Using both the provider's name and provider GUID:

{
  "LogConfig": {
    "sources": [
      {
        "type": "ETW",
        "eventFormatMultiLine": false,
        "providers": [
          {
            "providerName": "Microsoft-Windows-WLAN-Drive",
            "providerGuid": "DAA6A96B-F3E7-4D4D-A0D6-31A350E6A445",
            "level": "Information"
          }
        ]
      }
    ]
  }
}

References

Event Log Monitoring

Description

Event log is a record of events related to the system, security, and application stored on a Windows operating system. The event log provides a standard, centralized way for apps and the operating system to record important software and hardware events, helping in tracking of potential threats and problems that could be potentially degrading performance.

Configuration

  • startAtOldestRecord (Required): This Boolean field indicates whether the Log Monitor tool should output event logs from the start of the container boot or from the start of the Log Monitor tool itself. If set true, the tool should output the event logs from the start of container boot, and if set false, the tool only outputs event logs from the start of log monitor.
  • eventFormatMultiLine (Optional): This is a Boolean field that is used to indicate whether the Log Monitor should format the logs to STDOUT as multi-line or single line. If the field is not set in the config file, by default the value is true. If the field is set true, the tool does not format the event messages to a single line (and thus event messages can span multiple lines). If set to false, the tool formats the event log messages to a single line and removes new line characters.
  • channels (Required): A channel is a named stream of events. It serves as a logical pathway for transporting events from the event publisher to a log file and possibly a subscriber. It is a sink that collects events. Each defined channel has the following properties:
    • name (Required): The name of the event channel
    • level (optional): This string field specifies the verboseness of the events collected. These include Critical, Error, Warning, Information and Verbose. If the level is not specified, level will be set to Error.

Examples

Example 1 (Application channel, verboseness: Error):

{
 "LogConfig": {
   "sources": [
     {
       "type": "EventLog",
       "startAtOldestRecord": true,
       "eventFormatMultiLine": false,
       "channels": [
         {
           "name": "application",
           "level": "Error"
         }
       ]
     }
   ]
 }
}

Example 2 (System channel, verboseness: Information):

{
 "LogConfig": {
   "sources": [
     {
       "type": "EventLog",
       "startAtOldestRecord": true,
       "eventFormatMultiLine": true,
       "channels": [
         {
           "name": "system",
           "level": " Information"
         }
       ]
     }
   ]
 }
}

References

Log File Monitoring

Description

This will monitor any changes in log files matching a specified filter, given the log directory location in the configuration. For instance, given a directory like C:\temp\logs and *.log filter, this will monitor any changes in .log files within the C:\temp\logs directory.

Configuration

  • type (required): "File"

  • directory (required): set to the directory containing the files to be monitored.

    ❕NOTE: Only works with absolute paths.

    To support long file name functionality, we prepend "\?" to the path. This approach extends the MAX_PATH limit from 260 characters to 32,767 wide characters. For more details, see Maximum Path Length Limitation.

    Due to this modification, the path must be an absolute path, beginning with a disk designator with a backslash, for example "C:" or "d:".

    UNC paths and DOS device paths are not supported.

    Ensure you identify the type of path and the path is correctly formatted to avoid issues.

    Example Path Type Allowed
    "c:" Absolute ✅
    "c:\" Absolute ✅
    "c:\temp" Absolute ✅
    "\tempdir" Absolute ❌
    "C:tempdir" Relative ❌
    "\\.\Volume{b75e2c83-0000-0000-0000-602f00000000}\Test" Volume GUID path ❌
    "\\.\c:\temp" DOS Device Path ❌
    "\\?\c:\temp" DOS Device Path ❌
    "\\127.0.0.1\c$\temp" UNC ❌
    "\\LOCALHOST\c$\temp" UNC ❌
    "\\.\UNC\LOCALHOST\c$\temp" UNC ❌
    "." Relative ❌
    "." Relative ❌
    "..\temp" Relative ❌

  • filter (optional): uses MS-DOS wildcard match type i.e.. *, ?. Can be set to empty, which will be default to "*".

  • includeSubdirectories (optional) : "true|false", specify if sub-directories also need to be monitored. Defaults to false.

  • includeFileNames (optional): "true|false", specifies whether to include file names in the logline, eg. sample.log: xxxxx. Defaults to false.

  • waitInSeconds (optional): specifies the duration to wait for a file or folder to be created if it does not exist. It takes integer values between 0-INFINITY. Defaults to 300 seconds, i.e, 5 minutes. It can be passed as a value or a string.

    • waitInSeconds = 0

      When the value is zero(0), this is means that we do not wait and LogMonitor terminates with an error

    • waitInSeconds = +integer

      When the value is a positive integer, LogMonitor will wait for the specified time. Once the predefined time elapses, LogMonitor will terminate with an error.

    • waitInSeconds = "INFINITY"

      In this case, LogMonitor will wait forever for the folder to be created.

      ❕NOTE

      • This field is case insensitive
      • When "INFINITY" is passed, it must be passed as a string.
      • The infinity symbol, ∞, is also allowed as a string or the symbol itself.

    Examples:

    1. Wait for 10 seconds
      • As a value: "waitInSeconds": 10
      • As a string: "waitInSeconds": "10"
    2. Wait forever/infinitely:
      • "waitInSeconds": "INFINITY" or "waitInSeconds": "inf" or "waitInSeconds": "∞"
      • This field is case-insensitive

    If a user provides an invalid value, a value less than 0, an error occurs:

    ERROR: Error parsing configuration file. 'waitInSeconds' attribute must be greater or equal to zero
WARNING: Failed to parse configuration file. Error retrieving source attributes. Invalid source

Sample FileMonitor LogMonitorConfig.json

Example 1

LogMonitor will monitor log files in the directory "c:\inetpub\logs" along with its subfolders. If the directory does not exist, it will wait for up to 10 seconds for the directory to be created.

{
  "LogConfig": {
    "sources": [
      {
        "type": "File",
        "directory": "c:\\inetpub\\logs",
        "filter": "*.log",
        "includeSubdirectories": true,
        "includeFileNames": false,
        "waitInSeconds": 10
      }
    ]
  }
}

Example 2

LogMonitor will monitor log files in the root directory, "C:".

When the directory is the root directory (e.g. "C:\" ) we can only monitor a file that is in the root directory, not a subfolder. This is due to access issues (even when running LogMonitor as an Admin) for some of the folders in the root directory. Therefore, includeSubdirectories must be false for the root directory.

See sample valid LogMonitorConfig.json below:

{
  "LogConfig": {
      "sources": [
          {
              "type": "File",
              "directory": "C:",
              "filter": "*.log",
              "includeSubdirectories": false
          }
      ]
  }
}

Example 3

This example shows an invalid file monitor configuration:

{
  "LogConfig": {
      "sources": [
          {
              "type": "File",
              "directory": "C:",
              "filter": "*.log",
              "includeSubdirectories": true
          }
      ]
  }
}

When the root directory is passed and includeSubdirectories = true, we get an error:

ERROR: LoggerSettings: Invalid Source File atrribute 'directory' (C:) and 'includeSubdirectories' (true).'includeSubdirectories' attribute cannot be 'true' for the root directory
WARNING: Failed to parse configuration file. Error retrieving source attributes. Invalid source

Process Monitoring

Description

Currently, this one does not need any configuration, it basically streams the output of tthe process/command that is provided as the argument to LogMonitor.exe. For the given entryproint command, a child process is created and links its STDIN and STDOUT to the LogMonitor process (through a pipe).

Examples

SHELL ["C:\\LogMonitor\\LogMonitor.exe", "cmd", "/S", "/C"] 
CMD "c:\\windows\\system32\\ping.exe -n 20 localhost"

The Process Monitor will stream the output for c:\windows\system32\ping.exe -n 20 localhost

Log Format Customization

Description

By default, logs will be displayed in JSON format. However, users can change the log format to either XML or their own custom defined format.

To specify the log format, a user needs to configure the logFormat field in LogMonitorConfig.json to either XML, JSON or Custom (the field value is not case-insensitive)
For JSON and XML log formats, no additional configurations are required. However, the Custom log format, needs further configuration. For custom log formats, a user needs to specify the customLogFormat at the source level.

Custom Log Format Pattern Layout

To ensure the different field values are correctly displayed in the customized log outputs, ensure to wrap the field names within modulo operators (%) and the field names specified matches the correct log sources' field names.

For example: %Message%, %TimeStamp%

Each log source tracked by log monitor (ETW, Log File, Events, and Process Monitor logs) has log field names specific to them:

Event Logs:

  • Source: The log source (Event Log)
  • TimeStamp: Time at which the event was generated
  • EventID: Unique identifier assigned to an individual event
  • Severity: A label that indicates the importance or criticality of an event
  • Message: The event message

ETW:

  • Source: The log source (ETW)
  • TimeStamp: Time at which the event was generated
  • Severity: A label that indicates the importance or criticality of an event
  • ProviderId: Unique identifier that is assigned to the event provider during its registration process.
  • ProviderName: Unique identifier or name assigned to an event provider
  • DecodingSource: Component or provider responsible for decoding and translating raw event data into a human-readable format
  • ExecutionProcessId: Identifier associated with a process that is being executed at the time an event is generated
  • ExecutionThreadId: Identifier associated with a thread at the time an event is generated
  • Keyword: Flag or attribute assigned to an event or a group of related events
  • EventId: Unique identifier assigned to an individual event
  • EventData: Payload or data associated with an event.

Log Files:

  • Source: The log source (File)
  • TimeStamp: Time at which the change was introduced in the monitored file.
  • FileName: Name of the file that the log entry is read from.
  • Message: The line/change added in the monitored file.

Process Monitor:

  • Source: The log source (Process Monitor)
  • TimeStamp: Time at which the process was executed
  • Message : The output of the process/command executed

Sample Custom Log Configuration

{
  "LogConfig": {
    "logFormat": "custom",
    "sources": [
      {
        "type": "ETW",
        "eventFormatMultiLine": false,
        "providers": [
          {
            "providerName": "Microsoft-Windows-WLAN-Drive",
            "providerGuid": "DAA6A96B-F3E7-4D4D-A0D6-31A350E6A445",
            "level": "Information"
          }
        ],
        "customLogFormat": "{'TimeStamp':'%TimeStamp%', 'Source':'%Source%', 'Severity':'%Severity%', 'ProviderId':'%ProviderId%', 'ProviderName':'%ProviderName%', 'EventId':'%EventId%', 'EventData':'%EventData%'}"
      },
      {
        "type": "File",
        "directory": "c:\\inetpub\\logs",
        "filter": "*.log",
        "includeSubdirectories": true,
        "customLogFormat": "{'Message':%Message%,'Source':%Source%,'fileName':%FileName%}"
      },
      {
        "type": "Process",
        "customLogFormat": "{'TimeStamp':'%TimeStamp%', 'Source':'%Source%', 'Message':'%Message%'}" 
      }
    ]
  }
}

For advanced usage of the custom log feature, a user can choose to define their own custom JSON log format. In such a case, The logFormat value should be custom.
To enable sanitization of the JSON output and ensure the the outputs displayed by the tool is valid, the user can add a suffix: '|json' after the desired custom log format.

For example:

{
  "LogConfig": {
	"logFormat": "custom",
        "sources": [
	      {
                "type": "ETW",
                "eventFormatMultiLine": false,
                "providers": [
                  {
                     "providerName": "Microsoft-Windows-WLAN-Drive",
                    "providerGuid": "DAA6A96B-F3E7-4D4D-A0D6-31A350E6A445",
                    "level": "Information"
                   }
                 ],
                "customLogFormat": "{'TimeStamp':'%TimeStamp%', 'Source':'%Source%', 'Severity':'%Severity%', 'ProviderId':'%ProviderId%', 'ProviderName':'%ProviderName%', 'EventId':'%EventId%', 'EventData':'%EventData%'}|json"
	      },
          {
            "type": "Process",
            "customLogFormat": "{'TimeStamp':'%TimeStamp%', 'Source':'%Source%', 'Message':'%Message%'}|JSON" 
          }
        ]
  }
}

Security Advisory for Config File

For extra security for cases where you have low privilege users for your container, you can restrict modification for the config file to only administrators. You can do this by running this in the container or adding it in your dockerfile.

# update the path if not using default
icacls C:/LogMonitor/LogMonitorConfig.json /grant Administrators:F /deny Users:W

This grants full access (F) to Administrators group and denies Write (W) permission to the Users group, which includes standard users (such as ContainerUser in the container).