Merge branch 'main' into plugins

Author: simonrob

README.md

@@ -172,6 +172,7 @@ It is often helpful to be able to view the raw connection details when debugging
 This can be achieved using `telnet`, [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/) or similar.
 For example, to test the Office 365 IMAP server from the [example configuration](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config), first open a connection using `telnet 127.0.0.1 1993`, and then send a login command: `a1 login [email protected] password`, replacing `[email protected]` with your email address, and `password` with any value you like during testing (see above for why the password is irrelevant).
 If you have already authorised your account with the proxy you should see a response starting with `a1 OK`; if not, this command should trigger a notification from the proxy about authorising your account.
+Note that POP and SMTP are different protocols, and while they can be tested in this way, they require different commands to be sent – see [this issue comment](https://github.com/simonrob/email-oauth2-proxy/issues/251#issuecomment-2133976839) for further details.
 
 If you are using a [secure local connection](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) the interaction with the remote email server is the same as above, but you will need to use a local debugging tool that supports encryption.
 The easiest approach here is to use [OpenSSL](https://www.openssl.org/): `openssl s_client -crlf -connect 127.0.0.1:1993`.

emailproxy.config

@@ -3,7 +3,11 @@ documentation = This is a sample Email OAuth 2.0 Proxy configuration file. Confi
 	[Server setup] and [Account setup] sections below. You may delete any servers or accounts that you do not intend to
 	use. Documentation is provided inline, with example setups for Gmail and Office 365 (though you will need to enter
 	your own desktop app API client credentials in the accounts section). Use the `Reload configuration file` menu
-	option or send a SIGHUP signal (or restart the proxy) to apply any changes.
+	option or send a SIGHUP signal (or quit the proxy before editing, then restart) to apply any changes.
+format = This file's format is documented at docs.python.org/library/configparser#supported-ini-file-structure. Values
+	that span multiple lines should be indented deeper than the first line of their key (as in this comment). Quoting
+	of values is not required. Documentation sections can be removed if needed (though it is advisable to leave these
+	in place for reference) - thw only required sections are the individual server and account items of your setup.
 warning = Do not commit changes to this file into a public repository (e.g., GitHub, etc). While the proxy encrypts the
 	OAuth 2.0 tokens it obtains and saves on your behalf, it cannot protect these against offline brute-force attacks.
 
@@ -143,6 +147,13 @@ documentation = Accounts are specified using your email address as the section h
 	in the example below) in order to allow the proxy to refresh its access token on your behalf. The proxy will still
 	work if this parameter is not included, but you will need to re-authenticate extremely often (about once per hour).
 
+	- The example Office 365 configuration entries below use an OAuth 2.0 scope that clearly specifies IMAP, POP and
+	SMTP permission. If you do not require one or more of these protocols, you may remove the relevant values to ensure
+	the access tokens obtained on your behalf are as precisely-targeted as possible. Conversely, it is also possible to
+	replace these specific scopes with the more generic `https://outlook.office365.com/.default`. Switching to a broader
+	scope value may also be needed if you are using a version of O365 delivered by a regional provider (e.g., 21Vianet).
+	See: https://github.com/simonrob/email-oauth2-proxy/issues/255 for more details and discussion.
+
 	- By default, new Entra (Azure AD) clients are accessible only within your own tenant. If you are registering a new
 	client to use with the proxy (and do not want to make it available outside your own organisation) you will need to
 	replace `common` with your tenant ID in the Office 365 `permission_url` and `token_url` values below. Alternatively,

emailproxy.py

@@ -6,7 +6,7 @@
 __author__ = 'Simon Robinson'
 __copyright__ = 'Copyright (c) 2024 Simon Robinson'
 __license__ = 'Apache 2.0'
-__version__ = '2024-05-25'  # ISO 8601 (YYYY-MM-DD)
+__version__ = '2024-06-28'  # ISO 8601 (YYYY-MM-DD)
 __package_version__ = '.'.join([str(int(i)) for i in __version__.split('-')])  # for pyproject.toml usage only
 
 import abc
@@ -322,7 +322,7 @@ def format_host_port(address):
         host, port, *_ = address
         with contextlib.suppress(ValueError):
             ip = ipaddress.ip_address(host)
-            host = '[%s]' % host if type(ip) is ipaddress.IPv6Address else host
+            host = '[%s]' % host if isinstance(ip, ipaddress.IPv6Address) else host
         return '%s:%d' % (host, port)
 
     @staticmethod
@@ -365,13 +365,13 @@ def _get_boto3_client(store_id):
         except ModuleNotFoundError:
             Log.error('Unable to load AWS SDK - please install the `boto3` module: `python -m pip install boto3`')
             return None, None
-        else:
-            # allow a profile to be chosen by prefixing the store_id - the separator used (`||`) will not be in an ARN
-            # or secret name (see: https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_CreateSecret.html)
-            split_id = store_id.split('||', maxsplit=1)
-            if '||' in store_id:
-                return split_id[1], boto3.session.Session(profile_name=split_id[0]).client('secretsmanager')
-            return store_id, boto3.client(service_name='secretsmanager')
+
+        # allow a profile to be chosen by prefixing the store_id - the separator used (`||`) will not be in an ARN
+        # or secret name (see: https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_CreateSecret.html)
+        split_id = store_id.split('||', maxsplit=1)
+        if '||' in store_id:
+            return split_id[1], boto3.session.Session(profile_name=split_id[0]).client('secretsmanager')
+        return store_id, boto3.client(service_name='secretsmanager')
 
     @staticmethod
     def _create_secret(aws_client, store_id):
@@ -824,7 +824,7 @@ def get_oauth2_credentials(username, password, reload_remote_accounts=True):
                             headers={
                                 'x5t#S256': base64.urlsafe_b64encode(jwt_certificate_fingerprint).decode('utf-8')
                             })
-                    except FileNotFoundError:
+                    except (FileNotFoundError, OSError):  # catch OSError due to GitHub issue 257 (quoted paths)
                         return (False, 'Unable to create credentials assertion for account %s - please check that the '
                                        '`jwt_certificate_path` and `jwt_key_path` values are correct' % username)
 
@@ -912,7 +912,7 @@ def get_oauth2_credentials(username, password, reload_remote_accounts=True):
 
         except OAuth2Helper.TokenRefreshError as e:
             # always clear access tokens - can easily request another via the refresh token (with no user interaction)
-            has_access_token = True if config.get(username, 'access_token', fallback=None) else False
+            has_access_token = bool(config.get(username, 'access_token', fallback=None))
             config.remove_option(username, 'access_token')
             config.remove_option(username, 'access_token_expiry')
 
@@ -986,6 +986,7 @@ def start_redirection_receiver_server(token_request):
                   Log.format_host_port((parsed_uri.hostname, parsed_port)))
 
         class LoggingWSGIRequestHandler(wsgiref.simple_server.WSGIRequestHandler):
+            # pylint: disable=arguments-differ
             def log_message(self, _format_string, *args):
                 Log.debug('Local server auth mode (%s): received authentication response' % Log.format_host_port(
                     (parsed_uri.hostname, parsed_port)), *args)
@@ -1150,22 +1151,22 @@ def get_service_account_authorisation_token(key_type, key_path_or_contents, oaut
             import requests
             import google.oauth2.service_account
             import google.auth.transport.requests
-        except ModuleNotFoundError:
-            raise Exception('Unable to load Google Auth SDK - please install the `requests` and `google-auth` modules: '
-                            '`python -m pip install requests google-auth`')
+        except ModuleNotFoundError as e:
+            raise ModuleNotFoundError('Unable to load Google Auth SDK - please install the `requests` and '
+                                      '`google-auth` modules: `python -m pip install requests google-auth`') from e
 
         if key_type == 'file':
             try:
-                with open(key_path_or_contents) as key_file:
+                with open(key_path_or_contents, mode='r', encoding='utf-8') as key_file:
                     service_account = json.load(key_file)
             except IOError as e:
-                raise FileNotFoundError('Unable to open service account key file %s for account %s',
+                raise FileNotFoundError('Unable to open service account key file %s for account %s' %
                                         (key_path_or_contents, username)) from e
         elif key_type == 'key':
             service_account = json.loads(key_path_or_contents)
         else:
-            raise Exception('Service account key type not specified for account %s - `client_id` must be set to '
-                            '`file` or `key`' % username)
+            raise KeyError('Service account key type not specified for account %s - `client_id` must be set to '
+                           '`file` or `key`' % username)
 
         credentials = google.oauth2.service_account.Credentials.from_service_account_info(service_account)
         credentials = credentials.with_scopes(oauth2_scope.split(' '))
@@ -2044,7 +2045,7 @@ def process_data(self, byte_data):
             if not re.search(' SASL-IR', updated_response, re.IGNORECASE):
                 updated_response = updated_response.replace(' AUTH=PLAIN', ' AUTH=PLAIN SASL-IR')
             updated_response = re.sub(' LOGINDISABLED', '', updated_response, count=1, flags=re.IGNORECASE)
-            byte_data = (b'%s\r\n' % updated_response.encode('utf-8'))
+            byte_data = b'%s\r\n' % updated_response.encode('utf-8')
 
         super().process_data(byte_data)
 
@@ -2842,9 +2843,9 @@ def create_config_menu(self):
         if len(self.proxies) <= 0:
             # note that we don't actually allow no servers when loading the config, so no need to generate a menu
             return items  # (avoids creating and then immediately regenerating the menu when servers are loaded)
-        else:
-            for server_type in ['IMAP', 'POP', 'SMTP']:
-                items.extend(App.get_config_menu_servers(self.proxies, server_type))
+
+        for server_type in ['IMAP', 'POP', 'SMTP']:
+            items.extend(App.get_config_menu_servers(self.proxies, server_type))
 
         config_accounts = AppConfig.accounts()
         items.append(pystray.MenuItem('Accounts (+ last authenticated activity):', None, enabled=False))
@@ -2977,7 +2978,7 @@ def create_authorisation_window(self, request):
         # pywebview 3.6+ moved window events to a separate namespace in a non-backwards-compatible way
         # noinspection PyDeprecation
         pywebview_version = pkg_resources.parse_version(pkg_resources.get_distribution('pywebview').version)
-        # the version zero check is due to a bug in the Ubuntu 22.04 python-pywebview package - see GitHub #242
+        # the version zero check is due to a bug in the Ubuntu 24.04 python-pywebview package - see GitHub #242
         # noinspection PyDeprecation
         if pkg_resources.parse_version('0') < pywebview_version < pkg_resources.parse_version('3.6'):
             # noinspection PyUnresolvedReferences
@@ -3174,10 +3175,10 @@ def macos_launchctl(command):
             output = subprocess.check_output(['/bin/launchctl', command, proxy_command], stderr=subprocess.STDOUT)
         except subprocess.CalledProcessError:
             return False
-        else:
-            if output and command != 'list':
-                return False  # load/unload gives no output unless unsuccessful (return code is always 0 regardless)
-            return True
+
+        if output and command != 'list':
+            return False  # load/unload gives no output unless unsuccessful (return code is always 0 regardless)
+        return True
 
     @staticmethod
     def started_at_login(_):