Merge pull request #867 from zonemaster/fix-release-v2021.1.3

Merge fix-release-v2021.1.3 into master (Backend)

Changes

@@ -1,5 +1,23 @@
 Release history for Zonemaster component Zonemaster-Backend
 
+v7.0.0 2021-09-15 (public security release)
+ [Fixes]
+ - By design adding a API user (needed for the batch function) is limited to
+   connections over localhost. With a default GUI installation with reverse
+   proxy all connections are over localhost, which means that adding a API
+   user is publicly available if the GUI is publicly available. If you can
+   add API users, then you can start several large batch jobs which may
+   overload the Zonemaster system. (#838, #850)
+   - Makes RPCAPI use the real remote IP for verification to restore the
+     limitation that the API key can only be added from localhost.
+   - Disables RPCAPI method "add_api_user" by default.
+   - Adds configuration key for "backend_config.ini" to enable RPCAPI method
+     "add_api_user".
+   - Adds configuration key for "backend_config.ini" to disnable RPCAPI method
+     "add_batch_job".
+ - Prevents RPCAPI daemon to recreate workers when workers crashed to do error
+   in configuration file. This is a FreeBSD specific error. (#813, #862)
+
 v6.2.0 2021-05-28 (public release version)
 
  [Features]
@@ -204,7 +222,7 @@ v3.0.0 2019-01-27 (pre-release version)
    * All link references on Github now to zonemaster/zonemaster instead
      of old dotse/zonemaster #443
    * Added missing support for "filter" in "get_test_history" #446
-   * Add a note about the empty string, stating that it's 
+   * Add a note about the empty string, stating that it's
      allowed but deprecated. #413
    * Fixed invalid Zonemaster::Backend::Config call #472
    * Remove geolocation code #462
@@ -249,14 +267,14 @@ v2.0.2 2018-02-23
 v2.0.1 2018-01-12
     Natural Language support
      * Update Translator.pm Add support for Danish "da" in Backend. (#346)
-    
+
     Fixed
      * Workaround for "query of death" problem (#287, #325)
      * Partial fix of leakage of system path information (#334)
      * Fixed the issue that validate_syntax and start_domain_test functions do not correctly validate
        IPv4 and IPv6 addresses (#173, #328)
      * Fixed: Use of uninitialized value $config/$policy (#268, #329)
-     * Fixed incomplete stored data for unit test and fixed bug in 
+     * Fixed incomplete stored data for unit test and fixed bug in
        TestAgent.pm (#337, #342)
      * Fix config and start files (#336)
      * Fixed: ipv4 || ipv4 must be ipv4 || ipv6 (#319, #326)
@@ -327,10 +345,10 @@ v1.0.6 2016-10-11
 	Fixes #155 - Change de preflight test to block only on Basic00
 	Fixes #153 - Improve the batch API (Fixed and added bulk testing methods)
 v1.0.5 2015-12-17
-	Fixes #148 - Use iana_profile.josn instead of iana.json as source file for IANA tests 
+	Fixes #148 - Use iana_profile.josn instead of iana.json as source file for IANA tests
 	Fixes #141 - Database initialisation files (.sql) not updated with the new hash_id column
 	Fixes #138 - The Bacakend's generated JSON is locale dependant
-	Fixes #134 - Bug fix of the crontab job runner 
+	Fixes #134 - Bug fix of the crontab job runner
 	Fixes #127 - The Bakend Translator does not handle non scalar message parameters
 	Fixes #125 - Non numeric IDs for tests
 	Fixes #124 - modified all instances of .SE to IIS
@@ -355,7 +373,7 @@ v1.0.2 2015-05-11
 	Fixes #99 - Fixes #59
 	Fixes #98 - Further updates for Debian instructions 2
 	Fixes #97 - Debian instructions for the backend updated
-	Fixes #96 - Debian-compatible start script 
+	Fixes #96 - Debian-compatible start script
 	Fixes #93 - Make test more robust
 	Fixes #92 - Updates backend install for Debian
 	Fixes #91 - API documentation needs to be improved
@@ -427,7 +445,7 @@ v1.0.0 2014-12-11 Public beta release.
 	Fixes #129 - History should differentiate from delegated, undelegated and batch
 	Fixes #152 - Delay in start of the test (when the same page is used for testing a second domain)
 	Fixes #121 - Does not support Swedish language
-	Fixes #132 - Does not run for all broken domains (e.g. broken.dnssec.ee) 
+	Fixes #132 - Does not run for all broken domains (e.g. broken.dnssec.ee)
 	Fixes #139 - No line-feed in output from GUI
 	Fixes #127 - Does not support IDN 2.0 domains
 	Fixes #117 - Disable both IPv4 and IPv6 possible

Makefile.PL

@@ -33,6 +33,7 @@ requires
   'Try::Tiny'                   => 0.12,
   'Zonemaster::Engine'          => 4.002,
   'Zonemaster::LDNS'            => 2.002,
+  'Plack::Middleware::ReverseProxy' => 0,
   ;
 
 test_requires 'DBD::SQLite';

docs/Configuration.md

@@ -10,6 +10,25 @@ Each section in `backend_config.ini` is documented below.
 Restart the `zm-rpcapi` and `zm-testagent` daemons to load the changes
 made to the `backend_config.ini` file.
 
+## RPCAPI section
+
+Available keys: `enable_add_batch_job`, `enable_add_api_user`.
+
+### enable_add_batch_job
+
+Boolean value to enable the `enable_add_batch_job` methods of the API.
+
+Accepted values: `yes` (or `true`) or `no` (or `false`),
+default to `yes` (enabled).
+
+### enable_add_api_user
+
+Boolean value to enable the `add_api_user` method of the API.
+
+Accepted values: `yes` (or `true`) or `no` (or `false`),
+default to `no` (disabled).
+
+
 ## DB section
 
 Available keys : `engine`, `user`, `password`, `database_name`,
@@ -330,5 +349,3 @@ Otherwise a new test request is enqueued.
 [SQLITE.database_file]:               #database_file
 [Zonemaster-Engine share directory]:  https://github.com/zonemaster/zonemaster-engine/tree/master/share
 [Zonemaster::Engine::Profile]:        https://metacpan.org/pod/Zonemaster::Engine::Profile#PROFILE-PROPERTIES
-
-

docs/Installation.md

@@ -70,7 +70,7 @@ sudo yum -y install jq perl-Class-Method-Modifiers perl-Config-IniFiles perl-DBD
 Install dependencies not available from binary packages:
 
 ```sh
-sudo cpanm Daemon::Control JSON::Validator Log::Any Log::Any::Adapter::Dispatch Starman
+sudo cpanm Daemon::Control JSON::Validator Log::Any Log::Any::Adapter::Dispatch Starman Plack::Middleware::ReverseProxy
 ```
 
 Install Zonemaster::Backend:
@@ -268,7 +268,7 @@ sv_SE.utf8
 Install dependencies available from binary packages:
 
 ```sh
-sudo apt install jq libclass-method-modifiers-perl libconfig-inifiles-perl libdbd-sqlite3-perl libdbi-perl libfile-sharedir-perl libfile-slurp-perl libhtml-parser-perl libio-stringy-perl libjson-pp-perl libjson-rpc-perl liblog-any-adapter-dispatch-perl liblog-any-perl liblog-dispatch-perl libmoose-perl libparallel-forkmanager-perl libplack-perl libplack-middleware-debug-perl librole-tiny-perl librouter-simple-perl libstring-shellquote-perl libtest-nowarnings-perl libtry-tiny-perl starman
+sudo apt install jq libclass-method-modifiers-perl libconfig-inifiles-perl libdbd-sqlite3-perl libdbi-perl libfile-sharedir-perl libfile-slurp-perl libhtml-parser-perl libio-stringy-perl libjson-pp-perl libjson-rpc-perl liblog-any-adapter-dispatch-perl liblog-any-perl liblog-dispatch-perl libmoose-perl libparallel-forkmanager-perl libplack-perl libplack-middleware-debug-perl libplack-middleware-reverseproxy-perl librole-tiny-perl librouter-simple-perl libstring-shellquote-perl libtest-nowarnings-perl libtry-tiny-perl starman
 ```
 
 > **Note**: libio-stringy-perl is listed here even though it's not a direct
@@ -447,7 +447,7 @@ su -l
 Install dependencies available from binary packages:
 
 ```sh
-pkg install jq p5-Class-Method-Modifiers p5-Config-IniFiles p5-Daemon-Control p5-DBI p5-File-ShareDir p5-File-Slurp p5-HTML-Parser p5-JSON-PP p5-JSON-RPC p5-Moose p5-Parallel-ForkManager p5-Plack p5-Role-Tiny p5-Router-Simple p5-Starman p5-String-ShellQuote p5-DBD-SQLite p5-Log-Dispatch p5-Log-Any p5-Log-Any-Adapter-Dispatch p5-JSON-Validator p5-YAML-LibYAML p5-Test-NoWarnings
+pkg install jq p5-Class-Method-Modifiers p5-Config-IniFiles p5-Daemon-Control p5-DBI p5-File-ShareDir p5-File-Slurp p5-HTML-Parser p5-JSON-PP p5-JSON-RPC p5-Moose p5-Parallel-ForkManager p5-Plack p5-Plack-Middleware-ReverseProxy p5-Role-Tiny p5-Router-Simple p5-Starman p5-String-ShellQuote p5-DBD-SQLite p5-Log-Dispatch p5-Log-Any p5-Log-Any-Adapter-Dispatch p5-JSON-Validator p5-YAML-LibYAML p5-Test-NoWarnings
 ```
 <!-- JSON::Validator requires YAML::PP, but p5-JSON-Validator currently lacks a dependency on p5-YAML-LibYAML -->
 

lib/Zonemaster/Backend.pm

@@ -1,6 +1,6 @@
 package Zonemaster::Backend;
 
-our $VERSION = '6.2.0';
+our $VERSION = '7.0.0';
 
 use strict;
 use warnings;

lib/Zonemaster/Backend/Config.pm

@@ -112,7 +112,7 @@ sub parse {
 
     # Validate section names
     {
-        my %sections = map { $_ => 1 } ( 'DB', 'MYSQL', 'POSTGRESQL', 'SQLITE', 'LANGUAGE', 'PUBLIC PROFILES', 'PRIVATE PROFILES', 'ZONEMASTER' );
+        my %sections = map { $_ => 1 } ( 'DB', 'MYSQL', 'POSTGRESQL', 'SQLITE', 'LANGUAGE', 'PUBLIC PROFILES', 'PRIVATE PROFILES', 'ZONEMASTER', 'RPCAPI');
         for my $section ( $ini->Sections ) {
             if ( !exists $sections{$section} ) {
                 die "config: unrecognized section: $section\n";
@@ -128,6 +128,8 @@ sub parse {
     $obj->_set_ZONEMASTER_number_of_processes_for_batch_testing( '20' );
     $obj->_set_ZONEMASTER_lock_on_queue( '0' );
     $obj->_set_ZONEMASTER_age_reuse_previous_test( '600' );
+    $obj->_set_RPCAPI_enable_add_api_user( 'no' );
+    $obj->_set_RPCAPI_enable_add_batch_job( 'yes' );
 
     # Assign property values (part 1/2)
     if ( defined( my $value = $get_and_clear->( 'DB', 'engine' ) ) ) {
@@ -242,6 +244,12 @@ sub parse {
     if ( defined( my $value = $get_and_clear->( 'ZONEMASTER', 'age_reuse_previous_test' ) ) ) {
         $obj->_set_ZONEMASTER_age_reuse_previous_test( $value );
     }
+    if ( defined( my $value = $get_and_clear->( 'RPCAPI', 'enable_add_api_user' ) ) ) {
+        $obj->_set_RPCAPI_enable_add_api_user( $value );
+    }
+    if ( defined( my $value = $get_and_clear->( 'RPCAPI', 'enable_add_batch_job' ) ) ) {
+        $obj->_set_RPCAPI_enable_add_batch_job( $value );
+    }
 
     $obj->{_LANGUAGE_locale} = {};
     for my $locale_tag ( split /\s+/, $get_and_clear->( 'LANGUAGE', 'locale' ) || 'en_US' ) {
@@ -363,6 +371,24 @@ sub _set_DB_engine {
     return;
 }
 
+sub _set_RPCAPI_enable_add_api_user {
+    my ( $self, $value ) = @_;
+
+    $value = untaint_bool( $value ) // die "Invalid value for RPCAPI.enable_add_api_user: $value\n";
+    $self->{_RPCAPI_enable_add_api_user} = $value;
+    return;
+}
+
+sub _set_RPCAPI_enable_add_batch_job {
+    my ( $self, $value ) = @_;
+
+    $value = untaint_bool( $value ) // die "Invalid value for RPCAPI.enable_add_batch_job: $value\n";
+    $self->{_RPCAPI_enable_add_batch_job} = $value;
+    return;
+}
+
+
+
 =head2 DB_polling_interval
 
 Get the value of L<DB.polling_interval|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#polling_interval>.
@@ -461,6 +487,24 @@ Returns an integer.
 
 =cut
 
+=head2 RPCAPI_enable_add_api_user
+
+Get the value of
+L<RPCAPI.enable_add_api_user|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#enable_add_api_user>.
+
+Return 0 or 1
+
+=cut
+
+=head2 RPCAPI_enable_add_batch_job
+
+Get the value of
+L<RPCAPI.enable_add_batch_job|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#enable_add_batch_job>.
+
+Return 0 or 1
+
+=cut
+
 # Getters for the properties documented above
 sub DB_polling_interval                                 { return $_[0]->{_DB_polling_interval}; }
 sub MYSQL_host                                          { return $_[0]->{_MYSQL_host}; }
@@ -478,6 +522,8 @@ sub ZONEMASTER_lock_on_queue                            { return $_[0]->{_ZONEMA
 sub ZONEMASTER_number_of_processes_for_frontend_testing { return $_[0]->{_ZONEMASTER_number_of_processes_for_frontend_testing}; }
 sub ZONEMASTER_number_of_processes_for_batch_testing    { return $_[0]->{_ZONEMASTER_number_of_processes_for_batch_testing}; }
 sub ZONEMASTER_age_reuse_previous_test                  { return $_[0]->{_ZONEMASTER_age_reuse_previous_test}; }
+sub RPCAPI_enable_add_api_user                          { return $_[0]->{_RPCAPI_enable_add_api_user}; }
+sub RPCAPI_enable_add_batch_job                         { return $_[0]->{_RPCAPI_enable_add_batch_job}; }
 
 # Compile time generation of setters for the properties documented above
 UNITCHECK {

lib/Zonemaster/Backend/Validator.pm

@@ -12,12 +12,14 @@ use Readonly;
 
 our @EXPORT_OK = qw(
   untaint_engine_type
+  untaint_bool
 );
 
 our %EXPORT_TAGS = (
     untaint => [
         qw(
           untaint_engine_type
+          untaint_bool
           )
     ],
 );
@@ -38,6 +40,11 @@ Readonly my $RELAXED_DOMAIN_NAME_RE => qr/^[.]$|^.{2,254}$/;
 Readonly my $TEST_ID_RE             => qr/^[0-9a-f]{16}$/;
 Readonly my $USERNAME_RE            => qr/^[a-z0-9-.@]{1,50}$/i;
 
+# Boolean
+Readonly my $BOOL_TRUE_RE           => qr/^(true|yes)$/i;
+Readonly my $BOOL_FALSE_RE          => qr/^(false|no)$/i;
+Readonly my $BOOL_RE                => qr/^$BOOL_TRUE_RE|$BOOL_FALSE_RE$/i;
+
 sub joi {
     return JSON::Validator::Joi->new;
 }
@@ -136,6 +143,15 @@ sub untaint_engine_type {
     return _untaint_pat( $value , $ENGINE_TYPE_RE );
 }
 
+sub untaint_bool {
+    my ( $value ) = @_;
+
+    my $ret;
+    $ret = 1 if defined _untaint_pat( $value, $BOOL_TRUE_RE );
+    $ret = 0 if defined _untaint_pat( $value, $BOOL_FALSE_RE );
+    return $ret;
+}
+
 sub _untaint_pat {
     my ( $value, $pattern ) = @_;
 

script/zonemaster_backend_rpcapi.psgi

@@ -121,29 +121,33 @@ my $router = router {
         action => "get_test_history"
     };
 
-############ BATCH MODE ####################
-
-    connect "add_api_user" => {
+    connect "get_batch_job_result" => {
         handler => $handler,
-        action => "add_api_user"
+        action => "get_batch_job_result"
     };
+};
 
-    connect "add_batch_job" => {
+if ($config->RPCAPI_enable_add_api_user) {
+    $log->info('Enabling add_api_user method');
+    $router->connect("add_api_user", {
         handler => $handler,
-        action => "add_batch_job"
-    };
+        action => "add_api_user"
+    });
+}
 
-    connect "get_batch_job_result" => {
+if ($config->RPCAPI_enable_add_batch_job) {
+    $log->info('Enabling add_batch_job method');
+    $router->connect("add_batch_job", {
         handler => $handler,
-        action => "get_batch_job_result"
-    };
-};
+        action => "add_batch_job"
+    });
+}
 
 my $dispatch = JSON::RPC::Dispatch->new(
     router => $router,
 );
 
-sub {
+my $rpcapi_app = sub {
     my $env = shift;
     my $req = Plack::Request->new($env);
     my $res = {};
@@ -164,7 +168,7 @@ sub {
           $res->body( encode_json($errors) );
           $res->finalize;
         } else {
-            $dispatch->handle_psgi($env, $env->{REMOTE_HOST});
+            $dispatch->handle_psgi($env, $env->{REMOTE_ADDR});
         }
     } else {
         $res = Plack::Response->new(200);
@@ -181,3 +185,8 @@ sub {
 
     }
 };
+
+builder {
+    enable "Plack::Middleware::ReverseProxy";
+    mount "/" => $rpcapi_app;
+};

share/backend_config.ini

@@ -31,6 +31,13 @@ database_file = /var/lib/zonemaster/db.sqlite
 #
 #maximal_number_of_retries=3
 
+[RPCAPI]
+
+# Uncomment to enable API method "add_api_user"
+#enable_add_api_user = yes
+# Uncomment to disable API method "add_batch_job"
+#enable_add_batch_job = no
+
 [LANGUAGE]
 locale = da_DK en_US fi_FI fr_FR nb_NO sv_SE
 
@@ -40,4 +47,3 @@ locale = da_DK en_US fi_FI fr_FR nb_NO sv_SE
 
 [PRIVATE PROFILES]
 #example_profile_2=/example/directory/test2_profile.json
-

share/zm-rpcapi.lsb

@@ -27,7 +27,7 @@ STARMAN=`PATH="$PATH:/usr/local/bin" /usr/bin/which starman`
 . /lib/lsb/init-functions
 
 start () {
-    $STARMAN --user=$USER --group=$GROUP --error-log=$LOGFILE --pid=$PIDFILE --listen=$LISTENIP:$LISTENPORT --preload-app --daemonize $BINDIR/zonemaster_backend_rpcapi.psgi || exit 1
+    $STARMAN --listen=$LISTENIP:$LISTENPORT --preload-app --user=$USER --group=$GROUP --pid=$PIDFILE --error-log=$LOGFILE --daemonize $BINDIR/zonemaster_backend_rpcapi.psgi || exit 1
 }
 
 stop () {

share/zm_rpcapi-bsd

@@ -21,7 +21,7 @@ export ZONEMASTER_BACKEND_CONFIG_FILE="/usr/local/etc/zonemaster/backend_config.
 #export ZM_BACKEND_RPCAPI_LOGLEVEL='warning'  # Set this variable to override the default log level
 
 command="/usr/local/bin/starman"
-command_args="--daemonize --user=${zm_rpcapi_user} --group=${zm_rpcapi_group} --pid=${zm_rpcapi_pidfile} --error-log=${zm_rpcapi_logfile} --listen=${zm_rpcapi_listen} --app /usr/local/bin/zonemaster_backend_rpcapi.psgi"
+command_args="--listen=${zm_rpcapi_listen} --preload-app --user=${zm_rpcapi_user} --group=${zm_rpcapi_group} --pid=${zm_rpcapi_pidfile} --error-log=${zm_rpcapi_logfile} --daemonize /usr/local/bin/zonemaster_backend_rpcapi.psgi"
 pidfile="${zm_rpcapi_pidfile}"
 required_files="/usr/local/etc/zonemaster/backend_config.ini /usr/local/bin/zonemaster_backend_rpcapi.psgi"