Metrics based custom scheduler plugin #24439

jp-sivaprasad · 2025-01-27T12:48:17Z

Description

Metrics based custom scheduler plugin

Motivation and Context

Metrics based custom scheduler plugin. Schedules cluster with lowest number of queries (running + queued queries).
Support for implementation of third party custom schedulers to be used by Presto router.

Impact

Test Plan

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Metrics based custom scheduler plugin Changes
* Add support for Metrics based custom scheduler plugin

tdcmeehan · 2025-01-27T17:25:52Z

presto-router-custom-scheduler/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+


You need to add the parent tag to make this module consistent with the others:

<parent> <groupId>com.facebook.presto</groupId> <artifactId>presto-root</artifactId> <version>0.291-SNAPSHOT</version> </parent>

steveburnett

LGTM! (docs)

tdcmeehan · 2025-01-30T16:19:13Z

@auden-woolfson can you PTAL?

auden-woolfson · 2025-02-03T17:20:25Z

presto-router/etc/router-config/router-scheduler.properties

@@ -0,0 +1 @@
+router-scheduler.name=metricsBased


Would it be better to set this as a default value in the code, and leave this file empty?

The property name (router-scheduler.name) is added in the property file so that clients implementing a new scheduler plugin can just update the value to the name of the new scheduler. Also since the Metric based scheduler is just a sample implementation, it may not be good to set it as the default value in code.

Ok... maybe it would make more sense to add a new optional property to the existing router-config.json file ("custom-scheduler-name" or something similar) instead of creating an entirely new file for this one property. This way would also allow for customers to use multiple different custom schedulers in the future.

There can be more properties in future for new schedulers, like cpu/memory thresholds related to specific schedulers, which can use this property file

steveburnett · 2025-02-03T17:31:25Z

Thanks for the release note!

New release note guidelines. Please remove the manual PR link in the following format from the release note entries for this PR.

:pr:`12345`

I have updated the Release Notes Guidelines to remove the examples of manually adding the PR link.

aaneja · 2025-02-20T10:46:17Z

presto-router-custom-scheduler/pom.xml

+    <artifactId>scheduler</artifactId>
+    <version>0.291-SNAPSHOT</version>
+    <packaging>jar</packaging>
+    <name>scheduler</name>


Please use the full name

Suggested change

<name>scheduler</name>

<name>presto-router-custom-scheduler</name>

aaneja · 2025-02-20T10:48:45Z

presto-router-custom-scheduler/pom.xml

+    <!-- Build configurations -->
+    <build>
+        <plugins>
+            <plugin>


We do not support JDK 11 as target bytecode. Code must compile at JDK 8 level right now

aaneja · 2025-02-20T10:51:09Z

presto-router-custom-scheduler/pom.xml

+        <dependency>
+            <groupId>org.testng</groupId>
+            <artifactId>testng</artifactId>
+            <version>7.10.2</version>


We cannot use 7.10.2 here. We upgrade Presto's TestNG version in the root pom. 7.10.2 requires a JDK version higher than what we support at the moment

aaneja · 2025-02-20T10:53:54Z

presto-router-custom-scheduler/pom.xml

+        <version>0.291-SNAPSHOT</version>
+    </parent>
+
+    <groupId>com.facebook.presto.router</groupId>


You need to add a reference in the presto-root pom.xml to get root level dependencies, i.e add

<module>presto-router-custom-scheduler</module>

in the <modules> xml

aaneja · 2025-02-20T10:59:24Z

presto-router/src/main/java/com/facebook/presto/router/scheduler/Scheduler.java

+    {
+        return Optional.empty();
+    };
+    /**


nit: Add a newline above this

aaneja · 2025-02-20T11:02:02Z

...m-scheduler/src/main/java/com/facebook/presto/router/scheduler/CustomSchedulerConstants.java

+ */
+package com.facebook.presto.router.scheduler;
+
+public class CustomSchedulerConstants


Class can be an enum instead

aaneja · 2025-02-20T11:02:33Z

presto-router-custom-scheduler/README.md

@@ -0,0 +1,17 @@
+# Presto Router Custom Scheduler Plugin


You neeed to fix line endings with double space. This does not render correctly

aaneja · 2025-02-20T11:03:12Z

presto-router/src/main/java/com/facebook/presto/router/scheduler/CustomPluginScheduler.java

+public class CustomPluginScheduler
+        implements Scheduler
+{
+    private SchedulerManager schedulerManager;


This can be made private final

aaneja · 2025-02-20T11:06:31Z

presto-router/src/main/java/com/facebook/presto/router/cluster/ClusterManager.java

@@ -105,4 +150,14 @@ private void initializeServerWeights()
            }
        });
    }
+
+    public ConcurrentHashMap<URI, RemoteClusterInfo> getRemoteClusterInfos()


These are unused

aaneja · 2025-02-20T11:15:06Z

presto-router/src/main/java/com/facebook/presto/router/cluster/ClusterManager.java

    {
-        RouterSpec routerSpec = parseRouterConfig(config)
-                .orElseThrow(() -> new PrestoException(CONFIGURATION_INVALID, "Failed to load router config"));
+        this.routerConfig = config;


I see some duplicate of code from #24449. Can you coordinate with Auden and decide which PR goes first

jp-sivaprasad self-assigned this Jan 27, 2025

jp-sivaprasad requested a review from a team as a code owner January 27, 2025 12:48

jp-sivaprasad requested a review from presto-oss January 27, 2025 12:48

prestodb-ci added the from:IBM PR from IBM label Jan 27, 2025

prestodb-ci requested review from a team, pramodsatya and nishithakbhaskaran and removed request for a team January 27, 2025 12:48

jp-sivaprasad requested a review from tdcmeehan January 27, 2025 12:50

jp-sivaprasad force-pushed the presto-router-metrics-based-custom-scheduler-plugin branch from 4e47db2 to 83cab5c Compare January 27, 2025 13:04

tdcmeehan requested changes Jan 27, 2025

View reviewed changes

steveburnett previously approved these changes Jan 27, 2025

View reviewed changes

jp-sivaprasad dismissed steveburnett’s stale review via 745c65b January 28, 2025 22:01

jp-sivaprasad force-pushed the presto-router-metrics-based-custom-scheduler-plugin branch from 83cab5c to 745c65b Compare January 28, 2025 22:01

jp-sivaprasad requested a review from tdcmeehan January 28, 2025 22:02

Metrics based custom scheduler plugin

f1fc17d

jp-sivaprasad force-pushed the presto-router-metrics-based-custom-scheduler-plugin branch from 745c65b to f1fc17d Compare January 29, 2025 00:30

tdcmeehan requested a review from auden-woolfson January 30, 2025 16:19

ethanyzhang added the presto-router label Jan 31, 2025

auden-woolfson reviewed Feb 3, 2025

View reviewed changes

aaneja requested changes Feb 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metrics based custom scheduler plugin #24439

Metrics based custom scheduler plugin #24439

jp-sivaprasad commented Jan 27, 2025 •

edited

Loading

tdcmeehan Jan 27, 2025

steveburnett left a comment

tdcmeehan commented Jan 30, 2025

auden-woolfson Feb 3, 2025

jp-sivaprasad Feb 3, 2025

auden-woolfson Feb 3, 2025

jp-sivaprasad Feb 4, 2025

steveburnett commented Feb 3, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

aaneja Feb 20, 2025

	<name>scheduler</name>
	<name>presto-router-custom-scheduler</name>

		@@ -0,0 +1 @@
		router-scheduler.name=metricsBased

Metrics based custom scheduler plugin #24439

Are you sure you want to change the base?

Metrics based custom scheduler plugin #24439

Conversation

jp-sivaprasad commented Jan 27, 2025 • edited Loading

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

tdcmeehan commented Jan 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett commented Feb 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jp-sivaprasad commented Jan 27, 2025 •

edited

Loading