KTL-4174 Create a job to collect index requests from GH issues by zwiora · Pull Request #306 · JetBrains/klibs-io

Zofia Wiora (zwiora) · 2026-05-25T14:24:32Z

The previous naming conventions used for this feature (such as RequestIndexingService or ScraperType.MANUAL_REQUEST) were inconsistent and could be confused with distinct parts of the project, like IndexingRequestRepository. These occurrences were renamed to user requests now.
io.klibs.integration.maven.search.impl.BaseMavenSearchClient#getRemoteFileUrl now also processes untrusted data directly provided by users. That's why an URLBuilder was added to it.
The number of GitHub issues fetched in a single execution loop is limited. It should prevent potential API quota exhaustion in case anyone attempts to flood the service with spam issues.
The duplication verification logic has two stages:
1. UserRequestCheckService.findDuplicateIssueNumber filters out duplicate requests within the current batch;
2. UserRequestIndexingService.isAlreadyIndexedOrQueued checks if request is not already saved in the database.

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:20:38Z

+     * Updated only when a polling run completes without any server errors
+     */
+    @Column(name = "user_request_check_timestamp")
+    var userRequestCheckTimestamp: Instant


It is better to have a field called "log_type" or something similar, and to have two separate records in the DB for the Maven index update and the user request check.

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:26:46Z

+        val since = mavenCentralLogRepository.retrieveUserRequestCheckTimestamp()
+        val runStartedAt = Instant.now()
+
+        // Get batch of issues from GitHub


Please don't leave excessive comments. Code should be self-documenting. Comment should be only in non-obvious places and should answer question "why", not "how".
Fix in other places as well please.

I removed some comments, please let me know if you find any more of them that can be deleted

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:35:41Z

@@ -0,0 +1,336 @@
+package io.klibs.app.indexing


move the service to io.klibs.app.service.impl

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:36:39Z

+        }
+    }
+
+    internal data class ParsedRequest(


Let's move these data calsses to a separate files in io.klibs.app.dto package.

I moved ParsedRequest to io.klibs.integration.maven.dto, but I left ProcessedRequestInfo in io.klibs.app.dto. Please let me know, if that's ok

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:38:43Z

+        }
+    }
+
+    internal data class ParsedRequest(


It also could be named as MavenArtifactDto and lay in the io.klibs.integration.maven.dto

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:46:51Z

+     * Updates the timestamp in the database only if all issues were processed
+     * without server-side errors.
+     */
+    fun checkUserRequests() {


try to divide such big methods, the best if it fits one screen without scrolling.

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:47:48Z

+            return
+        }
+
+        val issuesToProcess = issuesBatch.issues.mapNotNull { issue ->


Could be shortened to val issuesToProcess = issuesBatch.issues.associateWith { issue -> convertToValidMavenArtifact(issue) } if the whole body moves to a separate method

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:51:59Z

+     *
+     * Returns null if the request is valid, or an error message if it is not
+     */
+    internal fun validateRequest(parsed: ParsedRequest): String? {


This method could be extracted to a seperate MavenArtifactUtils class

Dmitrii Krasnov (dkrasnoff) · 2026-06-02T09:55:56Z

@@ -17,22 +17,22 @@ import org.springframework.transaction.annotation.Transactional
 import org.springframework.web.server.ResponseStatusException

 @Service
-class RequestIndexingService(
+class UserRequestIndexingService(


Please move this service to io.klibs.app.service.impl package as well

Zofia Wiora (zwiora) requested review from Dmitrii Krasnov (dkrasnoff) and Nikita Vlaev (nikitavlaevjb) and removed request for Dmitrii Krasnov (dkrasnoff) May 25, 2026 15:10

Zofia Wiora (zwiora) force-pushed the feature/KTL-4062-add-index-my-package-endpoint branch 2 times, most recently from cebe075 to 8e094b1 Compare June 1, 2026 12:22

Dmitrii Krasnov (dkrasnoff) removed the request for review from Nikita Vlaev (nikitavlaevjb) June 2, 2026 09:30

Dmitrii Krasnov (dkrasnoff) reviewed Jun 2, 2026

View reviewed changes

Dmitrii Krasnov (dkrasnoff) requested changes Jun 2, 2026

View reviewed changes

Zofia Wiora (zwiora) added 3 commits June 4, 2026 13:43

KTL-4174 Create a job to collect index requests from GitHub

5a79341

KTL-4174 Rename Index Request to User Request

7eb1553

KTL-4174 Refine UserRequestCheckJob and address security threats

c778cf2

Zofia Wiora (zwiora) force-pushed the feature/KTL-4174-Create-a-job-to-collect-index-requests branch from 7c16089 to d51c715 Compare June 5, 2026 09:33

Zofia Wiora (zwiora) force-pushed the feature/KTL-4062-add-index-my-package-endpoint branch from 8e094b1 to c4033c7 Compare June 5, 2026 09:34

fixup! KTL-4174 Refine UserRequestCheckJob and address security threats

650588b

Zofia Wiora (zwiora) force-pushed the feature/KTL-4174-Create-a-job-to-collect-index-requests branch from d51c715 to 650588b Compare June 5, 2026 09:34

Zofia Wiora (zwiora) requested a review from Dmitrii Krasnov (dkrasnoff) June 5, 2026 09:44

Conversation

Zofia Wiora (zwiora) commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Zofia Wiora (zwiora) commented May 25, 2026 •

edited

Loading