Pipeline support

[proski]

This comment is here to reflect the current state of the PR, it is updated after every push

This is a rebased commit by @rhencke to support pipelines. It has been successfully tested both with pipelines and with traditional projects (Free Style, Matrix and Maven).

Most changes are already in the master branch. The remaining part is a commit to enable pipeline support in global configuration. That is necessary to make users understand that the support is incomplete and there are other ways to access Bitbucket Server from a pipeline. Also, a section about pipeline has been added to README.md.

[rhencke]

Thanks for taking this up (I am the original author of ea4a885). If there's any questions I can help answer feel free to ping me here or email me directly.

[proski]

Thanks for taking this up (I am the original author of ea4a885). If there's any questions I can help answer feel free to ping me here or email me directly.

Thank you, I've updated the description. Sorry for misattributing the original work!

I went through the code line-by-line, it's actually a very clean implementation. I made it even less intrusive by requiring Job to implement ParameterizedJob and Queue.Task. That way, I can cast it without fallbacks.

I also made changes to reduce interdependencies between classes. That way, I could split the commit into smaller parts.

The customized messages cannot even be configured in the pipeline, so I don't consider it a defect in this PR. Rather, we should find a way to post arbitrary messages to Stash from the pipeline, which would be a large separate task.

[proski]

I have fixed a major bug I introduced while applying to original patch manually (enabled projects were treated as disabled and vice versa). I've done some testing, everything appears to be working correctly. Now it's time for others to test.

[proski]

Changes to variable names and comments only.

[rhencke]

@proski Thank you - I appreciate the kind words. :)
Looking over everything this looks good. I noticed you've been making other improvements to the plugin as well in other PRs. We use this plug-in pretty heavily where I work, and there are actually quite a few things that could be done to improve the speed and functionality of this plugin while keeping with its spirit, but I've been holding off on working on it because there really hasn't been an active maintainer. If you I think that it would be worth it I'd be happy to file some issues and implement PR's to improve things as well.

One of the ideas is to revamp the way that this plug-in tracks build status - it currently traverses the comment chain to determine when things need to build and this ends up not being very efficient at scale - this could be switched to use the native build status API in Bitbucket Server. That would reduce the size of this codebase, make it much faster on more active repos, and also take advantage of build status API which currently requires adding the stash notifier plugin if you want to use it.

Another is adding native support for Bitbucket Server webhooks - this would eliminate polling completely and also instantly start builds when pull requests triggered them instead of having to heavily poll the server via the comments API.

If you think this would be something you be open to let me know, and I can dig in as well. :)

[rhencke]

(also also, if you like, Git maintains author and committer separately in commits, though they are normally the same - this is the use case it helps track. When you are rebasing you can do a git commit --amend --author='Robert Hencke <[email protected]>' and it should then show the author as me and the committer as you)

[proski]

@rhencke, I've set your authorship on the commits that are derived from your patch.

It's good to know that your patch is heavily used. My changes are mostly to avoid any fallbacks. For instance, I was uncomfortable about passing null to CredentialsMatchers.firstOrNull. Those fallbacks are not needed for traditional projects and pipelines, but I don't wan't to cause a security issue to some other kind of projects.

PRs are getting approved slowly here, so I don't feel good about asking you to do more changes that may wait for a long time.

I believe traversing the comments is unavoidable with the current design, as the plugin is looking for "test this please" and "NO test". We can tell the plugin to stop looking for PRs with the successful build status, but then "test this please" would stop working for the projects that have been tested successfully. It's not a typical case ("test this please" is more commonly used in case of build failure), but it can be confusing to the users.

Speaking of the build status, do you set it from the pipeline? Stash Notifier is a post-build publisher, it cannot be configured from the pipeline GUI. Are you doing it from the pipeline code? If we add Stash Notifier functionality to this plugin, we could configure it from the GUI.

I believe Bitbucket plugin supports webhooks, so it may be better just to migrate to it. But the last commit in the code was on December 6, 2017. https://github.com/jenkinsci/bitbucket-plugin

There is also Bitbucket pull request builder. It is being actively developed. It only claims to support Bitbucket.org in README, but I see a reference to Bitbucket server in the code.

[rhencke]

Hey @proski :)

It's good to know that your patch is heavily used. My changes are mostly to avoid any fallbacks. For instance, I was uncomfortable about passing null to CredentialsMatchers.firstOrNull. Those fallbacks are not needed for traditional projects and pipelines, but I don't wan't to cause a security issue to some other kind of projects.

That's fair - that sounds like a good idea. I'll be the first to admit the Jenkins API is not my strong suit, so please feel free to improve as you see fit. :)

PRs are getting approved slowly here, so I don't feel good about asking you to do more changes that may wait for a long time.

That's okay! The fact they're getting approved and merged at all is enough for me. :)

I believe traversing the comments is unavoidable with the current design, as the plugin is looking for "test this please" and "NO test". We can tell the plugin to stop looking for PRs with the successful build status, but then "test this please" would stop working for the projects that have been tested successfully. It's not a typical case ("test this please" is more commonly used in case of build failure), but it can be confusing to the users.

I agree, this is true for polling style, unfortunately. But, if web hook support was implemented (would be happy to work on this), then for those who enabled it, Bitbucket Server will just POST the comments to Jenkins as they happened, and all the build would have to do is look at each comment as it arrived. (basically, it'd be a model akin to /git/notifyCommit from the Git plugin, but specific to receiving events for Bitbucket).

Speaking of the build status, do you set it from the pipeline? Stash Notifier is a post-build publisher, it cannot be configured from the pipeline GUI. Are you doing it from the pipeline code? If we add Stash Notifier functionality to this plugin, we could configure it from the GUI.

Yes - exactly. I don't have the code in front of me. It's not pretty, but it's something to the effect of (pseudocode):

StashNotifier.notify('INPROGRESS')
def error
try {
    /* build */
} catch(err) {
	error = err
	throw err
} finally {
	if (error) {
		try {
			StashNotifier.notify('FAILED', error)
		catch (err) {
			error.addSuppressed(err)
		}
	} else {
		StashNotifier.notify('SUCCESS')
	}
}

...which I would love to delete and just configure from the pipeline GUI instead. :)

I believe Bitbucket plugin supports webhooks, so it may be better just to migrate to it. But the last commit in the code was on December 6, 2017. jenkinsci/bitbucket-plugin

Unfortunately, yes. I can't speak for others, but at least in my environment we end up having to install several plugins together (this one, Stash notifier, etc) to stitch together a solution. But it's honestly been a while since I went and looked at what else was available, and tried the latest versions of everything - maybe there's better options now for where some of this work could be focused, or maybe there's something out there that'd be a good swap-out replacement.

There is also Bitbucket pull request builder. It is being actively developed. It only claims to support Bitbucket.org in README, but I see a reference to Bitbucket server in the code.

I'll check that out - thank you! :)

[proski]

@rhencke, thank you for your detailed reply. There are some recipes in the StashNotifier README

Scripting is a mixed bag. It's a barrier for entry, but it gives a lot of power if it's needed. Scripting works with multiple components just fine, but GUI doesn't work well if there are multiple components with their own configurations. Either there should be one plugin dealing with Stash-Jenkins interactions, or there should be scriptable low-level plugins and a high-level GUI capable plugin that depends on them.

I also plan to test every plugin with "Bitbucket" in its name as long as it's maintained or at least has an active community.

Back to this PR, my only unresolved concern so far is whether the environment variables are provided to all jobs in the pipeline. I want to have a way to identify the top level project for every Run. That's not your code, the environment variables were handled as parameters until recently.

[MyroslavSeliverstov]

Thank you for this PR. It's really valued option for this plugin
I build artifact from your repo from pipeline-support branch. And deploy plugin in my Jenkins.

Looks like I faced two issues.
I had two PR in bitbucket.
I expect that job will be triggered two times. And each build will have just one badge about PR.
Issues:

1. Job runed just one time. Build marked with two badges (two PRs). In Bitbucket both PRs receive comment about this build, but just one has comment that build - success.
   
   
   

2. Job run two times. But first build has two badges as well. Comments in PR time to time like it's described above, and sometimes as expected (comment that - build success).
   

Jenkins ver. 2.138.1

Thanks in advance.

[proski]

@MyroslavSeliverstov, I'm sorry, I don't understand either issue. Even the GUI in the screenshots looks unfamiliar to me, except the Stash comments.

I don't think those issues can be fixed in this PR. This PR allows Stash Pull Request Builder to operate on pipelines by using less specific interfaces. There are no other changes here. In particulate, there are no changes to how comments are posted.

There is one change that could actually fix the disappearing job. That's the change from isDisabled to isBuildable. With the current master branch, a job can be disabled but not buildable. In that case, the build would never complete, and the comment about the build result would never get posted. This PR should fix such possibility.

Let's get back to those issues once this PR gets merged. Jenkins logs should help figure out what is happening.

[proski]

I've added commits from PRs #67, #72 and #73 to provide full pipeline support. This PR is marked WIP anyway, we can always merge other PRs and rebase this one to exclude already merged changes.

I have found a document how to write a pipeline-compatible plugin: https://jenkins.io/doc/developer/plugin-development/pipeline-integration/

I plan to go through it carefully, it looks like some issues still need to be addressed, e.g. changing BuildListener to TaskListener.

[jakub-bochenski]

I've added commits from PRs #67, #72 and #73 to provide full pipeline support.

Just FTR, and as mentioned in #72 I do like the approach of having several self-contained PRs

[proski]

I've added commits from PRs #67, #72 and #73 to provide full pipeline support.

Just FTR, and as mentioned in #72 I do like the approach of having several self-contained PRs

I've removed the commits that come from other PRs.

For the purpose of testing, I have a branch in my fork that integrates all those PRs and some other work (mostly warning fixes): https://github.com/proski/stash-pullrequest-builder-plugin/tree/merge-all

I'm open to splitting this PR even further (e.g. taking my environment commit elsewhere) if it helps with merging.

[jakub-bochenski]

I also plan to test every plugin with "Bitbucket" in its name as long as it's maintained or at least has an active community.

I've spent some time trying to get https://wiki.jenkins.io/display/JENKINS/Bitbucket+Branch+Source+Plugin to work. I'm still seeing some issues, probably because we are stuck using a 4.x version of Stash/BB

You can configure it with a dsl-like the one seen below. The code is not pretty with all the configuration blocks, but more importantly:

• it handles both regular branch builds and PRs (bugged, but still)
• it sends notifications (could be improved)
• it's configured centrally, no need to repeat the same trigger config in each Jenkinsfile
• if I'm not mistaken you can also add other common traits at this level like timestamps (?)

I appreciate all the work done here, and I'm sad to be making a fuss here.
Yet I think before proceeding with this we need to figure out why and if it makes sense to duplicate the functionality already present in another plugin.

It seems to me that the branch source/organization approach is "the way to go" with Pipelines. This is only a gut feeling though. I would be happy to see arguments that show otherwise, or use cases that are not supposed to be/can't be handled by the branch source approach.

Even if this plugin is not going to support pipelines I think it's still going to be useful for many years to come. I don't expect the old-style jobs to disappear any time soon.

    organizations {
        bitbucket {
            repoOwner('acme')
            credentialsId('secret')
            serverUrl('https://stash.acme.com/')
            traits {
                sourceRegexFilter {
                    regex('.*')
                }
                headWildcardFilter {
                    includes("*")
                    excludes("stuff")
                }
            }
        }
    }

    // discover Branches (workaround due to JENKINS-46202)
    configure { node ->
        // node represents <jenkins.branch.OrganizationFolder>
        final traits = node / 'navigators' / 'com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator' / 'traits'
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
            strategyId(1) // Exclude branches that are also filed as PRs
        }
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait' {
            strategyId(1)
            trust(class: 'com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait$TrustEveryone')
        }
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
            strategyId(1)
        }
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.SSHCheckoutTrait' {
            credentialsId('key')
        }
    }

[atzoum]

How far is this PR from being merged? Any possibility to be included in the next version?

[jakub-bochenski]

1 of the commits have already been merged, 2 others have been approved by me and are waiting for review by @jimklimov

[HB-2012]

@HB-2012: thank you for reporting the issue! The pipeline support was disabled because the code is not ready to support it. It's better not to show the trigger in GUI if it won't work.

Why did you need to revert to an older version? Did it resolve any issue for you? What were the versions?

The reason we had to revert is because we were on an older version that didn't have the check. Some teams are using this plugin with additional (Power)shell scripts to make it all function somewhat decently. We run quite a lot of plugins and workflows so I'm unsure of the specific use cases, I just know that our users wanted to downgrade after this plugin no longer worked inside a pipeline.

Looking forward to the release! :)

[proski]

The reason we had to revert is because we were on an older version that didn't have the check. Some teams are using this plugin with additional (Power)shell scripts to make it all function somewhat decently. We run quite a lot of plugins and workflows so I'm unsure of the specific use cases, I just know that our users wanted to downgrade after this plugin no longer worked inside a pipeline.

@HB-2012: Thank you for the reply. It's actually good that pipeline support was the only reason to revert to an older version. It means that merging this PR would likely fix your setup.

I don't understand how it's possible to use a plugin without pipeline support with pipelines. The only explanation I have is that the Jenkins Pipeline code has a compatibility mode for plugins that only support freestyle projects. Perhaps that mode was activated with the scripts you mentioned.

[proski]

This PR is down to one commit now. Getting really close!

[proski]

Added a commit to rename the descriptor class, also submitted separately as #116.