Salesforce DX - Jenkins Shared Library

Why?
Prerequisites
Pipelines
- sfdxBuildPipeline
- sfdxDeployPipeline
Steps
Multiple Orgs
Org Bean
Running Library Tests Locally

Why?

The two aims of this library are, for Salesforce DX Jenkins builds:

To avoid the duplication of 150+ lines of Jenkinsfile logic across builds (at ClaimVantage we have dozens of builds) so a reliable pattern can be applied and maintained.

This is accomplished by providing custom pipeline steps that hide some of the detail. A default pipeline that sits on top of these steps is also provided and is recommended for projects that fit its pattern.
To make the process of testing against various org configurations - e.g. person Accounts turned on or Platform Encryption turned on - simple, and not require additional builds to be setup.

When multiple Scratch Org Definition File files are provided that match a regular expression, parallel builds are done using scratch orgs created from the files.

The resulting Jenkinsfile can be as simple as:

#!groovy
@Library('sfdx-jenkins-shared-library')_

sfdxBuildPipeline()

Note the required _ for this case.

For some background information including how to hook up this library, see e.g. Extending your Pipeline with Shared Libraries, Global Functions and External Code. Libraries are pulled directly from Git for each new build, so setup is simple.

Prerequisites

Unix Only

Makes use of shell scripting so only Unix platforms are supported.

Jenkins

Use a recent “Long Term Support” (LTS) version of Jenkins. Add these:

Credentials Binding Plugin
Workspace Cleanup plugin
SSH Agent
- Only required when the retrieveExternals or processHelp step is used

Tools

Requires Salesforce SFDX CLI to be installed where Jenkins is running (and also a Git client). If Git externals are being used, also requires develersrl/git-externals.

This library has only been tested with GitHub.

Jenkins Environment Variables

These must be set up for all the stages to work.

Name	Description	Example
CONFLUENCE_CREDENTIAL_ID^[1]	Confluence username/password credentials stored in Jenkins in “Credentials” under this name. Only used by the ClaimVantage proprietary processHelp step.	to-confluence
GITHUB_CREDENTIAL_ID^[2]	A GitHub generated private key and username stored in Jenkins in “Credentials” under this name.	to-github

[1] Used to extract help pages from Confluence.

[2] Used to retrieve Git externals.

Dev Hub Authentication

This library connects to the default Dev Hub configured on the build agent by using a JSON Web Token (JWT). See Authorize an Org Using the JWT-Based Flow for how to set that up.

Use the Client Id (the Consumer Key on the Connected App), the JWT Key file (private key that signed the certificate configured on Connected App), from the JWT setup process in the following steps. Also a user must be setup in the Dev Hub for Jenkins e.g. jenkins@acdxgs0hub.org, and that username is also needed in the following steps.

The steps are:

Log into the build agent as the user that Jenkins run as (usually a user named Jenkins).
Save the JWT Key File in a folder that won’t get deleted by Jenkins builds e.g. /Users/jenkins/JWT/server.key. The server.key file content should look something like this:
```
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
```

Manually authenticate access to the Dev Hub using the command below:

sfdx force:auth:jwt:grant \
--clientid 04580y4051234051 \
--jwtkeyfile /Users/jenkins/JWT/server.key \
--username jenkins@acdxgs0hub.org \
--setdefaultdevhubusername \
--setalias my-hub-org

This authentication will stay in place until the certificate created as part of the setup expires.

Pipelines

sfdxBuildPipeline

This is a ready-made pipeline - recommended that you start with this - that runs these stages using both the steps listed in the Steps section below and standard steps:

stage("help") {...}                 // Only runs if help or helps beans are defined         
stage("checkout") {...}
stage("after checkout") {...}       // Only runs if afterCheckoutStage closure is defined
withOrgsInParallel() {
    stage("org create") {...}
    try {
        stage("org after create stage") {...}   // Only runs if afterOrgCreateStage closure is defined
        stage("org install") {...}              // Only runs if package or packages beans are defined
        stage("org before push") {...}          // Only runs if beforePushStage closure is defined
        stage("org push") {...}
        stage("org install after push") {...}   // Only runs if packageAfterPushStage or packagesAfterPushStage beans are defined
        stage("org before test") {...}          // Only runs if beforeTestStage closure is defined
        stage("org test") {...}
        stage("org after test") {...}           // Only runs if afterTestStage closure is defined
    } finally {
        stage("org delete") {...}
    }
}
stage("publish")  {...}
stage("clean") {...}
stage("finalStage") {...}           // Only runs if finalStage closure is defined

To use it, your Jenkinsfile should look like this (and you will need Scratch Org Definition File files that match the regular expression):

#!groovy
@Library('sfdx-jenkins-shared-library')
import com.claimvantage.sjsl.Help
import com.claimvantage.sjsl.Package

sfdxBuildPipeline(
    glob: 'config/project-scratch-def.*.json',
    help: new Help('cx', '33226968', 'extras-help'),
    packages: [
        new Package('cve', env.'cve.package.password.v12'),
        new Package('cvab', env.'cvab.package.password.v12')
    ]
)

Edit the Help and Package details to reflect the specific project.

You can use strings instead of IDs in sfdx-project.json to install packages, but if you’re comfortable using packaging IDs and prefer to not use aliases you can set IDs as input. The first parameter of the Package constructor may be an alias for the Package or the Package Id. See Package Alias for details.

To build a package that has multiple configurations that require additional components or data to be setup (before the tests are run):

#!groovy
@Library('sfdx-jenkins-shared-library')
import com.claimvantage.sjsl.Help
import com.claimvantage.sjsl.Package

sfdxBuildPipeline(

    glob: 'config/project-scratch-def*.json',

    help: new Help('ch', '22315079', 'claims-help'),

    beforeTestStage: { org ->
        echo "${org.name} before test stage"
        switch (org.name) {
            case 'content-notes':
                echo "${org.name} deploying extra components"
                shWithResult "sfdx force:source:deploy --sourcepath ${env.WORKSPACE}/config-components/content-notes --json --targetusername ${org.username}"
                break
            case 'accommodations':
                echo "${org.name} installing Accommodations"
                installPackage(org: org, package: new Package('cvawa', env.'cvawa.package.password.v12'))
                break
            case 'platform-encryption':
                echo "${org.name} setting up encryption"
                sh "sfdx force:user:permset:assign --permsetname Encryption --targetusername ${org.username}"
                sh "sfdx force:data:record:create --sobjecttype TenantSecret --values 'Description=Test' --targetusername ${org.username}"
                shWithResult "echo 'cve.SetupController.updateDefaultFieldEncryptedFlags(true);' | sfdx force:apex:execute --json --targetusername ${org.username}"
                break
        }
    },

    afterTestStage: { org ->
        sh "npm install"
        sh "./node_modules/karma/bin/karma start karma.conf.js"
    }
)

The named values available are:

afterCheckoutStage

An (optional) closure that is executed immediately after the checkout stage. See the Org Bean section below.

This example executes lint validation against a specific folder, which could prevent scratch org creation if it fails.
```
afterCheckoutStage: {
    sh "sfdx force:lightning:lint ./path/to/lightning/components/"
}
```
afterOrgCreateStage

An (optional) closure that is executed immediately after the org create stage. The org is passed in to this. See the Org Bean section below.

This is good point to perform operations before the possible packages installations.

For example, if you need to perform any operation on the new scratch org before any package is installed:
```
afterOrgCreateStage: { org ->
    sh "./setupOrg.sh ${org.username}"
}
```
beforePushStage

An (optional) closure that is executed immediately before the push stage. The org is passed in to this. See the Org Bean section below.

This is good point to insert extra content into the source tree, but that content has to apply to all org configurations as a common copy of the source tree is used.

See example below on how to set sfdx to use REST deployment instead of SOAP - really useful to avoid size limitation on static resource, for instance: ```groovy beforePushStage: { org -> // change the deployment from SOAP to REST to avoid size limitation sh ‘sfdx force:config:set restDeploy=true’ }
beforeTestStage

An (optional) closure that is executed immediately before the test stage. The org is passed in to this. See the Org Bean section below.

This example executes some Apex code included in the pushed code (in this case setting up a role):
```
beforeTestStage: { org ->
    sh "echo 'cveep.UserBuilder.ensureRole();' | sfdx force:apex:execute --targetusername ${org.username}"
}
```
cron

Optional. A map of branch name to cron expression. This allows a branch (or branches) to be built regularly, in addition to when changes are made in Git.

This example builds the master branch every day between midnight and 5:59 AM:
```
cron: ['master': 'H H(0-5) * * *']
```
daysToKeepPerBranch

Optional (defaults to 7 days). Defines how long builds are kept in Jenkins. This example keeps the master branch for 7 days.
```
daysToKeepPerBranch: ['master': 7]
```
notificationChannel

Optional. Defines the Slack channel notification about the build’s progress. Two messages are going to be sent if configured: One at the beginning and one at the end.
```
notificationChannel: '#tech-builds'
```
Note: You will need to have installed the following plugins:
- https://plugins.jenkins.io/build-user-vars-plugin/
- https://plugins.jenkins.io/slack/
glob

The matching pattern (see Ant style pattern for details - * matches zero or more characters, ? matches one character) used to find the Scratch Org Definition File files. Each matched file results in a separate parallel build. There are two naming conventions in place that can be used to define the scratch org names:
- config/project-scratch-def.*.json (default)
- config/project*-scratch-def.json (this is the same pattern used by the SFDX plugin in VS code)
This assumes that an extra part will be inserted into the file names and that part is used as a name for the parallel work. E.g: project-scratch-def.content-notes.json or project-content-notes-scratch-def.json

There are two ways to specify the matching pattern:
- to use the same matching pattern for all branches, just supply the matching pattern as a string
- to vary the matching pattern by branch, specify a map of branch names to matching patterns
A particularly useful map to use is this one, that identifies all org configurations for the master branch, but only one org configuration for other branches:
```
// Evaluate so a serializable string is used in the pipeline (the withDefault object is not serializable)
// See https://issues.jenkins-ci.org/browse/JENKINS-38186
glob: ['master': 'config/project-scratch-def*.json'].withDefault{'config/project-scratch-def.json'}[env.BRANCH_NAME],
```
and so reduces the number of Scratch Orgs consumed when multiple branches are being worked on at the same time.
help (or helps)

Disclaimer: Do not include this as it contains dependencies on private ClaimVantage repositories that we don’t plan to release to the community.

Reference a simple bean object (or an array of those objects) that holds the values needed to extract, process, and commit the help. When left out, no help processing is done.
keepOrg

Optional. When set to true, the scratch orgs are not deleted and instead login credentials are output via an echo in the stage. This allows the orgs to be examined after the build to e.g. reproduce test failures manually. The scratch orgs can then be manually deleted via the “Active Scratch Orgs” tab of the Dev Hub, or just left to expire.
keepWs

Optional. When set to true, the workspace is not deleted. This allows data such as raw test test result XML files to be examined after the build.
skipApexTests

Optional (default to false). When set to true, it skips apex test execution. This allows a build with no apex classes on it.
apexTestsTimeoutMinutes

Optional. Minutes that it should wait for test results.
apexTestsUsePooling

Optional. If the tests are executed using pooling or synchronously.
package (or packages)

Reference a simple bean object (or an array of those objects) that holds the values needed to install existing managed package versions BEFORE pushing the code. When left out, no package installation is done.
packageAfterPushStage (or packagesAfterPushStage)

Works exactly like the package (or packages), however it’s executed just AFTER pushing the code. When left out, no package installation is done.
stagger

Optional (defaults to 15 seconds). This is the number of seconds to delay before the next parallel set of steps is started. The aim is to smooth out the load a little both on the Jenkins machine and at the Salesforce side by staggering the the execution of the parallel logic.

sfdxDeployPipeline

This is a ready-made pipeline - recommended that you start with this - that runs these stages using both the steps listed in the Steps section below and standard steps:

stage("Checkout") {...}     
stage("Authenticate to org") {...}
stage("Install packages") {...}                // Only runs if packages beans are defined
stage("Install Unlocked Packages") {...}       // Only runs if unlockedPackages beans are defined
stage("Install unpackaged code") {...}         // Only runs if unpackagedSourcePath is defined
stage("Logout org")  {...}
stage("Clean") {...}

You can use it on pipeline definition like Jenkinsfile-deploy:

@Library('sfdx-jenkins-shared-library')
import com.claimvantage.sjsl.Package

sfdxDeployPipeline(
    sfdxUrlCredentialId: 'jeferson-winter21-sfdxurl',
    packages: [
        new Package('cve v19', env.'cve.package.password.v19')
    ],
    unlockedPackages: [
         new Package('dreamhouse v1', env.'dreamhous.password.v1')
    ],
    unpackagedSourcePath: 'force-app'
)

Note: Recommend using one build per environment, and using Git Parameter plugin to define the git tag to be deployed with very little effort - allowing the selection of git tag as parameter.

The named values available are:

sfdxUrlCredentialId

Required. The id of a credential stored on your Jenkins instance at Credential Storage (more info about Using credentials in Jenkins). The file must be in the format defined by SFDX auth URL (used by force:auth:sfdxurl:store): “force://@" or "force://::@".
packages

Optional. Reference an array of a simple bean object that holds the values needed to install managed package versions. This happens BEFORE unlocked packages are installed or unpackage source are deployed and only if the version to be installed is higher than the current one installed. When left out, no package installation is done.
unlockedPackages

Optional. Reference an array of a simple bean object that holds the values neeed to install Unlocked Packages. This happens BEFORE npackage source are deployed. When left out, no package installation is done.
unpackagedSourcePath

Optional. The path to the unpackaged source. This is not recommended approach any more because Unlocked Packages gives much better visibility and tracking. You can specify a comma-separeted list and they will be used. When left out, no unpackage source will be deployed.

Steps

The general pattern to use these steps is this, where the withOrgsInParallel step passes the org value into the closure:

#!/usr/bin/env groovy
@Library('sfdx-jenkins-shared-library')
import com.claimvantage.sjsl.Help
import com.claimvantage.sjsl.Org
import com.claimvantage.sjsl.Package

node {
    stage("checkout") {
        ...
    }
    withOrgsInParallel() { org ->
        stage("${org.name} create") {
            createScratchOrg org
        }
        ...
    }
    stage("publish") {
        ...
    }
}

createScratchOrg

Creates a scratch org and adds values relating to that to the supplied org object for use by later steps. This step has to come before most other steps. The org is created with the minimum duration value of --durationdays 1 as the number of active scratch orgs is limited, and failing builds might not get to their deleteScratchOrg step. You can change the duration day by changing the Org object. Details of the created org are output into the log via an echo step. It sets job name as the alias for the created scratch org.

org

Required. An instance of Org that has it’s projectScratchDefPath property set.

deleteScratchOrg

Deletes a scratch org identified by values added to the Org object by createScratchOrg. This step has to come after most other steps.

org

Required. An instance of Org that has been populated by createScratchOrg.

installPackage

Installs a package into a scratch org. Package installs typically take 2 to 20 minutes depending on the package size. The Package name and version are output via an echo in the stage.

org

Required. An instance of Org that has been populated by createScratchOrg.
package

Required. An instance of the Package bean object whose properties identify the package version to install.

Package Aliases

See Salesforce Project Configuration File for Packages for details. Use the example below to configure package aliases; the benefit is that the package version Id (that typically changes over time) is kept out of the Jenkinsfile that should not need to change over time.

The general pattern to use alias is to configure it on your sfdx-project.json file:

{
    "packageAliases": {
        "cve": "04t50000000AkkX",
        "cvab": "04t0V000000xEEs"
    }
}

processHelp

Disclaimer: Do not use this step as it contains dependencies on private ClaimVantage repositories that we don’t plan to release to the community.

This is a ClaimVantage proprietary stage that extracts help content from Confluence, processes that content and then adds the content to Git so that it can be pulled into a package via Git externals.

branch

The branch name for which this step runs. The default value is “master”.
help

Required. An instance of the Help bean object whose properties identify the help information.

pushToOrg

Pushes the components into a scratch org.

org

Required. An instance of Org that has been populated by createScratchOrg.

retrieveExternals

If a git_externals.json file is in the repository root, uses git-externals to pull in that content. If no file is present, the step does nothing (and git-externals does not have to be installed).

runApexTests

Runs Apex tests for an org and puts the test results in a unique folder based on the name of the org object. The test class names are also prefixed by that name so that when multiple orgs are tested, the test results are presented separated by the name.

To workaround frequent EAI_AGAIN errors while the tests are running, the current implementation polls to check whether the tests have finished rather than making the blocking SFDX call.

org

Required. An instance of Org that has been populated by createScratchOrg.
timeoutMinutes

Optional (Default to 300 minutes - 5h). Minutes that it should wait for test results.
usePooling

Optional (Default to true). If the tests are executed using pooling or synchronously.

runLightningTests

Runs Lightning tests for an org and puts the test results in a unique folder based on the name of the org object. The test class names are also prefixed by that name so that when multiple orgs are tested, the test results are presented separated by the name.

The Lightning Testing Service must be present in the org e.g. by making it part of the SFDX project.

org

Required. An instance of Org that has been populated by createScratchOrg.
appName

The name of the Lightning app used to test the application. For example “Test.app”.
configFile

Optional. The path to a test configuration file to configure WebDriver and other settings. There isn’t much official documentation on this; this Salesforce Stackexchange answer provides some information. An example file is:
```
{  
    "webdriverio":{  
        "desiredCapabilities": [{  
            "browserName": "chrome"
        }],
        "host": "hub.browserstack.com",
        "port": 80,
        "user": "username",
        "key": "password"
    }
}
```
runLwcTests

Runs Lightning Web Components for an org using sfdx-lwc-jest, puts the result in a unique folder and request junit to collect the results.

As part of the process it installs the LWC Test Runner for Jest.

This action expects tests to exist, currently looking for tests defined by glob in Jest, otherwise it fails to execute.

The jest-junit reporter must be present on devDependencies:

  "devDependencies": {
    "@salesforce/sfdx-lwc-jest": "^0.7.0",
    "jest-junit": "^10.0.0"
  }

org

Required. An instance of Org that has been populated by createScratchOrg.

runLint

Runs force:lightning:lint Tool in Parallel on multiple folders that contains Lightning components.

folders

Required. A list of folders that needs to be validated

It might look like this:

def LINT_FOLDERS = ["./sfdx-source/wiz/main/aura", "./sfdx-source/int/main/aura"]
runLint(folders:LINT_FOLDERS)

shWithResult

Typically not directly used as this is a building block for other steps.

Runs a shell command assumed to be an SFDX command that produces JSON output (typically via the --json argument), checks if the result is 0 (if it is not then it fails), and parses the output using readJSON step, returning the result object.

shWithStatus

Typically not directly used as this is a building block for other steps.

Runs a shell command and checks if the result is 0 (if it is not then it fails).

withOrgsInParallel

Finds matching Scratch Org Definition File files, and for each one uses the Jenkins Pipeline parallel step to execute the nested steps. This allows multiple org configurations to be handled at the same time.

glob

The matching pattern (see Ant style pattern for details - * matches zero or more characters, ? matches one character) used to find the Scratch Org Definition File files. Each matched file results in a separate parallel build. The default value is “config/project-scratch-def.*.json”; this assumes that an extra part will be inserted into the file names and that part is used as a name for the parallel work.

There are two ways to specify the matching pattern:
- to use the same matching pattern for all branches, just supply the matching pattern as a string
- to vary the matching pattern by branch, specify a map of branch names to matching patterns
A particularly useful map to use is this one, that identifies all org configurations for the master branch, but only one org configuration for other branches:
```
// Evaluate so a serializable string is used in the pipeline (the withDefault object is not serializable)
// See https://issues.jenkins-ci.org/browse/JENKINS-38186
glob: ['master': 'config/project-scratch-def*.json'].withDefault{'config/project-scratch-def.json'}[env.BRANCH_NAME],
```
and so reduces the number of Scratch Orgs consumed when multiple branches are being worked on at the same time.
stagger

Optional (defaults to 15 seconds). This is the number of seconds to delay before the next parallel set of steps is started. The aim is to smooth out the load a little both on the Jenkins machine and at the Salesforce side by staggering the the execution of the parallel logic.

Multiple Orgs

Each scratch org that is created and used (in parallel) is defined by a Scratch Org Definition File. The number of scratch orgs you can create per day is limited by Salesforce.

A single file, called project-scratch-def.json, might look like this:

{
    "orgName": "Jenkins Claims - (default)",
    "edition": "Developer",
    "namespace": "cve",
    "settings": {
        "orgPreferenceSettings": {
            "s1DesktopEnabled": true,
            "disableParallelApexTesting": true
        }
    }
}

Adding a second file called project-scratch-def.person-accounts.json (note the added features line) that looks like this:

{
    "orgName": "Jenkins Claims - (person-accounts)",
    "edition": "Developer",
    "namespace": "cve",
    "features": ["PersonAccounts"],
    "settings": {
        "orgPreferenceSettings": {
            "disableParallelApexTesting": true
        }
    }
  }

will result in the org-specific steps running in parallel for both orgs. Adding more files will result in more parallel work.

Not everything required can be configured via the project-scratch-def.json, so there is also an extension point attribute in the sfdxBuildPipeline called beforeTestStage where a closure can be added that executes arbitrary logic and is passed an org. Here is an example of using that extension point, in this case to setup platform encryption:

sfdxBuildPipeline(
    beforeTestStage: { org ->
        if (org.name == 'platform-encryption') {
            sh "sfdx force:user:permset:assign --permsetname Encryption --targetusername ${org.username}"
            sh "sfdx force:data:record:create --sobjecttype TenantSecret --values 'Description=Test' --targetusername ${org.username}"
            shWithResult "echo 'cve.SetupController.updateDefaultFieldEncryptedFlags(true);' | sfdx force:apex:execute --json --targetusername ${org.username}"
        }
    }
)

Org Bean

The attributes of the Org object (in order of usefulness) are:

Attribute	Description
`name`	Name of the scratch org. It is the unique part of the Scratch Org Definition File name extracted from the `projectScratchDefPath`.
`username`	Used to direct SFDX commands to the right org after the org has been created.
`password`	Allows interactive login if the org is kep after the build.
`instanceUrl`	Allows interactive login if the org is kep after the build.
`orgId`	Perhaps useful for looking up the scratch org in e.g. the Dev Hub.
`projectScratchDefPath`	The path to the specific Scratch Org Definition File.

Running Library Tests Locally

Test are automatically executed using Continuous Integration through Travis CI).

Install Gradle 4.10.2 (recommend using SDKMAN! listed on Gradle Installation guide).

Otherwise you can run the following in project root:

gradle assemble
gradle check

sfdx-jenkins-shared-library

Simplifies Jenkins Pipeline builds that use SFDX and scratch orgs.