Introduction
In the fast-paced development world, automation in Continuous Integration (CI) and Continuous Deployment (CD) pipelines is crucial. Integrating Bitbucket's PR system with Jenkins allows automatic build triggering whenever developers comment on the pull requests. This blog showcases a Proof of Concept (PoC) that demonstrates how to automate multibranch pipeline triggers using Docker containers, Bitbucket webhooks, and Jenkins.
Who Is This Relevant For?:
- Developers using Bitbucket and Jenkins for CI/CD
- Teams managing multibranch pipelines and looking to automate the build process based on pull request comments
To demonstrate how a Dockerized environment for Jenkins and Bitbucket can be integrated using webhooks, allowing multibranch pipelines to be automatically triggered whenever a pull request is opened or commented upon in Bitbucket we will be using the following
Tools & Setup:
- Docker Compose – Used to deploy Jenkins and Bitbucket in isolated containers.
- Bitbucket Webhooks – To detect PR events and trigger the appropriate Jenkins job.
- Jenkins Multibranch Pipelines – To automate builds based on the branch and repository.
- Python Flask Application – A webhook listener that parses incoming Bitbucket PR events and triggers the respective Jenkins pipeline using Jenkins API.
For demonstration, we have dockerized the example, into following folder structure
project-root/
│
├── docker-compose.yaml
│
├── webhook-listener/
│ ├── Dockerfile
│ ├── app.py
│ └── requirements.txt
│
└── README.md
docker-compose.yaml:
The file that defines and orchestrates the services (Jenkins, Bitbucket, and the webhook listener).webhook-listener:
Contains the Flask webhook listener code, a Dockerfile to create the image for the webhook app, and a requirements.txt to list the Python dependencies.
Step-by-Step Implementation
-
Docker Compose Setup
The
docker-compose.yaml
sets up all services in a single file. This allows you to bring up Jenkins, Bitbucket, and the Flask app with one command.
version: '3.8'
services:
jenkins:
image: jenkins/jenkins:lts-jdk11
container_name: jenkins
restart: always
ports:
- "8080:8080"
- "50000:50000"
volumes:
- jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock # To allow Jenkins to control Docker
networks:
- dev-net
bitbucket:
image: atlassian/bitbucket-server:7.6.0 # Replace with specific tag if needed
container_name: bitbucket
restart: always
ports:
- "7990:7990"
- "7999:7999"
volumes:
- bitbucket_data:/var/atlassian/application-data/bitbucket
networks:
- dev-net
environment:
- BITBUCKET_HOME=/var/atlassian/application-data/bitbucket
- JVM_SUPPORT_RECOMMENDED_ARGS=-Djava.security.egd=file:/dev/urandom
dind:
image: docker:19.03-dind
container_name: dind
privileged: true # Needed for DinD (Docker-in-Docker)
networks:
- dev-net
environment:
- DOCKER_TLS_CERTDIR=/certs
volumes:
- dind_data:/var/lib/docker
webhook-listener:
build: ./webhook # You will run your Python webhook here
container_name: webhook-listener
volumes:
- ./webhook:/app # Mount your Python webhook code
- ./logs:/app/logs # Mount your Python webhook code
networks:
- dev-net
ports:
- "5000:5000"
depends_on:
- bitbucket
- jenkins
environment:
FLASK_APP: app.py
FLASK_ENV: development # Enable development mode
networks:
dev-net:
driver: bridge
volumes:
jenkins_home:
bitbucket_data:
dind_data:
- Webhook Listener Setup webhook-listener/app.py contains the core logic for the webhook listener, it's idea is to get the REST API calls and detect for trigger commands (if present) in comments.
import logging
from flask import Flask, request, jsonify
import requests
from requests.auth import HTTPBasicAuth
import urllib.parse
# Configure logging to output to a file as well
logging.basicConfig(
level=logging.DEBUG, # Changed to DEBUG for more verbose logging
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler("webhook_listener.log"), # Log to a file
logging.StreamHandler() # Also log to the console
]
)
app = Flask(__name__)
JENKINS_BASE_URL = "http://jenkins:8080"
JENKINS_USER = "YOUR USERNAME"
JENKINS_TOKEN = "YOUR JENKINS TOKEN"
REPO_TO_JOB_MAPPING = {
'test1': 'TEST-JOB-CI/job/Test1',
'test2': 'TEST-JOB-CI/job/Test2',
}
jenkins_auth = HTTPBasicAuth(JENKINS_USER, JENKINS_TOKEN)
def trigger_jenkins_pipeline(repo_name, branch_name):
"""Triggers a Jenkins build for the specified branch with parameters."""
job_name = REPO_TO_JOB_MAPPING.get(repo_name)
if not job_name:
logging.error(f"No Jenkins job configured for repository: {repo_name}")
return
# Encode the branch name to handle special characters like '/'
encoded_branch_name = urllib.parse.quote(branch_name, safe='')
url = f"{JENKINS_BASE_URL}/job/{job_name}/job/{encoded_branch_name}/build"
logging.debug(f"Triggering Jenkins build at URL: {url}")
# Fetch Jenkins crumb for CSRF protection
crumb_url = "http://jenkins:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"
crumb_response = requests.get(crumb_url, auth=jenkins_auth)
if crumb_response.status_code == 200:
crumb_field, crumb_value = crumb_response.text.split(':')
headers = {crumb_field: crumb_value}
try:
response = requests.post(url, auth=jenkins_auth, headers=headers)
logging.debug(f"Response Code: {response.status_code}, Response Body: {response.text}")
if response.status_code == 201:
logging.info(f"Jenkins build triggered successfully for branch: {branch_name}")
else:
logging.error(f"Failed to trigger Jenkins build for branch: {branch_name}, Status Code: {response.status_code}, Reason: {response.reason}")
except requests.exceptions.RequestException as e:
logging.error(f"Request failed: {e}")
else:
logging.error(f"Failed to get Jenkins crumb, Status Code: {crumb_response.status_code}, Reason: {crumb_response.reason}")
@app.route('/bitbucket-webhook', methods=['POST'])
def bitbucket_webhook():
"""Handles incoming Bitbucket webhooks."""
data = request.json
logging.info("Webhook received from Bitbucket")
logging.debug(f"Received data: {data}")
if 'pullRequest' in data:
pr = data['pullRequest']
repo_name = pr['fromRef']['repository']['slug']
branch_name = pr['fromRef']['displayId']
comment = data.get('comment', {})
logging.info(f"PR from repo: {repo_name}, branch: {branch_name} received")
if comment: # Check if there is a comment
logging.info(f"Comment ID: {comment['id']}, Text: {comment['text']}, Author: {comment['author']['displayName']}")
if "trigger" in comment['text'].lower():
logging.info(f"Trigger word found in comment, triggering Jenkins build for repo: {repo_name}, branch: {branch_name}")
trigger_jenkins_pipeline(repo_name, branch_name)
else:
logging.info("No trigger word found in comment")
else:
logging.info("No comment found in pull request")
else:
logging.warning("Request does not contain 'pullRequest' key")
return jsonify({"status": "received"}), 200
if __name__ == '__main__':
logging.info("Starting Bitbucket Webhook Listener...")
app.run(host='0.0.0.0', port=5000)
Key Points:
- It listens for POST requests from Bitbucket's webhook system.
- Based on the repository name and branch from the webhook, it triggers the corresponding Jenkins pipeline.
- The
REPO_TO_JOB_MAPPING
dictionary maps repository names to Jenkins job names, allowing for easy extension and flexibility.
webhook-listener/requirements.txt
:
Flask==2.0.2
requests==2.26.0
-
Running the Setup
Once everything is configured, run the following command from the
project-root
directory to spin up the services:
docker-compose up --build -d
This command will:
- Build the Docker images for the webhook listener.
Start Jenkins, Bitbucket, and the webhook listener in isolated containers in detach mode.
Configure Test Repositories in Bitbucket
Setup multiple test repositories for the purpose of verification, saytest1
andtest2
with sampleJenkinsfile
in each, which is to be used for setting up multibranch pipeline later
Configure the Webhooks
Setup a webhook for the endpointhttp://<ip addr>:5000/bitbucket-webhook
where ip address for the bitbcuket container can be retrieved by executing the following command
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' bitbucket
Configure the webhook with following options
-
Jenkins Multibranch Pipeline Setup
Once Jenkins is running on
http://localhost:8080
, you'll need to create multibranch pipelines for the repositories (test1, test2, etc.). The Jenkins job name will match the job names specified in the repo_to_job_mapping dictionary. - Go to Jenkins > New Item > Multibranch Pipeline
- Configure each multibranch pipeline to scan the appropriate Bitbucket repository.
- Setup branch source(here bitbucket which is already added through
Manage Jenkins
). Further setup the following to supress automatic build trigger based on new commits.
- Test the Integration To test the webhook flow:
- Create a repository in Bitbucket named test1.
- Open a pull request for a branch (e.g., feature-branch) and provide the comment.
- The webhook listener will detect the event and trigger the appropriate Jenkins job for the branch.
- Monitor Jenkins to see the triggered job for that branch.
A sample docker webhook-listener log
$ docker-compose logs -f webhook-listener
webhook-listener | 2024-09-22 10:17:09,128 - INFO - 172.24.0.3 - - [22/Sep/2024 10:17:09] "POST /bitbucket-webhook HTTP/1.1" 200 -
webhook-listener | 2024-09-22 10:26:46,759 - INFO - Webhook received from Bitbucket
webhook-listener | 2024-09-22 10:26:46,759 - DEBUG - Received data: {'eventKey': 'pr:comment:added', 'date': '2024-09-22T10:26:46+0000', 'actor': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'pullRequest': {'id': 2, 'version': 0, 'title': 'Feature/KWS-0002', 'description': '* Add repo details\r\n* KWS-0002: test commit', 'state': 'OPEN', 'open': True, 'closed': False, 'createdDate': 1727000643753, 'updatedDate': 1727000643753, 'fromRef': {'id': 'refs/heads/feature/KWS-0002', 'displayId': 'feature/KWS-0002', 'latestCommit': '89e876d3e6d0b84c96876a8e364705a30cd78771', 'repository': {'slug': 'test1', 'id': 2, 'name': 'test1', 'hierarchyId': 'f7550b4c5c2d0f8afefd', 'scmId': 'git', 'state': 'AVAILABLE', 'statusMessage': 'Available', 'forkable': True, 'project': {'key': 'TEST', 'id': 2, 'name': 'test', 'public': False, 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST'}]}}, 'public': True, 'links': {'clone': [{'href': 'http://localhost:7990/scm/test/test1.git', 'name': 'http'}, {'href': 'ssh://git@localhost:7999/test/test1.git', 'name': 'ssh'}], 'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/browse'}]}}}, 'toRef': {'id': 'refs/heads/develop', 'displayId': 'develop', 'latestCommit': '7ea6d999d56df26f381be449426a323f3d963c4c', 'repository': {'slug': 'test1', 'id': 2, 'name': 'test1', 'hierarchyId': 'f7550b4c5c2d0f8afefd', 'scmId': 'git', 'state': 'AVAILABLE', 'statusMessage': 'Available', 'forkable': True, 'project': {'key': 'TEST', 'id': 2, 'name': 'test', 'public': False, 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST'}]}}, 'public': True, 'links': {'clone': [{'href': 'http://localhost:7990/scm/test/test1.git', 'name': 'http'}, {'href': 'ssh://git@localhost:7999/test/test1.git', 'name': 'ssh'}], 'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/browse'}]}}}, 'locked': False, 'author': {'user': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'role': 'AUTHOR', 'approved': False, 'status': 'UNAPPROVED'}, 'reviewers': [], 'participants': [], 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/pull-requests/2'}]}}, 'comment': {'properties': {'repositoryId': 2}, 'id': 31, 'version': 0, 'text': 'now trigger', 'author': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'createdDate': 1727000806737, 'updatedDate': 1727000806737, 'comments': [], 'tasks': [], 'severity': 'NORMAL', 'state': 'OPEN'}}
webhook-listener | 2024-09-22 10:26:46,759 - INFO - PR from repo: test1, branch: feature/KWS-0002 received
webhook-listener | 2024-09-22 10:26:46,759 - INFO - Comment ID: 31, Text: now trigger, Author: r m
webhook-listener | 2024-09-22 10:26:46,759 - INFO - Trigger word found in comment, triggering Jenkins build for repo: test1, branch: feature/KWS-0002
webhook-listener | 2024-09-22 10:26:46,760 - DEBUG - Triggering Jenkins build at URL: http://jenkins:8080/job/Axiado-Test-CI/job/Test1/job/feature%2FKWS-0002/build