Showing posts with label Automation Testing with Jenkins. Show all posts
Showing posts with label Automation Testing with Jenkins. Show all posts

Friday, April 5, 2019

Postman integration with Jenkins

Postman contains a full-featured testing sandbox that lets you write and execute JavaScript based tests for your API. You can then hook up Postman with your build system using Newman, the command line collection runner for Postman.
Newman allows you to run and test a Postman Collection. Newman and Jenkins are a perfect match. Let's review these topics to set up this operation.
Note: We are using Ubuntu as a target OS as in most cases your CI server would be running on a remote Linux machine.

Installation

Install NodeJS and npm. Newman is written in NodeJS and we distribute the official copy through npm. Install nodejs and npm for Linux.
Install Newman globally, to set up Newman as a command line tool in Ubuntu.
$ npm install -g newman

Run a collection in Postman

We assume you already have a Postman Collection with some tests. Run the collection in the Postman app. Here's an example of the output in Postman’s collection runner.
collection runner
Some of the tests are failing intentionally in the screenshot so we can show you the instructions for troubleshooting.

Run a collection using Newman

Run this collection inside Newman, using the command below. If everything is set up nicely, you should see the output below.
terminal output from collection run

Set up Jenkins

Jenkins exposes an interface at http://localhost:8080.
jenkins interface
Create a new job by clicking on the “New Item” link on the left sidebar > Select a “Freestyle Project” from the options > Name your project.
new Jenkins job
Add a build step in the project. The build step executes a Shell command.
execute shell command
Here is the command:
$ newman jenkins_demo.postman_collection --exitCode 1
Note here that we are using the Newman command parameter “exitCode” with the value 1. This denotes that Newman is going to exit with this code that will tell Jenkins that everything did not go well.
Click the Save button to finish creating the project.
source code management

Troubleshooting

Run this build test manually by clicking on the “Build Now” link in the sidebar.
run build
Jenkins indicates that the build has failed with a red dot in the title. We can check why with the console output from Newman.
build failed message
Click the “Console Output” link in the sidebar to see what Newman returned.
console output
Fix these tests inside Postman and then try again.
collection runner view
You can move on once you see green pass icons for all your tests like the screenshot above.
console output for all tests pass
Jenkins indicates that the build succeeded with a blue ball.

Configure frequency of runs

To set up the frequency with which Jenkins runs Newman, click on “Configure project” in the main project window and then scroll down.=. The syntax for setting the frequency is H/(30) * * * *.
build triggers
Note: 30 can be replaced with another number.
Jenkins will now run Newman at your desired frequency and will tell you whether the build failed or succeeded. In a bigger setup, Newman will be part of your build process and probably not the entire process. You can set up notifications and customize Jenkins as per your needs.
You can use a wide variety of other configurations to make your collection more dynamic. 

Thursday, June 25, 2015

Bitnami Jenkins Usage Steps

Jenkins, previously known as Hudson, is an open source continuous integration server. Built with Java, it provides over 400 plugins to support building and testing virtually any project. It supports SCM tools including CVS, Subversion, Git, Mercurial, Perforce and Clearcase, and can execute Apache Ant and Apache Maven based projects as well as arbitrary shell scripts and Windows batch commands. It also can monitor executions of remote tasks.
Please, take a look to the Quick Start Guide to know the basic use of this Stack.

How to start/stop the servers?

Graphical tool
The native installers include a graphical tool to manage the servers easily. You can find the "manager-windows.exe", "manager-osx" or "manager-linux" tool in your installation directory. Using this tool, you can Start, Stop or Restart the servers and check the log files. You can click on the icon to start it.
manager-servers.png

manager-osx.png










win_platform.png
On Windows: You can also start the Manager tool from shortcuts: Start -> Program Files -> Bitnami Stack -> Manager tool
Command line tool
If you prefer, you can use the "ctlscript.sh" utility from the command line. This script is in the installation directory.
linux_platform.png
On Virtual Machines and Cloud images:
$ sudo /opt/bitnami/ctlscript.sh start
A native installer on Linux:
$ cd ~/application-version
$ ./ctlscript.sh start
mac_platform.png

On OS X: You can start the Manager tool from the installation directory or you can use the "ctlscript.sh" utility from a Terminal.
host:~ user$ cd /Applications/application-version
host:~ user$ ./ctlscript.sh start

Where is JENKINS_HOME variable pointing to?

We define JENKINS_HOME in your installation directory in the apache-tomcat/conf/Catalina/localhost/jenkins.xml file. By default it is pointing to apps/jenkins/jenkins_home.
Notice that the installation directory is /opt/bitnami in the Bitnami virtual appliances and Bitnami Amazon Images.

How to start with Git and Jenkins?

Jenkins application is running as "tomcat" user so it is necessary to configure this user in case you want to access to a private repository. If you use a public repository is not necessary to configure the Tomcat user.
Configure Tomcat user to have access to the Git repository
Previous versions of Jenkins require to create the Tomcat user folder:
$ sudo mkdir /home/tomcat
$ sudo mkdir /home/tomcat/.ssh
Then copy the private key to access to the repository into the "/home/tomcat/.ssh" folder:
$ sudo cp /path/to/id_rsa /home/tomcat/.ssh
And change the permissions
$ sudo chown -R tomcat:tomcat /home/tomcat
It is also necessary to save the hostname of your repository into the "known_hosts" file:
$ sudo su tomcat -c "/opt/bitnami/git/bin/git ls-remote -h git@YOUR_REPOSITORY.git HEAD"
The authenticity of host 'ec2-xx-yy-zz.compute-1.amazonaws.com (xx.yy.zz)' can't be established. 
ECDSA key fingerprint is e5:xx:9d:yy:ff:89:55:47:aa:fd:b5:ee:0a:pp:6f:8d.
Are you sure you want to continue connecting (yes/no)? yes
Configure Git repository in Jenkins 
Before creating the first Job, it is necessary to configure the Git user and email that Jenkins will use to clone the repository. You can configure these options in "Manage Jenkins" -> "Configure System" -> "Git plugin" section
configuring-jenkins-git-2a.jpg
Then create a new Job in Jenkins and specify the Git repository.
create-job-jenkins-3.jpg
And run the Build. If everything was properly configured, Jenkins will checkout the code from the Git repository.

How to upgrade Jenkins?

If you want to upgrade Jenkins keeping your data, you just need to download the latest jenkins.war file, copy it in your apache-tomcat/webapps directory and restart Tomcat. It is recommended to create a backup of the previous jenkins.war file. You may want to revert to the previous version if you find any issues after upgrading.
Upgrading only Jenkins
On Linux and OSX you can execute the following steps to perform the above (notice that the installation directory is /opt/bitnami in the Bitnami virtual appliances and Bitnami Amazon Images):
 
$ cd
## Stop the server
$ ./ctlscript.sh stop
 
## Copy your previous .war file in case you need to restore it
$ cp apache-tomcat/webapps/jenkins.war apps/jenkins/jenkins.war.bak
 
## Remove jenkins all files to force Tomcat to deploy the new jenkins.war file
$ rm -r apache-tomcat/webapps/jenkins
$ rm -r apache-tomcat/webapps/jenkins.war
$ rm -r apache-tomcat/work/Catalina/localhost/jenkins
 
## Donwload the new version of jenkins
$ mv jenkins.war apache-tomcat/webapps/
 
## Start the server
$ ./ctlscript.sh start
On Windows you can stop the servers from the Start Menu entry or using the graphical tool that you can find in your installation directory. Using the file explorer go to where you installed the Bitnami Jenkins Stack. Copy the jenkins.war file from the apache-tomcat/webapps directory to apps/jenkins. Download the latest jenkins.war file and copy it in apache-tomcat/webapps. Now just start the server from the Start Menu or using the graphical tool.
Upgrading all the components
Alternatively to the process above, you can download a new Bitnami Jenkins Stack. You can install it to a different location (remember to stop the previous install so there are no port conflicts) and point it to your previous JENKINS_HOME path by editing the file  apache-tomcat/conf/Catalina/localhost/jenkins.xml.
If you are using a Bitnami virtual appliance or AMI you can start a new image with the latest jenkins version and move the content of /opt/bitnami/apps/jenkins/jenkins_home in your current installation to the new installation. Then restart the server.

How to enable SSL in Jenkins?

You can configure Apache Tomcat to enable SSL connections for accessing to the application using the HTTPS protocol. It is necessary to specify the Apache SSL certificate. If you do not have the cert you can create following these steps in the section below.
Check that you have the certificate file at "/installdir/apache-tomcat/conf/ssl/tomcat.cert.pem" and the certificate key file at "/installdir/apache-tomcat/conf/ssl/tomcat.key.pem".
To enable SSL you should uncomment the following line in the /installdir/apache-tomcat/conf/server.xml file:
8443
" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" disableUploadTimeout="true" acceptCount="100" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" keystoreFile="conf/myKeystore" keystorePass="your_password" SSLCertificateFile="installdir/apache-tomcat/conf/ssl/tomcat.cert.pem" SSLCertificateKeyFile="installdir/apache-tomcat/conf/ssl/tomcat.key.pem"/>
Then restart the Apache server and try to access at "https://localhost:8443".

How to create a SSL certificate for Jenkins?

You can find a detailed guide in the official Apache Tomcat documentation at http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html

Bitnami Jenkins Stack Documentation

      Bitnami Jenkins Stack 1.617-0
    ============================

1. OVERVIEW

The Bitnami Project was created to help spread the adoption of freely
available, high quality open source web applications. Bitnami aims to make
it easier than ever to discover, download and install Open Source software 
such as document and content management systems, wikis and blogging 
software.

You can learn more about Bitnami at https://bitnami.com

Jenkins, previously known as Hudson, is an open source continuous integration
server. Built with Java, it provides over 400 plugins to support building and 
testing virtually any project. It supports SCM tools including CVS, Subversion, 
Git, Mercurial, Perforce and Clearcase, and can execute Apache Ant, Apache Maven
and Gradle based projects as well as arbitrary shell scripts and Windows batch
commands. It also can monitor executions of remote tasks.

You can learn more about Jenkins at http://jenkins-ci.org

The Bitnami Jenkins Stack is an installer that greatly simplifies the
installation of Jenkins and runtime dependencies. It includes ready-to-run
versions of Apache-Tomcat and Java. Jenkins Stack is distributed for 
free under the Apache 2.0 license. Please see the appendix for the specific
licenses of all Open Source components included.

You can learn more about Bitnami Stacks at https://bitnami.com/stacks/

2. FEATURES

- Easy to Install

Bitnami Stacks are built with one goal in mind: to make it as easy as
possible to install open source software. Our installers completely automate
the process of installing and configuring all of the software included in
each Stack, so you can have everything up and running in just a few clicks.

- Independent

Bitnami Stacks are completely self-contained, and therefore do not interfere
with any software already installed on your system. For example, you can
upgrade your system's Java or Apache Tomcat without fear of 'breaking' your
Bitnami Stack.

- Integrated

By the time you click the 'finish' button on the installer, the whole stack
will be integrated, configured and ready to go. 

- Relocatable

Bitnami Stacks can be installed in any directory. This allows you to have
multiple instances of the same stack, without them interfering with each other. 

3. COMPONENTS

Bitnami Jenkins Stack ships with the following software versions:
 
  - Jenkins 1.617
  - Apache 2.4.12
  - Apache Tomcat 7.0.62
  - Git 1.9.5
  - Git 2.4
 
On Linux and Windows thea version of the stack includes a bundled JDK 1.7.0_76. 
On OS X, it is required that you have Java 1.5 or later installed in your system. 
It can be downloaded from http://www.apple.com/macosx/features/java/

4. REQUIREMENTS

To install Bitnami Jenkins Stack you will need:

    - Intel x86 or compatible processor
    - Minimum of 512 MB RAM
    - Minimum of 400 MB hard drive space
    - TCP/IP protocol support
    - Compatible operanting systems:
      - An x86 or x64 Linux operating system.
      - A 32-bit Windows operating system such as Windows 2000, XP, Vista, 7, 
      Windows Server 2008 or Windows Server 2003.
      - An OS X operating system.


5. INSTALLATION

The Bitnami Jenkins Stack is distributed as a binary executable installer.
It can be downloaded from:

https://bitnami.com/stacks/

The downloaded file will be named something similar to:

bitnami-jenkins-1.617-0-linux-installer.run on Linux or
bitnami-jenkins-1.617-0-linux-x64-installer.run on Linux 64 bit or
bitnami-jenkins-1.617-0-windows-installer.exe on Windows or
bitnami-jenkins-1.617-0-osx-x86-installer.dmg on OS X x86.

On Linux, you will need to give it executable permissions:

chmod 755 bitnami-jenkins-1.617-0-linux-installer.run on Linux


To begin the installation process, invoke from a shell or double-click on
the file you have downloaded, and you will be greeted by the 'Welcome'
screen. You will be asked to choose the installation folder. If the
destination directory does not exist, it will be created as part of the
installation.

The default listening port for the bundled Apache is 8080, Apache Tomcat port is
not enabled by default. Apache Tomcat and Apache are connected through the AJP 
Proxy Connector that uses the 8009 port. If those ports are already in use by 
other applications, you will be prompted for alternate ports to use.

The next screen will ask you for the credentials for the administrator user 
that will be created for your jenkins installation. 

On OS X x86, if the installer has not been able to find a suitable JDK installation 
in your system, you will have to provide the location of a jdk1.5 or jdk1.6 
installation in the next page.

Once the installation process has been completed, you will see the
'Installation Finished' page. You can choose to launch Bitnami
Jenkins Stack at this point. If you do so, your default web browser
will point you to the Bitnami local site.

If you receive an error message during installation, please refer to
the Troubleshooting section.

The rest of this guide assumes that you installed Bitnami Jenkins
Stack in /home/user/jenkins-1.617-0 on Linux or C:\Program Files\Bitnami Jenkins Stack
on Windows or /Applications/jenkins-1.617-0 on OS X and you use port 8080 for Apache 
Tomcat.

6. STARTING AND STOPPING BITNAMI JENKINS STACK

The Bitnami Application Manager is a simple graphical interface included in
the stacks that can start and stop the Bitnami servers. It is are located
in the same installation directory.

To start the utility, double click the file named ‘manager-linux’,
'manager-windows' or 'manager-osx' from your file browser.

Alternatively, you can also start and stop the services manually, as explained below.

To start/stop/restart application on Linux or OS X you can use the included 
ctlscript.sh utility, as shown below:

       ./ctlscript.sh (start|stop|restart)
       ./ctlscript.sh (start|stop|restart) tomcat
       ./ctlscript.sh (start|stop|restart) apache

  start      - start the service(s)
  stop       - stop  the service(s)
  restart    - restart or start the service(s)

You can start and stop Bitnami Jenkins Stack on Windows using the shortcuts
created in the Start Menu, under Programs -> Bitnami -> Bitnami Service

That will start Apache Tomcat service. Once started, you can open your
browser and access the following URL:

http://127.0.0.1:8080/jenkins/

If you selected an alternate port during installation, for example 18080, the
URL will look like:

http://127.0.0.1:18080/jenkins/


7. DIRECTORY STRUCTURE

The installation process will create several subfolders under the main
installation directory:

 apache2/: Apache Web server.
    gradle/: Gradle build automation tool.
 apache-tomcat/: Apache Tomcat Web server.
        git/: Git version control system.
 java/: Java SE Development Kit.
        perl/: Perl (only on Unix)
 apps/jenkins: Applications folder
        apps/jenkins/jenkins_home: Folder in which Jenkins keep archives, configuration files, etc.

(In Bitnami Virtual Appliances and AMIs the default installation directory is /opt/bitnami).

8. DEFAULT CONFIGURATION

8.1 Default usernames and passwords

The Jenkins administrative user and password are the ones you set at installation time.
(In Bitnami Virtual Appliances and AMIs this default to user/bitnami).

By default Jenkins access control is configured to use Jenkins's own user database and
Project-based Matrix Authorization Strategy. Anonymous user doesn't have access permissions by default and
users sign up is disabled. You can easily change this from the Jenkins interface once you login as the
administrator user created during the installation.

8.2. Setting JENKINS_HOME

The Bitnami Jenkins Stack defines the JENKINS_HOME variable is the Tomcat context descriptor, 
in apache-tomcat/config/Catalina/localhost/jenkins.xml 

8.3 Git Plugin

Git plugin is already installed by default. It is configured to use the git binary bundled with the stack which 
is located in your installation directory git/bin/ folder.

8.4 Gradle Plugin

Gradle plugin is also installed by default. It is configured to use the gradle binary bundled with the stack which 
is located in your installation directory gradle/bin/ folder.

9. TROUBLESHOOTING

You can find more information about this product at:

https://wiki.bitnami.com

We also encourage you to post your questions and suggestions at:

https://community.bitnami.com

We also encourage you to sign up for our newsletter, which we'll use to
announce new releases and new stacks. To do so, just register at:
https://bitnami.com/newsletter.


9.1 Installer

# Installer Payload Error 

If you get the following error while trying to run the installer from the
command line:

"Installer payload initialization failed. This is likely due to an
incomplete or corrupt downloaded file" 

The installer binary is not complete, likely because the file was
not downloaded correctly. You will need to download the file and
repeat the installation process.

# Installer execution error on Linux

If you get the following error while trying to run the installer:

"Cannot open bitnami-jenkins-1.617-0-linux.run: No application suitable for 
automatic installation is available for handling this kind of file."

In some operatings systems you can change permissions with right click ->
properties -> permissions -> execution enable. 

Or from the command line:

$ chmod 755 bitnami-jenkins-1.617-0-linux.run 

9.2 Java 

# Customize application on Linux or on OS X

If you want to reduce application delay, you can change apache-tomcat/bin/setenv.sh
by replacing this line:

JAVA_OPTS="-XX:MaxPermSize=512m -Xms256m -Xmx512m"

with

JAVA_OPTS="-XX:MaxPermSize=512m -Xms512m -Xmx1024m"

9.3 Apache Tomcat

If you find any problem starting Apache Tomcat, the first place you
should look at is the error log file that will be created at
jenkins-1.617-0/apache-tomcat/logs/catalina.out. There you will
likely find useful information to determine what the problem is. For issues
not covered in this Quick Start guide, please refer to the Bitnami forums
and to the Apache Tomcat documentation, which is located at
http://tomcat.apache.org/tomcat-5.5-doc/index.html.

9.4 Jenkins

For any problem related to Jenkins, please visit
http://wiki.jenkins-ci.org/display/JENKINS/Home

# Automated GUI Testing in Windows

As Tomcat is configured as a service you may find issues when running 
Automated GUI Test (https://wiki.jenkins-ci.org/display/JENKINS/Tomcat). A typical error might look similar to this:

[junit] \# An unexpected error has been detected by HotSpot Virtual Machine:
[junit] \#
[junit] \# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d07baf4, pid=3260, tid=288
[junit] \#
[junit] \# Java VM: Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode, sharing)
[junit] \# Problematic frame:
[junit] \# C [awt.dll+0xbaf4|awt.dll+0xbaf4]
[junit] \#

10. LICENSES

Apache Web Server is distributed under the Apache License v2.0, which
is located at http://www.apache.org/licenses/LICENSE-2.0

Apache Tomcat is distributed under the Apache License v2.0, which
is located at http://www.apache.org/licenses/LICENSE-2.0

MySQL is distributed under the GNU General Public License v2, which is
located at http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

Java and related libraries are distributed under the Common Development 
and Distribution License (CDDL), Version 1.0 and the Sun Microsystems, Inc. 
("Sun") Software License Agreement, wich are located at
http://java.sun.com/j2se/1.5.0/docs/relnotes/license.html

Jenkins is distributed under the Apache License v2.0, which
is located at http://www.apache.org/licenses/LICENSE-2.0

Hibernate is distributed under the GNU General Public License v2, which 
is located at http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
 


Sunday, March 29, 2015

Jenkins as the Test Reporting Framework

Very few and matured companies are able to invest on the development of the Test Reporting Frameworks. At Komli we realised the benefits & value it offers.Instead of buying some commercial tools we went with the approach of extending Jenkins for the same. This talk is about our experience regarding the same.
Test Reporting Framework are the ultimate utilities reached by very few companies in the world likes of Google & Facebook. It require significant investment in terms money and resources. To optimize on both we tried the approach of extending Jenkins for the same purpose.
We like to share our experiences with respect to this journey. We will talk about different challenges we faced in doing so and how we solved them.
We strongly believe that other startup can use our experience and get benefit from this immensely.

WHY TEST REPORTING FRAMEWORK?

  1. To help with informed decision, whether to go ahead with production deployment of the build
  2. To analyze whether sufficient testing being done
  • What are code changes?
  • What are the corresponding automation changes w.r.t test-cases added/deleted/modified
  • What is the test automation run trend w.r.t trend on Release basis

REPORTING FRAMEWORK REQUIREMENTS

  • Web based Tool
  • Cost Effective
  • Easy to maintain
  • Integration with Test Case Execution Frameworks

SOME OF PAID TEST REPORTING FRAMEWORKS AVAILABLE
  • TestRail
  • Xqual
  • Test Collab

SOLUTION TO THE ABOVE PROBLEMS :- JENKINS

  • Continuous Integration server that allows to run multiple automation suites at the same time.
  • Supports SVN, Git and about any other SCM tools you can think of via plugins

WHY JENKINS

  • Web based
  • No additional cost, we are already dependent on it as Test-Case Execution Framework
  • Well supported
  • Highly customizable

JENKINS ON THE RELEASE BASIS








COMMON JENKINS TERMS

  • Project
  • Builds

NEW TERMS INTRODUCED

  • Benchmark Project
  • Benchmark Builds 

BENCHMARK PROJECT

  • The project in Jenkins where we maintain the automation results on the basis of releases done to production.


BENCHMARK BUILD

  • Build which has best results
  • The one whose final status we want to maintain in the benchmark project

WHAT IS ALREADY AVAILABLE IN JENKINS

  • Show automation run trend
  • Integration with SCM tool show code diffs


CHALLENGES

  1. Integration : Want everything in the single screen (single project in Jenkins)
  2. Benchmark Build : We don't know priori that given build is the best build to run
  • Don't want to waste resources and time to execute automation run again once we have decided on the build to deploy on production.
  • Intermittent Failures : Want to maintain their status as Passed for a given release
  • If we have taken the approach of copying results from one project to other project (benchmark project).

OUR APPROACH

  • Git Plugin for Code Diff
  • Custom Java Script for Test-Cases diff
  • Custom Java Script for overriding intermittent failure status
  • CopyArtifact Plugin to migrate results
  • Custom Groovy Script for re-generating the TestNG result trends

INTERMITTENT FAILURES


 

PARAMETERS TO COPY BENCHMARK BUILD






Slideshare Link for Presentation at RootConf


My Profile

My photo
can be reached at 09916017317