Appium Architecture - Core Concepts

What is Appium?

It’s a NodeJS based open-source tool for automating mobile applications. It supports native, mobile web, and hybrid applications on iOS mobile, Android mobile, and Windows desktop platforms.

Using Appium, you can run automated tests on physical devices or emulators, or both.

Let’s understand the above Appium architecture diagram.

• Appium is a client-server architecture. The Appium server communicates with the client through the HTTP JSONWire Protocol using JSON objects.
• Once it receives the request, it creates a session and returns the session ID, which will be used for communication so that all automation actions will be performed in the context of the created session.
• Appium uses the UIAutomator test framework to execute commands on Android devices and emulators.
• Appium uses the XCUITest test framework to execute commands on Apple mobile devices and simulators.
• Appium uses WinAppDriver to execute commands for Windows Desktop apps. It is bundled with Appium and does not need to be installed separately.

Appium - Android visual interaction flow

Let’s understand the interaction flow between the code and the Android device via the Appium server.

• The client sends the request to the Appium server through the HTTP JSONWire Protocol using JSON objects.
• Appium sends the request to UIAutomator2.
• UIAutomator2 communicates to a real device/simulator using bootstrap.jar which acts as a TCP server.
• bootstrap.jar executes the command on the device and sends the response back.
• Appium server sends back the command execution response to the client.

Appium - iOS visual interaction flow

Let’s understand the interaction flow between the code and the iOS device via the Appium server.

• The client sends the request to the Appium server through the HTTP JSONWire Protocol using JSON objects.
• Appium sends the request to XCUITest.
• XCUITest communicates to a real device/simulator using bootstrap.js which acts as a TCP server.
• bootstrap.js executes the command on the device and sends the response back.
• The Appium server sends the command execution response to the client.

Whiteboard Sessions

• IOS flow architecture

• Android flow architecture

• Drivers which appium supports
• UI Automator2 (Android)
• Espresso (Android)
• WinApp (Windows)
• MAC Driver (Mac OS)
• XCUITest (IOS above 9.3 version)
• UI Automation (IOS below 9.3 version)
• Tizen (for samsung)

Happy Learning :) 

How Netflix onboards new content - Video Processing at scale🎥

Everyday, #Netflix handles billions of requests regarding movies, trailers and other video content. Delivering at such a large scale needs an #engineering marvel. This #video will talk about how Netflix is able to onboard new video content onto their platform. We go from video chunking to collating 4 second shots into scenes.

Amazon S3 is used to store the video chunks. Netflix also provides open connect servers to internet service providers, which acts like a cache of movies. Most requests to Netflix can be served by this cache, and the remaining are sent over the network. This reduces the bandwidth and time required for Netflix to operate at scale. Synergy at it's finest.

Video Formats and Resolutions

• Different formats
• High quality
• Medium quality
• Low quality
• Different resolutions
• 1080p
• 720p
• 480p
• Storage combination comes F x R -> V
• What netflix does is like broken down a video into smaller parts so that it can be deal with effectively per processor
• One resolution, one format, one chunk - that's one task

Chunk Processing

• What they are doing intelligently is breaking the chunks not based on timestamps but based on scenes to have seamless user experience.
• Based os scenes so you can make instead of 3 min thing, you can make it more fine grained 4 sec each, it's called a shot and you can collate shot, put them all together to create a scene.
• Prediction algorithm is pretty smart to understand if user is not watching with engaged mode and clicking fwd to see the movie, instead of giving the whole content, it gives only data to the user has asked for because they are probably clicking on different points in that buffer you get.
• On the other hand, if user if watching in engaged mode, so instead of sending just the part user has asked for, it redictively proactively fetches the future parts, gets onto your computer and shows it you.

Storage

• Amazon S3 is what Netflix uses to store that video content.
• This is where people store their static data meaning that you don't change that data.
• It's extremely cheap compared to a database.

OpenConnect for video caching

• Netflix servers are usually in the U.S which means they are geographically concentrated and in a place like India which is really far, it's going to take lot of time to send the signal and receive it especially if it's video because there is lot of data which is going to be coming in and it's going to be slow.
• What Netflix did intelligently to extend the concept of caching and apply it to ISP's. When request comes to ISP for the movie, it looks for local cache say cache for Indian movies.
• Cache has been called as OpenConnect
• Lots of bandwidth saved
• Lots of time saved
• Much better user experience
• 90% of the Netflix traffic is taken care by these ISP boxes that they provide.
• W.r.t new content overnight job can be run to copy the new contents when there is less load on server for the requests.

Database Sharding - Key Concepts

Sharding a database is a common scalability strategy used when designing server side systems. The server side system architecture uses concepts like sharding to make systems more scalable, reliable and performant.

Sharding is horizontal partitioning of data according to a shard key. This shard key determines which database the entry to be persisted is sent to. Some common strategies for this are reverse proxies.

Let's take a common example of pizza and break it into slices and call your friends over. Each of your friend is going to get one slice of pizza. What you have done effectively is partitioned the pizza according to each friend's share. Just like that we can have serves which are going to be taking the load of the requests.

How we can convert our tech requirement to the pizza model.

• Basically each server going to handle requests based on partition say 1-100 userIds on 1st server and 101-200 on 2nd server and so on.
• This kind of partitioning which uses some sort of a key to break the data into pieces and allocate that to different servers is called horizontal partitioning.
• Servers which we are talking about here are database servers.

While we are discussing about database tuning, we always have to make sure we are maintaining key attributes of the database.

• Consistency
• Availability

What should be shard your data on?

• We are using userId in our case but in applications like tinder which use location, you could shard on the location and if a person says find me all the users in city X and X may fall in one specific sharda and all you need to do just read through this shard.

Problems doing Sharding

• Joins between shards
• Fixed number of shards
• With hierarchichal sharding approach we can overcome this problem, we can break one shard into sub-shards and there can one master sort of thing which can decide which mini shard request needs to route.

Best Practices

• Create index on shards
• This index can be on completely different attribute compared to userId.
• One of the good example is like find me all the people on NewYork having age greater than 50.
• Use Master Slave architecture on the shards to avoid SPOF.
• Read requests can go to slaves.
• Write requests always goes to master.
• In case master fails, slaves choose one master among themselves.

Conclusion

• Conceptually it's easy but in terms of doing practically it's quite tricky because consistency is really tough to do.
• If starting with a new system, take other mechanisms into consideration like indexing, noSQL databases which internally uses these these kind of concepts.
• Indexing and ready made solutions would be way to go before to think of implementing sharding by our own.

Monolith vs Microservice Architecture

Microservices are a hot topic these days. It is important to know why we use them instead of monolithic systems. The short answer is: Scalability. The detailed one would be:

Advantages:

• The microservice architecture is easier to reason about/design for a complicated system.
• They allow new members to train for shorter periods and have less context before touching a system.
• Deployments are fluid and continuous for each service.
• They allow decoupling service logic on the basis of business responsibility
• They are more available as a single service having a bug does not bring down the entire system. This is called a single point of failure.
• Individual services can be written in different languages.
• The developer teams can talk to each other through API sheets instead of working on the same repository, which requires conflict resolution.
• New services can be tested easily and individually. The testing structure is close to unit testing compared to a monolith.

Microservices are at a disadvantage to Monoliths in some cases. Monoliths are favorable when:

• The technical/developer team is very small
• The service is simple to think of as a whole.
• The service requires very high efficiency, where network calls are avoided as much as possible.
• All developers must have context of all services.

Kafka - Core Concepts

 Let's discuss about common terms been used in kafka and about their roles in distributed architecture.

Producer

• An application that sends message to Kafka

Message

• Small to medium sized piece of data

Consumer

• An application that reads data from Kafka

Cluster

• A group of computers sharing workload for a common purpose

Topic

• A topic is a unique name for Kafka stream

Partition

• Kafka topics are divided into several partitions. While the topic is a logical comcept in Kafka, a partitin is the smallest storage unit that holds a subset of records owned by a topic. Each partition is a single log file where records are written to it in an append-only fashion.

Offset

• A sequence id given to messages as they arrive in a partition

Global Unique identifier of the a message?

• Topic Name -> Partition Number -> Offfset

Consumer Group

• A group of consumers acting as a single logical unit

Can multiple kafka consumers read same message from the partition?

• It depends on group ID. Suppose you have a topic with 12 partitions. If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set of partitions = different set of messages. If you have 4 Kafka cosnumers with the same Group Id, each of them will all read three different partitions etc.
• But when you set different Group Id, the situation changes. If you have two Kafka consumers with different Group Id they will read all 12 partitions without any interference between each other. Meaning both consumers will read the exact same set of messages independently. If you have four Kafka consumers with different Group Id they will all read all partitions etc.

Within same group: NO

• Two consumers (Consumer 1, 2) within the same group (Group 1) CAN NOT consume the same message from partition (Partition 0).

Across different groups: YES

• Two consumers in two groups (Consumer 1 from Group 1, Consumer 1 from Group 2) CAN consume the same message from partition (Partition 0).

Framework Evaluation & Selection

Framework Evaluation Criteria

1. Support automation integration testing for UI (real/headless browser/device), API,  performance in cross platforms.

   • For the integration test, we don’t test the front-end or back-end individually. We need to verify the data with the integration between system, front-end & back-end. So the framework needs to be able to support all testing type and especially crossing browser, device & system.

2. Support CI integration with parallel execution.

   • The testing framework needs to be able to integrate with our current CI/CD. Parallel execution will help reduce the build time, so we will be able to deploy quickly and have faster turnaround times for bugs and features.

3. Supports the concept of executable documentation for BDD (behavior-driven development)& DDT (data-driven testing) in modularization, maintainable and understandable test suites.

   • BDD will help the test case & test suite easy to maintaining and understanding. It also helps communication between business and development is extremely focused as a result of common language.
   • One of the most important concepts for effective test automation is modularization, it will help to create a sequence of test case only one and reuse as often as required in test script without rewriting the test all the time.

BDD Testing Framework Selection

	Gauge	Cucumber
Language	Markdown	Gherkin
IDE & plugin support	Yes	Yes
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	No	No
Reusable, easy to maintain	Yes	Yes
Parallel execution	Built-in	3rd party plugin
Customize reporting	Yes	Yes
Price	Open source & free	Open source & free

Winner - Gauge:

• An open source lightweight cross-platform test automation tool with the ability to author test cases in the business language and have built-in parallel execution feature.
• Support BDD (Behavior-Driven Development) & CI(Continuous Integration) & report customization.

API Testing Framework Selection

	REST Assured	Postman
Support BDD	Yes	3rd third party
Support DDT	Yes	Limit
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	No	Yes
Reusable, easy to maintain	Yes	No
Customize reporting	Can be used with any customized/open source reporting tool.	No
Price	Open source & free	8-21$ per user/month for professional collaboration  & advanced features

Winner - REST Assured:

• An open source Java-based Domain-Specific Language (DSL) that allows writing powerful, readable, and maintainable automated tests for RESTful APIs. 
• Support testing and validating REST services in BDD (Behavior-Driven Development) / Gherkin format.

API Performance Testing Framework  Selection

	Apache JMeter	Loadrunner
Support DDT	Yes	Yes
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	Yes	No
Cross-platform	Yes	Windows, Linux
Reusable, easy to maintain	Yes	No
Customize reporting	Yes	Yes
Price	Open source & free	Free for first 50 virtual User

Winner - Apache JMeter : 

• Open source performance test runner & management framework may be used to test performance both on static and dynamic resources.
• Support load test functional behavior and measure performance. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.

Test Runner & Test Suite Management Framework Selection

	TestNG	JUnit
Support DDT	Yes	Yes
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	No	No
Reusable, easy to maintain	Yes	Yes
Parallel execution	Yes	Yes
Price	Open source & free	Open source & free
Annotation support	Yes	Limit
Suite Test	Yes	Yes
Ignore Test	Yes	Yes
Exception Test	Yes	Yes
Timeout	Yes	Yes
Parameterized Test	Yes	Yes
Dependency Test	Yes	No
Support executing before & after all tests in the suite	Yes	No
Support executing  before & after a test runs	Yes	No
Support executing  before the first & last test method is invoked that belongs to any of these groups is invoked	Yes	No

Winner - TestNG :

• Open source test runner framework which helps to run your tests in arbitrarily big thread pools with various policies available and flexible test configuration.
• Support DDT(Data-driven testing) & Test suite management.

Build Tool & Dependency ManagementFramework Selection

	Apache Maven	Grandle
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	Yes	No
Build Script Language	XML	Groovy
Reusable, easy to maintain	Yes	Yes
Dependency Management	Yes	Yes
Dependency Scopes	Built-in	Custom
IDE & plugin support	Many	A little
Price	Open source & free	Open source & free

Winner - Apache Maven 

• The leading open source dependency management and build tool. It standardizes the software build process by articulating a project’s constitution.
• Software project management and comprehension tool in the concept of a project object model (POM), Maven can manage a project's build, reporting, and documentation from a central piece of information.

All TestNG annotations with attributes

@BeforeSuite @AfterSuite @BeforeTest @AfterTest @BeforeGroups @AfterGroups @BeforeClass @AfterClass @BeforeMethod @AfterMethod		Configuration information for a TestNG class: @BeforeSuite: The annotated method will be run before all tests in this suite have run. @AfterSuite: The annotated method will be run after all tests in this suite have run. @BeforeTest: The annotated method will be run before any test method belonging to the classes inside the tag is run. @AfterTest: The annotated method will be run after all the test methods belonging to the classes inside the tag have run. @BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked. @AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked. @BeforeClass: The annotated method will be run before the first test method in the current class is invoked. @AfterClass: The annotated method will be run after all the test methods in the current class have been run. @BeforeMethod: The annotated method will be run before each test method. @AfterMethod: The annotated method will be run after each test method.
	alwaysRun	For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to. For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	enabled	Whether methods on this class/method are enabled.
	groups	The list of groups this class/method belongs to.
	inheritGroups	If true, this method will belong to groups specified in the @Test annotation at the class level.
@DataProvider		Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.
	name	The name of this data provider. If it's not supplied, the name of this data provider will automatically be set to the name of the method.
	parallel	If set to true, tests generated using this data provider are run in parallel. Default value is false.
@Factory		Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].
@Listeners		Defines listeners on a test class.
	value	An array of classes that extend org.testng.ITestNGListener.
@Parameters		Describes how to pass parameters to a @Test method.
	value	The list of variables used to fill the parameters of this method.
@Test		Marks a class or a method as part of the test.
	alwaysRun	If set to true, this test method will always be run even if it depends on a method that failed.
	dataProvider	The name of the data provider for this test method.
	dataProviderClass	The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	description	The description for this method.
	enabled	Whether methods on this class/method are enabled.
	expectedExceptions	The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
	groups	The list of groups this class/method belongs to.
	invocationCount	The number of times this method should be invoked.
	invocationTimeOut	The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.
	priority	The priority for this test method. Lower priorities will be scheduled first.
	successPercentage	The percentage of success expected from this method
	singleThreaded	If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with parallel="methods". This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called sequential (now deprecated).
	timeOut	The maximum number of milliseconds this test should take.
	threadPoolSize	The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount. Note: this attribute is ignored if invocationCount is not specified

Bitnami Jenkins Usage Steps

Jenkins, previously known as Hudson, is an open source continuous integration server. Built with Java, it provides over 400 plugins to support building and testing virtually any project. It supports SCM tools including CVS, Subversion, Git, Mercurial, Perforce and Clearcase, and can execute Apache Ant and Apache Maven based projects as well as arbitrary shell scripts and Windows batch commands. It also can monitor executions of remote tasks.

Please, take a look to the Quick Start Guide to know the basic use of this Stack.

How to start/stop the servers?

Graphical tool

The native installers include a graphical tool to manage the servers easily. You can find the "manager-windows.exe", "manager-osx" or "manager-linux" tool in your installation directory. Using this tool, you can Start, Stop or Restart the servers and check the log files. You can click on the icon to start it.

On Windows: You can also start the Manager tool from shortcuts: Start -> Program Files -> Bitnami Stack -> Manager tool

Command line tool

If you prefer, you can use the "ctlscript.sh" utility from the command line. This script is in the installation directory.

On Virtual Machines and Cloud images:

$ sudo /opt/bitnami/ctlscript.sh start

A native installer on Linux:

$ cd ~/application-version
$ ./ctlscript.sh start

On OS X: You can start the Manager tool from the installation directory or you can use the "ctlscript.sh" utility from a Terminal.

host:~ user$ cd /Applications/application-version
host:~ user$ ./ctlscript.sh start

Where is JENKINS_HOME variable pointing to?

We define JENKINS_HOME in your installation directory in the apache-tomcat/conf/Catalina/localhost/jenkins.xml file. By default it is pointing to apps/jenkins/jenkins_home.

Notice that the installation directory is /opt/bitnami in the Bitnami virtual appliances and Bitnami Amazon Images.

How to start with Git and Jenkins?

Jenkins application is running as "tomcat" user so it is necessary to configure this user in case you want to access to a private repository. If you use a public repository is not necessary to configure the Tomcat user.

Configure Tomcat user to have access to the Git repository

Previous versions of Jenkins require to create the Tomcat user folder:

$ sudo mkdir /home/tomcat
$ sudo mkdir /home/tomcat/.ssh

Then copy the private key to access to the repository into the "/home/tomcat/.ssh" folder:

$ sudo cp /path/to/id_rsa /home/tomcat/.ssh

And change the permissions

$ sudo chown -R tomcat:tomcat /home/tomcat

It is also necessary to save the hostname of your repository into the "known_hosts" file:

$ sudo su tomcat -c "/opt/bitnami/git/bin/git ls-remote -h git@YOUR_REPOSITORY.git HEAD"
The authenticity of host 'ec2-xx-yy-zz.compute-1.amazonaws.com (xx.yy.zz)' can't be established. 
ECDSA key fingerprint is e5:xx:9d:yy:ff:89:55:47:aa:fd:b5:ee:0a:pp:6f:8d.
Are you sure you want to continue connecting (yes/no)? yes

Configure Git repository in Jenkins 

Before creating the first Job, it is necessary to configure the Git user and email that Jenkins will use to clone the repository. You can configure these options in "Manage Jenkins" -> "Configure System" -> "Git plugin" section

Then create a new Job in Jenkins and specify the Git repository.

And run the Build. If everything was properly configured, Jenkins will checkout the code from the Git repository.

How to upgrade Jenkins?

If you want to upgrade Jenkins keeping your data, you just need to download the latest jenkins.war file, copy it in your apache-tomcat/webapps directory and restart Tomcat. It is recommended to create a backup of the previous jenkins.war file. You may want to revert to the previous version if you find any issues after upgrading.

Upgrading only Jenkins

On Linux and OSX you can execute the following steps to perform the above (notice that the installation directory is /opt/bitnami in the Bitnami virtual appliances and Bitnami Amazon Images):
 

$ cd

## Stop the server

$ ./ctlscript.sh stop

 

## Copy your previous .war file in case you need to restore it

$ cp apache-tomcat/webapps/jenkins.war apps/jenkins/jenkins.war.bak

 

## Remove jenkins all files to force Tomcat to deploy the new jenkins.war file

$ rm -r apache-tomcat/webapps/jenkins

$ rm -r apache-tomcat/webapps/jenkins.war

$ rm -r apache-tomcat/work/Catalina/localhost/jenkins

 

## Donwload the new version of jenkins

$ wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war

$ mv jenkins.war apache-tomcat/webapps/

 

## Start the server

$ ./ctlscript.sh start

On Windows you can stop the servers from the Start Menu entry or using the graphical tool that you can find in your installation directory. Using the file explorer go to where you installed the Bitnami Jenkins Stack. Copy the jenkins.war file from the apache-tomcat/webapps directory to apps/jenkins. Download the latest jenkins.war file and copy it in apache-tomcat/webapps. Now just start the server from the Start Menu or using the graphical tool.

Upgrading all the components

Alternatively to the process above, you can download a new Bitnami Jenkins Stack. You can install it to a different location (remember to stop the previous install so there are no port conflicts) and point it to your previous JENKINS_HOME path by editing the file  apache-tomcat/conf/Catalina/localhost/jenkins.xml.

If you are using a Bitnami virtual appliance or AMI you can start a new image with the latest jenkins version and move the content of /opt/bitnami/apps/jenkins/jenkins_home in your current installation to the new installation. Then restart the server.

How to enable SSL in Jenkins?

You can configure Apache Tomcat to enable SSL connections for accessing to the application using the HTTPS protocol. It is necessary to specify the Apache SSL certificate. If you do not have the cert you can create following these steps in the section below.

Check that you have the certificate file at "/installdir/apache-tomcat/conf/ssl/tomcat.cert.pem" and the certificate key file at "/installdir/apache-tomcat/conf/ssl/tomcat.key.pem".

To enable SSL you should uncomment the following line in the /installdir/apache-tomcat/conf/server.xml file:

8443

" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" disableUploadTimeout="true" acceptCount="100" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" keystoreFile="conf/myKeystore" keystorePass="your_password" SSLCertificateFile="installdir/apache-tomcat/conf/ssl/tomcat.cert.pem" SSLCertificateKeyFile="installdir/apache-tomcat/conf/ssl/tomcat.key.pem"/>

Then restart the Apache server and try to access at "https://localhost:8443".

How to create a SSL certificate for Jenkins?

You can find a detailed guide in the official Apache Tomcat documentation at http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html