Everyday, #Netflix handles billions of requests regarding movies, trailers and other video content. Delivering at such a large scale needs an #engineering marvel. This #video will talk about how Netflix is able to onboard new video content onto their platform. We go from video chunking to collating 4 second shots into scenes.
Amazon S3 is used to store the video chunks. Netflix also provides open connect servers to internet service providers, which acts like a cache of movies. Most requests to Netflix can be served by this cache, and the remaining are sent over the network. This reduces the bandwidth and time required for Netflix to operate at scale. Synergy at it's finest.
Video Formats and Resolutions
Different formats
High quality
Medium quality
Low quality
Different resolutions
1080p
720p
480p
Storage combination comes F x R -> V
What netflix does is like broken down a video into smaller parts so that it can be deal with effectively per processor
One resolution, one format, one chunk - that's one task
Chunk Processing
What they are doing intelligently is breaking the chunks not based on timestamps but based on scenes to have seamless user experience.
Based os scenes so you can make instead of 3 min thing, you can make it more fine grained 4 sec each, it's called a shot and you can collate shot, put them all together to create a scene.
Prediction algorithm is pretty smart to understand if user is not watching with engaged mode and clicking fwd to see the movie, instead of giving the whole content, it gives only data to the user has asked for because they are probably clicking on different points in that buffer you get.
On the other hand, if user if watching in engaged mode, so instead of sending just the part user has asked for, it redictively proactively fetches the future parts, gets onto your computer and shows it you.
Storage
Amazon S3 is what Netflix uses to store that video content.
This is where people store their static data meaning that you don't change that data.
It's extremely cheap compared to a database.
OpenConnect for video caching
Netflix servers are usually in the U.S which means they are geographically concentrated and in a place like India which is really far, it's going to take lot of time to send the signal and receive it especially if it's video because there is lot of data which is going to be coming in and it's going to be slow.
What Netflix did intelligently to extend the concept of caching and apply it to ISP's. When request comes to ISP for the movie, it looks for local cache say cache for Indian movies.
Cache has been called as OpenConnect
Lots of bandwidth saved
Lots of time saved
Much better user experience
90% of the Netflix traffic is taken care by these ISP boxes that they provide.
W.r.t new content overnight job can be run to copy the new contents when there is less load on server for the requests.
Support automation integration testing for UI (real/headless browser/device), API, performance in cross platforms.
For the integration test, we don’t test the front-end or back-end individually. We need to verify the data with the integration between system, front-end & back-end. So the framework needs to be able to support all testing type and especially crossing browser, device & system.
Support CI integration with parallel execution.
The testing framework needs to be able to integrate with our current CI/CD. Parallel execution will help reduce the build time, so we will be able to deploy quickly and have faster turnaround times for bugs and features.
Supports the concept of executable documentation for BDD (behavior-driven development)& DDT (data-driven testing) in modularization, maintainable and understandable test suites.
BDD will help the test case & test suite easy to maintaining and understanding. It also helps communication between business and development is extremely focused as a result of common language.
One of the most important concepts for effective test automation is modularization, it will help to create a sequence of test case only one and reuse as often as required in test script without rewriting the test all the time.
BDD Testing Framework Selection
Gauge
Cucumber
Language
Markdown
Gherkin
IDE & plugin support
Yes
Yes
Easy to integrate with CI/CD
Yes
Yes
Easy to use, quick to learn
No
No
Reusable, easy to maintain
Yes
Yes
Parallel execution
Built-in
3rd party plugin
Customize reporting
Yes
Yes
Price
Open source & free
Open source & free
Winner - Gauge:
An open source lightweight cross-platform test automation tool with the ability to author test cases in the business language and have built-in parallel execution feature.
Support BDD (Behavior-Driven Development) & CI(Continuous Integration) & report customization.
API Testing Framework Selection
REST Assured
Postman
Support BDD
Yes
3rd third party
Support DDT
Yes
Limit
Easy to integrate with CI/CD
Yes
Yes
Easy to use, quick to learn
No
Yes
Reusable, easy to maintain
Yes
No
Customize reporting
Can be used with any customized/open source reporting tool.
No
Price
Open source & free
8-21$ per user/month for professional collaboration & advanced features
Winner - REST Assured:
An open source Java-based Domain-Specific Language (DSL) that allows writing powerful, readable, and maintainable automated tests for RESTful APIs.
Support testing and validating REST services in BDD (Behavior-Driven Development) / Gherkin format.
API Performance Testing Framework Selection
Apache JMeter
Loadrunner
Support DDT
Yes
Yes
Easy to integrate with CI/CD
Yes
Yes
Easy to use, quick to learn
Yes
No
Cross-platform
Yes
Windows, Linux
Reusable, easy to maintain
Yes
No
Customize reporting
Yes
Yes
Price
Open source & free
Free for first 50 virtual User
Winner - Apache JMeter :
Open source performance test runner & management framework may be used to test performance both on static and dynamic resources.
Support load test functional behavior and measure performance. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.
Test Runner & Test Suite Management Framework Selection
TestNG
JUnit
Support DDT
Yes
Yes
Easy to integrate with CI/CD
Yes
Yes
Easy to use, quick to learn
No
No
Reusable, easy to maintain
Yes
Yes
Parallel execution
Yes
Yes
Price
Open source & free
Open source & free
Annotation support
Yes
Limit
Suite Test
Yes
Yes
Ignore Test
Yes
Yes
Exception Test
Yes
Yes
Timeout
Yes
Yes
Parameterized Test
Yes
Yes
Dependency Test
Yes
No
Support executing before & after all tests in the suite
Yes
No
Support executing before & after a test runs
Yes
No
Support executing before the first & last test method is invoked that belongs to any of these groups is invoked
Yes
No
Winner - TestNG :
Open source test runner framework which helps to run your tests in arbitrarily big thread pools with various policies available and flexible test configuration.
Support DDT(Data-driven testing) & Test suite management.
The leading open source dependency management and build tool. It standardizes the software build process by articulating a project’s constitution.
Software project management and comprehension tool in the concept of a project object model (POM), Maven can manage a project's build, reporting, and documentation from a central piece of information.
Mountebank is the first open source tool to provide cross-platform, multi-protocol test doubles over the wire. Simply point your application under test to mountebank instead of the real dependency, and test like you would with traditional stubs and mocks.
Basically it provides a server that can be configured using a DSL to simulate requests and responses over variety of protocols (http, https, tcp and smtp).
One of the hardest part in learning Spring is to understandwhy do you need springandwhat is it exactly. Once this is clear, rest all becomes easy. Spring is generally described as ” A light weight framework for building Java applications”. This means any type of Java application – standalone java, JEE applications , Web applications etc. The light weight means you have to make only a few change or none at all to your application code to get the benefits of Spring. This is very important. You don’t need to extend your classes from any particular Spring framework component to get the framework features. This statement is applicable to Spring core.
Spring provides IOC ( Inversion of Control) and DI ( Dependency Injection ) Capabilities. Minimizing the Dependencies is one of most important factors in creating maintainable, extensible software and Spring solves this beautifully. Let us explore IOC and Dependency Injection in the next section.
Inversion of Control ( IOC )
IoC is a technique that externalizes the creation and management of component dependencies. Inversion of Control is best understood through the term the “Hollywood Principle,” which basically means“Don’t call me, I’ll call you.”
Consider the case where Class Persondepends on an instance ofClass Address. In traditional programming practice, Class Person either instantiates an instance of Address or Class Person uses a factory class to get the instance of Address. With the IOC practice, an instance of Address is given to Person at run time by the framework. Thus the dependency is injected by the framework at run time and hence it is also calledDependency Injection. Effectively Inversion of Control is a means to achieve Dependency Injection. Many people use IOC and Dependency Injection interchangeably. In fact Spring acts like a container providing your application classes with all dependencies it needs.
Let us explore the beauty of Dependency Injection with an example
In the above Code, we simply tell spring that Person requires an instance of Address and that dependency should be satisfied using the setter method annotated with @Autowire.
@Autowired will instruct Spring framework to look for a bean that implements the required interface and inject it automatically via the setter.
The spring framework will instantiate a valid instance of Address and wire it to the Person class. This is the beauty of Dependency Injection / IOC. There is no plumbing code put by the developer to satisfy the dependency. In my opinion IOC/ DI is the most important reason for the popularity and usefulness of Spring framework. Spring framework provides better means of non intrusive, plug-gable programming model.
There is more to Spring – it also provides declarative programming withAOP ( Aspect Oriented Programming). This is a better way to implement cross cutting concerns without the needs to use plumbing code all over the core business classes.
AOPenables you to think about concerns or aspects in your system. Typical concerns are transaction management, logging etc. AOP enables you to capture the cross-cutting code in modules such as interceptors that can be applied declaratively wherever the concern they express applies.Spring includes a proxy-based AOP framework.
But is that all that Spring offers ?. The answer is a BIG NO.Spring provides a lot of components that help with data access / integration, Web programming and other common programming requirements.
Spring also has a number of projects than utilizes the spring core and makes programming easier and cleaner. Few of the Spring projects are
Spring Batch
Spring AMQP
Spring Boot
Spring Data Commons
Spring Data GemFire
Spring Data JDBC Extensions
Spring Data JDBC Extensions
Spring Data JPA
Spring Data MongoDB
Spring Data Neo4J
Spring Data Redis
Spring Data REST
Spring Data Solr
Spring Flex
Spring for Android
Spring for Apache Hadoop
Spring Framework
Spring HATEOAS
Spring Integration
Spring LDAP
Spring Mobile
Spring Roo
Spring Security
Spring Security OAuth
Spring Shell
Spring Social
Spring Social Facebook
Spring Social Twitter
Spring Web Flow
Spring Web Services
Basically Spring is a framework fordependency-injectionwhich is a pattern that allows to build very decoupled systems. I'll try to explain you the simplest I can (this isn't a short answer).
The problem
For example, suppose you need to list the users of the system and thus declare an interface calledUserLister:
public interface UserLister {
List getUsers();
}
And maybe an implementation accessing a database to get all the users:
public class UserListerDB implements UserLister {
public List getUsers() {
// DB access code here
}
}
In your view you'll need to access an instance (just an example, remember):
public class SomeView {
private UserLister userLister;
public void render() {
List users = userLister.getUsers();
view.render(users);
}
}
Note that the code above doesn't have initialized the variableuserLister. What should we do? If I explicitly instantiate the object like this:
UserLister userLister = new UserListerDB();
...I'd couple the view with my implementation of the class that access the DB. What if I want to switch from the DB implementation to another that gets the user list from a comma-separated file (remember, it's an example)? In that case I would go to my code again and change the last line by:
UserLister userLister = new UserListerCommaSeparatedFile();
This has no problem with a small program like this but... What happens in a program that has hundreds of views and a similar number of business classes. The maintenance becomes a nightmare!
Spring (Dependency Injection) approach
What Spring does is towirethe classes up by using a XML file, this way all the objects are instantiated and initialized by Spring andinjectedin the right places (Servlets, Web Frameworks, Business classes, DAOs, etc, etc, etc...).
Going back to the example in Spring we just need to have a setter for theuserListerfield and have an XML like this:
This way when the view is created itmagicallywill have aUserListerready to work.
List users = userLister.getUsers(); // This will actually work
// without adding any line of code
It is great! Isn't it?
What if you want to use another implementation of yourUserListerinterface?Just change the XML
What if don't have aUserListerimplementation ready?Program a temporal mock implementation ofUserListerand ease the development of the view
What if I don't want to use Spring anymore?Just don't use it! Your application isn't coupled to it.Inversion of Controlstates: "The application controls the framework, not the framework controls the application".
There are some other options for Dependency Injection around there, what in my opinion has made Spring so famous besides its simplicity, elegance and stability is that the guys of SpringSource have programmed many many POJOs that help to integrate Spring with many other common frameworks without being intrusive in your application. Also Spring has several good subprojects like Spring MVC, Spring WebFlow, Spring Security and again a loooong list of etceteras.
Hibernate provides three different ways to retrieve data from database. TheCriteria Query APIlets you build nested, structured query expressions in Java, providing a compile-time syntax checking that is not possible with a query language like HQL or SQL. The Criteria API also includesquery by example (QBE)functionality. This lets you supply example objects that contain the properties you would like to retrieve instead of having to step-by-step spell out the components of the query. It also includes projection and aggregation methods, includingcount(). Let’s explore it’s different features in detail.y
Basic Usage Example
The Criteria API allows you to build up a criteria query object programmatically; theorg.hibernate.Criteriainterface defines the available methods for one of these objects. The HibernateSessioninterface contains severalcreateCriteria()methods. Pass the persistent object’s class or its entity name to thecreateCriteria()method, and Hibernate will create aCriteriaobject that returns instances of the persistence object’s class when your application executes a criteria query.
The simplest example of a criteria query is one with no optional parameters or restrictions—the criteria query will simply return every object that corresponds to the class.
Moving on from this simple example, we will add constraints to our criteria queries so we can whittle down the result set.
Using Restrictions with Criteria
The Criteria API makes it easy to use restrictions in your queries to selectively retrieve objects; for instance, your application could retrieve only products with a price over $30. You may add these restrictions to aCriteriaobject with theadd()method. Theadd()method takes anorg.hibernate.criterion.Criterionobject that represents an individual restriction. You can have more than one restriction for a criteria query.
i) Restrictions.eq() Example
To retrieve objects that have a property value that “equals” your restriction, use theeq()method onRestrictions, as follows:
Above query will search all products having description anything but not “Mouse”.
You cannot use the not-equal restriction to retrieve records with a NULL value in the database for that property (in SQL, and therefore in Hibernate, NULL represents the absence of data, and so cannot be compared with data). If you need to retrieve objects with NULL properties, you will have to use theisNull()restriction.
iii) Restrictions.like() and Restrictions.ilike() Example
Instead of searching for exact matches, we can retrieve all objects that have a property matching part of a given pattern. To do this, we need to create an SQL LIKE clause, with either thelike()or theilike()method. Theilike()method is case-insensitive.
Above example uses anorg.hibernate.criterion.MatchModeobject to specify how to match the specified value to the stored data. TheMatchModeobject (a type-safe enumeration) has four different matches:
ANYWHERE: Anyplace in the string
END: The end of the string
EXACT: An exact match
START: The beginning of the string
iv) Restrictions.isNull() and Restrictions.isNotNull() Example
TheisNull()andisNotNull()restrictions allow you to do a search for objects that have (or do not have) null property values.
v) Restrictions.gt(), Restrictions.ge(), Restrictions.lt() and Restrictions.le() Examples
Several of the restrictions are useful for doing math comparisons. The greater-than comparison isgt(), the greater-than-or-equal-to comparison isge(), the less-than comparison islt(), and the less-than-or-equal-to comparison isle(). We can do a quick retrieval of all products with prices over $25 like this, relying on Java’s type promotions to handle the conversion toDouble:
Moving on, we can start to do more complicated queries with the Criteria API. For example, we can combine AND and OR restrictions in logical expressions. When we add more than one constraint to a criteria query, it is interpreted as an AND, like so:
If we want to have two restrictions that return objects that satisfy either or both of the restrictions, we need to use theor()method on the Restrictions class, as follows:
The orExp logical expression that we have created here will be treated like any other criterion. We can therefore add another restriction to the criteria:
If we wanted to create an OR expression with more than two different criteria (for example, “price > 25.0 OR name like Mou% OR description not like blocks%”), we would use anorg.hibernate.criterion.Disjunctionobject to represent a disjunction.
You can obtain this object from thedisjunction()factory method on theRestrictionsclass. The disjunction is more convenient than building a tree of OR expressions in code. To represent an AND expression with more than two criteria, you can use theconjunction()method, although you can easily just add those to theCriteriaobject. The conjunction can be more convenient than building a tree of AND expressions in code. Here is an example that uses the disjunction:
sqlRestriction()restriction allows you to directly specify SQL in the Criteria API. It’s useful if you need to use SQL clauses that Hibernate does not support through the Criteria API.
Your application’s code does not need to know the name of the table your class uses. Use {alias} to signify the class’s table, as follows:
crit.add(Restrictions.sqlRestriction("{alias}.description like 'Mou%'"));
List results = crit.list();
Paging Through the ResultSet
One common application pattern that criteria can address is pagination through the result set of a database query. There are two methods on theCriteriainterface for paging, just as there are for Query:setFirstResult()andsetMaxResults(). ThesetFirstResult()method takes an integer that represents the first row in your result set, starting with row 0. You can tell Hibernate to retrieve a fixed number of objects with thesetMaxResults()method. Using both of these together, we can construct a paging component in our web or Swing application.
As you can see, this makes paging through the result set easy. You can increase the first result you return (for example, from 1, to 21, to 41, etc.) to page through the result set.
Obtaining a Unique Result
Sometimes you know you are going to return only zero or one object from a given query. This could be because you are calculating an aggregate or because your restrictions naturally lead to a unique result. If you want obtain a single Object reference instead of a List, theuniqueResult()method on theCriteriaobject returns an object or null. If there is more than one result, theuniqueResult()method throws aHibernateException.
The following short example demonstrates having a result set that would have included more than one result, except that it was limited with thesetMaxResults()method:
Again, please note that you need to make sure that your query returns only one or zero results if you use theuniqueResult()method. Otherwise, Hibernate will throw aNonUniqueResultExceptionexception.
Obtaining Distinct Results
If you would like to work with distinct results from a criteria query, Hibernate provides a result transformer for distinct entities,org.hibernate.transform.DistinctRootEntityResultTransformer, which ensures that no duplicates will be in your query’s result set.Rather than using SELECT DISTINCT with SQL, the distinct result transformer compares each of your results using their defaulthashCode()methods, and only adds those results with unique hash codes to your result set. This may or may not be the result you would expect from an otherwise equivalent SQL DISTINCT query, sobe careful with this.
An additional performance note: the comparison is done in Hibernate’s Java code, not at the database, so non-unique results will still be transported across the network.
Sorting the Query’s Results
Sorting the query’s results works much the same way with criteria as it would with HQL or SQL. The Criteria API provides theorg.hibernate.criterion.Orderclass to sort your result set in either ascending or descending order, according to one of your object’s properties.
This example demonstrates how you would use the Order class:
You may add more than oneOrderobject to theCriteriaobject. Hibernate will pass them through to the underlying SQL query. Your results will be sorted by the first order, then any identical matches within the first sort will be sorted by the second order, and so on. Beneath the covers,Hibernate passes this on to an SQL ORDER BY clause after substituting the proper database column name for the property.
Performing Associations (Joins)
The association works when going fromeither one-to-many or from many-to-one. First, we will demonstrate how to use one-to-many associations to obtain suppliers who sell products with a price over $25. Notice that we create a new Criteria object for the products property, add restrictions to the products’ criteria we just created, and then obtain the results from the supplier Criteria object:
suppCrit.add(Restrictions.eq("name","Hardware Are We"));
List results = crit.list();
Adding Projections and Aggregates
Instead of working with objects from the result set, you can treat the results from the result set as a set of rows and columns, also known as a projection of the data. This is similar to how you would use data from a SELECT query with JDBC.
To use projections, start by getting theorg.hibernate.criterion.Projectionobject you need from theorg.hibernate.criterion.Projectionsfactory class. TheProjectionsclass is similar to theRestrictionsclass in that it provides several static factory methods for obtainingProjectioninstances. After you get aProjectionobject, add it to your Criteria object with thesetProjection()method. When theCriteriaobject executes, the list contains object references that you can cast to the appropriate type.
Example 1 : Single Aggregate ( Getting Row Count )
Other aggregate functions available through theProjectionsfactory class include the following:
avg(String propertyName): Gives the average of a property’s value
count(String propertyName): Counts the number of times a property occurs
countDistinct(String propertyName): Counts the number of unique values the property contains
max(String propertyName): Calculates the maximum value of the property values
min(String propertyName): Calculates the minimum value of the property values
sum(String propertyName): Calculates the sum total of the property values
Example 2 : Multiple Aggregates
We can apply more than one projection to a given Criteria object. To add multiple projections, get a projection list from theprojectionList()method on the Projections class. Theorg.hibernate.criterion.ProjectionListobject has anadd()method that takes aProjectionobject. You can pass the projections list to thesetProjection()method on theCriteriaobject becauseProjectionListimplements theProjectioninterface.
Another use of projections is to retrieve individual properties, rather than entities. For instance, we can retrieve just the name and description from our product table, instead of loading the entire object representation into memory.
In QBE, instead of programmatically building aCriteriaobject withCriterionobjects and logical expressions, you can partially populate an instance of the object. You use this instance as a template and have Hibernate build the criteria for you based upon its values. This keeps your code clean and makes your project easier to test.
For instance, if we have a user database, we can construct an instance of a user object, set the property values for type and creation date, and then use theCriteriaAPI to run a QBE query. Hibernate will return a result set containing all user objects that match the property values that were set. Behind the scenes, Hibernate inspects the Example object and constructs an SQL fragment that corresponds to the properties on the Example object.
The following basic example searches for suppliers that match the name on the example Supplier object:
Using the Criteria API is an excellent way to get started developing with HQL. The developers of Hibernate have provided a clean API for adding restrictions to queries with Java objects. Although HQL isn’t too difficult to learn, some developers prefer the Criteria Query API, as it offers compile-time syntax checking—although column names and other schema-dependent information cannot be checked until run time.
