How to do NFR (Non Functional Requirement) Testing

 

What is NFR testing?

NFR testing serves to validate that your system meets all the non-functional requirements (e.g concurrent requests, transactions per second, response times, sub-service failures etc) expected of it.

Why is NFR testing so important?

Today, applications and the eco-systems in which they run have changed drastically. Older applications used to run in closed, barely-distributed, named-(pet)-system environments where everything was most probably largely within your control. Now, with the arrival of clouds and microservices the eco-system has changed drastically, and the importance of testing NFRs has risen considerably.

Why should you care?

I think you should care because:

1) It’s important - actually very important : Its importance can simply be understood from the point of view that if a system can fulfill all its functional requirements, but not fulfill is just one critical NFR (be it related to capacity, speed, security, availability, scalability etc.), then that system will soon get outdated or fail to deliver at all in today’s fast-paced digital world.

2) It’s free : Its free in the sense that you don’t have to invest heavily (for tools or manpower) to do NFR testing. It’s best done by the developers themselves, with freely available open source tools.

3) It’s fun : It is fun for the developers to test the system they build; to see how it performs in the actual world and fine tune, improve and re-engineer their code before it’s too late.

NFR testing steps

Now you know what NFR testing is as well as its importance and why you should care about it, let me explain how you might do it.

0) Be Agile - Don’t wait till the end.

NFR testing should be planned from the beginning of the project as NFRs can have a big effect on the coding/architecture of your application.

Suppose you have an NFR which states that your application should handle very high traffic of say 1000 requests per second. Now, if you start NFR testing from the beginning of the project, you may come to know early in the development cycle that your application architecture can or can’t handle it. This may lead to changes in the code or design or adoption of some coding practice which allows you to achieve this. For example, you may have to use thread pools to minimize the time in creating and spawning new threads. You may also use multi casting and aggregation patterns, or even swap out one framework for another to achieve better response times.

Say another NFR states that the system should not be overloaded in case of failure in any one components of the system. Again, if you test this NFR from the beginning you can find whether your system can cope with this requirement or not. You may decide to build your application to fail fast in case of any error using the hystrix circuit breaker from Netflix.

The above example clearly show that if we are Agile (and early) with our NFR testing then it can assist in verifing the coding approach / architecture of our application and help us to change if required early in the development cycle; thereby saving us a lot of time and money later.

1) Plan

Being Agile in your NFR testing is not enough - you have to plan for it properly. Planning is important as will help to decide which NFR tests to run when you get constrained by time or resources. Believe me - you will. When Planning your NFR testing, you should consider the following points to get the most out of your NFR testing:

a) Be requirement-driven - Locate all NFRs and map them to your tests.

To better understand this, when creating an NFR test plan make sure you map all your NFR tests with the corresponding NFRs.

Suppose you created a test in which you are going to verify that in the circumstance where one of your sub-services fails, your system fails fast without overloading. This is a really important and good test as it validates your system resilience in case of failures and also how your system resources are used in such a scenario.

But if you can map this with a corresponding NFR which states ‘Failure to connect or receive a response from the Sub-Service is handled gracefully, logged, and reported back to the consuming application within X seconds of the call’ then not only will this achieve the above said goals, it will also help you to provide your stakeholders with statistics and data for the corresponding NFR and boosting their confidence in the system being built.

b) Be fear driven too – If something is new, test the hell out of it

Suppose you have a system which uses MongoDB 2.6 as the database and now you’re told to upgrade to MongoDB 3.0. In this scenario MongoDB 3.0 is new to your solution and may impact you in unknown ways - most probably in a positive way, but you don’t know. There will be always be concern/fear about the effect of this change. You should address this concern/fear with high priority. It is therefore better to run all NFR tests which are related to your Database as compared to other NFR test which are non related such as sub-service failure.

c) Prioritize

When you define your NFR tests, make sure you prioritize them too, as you may/will be constrained by time and manpower.

Let say you have three NFR tests:

1. DB failover and recovery
2. Handling transaction at 100tps
3. Graceful handling of Sub service failure.

Now, taking the above example, since your last test you have recently changed your Database from MongoDB 2.6 to MongoDB 3.0. Suppose when the system was built all the NFRs were “MUST”, but in this scenario you may not have enough time to re-run them all, so you have to prioritize them. In this example, it is clear that we “MUST” run NFR 1 and 2 as they specifically depend on the Database. NFR 3 can be termed as nice-to-have, as the system has not changed this test is less likely to be affected.

NFRs can also be prioritized with respect to an environment you are running them. Suppose you have three Database related NFRs:

1. DB failover and recovery
2. DB Handling transaction at 100tps
3. Data center failure and recovery.

Now NFR 1 and 2 were already tested on the Dev and Test environments, but NFR 3 was not tested, as these environments don’t have the infrastructure to support it. Well, Pre-prod does have this infrastructure, so when doing NFR testing on pre-prod, you should prioritize the NFR 3 as this is the one which you have never tested before and has a high significance in production.

2) Setup

Setting up the proper environment for the your NFR testing is very important as it may nullify/invalidate your testing or create doubt around the test results. Consider the points below when setting up your NFR test environment.

a) Only prod-like truly counts - in Config, Data, Platform and Monitoring

When setting up your NFR test environment you should make sure it has the same configuration, physical architecture and monitoring as you will have in production. (Or as close to it as you can get to tell you what you want to know.) If any aspect of this (CPU, RAM, network settings / config) deviates from the production configuration, then results which you will get by running your test will be skewed in some way and be less valid when you present them to your stakeholders.

Take the example of NFR where your system is supposed to handle 100tps. Perhaps you set up your environment with 2 CPU with 4 cores. But, your production haS 4 CPUs and 8 cores. In this circumstance, any test which you will run could have two outcomes:

1. you will get good results (system meets or exceeds your NFR target)
2. you will get bad results (system fails to meet your NFR target).

In the case of a good result, you should be happy that you have written very optimized code which is working good on limited hardware. But, if get a bad result you may unnecessarily suspect that there is some issue with your application. You may waste time and manpower in optimizing code which is already optimized, but the result is skewed due to hardware limitation.

Also, if you reverse the scenario and use a more superior configuration in your test environment than the production box you might gain a false sense of security. You can easily understand the impact.

b) N.b. Run-state - Normal? Peak Load? Failure/DR scenario? Latency?

A system can operate under different conditions and loads. You ideally set up your test environment to simulate all possible conditions (normal load, peak load, failure scenario and n/w latency etc). Environment setup may also include creating data, tests and scripts to simulate these conditions. You should note down the run-state as part of NFR result. Run your NFR tests in all such environments to prove the predictable functioning of the system.

c) Be repeatable - config, dataset, request load, and scenario steps

You should ensure that test environment setup should be easily configurable and repeatable. If your test requires some data to be preloaded into the system, then that data should be easy to re-create and should be available to anyone going to run the test. All configuration files should be stored in a central repository so that anyone running the tests can access them as and when required.

3) Perform - Run your NFR tests

The next obvious thing after planning, prioritizing your test runs and setting up the test environment is to actually run them. Again there are several aspects to this:

a) Request Mix - Use case & background, success & failure

When running your NFR tests make sure they reflect the real use cases as closely as possible. Suppose, your system caters for four types of request (add, update, delete and read). As we can see the nature of each of these requests are different, and can have a significant effect on overall response times. Now suppose your test is to validate the NFR that you must support 100tps, then your NFR test should not just send one type of request at 100tps, but it should send a request mix of all four request types. You should also take care of the percentage of each request type to be sent for proper request mix (e.g. 20% add, 5% update, 1% delete and 74% Read. You can get this info from your product owner, a user or a business analyst). You should also consider sending bad and invalid requests mixed into your test load to simulate the real life scenario where not everything is going according to plan elsewhere.

There may also be a test to just send successful requests, in such scenario also you should pass all the four types of request with agreed percentage for each type.

This brings up a key point. Request mix is very important, as it doesn’t just validate that your system can handle all sorts of requests, but also how your system behaves when all sorts of requests come at the same time. A lack of request mix can skew your request and give a false impression of high performance. For example, if you just send read requests, the response time will be quick, but if you send all add requests the response time will be longer than the read. And if you add in some bad requests, resultant error handling and compensatory transactions, then you get a very different picture.

So, take care of the request mix when creating test case for your NFRs.

b) Load - N.b Req/sec & number of users

Again the transactions per second and the number of concurrent users have a huge impact on the performance of the system, hence take care with it and make sure your run your test with the same transactions per second specified in NFRs or what you expect in Production. Suppose you are aware that your system will at max hit with 6tps in Production (tps being controlled by your load balancer), then you should run your test at 6tps only.

c) Events - Fail/Slow down & Recover/Speed up.

You will have NFRs related to system failure/recovery or slowdown/speedup. So, your tests should be run under conditions of sub-system failure and recovery, system slow down and speed up to cover these NFRs.

4) Record

So you’ve run your tests and have your results - you’re done right? Well not really. Recording your NFR test results is a must, without it you will never be able to prove your system performance and go back to it in the future when you learn something new. The following are some basic guidelines about what you should record in your NFR findings and how.

a) Measure - Throughput and Response time (inc. the percentiles).

The key elements to measure in your NFR testing are most usually throughput and response time, and to view these in relation to the 95th and 99th percentiles. Percentiles are important too (especially in a microservice based system) so its important you know about how they work. A good introduction can be found on Wikipedia.

b) Write up using templates - Recording the same thing, in the same way, every time

The best way to record test results is to use templates. Create a generic template which covers what what you want to capture and use it whenever recording the results. Templates gives many benefits, a few of them are:

• create once, use many times
• they help to easily compare the results
• you get consistency
• they are easy to understand

Recording your NFR test results is important, and it is most important to record the measured variables and environment settings used after each run (e.g throughput, response time, JVM and CPU usage, http and tomcat thread pool and response codes). If you don’t record the same set of parameters in each run it will become hard to validate the test result and see any improvement or side effect after each new release of code.

A practical example will be, for NFR test of 1000req/sec you ran a first test and capture the throughput, response time and CPU and Memory usage and got values of 70tps, 2000ms per request average, and with an average CPU usage of 30% and average Memory usage of 20%. Now to improve the throughput you decided to use more http and tomcat threads. This time you captured only the throughput and response times (100tps and 1800ms). This seems to be good and you have achieved your NFR of 100tps. But you have not captured the CPU or Memory related information. To your surprise the CPU usage was very high 90% and Memory use was almost 100%, which is not desirable state of system are normal load. Therefore, your approach to increase the thread pool was not effective as you expected. This can trip you up later on when you scale even more.

So remember to capture the same set of data every time and have a fixed template to captured it.

c) Save Evidence - logs, Graphs etc.

It is also important to capture the NFR test results in the raw, as this is the data used to validate your findings. There is no point in just writing on a wiki page saying that you ran the test and achieved the throughput of 100tps with response time being less then 1 sec with no errors, until you can back it up with some sort of data (logs, graphs, etc.).

If you can capture logs and graphs from a test tool such as JMeter, this is strong evidence that can be very helpful in supporting your conclusions.

d) Summarise - For the TL;DR crew

In the end, you should summarise the NFR test result to your TL;DR crew. If they can understand and agree with the NFR test results then most probably your stakeholders will too.

5) Investigate

One of the most important tasks of NFR testing is not just to run the tests and capture the data, but also to capture all failures and errant behaviours of the system and attribute explanations to each of them.

a) Attribute all errors - write them up too

Every system is supposed to throw errors, but if we don’t have an explanation for each, then the system is unreliable. So you should make sure that not only do you capture the errors, but also that you attribute them each with a proper explanation. Without a valid explanation for failures your NFR testing becomes invalid. If you don’t have any explanation for the error or errant behaviour of the system, then you’ll need to investigate. This may lead to code changes, documentation updates or simply a deeper understanding of the system. More importantly it can lead to finding bugs in system or at least tuning of the logs being captured. Again, going back to the example of the 100tps NFR. When running this test, one of your requests failed. If you don’t provide any explanation for this error, your NFR test becomes invalid. But if you check the logs and you find an error related to “tcp connection timeout”, then you can attribute that failure to this error and can proceed.

b) Re-run - only changing one thing at a time

If your investigation requires, you should re-run your test to validate your findings. It is good practice to level up the logging when investigating. When it comes to configuration changes, make sure you change only one parameter at a time, to verify the effect of the changes made. To raise the capacity from an observed 75tps to 100tps, suppose you changed the default thread pool size of 10 to 20 and also modified the memory setting from 200mb to 500mb; here you have modified two things. Now, suppose after his change you were able to get 100tps. So, what will you do? will you keep 20 as default thread pool size or memory size to 500mb. You can’t decide until you know the effect of each change individually. So the best way is to make one change at a time and re-run your test to verify the effect of the change made.

6) Improve

NFR testing is not just to check that your application meets NFR specification but also to improve your application. NFR testing always leads to finding bottlenecks in a system and help to improve it. When improving the system we should use the helpers below.

a) Tune - pool sizes, timeouts, queries, caches, indexes etc.

When improving/tuning the system to meet NFRs specification, you should start with configurable settings such as thread pool size, timeouts, queries, caches etc. A code change should be amongst your last options unless you can clearly associate an issue with it.

b) N.b Run the slowest step.

When it come to tuning, it is always good to find the slowest bit and tune that. If you are able to fine tune this then most probably other bits too will improve. This is not a universal law - sometimes this can lead to a flood cascading on downstream elements which in turn has an even more detrimental effect on performance. The best way to find the slowest bit is to view the response time graphs for each request. Through it you can easily figure out which service is taking more time and you can start tuning it.

c) Fix - Bugs, slowness, weakness etc.

When running NFR tests, if you find any error either there will be a valid explanation for it or there will be some bug which is causing it. In the latter case, fix it. Similarly, if your system is slow and does not meet your NFR specification, then to look for improving the system as explained in 6.a.

d) N.b your NFR target - Don’t get carried away if NFRs == met, break

The key thing to note when improving the system is, if your system is able to meet your NFRs, we should not over-optimize it. Suppose, when you were running a NFR test at 30tps you were getting Hystrix thread denied for thread group size of 10. Then, you changed it to thread group size to 15 and it was working fine. Now there is no need to over optimize and test it whether it works with thread group size of 11, 12, 13 -that is unnecessary.

7) Goto 0

If there are still some NFRs to be run, repeat the cycle.

API Security Testing - How we can do/stop hacking to the APIs : Part-3

 

The first two parts in this series have set us well on our path to API security/intrusion nirvana; we first got to grips with current API technology basics, moved on to API Attack surface detection and then looked at a series of common and potent vulnerability attacks – all sharing the fact that they are pretty simple to perform using a little wit and available tools. Let’s dig a little more in our arsenal before concluding with some hands on tips on how to work with API Security Testing.

Cross Site Request Forgery (CSRF)

The last vulnerabilities we looked at in the previous installment were related to cross-site scripting attacks (XSS). Now, let’s have a look at another vulnerability / attack that plays with similar cards; cross-site request forgery. Just like XSS, this is an attack that originates from a web-browser – but since it often follows through with an API call it is definitely something you should be aware of:

Let’s walk through this so to provide an understanding of how this type of attack works:

• A user logs in to his/her bank account – which sets a session cookie on the client
• The bank shows a (malicious) ad for the latest vitamin pills, which the user clicks on without first logging out of the bank
• The page selling vitamin pills also contains a hidden form performing a bank-transfer, which is submitted automatically without the user noticing
• Since the user is still logged in to the bank the “forged request” succeeds and makes the transfer without the user noticing.

While this example may seem somewhat contrived (what bank shows vitamin ads?), I’m sure you get the idea, and I’m also sure I’m not the only one that has navigated away from an authenticated site without logging out first.

A common way to attempt to prevent these attacks is to look for specific http headers related to request referral, but headers can easily be spoofed using various techniques described online. The much better way to guard for these attacks from a development point-of-view is therefore to require a unique request token to be added to each incoming request to an API – either as an HTTP header or as a query parameter. The value is generated by the server when preparing the underlying webpage using a secret algorithm - which the attacker presumably won’t know – and thus won’t be able to.

Testing this becomes straightforward; are tokens enforced and validated correctly? What if you use one of the previously discussed techniques for the token itself; fuzzing, injection, etc.? Is this all handled gracefully or does an error message give away a little too much about how things are being done on the server side?

Insufficient SSL Configuration

Let’s shift our focus a little and have a look at some common vulnerabilities and attack opportunities that relate to the security stack of your API itself.

Using a self-signed SSL certificate is an easy mistake to make – the convenience of generating your own certificate is high – but the dangers are lurking, especially when it comes to APIs; if your app is the only one consuming your API your developers might even have hard-coded the client to accept the certificate – and then forgot all about it. Unfortunately, using a self-signed certificate leaves your API wide open for man-in-the-middle attacks – where an attacker inserts him/herself between the API and its client by replacing the client-facing certificate – allowing them to listen in on (and modify) traffic that was meant to be encrypted.

Testing for insufficient SSL configurations is straightforward – make sure your tests accept only valid certificates. Taking an extra precaution for MITM attacks is also advisable – for example by adding signatures to a message, which makes it impossible (well, almost) for a eavesdropper to modify messages on the wire, even if they manage to insert themselves in the communication pipeline. Testing with signatures, that they are enforced, correctly validated, etc. – is equally possible.

Insecure Direct Object References (IDOR)

Including object references as arguments to an API call is common practice – and consequently a very common API vulnerability is that access rights are not enforced for objects a user does not have access to. Identifying id parameters and sequentially looping through possible values can be an easy way to access data that isn’t meant for a user – once again API metadata can be a natural source for more information in this regard, making it easy for both hackers and tester to find possible attack surfaces within an API topology.

As a tester you need to validate that security is correctly enforced for objects one should not have access to – which is easily done by automating requests to resource not accessible and validating that the correct error messages are shown. At a higher level – you should perhaps also question the usage of identifier values that can easily be guessed – especially if the data is sensitive. Perhaps using GUID identifiers could be a better solution as identifiers in applicable cases.

Bad Session / Authentication Handling

The previous vulnerability is a specific example of whole slew of vulnerabilities related to bad session, security and authentication handling. Consider some more examples:

• Are session tokens re-used or sequential?
• Do session tokens timeout correctly?
• Are tokens exposed in unencrypted traffic?
• Are tokens added to URLs when sending links?
• Are login endpoints restricted?

All of these will be obvious attack points for a hacker – and should be as such for you as a Security Tester. Make sure that the described situations are correctly handled, do not give “bad” error messages if charged with invalid / unexpected input (using the arsenal of tests we have been looking at) – and do not provide an entry point for a hacker to start prying away at the walls of your APIs defense mechanisms!

Recap – and some tips to go!

Some hands-on tips to ensure that your APIs are fully security tested before we say our goodbyes:

• Automate: Most of the testing approaches for security vulnerabilities that we have covered can be automated and should definitely be done so – preferably against your production environment in some way. A system upgrade or server reconfiguration could be just as much a cause for a security vulnerability as bad development in the code of your APIs – and you definitely want to find out if such changes open up for unexpected vulnerabilities before anyone else.
• Stay on top: OWASP provides invaluable services and information on Internet related security vulnerabilities – and how to work against them. Make it a habit to check out their website and match their recommendations and findings against your APIs and infrastructure. Follow their guidelines – specifically those geared at APIs - and make sure you are doing what you can base on their resources.
• Finally - It’s your problem: Security is probably not going to be fixed or worried by anyone else than you, – make sure you give it the same attention and focus as you do to functional, performance, usability, A/B testing, etc. The price to pay for leaked passwords or erased user data is too high to pay for you not to care (which I know you do).

And there you have it - that’s it for the last part of the series. Hopefully you have learned plenty; basic underpinnings of API security vulnerabilities, how easy it is too perform common API attacks and how equally easy it is in most cases to test for corresponding vulnerabilities. Now get out there and hack an API!

API Security Testing - How we can do/stop hacking to the APIs : Part-2

In our last post, we prepared our API hacking weaponry – we looked at the basics of Web-based APIs (HTTP, Message Formats, Security Standards) and how to discover the attack surface of an API. This time around we’re going to start with some basic attacks. Let’s see if we can get our API out of balance and joviality – welcome back!

The immediate purpose of an attack

Before we fire our trebuchets and spout our boiling canola oil, let’s take a quick step back and consider the actual immediate purpose of an attack. Although the high-level goal of an API hack might be to get access to credit card numbers or user passwords – a single attack is often just a step on the way. To get to those credit card numbers, we have to learn about a system’s underpinnings and its weaknesses. We have to pry around to find out how it works and what its vulnerabilities are.

A common approach is to provoke an API with unexpected content in the hope that its inability to handle it correctly will teach us about its inner workings. For example, getting the target API to return a database error for an unexpected input is of great value; we now know that there is a database behind the scenes (SQL Injection vulnerabilities?) and, if we’re “lucky”, we might even know a little about its vendor / version / schema / etc. – all pieces that help us put the vulnerability-puzzle. And the more verbose the error message, the better for us.  There is plenty of information online that will tell us of any known security vulnerabilities of any specific server/framework/OS setup.

As a Security Tester, you would most likely be doing the same; trying to get your target system to behave in a way that helps hackers get under its skin – but in your case you’ll tell your development team so they can fix the issues instead. Right?

Fuzzing

Fuzzing is a classic way of attacking (and testing!) a target system; the basic idea is to repeatedly generate randomized input for a target API until we stumble upon something that provokes an error message or system. With today’s tools, it’s easy to set up a script that runs millions of requests in the hope that some combination of input parameters will achieve our purpose.

As a tester, set up assertions that validate the response to be consistent and concise, use the correct HTTP error code, and do not expose any sensitive system information.

Invalid Input Attacks

Where fuzzing has little structure to it, invalid input attacks are slightly more intelligent as they aim to provoke a target system by sending input that it doesn’t expect. The better an API is described via metadata or documentation, the easier it is to do this efficiently. Examples would be sending strings when the API expects numbers, sending text when it expects dates, or for any field, send nothing or send something too long. Given our knowledge of HTTP this can be done at the protocol level also by sending invalid HTTP Headers and values – targeting both the APIs HTTP layer and its own logic.

Just as for fuzzing, a security tester can automate boundary/invalid-input tests and configure assertions that validate the error message. Be sure to take into account that, for usability reasons, your API might provide “friendly” error messages – make sure that these are “generic” and don’t give away any details of the underlying technologies.

Malicious input

While the above mentioned attacks generally hope to provoke unexpected system behavior and “bad” error messages, some input can be directly malicious and target both the API itself and its hosting infrastructure. Message parsers have been a common target for these attacks, with XML parsers being the prime example. Consider the following “XML Bomb” attack:

 When a misconfigured or vulnerable XML parser attempts to expand the lol9 entity in the actual document, the recursive nature of the “lol” entities will most likely result in out-of-memory errors. This could possibly bring down the entire server and leave it in a state where it might be vulnerable to attacks.

Another possibility for vulnerabilities and malicious input is where files are uploaded – does that target system process these files “safely”? What if the system expects a PDF but you upload a shell script instead – will that get executed on the server? Or get propagated to a 3rd party when they try to access the file (that they thought was a PDF) via another API call? Will corrupt files be handled gracefully or give away information about the underlying file system?

As a Security Tester, you can easily attempt to provoke your system by “playing the hacker” and sending malicious data as described. Obviously you need to be careful so you don’t bring down your system (unless that’s your goal) – but on the other hand going “full throttle” might be the only way of ensuring that your system isn’t vulnerable to these kinds of attacks. At least you’ll know what hit you.

Injection Attacks

According to OWASP, injection vulnerabilities are the most common types of security vulnerabilities out there. Many of the attacks that have gained media attention in recent times exploited related weaknesses. In OWASP-speak, an injection attack is one where an “Attacker sends simple text-based attacks that exploit the syntax of the targeted interpreter. Almost any source of data can be an injection vector, including internal sources.”

Let’s have a look at a simple SQL Injection example. Let’s say we have a REST API for a pet store (the one described in the Swagger document in the previous post of this series):

http://petstore.com/api/v1/pet/123

The actual implementation of the API uses the ID of the pet (“123” in the example URL) to look up the data in a database using the following SQL statement:

“SELECT * FROM pets WHERE petID=’” + petId +”‘”;

When the request is as above, this expands to

“SELECT * FROM pets WHERE petID = ‘123’”

Which is totally fine.

Now consider what happens if an attacker sends the following instead

http://petstore.com/api/v1/pet/’%20or%20’1’=’1

which with the above (admittedly naive) logic becomes

SELECT * FROM pets WHERE petID = ‘’ or ‘1’ = ‘1’

Whoa!! Suddenly we have a SQL statement that will give us all pets in the database, which could result in both a severely overloaded server (if the database of pets is large) and pet owners gaining access to other clients’ pet information – the horror! Although somewhat contrived, this simple example shows the danger; a more malicious SQL injection will attempt to augment SQL statements to delete data, change passwords, etc.

As hinted by the OWASP description above, injection attacks aren’t specific to SQL; any language interpreter used to process (API) input is at risk; Json, XPath, XSLT, JavaScript, etc. are all technologies that require parsers/processors which can be buggy and “compromisable” and should be seen as a risk (or an opportunity, depending on your point of view). As before, collections and examples of common injection attack vectors for common systems and technologies can easily be found online – use this to your advantage and make sure your API isn’t vulnerable to them.

As a Security Tester, injection attacks are a little bit more challenging than the invalid-input related attacks we looked at to start with. First of all, you need to know and understand a little about the inner workings of the target API to choose the “right” injection attacks (this is what a hacker would use other attacks to find out).  Then you have to decide on what a “successful” injection attack would look like. For the above example, you would probably want to validate that you either get a 404 (Not Found) or a 400 (Bad Request).

Cross-Site Scripting Attacks

Let’s finish this post with a look at another very common way to attack a system and its users – XSS (cross-site-scripting) attacks. The name of the attack is somewhat misleading; you might spontaneously think that this applies to web applications only, but since APIs are often driving the interface of a web application, it makes sense to test that the API keeps things as invulnerable as possible.

XSS attacks can be either “reflective” or “persistent” – the underlying “objective” of either being the insertion of a script that is executed in the browser/system of an unsuspecting user.

For reflective XSS, the script is often included in link in an email – when the user clicks the link the script is sent to a target server and returned to the client where it is executed:

So what is the API’s role here? Well, in today’s world, the actual request to the victim’s server that returns data is an API call that is formatted on the client – if your API “reflects” its input in any way (example: a search API might include the search-string in the returned result), you will need to decide if it’s up to the API or the client to handle input that is potentially malicious. Then test the API accordingly.

For persistent XSS – the malicious script is injected into a backend system. It is  then retrieved and executed by a separate client at a later time (this could also be seen as a client-injection attack):

Once again, it’s not obvious if an involved API should be handling this – or the client UI. For example, let’s say you have an issue tracker with an API where you can create issues. If someone inserts a script into the description of an issue, is it up to the API backend to remove/escape that script before it is returned to an API client? Or is it up to the client to perform this handling?

One way to look at it is that this XSS attacks are basically injection attacks – a script is injected into the system – and then either executed on the client, or somewhere during the processing on the backend, and in either case it’s a potential vulnerability and something you need to assess (before a hacker does it for you).

Whatever you decide, it is easy to set up the appropriate tests, Vectors of common cross-site scripting attacks are available online, allowing you to set up data-driven tests with these vectors using your favorite API testing tool to run and perhaps even automate corresponding tests.

Moving ahead

Thus concludes our lesson outlining the various weapons in your API hacking arsenal. But this is only a portion of the methods at your disposal – next part to look at some more common attacks and how to test for them. More importantly, I’ll provide advice on how to set up your API security tests and how to stay up-to-date with the latest security vulnerabilities so that the only person hacking your API is you.