Rest API Design - OTP based Auth API

Learn how to design REST API with a real world example. We build an OTP based auth API with /signup, /login, /send-otp and /validate-otp features.

Problem statement:

Design the auth flow apis (signup, login, etc) for an OTP based user authentication. Assume OTP will be created and presented to users upon a successful login.

API Signature:

- URL

- Request Method

- Request Headers

- Request Body

- Response Status code

- Response Headers

- Response Body

Signup API:

• URL
• https://<name>.com/signup
• Will this be the URL on browser?
• Can we have same URL for UI and backend?
• https://<name>.com/api/signup or https://api.<name>.com/signup Are we missing anything?
• Versioning of APIs
• Path or via headers
• https://<name>.com/api/v1/signup or via request headers
• Request method: GET, PUT, POST, DELETE, PATCH, OPTIONS
• Which method should we use here?
• POST
• POST vs PUT? Idempotency
• Request Headers:
• Content Type: application/json
• any others?
• Request Body: Form Data, XML, JSON, i.e. Content Type
• JSON body:
•   {
•     "email": "foo@bar.com",
•     "password": "supersecret", // plaintext password, yikes!
•     "phone": "+91-9876543210"
•   }
• 
  
• Logic on server side?
• Validate request body
• Insert record in DB
• Handle errors
• Response Status code: 2xx, 3xx, 4xx, 5xx
• 200 ok or
• 201 created
• Response Headers:
• Content Type if we are sending any content back
• Any other headers?
• Response Body:
• Empty body or
• Some useful content (user id?)

Login API:

• URL https://<name>.com/api/v1/login
  
• Request method: POST
  
• Request Headers:
• Content Type: application/json
• any others?
• Request Body:
• JSON body:
•   {
•     "email": "foo@bar.com",
•     "password": "supersecret", // plaintext password, yikes!
•   }
• 
  
• Logic on server side?
• Validate request body
• Check email and password combination
• Handle errors
• Create a token and send token back to user. Why?
• Response Status code: 2xx, 3xx, 4xx, 5xx
• 200 ok
• 4xx client side errors (validation of email, invalid credentials or unauthorized)
• 5xx server side errors (DB down so not able to login the user)
• Response Headers:
• Content Type if we are sending any content back
• Any other headers?
• Return a time limited token in headers
• Why do we need this? Let's come back to it when we talk about validate OTP API
• Response Body:
• Empty body or
• Send OTP as part of response?
• When do we actually send the OTP to user? Is it a UI triggered action? Or a "side-effect" of successful login

Send OTP API:

• URL https://<name>.com/api/v1/send-otp
  
• Request method: POST
  
• Request Headers:
• Token received from Login API reponse. Why?
• any others?
• Request Body:
• Empty body or
• Should we send phone number to send OTP to?
• Logic on server side?
• Validate token from headers
• Figure out phone number from token
• Sent OTP, store OTP in some storage for validation
• Response Status code: 2xx, 3xx, 4xx, 5xx
• 200 ok
• 4xx client side errors (invalid token, i.e. unauthorized)
• 5xx server side errors (Third party SMS vendor down, DB down, etc. so not able to send OTP the user)
• Response Headers:
• None that I can think of
• Response Body:
• Empty body

Validate OTP API:

• URL https://<name>.com/api/v1/validate-otp
  
• Request method: POST, or GET. Any query params?
  
• Request Headers:
• Token received from Login API reponse. Why?
• any others?
• Request Body:
• JSON body
• {
•     "otp": "123456"
• }
• 
  
• Logic on server side?
• Validate token
• Check token and OTP combination are correct
• Handle errors
• Response Status code: 2xx, 3xx, 4xx, 5xx
• 200 ok
• 4xx client side errors (invalid token, i.e. unauthorized, invalid otp, expired otp, etc.)
• 5xx server side errors (DB down so not able to validate OTP)
• Response Headers:
• None that I can think of
• Response Body:
• Empty body

Do you see any challenges/unknowns?

• Token before and after OTP validation is same. Is it okay? What problems can it cause?
• What if user requests for OTP multiple times? (both legit and abuser use case)
• Where do we store OTPs, Tokens?
• Can this API be used by both web and mobile?

Kubernetes daily use commands

 Here is a quick view on k8 commands which are ideally useful for daily interaction work

• kubectl config get-contexts
• display list of contexts
• kubectx <env-name>
• to switch the env
• kubectl get nodes
• Get all nodes
• kubectl get namespaces
• Get all namespaces for a environment
• kubectl -n services get pod <pod-name>
• Get specific pod details
• kubectl -n services get pods
• Get all pods
• kubectl -n services delete pod <pod-name>
• Delete specific pod name
• -o wide
• To get data in more details like which node etc
• -o yaml
• To open specific file to see the details
• kubectl -n services describe pod <pod-name>
• To see description of the pod
• kubectl -n services exec -it  <pod-name> /bin/bash
• Logging into the pod to see details at code level
• kubectl -n services get hpa <hpd-pod-name>
• To check hpa details if hpa enabled
• kubectl -n services scale deploy <pod-name> --replicas=3
• To manual scale
• kubectl -n services get pods -o wide | grep ½
• Get all pods which are in unhealthy state
• kubectl -n services get deployment <pod-name> -o yaml
• To check min/max replicas or any other data point w.r.t deployment
• kubectl -n services get deployment <pod-name> -o yaml
• To edit min/max replicas or any resources
• kubectl -n services rollout restart deployment <po-name>
• To restart a service
• kubectl -n services logs <pod-name> -c install -f
• This could be used if our pod is in Init stage. Gives us the keys that are missing from config(Search for nil after running this command)
• kubectl -n configuration get pods
• Consul and Vault status
• stern -n configuration <name>
• Check logs of vault
• stern -n services <pod-name> -c <app-name> -t --since 1m
• Prints logs of all the pods of last 1 min
• stern -n services geolayers-api-primary -c geolayers-api -t --since 1m | grep '<text>' 
• Prints log only for the text that is in grep
• kubectl describe canary <pod-name> -n services
• To check canary deployment of specific pod
• kubectl get canary -n services
• To check all pods for which canary enabled

Kubectl commands complete cheatsheet : https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Happy Learning !! :)

Design Parking Garage

 Let's discuss designing parking garage as a problem statement.

At high level what all we need to consider and think about while designing a parking garage.

Product Requirements:

• Need to be able to reserve a parking spot and receive some kind of ticket or receipt.
• Need to be able to pay for a parking spot.
• System needs to have a high consistency (no two people should be able to reseve the same spot at the same time).
• 3 types of vehicles (compact, regular and large).
• flat rate, but different rates depending of the type of parking.

APIs:

• Public Endpoints
• /reserve
• Params : garage_id, start_time, end_time
• Returns : (spot_id, reservation_id)
• /payment
• Params : reservation_id
• Note : Likely using an existing API to handle (Stripe, Square etc...)
• /cancel
• Params : reservation_id
• Internal Endpoints
• /calculate_payment
• Params : reservation_id
• /freespots
• Params : garage_id, vehicle_type, time
• Note : Smaller vehicles can fit into larger spots if necessary and therefore should be included in the overall number of spots
• /allocate_spot
• Params : garage_id, vehicle_type, time
• /create_account
• Params ; email, password, first_name, last_name
• /login
• Params : email, password

Data Schema:

• Reservations
• id : primary key
• garage_id : foreign key
• spot_id : foreign key
• start: timestamp
• end : timestamp
• paid : boolean
• Garage
• id : primary key
• zipcode : varchar
• rate_compact : decimal
• rate_regular : decimal
• rate_large : decimal
• Spots
• id : primary key
• garage_id : foreign key
• vehicle_type : enum
• status : enum
• Users
• id : primary key
• email : varchar
• password : varchar (note that this probably SHA-256 hash)
• first_name : varchar
• last_name : varchar
• Vehicles
• id : primary key
• user_id : foreign key
• licence : varchar

How to start with distributed systems

 Let's discuss at high level what we need to think in terms of designing distributed system.

Taking a very common example of opening pizza shop and converting it into technical terms. Let's say we wanna open a pizza shop so what all we need to consider to run it well and successfully.

Few tips/terms used by engineers when designing systems

Vertical Scaling:

• After opening your restaurant you started getting more orders so now you need to scale your business.
• Optimise processes and increase throughput using the same resource. 

Preprocessing using cron jobs:

• You want to do something beforehand which can save time during peaktime like may be preparing pizza bases etc
• Preparing beforehand at non-peak hours.

Backup Servers:

• What if you only have one chef and if he falls sick, your business won't run that day so you need to hire one more chef as a backup.
• Keep backups and avoid single point of failures.

Horizontal Scaling:

• As business is running quote, it's time to hire more chefs and do more business with more orders
• Hire more resources

Microservice Architecture:

• As your business expanded and you have chefs with expertise like making pizza and garlic breads so you want to route specific requests to speciality chef and diving responsibilities
• You need to build microservice architecture to achive this use-case.

Distributed Systems:

• Let's say there is a powercut for one day or some licence issue happens in one day, your business won't run that day so you need to have one more shop may be with small scale but your business can run well.
• You need to design distributed system to achive the above use-case.

Load Balancing:

• Now you have two shops so someone should be intelligent enough to decide where to route order request to get efficient result.
• You need load balancer to achieve above use-case.

Decoupling:

• As an example delivery agent doesn't need to know what he is delivering, it is pizza today and might be burger tomorrow if you you expand your business with other verticals down the line.
• In technical terms, it's decoupling the system and separating out concerns so that you can handle separate systems more efficiently.

Logging and metrics calculation:

• Let's say one day there was some faulty oven or delivery agent having fault in bike and total order rate went down so you want to log everything so you can debug at any point of time later to know the reason and act on it accordingly.
• Analytics
• Auditing
• Reporting
• Machine Learning

Extensibility:

• As a backend engineer you don't want to reqrite all this code again and again to serve a different purpose

What we have done is taken a business scenario try to find solutions to all the problems that it came up with and then just map them into technical terms. Now if you think of these, there are solutions in themseleves for the technical counterparts of the problems.

Happy Learning !! :)

Kubernetes Basic Components

 Here is a quick view of few basic components of K8

Pod:

• Smallest unit of K8
• Abstraction over container
• Usually one application per Pod
• Each Pod gets his own IP address
• New IP address on re-creation

Services:

• An abstract way to load balance across the pods and expose an application deployed on a set of Pods.
• Permanent IP address

ConfigMap:

• External configuration of your application like DBs config etc

Secret:

• Used to store secret data like DBs username/pwd etc which can't be store in plain text in ConfigMap
• base64 encoded

Kubernetes Basics

 Official definition of Kubernetes

Open source container orchestration tool developed by Google and helps you manage containerized applications in different deployment environments like physical machines, virtual machines or cloud environment or even hybrid deployment environments.

The need for a container orchestration tool

• Trend from Monolith to Microservices
• Increased usage of containers
• Demand for a proper way of managing those hundreds of containers

What features do orchestration tools offer?

• High Availability or no downtime
• Scalability or high performance
• Disaster recovery - backup and restore

Kubernetes Basic Architecture

Made up with master node and couple of worker nodes where each node has kubelet process running on it. Kubelet is basically a kubertnetes process that makes it possible for cluster to talk to each other and actually executes some tasks on those nodes like running application processes. Each worker node has docker containers of different applications deployed on it. On worker nodes your applications are running.

Master node runs several K8 processes

• API Server which also a container and an entrypoint to K8 cluster.
• Controller Manager keeps track of whats happening in the cluster, may be if container died and needs a restart etc.
• Scheduler ensures Pods placement.
• etcd (key value storage) which basically holds at anytime current state of K8 cluster. It has all the configuration data inside and all the status data of each node and each container inside of that node and backup restore actually made of this etcd snapshot.

Kubernetes Basics Concepts

Pod is a smallest unit which you as a user will configure and interact with and basically a wrapper of container. In a worker node you gonna have multiple pods. Each pod is it's own self contained server with it's own IP address. Pods are recreated frequently and it gets new IP address on creation hence another component of K8 called service is used along with pod.

Horizontal vs Vertical Scaling

Horizontal scaling is adding more machines to deal with increasing requirements. These machines handle requests in parallel to improve user experience. 

Vertical scaling is replacing the current machines with more advanced machines to improve throughput and hence response time. The techniques are used in conjunction in real world systems.

Horizontal Scaling Points

1. Load Balancing Required
2. Resilient
3. Network Calls
4. Data Inconsistency
5. Scales well as users increase

Vertical Scaling Points

1. Load Balancing not required
2. SPOF
3. Inter process communication
4. Consistent
5. Hardware Limit