Skip to content

vangav open source - backend; a backend generator (generates more than 90% of the code needed for big scale backend services)

License

Notifications You must be signed in to change notification settings

vangav/vos_backend

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

vangav backend

built on top tech

tech ref
cassandra apple's cassandra deployment has over 75,000 nodes storing over 10PB of data
play framework play framework powers linkedin's 500,000,000 members backend
akka akka helps power e-shopping for hundreds of millions of amazon members
datastax drivers datastax processes over 1,000,000,000,000 cassandra requests for netflix per day

fully-loaded

utility options
backend generator design in json and generate 90+% of the code for a service + several utilities
security facebook/google auth, oauth 2, transaction tokens, encryption, ...
networks twilio, mail gun, rest client, rest jobs, ...
push notifications ios, android, ...
geo services geo grids, reverse geo coding, geo hashing, ...
dispatcher/worker dispatchable queries, push notifications, emails, sms, ...
and more ... periodic jobs, thread pools, facebook graph api, snow flake ids (twitter's ids), compression, client generator, exceptions, geometry, metrics, deep vangav mighty integration, ...

illustration

template services features db tables code: total code: generated
instagram 7 47 52 67,715 + 190,996 238,465 (92.17%)
analytics writer & reader 2 9 4 67,715 + 10,252 74,552 (95.62%)
whatsapp 3 7 7 67,715 + 7,834 74,408 (98.49%)
geo server 1 3 3 67,715 + 2,691 70,300 (99.85%)
calculate sum 1 1 0 67,715 + 544 68,258 (99.99%)

roadmap

time effect
5 min first service (vos_calculate_sum) generated and running
20 min second service (vos_geo_server) generated and running
5-7 days finished all templates: command of all utilities, multi-service backends
1-2 weeks full command of modifying/extending the underlying backend
thereafter best-in-class-level in implementing top class backend services, finishing a year's worth of work every month

in a nutshell

illustration

system requirements

  • unix operating system (e.g.: mac os, ubuntu, etc ...)
  • java 8
  • python 2.7.10 (only for using cassandra)
    • to check python version type in the following command in a terminal session python2.7 -V and the output should be Python 2.7.10
    • installing python on mac os
    • installing python on ubuntu
      • cd ~/Downloads
      • wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
      • tar -zxvf Python-2.7.10.tgz
      • sudo apt-get install build-essential checkinstall
      • sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libgdbm-dev libc6-dev libbz2-dev libsqlite3-dev tk-dev libssl-dev
      • cd ~/Downloads/Python-2.7.10/
      • sudo ./configure
      • sudo make altinstall
      • python2.7 --version

prerequisites

  • basic java
  • basic cassandra (optional for using cassandra database)

quick start example: calculate sum

YouTube Play Icon on YouTube

5-10 min: this tutorial explains how to generate and use the first vangav backend service

vos_calculate_sum is a service that takes a two floats (a and b) request and returns a double (c) response representing the summation of a and b

init

  1. create a workspace directory my_services - this is the directory to contain both of vos_backend and all the services generated using it
  2. download this vos_backend.zip project (from the green clone or download button up there) inside the workspace directory created in (1) and unzip it
  3. rename downloaded vos_backend-master to vos_backend

generate a new service

  1. create a new directory my_services/vos_calculate_sum
  2. copy controllers.json from vos_backend/vangav_backend_templates/vos_calculate_sum/ to the directory my_services/vos_calculate_sum created in (1)
  3. open a terminal session and cd to my_services/vos_backend/tools_bin
  4. execute the command java -jar backend_generator.jar new vos_calculate_sum to generate the service
  5. enter y for using the config directory in order to use controllers.json for generating
  6. enter n for generating a worker service (using workers is explained in a separate section)

writing the service's logic code

  • optionally for eclipse users: open eclipse and import vos_calculate_sum project
    • file > import > general > existing projects into workspace > next > set "select root directory" to my_services > under projects make sure that vos_calculate_sum is selected > finish
    • double check the java version used for compiling the project: right click the project > properties > java compiler > enable project specific settings > compiler compliance level > 1.7 or 1.8
  • open class HandlerCalculateSum.java under package com.vangav.vos_calculate_sum.controllers.calculate_sum, method processRequest should be as follows in order to complete the request-to-response logic
  @Override
  protected void processRequest (final Request request) throws Exception {

    // use the following request Object to process the request and set
    //   the response to be returned
    RequestCalculateSum requestCalculateSum =
      (RequestCalculateSum)request.getRequestJsonBody();
    
    // set response's value
    ((ResponseCalculateSum)request.getResponseBody() ).set(
      requestCalculateSum.a + requestCalculateSum.b);
  }

start the service

  1. cd to my_services/vos_calculate_sum
  2. execute the command ./_run.sh

try it out

  1. open an internet browser page and type http://localhost:9000/calculate_sum?a=1.2&b=2.3 - this returns 3.5
  2. play with a and b values in the request string in (1)
  3. try issuing an invalid request (e.g.: set a to "xyz", don't set b, ...) to get a sense of how the default error response looks like (error responses are explained in depth in a separate section)

stop the service

in the terminal session where you started the service press control + d

vangav backend tutorials

YouTube Play Icon on YouTube

  • shows how to add controllers (api entry points) before and after service generation | on YouTube
  • explains the building blocks of vangav backend | on YouTube
  • generates a service that has a backend database and also uses the geo services utility | on YouTube
  • explains the building blocks of a generated service | on YouTube
  • explains the building blocks of the config used to generate a service | on YouTube
  • when implementing a service's logic, the passed request object keeps all the request's information from start to finish; this tutorial shows how to use that object | on YouTube
  • explains what happens whenever an error happens during request processing and how to manually return various types of error response | on YouTube
  • shows why and how to us the dispatcher with one or more worker service(s) | on YouTube
  • explains how to start and use the debugger
  • once your service is ready for release, this tutorial has the step-by-step process till your service is up and running on a production server - as well as how to scale it up | on YouTube
  • vangav backend detects various types of errors (e.g.: invalid request param, invalid generator config, wrong utility method arguments, unauthorized third-party authentication, invalid vangav mighty solution, ...); this tutorial lists all of vangav backend's error codes/sub-codes with reference to the code producing them for ease of tracing if you get one | on YouTube

utilities, design and src code tutorials

YouTube Play Icon on YouTube

  • handles generating java clients and contains vangav backend's client framework
  • manages all vangav backend generators (api, database client, worker and java client)
  • handles generating database clients, contains the framework for generated clients and handles all cassandra operations
  • contains different compression algorithms
  • handles content (code, phriction-wiki, text, ...) checking, formatting and generation; vangav backend relies on this package to verify generation config and format generated code, scripts, wiki, ...
  • has data structures (e.g.: heap, kd-tree, tuples, ...) and algorithms (collections, matricies, strings, arrays, ...)
  • handles generating worker services, contains the framework for dispatchers/workers and handles all of their operations
  • vangav exceptions are used to handle bad request and internal error; those exceptions can be returned to the client and loggable (in database, text files, ...)
  • simplifies various types of files needed for vangav backend services like: properties, json config, images, http response files, ommiting comments, directory operations, ...
  • handles reverse geo coding, geo grids and geo hashing
  • handles uuid operations, sequential ids and twitter's snow flake ids
  • handles mathematical and geometric operations like: numeric, ranges, circles, line segments, straight lines, ...
  • handles distance, time, date and calendar operations
  • has sync/async rest client, rest jobs, email clients, twilio messaging and download utility
  • handles generating api code and contains the framework for generated vangav backend services; vangav backend services' backbone
  • handles loading and extracting data from properties files
  • handles fetching data from facebook graph api and car2go api
  • handles building and sending apple and android notifications
  • handles authentication (facebook, google, oauth 2 and transaction tokens) and cryptography (asymmetric, hashing and two-way encryption)
  • handles fetching system info (cpu usage, free ram/disk, os type, number of cores, ...) and console operation (interactive, commands, ...)
  • contains latch threads, periodic jobs and the thread pools responsible for maintaining top performance for vangav backend services (in-memory threads, cassandra, dispatcher and rest client)

templates

  • simple service | on YouTube | but my phone was under 10%
  • a more advanced service with a databse and uses geo services utility | on YouTube | I tried to make it slow!
  • service oriented architecture (main + worker + analytics), multi-keyspace database and basic authentication | on YouTube | am I a hologram?
  • service oriented architecture (writer + reader) and generic service design (handles any type of analytics) | on YouTube | it may be foolish to turn this down!
  • on YouTube | [in-spuh-rey-shuh n]

covered topics

  • service oriented architecture (main + dispense + jobs + worker + dash board)
  • multi-keyspace database
  • oauth 2 and facebook authentication
  • facebook graph api
  • rest jobs
  • periodic jobs
  • vangav mighty
  • push notifications
  • logging
  • analytics
  • client generator
  • test and bots services

share

facebook share twitter share pinterest share google plus share linkedin share

free consulting

vangav's consultant