Skip to content

vangav/vos_instagram

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

why? instagram is the biggest vangav backend template designed for big scale and covering all the core features/utilities: service oriented architecture, worker service, multi-entry-point api, multi-keysapce database, oauth 2, facebook authentication, facebook graph api, rest jobs, periodic jobs, vangav mighty, push notifications, logging, analytics, client generator, test service, bots service, ...

instagram

instagram backend design

prerequisite

functionality

  • signup/login using email/facebook
  • request authentication using oauth 2
  • fetching/updating users' information
  • follow/unfollow and get followers/following lists for users
  • posting photos
  • get posts by feed and by user
  • interacting with posts (like, comment, ...)
  • get top users/posts
  • searching for users
  • get server time
  • is a helper service for instagram; responsible for distributing posted photos to followers (because on photo can be posted to tens of millions of followers; such an expensive operation should be distributable and shouldn't overload the main service)
  • crawls for failed jobs in the database to retry executing them (e.g.: a dispense job, ...)
  • uses vangav mighty to update top users (per geo grid, per country and world wide) every week
  • uses vangav mighty to update top posts (per geo grid, per country and world wide) every day
  • writes logs and analytics in the database
  • sends push notifications (ios and android)
  • handles fetching logs and analytics for system admin
  • has a full-coverage test for all the services above
  • simulates incremental usage behavior from many cities around the world with some celebrity users

overview

  • this service is based on vangav backend's instagram template
  • this service has the 90+% of the vangav backend's generated code + the 10-% of the logic code needed to complete the service

try this service

  1. for first timers - follow the steps in the system requirements tutorial
  2. for first timers - follow the steps in the workspace initialization tutorial
  3. download instagram, instagram dispense, instagram jobs, instagram worker and instagram dash board projects (from the green clone or download button) inside the workspace directory created previously (my_services) and unzip them
  4. rename unzipped directories, remove the -master from their names
  5. in the terminal cd to vos_instagram/cassandra/cql/
  6. execute ./_start_cassandra.sh to start cassandra
  7. cd to vos_instagram/cassandra/cql/drop_and_create/
  8. execute the commands ./_execute_cql.sh ig_....cql to initialize the services' database tables; repeat this step for all the .cql files
  9. cd to vos_instagram_worker and execute ./_run.sh to start the instagram worker service on port 8000
  10. cd to vos_instagram_dispense and execute ./_run.sh 3000 to start the instagram dispense service on port 3000
  11. cd to vos_instagram_jobs and execute ./_run.sh 4000 to start the instagram jobs service on port 4000
  12. cd to vos_instagram_dash_board and execute ./_run.sh 5000 to start the instagram dash board service on port 5000
  13. cd to vos_instagram and execute ./_run.sh to start the instagram service on port 9000
  14. make a call to the start_jobs controller from the vos_instagram_jobs backend service in order to start the periodic jobs
  15. from your prefered client (we recommned postman) start trying the service; refer to the features and service references sections for reference
  • at the end to stop the services: press control + d in the terminal session where each service was started in (9, 10, 11, 12 and 13)
  • to stop cassandra: execute ps auwx | grep cassandra to get cassandra's (pid) then kill -9 (pid) to stop cassandra

eclipse

  • follow the following steps to import the downloaded backend service in eclipse
  1. cd to the service's directory and execute the ./_eclipsify.sh script
  2. file > import > general > existing projects into workspace > next > set "select root directory" to my_services > under projects make sure that vos_calculate_sum is selected > finish
  3. double check the java version used for compiling the project: right click the project > properties > java compiler > enable project specific settings > compiler compliance level > 1.7 or 1.8

covered topics

  • generate multiple services (main + dispense + jobs + worker + dash board) to work together in a service oriented architecture
  • generate a multi-keyspace database
  • oauth 2 and facebook authentication
  • facebook graph api
  • rest jobs
  • periodic jobs
  • vangav mighty
  • push notifications
  • logging
  • analytics
  • client generator
  • test and bots services

features

controller(s) feature
signup email, login email, login facebook, get access token, refresh access token and logout handle user signup, loging and requests' authentication
update facebook info, update last active time, update location, update profile picture, get user info and get profile picture handle setting/updating users' info and getting it
follow, unfollow, get followers and get following handle building the social graph and exploring it
post photo handles posting a new photo to a user's followers
get feed posts, get user posts, get post info, get post photo id, get photo, get photo thumbnail, get post likes and get post comments handle getting feed/users' posts and posts' info/content/likes/comments
like post, unlike post, comment and delete comment handle interacting with posts
get top users and get top posts handle exploring top users/posts
search by email handles searching for users by their e-mail
get server time getting server's time is important for various reasons like enabling the client to set how old a post is (2h, 1w, ...)
controller feature
post photo to followers handles distributing new posts to followers; used by instagram: post photo
controller feature
start jobs stars the periodic job for retrying failed rest jobs and the periodic jobs for ranking users and posts
controller(s) feature
get daily users error logs, get daily users logs and get hourly controllers error logs handle getting logs per user/controller
get annual regional counters, get daily regional counters, get daily requests counters and get hourly requests counters handle getting backend-wide analytics
get top users and get top posts gets in-depth information about top users/posts on specified dates/locations
controller feature
test all controllers runs a full-coverage test for the instagram services above and returns a full report (success, failure, request-to-response time, ...)
controller feature
start bots starts simulating incremental usage behavior from many cities around the world with some celebrity users

service references

reference explanation
routes api routes
prop in addition to vangav backend's properties files, this directory contains instagram properties files: constants and dispense
reverse_geo_coding reverse geo coding data
controllers.json api request/response's elements
ig_auth.keyspace ig_auth is the keyspace used for all authentication-related tables
ig_app_data.keyspace ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...)
ig_blobs.keyspace ig_blobs is the keyspace used for all blob-related tables (photos, thumbnails, ...)
ig_jobs.keyspace ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted
ig_logging.keyspace ig_logging is the keyspace used for all logging-related tables
ig_analytics.keyspace ig_analytics is the keyspace used for all analytics-related tables
Global.java initializes reverse geo coding data
common handles common operations like authentication, cassandra's frequent-operations, omitting blobs from requests/responses for logging, ...
controllers api implementation
ig_auth ig_auth cassandra's keyspace client
ig_app_data ig_app_data cassandra's keyspace client
ig_blobs ig_blobs cassandra's keyspace client
ig_jobs ig_jobs cassandra's keyspace client
ig_logging ig_logging cassandra's keyspace client
ig_analytics ig_analytics cassandra's keyspace client
reference explanation
routes api routes
controllers.json api request/response's elements
ig_auth.keyspace ig_auth is the keyspace used for all authentication-related tables
ig_app_data.keyspace ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...)
ig_jobs.keyspace ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted
ig_logging.keyspace ig_logging is the keyspace used for all logging-related tables
ig_analytics.keyspace ig_analytics is the keyspace used for all analytics-related tables
controllers api implementation
ig_auth ig_auth cassandra's keyspace client
ig_app_data ig_app_data cassandra's keyspace client
ig_jobs ig_jobs cassandra's keyspace client
ig_logging ig_logging cassandra's keyspace client
ig_analytics ig_analytics cassandra's keyspace client
reference explanation
routes api routes
prop in addition to vangav backend's properties files, this directory contains instagram jobs properties files: constants, users rank and posts rank
reverse_geo_coding reverse geo coding data
controllers.json api request/response's elements
ig_app_data.keyspace ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...)
ig_jobs.keyspace ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted
ig_logging.keyspace ig_logging is the keyspace used for all logging-related tables
Global.java initializes reverse geo coding data and periodic jobs
common handles common operations like instagram-jobs-service's properties
periodic_jobs handles retrying failed jobs as well as ranking users and posts
controllers api implementation
ig_app_data ig_app_data cassandra's keyspace client
ig_jobs ig_jobs cassandra's keyspace client
ig_logging ig_logging cassandra's keyspace client
reference explanation
routes api routes
controllers.json api request/response's elements
ig_auth.keyspace ig_auth is the keyspace used for all authentication-related tables
ig_app_data.keyspace ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...)
ig_logging.keyspace ig_logging is the keyspace used for all logging-related tables
ig_analytics.keyspace ig_analytics is the keyspace used for all analytics-related tables
common handles common operations like service-wide constants
controllers api implementation
ig_auth ig_auth cassandra's keyspace client
ig_app_data ig_app_data cassandra's keyspace client
ig_logging ig_logging cassandra's keyspace client
ig_analytics ig_analytics cassandra's keyspace client
reference explanation
routes api routes
prop in addition to vangav backend's properties files, this directory contains the clients properties file clients
controllers.json api request/response's elements
vos_instagram.client_java instagram client's api request/response's elements
vos_instagram_dash_board.client_java instagram dash board client's api request/response's elements
common handles common operations like representing instagram users
clients contains the generated clients for instagram and instagram dash board services
controllers api implementation
reference explanation
routes api routes
setup_data contains a one-time setup data with all the cities (id, location, population, ...) where the bots are created
prop in addition to vangav backend's properties files, this directory contains the clients properties file clients
controllers.json api request/response's elements
vos_instagram.client_java instagram client's api request/response's elements
ig_bots.keyspace ig_bots is the keyspace used for all bots-related vos_instagream tables
Global.java initializes cities and bots
cities has the cities' json representation and loader
bots has the bots' periodic jobs and the bots creator
clients contains the generated clients for instagram service
controllers api implementation
ig_bots ig_bots cassandra's keyspace client

change log

  • this section lists the 10-% code added after vangav backend generated 90+% of the code
file/dir change
prop added instagram properties files: constants and dispense
reverse_geo_coding added reverse geo coding data
Global.java added the initialization of reverse geo coding data
common added common operations like authentication, cassandra's frequent-operations, omitting blobs from requests/responses for logging, ...
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
prop added instagram jobs properties files: constants, users rank and posts rank
reverse_geo_coding added reverse geo coding data
Global.java added the initialization of reverse geo coding data and periodic jobs
common added common operations like instagram-jobs-service's properties
periodic_jobs added the periodic jobs for retrying failed jobs as well as ranking users and posts
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
common added common operations like service-wide constants
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
common added common operations like representing instagram users
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
setup_data added a one-time setup data with all the cities (id, location, population, ...) where the bots are created
Global.java added the initialization of cities and bots
cities added the cities' json representation and loader
bots added the bots' periodic jobs and the bots creator
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages

error codes

  • following are the error codes of instagram services
class code : sub_code explanation
Constants
300 : 1 couldn't initialize kAuthCodeLifeTime
300 : 2 couldn't initialize kAccessTokenLifeTime
300 : 3 couldn't initialize kFacebookAppId
300 : 4 couldn't initialize kDefaultRegion
300 : 5 couldn't initialize kWorldRegion
300 : 6 couldn't initialize kDefaultGridId
300 : 7 couldn't initialize kGeoGridDimensionMetres
300 : 8 couldn't initialize kGeoGridsConfig
300 : 9 couldn't initialize kThumbnailDimension
CommonPlayHandler
400 : 1 no access token for user id and device token
400 : 2 wrong access token
400 : 3 already expired access token
400 : 4 just expired access token
HandlerComment
401 : 1 can't comment on a post that doesn't exist
401 : 2 non-fatal: invalid device type
DeleteComment
402 : 1 post/comment doesn't exist
HandlerFollow
403 : 1 can't follow self
403 : 2 can't follow an non-existing user
403 : 3 already following user
403 : 4 non-fatal: invalid device type
HandlerGetAccessToken
404 : 1 auth code doesn't exist
404 : 2 wrong auth code
HandlerGetFollowers
406 : 1 can't get followers for non-existing user
HandlerGetFollowing
407 : 1 can't get following for non-existing user
HandlerGetPhoto
408 : 1 photo doesn't exist
HandlerGetPhotoThumbnail
409 : 1 photo doesn't exist
HandlerGetPostComments
410 : 1 post doesn't exist
HandlerGetPostInfo
411 : 1 post doesn't exist
HandlerGetPostLikes
412 : 1 post doesn't exist
HandlerGetPostPhotoId
413 : 1 post doesn't exist
HandlerGetProfilePicture
414 : 1 user doesn't exist
HandlerGetUserInfo
418 : 1 user doesn't exist
HandlerGetUserPosts
419 : 1 user doesn't exist
HandlerLikePost
420 : 1 post doesn't exist
420 : 2 already liked
420 : 3 non-fatal: invalid device type
HandlerLoginEmail
421 : 1 e-mail isn't signed up
421 : 2 wrong password
421 : 3 e-mail isn't signed up - triggers in case authentication is disabled
HandlerLoginFacebook
422 : 1 couldn't get user's name from facebook graph api
HandlerLogout
423 : 1 no refresh token for this user/device_token
423 : 2 wrong refresh token
HandlerRefreshAccessToken
425 : 1 no access token for user
425 : 2 wrong access token
425 : 3 no refresh token
425 : 4 wrong refresh token
HandlerSignupEmail
427 : 1 e-mail already signed up
HandlerUnfollow
428 : 1 can't unfollow a user that doesn't exist
428 : 2 a user can't unfollow another unless she/he is following that user already
HandlerUnlikePost
429 : 1 can't unlike a post that doesn't exist
429 : 2 user didn't like this post before to unlike it
class code : sub_code explanation
HandlerGetDailyUsersErrorLogs
404 : 1 e-mail isn't registered
HandlerGetDailyUsersLogs
405 : 1 e-mail isn't registered
class code : sub_code explanation
Constants
1000 : 1 couldn't initialize kVosInstagramUrl
1000 : 2 couldn't initialize kVosInstagramDashBoardUrl
class code : sub_code explanation
Constants
1000 : 1 couldn't initialize kVosInstagramUrl

notes

share

facebook share twitter share pinterest share google plus share linkedin share

free consulting

vangav's consultant