SOLIDATA
backend

WARNING : almost deprecated

From now please refer to the cloned and enhanced repo : solidata-backend

---

PRESENTATION

part 2a/3 of the TADATA! sofware suite (ApiViz / Solidata_frontend / OpenScraper )

Building a public service for managing, consolidating, and sharing "api-fied" data

---

GOALS

• allow you to consume data from your own csv/xls files or external APIs ;
• consolidate your data : apply your own datamodel, simplify columns, apply metadatas, ... ;
• share the consolidated datas on API endpoints with the level of openness you decide (opendata, commons, collective, private) ;
• manage all your data and your recipes by projects ;
• be able to manage projects by teams and share data/recipes/datamodels...

---

DEVELOPPERS

Nice to see you around ! For a more complete tour check out :

• our guidelines
• our changelog
• our contribution guide

---

TECHNICAL POINTS

Tech stack

• Language : Python 3.6... yes ... I know ... Python again ... hmmm ... gnnn ... don't judge me ?!
• Framework : Flask... minimalistic Python framework
• API : Flask-RestPlus... Swagger documentation integrated
• Security : Flask-JWT-extended... wrapper JWT for Flask
• Emailing : Flask-email... templating, sending, etc...
• Data management : Pandas... to do operations on datasets...
• Geocoding : Geopy... to use several geocoders : Nominatim, BAN, etc...

Features :

• JWT (JSON Web Tokens) :
  • access and refresh token for security over all the app
• user :
  • anonymous login (automatic)
  • login / register user
  • confirm user by sending a confirmation link (protected) in an email
  • password forgotten by sending a link (protected) in an email with redirection to new password form
  • reset password from client interface (protected) ...
• Documentation
  • on all API endpoints with Swagger (and some patience from the developer)

Current features TO DO :

• user :
  • edit user
  • edit email
• add/edit a project :
  • edit team and access levels
  • add project to users involved

---

WHAT'S NEW ?

• check out our our changelog for more details

---

INSTALLATION WALKTHROUGH

LOCALLY

• clone / fork the depo ...

• create a virtual environment for Python3

   python3 -m venv venv
   pip install --upgrade pip
   pip install -r requirements.txt

• create a secret config_file config_prod.py in the folder ./solidata_api based

   cp ./solidata_api/config_pro_exampled.py ./solidata_api/config_prod.py
   nano ./solidata_api/config_prod.py 

• pay attention at the MONGO_URI variable depending on your local mongodb configuration...

  • The following is the standard URI connection scheme (from mongo documentation):

     mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
    

• run the app in dev mode (without possibility to send email) :

   python appserver.py --mode=dev

• run the app in dev mode (with possibility to send email) :

   python appserver.py --mode=dev_email

PRODUCTION

• Check our snippets folder for more instructions

• for now we are using the following configuration

  • droplet in digitalocean.com
  • ubuntu 18.04
  • 3Go RAM / 2CPU
  • 60Go memory

• configure your server (user, firewall...):

  • cf : docs 1
  • cf : docs 2
  • cf : docs 3

• install mongodb :

  • cf : docs 1
  • cf : docs 2
  • cf : docs 3
  • cf : docs 4

• install nginx :

  • cf : docs 1
  • cf : docs 2
  • cf : docs 3

• install git on your server

  • cf : docs 1 - basically :

     sudo apt-get update
     sudo apt-get install git
     git --version

  • go to your directory and init git :

     git init . 
     git remote add origin https://github.com/entrepreneur-interet-general/solidata_backend.git
     git pull origin master
    

• same steps than for local installation (virtual env, install dependencies, config_prod.py file, ) ...

• run the app in production mode (with possibility to send email) :

   python appserver.py --mode=production

---

INSPIRATIONS / BENCHMARK

• Dataiku, Parabola, but they are proprietary solutions (and too complex for our pruposes)... Not to mention they are very expensive...
• Match ID, but the backend doesn't resolve the "sharing" part with levels as ["opendata", "commons", "collective", "private"], and no BDD to back up recipes

---

CREDITS

Solidata's backend team thanks :

• the SocialConnect project, aka "Carrefour des Innovations Sociales"
• the EIG program by Etalab
• the CGET
• Christophe Ninucci for his help, advices and taste for memes
• the creator and community of Flask RestPlus

Contacts - maintainance :

• Julien Paris, developer (aka JPy on Twitter, or Jpy on Github)

---

SCREENSHOTS

---

---

---

---