Skip to content

An Elasticsearch client tailored to data science workflows.

License

Notifications You must be signed in to change notification settings

uptake/uptasticsearch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

uptasticsearch

GitHub Actions Build Status codecov CRAN_Status_Badge CRAN_Download_Badge

Introduction

uptasticsearch tackles the issue of getting data out of Elasticsearch and into a tabular format in R and Python. It should work for all versions of Elasticsearch from 1.0.0 onwards, but is not regularly tested against all of them. If you run into a problem, please open an issue.

Table of contents

How it Works

The core functionality of this package is the es_search() function. This returns a data.table containing the parsed result of any given query. Note that this includes aggs queries.

Installation

R

Lifecycle Maturing

Releases of this package can be installed from CRAN:

install.packages(
  'uptasticsearch'
  , repos = "http://cran.rstudio.com"
)

or from conda-forge

conda install -c conda-forge r-uptasticsearch

To use the development version of the package, which has the newest changes, you can install directly from GitHub

remotes::install_github(
  "uptake/uptasticsearch"
  , subdir = "r-pkg"
)

Python

Lifecycle Dormant

This package is not currently available on PyPi. To build the development version from source, clone this repo, then :

cd py-pkg
pip install .

Usage Examples

The examples presented here pertain to a fictional Elasticsearch index holding some information on a movie theater business.

Example 1: Get a Batch of Documents

The most common use case for this package will be the case where you have an Elasticsearch query and want to get a data frame representation of many resulting documents.

In the example below, we use uptasticsearch to look for all survey results in which customers said their satisfaction was "low" or "very low" and mentioned food in their comments.

library(uptasticsearch)

# Build your query in an R string
qbody <- '{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "customer_comments"
              }
            },
            {
              "terms": {
                "overall_satisfaction": ["very low", "low"]
              }
            }
          ]
        }
      }
    },
    "query": {
      "match_phrase": {
        "customer_comments": "food"
      }
    }
  }
}'

# Execute the query, parse into a data.table
commentDT <- es_search(
    es_host = 'http://mydb.mycompany.com:9200'
    , es_index = "survey_results"
    , query_body = qbody
    , scroll = "1m"
    , n_cores = 4
)

Example 2: Aggregation Results

Elasticsearch ships with a rich set of aggregations for creating summarized views of your data. uptasticsearch has built-in support for these aggregations.

In the example below, we use uptasticsearch to create daily timeseries of summary statistics like total revenue and average payment amount.

library(uptasticsearch)

# Build your query in an R string
qbody <- '{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "pmt_amount"
              }
            }
          ]
        }
      }
    }
  },
  "aggs": {
    "timestamp": {
      "date_histogram": {
        "field": "timestamp",
        "interval": "day"
      },
      "aggs": {
        "revenue": {
          "extended_stats": {
            "field": "pmt_amount"
          }
        }
      }
    }
  },
  "size": 0
}'

# Execute the query, parse result into a data.table
revenueDT <- es_search(
    es_host = 'http://mydb.mycompany.com:9200'
    , es_index = "transactions"
    , size = 1000
    , query_body = qbody
    , n_cores = 1
)

In the example above, we used the date_histogram and extended_stats aggregations. es_search() has built-in support for many other aggregations and combinations of aggregations, with more on the way. Please see the table below for the current status of the package. Note that names of the form "agg1 - agg2" refer to the ability to handled aggregations nested inside other aggregations.

Agg type R support? Python support?
"cardinality" YES NO
"date_histogram" YES NO
date_histogram - cardinality YES NO
date_histogram - extended_stats YES NO
date_histogram - histogram YES NO
date_histogram - percentiles YES NO
date_histogram - significant_terms YES NO
date_histogram - stats YES NO
date_histogram - terms YES NO
"extended_stats" YES NO
"histogram" YES NO
"percentiles" YES NO
"significant terms" YES NO
"stats" YES NO
"terms" YES NO
terms - cardinality YES NO
terms - date_histogram YES NO
terms - date_histogram - cardinality YES NO
terms - date_histogram - extended_stats YES NO
terms - date_histogram - histogram YES NO
terms - date_histogram - percentiles YES NO
terms - date_histogram - significant_terms YES NO
terms - date_histogram - stats YES NO
terms - date_histogram - terms YES NO
terms - extended_stats YES NO
terms - histogram YES NO
terms - percentiles YES NO
terms - significant_terms YES NO
terms - stats YES NO
terms - terms YES NO