integrate meza

[reubano]

There is a lot of overlap in functionality with the utils module and meza. Where possible, I should merge redundant functions and move new ones.

[COLABORATI]

Am I right to hope that the integration with sqlalchemy will bring a nice save-to-database feature to riko? I am currently looking what would be the best way to store things in db without breaking a pipeline, so that 'save-to-db' could be done seamless in different stages of the data pipeline.

[reubano]

Technically, you can already do something similar to that example with riko since it shares a similar data structure.

>>> from riko.modules.fetch import pipe
>>> from meza import fntools as ft
>>> from .models import Table

# Table is a sqlalchemy.Model class
# db is a sqlalchemy database instance
>>> stream = pipe(conf={'url': 'https://news.ycombinator.com/rss'})
>>> for data in ft.chunk(stream, chunk_size):
...     db.engine.execute(Table.__table__.insert(), data)

But you are correct in that this will "break" the pipeline since you are consuming the iterator. I do have future plans to have a pipe that outputs to sqlalchemy supported dbs. Maybe something like this:

>>> from riko.collections.sync import SyncPipe
>>> flow = (
...     SyncPipe('fetch', conf={'url': 'https://news.ycombinator.com/rss'})        
...         .write(conf={'uri': 'sqlite:////tmp/test.db', 'table': 'raw'})
...         .filter(conf={'rule': {'field': 'link', 'op': 'contains', 'value': 'python'}})
...         .write(conf={'uri': 'sqlite:////tmp/test.db', 'table': 'filtered'})
...         .sort(conf={'rule': {'sort_key': 'title'}}))

But this doesn't really have anything to do w.r.t meza.

[reubano]

Add feel free to add this request to #8.