Export document list as CSV

[mtettke123]

Story:
I'm actually preparing my tax declaration - the first one since using paperless.
I've all documents sorted and tagged in paperless. So, I can search for all invoices a.e. which works great.

Wish:
As I now have a list of all invoices, an "export as CSV" button would be great, to export the list of all filtered elements. I could just use this list, import it to whatever and would have saved so much time and monkey work!

Wish 2 (much in the future and handled in another request):
If there was a custom field like "amount", it would be even greater....

[nlspln]

I would use this to print an inventory sheet to add to each binder so I can look at a glance which documents are in the binder and under what number I can find them (in case I need to do a lookup and I don't have access to the web interface)

[nodecentral]

Can something like that be done with the REST API, maybe not a CSV, but a list of files with a defined document type/tag etc ?

[23ewrdtf]

I'm looking for this too. I receive invoices in PDF and I would like to OCR them and then transfer them (tables in them) to csv or excel tables. There is https://github.com/invoice-x/invoice2data but it's clunky.

Maybe we can connect the two?

[mtettke123]

While I would really love this feature - I don't think we should put them together in one feature request: What I would love to have here, is just a generation of a CSV from listed items. Nothing fancy, pretty basic.

What you like to achieve - while being a great idea - is muuuch more complex (if I understand it correctly): analything the content of invoices and exporting... I know there are packages for this and DMS that can do this, but this is really complex!

[23ewrdtf]

Yes, I know it's not easy. Been trying to setup invoice2data but the results varies.

For now I can add invoice2data to Post-consumption script.

Detecting tables and basic csv export would be a huge help.

[mtettke123]

Just read a little bit into invoice2data. Seems like a fun project and good idea to integrate into paperlesspaperless combined with custom fields (we all are waiting for) to automagically insert a.e. amount of an invoice!

[23ewrdtf]

happy to test. I have lot's of invoices in various types and quality.

[Frogy76]

Is it possible to add invoice2data to paperless?

[pilot479]

I am another big supporter for this type of feature request please...csv export (ability to filter as well) of labelling records of the documents (titles, types, correspondents, etc).

[enricohessel]

I would like to see this feature as well for prepping taxes.
Best practice:

• Adding the new custom fields to ALL selected documents
• typing in the values if ocr does not work (e.g. price)
• generate a csv, e.g. filename + custom fields + dates

[georgniemeyer]

I have the same use case as @enricohessel and would really appreciate an easy way to generate an overview and perhaps even a sum of the custom field "invoice amount" for selected documents. That would make the annual tax declaration much easier.

[Radnabe]

Me too

[mtettke123]

If it only was possible to fund something like this!

[dast5022]

I also would like to see that function in the ui of paperless-ngx. In the meantime, perhabs a little python script could help:

from ast import Slice, Try
import csv
from inspect import cleandoc
from xml.etree.ElementTree import tostring
import requests
import yaml
import sys


def load_config():

    # Change your values here:
    config={}
    config['token']="yourtoken"
    config['api_url']="http://your-ip/api/"
    config['search_string']="tags__id__all=10"
    config['result_size']=1000
    config['delimiter_fields']=";"
    config['delimiter_tags']= ","
    config['output_file']="output.csv"

    return config


def load_enums(api_url, token, keyfield):
    headers = {'Authorization': f'Token {token}'}
    response = requests.get(api_url+"?page_size=10000", headers=headers)
    #print(response)
    enums = response.json()
    filtered_enums = {}
    for k in enums["results"]:
        filtered_enums[k["id"]] = k[keyfield]

    return filtered_enums


def load_documents(api_url, token, search_string, size):
    headers = {'Authorization': f'Token {token}'}
    url = api_url+"?"+str(search_string)+"&page_size="+str(size)
    print("trying", url)
    response = requests.get(url, headers=headers)
    document = response.json()

    if "results" in document and document["count"] >= 1:
        return document["results"]
    else:
        raise ExNoDocument(search_string)


def replace_field_value(value, mapping, categorie):
    if value != "" and value != None:
        try:  
            return mapping[value]
        except:
            print(f'Fehler: "{value}" in {categorie} nicht gefunden.')
            raise ExNoItem(str(value))
    else:
        return ""

class ExNoDocument(Exception):
    def __init__(self, filename):
        self.message = "No document found in paperless"
        self.filename = filename
        super().__init__(self.message, self.filename)    

class ExNoItem(Exception):
    def __init__(self, itemname):
        self.message = "item not found in paperless"
        self.itemname = itemname
        super().__init__(self.message, self.itemname)    

def main():
 
    config=load_config()
    token = config['token']

    # read enums from paperless
    try:
        correspondents = load_enums(config["api_url"]+"correspondents/", token, "name")
        document_types = load_enums(config["api_url"]+"document_types/", token, "name")
        paths= load_enums(config["api_url"]+"storage_paths/", token, "name")
        tag_list= load_enums(config["api_url"]+"tags/", token, "name")
    except ExNoItem:
        print("replacing enum failed")
        exit()

    # read documents from paperless
    try:
        documents = load_documents(config["api_url"]+"documents/", token, config["search_string"], config['result_size'])
    except ExNoDocument as e:
        print("No matching document was found")
        exit()

    # loop through all documents
    documents_out=[]
    i=0
    for doc in documents:

        # replace enums
        doc['correspondent'] = replace_field_value(doc['correspondent'], correspondents, "correspondents")
        doc['document_type'] = replace_field_value(doc['document_type'], document_types, "document_types")
        doc['storage_path'] = replace_field_value(doc['storage_path'], paths, "paths")
        
        # replace tags
        tags_in=[]
        tags_out=[]
        tags_in=doc['tags']
        for tag in tags_in:
            tag = replace_field_value(tag, tag_list, "tags")
            tags_out.append(tag)
        
        # transfer fields to destination list
        row={}
        row['id'] = doc['id']
        row['original_file_name'] = doc['original_file_name']
        row['correspondent'] = doc['correspondent']
        row['storage_path'] = doc['storage_path']
        row['document_type'] = doc['document_type']
        row['created_date'] = doc['created_date']    
        row['title'] = doc['title']
        row['tag_list']=config['delimiter_tags'].join(tags_out)
        documents_out.append(row)
        i+=1
    # export to csv
    csv_file_path = config['output_file']
    print("found "+str(i)+" lines and trying to write to "+csv_file_path)
    
    fieldnames = ['id','original_file_name', 'storage_path', 'correspondent', 'document_type', 'title', 'tag_list', 'created_date']
    
    with open(csv_file_path, 'w', newline='') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames, delimiter = config['delimiter_fields'])
        writer.writeheader()
        writer.writerows(documents_out)

if __name__ == '__main__':
    main()

[mtettke123]

just tested (path mounted into paperless-docker), works. Thank you! :-)

Do you have an idea how to add custom fields to the export?

[mtettke123]

Answering to myself, as solved with the help of ChatGPT and the API documentation:

adding field ID of dynamic field to config (guessed the ID...)
config['custom_field_id_for_betrag']=1

in main, when looping through the documents:

    for doc in documents:

        # custom fields
        betrag_value = None

        # Check if 'custom_fields' exists in the document
        if 'custom_fields' in doc:
            # Iterate through each custom field to find 'Betrag'
            for custom_field in doc['custom_fields']:
                if custom_field['field'] == config['custom_field_id_for_betrag']:
                    betrag_value = custom_field['value']
                    break  # Stop searching once 'Betrag' is found

finally adding the row
row['Betrag']=betrag_value if betrag_value is not None else ''

and the index:
fieldnames = ['id','original_file_name', 'storage_path', 'correspondent', 'document_type', 'title', 'tag_list', 'created_date', 'Betrag']

"works for me".

[docb7]

Thank you very much, that was exactly what I was looking for. I also would like to add media_filename to the output, but I just get error messages when trying to add that (to the enums block or in the row block) - as you might already have guessed, I have no clue about api but I guess, as I tried to understand the documentation, the media_filename value is under a different path (/api/documents//metadata/) while all other values are under /api/documents//. But I do not know, how to change that. It would be great if @dast5022 or you could point me in the right direction.

[andi0b]

I'm also missing this feature.

Some additional thoughts:

1. I would like to include this document list into the existing multi document ZIP file export functionality.
   The document list should be part of the ZIP file, one CSV in the root directory should be enough. There should be a column with a relative file path to the document.

2. It should support not only CSV, but also JSON and JSONL (json lines), and preferrably all formats can be added to the export at the same time

3. CSV separator should be locale (language) specific or changable
   In some locales the separator is a semicolon, because the decimal seperator is a comma. This is also called "European format" and an often missed crucial requirement from US based developers. European Excel versions for example can't simply open comma separated CSV files, this often causes big issues if you send it to non-tech people (simple double-clicking doesn't work). There is an example on the Wikipedia page about CSV.

4. Custom fields should be included, maybe notes too.

[bembembene]

mtettke123 how did u mount the path in docker? greetings ben