v24.5.4.0

This release includes several important updates and improvements:

• Updates and fixes to existing integrations, including LangChain, Vertex AI, ChromaDB, MindsDB Inference, and more.
• Enhanced test coverage for greater reliability.
• A schema refactor that includes changes to the structure for storing metadata of all objects like models, views, tables, etc.
• Bug fixes to the SQL API, including improvements to the LAST keyword.

All relevant updates have been accompanied by corresponding documentation updates.

What's Changed

• removing redundant package lists and sentence-transformers library by @QuantumPlumber in #8971
• Safe extraction for all tar files by @StpMax in #8928
• Fix tarfile safe extract by @StpMax in #9000
• Refactor Langchain Handler by @tmichaeldb in #8998
• Tests for sql features by @ea-rus in #8952
• Update query format in quickstart-tutorial.mdx by @vlad-ro in #9011
• [404] fix url in huggingface_inference_api.mdx by @ArnavK-09 in #9016
• Fix hardcoded API base by @ZoranPandovski in #9019
• Added Check for Import Errors When Running CREATE MODEL Statement by @MinuraPunchihewa in #9020
• updated hostname of sample dbs by @martyna-mindsdb in #9025
• Fixed Installation of Dependencies for Integrations That Refer to Other requirements.txt Files by @MinuraPunchihewa in #9022
• fixed slack chatbot tutorial by @martyna-mindsdb in #9026
• Fix delete from table in database by @ea-rus in #9017
• Added File Path and URL Support to the Vertex AI Integration by @MinuraPunchihewa in #9007
• clean up handler install log messages by @hamishfagg in #8720
• Docker build improvements by @hamishfagg in #8999
• Add 'seed db' function to MSSQL tests by @hamishfagg in #8742
• Fixed Installation of Dependencies for Integrations That Refer to requirements.txt Files (Docker-supported) by @MinuraPunchihewa in #9034
• Fixed Installation of Dependencies When requirements.txt Files Contain Comments by @MinuraPunchihewa in #9038
• Fixed the Initializer for the EventStoreDB Integration by @MinuraPunchihewa in #9029
• Fixed Unused Imports and Dependencies in Rockset Integration by @MinuraPunchihewa in #9031
• updated email integration docs by @martyna-mindsdb in #9035
• Make alembic use mindsdb log level by @hamishfagg in #9028
• Bump version for release by @hamishfagg in #9041
• Fix LAST with slack by @ea-rus in #9036
• Updated LightFM Integration Docs with Tip to Install Linux Dev Pakcages by @MinuraPunchihewa in #9042
• updated ollama docs by @martyna-mindsdb in #8995
• Updated Snowflake Integration Docs with Tip to Install Git by @MinuraPunchihewa in #9032
• Add RAG model and utilities for Mindsdb integration by @dusvyat in #8996
• Update requirements.txt by @martyna-mindsdb in #9048
• Update requirements.txt by @martyna-mindsdb in #9050
• update sampledatabases to new k8s server by @lucas-koontz in #8986
• The "database" field in the form is changed to not required. by @alejandrovillegas in #9052
• Fixed Column Structure of chat_messages Table by @MinuraPunchihewa in #9015
• Fix BYOM engine create by @StpMax in #9054
• updated agents docs by @martyna-mindsdb in #9058
• updated broken links in mongo docs by @martyna-mindsdb in #9059
• Bump version to 24.4.2.1 by @StpMax in #9061
• Feature/update dd docs by @MinuraPunchihewa in #9062
• Clear LAST value in editor by @ea-rus in #9049
• bump litellm version by @lucas-koontz in #9064
• Managing knowledge base columns by @ea-rus in #9005
• Fix typo by @vlad-ro in #9065
• Initial RAG 2.0 integration by @dusvyat in #9068
• Mindsdb inference endpoints by @ZoranPandovski in #9044
• Disable storage backup by @StpMax in #9073
• Bump version to 24.4.3.0 by @StpMax in #9074
• Kb updates 04.2024 by @ea-rus in #9069
• 9078-bug-cannot-create-a-langchain-engine by @dusvyat in #9080
• Remove custom GH actions and use shared ones by @hamishfagg in #9072
• updated agent skills in docs by @martyna-mindsdb in #9082
• Сhanged default agent type for supporting conversational history by @ea-rus in #9087
• Switch to new GHA runners by @hamishfagg in #9085
• Remove prometheus dir from dockerfile by @hamishfagg in #9092
• Mysql improvments by @ZoranPandovski in #9081
• Infernece docs by @ZoranPandovski in #9100
• updated mindsdb readme by @martyna-mindsdb in #9055
• Fixed Load Tests and Test Docs (README) by @MinuraPunchihewa in #9094
• use cases docs plus docs updates by @martyna-mindsdb in #9095
• Minor tweaks to retrieval skill and file ingestion by @dusvyat in #9093
• Fixes for RAG Integration to Python SDK by @tmichaeldb in #9098
• mysql and postgres docs + readme updates by @martyna-mindsdb in #9102
• MS SQL Handler Improvements by @MinuraPunchihewa in #9083
• Fix REST API Endpoints URL by @ZoranPandovski in #9108
• docs cleanup by @martyna-mindsdb in #9112
• add bigquery to cloud image by @hamishfagg in #9114
• updated triggers docs by @martyna-mindsdb in #9116
• Mindsdb provider for langchain by @ea-rus in #9046
• Update README.md by @martyna-mindsdb in #9125
• removed mindsdb cloud ref in docs by @martyna-mindsdb in #9118
• openapi docs for agents/skills/chatbots by @martyna-mindsdb in #9113
• updated use cases docs by @martyna-mindsdb in #9110
• [fix] dataframe encode in byom describe by @paxcema in #9124
• Fixed Script for Running Handler Unit Tests in Workflow by @MinuraPunchihewa in #9103
• [docs] Improved readme for anomaly detection handler by @paxcema in #9130
• MindsDB Inference Handler improvements by @MinuraPunchihewa in #9099
• Drop semaphor table by @StpMax in #9134
• Support New Retrieval Skill for RAG by @tmichaeldb in #9119
• Update Web Handler README by @ZoranPandovski in #9138
• Schema refactor by @ea-rus in #9001
• Remove permanent ml handlers by @ea-rus in #9128
• Update Avaiability of handlers by @ZoranPandovski in #9117
• updated docs for LAST by @martyna-mindsdb in #9147
• Clickhouse improvments by @ZoranPandovski in #9111
• updated docs links in completed readmes&docs for integrations by @martyna-mindsdb in #9150
• added info to langchain about langfuse support by @martyna-mindsdb in #9153
• updated lightwood docs by @martyna-mindsdb in #9148
• Chromadb: sync handler storage on rw operations by @ea-rus in #9140
• Use SQLDatabaseToolkit for Agent sql Skill by @tmichaeldb in #9132
• Fully Support Agents File & Web Retrieval by @tmichaeldb in #9137
• updated a link in openai finetuning tutorial by @martyna-mindsdb in #9156
• remove random file by @hamishfagg in #9157
• Prepare to rename default branch to "main" by @hamishfagg in #9089
• Update doc links for branch rename by @hamishfagg in #9158
• Added Tip for Enabling Dev Mode (Windows) Before Installing Extension by @MinuraPunchihewa in #9146
• Fix drop ml-engine by @ea-rus in #9143
• Project and database can't have same name by @ea-rus in #9127
• removed ref to mindsdb (cloud) pro by @martyna-mindsdb in #9154
• updated use cases diagrams by @martyna-mindsdb in #9167
• Anomaly detection docs by @martyna-mindsdb in #9152
• Moved the File Splitter Imports to the Function They are Used in by @MinuraPunchihewa in #9162
• Mongo shell: describe model error shows right command by @ea-rus in #9126
• Model column mapping by @ea-rus in #9166
• remove references to removed table by @hamishfagg in #9165
• Fix mysql connection status tracking by @hamishfagg in #9120
• [WIP] Fix langchain handler install by @hamishfagg in #9115
• Restrict concurrent gha workflows by @hamishfagg in #9091
• Rest api for jobs by @ea-rus in #9142
• Bump type_infer to 0.0.20 by @hamishfagg in #9170
• add docker commands to makefile by @hamishfagg in #9173
• Update pydantic to 2.0 by @hamishfagg in #9121
• updated handler link by @martyna-mindsdb in #9175
• Added the Troubleshooting Guide for the MindsDB Inference Handler by @MinuraPunchihewa in #9171
• MindsDB Inference - Fixed Validation of API Keys on Model/Engine Creation by @MinuraPunchihewa in #9174
• AltibaseHandler improvements by @colehaanlee in #9159
• python sdk docs for agents and jobs by @martyna-mindsdb in #9168
• Snowflake Integration Improvements by @MinuraPunchihewa in #9139
• Remove unused files by @ZoranPandovski in #9002
• added feedback question to all docs by @martyna-mindsdb in #9182
• Prepare Agents for Release by @tmichaeldb in #9172
• Fix concurrency settings by @hamishfagg in #9177
• updated mongo syntax by @martyna-mindsdb in #9183
• removing agents from python sdk docs by @martyna-mindsdb in #9184
• Fixed Authentication Errors Raised by Anyscale Handler by @MinuraPunchihewa in #9191
• Fix pydantic warnings by @hamishfagg in #9186
• Improve caching of docker builds by @hamishfagg in #9194
• Twelvelabs fix by @ZoranPandovski in #9199
• Fix extras arg in docker builds by @hamishfagg in #9202
• updates to sql api docs by @martyna-mindsdb in #9200
• removed rest api for agents by @martyna-mindsdb in #9201
• Pydantic V2 migration by @hamishfagg in #9193
• unindexed agents docs for rest api and python sdk by @martyna-mindsdb in #9206
• updated automated finetuning tutorials by @martyna-mindsdb in #9207
• Verify web by @ZoranPandovski in #9190
• enhanced user journey through docs by @martyna-mindsdb in #9204
• updated mql tutorials by @martyna-mindsdb in #9214
• Hide sensitive connection args by @StpMax in #9192
• bump lightwood version by @hamishfagg in #9212
• Anyscale Endpoints Integration Improvements by @MinuraPunchihewa in #9205
• Update docker-compose.yml by @ZoranPandovski in #9213
• Minor updates to contributors docs by @ZoranPandovski in #9224
• Project as database for create/retrain/finetune by @ea-rus in #9135
• Access All Tables of Given Database Only for SQL Completions (fix) by @tmichaeldb in #9223
• Upgrade Werkzeug to 2.3.8 by @ZoranPandovski in #9217
• Upgrade mintlify in docs by @ZoranPandovski in #9230
• Upgrade llama-hub version by @ZoranPandovski in #9229
• Model Validation Fixes for Twelve Labs and ChromaDB by @MinuraPunchihewa in #9228
• updated trigger docs by @martyna-mindsdb in #9219
• Fixed Auth Mechanism for ScyllaDB/Cassandra by @MinuraPunchihewa in #9231
• fix wrong sensitive_info arg by @StpMax in #9233
• New release by @ZoranPandovski in #9235
• Fixed OpenAI Conersational Model by @MinuraPunchihewa in #9227
• Add gpt-4o by @ZoranPandovski in #9211
• Fix tests by @ZoranPandovski in #9236

New Contributors

• @vlad-ro made their first contribution in #9011
• @ArnavK-09 made their first contribution in #9016
• @colehaanlee made their first contribution in #9159

Full Changelog: v24.4.2.0...v24.5.4.0