In today's rapidly advancing landscape of machine learning and artificial intelligence (AI), ensuring the security of AI models has become an imperative. These models serve as the driving force behind decision-making in a myriad of applications. As such, safeguarding their integrity and protection against potential attacks remains a top priority. Unfortunately, the critical components of this AI ecosystem, namely Jupyter notebooks and models, are often overlooked in routine security assessments, leaving them vulnerable and appealing targets for potential attackers. Initial tests with open-source tools painted a grim picture, underscoring severe limitations and an urgent need for a bespoke solution. As AI/ML models continue to reshape industries and drive innovation, the significance of model security is paramount. However, amidst the excitement of AI's potential, the security of Jupyter notebooks and AI models has been consistently relegated to the shadows.
A compelling illustration of this security blind spot revolves around the widespread usage of Jupyter notebooks. Within these digital pages lie the blueprints, codes, and algorithms that breathe life into many of today's AI-driven products and services. However, like any other codebase, Jupyter notebooks are not immune to vulnerabilities that could inadvertently find their way into the final product. These vulnerabilities, if left unaddressed, can become gateways for cyber-attacks, risking not only the integrity of AI-driven products but also the security of user data.
AIShield Watchtower is designed to automate model and notebook discoveries and conduct comprehensive vulnerability scan. Its capabilities go beyond merely identifying all the models and notebooks within your repository. It assesses risks, such as hard-coded secrets, PIIs, outdated/unsafe libraries, model serialization attacks, custom unsafe operations etc.
AIShield Watchtower stands out with its capability to categorize scans into four distinct risk levels: "low," "medium," "high," and "critical." This classification equips organizations with the ability to tailor their security efforts to the level of risk detected. Its adaptive approach and meticulous risk categorization significantly bolster security efforts, fortifying them effectively. Watchtower's alignment with industry standards such as OWASP, MITRE, CWE, NIST AI RMF MAP functions further enhances its market standing by providing advanced security solutions.
AIShield Watchtower can be used to inspect vulnerabilities in Jupyter notebooks and AI/ML Models (.h5, .pkl and .pb file formats).
For using AIShield Watchtower, clone Watchtower repo. Install prerequisites and scan your notebooks and AI/ML models. Some starting sample test files are available within the Watchtower repo.
- For running Watchtower in CLI or UI version, python3 and pip should be installed in the host system.
- For running UI-Docker version, docker and docker-compose should be installed in the host system
Cloning Watchtower repo
git clone https://github.com/bosch-aisecurity-aishield/watchtower.git
Once Git repositories cloned, change directory.
NOTE: For docker users, refer UI-Docker
cd watchtower/srcInstall Watchtower related dependency libraries using following commands
pip install -r requirements.txt
python -m spacy download en_core_web_lgInspection of Jupyter Notebooks and ML/DL models can be done by any of the three methods:
To View the available options in CLI mode
python watchtower.py -h python watchtower.py --repo_type=github --repo_url=<Enter Repo Url> --branch_name=<Enter Branch Name> --depth=<Enter a number>NOTE: branch_name and depth parameters are optional. Default value of branch_name is main and default depth value is 1
python watchtower.py --repo_type=huggingface --repo_url=<Enter Hugging Face Url>python watchtower.py --repo_type=s3 --bucket_name="<Enter Bucket name>" --region="<Enter region of s3 bucket>" --aws_access_key_id="<Enter aws access key>" --aws_secret_access_key="<Enter aws secret key>"#Select Repo_type = file for scanning individual file
python watchtower.py --repo_type=file --path=<Enter path of File>
#Select Repo_type = file for scanning individual file
python watchtower.py --repo_type=folder --path=<Enter path of Folder>
For using Watchtower UI, execute following command
python watchtower_webapp.py open browser and paste: http://localhost:5015/watchtower-aishield
For using Watchtower UI, build docker image for Watchtower and run Watchtower image
cd watchtower
docker-compose build
docker-compose up open browser and paste: http://localhost:5015/watchtower-aishield
On successful completion of scan, Watchtower vulnerability reports will be available in reports folder in Watchtower root folder.
For stopping and removing Watchtower image execute following
docker-compose downFor quick getting started, you may try Watchtower Playground by visiting https://app-watchtower.boschaishield.com
In the Watchtower Playground, users can scan Notebooks and AI/ML models available in public Github Repos by providing public Github Repo URL. After completion of vulnerability scan, reports will be available to download in the Playground screen.
On successful completion of the Watchtower scan, three reports will be generated in the following path :
-
For CLI Mode - all three reports will be available inside the Watchtower src folder. Users may refer last line of the summary report in the console for the complete path of the reports location
-
For UI Mode - all three will be reports will be available inside the Watchtower src folder. Users may refer to the success message on the UI to get the path of the reports location
-
For UI-Docker Mode - all three reports will be available inside the Watchtower reports folder. Users may refer to the success message on the UI to get the path of the reports location
- Summary Report - In summary report will provide information on number of model files and notebook files detected, Number of vulnerabilities detected and Count of those vulnerabilities mapped to Critical, High, Medium and low. Sample snippet of Summary Report:
{
"Repository Type": "github",
"Repository URL": "https://github.com/bosch-aisecurity-aishield/watchtower.git",
"Total Number of Model Found": 17,
"Total Number of Notebooks & Requirement files Found": 6,
"Total Number of Model Scanned": 17,
"Total Number of Notebooks & Requirement files Scanned": 6,
"Total Notebook & Requirement files Vulnerabilities Found": {
"Critical": 0,
"High": 40,
"Medium": 17,
"Low": 1
},
"Total Model Vulnerabilities Found": {
"Critical": 0,
"High": 4,
"Medium": 2,
"Low": 8
}
}- Severity Mapping Report - In this report, details of High, Medium and Low mapping to Model or Notebooks is reported. Sample snippet of Severity Mapping Report:
{
"type": "Hex High Entropy String",
"filename": "repo_dir_1696827949/sample_test_files/sample_notebook_files/classification_notebook.py",
"hashed_secret": "8d1e60a0b91ca2071dc4027b6f227990fb599d27",
"is_verified": false,
"line_number": 47,
"vulnerability_severity": "High"
},
{
"type": "Secret Keyword",
"filename": "repo_dir_1696827949/sample_test_files/sample_notebook_files/classification_notebook.py",
"hashed_secret": "8d1e60a0b91ca2071dc4027b6f227990fb599d27",
"is_verified": false,
"line_number": 47,
"vulnerability_severity": "High"
}- Detailed Report - In this report, Watchtower users will be able to find all logs generated during the vulnerability scan.
- Model and Notebook Detection: Automatically recognizes AI/ML models and Notebooks within a provided repository. Supported file format is H5, pickle, saved model, .ipynb
- Scanning:Executes thorough scans of the models and notebooks to detect potential safety and security concerns.
- Report Generation: Produces comprehensive reports that classify the scanned files containing "low," "medium,", "high" and "critical" risk.
- Supported Repositories : AIShield Watchtower supports integration with GitHub and AWS S3 buckets, allowing for automated scanning of Git repositories and AWS S3 buckets to identify potential risks.
- User Interface (UI): Offers an intuitive user interface for conducting repository scans.
- Real-time scanning is an essential component for the swift identification and mitigation of emerging threats. This feature ensures that any changes or additions to AI/ML models and notebooks are promptly analyzed. It enables immediate action against potential vulnerabilities, thereby preserving the integrity of AI/ML applications.
- Versatile Framework Support allows Watchtower to cater to a wide array of AI/ML projects by ensuring compatibility with diverse model frameworks. This versatility enables organizations to leverage Watchtower regardless of the frameworks they utilize, making it a universally applicable security tool.
- Dynamic Risk Identification plays a pivotal role in adapting to evolving threats and vulnerabilities within the ever-changing landscape of AI/ML. This adaptability is vital as new threat types continually emerge, demanding a security tool that can evolve and adapt in tandem.
- Comprehensive Assessment offered by Watchtower covers a wide spectrum of vulnerability assessments, identifying and analyzing a broad range of risks. It meticulously examines all aspects of potential threats, ensuring that individuals and organizations possess comprehensive knowledge of their security landscape.
- Industry Standards Compliance is ensured through alignment with renowned security standards such as OWASP, MITRE, and CWE. This compliance guarantees that Watchtower adheres to globally recognized security practices, establishing a baseline of security and fostering trust among stakeholders.
- Efficiency and Scalability are achieved through automated assessments, significantly accelerating security workflows. This efficiency is critical for organizations expanding their AI/ML initiatives, ensuring that security does not hinder development processes during scaling.
- Seamless Integration is a valuable feature that allows Watchtower to integrate effortlessly with popular AI/ML platforms and repositories. It simplifies the incorporation of Watchtower into existing development ecosystems, streamlining security implementation and ensuring consistency across platforms.
- Informed Decision-Making is facilitated by detailed vulnerability reports generated by Watchtower. These reports empower organizations to prioritize actions and allocate resources effectively, enabling prompt addressing of critical vulnerabilities and optimal resource utilization.
- Competitive Advantage is gained by leveraging advanced security tools like Watchtower in a market where security is of paramount concern. This advantage not only appeals to clients and end-users but also instills confidence among stakeholders. It underscores the organization's commitment to securing its AI/ML assets against a wide range of risks, from model tampering to unauthorized data access. This comprehensive assessment ensures a thorough examination of all possible vulnerabilities, leaving no stone unturned in safeguarding AI/ML assets.
- Support only for public Git and huggingface repository.
- Enable AWS S3 bucket support by configuring role-based credentials, where a specialized role is crafted to grant minimal read-only access.
- Doesn't support scanning of .pb file from s3 buckets.
- Presidio analyser has 1000000 (1GB) has max length. Any data greater than 1GB will not be captured in reports.
- Possible miss-match in severity levels from Whispers library and Watchtower severity levels.
- Watchtower application is tested in Ubuntu LTS 22.04.
- Reduce false positives by using these customized versions that detect vulnerabilities more accurately.
- Refine PII severity assessment with contextual rules for varied levels.
- Enhance model vulnerability detection by adding checks for the embedding layer's potential issues.
- Integration with Github actions
- Repositories cloned from GitHub and Hugging Face during watchtower analysis will not be automatically removed post-analysis. It is advisable to manually delete these folders found within the 'src' directory.
Contributions are always welcome! See the Contribution Guidelines for more details.
- Parmar Manojkumar Somabhai
- Amit Phadke
- Deepak Kumar Byrappa
- Pankaj Kanta Maurya
- Ankita Kumari Patro
- Yuvaraj Govindarajulu
- Sumitra Biswal
- Amlan Jyoti
- Mallikarjun Udanashiv
- Manpreet Singh
- Shankar Ajmeera
- Aravindh J
This project is licensed under the Apache License. See LICENSE for details.
First and foremost, we want to extend our deepest gratitude to the vibrant open-source community. The foundation of AI Watchtower by AIShield is built upon the collective wisdom, tools, and insights shared by countless contributors. It's a privilege to stand on the shoulders of these giants:
- Yelp's detect-secrets: Used for scanning configurations to prevent accidental commitment of sensitive data.
- Safety: Used for scanning python dependencies for known security vulnerabilities and suggests the proper remediation for vulnerabilities detected.
- Whispers : Identify hardcoded secrets in static structured text
- Whispers: Advanced secrets detection
- Presidio Analyzer: The Presidio analyzer is a Python based service for detecting PII entities in text
- PII Confidentiality Impact Levels
- Top open source licenses and legal risk for developers
- Secure API keys
- OWASP Top 10: Vulnerability and Outdated Components
- Chris Anley - Practical Attacks on Machine Learning Systems
- Abhishek Kumar - Insecure Deserialization with Python Pickle module
- Shibin B Shaji - Using Python’s pickling to explain Insecure Deserialization
- TensorFlow Model Comparison: Insights from Saturn Cloud [1, 2, 3 ]
- TensorFlow Remote Code Execution with Malicious Model
- TensorFlow layers embedding causes memory leak
- Adding embeddings for unknown words in Tensorflow
- Data Leakage in Machine Learning
- Race Conditions and Secure File Operations
- Nuthdanai Wangpratham - Feature Scaling Matters for Accurate Predictions
- Predicting how and when hidden neurons skew measured synaptic interactions
- Deserialization bug in TensorFlow machine learning framework allowed arbitrary code execution
- Tom Bonner - A Deep Dive into Security Risks in TensorFlow and Keras
- Security in MLOps Pipeline
- A traditional attack vector applied to AI/ML Models
... and to many others who have contributed their knowledge on open-source licenses, API key security, MLOps pipeline security, and more.
In creating AI Watchtower, it's our humble attempt to give back to this incredible community. We're inspired by the spirit of collaboration and are thrilled to contribute our grain of sand to the vast desert of open-source knowledge. Together, let's continue to make the AI landscape safer and more robust for all!