Devcontainer for development environment

[g-kartik]

Many new contributors face problems while setting up the development environment on their system. Devcontainers can simplify the process and save their valuable time. This pull request configures devcontainer for running cvat both on local system as well as on GitHub Codespaces.

Motivation and context

closes #6230

Guide from the documentation

Dependencies

• Install Chrome
• Install VS Code or VS Code Insiders in case when extensions are not installed after start of the dev container
  • Install Dev Container extension for local dev container with all the system requirements as specified in the documentation.
  • Install GitHub Codespaces extension for Codespaces
• Install git
  • For Windows users following guides may be helpful
    • Get started to use Git on Windows Subsystem for Linux
• Connect to GitHub using SSH. (guide)
• The system should have POSIX sh shell for running initial shell scripts

Optional steps

• In the root directory of the repository create a .env file with the variables mentioned in dist.env file and modify their values
  • If not created, default values are taken from the docker-compose file inside the .devcontainer/docker-compose.yml
• In .devcontainer directory create a .env file with variables mentioned in dist.env file
• The GIT_BRANCH_ISOLATION environment variable is used at the build time of the dev container. It enables the persistence of container data between builds by using Docker volumes namespaced by the Git branch name. It is set to true by default. More details on this are provided in a later section.

Local Dev-Container guide

• Upon opening the repository in VS Code, click on the green color icon in the bottom left corner of the window labeled Open Remote Window and select reopen in container
  Initially it shall take some time to set up as it will build the dev container image and pull all the required docker images. Subsequent builds shall be fast and use cache to build and run the container
• The container data is persisted between builds with the help of named volumes and each volume is namespaced by git branch name. Therefore one can create separate dev container environment specific to the current working git branch. This can be helpful for reviewing pull requests and making quick bug fixes

GitHub Codespaces guide

• Sign into GitHub from VS Code account panel or using GitHub Codespace extension
• Click on the green color icon in the bottom left corner of the window labeled Open Remote Window and select create new codespace
• Select your forked CVAT repository and the branch name
• Select the machine type
• Again from the Open Remote Window menu, select connect to codespace and select the codespace you created in the previous step amd the container shall start building
• The container data is persisted between builds with the help of named volumes and each volume is namespaced by git branch name. Therefore one can create separate dev container environment specific to the current working git branch
• One can also create separate codespace for each branch by repeating above steps with the new chosen git branch
• The codespace shall automatically stop after 30 minutes of inactivity. One can manually stop and delete it using the VS code command panel

Run and debug guide

Steps are common for both local and codespaces remote development

• Open the Run and Debug panel in VS Code and launch the following configurations:
  • devcontainer: server
    It shall do the following operations:
    • Check availability of dependent docker containers
    • Migrate the migrations to the postgres database
    • Create a superuser as per the environment variables specified in .env file or use the default values
    • Run a background django server at port 8080 for serving the opa container
    • Start debug process for django server at port 7000
    • Start debug process for a rq-worker process listening on all the queues
  • devcontainer: ui
    It shall do the following operations:
    • Compile the node modules and keep the webpack-dev-server running in the background
    • Launch Chrome window in debug mode and open the CVAT web application running on port 3000
  • devcontainer: server: unit tests
    This launch config is used to run and debug python unit tests for the django apps and cvat-cli.
• One can debug the django code, rq-worker code and javascript code from the devcontainer: django and devcontainer: rq worker, devcontainer: ui panels in the debug console respectively
• After launching the server and ui launch configs, one should be able to log in to the cvat website running on the localhost:3000 with username and password as admin

Dev-Container Features

• The dev container image is based on the official CVAT docker image at cvat/server:dev Upon every rebuild, the dev container shall try to pull the latest base image and therefore it will always have the latest upstream changes without any user intervention
• The dev container pre-installs all the extensions specified in devcontainer.json file
• The default python virtual environment contains packages installed from testing.txt requirements file which inherits development, production and dataset_manifest requirements files
• Contains an additional python virtual environment for running pytest. It can be activated by selecting /opt/venv-test/bin/python in the Python: Select interpreter command pallette menu
• All the python packages in the virtual environments as well node dependencies are updated when the container runs for the first time after rebuilding the container. This is done via the postCreateCommand specified in devcontainer.json file
• Since the base image does not contain installation metadata for the datumaro, it clones the git repo every time the packages are updated; this takes a long time for the update to finish. To avoid this, its git commit_hash value from the base image is saved and used to check if an update is required
• Pip cache is persisted between rebuilds
• The default shell is zsh
• The container user is devcontainer and it is a non-root user; however, one can access the root user via sudo without any password to perform root operations like installing development-specific applications
• To permanently include an ubuntu package in the dev container such that all other users can access them, one can add it to apt package installation section in .devcontainer/Dockerfile, and rebuild the container
• Additional python packages can be installed into the virtual environment by command pip install. They shall not persist between container builds; therefore, it is just useful for testing new packages. They can be made to persist by adding them to the requirements file and rebuilding the image
• Git configurations on the host machine are mounted into the container. So things like user.name and user.email are already configured inside the container
• SSH keys on the host machine are mounted into the container. So one should be able to use your GitHub ssh keys to access GitHub
• SQLTool extension is pre-configured for the cvat-db database
• Local terminal can be accessed from the dev container window by selecting Terminal: Create New Integrated Terminal (Local) from the command pallette. It can be used to run command on the host machine. It can be useful for accessing docker running on the host machine
• Supports docker in docker to start and stop docker containers for running pytest tests and debug it without disturbing the main development containers and its ports
• Serverless components can be deployed from within the dev container as per the documentation
• User profile data including shell history are synced between the host and the container
• Uses dependabot for creating automatic PRs for bumping dev container feature versions
• It contains OPA and nuctl cli applications preinstalled

Limitations

• Cypress testing with browser is not supported in dev container and has to be run externally
• In GitHub Codespaces, running recursive chown on an existing directory in Dockerfile is very slow. So it shall take a while to build the container for the first time

How has this been tested?

Manual testing on Linux and Windows machines, and GitHub Codespaces.

Checklist

• I submit my changes into the develop branch
• I have created a changelog fragment
• I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• I have linked related issues (see GitHub docs)
• [ ] I have increased versions of npm packages if it is necessary
  (cvat-canvas,
  cvat-core,
  cvat-data and
  cvat-ui)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

• New Features

  • Introduced Git branch isolation for enhanced development environment management.
  • Added Docker Compose configurations for CVAT development and Nuclio service setup.
  • Automated scripts for setting up and initializing the CVAT server.
  • Launched comprehensive guides and updated documentation for setting up remote development environments and running tests.

• Documentation

  • Updated guides for contributing to development environments and running tests with new instructions and configurations for remote development.

• Bug Fixes

  • Adjusted configurations and task initialization patterns in VS Code to enhance development workflow and error handling.

• Chores

  • Updated .gitignore to include new development environment files and configurations.

[coderabbitai]

Walkthrough

This update brings significant enhancements to CVAT's development environment by introducing a structured setup within a development container. It streamlines the development process, integrates essential tools, and optimizes configurations for efficient development workflows.

Changes

File Path	Change Summary
.devcontainer/...	Introduces configuration files and scripts for Git branch isolation and setting up essential tools for CVAT development.
.vscode/...	Updates VS Code configurations to support the new development environment, enhancing launch and task settings for various components.
site/content/en/docs/...	Updates documentation to provide detailed guidance on utilizing the new development environment for contributors.
cvat/apps/lambda_manager/views.py, cvat/settings/development.py	Makes minor code adjustments to better integrate with the new development container environment.

Assessment against linked issues

Objective	Addressed	Explanation
Create a config for dev container to make development easier (#6230)	✅	The introduction of .devcontainer configurations and scripts simplifies the development setup, aligning with the objective to enhance accessibility for contributors.

Poem

Within Docker's realm, CVAT takes flight,
Codespaces await, shining bright.
🐰🎉 Scripts and files, a developer's delight,
CVAT's journey, a tale of might.
Setup hurdles fade, coding with ease,
Progress celebrated under digital trees!

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 17

Out of diff range and nitpick comments (5)

.devcontainer/docker-compose.yml (3)

8-12: Clarify the purpose of the pull argument in the build context.

The pull argument is used in the build context, but its purpose is not immediately clear. Consider adding a comment to explain why it is necessary to fetch upstream base image changes on container rebuild.

---

24-33: Ensure volume mappings do not cause performance issues.

The use of named volumes for static files and node_modules is intended to improve write performance on non-Linux machines. Monitor the performance to ensure that these settings do not inadvertently cause slowdowns or other issues in different environments.

---

85-85: Document the use of git branch values in volume names.

The use of git branch values to parameterize volume names is a clever way to isolate the development environment. Consider adding documentation or comments in the file to explain this setup to new developers.

.devcontainer/init.sh (1)

74-86: Document the manual merging of Docker Compose files.

The script manually merges Docker Compose files due to limitations in VS Code Remote. Consider documenting this workaround both in the script and in the project documentation to inform other developers.

site/content/en/docs/contributing/running-tests.md (1)

147-151: Update the documentation to reflect the non-requirement of certain steps in devcontainer.

The documentation mentions that building the CVAT server image and running the cvat_opa container are not required for devcontainer. Consider highlighting or separating these instructions more clearly to avoid confusion among developers using devcontainers.

[coderabbitai]

Actionable comments posted: 12

Out of diff range and nitpick comments (3)

.devcontainer/init.sh (1)

39-41: The command substitution in docker inspect should handle potential errors more gracefully to avoid script failure.

Consider checking the exit status of the docker inspect command explicitly and handle the error scenario appropriately.

site/content/en/docs/contributing/running-tests.md (1)

56-57: Clarify the activation instructions for the test-venv virtual environment to ensure users can easily follow the steps.

Consider adding a more explicit command or script snippet that shows how to activate the environment before executing the code.

site/content/en/docs/contributing/development-environment.md (1)

Line range hint 206-206: Change "need install" to "need to install" to correct the grammatical structure.

- you may need install `hdf5`
+ you may need to install `hdf5`

[coderabbitai]

Actionable comments posted: 2

Out of diff range and nitpick comments (1)

.devcontainer/postcreate.sh (1)

24-24: Consider the security implications of marking all directories as git safe globally.

Using git config --global --add safe.directory '*' marks all directories as safe, which can potentially expose the system to malicious repositories. It's safer to specify directories explicitly or use a more restrictive setting.

[coderabbitai]

Actionable comments posted: 9

[g-kartik]

Applied PR review suggestions from CodeRabbit Bot

[coderabbitai]

Actionable comments posted: 1

[coderabbitai]

Actionable comments posted: 1

[coderabbitai]

Actionable comments posted: 1

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ef27186 and b468d3d.

Files selected for processing (1)

• site/content/en/docs/contributing/running-tests.md (5 hunks)

Additional Context Used

LanguageTool (4)

site/content/en/docs/contributing/running-tests.md (4)

Near line 50: ‘necessary requirements’ might be wordy. Consider a shorter alternative.
Context: ...d cvat-cli source code 1. Install all necessary requirements before running REST API tests: ``` ...

---

Near line 107: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ST API tests: Attach to RQ lowfor the low priority queue worker -REST API tests: Attach ...

---

Near line 175: You have already used this phrasing in nearby sentences. Consider replacing it to add variety to your writing.
Context: ...sts` when using dev container 1. If you want to debug particular tests then change the ...

---

Near line 175: Possible missing comma found.
Context: ...iner 1. If you want to debug particular tests then change the configuration of the co...

Additional comments not posted (4)

site/content/en/docs/contributing/running-tests.md (4)

59-59: This clarification is helpful for users in a devcontainer environment.

---

147-147: Correctly noted as unnecessary in the devcontainer setup.

---

151-151: Correctly noted as unnecessary in the devcontainer setup.

---

174-174: Enhances the debugging experience by providing specific instructions for devcontainer environments.