Troubles with pydantic models and VSCode and pyright in neovim

[strayer]

Hey,

I'm currently testing Piccolo in a side-project and ran into issues when trying to use my IDE-like neovim features with the pyright language server in combination with the tables.

My project is split into two parts, a backend with a database and a CLI that runs on other systems interacting with the backend through REST APIs. Ideally I'd like to have the tables and models in shared code so I can just create pydantic models in the CLI, serialize them and pass them on to the backend which uses the same tables and models.

Right now pyright doesn't seem to understand the models created with piccolo.utils.pydantic.create_pydantic_model:

It seems to think that DishModel is of type BaseModel:

I tried the same with the class DishModel(create_pydantic_model(...)): pass. In that case the type is detected as DishModel, but still doesn't now anything about the fields and especially complains about the constructor:

The code itself is working fine, required fields are checked and cause an exception. But I really like to lean on correct completions and linting. The same is happening in Visual Studio Code with the Python extension.

Any ideas what I might be doing wrong? Based on other discussions here in the issues I assume this should work…

[dantownsend]

Yeah, there are limitations when we create Pydantic models dynamically.

In my own projects, I tend to use a mix of create_pydantic_model and manually created Pydantic models, depending on the use case.

I've explored added better typing support for create_pydantic_model, but haven't found a perfect solution.

One thing we could do instead is have a command - something like piccolo models generate which just outputs a models.py file containing the Pydantic models.

[strayer]

Ahh, I didn't really think about that. My tables will be rather simple, so I guess I'll just implement the pydantic models for the CLI myself. Maybe I can get some tests going that at least validate that the generated models equal the manually implemented ones.

[strayer]

I'll create them manually for now. If I ever get into testing those I'll update this discussion.

import decimal
from piccolo.columns import Decimal, Integer, Varchar
from piccolo.table import Table
from pydantic import BaseModel


class Dish(Table):
    remote_id = Integer(required=True, unique=True)
    name = Varchar(required=True)
    original_name = Varchar(required=True)
    price = Decimal(required=True)

class DishModel(BaseModel):
    remote_id: int
    name: str
    original_name: str
    price: decimal.Decimal

[dantownsend]

They look right to me.

[strayer]

I might have gone a bit overboard, but I'm now generating the pydantic models instead of writing them by hand and implemented a simple test that keeps me from forgetting to keep them up-to-date:

import os
import json
from pathlib import Path

import typer
from piccolo.utils.pydantic import create_pydantic_model

from ..shared.tables import Dish

cli_app = typer.Typer()


@cli_app.command()
def generate_pydantic_models():
    from datamodel_code_generator import DataModelType, InputFileType, generate

    output = Path(os.path.dirname(__file__)) / ".." / "shared" / "models.py"

    json_schema = json.dumps(
        create_pydantic_model(Dish, model_name="DishModel").model_json_schema()
    )

    generate(
        json_schema,
        input_file_type=InputFileType.JsonSchema,
        input_filename="example.json",
        output=output,
        # set up the output model types
        output_model_type=DataModelType.PydanticV2BaseModel,
    )

from typing import Any
from piccolo.utils.pydantic import create_pydantic_model

from ...shared.tables import Dish
from ...shared.models import DishModel


def remove_keys_named_extra(schema: dict[str, Any]) -> dict[str, Any]:
    filtered_schema = {key: value for key, value in schema.items() if key != "extra"}
    filtered_schema["properties"] = {
        property_key: {
            key: value for key, value in property_value.items() if key != "extra"
        }
        for property_key, property_value in schema["properties"].items()
    }

    return filtered_schema


class TestTableModels:
    def test_dish(self):
        handmade_model = DishModel.model_json_schema()
        generated_model = create_pydantic_model(
            Dish, model_name="DishModel"
        ).model_json_schema()

        assert handmade_model == remove_keys_named_extra(generated_model)

Needs some more work done to support multiple tables instead of just one, but step by step.

[dantownsend]

That's cool - nice work!