Migration from LoopBack 3.x

[bajtos]

When LoopBack 4 gets more mature and reaches feature parity with LoopBack 3, we should write a migration guide and automated tooling to help existing LoopBack 3 users to upgrade their projects to LoopBack 4.

Spike tasks:

• SPIKE - Migrate LB3 to LB4 is possible SPIKE - Migrate LB3 to LB4 is possible #485
• Drop LB3 model files into LB4 app [Spike] Drop LB3 model files into LB4 app #2224
• RFC: use LB3 models in an LB4 app RFC: use LB3 models in an LB4 app [SPIKE - DO NOT MERGE] #2274
• Mount LB3 app and include its REST API in OpenAPI spec Spike: Mount LB3 app and include its REST API in OpenAPI spec #2301
• PoC showing how to mount an LB3 app and include its REST API in OpenAPI spec PoC showing how to mount an LB3 app and include its REST API in OpenAPI spec [DO NOT MERGE] #2318

[bajtos]

Cross posting my earlier thoughts - please note some parts are outdated and need tweaks to make them compatible with the latest LB4 design.

---

#485 (comment)

Here are my thoughts on the migration story.

At high-level, LoopBack 3.x applications consist of three big "parts"

• Persistence layer (this includes talking to backend services like SOAP/REST)
• Outwards facing REST API
• App-wide setup - Express middleware, boot scripts, etc.

In the persistence layer, users can contribute the following artifacts:

1. Definitions of Model data types (properties, validations)
2. Definition of data sources
3. Configuration of models (which datasource are they attached to)
4. Operation hooks

At the public API side, users can define:

1. Which built-in methods should be exposed (think of disableRemoteMethodByName)
2. Custom remote methods
3. before/after/afterError hooks at application-level
4. before/after/afterError hooks at model-level
5. before/after/afterError hooks at model method level

LoopBack Next was intentionally designed to allow users to choose their ORM/persistence solution, and our initial version of @loopback/repository is based on juggler 3.x. That makes it possible for users to reuse their existing model definitions, migrating their application incrementally.

The proposal

Here is my proposal:

• Keep the existing model definitions in .json and .js files, datasources configuration, etc. (But consider moving them around to get a saner file structure - this should be done by our migration tool we need to build.)
• Write a small amount of code to bind LB 3.x Models with LB Next Context, e.g. for each {Model} create models.{Model} binding.
• This allows us to reuse model definitions, datasources & connectors, operation hooks.
• The biggest difference is in the REST API. Fortunately, we have tools to inspect all remote methods and generate Swagger specification for them.
• For each public model, use introspection and swagger generator to code-generate a controller class implementing the same public REST API as the original LB 3.x model. This is a tool we need to build.
• Each controller method should be a simple wrapper calling the original model method.

A simplified example:

@api({basePath: '/products'})
export class ProductController {
  constructor(
    @inject('models.Product') private Product: Entity
  ){}

  @get('/')
  @param.query.object('where')
  // TODO describe response type
  find(where: Object): Product[] {
    return await this.Product.find(where);
  }

  @get('/custom')
  // ...
  customMethod(arg: number): Object {
    return await this.Product.customMethod(arg);
  }

  @patch('/{id}')
  // ...
  prototypeUpdateAttributes(id: number, data: Partial<Model>) {
    const instance = await this.Model.sharedCtor(id);
    return await instance.updateAttributes(data);
  }

  // ...
}

Remoting hooks

Let's talk about remoting hooks now and how to translate them to LB Next:

1. Application-level hooks should be invoked as part of the Sequence.
2. For model-level hooks, I am proposing the convention that a controller class can provide before, after and afterError methods. These methods can be either invoked explicitly from each method (which is brittle), or we need to improve our framework to invoke them automatically as part of invokeMethod.
3. For method-level hooks, the code should go directly into the controller method itself.

@api({basePath: '/products'})
export class ProductController {
  constructor(
    @inject('models.Product') private Product: Entity
  ){}

  before(route: Route, args: OperationArgs) {
    console.log('Invoking remote method %s with arguments %s', 
      route.describe(); JSON.stringify(args));
  }

  @get('/')
  @param.query.object('where')
  find(where: Object): Product[] {
    console.log('Here we can modify `where` filter like we were in a before-remote hook');

    return await this.Product.find(where);
  }

  // etc.
}

I don't think we can automate this migration step, because remoting hooks expect to receive a strong-remoting context that we don't have in LoopBack next. A good documentation with helpful examples is needed.

Alternatively, we can let the controller to provide invokeMethod where it's up to users to define whatever phases they need, so that they are not constrained by three phases before/after/afterError. (However, they can implement invokeMethod calling before/after/afterError if they like it.)

class MyController {
  invokeMethod(route: Route, args: OperationArgs) {
    console.log('Invoking remote method %s with arguments %s', 
      route.describe(); JSON.stringify(args));
    return await this[route.methodName](...args);
  }
}

// alternate invokeMethod implementation supporting before/after/afterError hooks
// this can be a Controller Mixin provided by an extension
  invokeMethod(route: Route, args: OperationArgs) {
    if (this.before)
      await this.before(route, args);

    try {
      const result = await this[route.methodName](...args);
      if (this.after)
        result = await this.after(route, args, result);
      return result;
    } catch(err) {
     if (this.afterError)
       const handled = await this.afterError(route, args, error);
       if (handled) return;
    }
    throw err;
  }

Summary:

• A tool for shuffling files around to get a sane structure for LB3 + LB4 files
• A tool for generating controllers from remoting metadata
• Runtime support for controller hooks (before, after, afterError)
• Instructions on how to migrate remoting hooks

Remaining things to figure out:

• how to migrate middleware into Sequence
• how to migrate boot scripts
• where to keep Swagger definitions for Models so that we can reference them in controller specs

Possibly out of scope:

• how to migrate ACLs that we don't support yet
• how to migrate built-in models like User, AccessToken, etc.
• change-tracking, replication and offline-sync
• push notifications
• storage component

Next level

• Migrate LB 3.x .json and .js files to use @loopback/repository. The generated controllers should use typed Repositories instead of juggler 3.x based Model classes.

Justification

You may ask why to code-generate all those controller files? In my experience, one of the most frequently sought feature of LB 3.x is the ability to control and customize the public API. Users start with a wide built-in CRUD API provided by the LoopBack, but soon enough then need to lock the API down, expose only subset of endpoints, and customize how exactly these endpoints work.

In LoopBack 3.x, this was a complex process that required hooks at many different levels (middleware/strong-remoting phases, remoting hooks, operation hooks) and unintuitive API like disableRemoteMethodByName.

In my proposal, customization of the REST API is as easy as editing the generated code.

• Need to hide a built-in (generated) endpoint? Just delete the controller method!
• Need to modify the request arguments before executing the underlying CRUD method? Add that code into the controller method.
• Need to add more parameters to a built-in (generated) endpoint? Just modify the controller method.
• Need to post-process the value returned from the model? Add the transformation after you awaited the result of the model method.

---

#485 (comment)

Remaining things to figure out:

• how to migrate middleware into Sequence
• where to keep Swagger definitions for Models so that we can reference them in controller specs

Possibly out of scope:

• how to migrate boot scripts
• how to migrate ACLs that we don't support yet
• how to migrate built-in models like User, AccessToken, etc.
• change-tracking, replication and offline-sync
• push notifications
• storage component

Next level

Migrate LB 3.x .json and .js files to use @loopback/repository. The generated controllers should use typed Repositories instead of juggler 3.x based Model classes.

[raymondfeng]

I have some initial ideas as follows:

1. Create the swagger.json from an existing LB3 app
2. Use lb4 openapi to generate controllers and models
3. Capture datasources from LB3 as json
4. Use lb4 datasource to recreate it from 3
5. Use lb4 repository/service to create repos/services from model-config.json
6. Handle other artifacts such as components, middleware, boot-scripts, hooks, ACLs, .... (TBD)

[marioestradarosa]

@bajtos , @raymondfeng for CRUD,

• I believe that we should be able to read hook definitions and generate the controller automatically
• The models can also be read and use either @model() decorator or generate the model class

In other words, we can run a command inside an lb3 app like so:

lb4 migrate --output ../mylb4app

The CLI could read the datasources, models, hooks for CRUD and generate the appropriate files.

thoughts?

[yagobski]

any progress about this task. How we can migrate large app from LB3?

[hacksparrow]

@yagobski you can track the progress being made on feature parity with LB 3 at #1920.

[bajtos]

For everybody watching this issue: please take a look at my proof-of-concept in #2274, it shows a compatibility layer allowing LB3 models to be added to LB4 applications and exposed via LB4 REST layer, with (hopefully) no changes in the LB3 model scripts needed

I would appreciate your feedback.

• Do you find such compatibility layer useful, would you use it in your project?
• The PoC version provides only subset of LB3 features and a lot is missing. Which missing features would stop you from using the proposed compatibility layer to migrate your LB3 project to LB4?

Please leave your comments in the pull request, not here.

[bajtos]

In #2318, I am showing a different approach where the entire LB3 application (including it's runtime dependencies in LTS mode) is mounted on an LB4 application and the Swagger/OpenAPI specifications are merged to produce a single unified spec describing both LB3 and LB4 endpoints.

Would you find this feature useful and start running your LB3 application mounted in an LB4 app?

[bajtos]

Created more stories:

• Import LB3 model JSON file to LB4 model class Import LB3 model JSON file to LB4 model class #2480
• From model definition to REST API with no repository/controller classes From model definition to REST API with no custom repository/controller classes #2036
• From datasource config to Service REST API with no proxy/controller classes Spike: From datasource config to Service REST API with no proxy/controller classes #2481
• Expose custom remote methods contributed by CRUD connectors on top of DAO methods Expose custom remote methods contributed by CRUD connectors on top of DAO methods #2482
• From Relation definition to REST API with no repository/controller classes [EPIC] From relation definition to REST API with auto-generated repository/controller classes #2483
• Spike: Discover and define models at runtime Spike: Discover and define models at runtime #2484
• Unregister runtime artifacts Unregister runtime artifacts #2485

[bajtos]

Current status:

It's possible to take an existing LB3 application and add it as-is to an LB4 project. See https://strongloop.com/strongblog/migrate-from-loopback-3-to-loopback-4/ and https://loopback.io/doc/en/lb4/Migrating-from-LoopBack-3.html.

While this does not remove dependencies on LB3 runtime, we hope it will make it easier for you to incrementally migrate your LB3 code to LB4.

[Sumit0581]

I have some ideas regarding Boot scripts in Loopback 4.
We can create a new file boot.ts in the /src folder which will export a Booter interface/class where users can add different boot scripts as functions.
Then this Booter interface/class can be called in index.ts and each function can be called separately.
I have attached a sample for same and it is working perfectly in my LB4 application.
PS: I have learned Loopback as technology in the last 4 -5 weeks so there are chances of mistakes.

[bajtos]

I have some ideas regarding Boot scripts in Loopback 4.

Thank you @Sumit0581 for sharing your solution with us.

Recently, @raymondfeng added support for Lifecycle observers (see docs), I believe they are intended to replace boot scripts. They are actually better than LB3 boot scripts, because they allow you to run some code not only at app startup, but also when the app is shutting down.

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.

[amitinvizio]

code: 'ER_NOT_SUPPORTED_AUTH_MODE',
errno: 1251,
sqlMessage: 'Client does not support authentication protocol requested by server; consider upgrading MySQL client',
sqlState: '08004',
fatal: true
}

i am using
node : 16.13.1
loopback4: 3.1.0
mysql: 8.0
"loopback-connector-mysql": "^5.4.4"
why i am not able to connect to database