A framework to support digital engineering in the Python language
The Digital Architecture Framework (DAF) Metamodel is a modeling language which defines the notation for all elements used in the different stages of an organization life cycle to support the concept of Digital Engineering (DE), from the requirement to the application. DAF is intended to be used within a DE scope. The aim of the DE is to create a Digital Twin of an organization or domain, by holding all knowledge of the enterprise in an Authoritate Source of Truth (ASoT). The products of this knowledge (text documents, code configuration, deployment files, etc.) are generated from the model rather than produced, managed, and stored separately.
To be able to transform this abstract model into running code the support of a special class of frameworks is required, called Aphrodites (Alfa). DigitalPy is a Python framework that implements the Aphrodites 2.0 specifications. Other Aphrodites frameworks and the original specification are maintained under the Olympos MDA project (e.g. VenusSharp a C# implementation). The most notable being the WCMF, an Aphrodite framework for PHP.
An Aphrodites framework consists of frozen spots and hot spots. On one hand, frozen spots define the overall architecture of a software system, that is to say, its basic components and the relationships between them. These remain unchanged (frozen) in any instantiation of the application framework. On the other hand, hot spots represent those parts where the programmers using the framework add their own code to add the functionality specific to their own project.
The aim of a DigitalPy is to define all the frozen spots, so that is hot spot can be generated from the ASOT. While DigitalPy is well suited for Digital Enginering, there are no dependencies with the ASoT or the DAF Generator (DAFGen) that are maintained as a separated project.
This is not about “how easy the Hello World example is” rather “How long you need to understand enough to use it”. Design simplicity is a key principle and unnecessary complexity is avoided to the user. DigitalPy hides the complexity behind a clear defined, simple API.
A lesson learned from “Ruby on rails” in positive and the typical old J2EE application in negative sense. DigitalPy runs with minimal or no configuration. Even if a configurable entity was not configured, there should be some proper default to attach it with.
DigitalPy follows
- Abstraction: Alpha abstraction, supports the KISS principle, by hiding internal implementation and showing only the required features or set of services that are offered.
- Encapsulation: Alpha binds data and attributes or methods and data members in classes, also implements the Facade structural pattern (see below)
- Inheritance: Alpha allocates features to super classes that can be inherited from children classes
- Polymorphism: Alpha describe objects with polymorphic charatteristics.
provides a way to describes concepts like “Domain Object”, Services , View . See also the MVC point below.
is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other.
An application is usually based on a domain model that represents the real-world concepts of the domain of interest. In object oriented programming this model is implemented using classes. Depending on the application requirements the instances of some of these classes have to be persisted in a storage to keep the contained data. These classes represent the data model. The classes that provide the infrastructure for storing data form the persistence layer.
the dModelclasses support the ability to Create, Read, Update and Delete (CRUD) tree of elements taking in account the model information.
DigitalPy defines PersistentObject as base class for persistent domain classes. It mainly implements an unique identifier for each instance (see ObjectId), tracking of the persistent state, methods for setting and getting values as well as callback methods for lifecycle events. For the composition of object graphs the derived class Node is used as base class. It implements relation support for persistent objects.
To retrieve persisted objects PersistenceFacade is used. The actual operations for creating, reading, updating and deleting objects (e.g. SQL commands) are defined in classes implementing the PersistenceMapper interface (see Data Mapper Pattern). Although not necessary there usually exists one mapper class for each persistent domain class. Mapper classes are introduced to the persistent facade by configuration.
Associations are an important concept in a Chronos model, since they describe the relations between entities, namely the parent-child relation. There are three different types of Relationships supported: Composite, Shared and normal Associations. It is important to understand how DAFGen recognizes parents and children in an association with composite or shared ends the parent is always the class, that is connected with the diamond end of the association, even if that is placed on the target side of the association.
Normal associations are treated as parent-child relation, if they are only navigable in one direction, which is then defined as the child to parent direction.
Different association types shown below result in different behaviors regarding the operations allowed on child entities
| **Association type ** | **Delete children ** | **Create children ** | **Associate children ** |
|---|---|---|---|
| Composite association | Yes | Yes | No |
| Shared association | No | Yes | Yes |
| Association | No | No | Yes |
Another question that may arise is how to model multiple associations between two types. Given that you have Person and Task entity types and you want to associate tasks with persons. To achieve this, you'll draw a shared association between them and place the aggregation (diamond) end at the Person class.
But what if you want to have a task a task owner and a task creator? In this case you'll have to draw a second association.
The next question is now how the generator can distinguish between a task creators and task owners. This is, where the concept of roles comes into play. To resolve this conflict, you'll have to assign role names to the person and task association ends. The generator will generate different entity and mapper classes for each role, which allows the framework to know which role a person has in relation to a given task.
If different entity types should have the same attributes, one may define a common base class that makes these available to inheriting classes. An example may be a EntityBase type, which defines meta information like creation date and last modification date for all entity types
The DAFGen expects to have a target class/mapper definitions for each class in the model. In case of inheritance the subclasses will contain the attributes of the parent classes too. For abstract base classes no mapper class will be generated.
Inheritance will cause the generator to draw new relations in the following cases: • Subclasses of parent classes of the current entity will also become parents . • Parent classes of the superclass of the current entity will also become parents. • Subclasses of child classes of the current entity will also become children . • Child classes of super classes of the current entity will also become children .
Another aspect of associations is the use of arrow ends that describe the navigability. If an arrow end is omitted on one end, that means that the appropriate entity type is not seen from the other end.
This association information is written into the generated mapper classes.
Modular design is a generally recognized “Good Thing(tm)” in software engineering. As in science in general, breaking a problem down to smaller, bite-sized pieces makes it easier to solve. It also allows different people to solve different parts of the problem and still have it all work correctly in the end. Each component is then self-contained and, as long as the interface between different components remains constant, can be extended or even gutted and rewritten as needed without causing a chaotic mess. DigitalPy supports this principle with his component Architecture (see below).
Aphrodite 1.0 describes a monolithic architecture, to support a better modular design, the 2.0 specs introduces the concept of Component Architecture.
- Independent − Alfa Components are designed to have minimal dependencies on other components thanks to the DigitalPy routing system.
- Reusability − Alfa Components are designed to be reused in different situations in different applications. However, some components may be designed for a specific task.
- Replaceable − Alfa Components may be substituted with other similar components.
- Not context specific − Alfa Components are designed to operate in different environments and contexts and can be deployed anywhere.
- Encapsulated − A component depicts a Facade, which allow the caller to use its functionality, and do not expose details of the internal processes or any internal variables or state.
- Deployability - Alfa Components can de deployed at runtime, registering themselves to the application core.
each component exposes a Facade, that is inheriting from the framework Facade. All the messages are routed trough the facade to the Component Actionmapper that allocates the action to the proper controller as defined in internalActionMapping.ini.
this allows:
- Isolation: We can easily isolate our code from the complexity of a subsystem.
- Testing Process: Using Facade Method makes the process of testing comparatively easy since it has convenient methods for common testing tasks.
- Loose Coupling: Availability of loose coupling between the clients and the Subsystems.
each component exposes a Facade, that is inheriting from the framework Facade. All the messages are routed trough the facade to the Component Actionmapper that allocates the action to the proper controller as defined in internalActionMapping.ini.
Here is an example of how you could create a facade for a "emergency" component in Python :
class EmergencyFacade:
def received(self, request, response):
# Implement the logic for handling received emergency alerts here
pass
class Emergency:
@staticmethod
def Received(request, response):
emergency_facade = EmergencyFacade()
emergency_facade.received(request, response)
The Emergency class acts as a facade, providing a single point of entry for handling emergency alerts. The Received method delegates the actual handling of the alert to the EmergencyFacade class. This allows you to encapsulate the implementation details of the emergency alert handling logic within the EmergencyFacade class, while providing a simpler interface for interacting with it through the Emergency facade.
You can then use the Emergency class in your actionmapping configuration like this:
[actionmapping]
EmergencyAlert = FreeTAKServer.components.extended.emergency.Emergency.Received
EmergencyInContact = FreeTAKServer.components.extended.emergency.Emergency.Received
EmergencyRingTheBell = FreeTAKServer.components.extended.emergency.Emergency.Received
EmergencyGeoFenceBreached = FreeTAKServer.components.extended.emergency.Emergency.Received
This allows you to keep the implementation details of the emergency alert handling logic separate from the rest of the application, and makes it easier to maintain and modify the code in the future.
DigitalPy supports a base API which can register flask blueprints dynamically.
This enables component developpers to quickly add exposed functionality without the need to develop a new service from scratch. These blueprint's must be put in the path DPApplicationRoot/Blueprints/your_blueprint.py. From there, they will be dynamically registered.
Routing is the process of selecting the correct Controller for a given Request. DigitalPy distinguishes between internal and external routing. DP inspects the action key formed from it's sender, context and action name and parameters (if any).
- Sender: the original emitter of the request (e.g. a component in external routing)
- Context: in certain circostances the same Sender can reuse the same action name
- Action: the name of the Action that is called
Internal routing is executed within the scope of a Component
External routing can be executed Between:
- a Service and a Component
- Two Components
- a Core function and a component
example of extenal routing
[actionmapping]
Health?api?CreateHealthPOST = components.health.health_facade.Health.create_health_post
Caution
Action Keys must not use the ~ character as this will break routing
DigitalPy core contains the following component:
| Component Name | Description |
|---|---|
| Integration Manager | part of the Z-managerThe Integration manager receives all reponses from all workers, prints them, and sends a message to the workers to shut down when all tasks are complete.Uses a ZMQ_PULL socket to receive answers from the workers.Uses a ZMQ_PUB socket to send the FINISH message to the workers. |
| RoutingWorker | part of the Z-managerThe Integration manager receives all reponses from all workers, prints them, and sends a message to the workers to shut down when all tasks are complete.Uses a ZMQ_PULL socket to receive answers from the workers.Uses a ZMQ_PUB socket to send the FINISH message to the workers. |
| Subject | part of the Z-managerDispatches events to listeners and sends messages with payloads from services, acting like a load balancer.Uses a ZMQ_PUSH socket to send messages to workers. |
| serializer KML | |
| SerializerJSON | |
| serializerProtobuff | |
| SerializerXML | |
| Registration | part of component management. Manages component registration. |
| DigPy Core | Core capabilities of DigitalPy: there are divided in two type:Minimal: the capabilities directly instantiated by Main that are strictly necessary for DigitalPy to start. Include basic internal (zmanager) and external (service) routing processes.Extended: additional capabilities that allows full functionality of the framework |
| Component Management | This core package contains functions related to Management of DigitalPy Components, this includes Discovery, Registration, installation and de-installation.Discovery exposes an end point in the rest API that goes trough all the folder in the component and search for non installed ones.Also describes the interfaces required to be implemented by components. |
| DigiPy Configuration | Provides centralized access to configuration of all components and central core independent from their format (e.g. DB, YAML, INI, .Py) |
| Files | The file package manages the access to the file system including reading and saving different files format such as ini, JSON and YAML |
| Health | The Health Core package in DigitalPy is designed to aggregate and provide system information for monitoring purposes. This package works in tandem with the Telemetry component, offering a comprehensive view of the system's health and performance.Data Collection: Gathers critical system information, including hardware status, system performance metrics, application health indicators, and more.Integration with Telemetry: Seamlessly integrates with the Telemetry component to include telemetry data in the health monitoring information.Health Monitoring Interface:Monitoring Services Integration: Provides a unified interface to deliver system health data to various monitoring services and tools.Real-Time Data Access: Facilitates real-time access to health metrics, enabling timely detection and response to potential issues.System Health Reporting:Comprehensive Reports: Generates detailed health reports, including both current status and historical data analysis.Alerts and Notifications: Implements mechanisms for sending alerts and notifications based on predefined thresholds or detected anomalies.Performance Metrics:Resource Utilization: Monitors key performance indicators such as CPU usage, memory consumption, network bandwidth, and disk utilization.Application Health: Tracks application-specific metrics like response times, error rates, and throughput. |
| IAM | The IAM component in DigitalPy is designed to support standard authentication and authorization protocols, catering to both internal and external applications, API developers, and end-users. It ensures secure access control and identity management within the framework.Authentication Support:Protocols: Supports a variety of standard authentication protocols, including LDAP (Lightweight Directory Access Protocol), SAML (Security Assertion Markup Language), OAuth, and WS-Federation.Process: Handles the verification of user identities through secure methods, ensuring that each user is who they claim to be.Authorization Mechanism:Access Control: Determines user permissions and access levels, ensuring users are only able to perform actions they are authorized for.Role-Based Access Control (RBAC): Implements RBAC to manage user roles and permissions, enabling fine-grained access control. |
| Installation | Provide ability to create the physical machine, install, upgrade, destroy (kill switch) and de-install current the system |
| Logic | The Logic core component in DigitalPy is designed to abstractly define and implement business logic within the framework. It organizes business logic into two distinct levels: flow logic and atomic logic. Flow Logic:Definition: Represents the sequence of actions or methods, along with decision points interspersed within these sequences.Implementation: Involves constructing a series of interconnected steps where each step might involve invoking an action (method execution) and making decisions based on certain conditions or data states.Control Structures: Utilizes various control structures (like conditional statements, loops, etc.) to guide the flow of execution based on business rules and conditions.Atomic Logic:Definition: Refers to the granular level of business logic that governs micro-decisions within the broader process flow.Implementation: Consists of individual, discrete rules or conditions that determine specific outcomes or actions in a given context.Rule-Based Approach: Often implemented using a rule-based system where individual rules are defined and applied to evaluate and affect specific parts of the process.The Logic Core can be directly imported by single components to make use of his features |
| Main | the main package is the part of DigitalPy Core that contains the crucial classes of the framework necessary to start a bare bone application. It directly import the related components. |
| Network | The Network package in DigitalPy is designed to provide fundamental network connectivity functionalities. It features an abstract interface that outlines the requirements for basic network connectivity solutions, such as web servers, FTP servers, and more. This package covers all aspects of network layers 1-7, ensuring comprehensive network communication capabilities. Layer 1 (Physical): Manages physical connectivity including cables, switches, and wireless signals.Layer 2 (Data Link): Handles protocols for node-to-node data transfer and error correction.Layer 3 (Network): Manages addressing and routing through IP protocols.Layer 4 (Transport): Ensures reliable data transfer, managing protocols like TCP and UDP.Layer 5 (Session): Controls dialogues (sessions) between computers, managing connections and terminations.Layer 6 (Presentation): Translates data formats and encrypts/decrypts data for application layer.Layer 7 (Application): Facilitates network services for end-user processes, supporting protocols like HTTP, FTP, and SMTP.Ensures a consistent approach to implementing different types of network communication protocols and services. Network is used by the Service Management to instantiate new services |
| Persistence | The Persistence package in DigitalPy is a core component designed to handle data persistence within the framework. It includes abstract classes and a base implementation using SQLAlchemy, with the flexibility to integrate additional implementations, (e.g. NoSQL database).Define the structure and contracts for data persistence operations, ensuring a consistent approach regardless of the underlying database technology.Components: Include abstract classes for basic CRUD (Create, Read, Update, Delete) operations, connection management, and transaction handling.SQLAlchemy Base Implementation: Utilizes SQLAlchemy, a popular ORM (Object-Relational Mapping) library in Python, for the base implementation.Functionality: Facilitates interaction with SQL databases, abstracting and automating database operations through an object-oriented approach.Flexibility: Supports various SQL databases, providing a versatile and robust solution for relational data persistence.Extensibility for Additional Implementations: Enables the addition of custom persistence solutions, aligning with specific project requirements or emerging database technologies. |
| Queries | Provides abstract access to query the persistency layer offered by the persistency component. Implementation of the query language can be specific to an application or a domain (e.g. GeoQuery) |
| Security | The Security component in DigitalPy is dedicated to the comprehensive management of security credentials, including certifications, tokens, passwords, and other sensitive authentication data. It handles all aspects of credential lifecycle, from creation and storage to disposal, ensuring high standards of security and data protection.Credential Management:Creation: Implements secure methods for generating and issuing various types of credentials, such as certificates, tokens, and passwords.Storage: Ensures secure storage of credentials, employing encryption and other security measures to protect sensitive data from unauthorized access.Disposal: Safely disposes of credentials that are no longer needed, following best practices to prevent data leakage or unauthorized retrieval.Certification Management:Certificate Authority Integration: Facilitates integration with Certificate Authorities (CAs) for issuing and managing digital certificates.Certificate Lifecycle Management: Manages the entire lifecycle of digital certificates, including issuance, renewal, revocation, and expiration.Token Management:Token Generation and Validation: Handles the generation and validation of tokens, such as JWT (JSON Web Tokens), for secure authentication and authorization processes.Token Expiry and Renewal: Manages token expiration and renewal, ensuring tokens are valid only for their intended lifespan.Password Management:Secure Password Handling: Implements secure practices for password creation, storage, and access, including hashing and salting.Password Policy Enforcement: Enforces strong password policies to enhance security, such as minimum complexity requirements and regular password changes. |
| Serialization | This core package includes the abstract implementation of ooriticilk and his serialization from and to:JSONXMLProtobuff....Analogous to serialization libraries in frameworks like .NET's JSON.NET, Google's Protocol Buffers, or Java's JAXB. |
| Service Management | The Service Management component in DigitalPy is a core function designed to manage the lifecycle and operations of various services within the framework. It provides abstract capabilities for installing, deinstalling, discovering, starting, and stopping services, aligning with the principles set in the Network component for network type and communication protocols.Lifecycle Management:Installation and Deinstallation: Allows for the installation and removal of services within the DigitalPy environment.Service Discovery: Facilitates the discovery of available services, aiding in dynamic service management and integration.Start/Stop Mechanisms: Provides the ability to start and stop services dynamically, ensuring flexibility and responsiveness in resource management.Service Isolation and Association:Ensures each service runs in a thread and is isolated from others, Associates each service with a specific port and network type, as defined in the Network component.Supports various data formats and protocols such as XML, JSON, Protobuf, etc.Integration with ZManager:Implements a Subscriber pattern of the ZManager, allowing services to subscribe to specific topics published by the Integration Manager.Message Handling:Implements a push pattern to forward received messages to the Subject (Ventilator).Interoperability and Standardization: Ensures compatibility with different network types and communication protocols, as defined in the Network component.Aligns with standardized practices for service management, ensuring a consistent and efficient operational environment. |
| Telemetry | The Telemetry component in DigitalPy is responsible for efficient collection, aggregation, processing, and export of telemetry data (metrics, logs, and traces) across the system, ensuring compatibility with a variety of monitoring and analysis tools.Data Collection: Collect diverse telemetry data types, including system metrics, logs, and traces, ensuring comprehensive coverage of system performance and behavior.Aggregation: Aggregate telemetry data from multiple sources within the system, summarizing and synthesizing data to provide an integrated view of system health and performance.Data Processing: Implement data processing techniques, including normalization, enrichment, and filtering, to enhance the relevance and context of the telemetry data.Production: Format and prepare telemetry data for consumption, ensuring compatibility with standard monitoring and analysis tools.Export and Integration: Facilitate the export of telemetry data in formats compatible with a variety of backend systems, emphasizing interoperability and ease of integration. |
| Translation | translation of messages to different languages.Internationalization of an application requires to identify all language dependent resources and make them exchangeable for the actual localization into a specific language. This includes static and dynamic texts as well as images. Since images are referenced by their filename or represented as text (e.g. base64 encoded), it is sufficient to focus on text. |
| Validation | The validation component of DigitalPy is responsible for ensuring the accuracy, completeness, and consistency of data within the application. It receives data from other components and validates it against predefined rules and criteria. The component ensures that data meets specified standards and formats, and that any required data is present and correct. The validation component also performs data type validation and cross-field validation to ensure that data is logically consistent. If any issues are found, the validation component generates error messages and notifies the appropriate component or user. The validation component plays a critical role in maintaining the integrity of the data and ensuring the reliability of the application. Implements robust input validation and filtering to prevent common web vulnerabilities such as SQL injection, cross-site scripting (XSS), etc. |
| ZManager | The ZManager component in DigitalPy is a core implementation of a decoupled communication strategy using ZeroMQ (ØMQ). It facilitates efficient, scalable, and distributed communication across the DigitalPy framework, forming part of the minimal set of services managed by DigitalPy Main.Decoupled Event-Based Architecture: Utilizes the observable pattern to enable a decoupled, event-based architecture, supporting multicore and multithreading capabilities.Distributed Communication: Can be distributed across multiple nodes, leveraging ØMQ's ability to function over various transports like in-process, inter-process, TCP, and multicast.Message-Oriented Sockets: Implements ØMQ sockets for N-to-N communication with patterns like publish-subscribe (pub-sub) and request-reply, carrying whole messages across different channels.Asynchronous I/O Model: Employs ØMQ's asynchronous I/O model to facilitate scalable, multicore application development through asynchronous message-processing tasks. |
| Util | the util package contains all the non-specific business or technical capabilities. |