Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix log formatting with salt logger #622

Open
Mryashbhardwaj opened this issue Sep 27, 2022 · 4 comments
Open

fix log formatting with salt logger #622

Mryashbhardwaj opened this issue Sep 27, 2022 · 4 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@Mryashbhardwaj
Copy link
Member

Describe the bug
current logger does not support argumentative logging as used on https://github.com/odpf/optimus/blob/main/job/deployer.go#L51
we need to format this log using fmt.Sprintf

To Reproduce
Steps to reproduce the behaviour:

  1. trigger deployment that is about to get failed
  2. check the log msg, it does not capture the error and deployment details

Expected behaviour
the logs should be formatted

@Mryashbhardwaj Mryashbhardwaj added bug Something isn't working good first issue Good for newcomers labels Sep 27, 2022
@camclark
Copy link

Hey team, interesting issue.
Just stumbled on your project; mind if I have this issue assigned to me?

@camclark
Copy link

Been playing around with the repo and I think I'm missing something to get local working.

tldr: Followed local guide, but unable to get ./optimus serve to work from default configs. Under the impression localhost should work near out of the box.

Followed steps as per: https://github.com/odpf/optimus#running-locally

$ ./optimus version
Client: v0.4.0-rc.6-next-17053a1

$ ./optimus serve  
WARN[0000] Shutting down server                         
INFO[0000] Server shutdown complete                     
Error: unable to create server: serve.ingress_host: required config missing
🔥 unable to complete request successfully

Since server isn't starting follow next steps within: https://odpf.github.io/optimus/docs/getting-started/configuration/

$ ./optimus init   

From here generated the optimus.yaml using values from this configuration page

version: 1
log:
  level: INFO
  format: ""
host: localhost:9100
project:
  name: local-project
  config: {}
namespaces:
- name: namespace1
  config: {}
  job:
    path: namespace1/jobs
  datastore:
  - type: bigquery
    path: namespace1/resources
    backup: {}

attempt deploy:

$ ./optimus deploy
Initializing client plugins
initialization finished!

Registering project [local-project] to [localhost:9100]
Error: Unable to reach optimus server at localhost:9100, this can happen due to following reasons:
1. Check if you are connected to internet
2. Is the host correctly configured in optimus config
3. Is Optimus server currently unreachable
🔥 unable to complete request successfully

See server issue and scroll down on that page to: server config

copy config.sample.yaml and uncomment as config.yaml

version: 1

# logging configuration
log:
  # debug, info, warning, error, fatal - default 'info'
  level: info

########################################
# SERVER CONFIG
########################################

# for configuring optimus service
serve:
 # port to listen on
 port: 9100

 # host to listen on
 host: localhost

 # this gets injected in compiled dags to reach back out to optimus service
 # when they run
 ingress_host: optimus.example.io:80

 # 32 char hash used for encrypting secrets
 # note: use a different one in production please
 app_key: Yjo4a0jn1NvYdq79SADC/KaVv9Wu0Ffc

 # database configurations
 db:
   # database connection string
   dsn: postgres://user:password@localhost:5432/database?sslmode=disable

   max_idle_connection: 5
   max_open_connection: 10

# optimus supports multiple scheduler types
scheduler:
 # name of the registered scheduler, default: airflow2
 name: airflow2


# application telemetry
telemetry:

 # configure the host:port used for
 # - exposing profiling metrics via pprof over /debug
 # - prometheus stats over /metrics
 profile_addr: ":9110"

 # jaeger collector address to send application traces
 jaeger_addr: "http://localhost:14268/api/traces"

# resource managers for job dependency enrichment
resource_managers:
# - name: other_optimus_server
 type: optimus # currently, only support optimus type
 description: neighbor optimus server
 config:
   host: # host of other optimus server
   headers: # might necessary for authorization

plugin:
  artifacts:
#     # refer : https://github.com/hashicorp/go-getter
    - ../transformers/dist/transformers_0.1.0_macos_arm64.tar.gz
    - https://github.com/odpf/optimus/releases/download/v0.2.5/optimus_0.2.5_linux_arm64.tar.gz

try serve again

$ ./optimus serve

2022/09/27 23:47:18 /Users/{myuser}/Documents/repos/open-source/optimus/internal/store/postgres/migration.go:165
[error] failed to initialize database, got error failed to connect to `host=localhost user=user database=database`: dial error (dial tcp [::1]:5432: connect: connection refused)
WARN[0000] Shutting down server                         
INFO[0000] Server shutdown complete                     
Error: unable to create server: error executing migration up: error initializing db client: error initializing db client: failed to connect to `host=localhost user=user database=database`: dial error (dial tcp [::1]:5432: connect: connection refused)
🔥 unable to complete request successfully

Please assist @Mryashbhardwaj

@irainia
Copy link
Member

irainia commented Oct 31, 2022

@Mryashbhardwaj, this issue will be addressed by this PR. to be more specific, under this commit.

@irainia
Copy link
Member

irainia commented Oct 31, 2022

@camclark , let me try to help you. in Optimus, we have two configurations (at the moment). one for client, which is named optimus.yaml and another is for server, named config.yaml. to run the server, make sure the config.yaml file is configured. I suggest you put in the same directory, otherwise you need to use flag -c (see --help flag).

in case of your error:

2022/09/27 23:47:18 /Users/{myuser}/Documents/repos/open-source/optimus/internal/store/postgres/migration.go:165
[error] failed to initialize database, got error failed to connect to `host=localhost user=user database=database`: dial error (dial tcp [::1]:5432: connect: connection refused)
WARN[0000] Shutting down server                         
INFO[0000] Server shutdown complete                     
Error: unable to create server: error executing migration up: error initializing db client: error initializing db client: failed to connect to `host=localhost user=user database=database`: dial error (dial tcp [::1]:5432: connect: connection refused)
🔥 unable to complete request successfully

try to check your database connection. it can be because you misconfigured it (like typos). or when reaching the database itself there is some problem, maybe connection issue or connection refused. if you don't have password to your database (such as for local), try to remove that, such as:

...
serve:
  db:
    dsn: postgres://user@localhost:5432/db_name?sslmode=disable
    max_idle_connection: 5
    max_open_connection: 10
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants