-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PostgreSQL] Add logs #2529
base: main
Are you sure you want to change the base?
[PostgreSQL] Add logs #2529
Conversation
connectors/sources/postgresql.py
Outdated
msg = ( | ||
"Fetching all tables as the configuration field 'tables' is set to '*'" | ||
if not is_filtering | ||
else "Fetching all tables as the advanced sync rules are enabled." | ||
) | ||
self._logger.info(msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
msg = ( | |
"Fetching all tables as the configuration field 'tables' is set to '*'" | |
if not is_filtering | |
else "Fetching all tables as the advanced sync rules are enabled." | |
) | |
self._logger.info(msg) | |
msg = "Fetching all tables" | |
self._logger.info(msg) |
return primary_keys | ||
|
||
async def get_table_last_update_time(self, table): | ||
self._logger.debug(f"Fetching last updated time for table: {table}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's log the time we fetched
connectors/sources/postgresql.py
Outdated
@@ -266,9 +274,12 @@ async def get_table_primary_key(self, table): | |||
) | |||
] | |||
|
|||
self._logger.debug(f"Found primary keys for '{table}' table") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's log what the primary keys were
connectors/sources/postgresql.py
Outdated
@@ -300,6 +311,7 @@ async def data_streamer( | |||
list: It will first yield the column names, then data in each row | |||
""" | |||
if query is None and row_count is not None and order_by_columns is not None: | |||
self._logger.debug(f"Streaming records from database using query: {query}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems wrong. query is None
was just asserted above this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the logs are swapped for if-else conditions.
connectors/sources/postgresql.py
Outdated
@@ -328,6 +340,7 @@ async def data_streamer( | |||
if row_count <= offset: | |||
return | |||
else: | |||
self._logger.debug(f"Streaming records from database for table: {table}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the sample log file, table is None
. That seems like an error.
[FMWK][18:58:23][DEBUG] [Connector id: 3S2VrY4BAqKLTyj6P7_3, index name: postgresql, Sync job id: KpLkvY4BBpk9ZgBA2rJn] Streaming records from database for table: None
Copying from another PR - that's applicable to all SQL connectors: The most useful connector logs are - which queries were actually executed by the connectors. Can you add debug logs that state which exact queries were executed? A good addition to it would be to also debug log the number of records returned by the query. |
@artem-shelkovnikov This is already present here connectors/connectors/sources/postgresql.py Line 348 in 43ee46c
Also, i've added a log for the number of records fetched from table/query. |
Part Of #2299
Adding more logs in PostgreSQL connector.
Log file: https://drive.google.com/file/d/12Pkq94J3ymnVGhXBtNc8S61xLqzDFCSK/view?usp=drive_link
Checklists
Pre-Review Checklist
config.yml.example
)v7.13.2
,v7.14.0
,v8.0.0
)