Skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob

[kdelee]

SUMMARY

Alternative to #15171
@AlanCoding suggested most speed up could be gained in just not updating parent for waiting jobs

Put all 39,000 jobs in my system back in pending and triggered task manager. Had START_TASK_LIMIT set to 10,000

Result was ~ 133 jobs/second -- close to the 142 jobs/second we get from bulk_update approach with exact same workload in #15171 (compared to ~ 20 jobs/second in devel)

# HELP task_manager_get_tasks_seconds Time spent in loading tasks from db
# TYPE task_manager_get_tasks_seconds gauge
task_manager_get_tasks_seconds{node="awx-1"} 18.265242898953147
# HELP task_manager_start_task_seconds Time spent starting task
# TYPE task_manager_start_task_seconds gauge
task_manager_start_task_seconds{node="awx-1"} 54.556849607266486
# HELP task_manager_process_running_tasks_seconds Time spent processing running tasks
# TYPE task_manager_process_running_tasks_seconds gauge
task_manager_process_running_tasks_seconds{node="awx-1"} 1.7329584807157516e-06
# HELP task_manager_process_pending_tasks_seconds Time spent processing pending tasks
# TYPE task_manager_process_pending_tasks_seconds gauge
task_manager_process_pending_tasks_seconds{node="awx-1"} 57.23840588296298
# HELP task_manager__schedule_seconds Time spent in running the entire _schedule
# TYPE task_manager__schedule_seconds gauge
task_manager__schedule_seconds{node="awx-1"} 75.57144767208956
# HELP task_manager__schedule_calls Number of calls to _schedule, after lock is acquired
# TYPE task_manager__schedule_calls gauge
task_manager__schedule_calls{node="awx-1"} 4
# HELP task_manager_recorded_timestamp Unix timestamp when metrics were last recorded
# TYPE task_manager_recorded_timestamp gauge
task_manager_recorded_timestamp{node="awx-1"} 0.0
# HELP task_manager_tasks_started Number of tasks started
# TYPE task_manager_tasks_started gauge
task_manager_tasks_started{node="awx-1"} 10000
# HELP task_manager_running_processed Number of running tasks processed
# TYPE task_manager_running_processed gauge
task_manager_running_processed{node="awx-1"} 0
# HELP task_manager_pending_processed Number of pending tasks processed
# TYPE task_manager_pending_processed gauge
task_manager_pending_processed{node="awx-1"} 39197
# HELP task_manager_tasks_blocked Number of tasks blocked from running
# TYPE task_manager_tasks_blocked gauge
task_manager_tasks_blocked{node="awx-1"} 67
# HELP task_manager_commit_seconds Time spent in db transaction, including on_commit calls
# TYPE task_manager_commit_seconds gauge
task_manager_commit_seconds{node="awx-1"} 0.04217028617858887

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

ADDITIONAL INFORMATION

@AlanCoding has indicated he'd like to solve this in potentially more universal way leveraging django-ansible-base