Commit Graph

147 Commits

Author SHA1 Message Date
beeankha
ef6297377b Enable Ansible version to be collected from EEs 2021-03-18 12:55:48 -04:00
beeankha
60827143bb Bump up unified_jobs_table version for new column addition 2021-03-03 18:52:55 -05:00
beeankha
0e80f663ab Add installed_collections column to unified job query 2021-03-03 18:52:55 -05:00
softwarefactory-project-zuul[bot]
3aba1e9db5 Merge pull request #8614 from jctanner/COLLECTION_PLAYBOOK_STATS
analytics: include modified column and full event_data for playbook_on_stats

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-12-08 13:08:41 +00:00
softwarefactory-project-zuul[bot]
d93d0f00ee Merge pull request #8721 from rooftopcellist/remove_available_instance
Remove redundant available_instances field

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-12-04 15:45:29 +00:00
Alan Rominger
44e6e9344b Remove mercurial support 2020-12-01 14:06:34 -05:00
Christian M. Adams
8e66172ed4 Remove redundant available_instances field 2020-12-01 11:20:21 -05:00
James Tanner
3e7554974a set a column name so it doesnt end up as "case" 2020-11-18 12:12:29 -05:00
James Tanner
153a1ecd39 bump minor 2020-11-18 11:30:48 -05:00
James Tanner
d6f9c5a0b6 get the modified column too 2020-11-18 11:28:22 -05:00
James Tanner
9cf294f3d7 implement ryanp suggestion 2020-11-18 09:57:22 -05:00
James Tanner
a1c2d458de get a csv of playbook_on_stats event data 2020-11-17 17:48:13 -05:00
Bill Nottingham
7b04fa114e Revert portions of 40a10dcc5f
Keep some of the collectors, while removing ones we calculate
in other ways.
2020-11-13 14:50:04 -05:00
Ryan Petrello
40a10dcc5f removed unused analytics metrics 2020-11-11 09:56:20 -05:00
Ryan Petrello
4445d096f5 reimplement licensing to work with RHSM and entitlement uploads
Co-authored-by: Christian Adams <chadams@redhat.com>
2020-10-30 16:39:57 -04:00
Bill Nottingham
18c27437b7 Fix analytics logging
The analytics change PR adjusted the logging for awx.analytics,
which solved the issue, but should have used the targeted awx.main.analytics.

Also flip a couple of loggers to use the regular awx.analytics (awx analytics)
logger instead of awx.main.analytics (the automation anayltics task system).
2020-09-17 13:39:14 -04:00
Bill Nottingham
13802fcf2b Don't return error messages for license errors
Just log the exception and return None.
2020-09-10 21:19:07 -04:00
Bill Nottingham
a604ecffb8 Adjust query_info to set the collection time based on what's passed. 2020-09-09 20:05:48 -04:00
Bill Nottingham
09f7d70428 Use isoformat() rather than strftime
Reformat SQL in unit tests because sqlite.
2020-09-09 20:05:32 -04:00
Bill Nottingham
c753324872 Move back to less frequent collections, and split large event tables
This should ensure we stay under 100MB at all times.
2020-09-09 17:42:40 -04:00
Bill Nottingham
9f67b6742c Fail more gracefully if analytics.ship() is called with a bad path,
or it's deleted out from under us.
2020-09-09 17:42:40 -04:00
Bill Nottingham
1a15f18be3 Stop using the TowerAnalyticsState solo model
This is now tracked in the AUTOMATION_ANALYTICS_LAST_GATHER setting.
2020-09-09 17:42:40 -04:00
Bill Nottingham
40309e6f70 Ensure we do not send large bundles, or empty bundles
Collect expensive collectors separately, and in a loop
where we make smaller intermediate dumps.

Don't return a table dump if there are no records, and
don't put that CSV in the manifest.

Fix up unit tests.
2020-09-09 17:42:40 -04:00
Bill Nottingham
1c4b06fe1e Refactor analytics collectors.
- Only have one registration class
- Add description fields
- Add automation collector information to /api/v2/config
2020-09-09 17:10:14 -04:00
Christian M. Adams
0681444294 Provide a distinct column name for inventory_name in unifiedjob analytics csv 2020-09-08 11:14:24 -04:00
softwarefactory-project-zuul[bot]
1c55d10d81 Merge pull request #7965 from rebeccahhh/devel
adding in inventory to unified job query for analytics

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-09-01 18:27:27 +00:00
Christian M. Adams
4ea648307e Accept all responses <300 from Insights API 2020-08-28 12:45:50 -04:00
Rebeccah
9a6f641df0 adding in inventory to unified job query for analytics 2020-08-26 16:32:02 -04:00
chris meyers
ac46013fcc fix sliding window per/minute calc
* Add tests to ensure correctness
2020-04-24 15:55:11 -04:00
Bill Nottingham
5b0bb4939f Allow subsets of table gathering for unit tests.
sqlite does not like some of our PG-isms.
2020-04-21 20:21:20 +02:00
Ladislav Smola
144cffe009 Send job and template nodes to analytics
Sending tables main_workflowjobnode and main_workflowjobtemplatenode
containing arrays of success/failure/always_nodes which is compatible
to what API call for nodes return.
2020-04-21 20:02:30 +02:00
Christian Adams
a899a147e1 Fix new flake8 from pyflakes 2.2.0 release 2020-04-20 09:50:50 -04:00
Bill Nottingham
f99a43ffa6 Collect information on inventory sources
Also remove one minor query from smart inventory collection that will
never return anything.
2020-04-14 19:15:19 -04:00
Bill Nottingham
cd6d2ed53a Move the comma so unit test can filter things properly. 2020-04-14 13:12:03 -04:00
Bill Nottingham
a296f64696 Cast the start/end times with timezone. 2020-04-14 10:53:57 -04:00
softwarefactory-project-zuul[bot]
c414fd68a0 Merge pull request #6176 from Ladas/send_also_workflows_as_part_of_unified_jobs
Send also workflows as part of unified jobs and send all changes to jobs

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:41:36 +00:00
Bill Nottingham
d651786206 Collect task timing, warnings, and deprecations from job events
Timing information requires ansible-runner >= 1.4.6.
2020-04-09 17:27:19 -04:00
Ladislav Smola
6a86af5b43 Use indexed timestamps
Use created and finished, which are indexed, to try to fetch all
states of jobs. If job is not finished, we might not get the
right terminal status, but that should be ok for now.
2020-04-09 15:20:27 +02:00
Ladislav Smola
6a503e152a Send also workflows as part of unified jobs
Workflows do not have a record in main_job, therefore the JOIN
was ignoring those. We need to do LEFT JOIN to include also
workflows.

It also seems like we are not able to get a link to organizations
from workflows? When looking at:
<tower_url>#/organizations?organization_search=page_size:20;order_by:name

We don't seem to list a relation to workflows. Is it possible to get it from
somewhere?
2020-04-09 15:20:27 +02:00
Ladislav Smola
b7227113be Use modified to check if job should be sent to analytics
It can take several hours for a job to go from pending to
successful/failed state and we need to also send the job with
a changed state, otherwise the analytics will be incorrect.
2020-04-09 15:20:27 +02:00
chris meyers
c00f1505d7 add broadcast websocket status command 2020-04-02 21:46:12 -04:00
Bill Nottingham
96576b0e3d Don't return different fields for smart vs non-smart inventories 2020-03-30 17:15:55 -04:00
chris meyers
d58df0f34a fix sliding window calculation 2020-03-18 16:10:19 -04:00
chris meyers
9e5fe7f5c6 translate Instance hostname to safe analytics name
* More robust translation of Instance hostname to analytics safe name by
replacing all non-alpha numeric characters with _
2020-03-18 16:10:19 -04:00
chris meyers
093d204d19 fix flake8 2020-03-18 16:10:19 -04:00
chris meyers
d6594ab602 add broadcast websocket metrics
* Gather brroadcast websocket metrics and push them into redis every
configurable seconds.
* Pop metrics from redis in web view layer to display via the api on
demand
2020-03-18 16:10:18 -04:00
softwarefactory-project-zuul[bot]
e210ee4077 Merge pull request #6301 from gamuniz/catch_analytics_failure
rework the gather() to always delete the leftover directories

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-16 20:45:36 +00:00
Gabe Muniz
80234c5600 rework the tar to always delete the leftover directories 2020-03-16 19:54:15 +00:00
AlanCoding
7d0b207571 Organization on JT as read-only field
Set JT.organization with value from its project

Remove validation requiring JT.organization

Undo some of the additional org definitions in tests

Revert some tests no longer needed for feature

exclude workflow approvals from unified organization field

revert awxkit changes for providing organization

Roll back additional JT creation permission requirement

Fix up more issues by persisting organization field when project is removed

Restrict project org editing, logging, and testing

Grant removed inventory org admin permissions in migration

Add special validate_unique for job templates
  this deals with enforcing name-organization uniqueness

Add back in special message where config is unknown
  when receiving 403 on job relaunch

Fix logical and performance bugs with data migration

within JT.inventory.organization make-permission-explicit migration

remove nested loops so we do .iterator() on JT queryset

in reverse migration, carefully remove execute role on JT
  held by org admins of inventory organization,
  as well as the execute_role holders

Use current state of Role model in logic, with 1 notable exception
  that is used to filter on ancestors
  the ancestor and descentent relationship in the migration model
    is not reliable
  output of this is saved as an integer list to avoid future
    compatibility errors

make the parents rebuilding logic skip over irrelevant models
  this is the largest performance gain for small resource numbers
2020-03-12 15:45:46 -04:00
Ben Thomasson
fc402aff29 Bumps the version of config.json to 1.1 2020-02-28 15:28:31 -05:00