this commit implements the bulk of `awx-manage run_dispatcher`, a new
command that binds to RabbitMQ via kombu and balances messages across
a pool of workers that are similar to celeryd workers in spirit.
Specifically, this includes:
- a new decorator, `awx.main.dispatch.task`, which can be used to
decorate functions or classes so that they can be designated as
"Tasks"
- support for fanout/broadcast tasks (at this point in time, only
`conf.Setting` memcached flushes use this functionality)
- support for job reaping
- support for success/failure hooks for job runs (i.e.,
`handle_work_success` and `handle_work_error`)
- support for auto scaling worker pool that scale processes up and down
on demand
- minimal support for RPC, such as status checks and pool recycle/reload
now that we have the CSRF middleware, we have a reliable token
available to us which we can use to verify individual ws_receive
payloads; this is _simpler_ than making sure you've properly configured
trusted origins, and it's also more secure than Origin header checks
see: https://github.com/ansible/tower/issues/2661
* Each time a route is needed (i.e. when a task is sumitted to celery).
The router will be queried. This is ideal. With the previous method we
had to consider how a change in the routes would propogate to all celery
workers and nodes.
* fully describe the default awx queue
* Our dynamic queue registration would correct awx_private_queue.
However, we don't want celery to even create an "invalid"/extra
queue-exchange-route. This change makes sure we don't create extranious
things in rabbitmq.
* reduce the cluster queue registration output. Only output when the
queue registration list changes.
refactor existing handlers to be the related
"real" handler classes, which are swapped
out dynamically by external logger "proxy" handler class
real handler swapout only done on setting change
remove restart_local_services method
get rid of uWSGI fifo file
change TCP/UDP return type contract so that it mirrors
the request futures object
add details to socket error messages
* Using Kombu's default Broadcast() constructor requires only 1
parameter. That parameter defines the exchange name and the queue name
is randomly generated per-node.
* This caused problems if/when celery enters an infinite restart loop
because too many rabbit queues get created and rabbit OOM's
(gracefully).
* To remedy this we tell Broadcast the queue name to use, which is
derived from some constant + the node name so that the per-node queue
name is stable.
* Before, we had a special group, tower, that ran any async work that
tower needed done. This allowed users fine grain control over which
nodes did background work. However, this granularity was too complicated
for users. So now, all tower system work goes to a special non-user
exposed celery queue. Tower remains the fallback instance group to
execute jobs on. The tower group will be created upon install and
protected from deletion.
* Separates test messages from application messages
* Removes test runner and groups, processes, and streams from network_ui
* Adds network_ui_test
* Fixes routing for network_ui_test
* Removes coverage_report tool from network_ui
* Fixes network_ui_test test workflow
* Sets width and height of the page during tests
The ansible-network-ui prototype project builds a standalone Network UI
outside of Tower as its own Django application. The original prototype
code is located here:
https://github.com/benthomasson/ansible-network-ui.
The prototype provides a virtual canvas that supports placing
networking devices onto 2D plane and connecting those devices together
with connections called links. The point where the link connects
to the network device is called an interface. The devices, interfaces,
and links may all have their respective names. This models physical
networking devices is a simple fashion.
The prototype implements a pannable and zoomable 2D canvas in using SVG
elements and AngularJS directives. This is done by adding event
listeners for mouse and keyboard events to an SVG element that fills the
entire browser window.
Mouse and keyboard events are handled in a processing pipeline where
the processing units are implemented as finite state machines that
provide deterministic behavior to the UI.
The finite state machines are built in a visual way that makes
the states and transitions clearly evident. The visual tool for
building FSM is located here:
https://github.com/benthomasson/fsm-designer-svg. This tool
is a fork of this project where the canvas is the same. The elements
on the page are FSM states and the directional connections are called
transitions. The bootstrapping of the FSM designer tool and
network-ui happen in parallel. It was useful to try experiemental
code in FSM designer and then import it into network-ui.
The FSM designer tool provides a YAML description of the design
which can be used to generate skeleton code and check the implementation
against the design for discrepancies.
Events supported:
* Mouse click
* Mouse scroll-wheel
* Keyboard events
* Touch events
Interactions supported:
* Pan canvas by clicking-and-dragging on the background
* Zooming canvas by scrolling mousewheel
* Adding devices and links by using hotkeys
* Selecting devices, interaces, and links by clicking on their icon
* Editing labels on devices, interfaces, and links by double-clicking on
their icon
* Moving devices around the canvas by clicking-and-dragging on their
icon
Device types supported:
* router
* switch
* host
* racks
The database schema for the prototype is also developed with a visual
tool that makes the relationships in the snowflake schema for the models
quickly evident. This tool makes it very easy to build queries across
multiple tables using Django's query builder.
See: https://github.com/benthomasson/db-designer-svg
The client and the server communicate asynchronously over a websocket.
This allows the UI to be very responsive to user interaction since
the full request/response cycle is not needed for every user
interaction.
The server provides persistence of the UI state in the database
using event handlers for events generated in the UI. The UI
processes mouse and keyboard events, updates the UI, and
generates new types of events that are then sent to the server
to be persisted in the database.
UI elements are tracked by unique ids generated on the client
when an element is first created. This allows the elements to
be correctly tracked before they are stored in the database.
The history of the UI is stored in the TopologyHistory model
which is useful for tracking which client made which change
and is useful for implementing undo/redo.
Each message is given a unique id per client and has
a known message type. Message types are pre-populated
in the MessageType model using a database migration.
A History message containing all the change messages for a topology is
sent when the websocket is connected. This allows for undo/redo work
across sessions.
This prototype provides a server-side test runner for driving
tests in the user interface. Events are emitted on the server
to drive the UI. Test code coverage is measured using the
istanbul library which produces instrumented client code.
Code coverage for the server is is measured by the coverage library.
The test code coverage for the Python code is 100%.
