Adds CONTRIBUTING docs

2026-05-06 16:32:01 -05:00 · 2018-03-13 14:19:36 -04:00
parent f8992e0edf
commit 297816b110
12 changed files with 572 additions and 509 deletions
--- a/awx/network_ui/CONTRIBUTING.md
+++ b/awx/network_ui/CONTRIBUTING.md
@@ -0,0 +1,133 @@
+Network UI
+==========
+
+See [awx/ui/client/src/network-ui/CONTRIBUTING.md](../ui/client/src/network-ui/CONTRIBUTING.md) for the introduction
+to the Network UI client-side development.
+
+Server-Side Development
+-----------------------
+
+This document covers the Network UI server-side development.
+
+The Network UI is a UX driven feature to provide a graphical user
+experience that fits well into the network engineer's normal workflow. Their
+normal workflow includes a diagram drawn in a graphical drawing program, a
+spreadsheet, and the command line interface of their network gear.  Network
+architects design the network on the graphical diagram and then hand off the
+architecture to network operators who implement the architecture on the network
+using spreadsheets to manage their data and manually converting the data into
+CLI commands using their networking expertise and expertise with their physical
+gear.
+
+The server-side code supports the persistence needed to provide this graphical
+user experience of architecting a network and using that information along with
+additional information (stored in vars files) to configure the network devices
+using the CLI or NETCONF using Ansible playbooks and roles.
+
+Network UI Data Schema
+----------------------
+
+For the 3.3 release the persistence needed includes the position information of
+the devices on the virtual canvas and the type of the devices as well as
+information about the interfaces on the devices and the links connecting those
+interfaces.
+
+These requirements determine the database schema needed for the network UI which
+requires these models: Topology, Device, Interface, Link, Client, and TopologyInventory.
+
+![Models](designs/models.png)
+
+This diagram shows the relationships between the models in the Network UI schema.
+
+The models are:
+
+* Device - a host, switch, router, or other networking device
+* Interface - a connection point on a device for a link
+* Link - a physical connection between two devices to their respective interfaces
+* Topology - a collection of devices and links
+* TopologyInventory - a mapping between topologies and Tower inventories
+* Client - a UI client session
+
+
+Network UI Websocket Protocol
+-----------------------------
+
+Persistence for the network UI canvas state is implemented using an
+asynchronous websocket protocol to send information from the client to the
+server and vice-versa.  This two-way communication was chosen to support future
+features for streaming data to the canvas, broadcast messaging between clients,
+and for interaction performance on the UI.
+
+
+Messages
+--------
+
+JSON messages are passed over the `/network_ui/topology` websocket between the
+test client and the test server.  The protocol that is used for all messages is
+in ABNF (RFC5234):
+
+
+    message_type    =  'DeviceMove' / 'DeviceCreate' / 'DeviceDestroy' / 'DeviceLabelEdit' / 'DeviceSelected' / 'DeviceUnSelected' / 'InterfaceCreate' / 'InterfaceLabelEdit' / 'LinkLabelEdit' / 'LinkCreate' / 'LinkDestroy' / 'LinkSelected' / 'LinkUnSelected' / 'MultipleMessage' / 'Snapshot'
+    message_data    = '{' 'msg_type' ': ' message_type ', ' key-value *( ', ' key-value ) '}'
+    message         = '[ id , ' posint ']' / '[ topology_id , ' posint ']' / '[' message_type ', ' message_data ']'
+
+See https://github.com/AndyA/abnfgen/blob/master/andy/json.abnf for the rest of
+the JSON ABNF.
+
+See [designs/messages.yml](designs/messages.yml) for the allowable keys and
+values for each message type.
+
+
+Initially when the websocket is first opened the server will send four messages
+to the client.  These are:
+
+* the client id using the `id` message type.
+* the topology id using the `topology` message type.
+* a Topology record containing data for the canvas itself.
+* a Snapshot message containing all the data of the data on the canvas.
+
+As the user interacts with the canvas messages will be generated by the client
+and the `network_ui.consumers.Persistence` class will update the models that
+represent the canvas.
+
+
+
+Persistence
+-----------
+
+The class `awx.network_uiconsumers.Persistence` provides persistence for the Network UI canvas.
+It does so by providing message handlers that handle storage of the canvas change events
+into the database. Each event has a message handle with name `onX` where `X` is the name of the message
+type.  The handlers use the `filter/values_list`, `filter/values`, `filter/update`, and `filter/delete`
+patterns to update the data in the database quickly with a constant O(1) number of queries per event
+often with only one query needed. With `filter/update` and `filter/delete` all the work is done
+in the database and Python never needs to instaniate and garbage collect the model objects.
+
+Bulk operations (`filter/values`) in `send_snapshot` are used to produce a constant number of
+queries produce a snapshot when the canvas is first loaded.  This method avoids creating
+the model objects since it only produces dicts that are JSON serializable which are bundled
+together for the `Snapshot` message type.
+
+This method of persistence uses Django as a database query-compiler for transforms from
+the event types to the database types.  Using Django in this way is very performant since
+Python does very little work processing the data and when possible the data never leaves
+the database.
+
+
+Client Tracking
+---------------
+
+Each user session to the network UI canvas is tracked with the `Client` model.  Multiple
+clients can view and interact with the network UI canvas at a time.  They will see each other's
+edits to the canvas in real time.  This works by broadcasting the canvas change events to
+all clients viewing the same topology.
+
+```
+    # Send to all clients editing the topology
+    Group("topology-%s" % message.channel_session['topology_id']).send({"text": message['text']})
+```
+
+API
+---
+
+There is no user accessible API for this feature in the 3.3 release.
--- a/awx/network_ui/designs/README.md
+++ b/awx/network_ui/designs/README.md
@@ -4,6 +4,5 @@ The design files in this directory are used in the database schema designer tool

 * [models.png](models.png) - An image of the database schema design for network UI.
 * [models.yml](models.yml) - Provides the main schema design for the network UI project.
-* [api.yml](api.yml) - Provides additional meta-data for the API.

 ![Models](models.png)
--- a/awx/network_ui/designs/api.yml
+++ b/awx/network_ui/designs/api.yml
@@ -1,65 +0,0 @@
-models:
- name: Device
-  api: true
-  v2_end_point: /api/v2/canvas/device/
-  topology_id_query: topology_id
-  v2_lookup_field: host_id
-  create_transform:
-      id: id
-      name: name
-      device_type: type
-      x: x
-      y: y
-      interface_id_seq: interface_id_seq
-      process_id_seq: process_id_seq
-      host_id: host_id
-      topology_id: topology_id
- name: Link
-  api: true
-  v2_end_point: /api/v2/canvas/link/
-  topology_id_query: from_device__topology_id
-  create_transform:
-      id: id
-      name: name
-      from_device__id: from_device_id
-      from_interface__id: from_interface_id
-      to_device__id: to_device_id
-      to_interface__id: to_interface_id
- name: Interface
-  api: true
-  v2_end_point: /api/v2/canvas/interface/
-  topology_id_query: device__topology_id
-  create_transform:
-      id: id
-      name: name
-      device__id: device_id
- name: Group
-  api: true
-  v2_end_point: /api/v2/canvas/group/
-  topology_id_query: topology_id
- name: GroupDevice
-  api: true
-  v2_end_point: /api/v2/canvas/groupdevice/
-  topology_id_query: group__topology_id
- name: Topology
-  api: true
-  v2_end_point: /api/v2/canvas/topology/
-  topology_id_query: topology_id
- name: TopologyInventory
-  api: true
-  v2_end_point: /api/v2/canvas/topologyinventory/
-  topology_id_query: topology_id
- name: Toolbox
-  api: true
-  v2_end_point: /api/v2/canvas/toolbox/
- name: ToolboxItem
-  api: true
-  v2_end_point: /api/v2/canvas/toolboxitem/
- name: Stream
-  api: true
-  v2_end_point: /api/v2/canvas/stream/
-  topology_id_query: from_device__topology_id
- name: Process
-  api: true
-  v2_end_point: /api/v2/canvas/process/
-  topology_id_query: device__topology_id
--- a/awx/network_ui/designs/messages.yml
+++ b/awx/network_ui/designs/messages.yml
@@ -1,7 +1,5 @@
 messages:
    - {msg_type: DeviceMove, fields: [msg_type, sender, id, x, y, previous_x, previous_y]}
-    - {msg_type: DeviceInventoryUpdate, fields: [msg_type, sender, id, host_id]}
-    - {msg_type: GroupInventoryUpdate, fields: [msg_type, sender, id, group_id]}
    - {msg_type: DeviceCreate, fields: [msg_type, sender, id, x, y, name, type, host_id]}
    - {msg_type: DeviceDestroy, fields: [msg_type, sender, id, previous_x, previous_y, previous_name, previous_type, previous_host_id]}
    - {msg_type: DeviceLabelEdit, fields: [msg_type, sender, id, name, previous_name]}
@@ -14,40 +12,8 @@ messages:
    - {msg_type: LinkDestroy, fields: [msg_type, id, sender, name, from_device_id, to_device_id, from_interface_id, to_interface_id]}
    - {msg_type: LinkSelected, fields: [msg_type, sender, id]}
    - {msg_type: LinkUnSelected, fields: [msg_type, sender, id]}
-    - {msg_type: Undo, fields: [msg_type, sender, original_message]}
-    - {msg_type: Redo, fields: [msg_type, sender, original_message]}
-    - {msg_type: Deploy, fields: [msg_type, sender]}
-    - {msg_type: Destroy, fields: [msg_type, sender]}
-    - {msg_type: Discover, fields: [msg_type, sender]}
-    - {msg_type: Layout, fields: [msg_type, sender]}
    - {msg_type: MultipleMessage, fields: [msg_type, sender, messages]}
-    - {msg_type: MouseEvent, fields: [msg_type, sender, x, y, type, trace_id]}
-    - {msg_type: MouseWheelEvent, fields: [msg_type, sender, delta, deltaX, deltaY, type, originalEvent, trace_id]}
-    - {msg_type: KeyEvent, fields: [msg_type, sender, key, keyCode, type, altKey, shiftKey, ctrlKey, metaKey, trace_id]}
-    - {msg_type: StartRecording, fields: [msg_type, sender, trace_id]}
-    - {msg_type: StopRecording, fields: [msg_type, sender, trace_id]}
-    - {msg_type: ViewPort, fields: [msg_type, sender, scale, panX, panY, trace_id]}
-    - {msg_type: CopySite, fields: [msg_type, site]}
-    - {msg_type: GroupMove, fields: [msg_type, sender, id, x1, y1, x2, y2, previous_x1, previous_y1, previous_x2, previous_y2]}
-    - {msg_type: GroupCreate, fields: [msg_type, sender, id, x1, y1, x2, y2, name, type, group_id]}
-    - {msg_type: GroupDestroy, fields: [msg_type, sender, id, previous_x1, previous_y1, previous_x2, previous_y2, previous_name, previous_type, previous_group_id]}
-    - {msg_type: GroupLabelEdit, fields: [msg_type, sender, id, name, previous_name]}
-    - {msg_type: GroupSelected, fields: [msg_type, sender, id]}
-    - {msg_type: GroupUnSelected, fields: [msg_type, sender, id]}
-    - {msg_type: GroupMembership, fields: [msg_type, sender, id, members]}
-    - {msg_type: ProcessCreate, fields: [msg_type, id, name, type, device_id, x, y]}
-    - {msg_type: StreamCreate, fields: [msg_type, sender, id, from_id, to_id, label]}
-    - {msg_type: StreamDestroy, fields: [msg_type, sender, id, from_id, to_id, label]}
-    - {msg_type: StreamLabelEdit, fields: [msg_type, sender, id, label, previous_label]}
-    - {msg_type: StreamSelected, fields: [msg_type, sender, id]}
-    - {msg_type: StreamUnSelected, fields: [msg_type, sender, id]}
-    - {msg_type: FSMTrace, fields: [msg_type, order, sender, trace_id, fsm_name, from_state, to_state, recv_message_type]}
-    - {msg_type: ChannelTrace, fields: [msg_type, sender, trace_id, from_fsm, to_fsm, sent_message_type]}
-    - {msg_type: Snapshot, fields: [msg_type, sender, devices, links, groups, streams, order, trace_id]}
-    - {msg_type: EnableTest, fields: [msg_type]}
-    - {msg_type: DisableTest, fields: [msg_type]}
-    - {msg_type: StartTest, fields: [msg_type]}
-    - {msg_type: TestCompleted, fields: [msg_type]}
-    - {msg_type: TestResult, fields: [msg_type, sender, id, name, result, date, code_under_test]}
-    - {msg_type: Coverage, fields: [msg_type, sender, coverage, result_id]}
-
+    - {msg_type: Snapshot, fields: [msg_type, sender, devices, links, order, trace_id]}
+    - {msg_type: id, type: int}
+    - {msg_type: topology_id, type: int}
+    - {msg_type: Topology, fields: [topology_id, name, panX, panY, scale, link_id_seq, device_id_seq]}
--- a/awx/network_ui/designs/models.png
+++ b/awx/network_ui/designs/models.png
--- a/awx/network_ui/designs/models.yml
+++ b/awx/network_ui/designs/models.yml
@@ -1,18 +1,7 @@
 app: awx.network_ui
 external_models: []
 models:
- api: true
-  create_transform:
-    device_type: type
-    host_id: host_id
-    id: id
-    interface_id_seq: interface_id_seq
-    name: name
-    process_id_seq: process_id_seq
-    topology_id: topology_id
-    x: x
-    y: y
-  display: name
+- display: name
  fields:
  - name: device_id
    pk: true
@@ -36,27 +25,13 @@ models:
  - default: 0
    name: interface_id_seq
    type: IntegerField
-  - default: 0
-    name: process_id_seq
-    type: IntegerField
  - default: 0
    name: host_id
    type: IntegerField
  name: Device
-  topology_id_query: topology_id
-  v2_end_point: /api/v2/canvas/device/
-  v2_lookup_field: host_id
  x: 348
  y: 124
- api: true
-  create_transform:
-    from_device__id: from_device_id
-    from_interface__id: from_interface_id
-    id: id
-    name: name
-    to_device__id: to_device_id
-    to_interface__id: to_interface_id
-  fields:
+- fields:
  - name: link_id
    pk: true
    type: AutoField
@@ -86,12 +61,9 @@ models:
    name: name
    type: CharField
  name: Link
-  topology_id_query: from_device__topology_id
-  v2_end_point: /api/v2/canvas/link/
-  x: 837
-  y: 10
- api: true
-  display: name
+  x: 731
+  y: -33
+- display: name
  fields:
  - name: topology_id
    pk: true
@@ -111,15 +83,7 @@ models:
  - default: 0
    name: link_id_seq
    type: IntegerField
-  - default: 0
-    name: group_id_seq
-    type: IntegerField
-  - default: 0
-    name: stream_id_seq
-    type: IntegerField
  name: Topology
-  topology_id_query: topology_id
-  v2_end_point: /api/v2/canvas/topology/
  x: 111
  y: 127
 - fields:
@@ -127,51 +91,9 @@ models:
    pk: true
    type: AutoField
  name: Client
-  x: -518
-  y: 138
- fields:
-  - name: topology_history_id
-    pk: true
-    type: AutoField
-  - name: topology
-    ref: Topology
-    ref_field: topology_id
-    type: ForeignKey
-  - name: client
-    ref: Client
-    ref_field: client_id
-    type: ForeignKey
-  - name: message_type
-    ref: MessageType
-    ref_field: message_type_id
-    type: ForeignKey
-  - name: message_id
-    type: IntegerField
-  - name: message_data
-    type: TextField
-  - default: false
-    name: undone
-    type: BooleanField
-  name: TopologyHistory
-  x: -205
+  x: -162
  y: 282
 - display: name
-  fields:
-  - name: message_type_id
-    pk: true
-    type: AutoField
-  - len: 200
-    name: name
-    type: CharField
-  name: MessageType
-  x: -501
-  y: 428
- api: true
-  create_transform:
-    device__id: device_id
-    id: id
-    name: name
-  display: name
  fields:
  - name: interface_id
    pk: true
@@ -186,170 +108,9 @@ models:
  - name: id
    type: IntegerField
  name: Interface
-  topology_id_query: device__topology_id
-  v2_end_point: /api/v2/canvas/interface/
-  x: 1157
-  y: 337
- api: true
-  fields:
-  - name: group_id
-    pk: true
-    type: AutoField
-  - name: id
-    type: IntegerField
-  - len: 200
-    name: name
-    type: CharField
-  - name: x1
-    type: IntegerField
-  - name: y1
-    type: IntegerField
-  - name: x2
-    type: IntegerField
-  - name: y2
-    type: IntegerField
-  - name: topology
-    ref: Topology
-    ref_field: topology_id
-    type: ForeignKey
-  - len: 200
-    name: group_type
-    type: CharField
-  - default: 0
-    name: inventory_group_id
-    type: IntegerField
-  name: Group
-  topology_id_query: topology_id
-  v2_end_point: /api/v2/canvas/group/
-  x: 407
-  y: -379
- api: true
-  fields:
-  - name: group_device_id
-    pk: true
-    type: AutoField
-  - name: group
-    ref: Group
-    ref_field: group_id
-    type: ForeignKey
-  - name: device
-    ref: Device
-    ref_field: device_id
-    type: ForeignKey
-  name: GroupDevice
-  topology_id_query: group__topology_id
-  v2_end_point: /api/v2/canvas/groupdevice/
-  x: 739
-  y: -234
- api: true
-  fields:
-  - name: stream_id
-    pk: true
-    ref: Stream
-    ref_field: stream_id
-    type: AutoField
-  - name: from_device
-    ref: Device
-    ref_field: device_id
-    related_name: from_stream
-    type: ForeignKey
-  - name: to_device
-    ref: Device
-    ref_field: device_id
-    related_name: to_stream
-    type: ForeignKey
-  - len: 200
-    name: label
-    type: CharField
-  - default: 0
-    name: id
-    type: IntegerField
-  name: Stream
-  topology_id_query: from_device__topology_id
-  v2_end_point: /api/v2/canvas/stream/
-  x: 709
-  y: 527
- api: true
-  fields:
-  - name: process_id
-    pk: true
-    type: AutoField
-  - name: device
-    ref: Device
-    ref_field: device_id
-    type: ForeignKey
-  - len: 200
-    name: name
-    type: CharField
-  - len: 200
-    name: process_type
-    type: CharField
-  - default: 0
-    name: id
-    type: IntegerField
-  name: Process
-  topology_id_query: device__topology_id
-  v2_end_point: /api/v2/canvas/process/
-  x: 654
-  y: 778
- api: true
-  fields:
-  - name: toolbox_id
-    pk: true
-    type: AutoField
-  - len: 200
-    name: name
-    type: CharField
-  name: Toolbox
-  v2_end_point: /api/v2/canvas/toolbox/
-  x: 179
-  y: 644
- api: true
-  fields:
-  - name: toolbox_item_id
-    pk: true
-    type: AutoField
-  - name: toolbox
-    ref: Toolbox
-    ref_field: toolbox_id
-    type: ForeignKey
-  - name: data
-    type: TextField
-  name: ToolboxItem
-  v2_end_point: /api/v2/canvas/toolboxitem/
-  x: 391
-  y: 645
+  x: 977
+  y: 312
 - fields:
-  - name: fsm_trace_id
-    pk: true
-    type: AutoField
-  - len: 200 
-    name: fsm_name
-    type: CharField
-  - len: 200 
-    name: from_state
-    type: CharField
-  - len: 200 
-    name: to_state
-    type: CharField
-  - len: 200 
-    name: message_type
-    type: CharField
-  - name: client
-    ref: Client
-    ref_field: client_id
-    type: ForeignKey
-  - default: 0
-    name: trace_session_id
-    type: IntegerField
-  - default: 0
-    name: order
-    type: IntegerField
-  name: FSMTrace
-  x: -872
-  y: 507
- api: true
-  fields:
  - name: topology_inventory_id
    pk: true
    type: AutoField
@@ -360,137 +121,10 @@ models:
  - name: inventory_id
    type: IntegerField
  name: TopologyInventory
-  topology_id_query: topology_id
-  v2_end_point: /api/v2/canvas/topologyinventory/
-  x: -226
-  y: -19
- fields:
-  - name: event_trace_id
-    pk: true
-    type: AutoField
-  - name: client
-    ref: Client
-    ref_field: client_id
-    type: ForeignKey
-  - default: 0
-    name: trace_session_id
-    type: IntegerField
-  - name: event_data
-    type: TextField
-  - name: message_id
-    type: IntegerField
-  name: EventTrace
-  x: -1087
-  y: 202
- fields:
-  - name: coverage_id
-    pk: true
-    type: AutoField
-  - name: coverage_data
-    type: TextField
-  - name: test_result
-    ref: TestResult
-    ref_field: test_result_id
-    type: ForeignKey
-  name: Coverage
-  x: -1068
-  y: -4
- fields:
-  - name: topology_snapshot_id
-    pk: true
-    type: AutoField
-  - name: client
-    ref: Client
-    ref_field: client_id
-    type: ForeignKey
-  - name: topology_id
-    type: IntegerField
-  - name: trace_session_id
-    type: IntegerField
-  - name: snapshot_data
-    ref: TopologySnapshot
-    ref_field: snapshot_data
-    type: TextField
-  - name: order
-    type: IntegerField
-  name: TopologySnapshot
-  x: -1123
-  y: -277
- fields:
-  - name: test_case_id
-    pk: true
-    type: AutoField
-  - len: 200
-    name: name
-    ref: TestCase
-    ref_field: name
-    type: CharField
-  - name: test_case_data
-    type: TextField
-  name: TestCase
-  x: -1642
-  y: -38
- fields:
-  - name: result_id
-    pk: true
-    type: AutoField
-  - len: 20
-    name: name
-    type: CharField
-  name: Result
-  x: -1610
-  y: 120
- fields:
-  - name: code_under_test_id
-    pk: true
-    ref: CodeUnderTest
-    ref_field: code_under_test_id
-    type: AutoField
-  - name: version_x
-    type: IntegerField
-  - name: version_y
-    type: IntegerField
-  - name: version_z
-    type: IntegerField
-  - name: commits_since
-    type: IntegerField
-  - len: 40
-    name: commit_hash
-    type: CharField
-  name: CodeUnderTest
-  x: -1612
-  y: 259
- fields:
-  - name: test_result_id
-    pk: true
-    type: AutoField
-  - name: test_case
-    ref: TestCase
-    ref_field: test_case_id
-    type: ForeignKey
-  - name: result
-    ref: Result
-    ref_field: result_id
-    type: ForeignKey
-  - name: code_under_test
-    ref: CodeUnderTest
-    ref_field: code_under_test_id
-    type: ForeignKey
-  - name: time
-    type: DateTimeField
-  - default: 0
-    name: id
-    type: IntegerField
-  - name: client
-    ref: Client
-    ref_field: client_id
-    type: ForeignKey
-  name: TestResult
-  x: -1336
-  y: -49
+  x: -204
+  y: 12
 modules: []
 view:
  panX: 213.729555519212
  panY: 189.446959094643
  scaleXY: 0.69
-