Browse Source

Initial import of osel code

This is an initial import of the osel codebase.  The osel tool is a tool that
initiates external security scans (initially through Qualys) upon reciept of
AMQP events that indicate certain sensitive events have occurred, like a
security group rule change.

The commit history had to be thrown away because it contained some non-public
data, so I would like to call out the following contributors:

This uses go 1.10 and vgo for dependency management.

Co-Authored-By: Charles Bitter <Charles_Bitter@cable.comcast.com>
Co-Authored-By: Olivier Gagnon <Olivier_Gagnon@cable.comcast.com>
Co-Authored-By: Joseph Sleiman <Joseph_Sleiman@comcast.com>

Change-Id: Ib6abe2024fd91978b783ceee4cff8bb4678d7b15
Nate Johnston 1 year ago
parent
commit
ca0e1ca769
48 changed files with 3740 additions and 0 deletions
  1. 1
    0
      .gitignore
  2. 18
    0
      Makefile
  3. 89
    0
      README.md
  4. 141
    0
      STORY.md
  5. 92
    0
      amqp.go
  6. 3
    0
      bindep.txt
  7. 57
    0
      events.go
  8. 299
    0
      events_json_fixtures_test.go
  9. 61
    0
      events_test.go
  10. 6
    0
      fixtures/README.md
  11. 90
    0
      fixtures/event_processor/icehouse/ignore_events/compute_instance_create_start.json
  12. 88
    0
      fixtures/event_processor/icehouse/ignore_events/compute_instance_delete_end.json
  13. 88
    0
      fixtures/event_processor/icehouse/ignore_events/compute_instance_shutdown_end.json
  14. 88
    0
      fixtures/event_processor/icehouse/ignore_events/compute_instance_shutdown_start.json
  15. 36
    0
      fixtures/event_processor/icehouse/port_create_with_default_security_group.json
  16. 38
    0
      fixtures/event_processor/icehouse/port_create_with_security_group_attached.json
  17. 26
    0
      fixtures/event_processor/icehouse/port_delete_with_default_security_group.json
  18. 28
    0
      fixtures/event_processor/icehouse/port_delete_with_security_group_attached.json
  19. 26
    0
      fixtures/event_processor/icehouse/security_group_rule_delete_with_ssh_open_to_the_internet.json
  20. 6
    0
      fixtures/viper/test.yml
  21. 20
    0
      go.mod
  22. 129
    0
      main.go
  23. 126
    0
      openstack.go
  24. 31
    0
      openstack_mock_test.go
  25. 133
    0
      processing.go
  26. 48
    0
      processing_test.go
  27. 104
    0
      qualys.go
  28. 170
    0
      qualys/assets.go
  29. 251
    0
      qualys/assets_test.go
  30. 68
    0
      qualys/assets_xml_fixtures_test.go
  31. 130
    0
      qualys/client.go
  32. 207
    0
      qualys/client_test.go
  33. 137
    0
      qualys/requests.go
  34. 186
    0
      qualys/scans.go
  35. 71
    0
      qualys/scans_test.go
  36. 44
    0
      qualys/utils.go
  37. 18
    0
      qualys/xml/scan_launch.xml
  38. 23
    0
      qualys/xml/scan_list_output.xml
  39. 81
    0
      qualys/xml/scan_results.xml
  40. 36
    0
      qualys_mock_test.go
  41. 24
    0
      releasenotes/notes/initial-import-8cdbc214e8596521.yaml
  42. 81
    0
      security_group_events.go
  43. 80
    0
      structs.go
  44. 81
    0
      syslog.go
  45. 24
    0
      syslog_mock_test.go
  46. 59
    0
      tools/test-setup.sh
  47. 55
    0
      viper.go
  48. 42
    0
      viper_test.go

+ 1
- 0
.gitignore View File

@@ -0,0 +1 @@
1
+osel

+ 18
- 0
Makefile View File

@@ -0,0 +1,18 @@
1
+# Makefile for the 'osel' project.
2
+
3
+# Note that the installation of go and vgo is accomplished by
4
+# tools/test-setup.sh
5
+
6
+SOURCE?=./...
7
+
8
+env:
9
+	@echo "Running build"
10
+	$(HOME)/go/bin/vgo build
11
+
12
+test: 
13
+	@echo "Running tests"
14
+	$(HOME)/go/bin/vgo test $(SOURCE) -cover
15
+
16
+fmt: 
17
+	@echo "Running fmt"
18
+	go fmt $(SOURCE)

+ 89
- 0
README.md View File

@@ -0,0 +1,89 @@
1
+OpenStack Event Listener
2
+========================
3
+
4
+What does this do?
5
+------------------
6
+
7
+The OpenStack Event Listener connects to the OpenStack message bus (RabbitMQ)
8
+and listens for certain kinds of events.  When it detects those events, it will
9
+gather additional data and forward the information to external systems for
10
+processing.  It integrates with syslog and the Qualys API.
11
+
12
+The initial use case that inspired this project was to detect when security
13
+group changes occurred and to trigger an external port scan of the affected IP
14
+addresses so that we could ensure that the change did not create a new
15
+vulnerability by opening something up to the Internet.
16
+
17
+For more background information on this project, see [the story of
18
+osel](STORY.md).
19
+
20
+Current State
21
+-------------
22
+Code maturity is considered experimental.
23
+
24
+Installation
25
+------------
26
+Use `go get git.openstack.org/openstack/osel`.  Or alternatively,
27
+download or clone the repository.
28
+
29
+The lib was developed and tested on go 1.10. 
30
+
31
+Configuration
32
+-------------
33
+
34
+Configuration resides in a YAML-format configuration file.  Before running the
35
+os_event_listener process set the EL_CONFIG environment variable to the
36
+absolute path to that file.
37
+
38
+This is an example of the configuration format:
39
+
40
+```yaml
41
+debug: true
42
+batch_interval: 2
43
+rabbit_uri: "amqp://amqp_user:amqp_password@amqp_host:amqp_port//"
44
+logfile: "/var/log/os_event_listener.log"
45
+syslog_server: your.syslog.server.fqdn
46
+syslog_port: "514"
47
+syslog_protocol: "tcp"
48
+retry_syslog: "false"
49
+openstack:
50
+  identity_endpoint: "https://keystone.url:5000/v2.0/"
51
+  tenant_name: "tenant_to_authenticate_against"
52
+  user: "username"
53
+  password: "password"
54
+  region: "region_name"
55
+qualys:
56
+  username: "qualys_username"
57
+  password: "qualys_password"
58
+  option: "Name Of The Qualys Scan Profile"
59
+  proxy_url: "http://in.case.you.need.to.proxy.to.reach.qualys/"
60
+  url: "https://qualysapi.qualys.com/api/2.0/fo/scan/"
61
+  drop6: true
62
+```
63
+
64
+Testing
65
+-------
66
+There is one type of test file.  The `*_test.go` are standard golang unit test
67
+files.  The examples can be run as integration tests.
68
+
69
+License
70
+-------
71
+Apache v2.
72
+
73
+Contributing
74
+------------
75
+The code repository utilizes the OpenStack CI infrastructure.  Please use the
76
+[recommended
77
+workflow](http://docs.openstack.org/infra/manual/developers.html#development-workflow).
78
+If you are not a member yet, please consider joining as an [OpenStack
79
+contributor](http://docs.openstack.org/infra/manual/developers.html).  If you
80
+have questions or comments, you can email the maintainer(s).
81
+
82
+Coding Style
83
+------------
84
+The source code is automatically formatted to follow `go fmt`.
85
+
86
+OpenStack Environment
87
+---------------------
88
+* Release note management is done using [reno](https://docs.openstack.org/reno/latest/user/usage.html)
89
+* Zuul CI jobs are defined in-repo, [using these techniques](https://docs.openstack.org/infra/manual/zuulv3.html#howto-in-repo)

+ 141
- 0
STORY.md View File

@@ -0,0 +1,141 @@
1
+# OpenStack Event Listener
2
+
3
+## What is STORY.md?
4
+
5
+Retweeted Safia Abdalla (@captainsafia):
6
+
7
+From now on, you can expect to see a "STORY.md" in each of my @github repos
8
+that describes the technical story/process behind the project.
9
+
10
+https://twitter.com/captainsafia/status/839587421247389696
11
+
12
+## Introduction
13
+
14
+I wanted to write a little about a project that I enjoyed working on, called
15
+the OpenStack Event Listener, or "OSEL" for short.  This project bridged the
16
+OpenStack control plane on one hand, and an external scanning facility provided
17
+by Qualys.  It had a number of interesting challenges.  I was never able to
18
+really concentrate on it - this project took about 20% of my time for a period
19
+of about 3 months.
20
+
21
+I am writing this partially as catharsis, to allow my brain to mark this part
22
+of my mental inventory as ripe for reclamation.  I am also writing on the off
23
+chance that someone might find this useful in the future.
24
+
25
+## The Setting
26
+
27
+Let me paint a picture of the environment in which this development occurred.  
28
+
29
+The Comcast OpenStack environment was transitioning from the OpenStack Icehouse
30
+release (very old) to the Newton release (much more current).  This development
31
+occurred within the context of the Icehouse environment.  
32
+
33
+Comcast's security team uses S3 RiskFabric to manage auditing and tracking
34
+security vulnerabilities across the board.  They also engage the services of
35
+Qualys to perform network scanning (in a manner very similar to Nessus) once a
36
+day against all the CIDR blocks that comprise Comcast's Internet-routable IP
37
+addresses.  Qualys scanning could also be triggered on-demand.
38
+
39
+## Technical Requirements
40
+
41
+First, let me describe the technical requirements for OSEL:
42
+
43
+* OSEL would connect to the OpenStack RabbitMQ message bus and register as a
44
+  listener for "notification" events.  This would allow OSEL to inspect all
45
+  events, including security group changes.
46
+* When a security group change occurred, OSEL would ensure that it had the
47
+  details of the change (ports permitted or blocked) as well as a list of all
48
+  affected IP addresses.
49
+* OSEL would initiate a Qualys scan using the Qualys API.  This would return a
50
+  scan ID.
51
+* OSEL would log the change as well as the Qualys scan ID to the Security
52
+  instance of Splunk to create an audit trail.
53
+* Qualys scan results would be imported into S3 RiskFabric for security audit
54
+  management.
55
+
56
+## Implementation Approach
57
+
58
+My group does most of it's development in Go, and this was no exception.  
59
+
60
+This is what the data I was getting back from the AMQP message looked like.
61
+All identifiers have been scrambled.
62
+
63
+```json
64
+{  
65
+   "_context_roles":[  
66
+      "Member"
67
+   ],
68
+   "_context_request_id":"req-f96ea9a5-435e-4177-8e51-bfe60d0fae2a",
69
+   "event_type":"security_group_rule.create.end",
70
+   "timestamp":"2016-10-03 18:10:59.112712",
71
+   "_context_tenant_id":"ada3b9b06482909f9361e803b54f5f32",
72
+   "_unique_id":"eafc9362327442b49d8c03b0e88d0216",
73
+   "_context_tenant_name":"BLURP",
74
+   "_context_user":"bca89c1b248e4a78282899ece9e744cc54",
75
+   "_context_user_id":"bca89c1b248e4a78282899ece9e744cc54",
76
+   "payload":{  
77
+      "security_group_rule_id":"bf8318fc-f9cb-446b-ffae-a8de016c562"
78
+   },
79
+   "_context_project_name":"BLURP",
80
+   "_context_read_deleted":"no",
81
+   "_context_tenant":"ada3b9b06482909f9361e803b54f5f32",
82
+   "priority":"INFO",
83
+   "_context_is_admin":false,
84
+   "_context_project_id":"ada3b9b06482909f9361e803b54f5f32",
85
+   "_context_timestamp":"2016-10-03 18:10:59.079179",
86
+   "_context_user_name":"admin",
87
+   "publisher_id":"network.osctrl1",
88
+   "message_id":"e75fb2ee-85bf-44ba-a083-2445eca2ae10"
89
+}
90
+```
91
+
92
+## Testing Pattern
93
+
94
+I leaned heavily on dependency injection to make this code as testable as
95
+possible.  For example, I needed an object that would contain the persistent
96
+`syslog.Writer`.  I created a `SyslogActioner` interface to represent all
97
+interactions with syslog.  When the code is operating normally, interactions
98
+with syslog occur through methods of the `SyslogActions` struct.  But in unit
99
+testing mode the `SyslogTestActions` struct is used instead, and all that does
100
+is save copies of all messages that would have been sent so they can be
101
+compared against the intended messages.  This facilitates good testing.
102
+
103
+## Fate of the Project
104
+
105
+The OSEL project was implemented and installed into production.  There were two
106
+problems with it.
107
+
108
+The first to become visible is that there was no exponential backoff for the
109
+AMQP connection to the OpenStack control plane's RabbitMQ.  When that RabbitMQ
110
+had issues - which was surprisingly often - OSEL would hanner away, trying to
111
+connect to it.  That would not be too much of an issue; despite what was
112
+effectively an infinite loop, CPU usage was not extreme.  The real problem was
113
+that connection failures were logged - and logs could become several gigabytes
114
+in a matter of hours.  This was mitigated by the OpenStack operations team
115
+rotating the logs hourly, and alerting if an hour's worth of logs exceeded a
116
+set size.  It was my intention to use one of the many [exponential backoff
117
+modules](https://github.com/cenkalti/backoff) available out there to make this
118
+more graceful.
119
+
120
+The second - and fatal - issue is that S3 RiskFabric was not configured to
121
+ingest from Qualys scans more than once a day.  Since Qualys was already
122
+scanning the CIDR block that corresponded to our OpenStack instances once a
123
+day, we were essentially just adding noise to the system.  The frequency of the
124
+S3-Qualys imports could not be easily altered, and as a result the project was
125
+shelved. 
126
+
127
+## Remaining Work
128
+
129
+If OSEL were ever to be un-shelved, here are a few of the things that I wish I
130
+had time to implement.
131
+
132
+- Neutron Port Events: The initial release of OSEL processed only security
133
+  group rule additions, modifications, or deletions.  So that covered the base
134
+  case for when a security group was already associated with a set of OpenStack
135
+  Networking (neutron) ports.   But a scan should be similarly launched when a
136
+  new port is created and associated to a security group.  This is what happens
137
+  when a new host is created.
138
+- Modern OpenStack: In order to make this work with a more modern OpenStack, it
139
+  would probably best to integrate with events generated through Aodh.  Aodh
140
+  seems to be built for this kind of reporting. 
141
+- Implement exponential backoff for AMQP connections as mentioned earlier.

+ 92
- 0
amqp.go View File

@@ -0,0 +1,92 @@
1
+package main
2
+
3
+/*
4
+
5
+amqp - This file includes all of the logic necessary to interact with the amqp
6
+library.  This is extrapolated out so that a AmqpInterface interface can be
7
+passed to functions.  Doing this allows testing by mock classes to be created
8
+that can be passed to functions.
9
+
10
+Since this is a wrapper around the amqp library, this does not need testing.
11
+
12
+*/
13
+
14
+import (
15
+	"fmt"
16
+	"log"
17
+
18
+	"github.com/streadway/amqp"
19
+)
20
+
21
+// AmqpActioner is an interface for an AmqpActions class.  Having
22
+// this as an interface allows us to pass in a dummy class for testing that
23
+// just returns mocked data.
24
+type AmqpActioner interface {
25
+	Connect() (<-chan amqp.Delivery, error)
26
+}
27
+
28
+// AmqpActions is a class that handles all interactions directly with Amqp.
29
+// See the comment on AmqpActioner for rationale.
30
+type AmqpActions struct {
31
+	Incoming       *<-chan amqp.Delivery
32
+	Options        AmqpOptions
33
+	AmqpConnection *amqp.Connection
34
+	AmqpChannel    *amqp.Channel
35
+	NotifyError    chan *amqp.Error
36
+}
37
+
38
+// AmqpOptions is a class to convey all of the configurable options for the
39
+// AmqpActions class.
40
+type AmqpOptions struct {
41
+	RabbitURI string
42
+}
43
+
44
+// Connect initiates the initial connection to the AMQP.
45
+func (s *AmqpActions) Connect() (<-chan amqp.Delivery, chan *amqp.Error, error) {
46
+	var err error
47
+
48
+	s.AmqpConnection, err = amqp.Dial(s.Options.RabbitURI)
49
+	if err != nil {
50
+		return nil, nil, fmt.Errorf("Failed to connect to RabbitMQ: %s", err)
51
+	}
52
+	s.NotifyError = s.AmqpConnection.NotifyClose(make(chan *amqp.Error)) //error channel
53
+
54
+	s.AmqpChannel, err = s.AmqpConnection.Channel()
55
+	if err != nil {
56
+		return nil, nil, fmt.Errorf("Failed to open a channel: %s", err)
57
+	}
58
+
59
+	amqpQueue, err := s.AmqpChannel.QueueDeclare(
60
+		"notifications.info", // name
61
+		false,                // durable
62
+		false,                // delete when usused
63
+		false,                // exclusive
64
+		false,                // no-wait
65
+		nil,                  // arguments
66
+	)
67
+	if err != nil {
68
+		return nil, nil, fmt.Errorf("Failed to declare a queue: %s", err)
69
+	}
70
+
71
+	amqpIncoming, err := s.AmqpChannel.Consume(
72
+		amqpQueue.Name, // queue
73
+		"osel",         // consumer
74
+		true,           // auto-ack
75
+		false,          // exclusive
76
+		false,          // no-local
77
+		false,          // no-wait
78
+		nil,            // args
79
+	)
80
+	if err != nil {
81
+		return nil, nil, fmt.Errorf("Failed to register a consumer: %s", err)
82
+	}
83
+	s.Incoming = &amqpIncoming
84
+	return amqpIncoming, s.NotifyError, nil
85
+}
86
+
87
+// Close closes connections
88
+func (s AmqpActions) Close() {
89
+	log.Println("Closing AMQP connection")
90
+	s.AmqpConnection.Close()
91
+	s.AmqpChannel.Close()
92
+}

+ 3
- 0
bindep.txt View File

@@ -0,0 +1,3 @@
1
+build-essential
2
+make
3
+clang

+ 57
- 0
events.go View File

@@ -0,0 +1,57 @@
1
+package main
2
+
3
+import (
4
+	"encoding/json"
5
+	"log"
6
+	"strings"
7
+)
8
+
9
+// EventProcessor is an Interface for event-specific classes that will process
10
+// events based on their specific fiends.
11
+type EventProcessor interface {
12
+	FormatLogs(*Event, []string) ([]string, error)
13
+	FillExtraData(*Event, OpenStackActioner) error
14
+}
15
+
16
+// Event is a class representing an event accepted from the AMQP, and the
17
+// additional attributes that have been parsed from it.
18
+type Event struct {
19
+	EventData          *openStackEvent
20
+	RawData            []byte
21
+	IPs                map[string][]string
22
+	SecurityGroupRules []*osSecurityGroupRule
23
+	LogLines           []string
24
+	Processor          EventProcessor
25
+	QualysScanID       string
26
+	QualysScanError    string
27
+}
28
+
29
+// ParseEvent takes the []byte that has been received from the AMQP message,
30
+// demarshals the JSON, and then returns the event data as well as an event
31
+// processor specific to that type of event.
32
+func ParseEvent(message []byte) (Event, error) {
33
+	var osEvent openStackEvent
34
+	if err := json.Unmarshal(message, &osEvent); err != nil {
35
+		return Event{}, err
36
+	}
37
+
38
+	e := Event{
39
+		EventData: &osEvent,
40
+		RawData:   message,
41
+	}
42
+
43
+	if Debug {
44
+		log.Printf("Event detected: %s\n", osEvent.EventType)
45
+	}
46
+
47
+	switch {
48
+	case strings.Contains(e.EventData.EventType, "security_group_rule.create.end"):
49
+		e.Processor = EventSecurityGroupRuleChange{ChangeType: "sg_rule_add"}
50
+	case strings.Contains(e.EventData.EventType, "security_group_rule.delete.end"):
51
+		e.Processor = EventSecurityGroupRuleChange{ChangeType: "sg_rule_del"}
52
+		// case strings.Contains(e.EventData.EventType, "port.create.end"):
53
+		// 	e.Processor = EventPortChange{ChangeType: "port_create"}
54
+	}
55
+
56
+	return e, nil
57
+}

+ 299
- 0
events_json_fixtures_test.go View File

@@ -0,0 +1,299 @@
1
+package main
2
+
3
+const (
4
+	portCreateWhenCreatingInstance = `
5
+  {
6
+    "_context_roles": [
7
+      "admin"
8
+    ],
9
+    "_context_request_id": "req-fdb23f2e-9c0e-46b1-802f-3194c1fad251",
10
+    "event_type": "port.create.end",
11
+    "timestamp": "2016-10-03 18:40:34.596836",
12
+    "_context_tenant_id": "0b65cf220eab4a3cbd68681d188d7dc7",
13
+    "_unique_id": "bca88f14c46e40559e981ac0b4ffebf5",
14
+    "_context_tenant_name": "services",
15
+    "_context_user": "31055c32b50442e5a4eb4c0f0cb3430b",
16
+    "_context_user_id": "31055c32b50442e5a4eb4c0f0cb3430b",
17
+    "payload": {
18
+      "port": {
19
+        "status": "DOWN",
20
+        "binding:host_id": "oscomp-ch2-a06",
21
+        "name": "",
22
+        "allowed_address_pairs": [
23
+
24
+        ],
25
+        "admin_state_up": true,
26
+        "network_id": "af33487a-4e96-4499-bfcd-4f741617a763",
27
+        "tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
28
+        "binding:vif_details": {
29
+          "port_filter": true,
30
+          "ovs_hybrid_plug": true
31
+        },
32
+        "binding:vnic_type": "normal",
33
+        "binding:vif_type": "ovs",
34
+        "device_owner": "compute:None",
35
+        "mac_address": "fa:16:3e:4a:ac:75",
36
+        "binding:profile": {
37
+        },
38
+        "fixed_ips": [
39
+          {
40
+            "subnet_id": "4a23cb36-b861-4daa-a8ef-c61360663669",
41
+            "ip_address": "162.150.0.117"
42
+          },
43
+          {
44
+            "subnet_id": "244c99a6-8011-4177-855b-dd493c5175c5",
45
+            "ip_address": "2001:558:fe21:403:f816:3eff:fe4a:ac75"
46
+          }
47
+        ],
48
+        "id": "a6c671d7-b4d5-4ebb-afaf-0c822bcc8948",
49
+        "security_groups": [
50
+          "0783a151-768c-49d3-a31d-178f70fabd51",
51
+          "46d46540-98ac-4c93-ae62-68dddab2282e"
52
+        ],
53
+        "device_id": "128bc33a-22ae-48b4-8283-093b6ec749d0"
54
+      }
55
+    },
56
+    "_context_project_name": "services",
57
+    "_context_read_deleted": "no",
58
+    "_context_tenant": "0b65cf220eab4a3cbd68681d188d7dc7",
59
+    "priority": "INFO",
60
+    "_context_is_admin": true,
61
+    "_context_project_id": "0b65cf220eab4a3cbd68681d188d7dc7",
62
+    "_context_timestamp": "2016-10-03 18:40:34.477012",
63
+    "_context_user_name": "neutron",
64
+    "publisher_id": "network.osctrl-ch2-a03",
65
+    "message_id": "71047538-531f-4aca-be09-a31bec441d16"
66
+  }
67
+
68
+  `
69
+
70
+	securityGroupRuleCreateWithCustomProtocall = `
71
+  {  
72
+     "_context_roles":[  
73
+        "Member"
74
+     ],
75
+     "_context_request_id":"req-a17c784c-fec9-4077-8908-44b6f56b6196",
76
+     "event_type":"security_group_rule.create.end",
77
+     "timestamp":"2016-10-03 17:50:59.982008",
78
+     "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
79
+     "_unique_id":"a7452605170c4979b2c6b76911d22026",
80
+     "_context_tenant_name":"VOIP",
81
+     "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
82
+     "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
83
+     "payload":{  
84
+        "security_group_rule":{  
85
+           "remote_group_id":null,
86
+           "direction":"ingress",
87
+           "protocol":10,
88
+           "remote_ip_prefix":"10.0.0.0/8",
89
+           "port_range_max":null,
90
+           "dscp":null,
91
+           "security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e",
92
+           "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
93
+           "port_range_min":null,
94
+           "ethertype":"IPv4",
95
+           "id":"3eff38bb-eb03-450b-aed4-019d612baeec"
96
+        }
97
+     },
98
+     "_context_project_name":"VOIP",
99
+     "_context_read_deleted":"no",
100
+     "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
101
+     "priority":"INFO",
102
+     "_context_is_admin":false,
103
+     "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
104
+     "_context_timestamp":"2016-10-03 17:50:59.925462",
105
+     "_context_user_name":"admin",
106
+     "publisher_id":"network.osctrl-ch2-a03",
107
+     "message_id":"6c93e24f-0892-494b-8e68-46252ceb9611"
108
+  }
109
+  `
110
+
111
+	securityGroupRuleCreateWithIcmpAndCider = `
112
+  {  
113
+     "_context_roles":[  
114
+        "Member"
115
+     ],
116
+     "_context_request_id":"req-c584fd21-9e58-4624-b316-b53487eed98e",
117
+     "event_type":"security_group_rule.create.end",
118
+     "timestamp":"2016-10-03 18:05:35.836029",
119
+     "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
120
+     "_unique_id":"cd280fd4f1474266bd0ad6e3ee5933a6",
121
+     "_context_tenant_name":"VOIP",
122
+     "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
123
+     "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
124
+     "payload":{  
125
+        "security_group_rule":{  
126
+           "remote_group_id":null,
127
+           "direction":"ingress",
128
+           "protocol":"icmp",
129
+           "remote_ip_prefix":"192.168.1.0/24",
130
+           "port_range_max":null,
131
+           "dscp":null,
132
+           "security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e",
133
+           "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
134
+           "port_range_min":null,
135
+           "ethertype":"IPv4",
136
+           "id":"66d7ac79-3551-4436-83c7-103b50760cfb"
137
+        }
138
+     },
139
+     "_context_project_name":"VOIP",
140
+     "_context_read_deleted":"no",
141
+     "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
142
+     "priority":"INFO",
143
+     "_context_is_admin":false,
144
+     "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
145
+     "_context_timestamp":"2016-10-03 18:05:35.769947",
146
+     "_context_user_name":"admin",
147
+     "publisher_id":"network.osctrl-ch2-a03",
148
+     "message_id":"f67b70d5-a782-4c5e-a274-a7ff197b73ec"
149
+  }
150
+  `
151
+	securityGroupRuleCreateWithports = `
152
+  {  
153
+     "_context_roles":[  
154
+        "Member"
155
+     ],
156
+     "_context_request_id":"req-1f17d667-c33f-4fa4-a026-8e2872dbf1d8",
157
+     "event_type":"security_group_rule.create.end",
158
+     "timestamp":"2016-10-03 17:32:25.723344",
159
+     "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
160
+     "_unique_id":"2fad8ecdd86e4748850d91bb0c83d625",
161
+     "_context_tenant_name":"VOIP",
162
+     "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
163
+     "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
164
+     "payload":{  
165
+        "security_group_rule":{  
166
+           "remote_group_id":null,
167
+           "direction":"ingress",
168
+           "protocol":"tcp",
169
+           "remote_ip_prefix":"10.0.0.0/8",
170
+           "port_range_max":443,
171
+           "dscp":null,
172
+           "security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e",
173
+           "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
174
+           "port_range_min":443,
175
+           "ethertype":"IPv4",
176
+           "id":"2b84d898-67b4-4370-9808-40a3fdb55a64"
177
+        }
178
+     },
179
+     "_context_project_name":"VOIP",
180
+     "_context_read_deleted":"no",
181
+     "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
182
+     "priority":"INFO",
183
+     "_context_is_admin":false,
184
+     "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
185
+     "_context_timestamp":"2016-10-03 17:32:25.665588",
186
+     "_context_user_name":"admin",
187
+     "publisher_id":"network.osctrl-ch2-a03",
188
+     "message_id":"4df01871-8bdb-4b85-bb34-cbff59ee6034"
189
+  }
190
+  `
191
+	securityGroupRuleCreateWithSecurityGroupAsRemoteIPPrefix = `
192
+  {  
193
+     "_context_roles":[  
194
+        "Member"
195
+     ],
196
+     "_context_request_id":"req-9e0360c7-786f-4a5b-84b6-7d2ccd23cbdd",
197
+     "event_type":"security_group_rule.create.end",
198
+     "timestamp":"2016-10-03 17:36:58.780554",
199
+     "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
200
+     "_unique_id":"b38fe8caed514eb2ba910e1ae74c6321",
201
+     "_context_tenant_name":"VOIP",
202
+     "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
203
+     "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
204
+     "payload":{  
205
+        "security_group_rule":{  
206
+           "remote_group_id":"0783a151-768c-49d3-a31d-178f70fabd51",
207
+           "direction":"ingress",
208
+           "protocol":"tcp",
209
+           "remote_ip_prefix":null,
210
+           "port_range_max":25,
211
+           "dscp":null,
212
+           "security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e",
213
+           "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
214
+           "port_range_min":20,
215
+           "ethertype":"IPv6",
216
+           "id":"7b14b6cd-f966-4b61-aaad-c03d8eacc830"
217
+        }
218
+     },
219
+     "_context_project_name":"VOIP",
220
+     "_context_read_deleted":"no",
221
+     "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
222
+     "priority":"INFO",
223
+     "_context_is_admin":false,
224
+     "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
225
+     "_context_timestamp":"2016-10-03 17:36:58.712962",
226
+     "_context_user_name":"admin",
227
+     "publisher_id":"network.osctrl-ch2-a03",
228
+     "message_id":"e2d7c089-8194-4523-8f84-ae22db497f60"
229
+  }
230
+  `
231
+	securityGroupRuleCreateWithSSHOpenToTheInternet = `
232
+  {  
233
+     "_context_roles":[  
234
+        "Member"
235
+     ],
236
+     "_context_request_id":"req-94df69c6-1c3f-48bd-b2f6-f47abdef5d9b",
237
+     "event_type":"security_group_rule.create.end",
238
+     "timestamp":"2016-10-03 18:09:11.938476",
239
+     "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
240
+     "_unique_id":"09412fff881543679f30412ef2342954",
241
+     "_context_tenant_name":"VOIP",
242
+     "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
243
+     "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
244
+     "payload":{  
245
+        "security_group_rule":{  
246
+           "remote_group_id":null,
247
+           "direction":"ingress",
248
+           "protocol":"tcp",
249
+           "remote_ip_prefix":"0.0.0.0/0",
250
+           "port_range_max":22,
251
+           "dscp":null,
252
+           "security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e",
253
+           "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
254
+           "port_range_min":22,
255
+           "ethertype":"IPv4",
256
+           "id":"bf288dfc-f9cb-446b-bacc-a8de016c9b11"
257
+        }
258
+     },
259
+     "_context_project_name":"VOIP",
260
+     "_context_read_deleted":"no",
261
+     "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
262
+     "priority":"INFO",
263
+     "_context_is_admin":false,
264
+     "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
265
+     "_context_timestamp":"2016-10-03 18:09:11.876789",
266
+     "_context_user_name":"admin",
267
+     "publisher_id":"network.osctrl-ch2-a03",
268
+     "message_id":"afb043b6-fa56-470b-b17e-984fb4cb6505"
269
+  }
270
+  `
271
+	securityGroupRuleDeleteWithIcmpAndCider = `
272
+  {
273
+    "_context_roles": [
274
+      "Member"
275
+    ],
276
+    "_context_request_id": "req-836eb80f-c6eb-459b-87b6-a093ebac3051",
277
+    "event_type": "security_group_rule.delete.end",
278
+    "timestamp": "2016-10-03 18:14:33.007074",
279
+    "_context_tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
280
+    "_unique_id": "04beeb34769b43bca09ec837d86ed18b",
281
+    "_context_tenant_name": "VOIP",
282
+    "_context_user": "bca89c1b248e4aef9c69ece9e744cc54",
283
+    "_context_user_id": "bca89c1b248e4aef9c69ece9e744cc54",
284
+    "payload": {
285
+      "security_group_rule_id": "7b14b6cd-f966-4b61-aaad-c03d8eacc830"
286
+    },
287
+    "_context_project_name": "VOIP",
288
+    "_context_read_deleted": "no",
289
+    "_context_tenant": "ada3b9b0dbac429f9361e803b54f5f32",
290
+    "priority": "INFO",
291
+    "_context_is_admin": false,
292
+    "_context_project_id": "ada3b9b0dbac429f9361e803b54f5f32",
293
+    "_context_timestamp": "2016-10-03 18:14:32.962116",
294
+    "_context_user_name": "admin",
295
+    "publisher_id": "network.osctrl-ch2-a03",
296
+    "message_id": "9bc5106c-a08b-4cda-9311-20bc16bc3008"
297
+  }
298
+  `
299
+)

+ 61
- 0
events_test.go View File

@@ -0,0 +1,61 @@
1
+package main
2
+
3
+import (
4
+	"reflect"
5
+	"testing"
6
+
7
+	"github.com/stretchr/testify/assert"
8
+)
9
+
10
+func TestParseEventWillReturnAnEventStruct(t *testing.T) {
11
+	event, err := ParseEvent([]byte(securityGroupRuleCreateWithports))
12
+	assert.Nil(t, err)
13
+	assert.Equal(t, "main.Event", reflect.TypeOf(event).String(),
14
+		"ParseEvent should return an Event struct")
15
+	assert.Equal(t, "security_group_rule.create.end", event.EventData.EventType)
16
+	assert.Equal(t, "bca89c1b248e4aef9c69ece9e744cc54", event.EventData.UserID)
17
+	assert.Equal(t, "admin", event.EventData.UserName)
18
+	assert.Equal(t, "ada3b9b0dbac429f9361e803b54f5f32", event.EventData.TenantID)
19
+	assert.Equal(t, "VOIP", event.EventData.TenantName)
20
+}
21
+
22
+func TestParseEventWillCreateTheProperEventProcessor(t *testing.T) {
23
+	e, err := ParseEvent([]byte(securityGroupRuleCreateWithports))
24
+	assert.Nil(t, err)
25
+	//assert.Equal(t, "main.EventSecurityGroupRuleChange", reflect.TypeOf(e.Processor).String(),
26
+	//	"ParseEvent should return the proper implementation of EventProcessor")
27
+	assert.Equal(t, EventSecurityGroupRuleChange{"sg_rule_add"}, e.Processor,
28
+		"ParseEvent should return the proper implementation of EventProcessor")
29
+
30
+	e, err = ParseEvent([]byte(securityGroupRuleDeleteWithIcmpAndCider))
31
+	assert.Nil(t, err)
32
+	assert.Equal(t, "main.EventSecurityGroupRuleChange", reflect.TypeOf(e.Processor).String(),
33
+		"ParseEvent should return the proper implementation of EventProcessor")
34
+
35
+	//	_, eventProcessor, err = ParseEvent([]byte(portCreateWhenCreatingInstance))
36
+	//	assert.Nil(t, err)
37
+	//	assert.Equal(t, "main.EventPortChange", reflect.TypeOf(eventProcessor).String(),
38
+	//		"ParseEvent should return the proper implementation of EventProcessor")
39
+
40
+}
41
+
42
+// func TestPortCreateEvent(t *testing.T) {
43
+//	fakeOpenStack := connectFakeOpenstack()
44
+//	event, eventProcessor, err := ParseEvent([]byte(portCreateWhenCreatingInstance))
45
+//	assert.Nil(t, err)
46
+//	eventProcessor.FillExtraData(&event, fakeOpenStack)
47
+//}
48
+
49
+func TestEventSecurityGroupRuleCreateEvent(t *testing.T) {
50
+	fakeOpenStack := connectFakeOpenstack()
51
+	event, err := ParseEvent([]byte(securityGroupRuleCreateWithports))
52
+	assert.Nil(t, err)
53
+	event.Processor.FillExtraData(&event, fakeOpenStack)
54
+}
55
+
56
+func TestEventSecurityGroupRuleDeleteEvent(t *testing.T) {
57
+	fakeOpenStack := connectFakeOpenstack()
58
+	event, err := ParseEvent([]byte(securityGroupRuleDeleteWithIcmpAndCider))
59
+	assert.Nil(t, err)
60
+	event.Processor.FillExtraData(&event, fakeOpenStack)
61
+}

+ 6
- 0
fixtures/README.md View File

@@ -0,0 +1,6 @@
1
+# Test Fixtures
2
+
3
+The fixtures folder is meant to hold test files that can be used to create positive and negative tests.
4
+
5
+The test files should be organized in a folder per _test.go file.   The folder should be named after the test.  For example foo_test.go 
6
+should use a folder called foo.  Sub folders are fine. Just don't go nuts and KISS 

+ 90
- 0
fixtures/event_processor/icehouse/ignore_events/compute_instance_create_start.json View File

@@ -0,0 +1,90 @@
1
+{
2
+  "_context_roles": [
3
+    "Member",
4
+    "admin"
5
+  ],
6
+  "_context_request_id": "req-49d3158a-1288-4b76-bed9-907666d7d8c5",
7
+  "_context_quota_class": null,
8
+  "event_type": "compute.instance.create.start",
9
+  "_context_service_catalog": [
10
+    {
11
+      "endpoints": [
12
+        {
13
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
14
+          "interface": "public",
15
+          "region": "ndc_ch2_a",
16
+          "id": "7d0a1277410648d4a3e0164ad233adea"
17
+        },
18
+        {
19
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
20
+          "interface": "admin",
21
+          "region": "ndc_ch2_a",
22
+          "id": "df06789d23684643a0b59bfa7ee26064"
23
+        }
24
+      ],
25
+      "type": "volume",
26
+      "id": "e90968e859474737a4ad7aadffb2f578"
27
+    }
28
+  ],
29
+  "timestamp": "2016-10-03 18:40:33.699871",
30
+  "_context_user": "bca89c1b248e4aef9c69ece9e744cc54",
31
+  "_unique_id": "ab304ee51028479d899f38cd100b9b36",
32
+  "_context_instance_lock_checked": false,
33
+  "_context_user_id": "bca89c1b248e4aef9c69ece9e744cc54",
34
+  "payload": {
35
+    "state_description": "scheduling",
36
+    "availability_zone": null,
37
+    "terminated_at": "",
38
+    "ephemeral_gb": 0,
39
+    "instance_type_id": 2,
40
+    "message": "",
41
+    "deleted_at": "",
42
+    "reservation_id": "r-vgn2gv93",
43
+    "instance_id": "128bc33a-22ae-48b4-8283-093b6ec749d0",
44
+    "user_id": "bca89c1b248e4aef9c69ece9e744cc54",
45
+    "hostname": "test-sc-group",
46
+    "state": "building",
47
+    "launched_at": "",
48
+    "metadata": {
49
+    },
50
+    "node": null,
51
+    "ramdisk_id": "",
52
+    "access_ip_v6": null,
53
+    "disk_gb": 1,
54
+    "access_ip_v4": null,
55
+    "kernel_id": "",
56
+    "image_name": "cirros-0.3.4",
57
+    "host": null,
58
+    "display_name": "test_sc_group",
59
+    "image_ref_url": "http://172.28.195.37:9292/images/caf23aab-f6ec-4691-8f50-b0c70a33eca3",
60
+    "root_gb": 1,
61
+    "tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
62
+    "created_at": "2016-10-03T18:40:33.000000",
63
+    "memory_mb": 512,
64
+    "instance_type": "m1.tiny",
65
+    "vcpus": 1,
66
+    "image_meta": {
67
+      "description": "cirros-0.3.4",
68
+      "container_format": "bare",
69
+      "min_ram": "0",
70
+      "disk_format": "qcow2",
71
+      "min_disk": "1",
72
+      "base_image_ref": "caf23aab-f6ec-4691-8f50-b0c70a33eca3"
73
+    },
74
+    "architecture": null,
75
+    "os_type": null,
76
+    "instance_flavor_id": "1"
77
+  },
78
+  "_context_project_name": "VOIP",
79
+  "_context_read_deleted": "no",
80
+  "_context_auth_token": "7b82d6dec2eb457d85e08940a7f7b33e",
81
+  "_context_tenant": "ada3b9b0dbac429f9361e803b54f5f32",
82
+  "priority": "INFO",
83
+  "_context_is_admin": false,
84
+  "_context_project_id": "ada3b9b0dbac429f9361e803b54f5f32",
85
+  "_context_timestamp": "2016-10-03T18:40:32.636697",
86
+  "_context_user_name": "admin",
87
+  "publisher_id": "compute.oscomp-ch2-a06",
88
+  "message_id": "0553fb1f-42d1-4959-8991-4b5034701d9d",
89
+  "_context_remote_address": "127.0.0.1"
90
+}

+ 88
- 0
fixtures/event_processor/icehouse/ignore_events/compute_instance_delete_end.json View File

@@ -0,0 +1,88 @@
1
+{
2
+  "_context_roles": [
3
+    "Member",
4
+    "admin"
5
+  ],
6
+  "_context_request_id": "req-dceb796f-f5cf-45b0-8f6c-78b669304f4d",
7
+  "_context_quota_class": null,
8
+  "event_type": "compute.instance.delete.end",
9
+  "_context_service_catalog": [
10
+    {
11
+      "endpoints": [
12
+        {
13
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
14
+          "interface": "public",
15
+          "region": "ndc_ch2_a",
16
+          "id": "7d0a1277410648d4a3e0164ad233adea"
17
+        },
18
+        {
19
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
20
+          "interface": "admin",
21
+          "region": "ndc_ch2_a",
22
+          "id": "df06789d23684643a0b59bfa7ee26064"
23
+        }
24
+      ],
25
+      "type": "volume",
26
+      "id": "e90968e859474737a4ad7aadffb2f578"
27
+    }
28
+  ],
29
+  "timestamp": "2016-10-03 18:25:18.941599",
30
+  "_context_user": "bca89c1b248e4aef9c69ece9e744cc54",
31
+  "_unique_id": "a4c13937ec624259856d11e0ae0717cf",
32
+  "_context_instance_lock_checked": false,
33
+  "_context_user_id": "bca89c1b248e4aef9c69ece9e744cc54",
34
+  "payload": {
35
+    "state_description": "",
36
+    "availability_zone": null,
37
+    "terminated_at": "2016-10-03T18:25:18.000000",
38
+    "ephemeral_gb": 0,
39
+    "instance_type_id": 2,
40
+    "deleted_at": "2016-10-03T18:25:18.857623",
41
+    "reservation_id": "r-osn0qo9l",
42
+    "instance_id": "563d0899-d1cc-4154-bb58-f932ad0b255c",
43
+    "user_id": "bca89c1b248e4aef9c69ece9e744cc54",
44
+    "hostname": "test",
45
+    "state": "deleted",
46
+    "launched_at": "2016-10-03T18:23:11.000000",
47
+    "metadata": {
48
+    },
49
+    "node": "openstack-host2",
50
+    "ramdisk_id": "",
51
+    "access_ip_v6": null,
52
+    "disk_gb": 1,
53
+    "access_ip_v4": null,
54
+    "kernel_id": "",
55
+    "host": "oscomp-ch2-a07",
56
+    "display_name": "test",
57
+    "image_ref_url": "http://172.28.195.38:9292/images/caf23aab-f6ec-4691-8f50-b0c70a33eca3",
58
+    "root_gb": 1,
59
+    "tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
60
+    "created_at": "2016-10-03 18:23:03+00:00",
61
+    "memory_mb": 512,
62
+    "instance_type": "m1.tiny",
63
+    "vcpus": 1,
64
+    "image_meta": {
65
+      "description": "cirros-0.3.4",
66
+      "container_format": "bare",
67
+      "min_ram": "0",
68
+      "disk_format": "qcow2",
69
+      "min_disk": "1",
70
+      "base_image_ref": "caf23aab-f6ec-4691-8f50-b0c70a33eca3"
71
+    },
72
+    "architecture": null,
73
+    "os_type": null,
74
+    "instance_flavor_id": "1"
75
+  },
76
+  "_context_project_name": "VOIP",
77
+  "_context_read_deleted": "no",
78
+  "_context_auth_token": "7b82d6dec2eb457d85e08940a7f7b33e",
79
+  "_context_tenant": "ada3b9b0dbac429f9361e803b54f5f32",
80
+  "priority": "INFO",
81
+  "_context_is_admin": false,
82
+  "_context_project_id": "ada3b9b0dbac429f9361e803b54f5f32",
83
+  "_context_timestamp": "2016-10-03T18:25:16.888382",
84
+  "_context_user_name": "admin",
85
+  "publisher_id": "compute.oscomp-ch2-a07",
86
+  "message_id": "cf848ea3-300e-4b9a-8672-2306e9413cbf",
87
+  "_context_remote_address": "127.0.0.1"
88
+}

+ 88
- 0
fixtures/event_processor/icehouse/ignore_events/compute_instance_shutdown_end.json View File

@@ -0,0 +1,88 @@
1
+{
2
+  "_context_roles": [
3
+    "Member",
4
+    "admin"
5
+  ],
6
+  "_context_request_id": "req-71eec143-a576-49aa-84f6-eeaccd4f6619",
7
+  "_context_quota_class": null,
8
+  "event_type": "compute.instance.shutdown.end",
9
+  "_context_service_catalog": [
10
+    {
11
+      "endpoints": [
12
+        {
13
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
14
+          "interface": "public",
15
+          "region": "ndc_ch2_a",
16
+          "id": "7d0a1277410648d4a3e0164ad233adea"
17
+        },
18
+        {
19
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
20
+          "interface": "admin",
21
+          "region": "ndc_ch2_a",
22
+          "id": "df06789d23684643a0b59bfa7ee26064"
23
+        }
24
+      ],
25
+      "type": "volume",
26
+      "id": "e90968e859474737a4ad7aadffb2f578"
27
+    }
28
+  ],
29
+  "timestamp": "2016-10-03 18:38:47.143591",
30
+  "_context_user": "bca89c1b248e4aef9c69ece9e744cc54",
31
+  "_unique_id": "2740b7baa2fb48fea3c3654bf94c304e",
32
+  "_context_instance_lock_checked": false,
33
+  "_context_user_id": "bca89c1b248e4aef9c69ece9e744cc54",
34
+  "payload": {
35
+    "state_description": "deleting",
36
+    "availability_zone": null,
37
+    "terminated_at": "",
38
+    "ephemeral_gb": 0,
39
+    "instance_type_id": 2,
40
+    "deleted_at": "",
41
+    "reservation_id": "r-s8qu0d07",
42
+    "instance_id": "8bca63d6-0d51-465b-9379-c20ccc8d3e17",
43
+    "user_id": "bca89c1b248e4aef9c69ece9e744cc54",
44
+    "hostname": "test-instance-one",
45
+    "state": "active",
46
+    "launched_at": "2016-10-03T17:23:59.000000",
47
+    "metadata": {
48
+    },
49
+    "node": "openstack-host1",
50
+    "ramdisk_id": "",
51
+    "access_ip_v6": null,
52
+    "disk_gb": 1,
53
+    "access_ip_v4": null,
54
+    "kernel_id": "",
55
+    "host": "oscomp-ch2-a06",
56
+    "display_name": "test_instance_one",
57
+    "image_ref_url": "http://172.28.195.37:9292/images/caf23aab-f6ec-4691-8f50-b0c70a33eca3",
58
+    "root_gb": 1,
59
+    "tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
60
+    "created_at": "2016-10-03 17:23:40+00:00",
61
+    "memory_mb": 512,
62
+    "instance_type": "m1.tiny",
63
+    "vcpus": 1,
64
+    "image_meta": {
65
+      "description": "cirros-0.3.4",
66
+      "container_format": "bare",
67
+      "min_ram": "0",
68
+      "disk_format": "qcow2",
69
+      "min_disk": "1",
70
+      "base_image_ref": "caf23aab-f6ec-4691-8f50-b0c70a33eca3"
71
+    },
72
+    "architecture": null,
73
+    "os_type": null,
74
+    "instance_flavor_id": "1"
75
+  },
76
+  "_context_project_name": "VOIP",
77
+  "_context_read_deleted": "no",
78
+  "_context_auth_token": "7b82d6dec2eb457d85e08940a7f7b33e",
79
+  "_context_tenant": "ada3b9b0dbac429f9361e803b54f5f32",
80
+  "priority": "INFO",
81
+  "_context_is_admin": true,
82
+  "_context_project_id": "ada3b9b0dbac429f9361e803b54f5f32",
83
+  "_context_timestamp": "2016-10-03T18:38:45.381205",
84
+  "_context_user_name": "admin",
85
+  "publisher_id": "compute.oscomp-ch2-a06",
86
+  "message_id": "1cffe568-1eee-4150-89a0-937973ce15e4",
87
+  "_context_remote_address": "127.0.0.1"
88
+}

+ 88
- 0
fixtures/event_processor/icehouse/ignore_events/compute_instance_shutdown_start.json View File

@@ -0,0 +1,88 @@
1
+{
2
+  "_context_roles": [
3
+    "Member",
4
+    "admin"
5
+  ],
6
+  "_context_request_id": "req-dceb796f-f5cf-45b0-8f6c-78b669304f4d",
7
+  "_context_quota_class": null,
8
+  "event_type": "compute.instance.shutdown.start",
9
+  "_context_service_catalog": [
10
+    {
11
+      "endpoints": [
12
+        {
13
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
14
+          "interface": "public",
15
+          "region": "ndc_ch2_a",
16
+          "id": "7d0a1277410648d4a3e0164ad233adea"
17
+        },
18
+        {
19
+          "url": "https://openstack-host:8776/v1/ada3b9b0dbac429f9361e803b54f5f32",
20
+          "interface": "admin",
21
+          "region": "ndc_ch2_a",
22
+          "id": "df06789d23684643a0b59bfa7ee26064"
23
+        }
24
+      ],
25
+      "type": "volume",
26
+      "id": "e90968e859474737a4ad7aadffb2f578"
27
+    }
28
+  ],
29
+  "timestamp": "2016-10-03 18:25:17.121727",
30
+  "_context_user": "bca89c1b248e4aef9c69ece9e744cc54",
31
+  "_unique_id": "b7c7128bd595423daeed9d6ad36c2b2a",
32
+  "_context_instance_lock_checked": false,
33
+  "_context_user_id": "bca89c1b248e4aef9c69ece9e744cc54",
34
+  "payload": {
35
+    "state_description": "deleting",
36
+    "availability_zone": null,
37
+    "terminated_at": "",
38
+    "ephemeral_gb": 0,
39
+    "instance_type_id": 2,
40
+    "deleted_at": "",
41
+    "reservation_id": "r-osn0qo9l",
42
+    "instance_id": "563d0899-d1cc-4154-bb58-f932ad0b255c",
43
+    "user_id": "bca89c1b248e4aef9c69ece9e744cc54",
44
+    "hostname": "test",
45
+    "state": "active",
46
+    "launched_at": "2016-10-03T18:23:11.000000",
47
+    "metadata": {
48
+    },
49
+    "node": "openstack-host2",
50
+    "ramdisk_id": "",
51
+    "access_ip_v6": null,
52
+    "disk_gb": 1,
53
+    "access_ip_v4": null,
54
+    "kernel_id": "",
55
+    "host": "oscomp-ch2-a07",
56
+    "display_name": "test",
57
+    "image_ref_url": "http://172.28.195.38:9292/images/caf23aab-f6ec-4691-8f50-b0c70a33eca3",
58
+    "root_gb": 1,
59
+    "tenant_id": "ada3b9b0dbac429f9361e803b54f5f32",
60
+    "created_at": "2016-10-03 18:23:03+00:00",
61
+    "memory_mb": 512,
62
+    "instance_type": "m1.tiny",
63
+    "vcpus": 1,
64
+    "image_meta": {
65
+      "description": "cirros-0.3.4",
66
+      "container_format": "bare",
67
+      "min_ram": "0",
68
+      "disk_format": "qcow2",
69
+      "min_disk": "1",
70
+      "base_image_ref": "caf23aab-f6ec-4691-8f50-b0c70a33eca3"
71
+    },
72
+    "architecture": null,
73
+    "os_type": null,
74
+    "instance_flavor_id": "1"
75
+  },
76
+  "_context_project_name": "VOIP",
77
+  "_context_read_deleted": "no",
78
+  "_context_auth_token": "7b82d6dec2eb457d85e08940a7f7b33e",
79
+  "_context_tenant": "ada3b9b0dbac429f9361e803b54f5f32",
80
+  "priority": "INFO",
81
+  "_context_is_admin": true,
82
+  "_context_project_id": "ada3b9b0dbac429f9361e803b54f5f32",
83
+  "_context_timestamp": "2016-10-03T18:25:16.888382",
84
+  "_context_user_name": "admin",
85
+  "publisher_id": "compute.oscomp-ch2-a07",
86
+  "message_id": "41963fa2-be81-42ee-b31e-e2f727f109b7",
87
+  "_context_remote_address": "127.0.0.1"
88
+}

+ 36
- 0
fixtures/event_processor/icehouse/port_create_with_default_security_group.json View File

@@ -0,0 +1,36 @@
1
+{  
2
+   "_context_roles":[  
3
+      "admin"
4
+   ],
5
+   "_context_request_id":"req-b22260da-ecd2-4eb6-a7bb-e07a216450de",
6
+   "event_type":"port.create.start",
7
+   "timestamp":"2016-10-03 18:23:04.894439",
8
+   "_context_tenant_id":"0b65cf220eab4a3cbd68681d188d7dc7",
9
+   "_unique_id":"d133f63bb45740f587b780d4dac7e2ee",
10
+   "_context_tenant_name":"services",
11
+   "_context_user":"31055c32b50442e5a4eb4c0f0cb3430b",
12
+   "_context_user_id":"31055c32b50442e5a4eb4c0f0cb3430b",
13
+   "payload":{  
14
+      "port":{  
15
+         "binding:host_id":"oscomp-ch2-a07",
16
+         "admin_state_up":true,
17
+         "network_id":"af33487a-4e96-4499-bfcd-4f741617a763",
18
+         "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
19
+         "device_owner":"compute:None",
20
+         "security_groups":[  
21
+            "0783a151-768c-49d3-a31d-178f70fabd51"
22
+         ],
23
+         "device_id":"563d0899-d1cc-4154-bb58-f932ad0b255c"
24
+      }
25
+   },
26
+   "_context_project_name":"services",
27
+   "_context_read_deleted":"no",
28
+   "_context_tenant":"0b65cf220eab4a3cbd68681d188d7dc7",
29
+   "priority":"INFO",
30
+   "_context_is_admin":true,
31
+   "_context_project_id":"0b65cf220eab4a3cbd68681d188d7dc7",
32
+   "_context_timestamp":"2016-10-03 18:23:04.892974",
33
+   "_context_user_name":"neutron",
34
+   "publisher_id":"network.osctrl-ch2-a02",
35
+   "message_id":"b256daa5-3eed-4fec-8c54-d93fa79ab753"
36
+}

+ 38
- 0
fixtures/event_processor/icehouse/port_create_with_security_group_attached.json View File

@@ -0,0 +1,38 @@
1
+{  
2
+   "_context_roles":[  
3
+      "admin"
4
+   ],
5
+   "_context_request_id":"req-662e5998-9dcd-4b76-a726-5d815a1ab9ef",
6
+   "event_type":"port.create.start",
7
+   "timestamp":"2016-10-03 17:25:04.189288",
8
+   "_context_tenant_id":"0b65cf220eab4a3cbd68681d188d7dc7",
9
+   "_unique_id":"edd46be30b524fe9a3af78cec809db0e",
10
+   "_context_tenant_name":"services",
11
+   "_context_user":"31055c32b50442e5a4eb4c0f0cb3430b",
12
+   "_context_user_id":"31055c32b50442e5a4eb4c0f0cb3430b",
13
+   "payload":{  
14
+      "port":{  
15
+         "binding:host_id":"oscomp-ch2-a07",
16
+         "admin_state_up":true,
17
+         "network_id":"af33487a-4e96-4499-bfcd-4f741617a763",
18
+         "tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
19
+         "device_owner":"compute:None",
20
+         "security_groups":[  
21
+            "0783a151-768c-49d3-a31d-178f70fabd51",
22
+            "46d46540-98ac-4c93-ae62-68dddab2282e"
23
+         ],
24
+         "device_id":"bb64fe08-0eae-4c83-8bc2-457b6cb7e9a3"
25
+      }
26
+   },
27
+   "_context_project_name":"services",
28
+   "_context_read_deleted":"no",
29
+   "_context_tenant":"0b65cf220eab4a3cbd68681d188d7dc7",
30
+   "priority":"INFO",
31
+   "_context_is_admin":true,
32
+   "_context_project_id":"0b65cf220eab4a3cbd68681d188d7dc7",
33
+   "_context_timestamp":"2016-10-03 17:25:04.187847",
34
+   "_context_user_name":"neutron",
35
+   "publisher_id":"network.osctrl-ch2-a02",
36
+   "message_id":"56253f06-7350-47c0-8284-d74701d11698"
37
+}
38
+

+ 26
- 0
fixtures/event_processor/icehouse/port_delete_with_default_security_group.json View File

@@ -0,0 +1,26 @@
1
+{  
2
+   "_context_roles":[  
3
+      "Member"
4
+   ],
5
+   "_context_request_id":"req-61fa3b99-0582-43a3-a5da-71465e5b56b6",
6
+   "event_type":"port.delete.end",
7
+   "timestamp":"2016-10-03 18:25:18.645306",
8
+   "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
9
+   "_unique_id":"7296f75a9e0f4993bde5cc2b6d5cbe7d",
10
+   "_context_tenant_name":"VOIP",
11
+   "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
12
+   "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
13
+   "payload":{  
14
+      "port_id":"d781dc1e-f940-4de8-ad2f-b1286b75efe0"
15
+   },
16
+   "_context_project_name":"VOIP",
17
+   "_context_read_deleted":"no",
18
+   "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
19
+   "priority":"INFO",
20
+   "_context_is_admin":false,
21
+   "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
22
+   "_context_timestamp":"2016-10-03 18:25:18.562793",
23
+   "_context_user_name":"admin",
24
+   "publisher_id":"network.osctrl-ch2-a02",
25
+   "message_id":"3375a032-8bd2-45b1-b3d8-29bdc5aacbf5"
26
+}

+ 28
- 0
fixtures/event_processor/icehouse/port_delete_with_security_group_attached.json View File

@@ -0,0 +1,28 @@
1
+{  
2
+   "_context_roles":[  
3
+      "Member"
4
+   ],
5
+   "_context_request_id":"req-d9daa316-5075-4702-961a-6a808db40c1e",
6
+   "event_type":"port.delete.end",
7
+   "timestamp":"2016-10-03 17:28:04.004860",
8
+   "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
9
+   "_unique_id":"e7a124d2056c476696a5efada8ddfbf2",
10
+   "_context_tenant_name":"VOIP",
11
+   "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
12
+   "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
13
+   "payload":{  
14
+      "port_id":"57545a0d-ab4e-4a18-bba2-d7ee8bc9c3c1"
15
+   },
16
+   "_context_project_name":"VOIP",
17
+   "_context_read_deleted":"no",
18
+   "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
19
+   "priority":"INFO",
20
+   "_context_is_admin":false,
21
+   "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
22
+   "_context_timestamp":"2016-10-03 17:28:03.916009",
23
+   "_context_user_name":"admin",
24
+   "publisher_id":"network.osctrl-ch2-a02",
25
+   "message_id":"9573b2e3-9f33-499d-ad31-43840f5c1c58"
26
+}
27
+
28
+

+ 26
- 0
fixtures/event_processor/icehouse/security_group_rule_delete_with_ssh_open_to_the_internet.json View File

@@ -0,0 +1,26 @@
1
+{  
2
+   "_context_roles":[  
3
+      "Member"
4
+   ],
5
+   "_context_request_id":"req-f96ea9a5-435e-4177-8e51-ebb60d0fae2a",
6
+   "event_type":"security_group_rule.delete.end",
7
+   "timestamp":"2016-10-03 18:10:59.112712",
8
+   "_context_tenant_id":"ada3b9b0dbac429f9361e803b54f5f32",
9
+   "_unique_id":"eafc9362327442b49d8c03b0e88d0216",
10
+   "_context_tenant_name":"VOIP",
11
+   "_context_user":"bca89c1b248e4aef9c69ece9e744cc54",
12
+   "_context_user_id":"bca89c1b248e4aef9c69ece9e744cc54",
13
+   "payload":{  
14
+      "security_group_rule_id":"bf288dfc-f9cb-446b-bacc-a8de016c9b11"
15
+   },
16
+   "_context_project_name":"VOIP",
17
+   "_context_read_deleted":"no",
18
+   "_context_tenant":"ada3b9b0dbac429f9361e803b54f5f32",
19
+   "priority":"INFO",
20
+   "_context_is_admin":false,
21
+   "_context_project_id":"ada3b9b0dbac429f9361e803b54f5f32",
22
+   "_context_timestamp":"2016-10-03 18:10:59.079179",
23
+   "_context_user_name":"admin",
24
+   "publisher_id":"network.osctrl-ch2-a03",
25
+   "message_id":"e75fb2ee-85bf-44ba-a083-2445eca2ae10"
26
+}

+ 6
- 0
fixtures/viper/test.yml View File

@@ -0,0 +1,6 @@
1
+required_string: "required_string value"
2
+test_alias: "test_alias value"
3
+nested:
4
+  one: "nested.one value"
5
+
6
+

+ 20
- 0
go.mod View File

@@ -0,0 +1,20 @@
1
+module "git.openstack.org/openstack/osel"
2
+
3
+require (
4
+	"github.com/fsnotify/fsnotify" v1.4.7
5
+	"github.com/google/go-querystring" v0.0.0-20170111101155-53e6ce116135
6
+	"github.com/hashicorp/hcl" v0.0.0-20171017181929-23c074d0eceb
7
+	"github.com/magiconair/properties" v1.7.6
8
+	"github.com/mitchellh/mapstructure" v0.0.0-20180220230111-00c29f56e238
9
+	"github.com/nate-johnston/viper" v1.0.1
10
+	"github.com/pelletier/go-toml" v1.1.0
11
+	"github.com/racker/perigee" v0.1.0
12
+	"github.com/rackspace/gophercloud" v1.0.0
13
+	"github.com/spf13/afero" v1.0.2
14
+	"github.com/spf13/cast" v1.2.0
15
+	"github.com/spf13/pflag" v1.0.0
16
+	"github.com/streadway/amqp" v0.0.0-20180131094250-fc7fda2371f5
17
+	"golang.org/x/sys" v0.0.0-20180302081741-dd2ff4accc09
18
+	"golang.org/x/text" v0.0.0-20171214130843-f21a4dfb5e38
19
+	"gopkg.in/yaml.v2" v1.1.1-gopkgin-v2.1.1
20
+)

+ 129
- 0
main.go View File

@@ -0,0 +1,129 @@
1
+package main // import "git.openstack.org/openstack/osel"
2
+
3
+import (
4
+	"log"
5
+	"net/url"
6
+	"os"
7
+	"time"
8
+
9
+	"github.com/fsnotify/fsnotify"
10
+	"github.com/nate-johnston/viper"
11
+)
12
+
13
+// OselVersion is exposed in the logged JSON in the "source_type" field.  This
14
+// will allow us to track the version of the logging specification.
15
+// 1.0: Initial revision
16
+// 1.1: Added qualys_scan_id and qualys_scan_error
17
+const OselVersion = "osel1.1"
18
+
19
+// Debug is a global variable to toggle debug logging
20
+var Debug bool
21
+
22
+// RabbitMQ URI
23
+var rabbitURI string
24
+
25
+func main() {
26
+	// Declare the configuration
27
+	viperConfigs := []ViperConfig{
28
+		ViperConfig{Key: "batch_interval", Description: "Interval of time in minutes for message batching"},
29
+		ViperConfig{Key: "debug", Description: "Output additional messages for debugging"},
30
+		ViperConfig{Key: "rabbit_uri", Description: "AMQP connection uri.  See: https://www.rabbitmq.com/uri-spec.html"},
31
+		ViperConfig{Key: "openstack.identity_endpoint", Description: "Openstack Keystone Endpoint"},
32
+		ViperConfig{Key: "openstack.user", Description: "Openstack user that has at least read only access to all tenants/ports/security groups in the region."},
33
+		ViperConfig{Key: "openstack.password", Description: "Password for the Openstack user"},
34
+		ViperConfig{Key: "openstack.region", Description: "The name of the region running this process"},
35
+		ViperConfig{Key: "qualys.drop6", Description: "Should IPv6 addresses be incorporated in Qualys scans?  true or false."},
36
+		ViperConfig{Key: "qualys.username", Description: "Username for credentials for the Qualys external scanning service"},
37
+		ViperConfig{Key: "qualys.password", Description: "Password for credentials for the Qualys external scanning service"},
38
+		ViperConfig{Key: "qualys.url", Description: "URL for thw Qualys service"},
39
+		ViperConfig{Key: "qualys.proxy_url", Description: "URL for an HTTP proxy that will permit access to the Qualys service"},
40
+		ViperConfig{Key: "syslog_server", Description: "FQDN of the server for events to log to over the network"},
41
+		ViperConfig{Key: "syslog_port", Description: "Port for communication to syslog, defaults to 514"},
42
+		ViperConfig{Key: "syslog_protocol", Description: "tcp or udp, defaults to tcp"},
43
+		ViperConfig{Key: "retry_syslog", Description: "Should the process keep trying if it cannot reach syslog?  true or false."},
44
+	}
45
+	configPath := os.Getenv("EL_CONFIG") //The config path comes from ENV.
46
+	if configPath == "" {
47
+		log.Fatalln("Fatal Error: The Config file was not set to EL_CONFIG.")
48
+	}
49
+	if err := InitViper(configPath, viperConfigs); err != nil {
50
+		log.Fatalf("Fatal Error: (%s) while reading config file %s", err, configPath)
51
+	}
52
+
53
+	// Set defaults
54
+	viper.SetDefault("batch_interval", 60)
55
+	viper.SetDefault("debug", true)
56
+	viper.SetDefault("qualys.drop6", true)
57
+	viper.SetDefault("qualys.url", "https://qualysapi.qualys.com/api/2.0/fo/scan/")
58
+	viper.SetDefault("syslog_port", "514")
59
+	viper.SetDefault("syslog_protocol", "tcp")
60
+
61
+	// Watch for config changes
62
+	viper.WatchConfig()
63
+	viper.OnConfigChange(func(fsnotify.Event) {
64
+		if err := ValidateConfig(viperConfigs); err != nil {
65
+			log.Printf("Fatal Error: %s while refreshing config file %s\n", err, configPath)
66
+		}
67
+	})
68
+
69
+	batchInterval := viper.GetInt("batch_interval")
70
+	Debug = viper.GetBool("debug")
71
+
72
+	// Initialize AMQP
73
+	rabbitURI = viper.GetString("rabbit_uri")
74
+	amqpBus := new(AmqpActions)
75
+	amqpBus.Options = AmqpOptions{
76
+		RabbitURI: rabbitURI,
77
+	}
78
+	amqpIncoming, amqpErrorNotify, err := amqpBus.Connect()
79
+	if err != nil {
80
+		log.Fatalln(err)
81
+	}
82
+
83
+	// Initialize Qualys
84
+	qualysURL, err := url.Parse(viper.GetString("qualys.url"))
85
+	if err != nil {
86
+		log.Fatal(err)
87
+	}
88
+	qualysProxyURL, err := url.Parse(viper.GetString("qualys.proxy_url"))
89
+	if err != nil {
90
+		log.Fatal(err)
91
+	}
92
+	qualys := new(QualysActions)
93
+	qualys.Options = QualysOptions{
94
+		DropIPv6:       viper.GetBool("qualys.drop6"),
95
+		Password:       viper.GetString("qualys.password"),
96
+		ProxyURL:       qualysProxyURL,
97
+		QualysURL:      qualysURL,
98
+		ScanOptionName: viper.GetString("qualys.option"),
99
+		MinRemaining:   viper.GetInt("qualys.min_remaining"),
100
+		UserName:       viper.GetString("qualys.username"),
101
+	}
102
+
103
+	// Initialize OpenStack
104
+	openstack := new(OpenStackActions)
105
+	openstack.Options = OpenStackOptions{
106
+		KeystoneURI: viper.GetString("openstack.identity_endpoint"),
107
+		Password:    viper.GetString("openstack.password"),
108
+		RegionName:  viper.GetString("openstack.region"),
109
+		UserName:    viper.GetString("openstack.user"),
110
+	}
111
+
112
+	// Initialize Syslog
113
+	logger := new(SyslogActions)
114
+	logger.Options = SyslogOptions{
115
+		Host:     viper.GetString("syslog_server"),
116
+		Port:     viper.GetString("syslog_port"),
117
+		Protocol: viper.GetString("syslog_protocol"),
118
+		Retry:    viper.GetBool("retry_syslog"),
119
+	}
120
+	err = logger.Connect()
121
+	if err != nil {
122
+		log.Fatal(err)
123
+	}
124
+
125
+	// run main loop
126
+	batchDuration := time.Duration(batchInterval) * time.Minute
127
+	mainLoop(batchDuration, amqpIncoming, amqpErrorNotify, openstack, logger, qualys)
128
+	defer amqpBus.Close()
129
+}

+ 126
- 0
openstack.go View File

@@ -0,0 +1,126 @@
1
+package main
2
+
3
+/*
4
+
5
+openstack - This file includes all of the logic necessary to interact with
6
+OpenStack.  This is extrapolated out so that an OpenStackActioner
7
+interface can be passed to functions.  Doing this allows testing by mock
8
+classes to be created that can be passed to functions.
9
+
10
+Since this is a wrapper around the gophercloud libraries, this does not need
11
+testing.
12
+
13
+*/
14
+
15
+import (
16
+	"fmt"
17
+	"log"
18
+
19
+	"github.com/rackspace/gophercloud"
20
+	"github.com/rackspace/gophercloud/openstack"
21
+	"github.com/rackspace/gophercloud/openstack/networking/v2/ports"
22
+	"github.com/rackspace/gophercloud/pagination"
23
+)
24
+
25
+// OpenStackActioner is an interface for an OpenStackActions class.
26
+// Having this as an interface allows us to pass in a dummy class for testing
27
+// that just returns mocked data.
28
+type OpenStackActioner interface {
29
+	GetPortList() []OpenStackIPMap
30
+	Connect(string, string) error
31
+}
32
+
33
+// OpenStackActions is a class that handles all interactions directly with
34
+// OpenStack.  See the comment on OpenStackActioner for rationale.
35
+type OpenStackActions struct {
36
+	gopherCloudClient *gophercloud.ProviderClient
37
+	neutronClient     *gophercloud.ServiceClient
38
+	Options           OpenStackOptions
39
+}
40
+
41
+// OpenStackOptions is a class to convey all of the configurable options for the
42
+// OpenStackActions class.
43
+type OpenStackOptions struct {
44
+	KeystoneURI string
45
+	Password    string
46
+	RegionName  string
47
+	TenantID    string
48
+	UserName    string
49
+}
50
+
51
+// OpenStackIPMap is a struct that is used to capture the mapping of IP address
52
+// to security group.  It is what is returned, in array form, from port list.
53
+type OpenStackIPMap struct {
54
+	ipAddress     string
55
+	securityGroup string
56
+}
57
+
58
+// GetPortList is a method that uses GopherCloud to query OpenStack for a
59
+// list of ports, with their associated security group. It returns an array of
60
+// OpenStackIPMap.
61
+func (s OpenStackActions) GetPortList() []OpenStackIPMap {
62
+	// Make port list request to neutron
63
+	var ips []OpenStackIPMap
64
+	portListOpts := ports.ListOpts{
65
+		TenantID: s.Options.TenantID,
66
+	}
67
+	if s.neutronClient == nil {
68
+		log.Println("Error: neutronClient is nil")
69
+	}
70
+	pager := ports.List(s.neutronClient, portListOpts)
71
+
72
+	// Define an anonymous function to be executed on each page's iteration
73
+	pager.EachPage(func(page pagination.Page) (bool, error) {
74
+		portList, err := ports.ExtractPorts(page)
75
+		if err != nil {
76
+			// ignore ?
77
+		}
78
+
79
+		for _, p := range portList {
80
+			// "p" will be a ports.Port
81
+			for _, fixedIP := range p.FixedIPs {
82
+				for _, securityGroup := range p.SecurityGroups {
83
+					ips = append(ips, OpenStackIPMap{
84
+						ipAddress:     fixedIP.IPAddress,
85
+						securityGroup: securityGroup,
86
+					})
87
+				}
88
+			}
89
+		}
90
+		return true, err
91
+	})
92
+	return ips
93
+}
94
+
95
+// Connect is the method that establishes a connection to the OpenStack
96
+// service.
97
+func (s *OpenStackActions) Connect(tenantID string, username string) error {
98
+	var err error
99
+	keystoneOpts := gophercloud.AuthOptions{
100
+		IdentityEndpoint: s.Options.KeystoneURI,
101
+		TenantID:         tenantID,
102
+		Username:         username,
103
+		Password:         s.Options.Password,
104
+		AllowReauth:      true,
105
+	}
106
+
107
+	log.Println(fmt.Sprintf("Connecting to keystone %q in region %q for tenant %q with user %q", s.Options.KeystoneURI,
108
+		s.Options.RegionName, tenantID, username))
109
+	s.gopherCloudClient, err = openstack.AuthenticatedClient(keystoneOpts)
110
+	if err != nil {
111
+		return fmt.Errorf("unable to connect to %s using user %s for tenant %s: %s",
112
+			s.Options.KeystoneURI, s.Options.UserName, s.Options.TenantID, err)
113
+	}
114
+	log.Println("Connected to gophercloud ", s.Options.KeystoneURI)
115
+
116
+	neutronOpts := gophercloud.EndpointOpts{
117
+		Name:   "neutron",
118
+		Region: s.Options.RegionName,
119
+	}
120
+	s.neutronClient, err = openstack.NewNetworkV2(s.gopherCloudClient, neutronOpts)
121
+	if err != nil {
122
+		return fmt.Errorf("unable to connect to neutron using user %s in region %s: %s",
123
+			s.Options.UserName, s.Options.RegionName, err)
124
+	}
125
+	return nil
126
+}

+ 31
- 0
openstack_mock_test.go View File

@@ -0,0 +1,31 @@
1
+package main
2
+
3
+type OpenStackTestActions struct {
4
+	regionName string
5
+	tenantID   string
6
+}
7
+
8
+func (s OpenStackTestActions) Connect(tenantID string, username string) error {
9
+	return nil
10
+}
11
+
12
+func (s OpenStackTestActions) GetPortList() []OpenStackIPMap {
13
+	return []OpenStackIPMap{
14
+		{
15
+			ipAddress:     "10.0.0.1",
16
+			securityGroup: "46d46540-98ac-4c93-ae62-68dddab2282e",
17
+		},
18
+		{
19
+			ipAddress:     "10.0.0.2",
20
+			securityGroup: "groupTwo",
21
+		},
22
+		{
23
+			ipAddress:     "10.0.0.3",
24
+			securityGroup: "46d46540-98ac-4c93-ae62-68dddab2282e",
25
+		},
26
+	}
27
+}
28
+
29
+func connectFakeOpenstack() *OpenStackTestActions {
30
+	return new(OpenStackTestActions)
31
+}

+ 133
- 0
processing.go View File

@@ -0,0 +1,133 @@
1
+package main
2
+
3
+import (
4
+	"fmt"
5
+	"log"
6
+	"net"
7
+	"time"
8
+
9
+	"github.com/streadway/amqp"
10
+)
11
+
12
+func processWaitingEvent(delivery amqp.Delivery, openstackActions OpenStackActioner) (Event, error) {
13
+	// executes when an event is waiting
14
+	event, err := ParseEvent(delivery.Body)
15
+	if err != nil {
16
+		return Event{}, fmt.Errorf("Failed to parse event due to error: %s", err)
17
+	}
18
+	if event.Processor == nil {
19
+		if !Debug {
20
+			return Event{}, nil
21
+		}
22
+		return Event{}, fmt.Errorf("Ignoring event type %s", event.EventData.EventType)
23
+	}
24
+	if Debug {
25
+		log.Printf("Processing event type %s\n", event.EventData.EventType)
26
+	}
27
+	err = event.Processor.FillExtraData(&event, openstackActions)
28
+	if err != nil {
29
+		return Event{}, fmt.Errorf("Error fetching extra data: %s", err)
30
+	}
31
+	return event, nil
32
+}
33
+
34
+func logEvents(events []Event, logger SyslogActioner, qualys QualysActioner) {
35
+	var ipAddresses []string
36
+	var qualysIPAddresses []string
37
+
38
+	if Debug {
39
+		log.Println("Timer Expired")
40
+	}
41
+
42
+	// De-dupe IP addresses and get them into a single struct
43
+	dedupIPAddresses := make(map[string]struct{})
44
+	for _, event := range events {
45
+		for _, IPs := range event.IPs {
46
+			for _, IP := range IPs {
47
+				if _, ok := dedupIPAddresses[IP]; !ok {
48
+					ipAddresses = append(ipAddresses, IP)
49
+				}
50
+				dedupIPAddresses[IP] = struct{}{}
51
+			}
52
+		}
53
+	}
54
+
55
+	// Disregard the scan if no targets have been found
56
+	if len(ipAddresses) == 0 {
57
+		if Debug {
58
+			log.Println("Nothing to scan, skipping...")
59
+		}
60
+		return
61
+	}
62
+
63
+	// Remove IPv6 addresses
64
+	if qualys.DropIPv6() {
65
+		for ipAddressIndex := range ipAddresses {
66
+			testIPAddress := ipAddresses[ipAddressIndex]
67
+			if net.ParseIP(testIPAddress).To4() != nil {
68
+				qualysIPAddresses = append(qualysIPAddresses, testIPAddress)
69
+			} else {
70
+				log.Println("Disregarded IPv6 address", testIPAddress)
71
+			}
72
+		}
73
+	}
74
+
75
+	// Execute Qualys scan
76
+
77
+	log.Println("Qualys Scan Starting")
78
+	scanID, scanError := qualys.InitiateScan(qualysIPAddresses)
79
+	log.Printf("Qualys Scan Complete: scan ID='%s'; scan_error='%v'", scanID, scanError)
80
+
81
+	// Iterate through entries and format the logs
82
+	log.Printf("Processing %d events\n", len(events))
83
+	for _, event := range events {
84
+		event.QualysScanID = scanID
85
+		if scanError != nil {
86
+			event.QualysScanError = scanError.Error()
87
+		}
88
+		event.LogLines, _ = event.Processor.FormatLogs(&event, qualysIPAddresses)
89
+
90
+		// Output the logs
91
+		log.Printf("Processing %d loglines\n", len(event.LogLines))
92
+		for lineToLog := range event.LogLines {
93
+			logger.Info(event.LogLines[lineToLog])
94
+		}
95
+	}
96
+}
97
+
98
+func mainLoop(batchInterval time.Duration, deliveries <-chan amqp.Delivery, amqpNotifyError chan *amqp.Error, openstackActions OpenStackActioner, logger SyslogActioner, qualys QualysActioner) {
99
+	var events []Event
100
+	ticker := time.NewTicker(batchInterval)
101
+	amqpReconnectTimer := time.NewTimer(1)
102
+	for {
103
+		select {
104
+		case e := <-deliveries:
105
+			event, err := processWaitingEvent(e, openstackActions)
106
+			if err != nil {
107
+				log.Printf("Event skipped: %s\n", err)
108
+				continue
109
+			}
110
+			events = append(events, event)
111
+		case <-ticker.C:
112
+			logEvents(events, logger, qualys)
113
+			events = nil
114
+		case err := <-amqpNotifyError:
115
+			// Reinitialize AMQP on connection error
116
+			log.Printf("AMQP connection error: %s\n", err)
117
+			amqpReconnectTimer = time.NewTimer(time.Second * 30)
118
+		case <-amqpReconnectTimer.C:
119
+			var err error
120
+			amqpBus := new(AmqpActions)
121
+			amqpBus.Options = AmqpOptions{
122
+				RabbitURI: rabbitURI,
123
+			}
124
+			deliveries, amqpNotifyError, err = amqpBus.Connect()
125
+			if err != nil {
126
+				log.Printf("AMQP retry connection error: %s\n", err)
127
+				amqpReconnectTimer = time.NewTimer(time.Second * 30)
128
+			} else {
129
+				log.Printf("AMQP reconnected\n")
130
+			}
131
+		}
132
+	}
133
+}

+ 48
- 0
processing_test.go View File

@@ -0,0 +1,48 @@
1
+package main
2
+
3
+import (
4
+	"fmt"
5
+	"os"
6
+	"testing"
7
+
8
+	"github.com/streadway/amqp"
9
+	"github.com/stretchr/testify/assert"
10
+)
11
+
12
+func TestProcessWaitingEvent(t *testing.T) {
13
+	var delivery amqp.Delivery
14
+	openstackActions := connectFakeOpenstack()
15
+
16
+	delivery.Body = []byte(securityGroupRuleCreateWithIcmpAndCider)
17
+	event, err := processWaitingEvent(delivery, openstackActions)
18
+	if err != nil {
19
+		t.Fatal(err)
20
+	}
21
+
22
+	_ = event
23
+}
24
+
25
+func TestLogEvents(t *testing.T) {
26
+	hostName, _ := os.Hostname()
27
+	IPList := []string{"10.0.0.1", "10.0.0.3"}
28
+	logLines := []string{fmt.Sprintf(`{"security_group_rule":{"remote_group_id":null,"direction":"ingress","protocol":"icmp","remote_ip_prefix":"192.168.1.0/24","port_range_max":null,"rule_direction":"","security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e","tenant_id":"ada3b9b0dbac429f9361e803b54f5f32","port_range_min":null,"ethertype":"IPv4","id":"66d7ac79-3551-4436-83c7-103b50760cfb"},"affected_ip_address":"10.0.0.1","change_type":"sg_rule_add","source_type":"osel","source_message_bus":"%s"}`, hostName), fmt.Sprintf(`{"security_group_rule":{"remote_group_id":null,"direction":"ingress","protocol":"icmp","remote_ip_prefix":"192.168.1.0/24","port_range_max":null,"rule_direction":"","security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e","tenant_id":"ada3b9b0dbac429f9361e803b54f5f32","port_range_min":null,"ethertype":"IPv4","id":"66d7ac79-3551-4436-83c7-103b50760cfb"},"affected_ip_address":"10.0.0.3","change_type":"sg_rule_add","source_type":"osel","source_message_bus":"%s"}`, hostName)}
29
+	logger := connectFakeSyslog()
30
+	qualys := connectFakeQualys()
31
+	IPs := make(map[string][]string)
32
+
33
+	IPs["46d46540-98ac-4c93-ae62-68dddab2282e"] = IPList
34
+	fakeEvent := Event{
35
+		RawData:   []byte(securityGroupRuleCreateWithIcmpAndCider),
36
+		LogLines:  logLines,
37
+		Processor: EventSecurityGroupRuleChange{ChangeType: "sg_rule_add"},
38
+		IPs:       IPs,
39
+	}
40
+	events := []Event{fakeEvent}
41
+
42
+	logEvents(events, logger, qualys)
43
+	savedLogs := logger.GetLogs()
44
+	assert.Equal(t, 2, len(savedLogs))
45
+
46
+	logLine1 := fmt.Sprintf(`{"affected_ip_address":"10.0.0.1","change_type":"sg_rule_add","qualys_scan_id":"","qualys_scan_error":"Not scanned by Qualys","security_group_rule":{"remote_group_id":null,"direction":"ingress","protocol":"icmp","remote_ip_prefix":"192.168.1.0/24","port_range_max":null,"rule_direction":"","security_group_id":"46d46540-98ac-4c93-ae62-68dddab2282e","tenant_id":"ada3b9b0dbac429f9361e803b54f5f32","port_range_min":null,"ethertype":"IPv4","id":"66d7ac79-3551-4436-83c7-103b50760cfb"},"source_type":"osel1.1","source_message_bus":"%s"}`, hostName)
47
+	assert.Equal(t, logLine1, savedLogs[0])
48
+}

+ 104
- 0
qualys.go View File

@@ -0,0 +1,104 @@
1
+package main
2
+
3
+/*
4
+
5
+qualys - This file includes all of the logic necessary to interact with the
6
+go-qualys library.  This is extrapolated out so that a QualysInterface
7
+interface can be passed to functions.  Doing this allows testing by mock
8
+classes to be created that can be passed to functions.
9
+
10
+Since this is a wrapper around the go-qualys library, this does not need
11
+testing.
12
+
13
+*/
14
+
15
+import (
16
+	"errors"
17
+	"fmt"
18
+	"log"
19
+	"net/http"
20
+	"net/url"
21
+
22
+	"git.openstack.org/openstack/osel/qualys"
23
+)
24
+
25
+// QualysActioner is an interface for an QualysActions class.  Having
26
+// this as an interface allows us to pass in a dummy class for testing that
27
+// just returns mocked data.
28
+type QualysActioner interface {
29
+	InitiateScan([]string) (string, error)
30
+	DropIPv6() bool
31
+}
32
+
33
+// QualysActions is a class that handles all interactions directly with Qualys.
34
+// See the comment on QualysActioner for rationale.
35
+type QualysActions struct {
36
+	Options QualysOptions
37
+}
38
+
39
+// QualysOptions is a class to convey all of the configurable options for the
40
+// QualysActions class.
41
+type QualysOptions struct {
42
+	DropIPv6       bool
43
+	MinRemaining   int
44
+	ProxyURL       *url.URL
45
+	Password       string
46
+	QualysURL      *url.URL
47
+	ScanOptionName string
48
+	UserName       string
49
+}
50
+
51
+// InitiateScan is the main method for the QualysActioner class, it
52
+// makes a call to the Qualys API to start a scan and harvests a scan ID, and
53
+// an optional error string if there is a problem contacting Qualys.
54
+func (s *QualysActions) InitiateScan(targetIPAddresses []string) (string, error) {
55
+	var err error
56
+
57
+	// create client with proxy so the qualys service can be accessed
58
+	qualysCreds := qualys.Credentials{
59
+		Username: s.Options.UserName,
60
+		Password: s.Options.Password,
61
+	}
62
+	c, err := qualys.NewClient(&http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(s.Options.ProxyURL)}}, &qualysCreds)
63
+	if err != nil {
64
+		return "", err
65
+	}
66
+	c.BaseURL = s.Options.QualysURL
67
+
68
+	// create the options
69
+	opts := qualys.LaunchScanOptions{
70
+		ScanTitle:   "osel",
71
+		ScannerName: "External",
72
+		OptionTitle: s.Options.ScanOptionName,
73
+		IP:          targetIPAddresses,
74
+	}
75
+
76
+	// launch the request
77
+	launchScanResponse, err := c.LaunchScan(&opts)
78
+	if err != nil {
79
+		return "", err
80
+	}
81
+
82
+	// process the request response
83
+	scanID := launchScanResponse.ScanReference
84
+	remainingQualysRequests := launchScanResponse.RateLimitations.Remaining
85
+	allowedQualysRequests := launchScanResponse.RateLimitations.Limit
86
+	if Debug {
87
+		log.Printf("Qualys Rate Limit: %d of %d total requests remaining, concurrency of %d out of %d, %d seconds remaining in limit window and %d seconds until a request can be made again\n",
88
+			remainingQualysRequests, allowedQualysRequests, launchScanResponse.RateLimitations.CurrentConcurrency,
89
+			launchScanResponse.RateLimitations.ConcurrencyLimit, launchScanResponse.RateLimitations.LimitWindow, launchScanResponse.RateLimitations.WaitingPeriod)
90
+	}
91
+	if launchScanResponse.Text != "" {
92
+		err = errors.New(launchScanResponse.Text)
93
+	}
94
+	if remainingQualysRequests <= s.Options.MinRemaining {
95
+		err = fmt.Errorf("halting Qualys processing!  Only %d Qualys calls remain out of a total of %d.  Waiting for %d seconds before resuming", remainingQualysRequests,
96
+			allowedQualysRequests, launchScanResponse.RateLimitations.LimitWindow)
97
+	}
98
+	return scanID, err
99
+}
100
+
101
+// DropIPv6 is an accessor method to allow other code to make decisions based on whether this flag is enabled.
102
+func (s *QualysActions) DropIPv6() bool {
103
+	return s.Options.DropIPv6
104
+}

+ 170
- 0
qualys/assets.go View File

@@ -0,0 +1,170 @@
1
+package qualys
2
+
3
+import (
4
+	"bytes"
5
+	"fmt"
6
+	"net"
7
+	"strings"
8
+)
9
+
10
+const (
11
+	assetsBasePath = "asset"
12
+	groupsBasePath = "group"
13
+)
14
+
15
+// AssetsService is an interface for interfacing with the Assets
16
+// endpoints of the Qualys API
17
+type AssetsService interface {
18
+	ListAssetGroups(*ListAssetGroupOptions) ([]AssetGroup, *Response, error)
19
+	GetAssetGroupByID(groupID string) (*AssetGroup, *Response, error)
20
+	AddIPsToGroup(*AddIPsToGroupOptions) (*Response, error)
21
+}
22
+
23
+// AssetsServiceOp handles communication with the asset related methods of the
24
+// Qualys API.
25
+type AssetsServiceOp struct {
26
+	client *Client
27
+}
28
+
29
+var _ AssetsService = &AssetsServiceOp{}
30
+
31
+// AssetGroup represents a Qualys HostGroup
32
+type AssetGroup struct {
33
+	ID          string        `xml:"ID"`
34
+	Title       string        `xml:"TITLE"`
35
+	OwnerUserID string        `xml:"OWNER_USER_ID"`
36
+	OwnerUnitID string        `xml:"OWNER_UNIT_ID"`
37
+	IPs         AssetGroupIPs `xml:"IP_SET"`
38
+}
39
+
40
+// AssetGroupIPs represents one or more IP addresses assigned to the AssetGroup
41
+type AssetGroupIPs struct {
42
+	IPs      []string `xml:"IP"`
43
+	IPRanges []string `xml:"IP_RANGE"`
44
+}
45
+
46
+type ipRange struct {
47
+	Min net.IP
48
+	Max net.IP
49
+}
50
+
51
+func newIPRange(rangeString string) *ipRange {
52
+	var r = strings.Split(rangeString, "-")
53
+	return &ipRange{Min: net.ParseIP(r[0]), Max: net.ParseIP(r[1])}
54
+}
55
+
56
+func (ip *ipRange) Contains(ipString string) bool {
57
+	var myIP = net.ParseIP(ipString)
58
+	if bytes.Compare(myIP, ip.Min) >= 0 && bytes.Compare(myIP, ip.Max) <= 0 {
59
+		return true
60
+	}
61
+	return false
62
+}
63
+
64
+// ContainsIP returns true when the AssetGroupIPs matches the provided IP
65
+func (agp *AssetGroupIPs) ContainsIP(ip string) bool {
66
+	if containsString(agp.IPs, ip) {
67
+		return true
68
+	}
69
+	if agp.IPRanges != nil && len(agp.IPRanges) > 0 {
70
+		for _, ipRange := range agp.IPRanges {
71
+			if newIPRange(ipRange).Contains(ip) {
72
+				return true
73
+			}
74
+		}
75
+	}
76
+	return false
77
+}
78
+
79
+// ContainsIP returns true when the AssetGroup has any assets matching the provided IP
80
+func (ag *AssetGroup) ContainsIP(ip string) bool {
81
+	return ag.IPs.ContainsIP(ip)
82
+}
83
+
84
+type assetGroupsRoot struct {
85
+	AssetGroups []AssetGroup `xml:"RESPONSE>ASSET_GROUP_LIST>ASSET_GROUP"`
86
+}
87
+
88
+// AssetGroupUpdateRequest represents a request to update a group
89
+type AssetGroupUpdateRequest struct {
90
+}
91
+
92
+// ListAssetGroupOptions represents the AssetGroup retrieval options
93
+type ListAssetGroupOptions struct {
94
+	Ids    []string `url:"ids,comma,omitempty"`
95
+	Action string   `url:"action,omitempty"`
96
+}
97
+
98
+// AddIPsToGroupOptions represents the update request for an AssetGroup
99
+type AddIPsToGroupOptions struct {
100
+	GroupID string   `url:"id,omitempty"`
101
+	IPs     []string `url:"add_ips,comma,omitempty"`
102
+}
103
+
104
+// ListAssetGroups retrieves a list of AssetGroups
105
+func (s *AssetsServiceOp) ListAssetGroups(opt *ListAssetGroupOptions) ([]AssetGroup, *Response, error) {
106
+	return s.listAssetGroups(opt)
107
+}
108
+
109
+// GetAssetGroupByID retrieves an AssetGroup by id.
110
+func (s *AssetsServiceOp) GetAssetGroupByID(groupID string) (*AssetGroup, *Response, error) {
111
+	return s.getAssetGroup(groupID)
112
+}
113
+
114
+// AddIPsToGroup adds the IPs in AddIPsToGroupOptions to the AssetGroup
115
+func (s *AssetsServiceOp) AddIPsToGroup(opt *AddIPsToGroupOptions) (*Response, error) {
116
+	return s.addIPsToGroup(opt)
117
+}
118
+
119
+func (s *AssetsServiceOp) getAssetGroup(groupID string) (*AssetGroup, *Response, error) {
120
+	opts := ListAssetGroupOptions{Ids: []string{groupID}}
121
+	groups, response, err := s.listAssetGroups(&opts)
122
+	if err != nil {
123
+		return nil, response, err
124
+	}
125
+	if len(groups) == 0 {
126
+		return nil, response, nil
127
+	}
128
+	return &groups[0], response, nil
129
+}
130
+
131
+func (s *AssetsServiceOp) addIPsToGroup(opt *AddIPsToGroupOptions) (*Response, error) {
132
+	path := fmt.Sprintf("%s/%s/?action=edit", assetsBasePath, groupsBasePath)
133
+
134
+	req, err := s.client.NewRequest("POST", path, opt)
135
+	if err != nil {
136
+		return nil, err
137
+	}
138
+
139
+	resp, err := s.client.MakeRequest(req, nil)
140
+	if err != nil {
141
+		return resp, err
142
+	}
143
+	return resp, err
144
+}
145
+
146
+// Helper method for listing asset groups
147
+func (s *AssetsServiceOp) listAssetGroups(listOpt *ListAssetGroupOptions) ([]AssetGroup, *Response, error) {
148
+	path := fmt.Sprintf("%s/%s/", assetsBasePath, groupsBasePath)
149
+	if listOpt == nil {
150
+		listOpt = &ListAssetGroupOptions{}
151
+	}
152
+	listOpt.Action = "list"
153
+	path, err := addURLParameters(path, listOpt)
154
+
155
+	if err != nil {
156
+		return nil, nil, err
157
+	}
158
+
159
+	req, err := s.client.NewRequest("GET", path, nil)
160
+	if err != nil {
161
+		return nil, nil, err
162
+	}
163
+
164
+	root := new(assetGroupsRoot)
165
+	resp, err := s.client.MakeRequest(req, root)
166
+	if err != nil {
167
+		return nil, resp, err
168
+	}
169
+	return root.AssetGroups, resp, err
170
+}

+ 251
- 0
qualys/assets_test.go View File

@@ -0,0 +1,251 @@
1
+package qualys
2
+
3
+import (
4
+	"fmt"
5
+	"net/http"
6
+	"reflect"
7
+	"testing"
8
+)
9
+
10
+func TestListAssetGroups(t *testing.T) {
11
+
12
+	cases := []struct {
13
+		name     string
14
+		response string
15
+		expected []AssetGroup
16
+		opts     *ListAssetGroupOptions
17
+		isErr    bool
18
+	}{
19
+		{
20
+			name:     "ListAssetGroups - single item, without list options",
21
+			response: assetGroupsXMLSingleGroup,
22
+			expected: []AssetGroup{
23
+				{
24
+					ID:    "1759735",
25
+					Title: "AG - Elastic Cloud Dynamic Perimeter",
26
+					IPs: AssetGroupIPs{
27
+						IPs:      []string{"10.1.1.1", "10.10.10.11"},
28
+						IPRanges: nil,
29
+					},
30
+				},
31
+			},
32
+			opts: nil,
33
+		},
34
+		{
35
+			name:     "ListAssetGroups - single item, with list options",
36
+			response: assetGroupsXMLSingleGroup,
37
+			expected: []AssetGroup{
38
+				{
39
+					ID:    "1759735",
40
+					Title: "AG - Elastic Cloud Dynamic Perimeter",
41
+					IPs: AssetGroupIPs{
42
+						IPs:      []string{"10.1.1.1", "10.10.10.11"},
43
+						IPRanges: nil,
44
+					},
45
+				},
46
+			},
47
+			opts: &ListAssetGroupOptions{Ids: []string{}},
48
+		},
49
+		{
50
+			name:     "ListAssetGroups - multi item",
51
+			response: assetGroupsXMLMultiGroups,
52
+			expected: []AssetGroup{
53
+				{ID: "1759734", Title: "AG - New"},
54
+				{ID: "1759735", Title: "AG - Elastic Cloud Dynamic Perimeter",
55
+					IPs: AssetGroupIPs{
56
+						IPs:      []string{"10.10.10.14"},
57
+						IPRanges: []string{"10.10.10.3-10.10.10.6"},
58
+					},
59
+				},
60
+			},
61
+			opts: &ListAssetGroupOptions{Ids: []string{"1", "2"}},
62
+		},
63
+	}
64
+
65
+	for _, c := range cases {
66
+		setup()
67
+		defer teardown()
68
+		mux.HandleFunc("/asset/group/", func(w http.ResponseWriter, r *http.Request) {
69
+			testMethod(t, r, "GET")
70
+			fmt.Fprint(w, c.response)
71
+		})
72
+
73
+		assetGroups, _, err := client.Assets.ListAssetGroups(c.opts)
74
+		if err != nil {
75
+			t.Errorf("Assets.ListAssetGroups returned error: %v", err)
76
+		}
77
+
78
+		if !reflect.DeepEqual(assetGroups, c.expected) {
79
+			t.Errorf("Assets.ListAssetGroups case: %s returned %+v, expected %+v", c.name, assetGroups, c.expected)
80
+		}
81
+	}
82
+}
83
+
84
+func TestGetAssetGroupByID(t *testing.T) {
85
+	setup()
86
+	defer teardown()
87
+
88
+	mux.HandleFunc("/asset/group/", func(w http.ResponseWriter, r *http.Request) {
89
+		testMethod(t, r, "GET")
90
+		fmt.Fprint(w, assetGroupsXMLSingleGroup)
91
+	})
92
+
93
+	groupID := "1759735"
94
+
95
+	assetGroup, _, err := client.Assets.GetAssetGroupByID(groupID)
96
+	if err != nil {
97
+		t.Errorf("Assets.GetAssetGroupByID(%s) returned error: %v", groupID, err)
98
+	}
99
+
100
+	expected := &AssetGroup{
101
+		ID:    "1759735",
102
+		Title: "AG - Elastic Cloud Dynamic Perimeter",
103
+		IPs: AssetGroupIPs{
104
+			IPs:      []string{"10.1.1.1", "10.10.10.11"},
105
+			IPRanges: nil,
106
+		},
107
+	}
108
+	if !reflect.DeepEqual(assetGroup, expected) {
109
+		t.Errorf("Assets.GetAssetGroupByID(%s) returned %+v, expected %+v", groupID, assetGroup, expected)
110
+	}
111
+}
112
+
113
+func TestAddIPsToGroup(t *testing.T) {
114
+	setup()
115
+	defer teardown()
116
+
117
+	groupID := "1759735"
118
+	ip := "10.10.10.10"
119
+
120
+	mux.HandleFunc("/asset/group/", func(w http.ResponseWriter, r *http.Request) {
121
+		testMethod(t, r, "POST")
122
+		if r.FormValue("add_ips") != ip {
123
+			t.Errorf("Request form data did not include the correct IP")
124
+		}
125
+		if r.FormValue("id") != groupID {
126
+			t.Errorf("Request form data did not include the correct asset group ID")
127
+		}
128
+		fmt.Fprint(w, assetGroupsAddIPsResponse)
129
+	})
130
+	opts := &AddIPsToGroupOptions{
131
+		GroupID: groupID,
132
+		IPs:     []string{ip},
133
+	}
134
+
135
+	_, err := client.Assets.AddIPsToGroup(opts)
136
+	if err != nil {
137
+		t.Errorf("Assets.AddIPsToGroup returned error: %v", err)
138
+	}
139
+}
140
+
141
+func TestAssetGroupContainsIP(t *testing.T) {
142
+	cases := []struct {
143
+		name     string
144
+		ip       string
145
+		group    *AssetGroup
146
+		expected bool
147
+	}{
148
+		{
149
+			name:     "AssetGroup.ContainsIP - nil",
150
+			ip:       "10.1.1.1",
151
+			group:    &AssetGroup{ID: "1759735", Title: "AG - Elastic Cloud Dynamic Perimeter"},
152
+			expected: false,
153
+		},
154
+		{
155
+			name: "AssetGroup.ContainsIP - empty",
156
+			ip:   "10.1.1.1",
157
+			group: &AssetGroup{
158
+				ID:    "1759735",
159
+				Title: "AG - Elastic Cloud Dynamic Perimeter",
160
+				IPs:   AssetGroupIPs{}},
161
+			expected: false,
162
+		},
163
+		{
164
+			name: "AssetGroup.ContainsIP - single item list",
165
+			ip:   "10.1.1.1",
166
+			group: &AssetGroup{
167
+				ID:    "1759735",
168
+				Title: "AG - Elastic Cloud Dynamic Perimeter",
169
+				IPs: AssetGroupIPs{
170
+					IPs:      []string{"10.1.1.1"},
171
+					IPRanges: []string{},
172
+				},
173
+			},
174
+			expected: true,
175
+		},
176
+		{
177
+			name: "AssetGroup.ContainsIP - multi item list",
178
+			ip:   "10.1.1.1",
179
+			group: &AssetGroup{
180
+				ID:    "1759735",
181
+				Title: "AG - Elastic Cloud Dynamic Perimeter",
182
+				IPs: AssetGroupIPs{
183
+					IPs:      []string{"10.1.1.1"},
184
+					IPRanges: []string{"10.10.1.1-10.10.10.10"},
185
+				},
186
+			},
187
+			expected: true,
188
+		},
189
+	}
190
+	for _, c := range cases {
191
+		contains := c.group.ContainsIP(c.ip)
192
+		if contains != c.expected {
193
+			t.Errorf("%s - AssetGroup.ContainsIP(%s) returned %v, expected %v", c.name, c.ip, contains, c.expected)
194
+		}
195
+	}
196
+}
197
+
198
+func TestAssetGroupIPsContainsIP(t *testing.T) {
199
+	group := AssetGroupIPs{IPs: []string{"10.0.1.1"}, IPRanges: []string{"10.10.10.3-10.10.10.6"}}
200
+
201
+	cases := []struct {
202
+		name     string
203
+		ip       string
204
+		group    AssetGroupIPs
205
+		expected bool
206
+	}{
207
+		{
208
+			name:     "AssetGroupIPs.ContainsIP - IP value match",
209
+			ip:       "10.0.1.1",
210
+			group:    group,
211
+			expected: true,
212
+		},
213
+		{
214
+			name:     "AssetGroupIPs.ContainsIP - IP value no match",
215
+			ip:       "192.0.1.1",
216
+			group:    group,
217
+			expected: false,
218
+		},
219
+		{
220
+			name:     "AssetGroupIPs.ContainsIP - IP Range value match",
221
+			ip:       "10.10.10.4",
222
+			group:    group,
223
+			expected: true,
224
+		},
225
+		{
226
+			name:     "AssetGroupIPs.ContainsIP - IP Range value no match",
227
+			ip:       "10.10.10.1",
228
+			group:    group,
229
+			expected: false,
230
+		},
231
+		{
232
+			name:     "AssetGroupIPs.ContainsIP - IP Range value match",
233
+			ip:       "10.10.0.4",
234
+			group:    AssetGroupIPs{IPs: []string{"10.0.1.1"}, IPRanges: []string{"10.10.0.0-10.10.10.6"}},
235
+			expected: true,
236
+		},
237
+		{
238
+			name:     "AssetGroupIPs.ContainsIP - IP Range value no match",
239
+			ip:       "10.10.0.4",
240
+			group:    AssetGroupIPs{IPs: []string{"10.0.1.1"}, IPRanges: []string{"10.10.1.3-10.10.10.6"}},
241
+			expected: false,
242
+		},
243
+	}
244
+
245
+	for _, c := range cases {
246
+		contains := c.group.ContainsIP(c.ip)
247
+		if contains != c.expected {
248
+			t.Errorf("%s - AssetGroupIPs.ContainsIP(%s) returned %v, expected %v", c.name, c.ip, contains, c.expected)
249
+		}
250
+	}
251
+}

+ 68
- 0
qualys/assets_xml_fixtures_test.go View File

@@ -0,0 +1,68 @@
1
+package qualys
2
+
3
+const (
4
+	assetGroupsXMLSingleGroup = `
5
+		<?xml version="1.0" encoding="UTF-8" ?>
6
+		<!DOCTYPE ASSET_GROUP_LIST_OUTPUT SYSTEM "https://qualysapi.qualys.com/api/2.0/fo/asset/group/asset_group_list_output.dtd">
7
+		<ASSET_GROUP_LIST_OUTPUT>
8
+			<RESPONSE>
9
+				<DATETIME>2016-10-05T19:00:22Z</DATETIME>
10
+				<ASSET_GROUP_LIST>
11
+					<ASSET_GROUP>
12
+						<ID>1759735</ID>
13
+						<TITLE><![CDATA[AG - Elastic Cloud Dynamic Perimeter]]></TITLE>
14
+						<IP_SET>
15
+							<IP>10.1.1.1</IP>
16
+							<IP>10.10.10.11</IP>
17
+						</IP_SET>
18
+					</ASSET_GROUP>
19
+				</ASSET_GROUP_LIST>
20
+			</RESPONSE>
21
+		</ASSET_GROUP_LIST_OUTPUT>
22
+		<!-- CONFIDENTIAL AND PROPRIETARY INFORMATION. Qualys provides the QualysGuard Service "As Is," without any warranty of any kind. Qualys makes no warranty that the information contained in this report is complete or error-free. Copyright 2016, Qualys, Inc. //-->
23
+	`
24
+
25
+	assetGroupsXMLMultiGroups = `
26
+		<?xml version="1.0" encoding="UTF-8" ?>
27
+		<!DOCTYPE ASSET_GROUP_LIST_OUTPUT SYSTEM "https://qualysapi.qualys.com/api/2.0/fo/asset/group/asset_group_list_output.dtd">
28
+		<ASSET_GROUP_LIST_OUTPUT>
29
+			<RESPONSE>
30
+				<DATETIME>2016-10-05T19:00:22Z</DATETIME>
31
+				<ASSET_GROUP_LIST>
32
+					<ASSET_GROUP>
33
+						<ID>1759734</ID>
34
+						<TITLE><![CDATA[AG - New]]></TITLE>
35
+						<DEFAULT_APPLIANCE_ID>105102</DEFAULT_APPLIANCE_ID>
36
+						<APPLIANCE_IDS>105102</APPLIANCE_IDS>
37
+					</ASSET_GROUP>
38
+					<ASSET_GROUP>
39
+						<ID>1759735</ID>
40
+						<TITLE><![CDATA[AG - Elastic Cloud Dynamic Perimeter]]></TITLE>
41
+						<IP_SET>
42
+							<IP_RANGE>10.10.10.3-10.10.10.6</IP_RANGE>
43
+							<IP>10.10.10.14</IP>
44
+						</IP_SET>
45
+					</ASSET_GROUP>
46
+				</ASSET_GROUP_LIST>
47
+			</RESPONSE>
48
+		</ASSET_GROUP_LIST_OUTPUT>
49
+		<!-- CONFIDENTIAL AND PROPRIETARY INFORMATION. Qualys provides the QualysGuard Service "As Is," without any warranty of any kind. Qualys makes no warranty that the information contained in this report is complete or error-free. Copyright 2016, Qualys, Inc. //-->
50
+	`
51
+
52
+	assetGroupsAddIPsResponse = `
53
+		<?xml version="1.0" encoding="UTF-8" ?>
54
+		<!DOCTYPE SIMPLE_RETURN SYSTEM "https://qualysapi.qualys.com/api/2.0/simple_return.dtd">
55
+		<SIMPLE_RETURN>
56
+			<RESPONSE>
57
+				<DATETIME>2016-10-12T14:16:22Z</DATETIME>
58
+				<TEXT>Asset Group Updated Successfully</TEXT>
59
+				<ITEM_LIST>
60
+					<ITEM>
61
+						<KEY>ID</KEY>
62
+						<VALUE>1759735</VALUE>
63
+					</ITEM>
64
+				</ITEM_LIST>
65
+			</RESPONSE>
66
+		</SIMPLE_RETURN>
67
+		`
68
+)

+ 130
- 0
qualys/client.go View File

@@ -0,0 +1,130 @@
1
+package qualys
2
+
3
+import (
4
+	"fmt"
5
+	"net/http"
6
+	"net/url"
7
+)
8
+
9
+const (
10
+	libraryVersion = "0.1.0"
11
+	defaultBaseURL = "https://qualysapi.qualys.com/api/2.0/fo/"
12
+	userAgent      = "go-qualys"
13
+	mediaType      = "application/xml"
14
+
15
+	headerUserAgent               = "X-Requested-With"
16
+	headerRateLimit               = "X-RateLimit-Limit"
17
+	headerRateLimitWindow         = "X-RateLimit-Window-Sec"
18
+	headerRateRemaining           = "X-RateLimit-Remaining"
19
+	headerRateLimitWait           = "X-RateLimit-ToWait-Sec"
20
+	headerConcurrencyLimit        = "X-Concurrency-Limit-Limit"
21
+	headerConcurrencyLimitRunning = "X-Concurrency-Limit-Running"
22
+)
23
+
24
+// Client for Qualys API
25
+type Client struct {
26
+	// Credentials used to authenticate to the Qualys API
27
+	Credentials *Credentials
28
+
29
+	// HTTP client used to communicate with the Qualys API
30
+	client *http.Client
31
+
32
+	// Base URL for API requests.
33
+	BaseURL *url.URL
34
+
35
+	// User agent for client
36
+	UserAgent string
37
+
38
+	// Rate contains the current rate limit for the client as determined by the most recent
39
+	// API call.
40
+	Rate Rate
41
+
42
+	// Services used for communicating with the API
43
+	Assets AssetsService
44
+}
45
+
46
+// Rate contains the rate limit for the current client.
47
+type Rate struct {
48
+	// The number of requests within the limit window of seconds the client is allowed
49
+	Limit int
50
+
51
+	// The number of seconds remaining in the limit window
52
+	LimitWindow int
53
+
54
+	// The number of remaining requests the client can make during the limit window period
55
+	Remaining int
56
+
57
+	// The number of seconds to wait before requests can be made again -- headerRateLimitWait
58
+	WaitingPeriod int
59
+
60
+	// The number of API calls permitted to be executed concurrrently
61
+	ConcurrencyLimit int
62
+
63
+	// The number of API calls currently running
64
+	CurrentConcurrency int
65
+}
66
+
67
+// Credentials holds the credentials and endpoint for the Qualys Client
68
+type Credentials struct {
69
+	Username string
70
+	Password string
71
+}
72
+
73
+// ClientOpt are options for New.
74
+type ClientOpt func(*Client) error
75
+
76
+// New returns a new API client instance.
77
+func New(httpClient *http.Client, credentials *Credentials, opts ...ClientOpt) (*Client, error) {
78
+	c, err := NewClient(httpClient, credentials)
79
+	if err != nil {
80
+		return nil, err
81
+	}
82
+	for _, opt := range opts {
83
+		if err := opt(c); err != nil {
84
+			return nil, err
85
+		}
86
+	}
87
+
88
+	return c, nil
89
+}
90
+
91
+// NewClient returns a new Qualys API client.
92
+func NewClient(httpClient *http.Client, credentials *Credentials) (*Client, error) {
93
+	if httpClient == nil {
94
+		httpClient = http.DefaultClient
95
+	}
96
+	if credentials == nil || credentials.Username == "" || credentials.Password == "" {
97
+		return nil, fmt.Errorf("Credentials must be provided")
98
+	}
99
+
100
+	baseURL, err := url.Parse(defaultBaseURL)
101
+	if err != nil {
102
+		return nil, err
103
+	}
104
+
105
+	c := &Client{client: httpClient, Credentials: credentials, BaseURL: baseURL, UserAgent: userAgent}
106
+
107
+	c.Assets = &AssetsServiceOp{client: c}
108
+	return c, nil
109
+}
110
+
111
+// SetBaseURL is a client option for setting the base URL.
112
+func SetBaseURL(bu string) ClientOpt {
113
+	return func(c *Client) error {
114
+		u, err := url.Parse(bu)
115
+		if err != nil {
116
+			return err
117
+		}
118
+
119
+		c.BaseURL = u
120
+		return nil
121
+	}
122
+}
123
+
124
+// SetUserAgent is a client option for setting the user agent.
125
+func SetUserAgent(ua string) ClientOpt {
126
+	return func(c *Client) error {
127
+		c.UserAgent = fmt.Sprintf("%s+%s", ua, c.UserAgent)
128
+		return nil
129
+	}
130
+}

+ 207
- 0
qualys/client_test.go View File

@@ -0,0 +1,207 @@
1
+package qualys
2
+
3
+import (
4
+	"bytes"
5
+	"fmt"
6
+	"net/http"
7
+	"net/http/httptest"
8
+	"net/url"