Boxing Octopus

infosec / Software Development

How to Make Nmap Output Prettier: Part 1 – nmap2json, yq, and xmllint

We all love Nmap. It’s an indispensable part of any red teamer’s arsenal, but we all know that. We all ALSO know that Nmap supports a few output formats aside from it’s standard output for easy reporting/record-keeping (“greppable”, XML, and “Script Kiddie”). These output formats (aside from “script kiddie”, of course) are meant to make the data easier to consume.

Now, maybe it’s just me, but all these formats are kind of lacking for a few reasons:

  1. These formats ABSOLUTELY SUCK as human-readable formats: The XML that’s outputted by Nmap is flat and ugly (as if XML were pretty in the first place), the greppable output is only really useful if you’re viewing it in a full-screen terminal session, and I won’t even give “skript kiddie” a mention, because…well…really??
  2. They’re archaic: Unless you’re some poor, unfortunate Java developer stuck having to use XML for ALL THE MANIFESTS, you’ve long since moved on from viewing XML as a useful (or desirable) data format.

With all this said, there are a few options for making Nmap data a bit easier to read. There are web-based tools like Dradis, or WebMap, but Dradis is heavy and quite franky, I’m not a fan of how it imports Nmap data into an existing reporting project. WebMap is, in my opinion a good tool, and I’ll likely cover setting it up some point in the future, but for the sake of this article, I wanted to focus more on text-based data.

Overall, in my opinion, a decent data format should have a good balance of human AND machine readability. And the only three formats I can think of which provide anything resembling that balance are JSON, YAML, and TOML.

Now, TOML is great for config files, but not so much for contextual data like you would see in an Nmap output, so that leaves JSON and YAML.

Thanks to a Ruby tool called nmap2json, converting Nmap XML to json is a snap. Installing it is simple as well:

# gem install nmap2json

Once the tool is installed, you can point it at any Nmap XML file and fire:

# nmap2json convert my_scan_data.xml

Buuut…wait…the output we get is, well…

{"nmaprun":{"scanner":"nmap","args":"nmap -sC -sV -T4 -v -oA my_scan_data -p1-65535 10.10.10.105","start":"1538194971","startstr":"Sat Sep 29 00:22:51 2018","version":"7.70","xmloutputversion":"1.04","scaninfo":{"type":"syn","protocol":"tcp","numservices":"65535","services":"1-65535"},"verbose":{"level":"1"},"debugging":{"level":"0"},"taskbegin":[{"task":"NSE","time":"1538194971"},{"task":"NSE","time":"1538194971"},{"task":"Ping Scan","time":"1538194971"},{"task":"Parallel DNS resolution of 1 host.","time":"1538194971"},{"task":"SYN Stealth Scan","time":"1538194971"},{"task":"Service scan","time":"1538195425"},{"task":"NSE","time":"1538195432"},{"task":"NSE","time":"1538195436"},{"task":"NSE","time":"1538195436"},{"task":"NSE","time":"1538195436"}],"taskend":[{"task":"NSE","time":"1538194971"},{"task":"NSE","time":"1538194971"},{"task":"Ping Scan","time":"1538194971","extrainfo":"1 total hosts"},{"task":"Parallel DNS resolution of 1 host.","time":"1538194971"},{"task":"SYN Stealth Scan","time":"1538195425","extrainfo":"65535 total ports"},{"task":"Service scan","time":"1538195432","extrainfo":"2 services on 1 host"},{"task":"NSE","time":"1538195436"},{"task":"NSE","time":"1538195436"},{"task":"NSE","time":"1538195436"},{"task":"NSE","time":"1538195436"}],"taskprogress":[{"task":"SYN Stealth Scan","time":"1538195002","percent":"9.86","remaining":"284","etc":"1538195285"},{"task":"SYN Stealth Scan","time":"1538195032","percent":"16.40","remaining":"311","etc":"1538195343"},{"task":"SYN Stealth Scan","time":"1538195062","percent":"27.13","remaining":"245","etc":"1538195306"},{"task":"SYN Stealth Scan","time":"1538195092","percent":"43.68","remaining":"157","etc":"1538195248"},{"task":"SYN Stealth Scan","time":"1538195122","percent":"46.34","remaining":"175","etc":"1538195297"},{"task":"SYN Stealth Scan","time":"1538195161","percent":"49.79","remaining":"192","etc":"1538195353"},{"task":"SYN Stealth Scan","time":"1538195230","percent":"55.09","remaining":"212","etc":"1538195441"},{"task":"SYN Stealth Scan","time":"1538195323","percent":"65.60","remaining":"185","etc":"1538195508"},{"task":"SYN Stealth Scan","time":"1538195353","percent":"77.48","remaining":"112","etc":"1538195464"},{"task":"SYN Stealth Scan","time":"1538195383","percent":"90.18","remaining":"45","etc":"1538195428"}],"host":{"starttime":"1538194971","endtime":"1538195436","status":{"state":"up","reason":"echo-reply","reason_ttl":"63"},"address":{"addr":"10.10.10.105","addrtype":"ipv4"},"hostnames":"\n","ports":{"extraports":{"state":"closed","count":"65532","extrareasons":{"reason":"resets","count":"65532"}},"port":[{"protocol":"tcp","portid":"21","state":{"state":"filtered","reason":"no-response","reason_ttl":"0"},"service":{"name":"ftp","method":"table","conf":"3"}},{"protocol":"tcp","portid":"22","state":{"state":"open","reason":"syn-ack","reason_ttl":"63"},"service":{"name":"ssh","product":"OpenSSH","version":"7.6p1 Ubuntu 4","extrainfo":"Ubuntu Linux; protocol 2.0","ostype":"Linux","method":"probed","conf":"10","cpe":["cpe:/a:openbsd:openssh:7.6p1","cpe:/o:linux:linux_kernel"]},"script":{"id":"ssh-hostkey","output":"\n  2048 15:a4:28:77:ee:13:07:06:34:09:86:fd:6f:cc:4c:e2 (RSA)\n  256 37:be:de:07:0f:10:bb:2b:b5:85:f7:9d:92:5e:83:25 (ECDSA)\n  256 89:5a:ee:1c:22:02:d2:13:40:f2:45:2e:70:45:b0:c4 (ED25519)","table":[{"elem":["15a42877ee130706340986fd6fcc4ce2","AAAAB3NzaC1yc2EAAAADAQABAAABAQDI2Jfx6VeMU2wFDys5YoSIVCu4U626/VDawUrXKa5SR+D8HaNvt6QFECtQumoFcYzxD7Jnd3PKw/dXTXvePTPnolDUNV3oimX8gEI3iY157v5scgrOKFjw39cTMuTfLc7/rM8e2TOeziN4yzzLfWAiTbe4wfiDe8cea7zJ1RFwvgGc398xiOA8bo1nwMD0wUkduJhtH4V98LpJZOVB4tMmtCdyb1T+e3HIR/1WbmMBLs0e6Cc/rf+K8vgqu6Tu/o4o8/TZ9aH9K5xoDRUXjU2R1w/Bi0HvYYHFRf664/NG9WcK/R0VlV6j92DOYL9wdUYwANyQPc4YCDfyuM6F6Bbd","2048.0","ssh-rsa"]},{"elem":["37bede070f10bb2bb585f79d925e8325","AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJToeoLQWJwkfcWBimMzO4E6BKOaHbTkWIk1uHoniOdaUaDL5C6MO2NeYYSaru/ikAYSHPU83p1p6hNcOJVy+OY=","256","ecdsa-sha2-nistp256"]},{"elem":["895aee1c2202d21340f2452e7045b0c4","AAAAC3NzaC1lZDI1NTE5AAAAIIN0vm7BcvmBgddJb7k1W7qUkBgn2n0T1bdOU6GV1JB8","256","ssh-ed25519"]}]}},{"protocol":"tcp","portid":"80","state":{"state":"open","reason":"syn-ack","reason_ttl":"62"},"service":{"name":"http","product":"Apache httpd","version":"2.4.18","extrainfo":"(Ubuntu)","method":"probed","conf":"10","cpe":"cpe:/a:apache:http_server:2.4.18"},"script":[{"id":"http-cookie-flags","output":"\n  /: \n    PHPSESSID: \n      httponly flag not set","table":{"key":"/","table":{"key":"PHPSESSID","elem":"httponly flag not set"}}},{"id":"http-methods","output":"\n  Supported Methods: GET HEAD POST OPTIONS","table":{"key":"Supported Methods","elem":["GET","HEAD","POST","OPTIONS"]}},{"id":"http-server-header","output":"Apache/2.4.18 (Ubuntu)","elem":"Apache/2.4.18 (Ubuntu)"},{"id":"http-title","output":"Login","elem":"Login"}]}]},"times":{"srtt":"131055","rttvar":"1916","to":"138719"}},"runstats":{"finished":{"time":"1538195436","timestr":"Sat Sep 29 00:30:36 2018","elapsed":"465.19","summary":"Nmap done at Sat Sep 29 00:30:36 2018; 1 IP address (1 host up) scanned in 465.19 seconds","exit":"success"},"hosts":{"up":"1","down":"0","total":"1"}}}}

That’s…Readable? Maybe?? Okay probably not. In fact that’s arguably just as bad as the original XML, but not to fear! We can pipe that ugly JSON elsewhere to clean it up considerably. We’ll need to grab the tool for that, however.

The tool in question is a little (but powerful) Python-based gadget called yq.
Now, yq is a wrapper for jq, a JSON parsing tool (ALSO written in Python), but it allows us to do a little something extra, which I will demonstrate in a moment. For now, let’s grab yq.

# pip install yq

Now finally:

# nmap2json convert my_scan_data.xml | jq .

And with that, our output should look something like this:

{
  "nmaprun": {
    "scanner": "nmap",
    "args": "nmap -sC -sV -T4 -v -oA my_scan_data -p1-65535 10.10.10.105",
    "start": "1538194971",
    "startstr": "Sat Sep 29 00:22:51 2018",
    "version": "7.70",
    "xmloutputversion": "1.04",
    "scaninfo": {
      "type": "syn",
      "protocol": "tcp",
      "numservices": "65535",
      "services": "1-65535"
    },
    "verbose": {
      "level": "1"
    },
    "debugging": {
      "level": "0"
    },
    "taskbegin": [
      {
        "task": "NSE",
        "time": "1538194971"
      },
      {
        "task": "NSE",
        "time": "1538194971"
      },
      {
        "task": "Ping Scan",
        "time": "1538194971"
      },
      {
        "task": "Parallel DNS resolution of 1 host.",
        "time": "1538194971"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538194971"
      },
      {
        "task": "Service scan",
        "time": "1538195425"
      },
      {
        "task": "NSE",
        "time": "1538195432"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      }
    ],
    "taskend": [
      {
        "task": "NSE",
        "time": "1538194971"
      },
      {
        "task": "NSE",
        "time": "1538194971"
      },
      {
        "task": "Ping Scan",
        "time": "1538194971",
        "extrainfo": "1 total hosts"
      },
      {
        "task": "Parallel DNS resolution of 1 host.",
        "time": "1538194971"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195425",
        "extrainfo": "65535 total ports"
      },
      {
        "task": "Service scan",
        "time": "1538195432",
        "extrainfo": "2 services on 1 host"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      },
      {
        "task": "NSE",
        "time": "1538195436"
      }
    ],
    "taskprogress": [
      {
        "task": "SYN Stealth Scan",
        "time": "1538195002",
        "percent": "9.86",
        "remaining": "284",
        "etc": "1538195285"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195032",
        "percent": "16.40",
        "remaining": "311",
        "etc": "1538195343"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195062",
        "percent": "27.13",
        "remaining": "245",
        "etc": "1538195306"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195092",
        "percent": "43.68",
        "remaining": "157",
        "etc": "1538195248"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195122",
        "percent": "46.34",
        "remaining": "175",
        "etc": "1538195297"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195161",
        "percent": "49.79",
        "remaining": "192",
        "etc": "1538195353"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195230",
        "percent": "55.09",
        "remaining": "212",
        "etc": "1538195441"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195323",
        "percent": "65.60",
        "remaining": "185",
        "etc": "1538195508"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195353",
        "percent": "77.48",
        "remaining": "112",
        "etc": "1538195464"
      },
      {
        "task": "SYN Stealth Scan",
        "time": "1538195383",
        "percent": "90.18",
        "remaining": "45",
        "etc": "1538195428"
      }
    ],
    "host": {
      "starttime": "1538194971",
      "endtime": "1538195436",
      "status": {
        "state": "up",
        "reason": "echo-reply",
        "reason_ttl": "63"
      },
      "address": {
        "addr": "10.10.10.105",
        "addrtype": "ipv4"
      },
      "hostnames": "\n",
      "ports": {
        "extraports": {
          "state": "closed",
          "count": "65532",
          "extrareasons": {
            "reason": "resets",
            "count": "65532"
          }
        },
        "port": [
          {
            "protocol": "tcp",
            "portid": "21",
            "state": {
              "state": "filtered",
              "reason": "no-response",
              "reason_ttl": "0"
            },
            "service": {
              "name": "ftp",
              "method": "table",
              "conf": "3"
            }
          },
          {
            "protocol": "tcp",
            "portid": "22",
            "state": {
              "state": "open",
              "reason": "syn-ack",
              "reason_ttl": "63"
            },
            "service": {
              "name": "ssh",
              "product": "OpenSSH",
              "version": "7.6p1 Ubuntu 4",
              "extrainfo": "Ubuntu Linux; protocol 2.0",
              "ostype": "Linux",
              "method": "probed",
              "conf": "10",
              "cpe": [
                "cpe:/a:openbsd:openssh:7.6p1",
                "cpe:/o:linux:linux_kernel"
              ]
            },
            "script": {
              "id": "ssh-hostkey",
              "output": "\n  2048 15:a4:28:77:ee:13:07:06:34:09:86:fd:6f:cc:4c:e2 (RSA)\n  256 37:be:de:07:0f:10:bb:2b:b5:85:f7:9d:92:5e:83:25 (ECDSA)\n  256 89:5a:ee:1c:22:02:d2:13:40:f2:45:2e:70:45:b0:c4 (ED25519)",
              "table": [
                {
                  "elem": [
                    "15a42877ee130706340986fd6fcc4ce2",
                    "AAAAB3NzaC1yc2EAAAADAQABAAABAQDI2Jfx6VeMU2wFDys5YoSIVCu4U626/VDawUrXKa5SR+D8HaNvt6QFECtQumoFcYzxD7Jnd3PKw/dXTXvePTPnolDUNV3oimX8gEI3iY157v5scgrOKFjw39cTMuTfLc7/rM8e2TOeziN4yzzLfWAiTbe4wfiDe8cea7zJ1RFwvgGc398xiOA8bo1nwMD0wUkduJhtH4V98LpJZOVB4tMmtCdyb1T+e3HIR/1WbmMBLs0e6Cc/rf+K8vgqu6Tu/o4o8/TZ9aH9K5xoDRUXjU2R1w/Bi0HvYYHFRf664/NG9WcK/R0VlV6j92DOYL9wdUYwANyQPc4YCDfyuM6F6Bbd",
                    "2048.0",
                    "ssh-rsa"
                  ]
                },
                {
                  "elem": [
                    "37bede070f10bb2bb585f79d925e8325",
                    "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJToeoLQWJwkfcWBimMzO4E6BKOaHbTkWIk1uHoniOdaUaDL5C6MO2NeYYSaru/ikAYSHPU83p1p6hNcOJVy+OY=",
                    "256",
                    "ecdsa-sha2-nistp256"
                  ]
                },
                {
                  "elem": [
                    "895aee1c2202d21340f2452e7045b0c4",
                    "AAAAC3NzaC1lZDI1NTE5AAAAIIN0vm7BcvmBgddJb7k1W7qUkBgn2n0T1bdOU6GV1JB8",
                    "256",
                    "ssh-ed25519"
                  ]
                }
              ]
            }
          },
          {
            "protocol": "tcp",
            "portid": "80",
            "state": {
              "state": "open",
              "reason": "syn-ack",
              "reason_ttl": "62"
            },
            "service": {
              "name": "http",
              "product": "Apache httpd",
              "version": "2.4.18",
              "extrainfo": "(Ubuntu)",
              "method": "probed",
              "conf": "10",
              "cpe": "cpe:/a:apache:http_server:2.4.18"
            },
            "script": [
              {
                "id": "http-cookie-flags",
                "output": "\n  /: \n    PHPSESSID: \n      httponly flag not set",
                "table": {
                  "key": "/",
                  "table": {
                    "key": "PHPSESSID",
                    "elem": "httponly flag not set"
                  }
                }
              },
              {
                "id": "http-methods",
                "output": "\n  Supported Methods: GET HEAD POST OPTIONS",
                "table": {
                  "key": "Supported Methods",
                  "elem": [
                    "GET",
                    "HEAD",
                    "POST",
                    "OPTIONS"
                  ]
                }
              },
              {
                "id": "http-server-header",
                "output": "Apache/2.4.18 (Ubuntu)",
                "elem": "Apache/2.4.18 (Ubuntu)"
              },
              {
                "id": "http-title",
                "output": "Login",
                "elem": "Login"
              }
            ]
          }
        ]
      },
      "times": {
        "srtt": "131055",
        "rttvar": "1916",
        "to": "138719"
      }
    },
    "runstats": {
      "finished": {
        "time": "1538195436",
        "timestr": "Sat Sep 29 00:30:36 2018",
        "elapsed": "465.19",
        "summary": "Nmap done at Sat Sep 29 00:30:36 2018; 1 IP address (1 host up) scanned in 465.19 seconds",
        "exit": "success"
      },
      "hosts": {
        "up": "1",
        "down": "0",
        "total": "1"
      }
    }
  }
}

MUCH nicer. Now, if you want YAML output, it’s as simple as adding a flag to your yq command:

# nmap2json my_scan_data.xml | yq -y .

The output of which looks something like this:

nmaprun:
  scanner: nmap
  args: nmap -sC -sV -T4 -v -oA my_scan_data -p1-65535 10.10.10.105
  start: '1538194971'
  startstr: Sat Sep 29 00:22:51 2018
  version: '7.70'
  xmloutputversion: '1.04'
  scaninfo:
    type: syn
    protocol: tcp
    numservices: '65535'
    services: 1-65535
  verbose:
    level: '1'
  debugging:
    level: '0'
  taskbegin:
task: NSE
time: '1538194971'
task: NSE
time: '1538194971'
task: Ping Scan
time: '1538194971'
task: Parallel DNS resolution of 1 host.
time: '1538194971'
task: SYN Stealth Scan
time: '1538194971'
task: Service scan
time: '1538195425'
task: NSE
time: '1538195432'
task: NSE
time: '1538195436'
task: NSE
time: '1538195436'
task: NSE
time: '1538195436'
taskend:
task: NSE
time: '1538194971'
task: NSE
time: '1538194971'
task: Ping Scan
time: '1538194971'
extrainfo: 1 total hosts
task: Parallel DNS resolution of 1 host.
time: '1538194971'
task: SYN Stealth Scan
time: '1538195425'
extrainfo: 65535 total ports
task: Service scan
time: '1538195432'
extrainfo: 2 services on 1 host
task: NSE
time: '1538195436'
task: NSE
time: '1538195436'
task: NSE
time: '1538195436'
task: NSE
time: '1538195436'
taskprogress:
task: SYN Stealth Scan
time: '1538195002'
percent: '9.86'
remaining: '284'
etc: '1538195285'
task: SYN Stealth Scan
time: '1538195032'
percent: '16.40'
remaining: '311'
etc: '1538195343'
task: SYN Stealth Scan
time: '1538195062'
percent: '27.13'
remaining: '245'
etc: '1538195306'
task: SYN Stealth Scan
time: '1538195092'
percent: '43.68'
remaining: '157'
etc: '1538195248'
task: SYN Stealth Scan
time: '1538195122'
percent: '46.34'
remaining: '175'
etc: '1538195297'
task: SYN Stealth Scan
time: '1538195161'
percent: '49.79'
remaining: '192'
etc: '1538195353'
task: SYN Stealth Scan
time: '1538195230'
percent: '55.09'
remaining: '212'
etc: '1538195441'
task: SYN Stealth Scan
time: '1538195323'
percent: '65.60'
remaining: '185'
etc: '1538195508'
task: SYN Stealth Scan
time: '1538195353'
percent: '77.48'
remaining: '112'
etc: '1538195464'
task: SYN Stealth Scan
time: '1538195383'
percent: '90.18'
remaining: '45'
etc: '1538195428'
host:
starttime: '1538194971'
endtime: '1538195436'
status:
  state: up
  reason: echo-reply
  reason_ttl: '63'
address:
  addr: 10.10.10.105
  addrtype: ipv4
hostnames: '
'
ports:
  extraports:
    state: closed
    count: '65532'
    extrareasons:
      reason: resets
      count: '65532'
  port:
protocol: tcp
portid: '21'
state:
  state: filtered
  reason: no-response
  reason_ttl: '0'
service:
  name: ftp
  method: table
  conf: '3'
protocol: tcpportid: '22'state:  state: open  reason: syn-ack  reason_ttl: '63'service:  name: ssh  product: OpenSSH  version: 7.6p1 Ubuntu 4  extrainfo: Ubuntu Linux; protocol 2.0  ostype: Linux  method: probed  conf: '10'  cpe:cpe:/a:openbsd:openssh:7.6p1
cpe:/o:linux:linux_kernel
script:
id: ssh-hostkey
output: "\n  2048 15:a4:28:77:ee:13:07:06:34:09:86:fd:6f:cc:4c:e2 (RSA)\n\
\  256 37:be:de:07:0f:10:bb:2b:b5:85:f7:9d:92:5e:83:25 (ECDSA)\n  256\
\ 89:5a:ee:1c:22:02:d2:13:40:f2:45:2e:70:45:b0:c4 (ED25519)"
table:
elem:15a42877ee130706340986fd6fcc4ce2
AAAAB3NzaC1yc2EAAAADAQABAAABAQDI2Jfx6VeMU2wFDys5YoSIVCu4U626/VDawUrXKa5SR+D8HaNvt6QFECtQumoFcYzxD7Jnd3PKw/dXTXvePTPnolDUNV3oimX8gEI3iY157v5scgrOKFjw39cTMuTfLc7/rM8e2TOeziN4yzzLfWAiTbe4wfiDe8cea7zJ1RFwvgGc398xiOA8bo1nwMD0wUkduJhtH4V98LpJZOVB4tMmtCdyb1T+e3HIR/1WbmMBLs0e6Cc/rf+K8vgqu6Tu/o4o8/TZ9aH9K5xoDRUXjU2R1w/Bi0HvYYHFRf664/NG9WcK/R0VlV6j92DOYL9wdUYwANyQPc4YCDfyuM6F6Bbd
'2048.0'
ssh-rsa
elem:37bede070f10bb2bb585f79d925e8325
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJToeoLQWJwkfcWBimMzO4E6BKOaHbTkWIk1uHoniOdaUaDL5C6MO2NeYYSaru/ikAYSHPU83p1p6hNcOJVy+OY=
'256'
ecdsa-sha2-nistp256
elem:895aee1c2202d21340f2452e7045b0c4
AAAAC3NzaC1lZDI1NTE5AAAAIIN0vm7BcvmBgddJb7k1W7qUkBgn2n0T1bdOU6GV1JB8
'256'
ssh-ed25519
protocol: tcpportid: '80'state:  state: open  reason: syn-ack  reason_ttl: '62'service:  name: http  product: Apache httpd  version: 2.4.18  extrainfo: (Ubuntu)  method: probed  conf: '10'  cpe: cpe:/a:apache:http_server:2.4.18script:id: http-cookie-flags
output: "\n  /: \n    PHPSESSID: \n      httponly flag not set"
table:
key: /
table:
  key: PHPSESSID
  elem: httponly flag not set
id: http-methodsoutput: "\n  Supported Methods: GET HEAD POST OPTIONS"table:key: Supported Methodselem:GET
HEAD
POST
OPTIONS
id: http-server-header
output: Apache/2.4.18 (Ubuntu)
elem: Apache/2.4.18 (Ubuntu)
id: http-title
output: Login
elem: Login
times:
srtt: '131055'
rttvar: '1916'
to: '138719'
runstats:
finished:
time: '1538195436'
timestr: Sat Sep 29 00:30:36 2018
elapsed: '465.19'
summary: Nmap done at Sat Sep 29 00:30:36 2018; 1 IP address (1 host up) scanned
in 465.19 seconds
exit: success
hosts:
up: '1'
down: '0'
total: '1'

Now okay, maybe I have a bit of a bias against XML, if you REALLY want to leave the XML as it is, but pretty it up, you can use xmllint to do so:

# xmllint --format carrier_all_ports.xml

The output will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
<?xml-stylesheet href="file:///usr/bin/../share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 7.70 scan initiated Sat Sep 29 00:22:51 2018 as: nmap -sC -sV -T4 -v -oA carrier_all_ports -p1-65535 10.10.10.105 -->
<nmaprun scanner="nmap" args="nmap -sC -sV -T4 -v -oA carrier_all_ports -p1-65535 10.10.10.105" start="1538194971" startstr="Sat Sep 29 00:22:51 2018" version="7.70" xmloutputversion="1.04">
  <scaninfo type="syn" protocol="tcp" numservices="65535" services="1-65535"/>
  <verbose level="1"/>
  <debugging level="0"/>
  <taskbegin task="NSE" time="1538194971"/>
  <taskend task="NSE" time="1538194971"/>
  <taskbegin task="NSE" time="1538194971"/>
  <taskend task="NSE" time="1538194971"/>
  <taskbegin task="Ping Scan" time="1538194971"/>
  <taskend task="Ping Scan" time="1538194971" extrainfo="1 total hosts"/>
  <taskbegin task="Parallel DNS resolution of 1 host." time="1538194971"/>
  <taskend task="Parallel DNS resolution of 1 host." time="1538194971"/>
  <taskbegin task="SYN Stealth Scan" time="1538194971"/>
  <taskprogress task="SYN Stealth Scan" time="1538195002" percent="9.86" remaining="284" etc="1538195285"/>
  <taskprogress task="SYN Stealth Scan" time="1538195032" percent="16.40" remaining="311" etc="1538195343"/>
  <taskprogress task="SYN Stealth Scan" time="1538195062" percent="27.13" remaining="245" etc="1538195306"/>
  <taskprogress task="SYN Stealth Scan" time="1538195092" percent="43.68" remaining="157" etc="1538195248"/>
  <taskprogress task="SYN Stealth Scan" time="1538195122" percent="46.34" remaining="175" etc="1538195297"/>
  <taskprogress task="SYN Stealth Scan" time="1538195161" percent="49.79" remaining="192" etc="1538195353"/>
  <taskprogress task="SYN Stealth Scan" time="1538195230" percent="55.09" remaining="212" etc="1538195441"/>
  <taskprogress task="SYN Stealth Scan" time="1538195323" percent="65.60" remaining="185" etc="1538195508"/>
  <taskprogress task="SYN Stealth Scan" time="1538195353" percent="77.48" remaining="112" etc="1538195464"/>
  <taskprogress task="SYN Stealth Scan" time="1538195383" percent="90.18" remaining="45" etc="1538195428"/>
  <taskend task="SYN Stealth Scan" time="1538195425" extrainfo="65535 total ports"/>
  <taskbegin task="Service scan" time="1538195425"/>
  <taskend task="Service scan" time="1538195432" extrainfo="2 services on 1 host"/>
  <taskbegin task="NSE" time="1538195432"/>
  <taskend task="NSE" time="1538195436"/>
  <taskbegin task="NSE" time="1538195436"/>
  <taskend task="NSE" time="1538195436"/>
  <host starttime="1538194971" endtime="1538195436">
    <status state="up" reason="echo-reply" reason_ttl="63"/>
    <address addr="10.10.10.105" addrtype="ipv4"/>
    <hostnames>
</hostnames>
    <ports>
      <extraports state="closed" count="65532">
        <extrareasons reason="resets" count="65532"/>
      </extraports>
      <port protocol="tcp" portid="21">
        <state state="filtered" reason="no-response" reason_ttl="0"/>
        <service name="ftp" method="table" conf="3"/>
      </port>
      <port protocol="tcp" portid="22">
        <state state="open" reason="syn-ack" reason_ttl="63"/>
        <service name="ssh" product="OpenSSH" version="7.6p1 Ubuntu 4" extrainfo="Ubuntu Linux; protocol 2.0" ostype="Linux" method="probed" conf="10">
          <cpe>cpe:/a:openbsd:openssh:7.6p1</cpe>
          <cpe>cpe:/o:linux:linux_kernel</cpe>
        </service>
        <script id="ssh-hostkey" output="
  2048 15:a4:28:77:ee:13:07:06:34:09:86:fd:6f:cc:4c:e2 (RSA)
  256 37:be:de:07:0f:10:bb:2b:b5:85:f7:9d:92:5e:83:25 (ECDSA)
  256 89:5a:ee:1c:22:02:d2:13:40:f2:45:2e:70:45:b0:c4 (ED25519)">
          <table>
            <elem key="fingerprint">15a42877ee130706340986fd6fcc4ce2</elem>
            <elem key="key">AAAAB3NzaC1yc2EAAAADAQABAAABAQDI2Jfx6VeMU2wFDys5YoSIVCu4U626/VDawUrXKa5SR+D8HaNvt6QFECtQumoFcYzxD7Jnd3PKw/dXTXvePTPnolDUNV3oimX8gEI3iY157v5scgrOKFjw39cTMuTfLc7/rM8e2TOeziN4yzzLfWAiTbe4wfiDe8cea7zJ1RFwvgGc398xiOA8bo1nwMD0wUkduJhtH4V98LpJZOVB4tMmtCdyb1T+e3HIR/1WbmMBLs0e6Cc/rf+K8vgqu6Tu/o4o8/TZ9aH9K5xoDRUXjU2R1w/Bi0HvYYHFRf664/NG9WcK/R0VlV6j92DOYL9wdUYwANyQPc4YCDfyuM6F6Bbd</elem>
            <elem key="bits">2048.0</elem>
            <elem key="type">ssh-rsa</elem>
          </table>
          <table>
            <elem key="fingerprint">37bede070f10bb2bb585f79d925e8325</elem>
            <elem key="key">AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJToeoLQWJwkfcWBimMzO4E6BKOaHbTkWIk1uHoniOdaUaDL5C6MO2NeYYSaru/ikAYSHPU83p1p6hNcOJVy+OY=</elem>
            <elem key="bits">256</elem>
            <elem key="type">ecdsa-sha2-nistp256</elem>
          </table>
          <table>
            <elem key="fingerprint">895aee1c2202d21340f2452e7045b0c4</elem>
            <elem key="key">AAAAC3NzaC1lZDI1NTE5AAAAIIN0vm7BcvmBgddJb7k1W7qUkBgn2n0T1bdOU6GV1JB8</elem>
            <elem key="bits">256</elem>
            <elem key="type">ssh-ed25519</elem>
          </table>
        </script>
      </port>
      <port protocol="tcp" portid="80">
        <state state="open" reason="syn-ack" reason_ttl="62"/>
        <service name="http" product="Apache httpd" version="2.4.18" extrainfo="(Ubuntu)" method="probed" conf="10">
          <cpe>cpe:/a:apache:http_server:2.4.18</cpe>
        </service>
        <script id="http-cookie-flags" output="
  /: 
    PHPSESSID: 
      httponly flag not set">
          <table key="/">
            <table key="PHPSESSID">
              <elem>httponly flag not set</elem>
            </table>
          </table>
        </script>
        <script id="http-methods" output="
  Supported Methods: GET HEAD POST OPTIONS">
          <table key="Supported Methods">
            <elem>GET</elem>
            <elem>HEAD</elem>
            <elem>POST</elem>
            <elem>OPTIONS</elem>
          </table>
        </script>
        <script id="http-server-header" output="Apache/2.4.18 (Ubuntu)">
          <elem>Apache/2.4.18 (Ubuntu)</elem>
        </script>
        <script id="http-title" output="Login">
          <elem key="title">Login</elem>
        </script>
      </port>
    </ports>
    <times srtt="131055" rttvar="1916" to="138719"/>
  </host>
  <taskbegin task="NSE" time="1538195436"/>
  <taskend task="NSE" time="1538195436"/>
  <taskbegin task="NSE" time="1538195436"/>
  <taskend task="NSE" time="1538195436"/>
  <runstats>
    <finished time="1538195436" timestr="Sat Sep 29 00:30:36 2018" elapsed="465.19" summary="Nmap done at Sat Sep 29 00:30:36 2018; 1 IP address (1 host up) scanned in 465.19 seconds" exit="success"/>
    <hosts up="1" down="0" total="1"/>
  </runstats>
</nmaprun>

*shudder*

In the follow-up to this article I’ll talk about a couple of web-based tools you can use to make Nmap’s output a little more aesthetically pleasing.

Social Issues

#BellLetsTalk and Mental Health in the world of Software Development

Today in Canada, it’s “Bell Let’s Talk Day”. A day of national recognition and awareness of mental health issues set up by one of the country’s big three telecom and broadcasting companies. On this day, if you tweet with the hashtag #BellLetsTalk, send a snap on Snapchat with Bell’s “Let’s Talk” geofilter, or send an SMS from a phone subscribed to Bell’s mobile carrier service, Bell will donate five cents to various mental health initiatives. That doesn’t sound like a lot, but last year the #BellLetsTalk hashtag was the top-most trending hashtag in Canada. Bell also puts some serious star power behind Let’s Talk Day. Canadian Olympic Medalist Clara Hughes was instrumental in kicking off the first campaign years ago, and it’s now attracted names like Ryan Reynolds, Kevin Smith, and even Alice Cooper. For my five cents, it’s possibly the most effective awareness campaign since people started dumping ice-cold water on their heads in the name of ALS awareness.

With all that said, I’d like to talk about how all of this applies to my own experiences with mental health, and how working in my field can lend itself to serious challenges when it comes to mental health.

Working in software development can be extremely stressful and demanding, and I don’t think people outside our community understand how quickly and easily it can be for someone to burn out, and how common mental health issues can become as a result. Society has come to a point now where we depend on technology like an infant depends on its mother. We cannot live without our smartphones, our day-to-day financial transactions are almost completely digital now. And at the heart of all that technology, lie the people who build it, and make it all work: Developers, Sysadmins, DevOps Engineers, Security Engineers, and more. With that in mind, as a Developer, Sysadmin, DevOps engineer, or as a Security Person, you are essentially tasked with keeping the company you work for running; you are the gears that power the machine.

In the software world, aside from normal project deadline stress, you’re also often dealing directly with clients. Aside from dealing with clients, you’re also often dealing with fixing stuff when it breaks, sometimes at crazy hours because you’re on-call. And sometimes, in nightmare scenarios that happen more often than you might think, those all converge. Here’s a fun little scenario to demonstrate a little of what I mean:

So you get that crazy-hour on-call page; it’s your manager, and they’re telling you that a client’s service has gone down. So you look into it, and while looking into the problem, you discover that a developer managed to push some bad code into production, which caused the outage. So now you’re worried that your manager is pissed, that the client is pissed, you start realizing that you’ll have to take time in lieu in order to catch up on sleep so that you don’t fall asleep at your desk the next morning, but that means you’ll have less time to work on the project which is due next week, you start having thoughts that all (or part) of this could cost you your job, you’re worried that the dev you just called out because of their bad code is going to lose their job and you don’t want that on your conscience. Meanwhile the dev is worried about the same things, your manager is worried that this will make them look bad, that they didn’t provide you with everything you and/or the dev team with everything they needed to prevent this scenario from happening, that they could lose the client, and the client is worried about losing money because their goddamn service is down, and everyone is starting to fantasize about running away to a small thatched roof hut on a small, remote island off the coast of Borneo…one little mistake has now resulted in a tumult of potential mental health triggers.

Oh yeah, there’s that too…the constant dread that “one little mistake” will set off the very scenario I just described. Sure, companies nowadays are striving to put processes and tools in place to help take some of the pressure off, but that still doesn’t eliminate nightmare scenarios like the one I described…far from it. Nor does it eliminate the constant dread and anxiety. To sum it up, working in software is a constant juggling of chainsaws, but the gasoline from those chainsaws all too often makes you want to sneeze.

With all that said, I will say that I am glad to see and hear about growing numbers of companies really making an effort to build cultures around their software teams that are committed to dealing with stress and mental health in a constructive, supportive manner.

I’m also glad that the community itself has come forward to provide places where people in the industry can talk about mental health within the context of the industry; because it’s one thing to go to a therapist, or talk to a friend…but unless those friends and therapists are also technical or work in the same field, you often end up feeling like you need to generalize when describing stressful situations, which can lead to you feeling like you weren’t able to express yourself fully, which can be really frustrating. So, to be able to bounce ideas and feelings off of colleagues and other people in your field can be (pardon the pun) just what the doctor ordered.

So to all my friends outside of the software field, now you’ve had the chance to see a little of what we deal with in order to keep your shit working. You’re Welcome. Also, encourage us to talk about what we do. Take an interest. Even if you don’t understand all the technical stuff at first, you’ll discover how much we appreciate immensely the simple effort of TRYING to understand what we do. If you’re already doing this, thank you. It really does mean a lot.

To my friends/colleagues WITHIN the industry, don’t ever feel like you have to suffer in silence. There are resources and people out there that can help, and if you can’t find those people, start a group of your own; because I can assure you, you’re not alone. And if all of that fails, I don’t mind personally lending an ear…so Let’s Talk.