The core mission in the Network Operations Center (NOC) is network resilience. We also provide integrated security, visibility and automation: a SOC (Security Operations Center) inside the NOC, with Grifter and Bart as the leaders.

In part one, Black Hat Asia 2023 NOC: Connecting Singapore, we covered the network:

• Designing the Black Hat Network
• AP (Access Points) Placement Planning, by Uros Mihajlovic
• Security Center Investigations, by Uros Mihajlovic
• Meraki and ThousandEyes, by Uros Mihajlovic
• Meraki Dashboards, by Steven Fan
• Meraki Alerting, by Connor Loughlin
• Meraki Systems Manager, by Paul Fidler
• A Better Way to Design Training SSIDs/VLANs, by Paul Fidler

In part two, we focus on security:

• Integration is Key to Security
• Integrating Secure Cloud Analytics into the Black Hat Ecosystem Story, by Ryan MacLennan
• What is Your VPN (Virtual Private Network) Doing in the Background, by Aditya Raghavan
• Script Kiddie gets a Timeout, by Ben Greenbaum and Shaun Coulter
• Correlating Meraki Scanning Data with Umbrella DNS (Domain Name Service) Security Events, by Christen Clauson
• Domain Name Service Statistics and Improved Visibility, by Alejo Calaoagan

Integration is Key to Security

For Black Hat Asia 2023, Cisco Secure was the official Mobile Device Management, DNS and Malware Analysis Provider.

As the needs of Black Hat evolved, so did the Cisco Secure Technologies in the NOC:

• Cisco XDR: eXtended Detection and Response actions / Automations
• Cisco Umbrella: DNS visibility and security
• Cisco Secure Malware Analytics (Formerly Threat Grid): for sandboxing and integrated threat intelligence
• Cisco Secure Cloud Analytics (Formerly Stealthwatch Cloud): for network traffic visibility and threat detection
• Cisco Webex: for incident delivery and collaboration
• Cisco Security Connector: iOS device security and visibility, connected with Meraki Systems Manager

The Cisco XDR dashboard made it easy to see the status of each of the connected Cisco Secure technologies, and the Meraki APs for the network.

Since joining the Black Hat NOC in 2016, I continually advocate for integration and automation. Black Hat 2023 was the most integrated NOC thus far.

This requires collaboration and open communication with the NOC partners.

Below are the Cisco XDR integrations for Black Hat Asia, empowering analysts to investigate Indicators of Compromise (IOC) very quickly, with one search.

The integrations comprised two screens. To add an integration, we merely click on the module in the list below and then add the API (Application Programming Interfaces) key.

We appreciate alphaMountain.ai, Pulsedive and Recorded Future donating full licenses to the Black Hat Asia 2023 NOC.

File Analysis and Teamwork in the NOC

Corelight and NetWitness extracted several PDFs from the conference network stream, which were sent for analysis in Cisco Secure Malware Analytics (Threat Grid). In the glovebox video, they were observed as quotes from an Audio-Visual rental company vendor working at the Black Hat conference. The quotes contained personal and proprietary business information, which would make it quite easy to craft spear phishing attacks against both the rental company and the customers.

Investigation by the Corelight team determined the user downloaded the first file via HTTP from an unsecure portal [http://imxx[.]netxxx.com[.]sg/login/login[.]cfm], with login credentials in the clear.

Then they emailed via unsecure SMTP protocol to the client. The Palo Alto Firewall team confirmed the SMTP email and files.

The NetWitness team reconstructed the emails. The NOC team created a findings report for the vendor, to aid them in securing their webserver and switching to a secure email protocol.

Integrating Secure Cloud Analytics into the Black Hat Ecosystem Story, by Ryan MacLennan

For Black Hat Asia, Cisco was able to add Secure Cloud Analytics (SCA) into the mix as a network analytics platform, to help enrich and provide an additional layer of security to the Black Hat conference.

To begin our deployment, we first wanted to deploy the new Cisco Telemetry Broker (CTB); however, this may have caused issues with resource management on our Intel NUC that was providing other critical infrastructure. To alleviate any resource management issues we could run into, we deployed a lightweight on-prem network sensor instead of CTB. At future conferences, we will be using another NUC with CTB deployed, as that is the recommended way to send on-prem network data to SCA.

After deploying the on-prem sensor we worked with the Arista team to get us a network tap and enabled our Meraki MXs to send Netflow data to the sensor.

With us getting data from Arista for anything going in or out of the network and Meraki providing NetFlow data on internal connections, we could then use the Umbrella and Meraki SCA integrations to enrich the network analytics within SCA.

With these two integrations enabled we started seeing the information about each host and saw the judgements of domains and URLs these hosts go to within SCA.

We added customized alerts for notification, added third party threat intelligence lists, configured countries we want to watch, and added groups of categorizations of our network to notify when sections of our network talk to each other when they should not be doing so.

After these configurations were put in place, we were now able to start getting meaningful alerts about what is happening in our network. In the image below, you can see that we have gotten multiple alerts during the conference and responded to each with an investigation.

After these configurations were put in place, we were now able to start getting meaningful alerts about what is happening in our network. In the image below, you can see that we have gotten multiple alerts during the conference and responded to each with an investigation.

What's your VPN Doing in the Background, by Aditya Raghavan

Secure Cloud Analytics was setup with integrations to third party watchlists, like OSINT (Open-Source Intelligence) Threat Feed, Emerging Threat Compromised IPs and Blocklist DE, in addition to the built-in Talos threat feed. Secure Cloud Analytics flagged a User Watchlist Alert detecting unusual traffic to an IP on the Blocklist DE list, highlighting the unusual traffic size of just 60 bytes to and from the watchlist IP which looked like malware beaconing.

We dug down deeper with our partners.

The Palo Alto Networks team confirmed this traffic on the firewall, which helped identifying the endpoint sourcing this traffic. Secure Cloud Analytics also flagged numerous Geographic Watchlist Observations of the same traffic from that endpoint to various countries across the world, so we saw repeated such behavior. The Corelight team was able to pinpoint this traffic to a single ICMP ping and response from the user endpoint. The hosts generating it were flagged by Corelight as VPNInsights::PIA.

Based on our analysis, we were able to pinpoint this traffic being produced by Private Internet Access (PIA) VPN client on the user endpoint. This VPN application was seen to send pings to thousands of IPs across the entire world every 60 seconds, to test latency to the VPN headend servers.

In the end, we found the underlying cause of the weird traffic that looked like malware going to thousand IPs across the world, and determined it was nothing malicious.

Script Kiddie gets a Timeout, by Ben Greenbaum and Shaun Coulter

One attendee tried to color outside the lines and had to be reminded that (through the power of the XDR approach, enabled by integration with multiple partners) the NOC sees all. Cisco XDR Analytics warned us about potential port scanning behavior emanating from the conference network against the outside world.

Within a few minutes, analysts from all the NOC partners were all alerted via their own tools about different activity against outside, "real world" targets, all from the same host: Log4j exploitation attempts, WordPress attacks against a well-known restaurant chain, SQL injection and other attacks against a prominent payment processor, and many others.

The Incident Description provided additional information to collaborate with the NOC partners.

The collated events from multiple relevant sources are detailed in the XDR Detections page below.

Network detection is a foundational pillar of security awareness and was the first telemetry widely available to security operators for a reason. It was the first, although far from only, technology to report in this instance as well.

The NetWitness team analyzed the PCAP (packet capture) of the attack.

The Palo Alto Networks Firewall team alerted on several attempted exploits.

The Meraki MX Security Team tracked the attacks in the Security Center.

The Corelight attack notices also confirmed the attacks.

In addition, this subject was seen performing various attack-adjacent activities, such as passive DNS research, CRL manipulation, HTTP scanning, port scanning and others.

The visualization in Cisco XDR helped the NOC team understand the scope of the attack, easily highlighting the most relevant information out of the sea of evidence.

Further analysis in these and other tools revealed a pattern of behavior that had a start earlier in the morning, a gap of about an hour, and then approximately 15 minutes of uninterrupted high volume attack activity that signified the use of automation.

While there are many things this team is tasked to observe but not interfere with, the Black Hat Code of Conduct expressly forbids attacking outside targets from anywhere within the Black Hat network. We provided the attacker's MAC address to the Palo Alto Networks Firewall team, who initiated a captive portal for the user a captive portal that politely reminded them of the Code of Conduct and ended with "if it continues we will come find you".

It did not continue.

More Scanning

In a similar encounter, Cisco XDR was alerted by network detection of an internal user scanning outside targets not associated with an approved curriculum. Notably, there were only two specific target IPs of this activity, and more notably, those targets were two of the IPs that host www.blackhat.com.

We investigated the device's network activity and found that the scanning comprised over 50% of their total network traffic at that time. The scans targeted exactly 1000 unique ports between 1 and 65389, and included all the usual service ports, as well as common secondary options. In this case, it was noted by the NOC leaders and not acted upon because Black Hat and hackers gonna hack, but the source was recorded as a subject of future closer scrutiny and a device of potential interest, should activity happen again. In a production environment, insider threats are very real and intentional reconnaissance against the organization's assets -including those outside the internal network -by anyone not specifically tasked with that activity, should be monitored and acted upon.

Correlating Meraki Scanning Data with Umbrella DNS Security Events, by Christian Clasen

Over the last three Black Hat events, we used Meraki scanning data to get location data for individual clients, as they roamed the conference. The project has slowly evolved from simply saving data off to flat text files for future analysis, to generating heatmaps using Python Folium, to populating a database, and finally correlating Umbrella DNS security events.

As the conference grew from the pandemic-era attendance (about 20% of previous events) back to full capacity, we had to make some adjustments to the process of ingesting the data from the Meraki streaming API. To aid with other integrations, we began writing the incoming data to files instead of directly to the database within the Flask app. We then added a scheduled job to read the files into the database every five seconds.

In past conferences, we would manually run the scripts to generate heatmaps (.html files) for analysis. This time, we wanted the maps to be generated automatically, always be up-to-date and be available to everyone over a web service. So, we created a new module that would host another Flask web app. In the module, we defined the bounds of each day in epoch time, and scheduled a job to create the maps every five minutes:

A map for each day was then generated and dropped into the "/templates" folder. By using the "render_template()" function, it displays the heatmap in the browser when navigating to the appropriate path. For example, we could make a request tohttps://webserver/wedand be served the heatmap for Wednesday, 10 May:

This way, anyone in the NOC could open the path to the current day in their browser and see the latest map up to the previous five minutes. But we did not want to have to manually refresh the page to get the latest map, so we added some JavaScript that would prompt the browser to refresh. First, we added a link to "refresh.js" in the map HTML:

Then we added a simple window refresh in the file, located in the "templates" directory:

Domain Name Service Statistics and Improved Visibility, by Alejo Calaoagan

Since 2018, we have been tracking the DNS stats at the Black Hat Asia conferences. This year's attendance saw well over 6.2 million total DNS queries.

This was the highest to date for Black Hat Asia.

This year's Black Hat saw over 1,100 apps connect to the network, nearly half of what was seen last year. This was the first time we have ever seen a decline in the number of Apps.

Should the need arise, we can block any application, such as any of the high-risk apps identified above.

Improving Network Visibility

At every Black Hat we support, we are always looking for ways to improve traffic visibility to help us identify malicious user activity more quickly. To facilitate better data, we worked with the network design team to define each room and area of the conference floor with their own VLAN and subnet.

By defining subnets and VLANs for each area in use at the show, we were now able to identify malicious events by the area the request was made. This added insight improved our data quality and helped us identify threats and trends much faster within our threat hunting duties.

Looking at the security events above, we see that these requests came from one of the Black Hat training rooms. In years past, we would have to jump through a couple different user interfaces (Meraki/Umbrella) to validate intent and location. Now, after a quick check-in with the training room instructor to make sure these requests were part of the course curriculum, we can safely move on to the next hunt.

Improving visibility even further, we worked with James Holland and the Palo Alto Networks firewall team to help us uncover data that is typically masked within Umbrella.

The savvier users out there may hard code DNS on their machines to maintain some level of control and privacy. To account for this, Palo Alto Networks NAT'ed (Network Address Translation) all this masked traffic through our Umbrella virtual appliances on site. Traffic previously masked was now visible and trackable within the VLANs and subnets defined above. This added visibility improved the quality of our statistics, supplying data that was previously a black box.

This is what it looked like inside the Palo Alto Networks Firewall.

This allowed us to detect traffic to a malicious domain.

Then use Umbrella Investigate to learn more and take appropriate action.

That is a wrap folks, another Black Hat Asia in the history books. With over 2,500 total attendees this year, it is safe to say that the show was a success. Learning from past events, we have truly streamlined our deployment and investigative processes.

We are proud of the collaboration of the Cisco team and the NOC partners. Black Hat USA will be in August 2023 at the Mandalay Bay... Hope to see you there!

Acknowledgments

Thank you to the Cisco NOC team:

• Cisco Secure: Christian Clasen, Alex Calaoagan, Ben Greenbaum, Ryan Maclennan, Shaun Coulter and Aditya Raghavan; with virtual support by Ian Redden and Adi Sankar
• Meraki Systems Manager:Paul Fidler and Connor Loughlin
• Meraki Network:Steven Fan, Uros Mihajlovic and Jeffrey Chua; with virtual support by Evan Basta and Jeffry Handal

Also, to our NOC partners: NetWitness (especially David Glover, Iain Davidson and Alessandro Zatti), Palo Alto Networks (especially James Holland), Corelight (especially Dustin Lee), Arista, MyRepublic and the entire Black Hat / Informa Tech staff (especially Grifter 'Neil Wyler,' Bart Stump, Steve Fink, James Pope, Mike Spicer, Jess Stafford and Steve Oldenbourg).

About Black Hat

For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: Black Hat.com. Black Hat is brought to you by Informa Tech.

We'd love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn