In part one of our Black Hat Asia 2022 NOC blog, we discussed building the network with Meraki: 

• From attendee to press to volunteer -coming back to Black Hat as NOC volunteer by Humphrey Cheung 
• Meraki MR, MS, MX and Systems Manager by Paul Fidler 
• Meraki Scanning API Receiver by Christian Clasen 

In this part two, we will discuss:  

• SecureX: Bringing Threat Intelligence Together by Ian Redden 
• Device type spoofing event by Jonny Noble 
• Self Service with SecureX Orchestration and Slack by Matt Vander Horst 
• Using SecureX sign-on to streamline access to the Cisco Stack at Black Hat by Adi Sankar 
• Future Threat Vectors to Consider -Cloud App Discovery by Alejo Calaoagan 
• Malware Threat Intelligence made easy and available, with Cisco SecureMalware Analytics and SecureX by Ben Greenbaum 

SecureX: Bringing Threat Intelligence Together by Ian Redden 

In addition to the Meraki networking gear, Cisco Secure also shipped two Umbrella DNS virtual appliances to Black Hat Asia, for internal network visibility with redundancy, in addition to providing: 

• SecureXextended detection and response platform, with orchestration and device insights integration with Meraki and Secure Endpoint 
• Umbrelladomain name service protection 
• Umbrella Investigate 
• Secure Endpoint for iOS/Security Connector 
• Secure Malware Analytics(formerly Threat Grid) 

Cisco Secure Threat Intelligence (correlated through SecureX)

• SecureX threat intelligence
• Cisco Secure Endpoint's File Reputation Database
• Cisco Talos Intelligence

Donated Partner Threat Intelligence (correlated through SecureX)

• Recorded Future threat intelligence

• alphaMountain.ai threat intelligence

Open-Source Threat Intelligence (correlated through SecureX)

• APIVoid
• CyberCrime Tracker
• Have I Been Pwned
• IBM X-Force Exchange
• Palo Alto Networks AutoFocus
• Pulsedive
• Shodan
• Threatscore | Cyberprotect| Cyberprotect

Continued Integrations from past Black Hat events

• NetWitness PCAP file carving and submission to Cisco Secure Malware Analytics (formerly Threat Grid) for analysis

New Integrations Created at Black Hat Asia 2022

• SecureX threat response and NetWitness SIEM: Sightings in investigations
• SecureX orchestration workflows for Slack that enabled:
  • Administrators to block a device by MAC address for violating the conference Code of Conduct
  • NOC members to query Meraki for information about network devices and their clients
  • NOC members to update the VLAN on a Meraki switchport
  • NOC members to query Palo Alto Panorama for client information
  • Notification if an AP went down
• NetWitness SIEM integration with Meraki syslogs
• Palo Alto Panorama integration with Meraki syslogs
• Palo Alto Cortex XSOAR integration with Meraki and Umbrella

Device type spoofing event by Jonny Noble

Overview

During the conference, a NOC Partner informed us that they received an alert from May 10 concerning an endpoint client that accessed two domains that they saw as malicious:

• legendarytable[.]com
• drakefollow[.]com

Client details from Partner:

• Private IP: 10.XXX.XXX.XXX
• Client name: LAPTOP-8MLGDXXXX
• MAC: f4:XX:XX:XX:XX:XX
• User agent for detected incidents: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/602.2.8 (KHTML, like Gecko) Version/11.0 Mobile/14B55c Safari/602.1

Based on the user agent, the partner derived that the device type was an Apple iPhone.

SecureX analysis

• legendarytable[.]com a Judgement of Suspicious by alphaMountain.ai
• drakefollow[.]com a Judgement of Malicious by alphaMountain.ai

Umbrella Investigate analysis

Umbrella Investigate positions both domains as low risk, both registered recently in Poland, and both hosted on the same IP:

Despite the low-risk score, the nameservers have high counts of malicious associated domains:

Targeting users in ASA, UK, and Nigeria:

Meraki analysis

Based on the time of the incident, we can trace the device's location (based on its IP address). This is thanks to the effort we invested in mapping out the exact location of all Meraki APs, which we deployed across the convention center with an overlay of the event map covering the area of the event:

• Access Point: APXX
• Room: Orchid Ballroom XXX
• Training course at time in location: "Web Hacking Black Belt Edition"

Further analysis and conclusions

The device name (LAPTOP-8MLGXXXXXX) and MAC address seen (f4:XX:XX:XX:XX:XX) both matched across the partner and Meraki, so there was no question that we were analyzing the same device.

Based on the useragent captured by the partner, the device type was an Apple iPhone. However, Meraki was reporting the Device and its OS as "Intel, Android"

A quick look up for the MAC address confirmed that the OUI (organizationally unique identifier) for f42679 was Intel Malaysia, making it unlikely that this was an Apple iPhone.

The description for the training "Web Hacking Black Belt Edition" can be seen here:

https://www.blackhat.com/asia-22/training/schedule/#web-hacking-black-belt-edition-day-25388

It is highly likely that the training content included the use of tools and techniques for spoofing the visibility of useragent or device type.

There is also a high probability that the two domains observed were used as part of the training activity, rather than this being part of a live attack.

It is clear that integrating the various Cisco technologies (Meraki wireless infrastructure, SecureX, Umbrella, Investigate) used in the investigation of this incident, together with the close partnership and collaboration of our NOC partners, positioned us where we needed to be and provided us with the tools we needed to swiftly collect the data, join the dots, make conclusions, and successfully bring the incident to closure.

Self Service with SecureX Orchestration and Slack by Matt Vander Horst

Overview

Since Meraki was a new platform for much of the NOC's staff, we wanted to make information easier to gather and enable a certain amount of self-service. Since the Black Hat NOC uses Slack for messaging, we decided to create a Slack bot that NOC staff could use to interact with the Meraki infrastructure as well as Palo Alto Panorama using the SecureX Orchestration remote appliance. When users communicate with the bot, webhooks are sent to Cisco SecureX Orchestration to do the work on the back end and send the results back to the user.

Design

Here's how this integration works:

1. When a Slack user triggers a '/' "slash command" or other type of interaction, a webhook is sent to SecureX Orchestration. Webhooks trigger orchestration workflows which can do any number of things. In this case, we have two different workflows: one to handle slash commands and another for interactive elements such as forms (more on the workflows later).
2. Once the workflow is triggered, it makes the necessary API calls to Meraki or Palo Alto Panorama depending on the command issued.
3. After the workflow is finished, the results are passed back to Slack using either an API request (for slash commands) or webhook (for interactive elements).
4. The user is presented with the results of their inquiry or the action they requested.

Workflow#1: Handle Slash Commands

Slash commands are a special type of message built into Slack that allow users to interact with a bot. When a Slack user executes a slash command, the command and its arguments are sent to SecureX Orchestration where a workflow handles the command. The table below shows a summary of the slash commands our bot supported for Black Hat Asia 2022:

Here's a sample of a portion of the SecureX Orchestration workflow that powers the above commands:

And here's a sample of firewall logs as returned from the "/pan_traffic_history" command:

Workflow#2: Handle Interactivity

A more advanced form of user interaction comes in the form of Slack blocks. Instead of including a command's arguments in the command itself, you can execute the command and Slack will present you with a form to complete, like this one for the "/update_vlan" command:

These forms are much more user friendly and allow information to be pre-populated for the user. In the example above, the user can simply select the switch to configure from a drop-down list instead of having to enter its name or serial number. When the user submits one of these forms, a webhook is sent to SecureX Orchestration to execute a workflow. The workflow takes the requested action and sends back a confirmation to the user:

Conclusion

While these two workflows only scratched the surface of what can be done with SecureX Orchestration webhooks and Slack, we now have a foundation that can be easily expanded upon going forward. We can add additional commands, new forms of interactivity, and continue to enable NOC staff to get the information they need and take necessary action. The goal of orchestration is to make life simpler, whether it is by automating our interactions with technology or making those interactions easier for the user. 

Future Threat Vectors to Consider -Cloud App Discovery by Alejo Calaoagan

Since 2017 (starting in Black Hat USA -Las Vegas), Cisco Umbrella has provided DNS security to the Black Hat attendee network, added layers of traffic visibility previously not seen. Our efforts have largely been successful, identifying thousands of threats over the years and mitigating them via Umbrella's blocking capabilities when necessary. This was taken a step further at Black Hat London 2021, where we introduced our Virtual Appliances to provide source IP attribution to the devices making requests.

Here at Black Hat Asia 2022, we've been noodling on additional ways to provide advanced protection for future shows, and it starts with Umbrella's Cloud Application Discovery's feature, which identified 2,286 unique applications accessed by users on the attendee network across the four-day conference.  Looking at a snapshot from a single day of the show, Umbrella captured 572,282 DNS requests from all cloud apps, with over 42,000 posing either high or very high risk.

Digging deeper into the data, we see not only the types of apps being accessed...

...but also see the apps themselves...

...and we can flag apps that look suspicious.

We also include risk downs breaks by category...

...and drill downs on each.

While this data alone won't provide enough information to take action, including this data in analysis, something we have been doing, may provide a window into new threat vectors that may have previously gone unseen. For example, if we identify a compromised device infected with malware or a device attempting to access things on the network that are restricted, we can dig deeper into the types of cloud apps those devices are using and correlate that data with suspicious request activity, potential uncovering tools we should be blocking in the future.

I can't say for certain how much this extra data set will help us uncover new threats, but, with Black Hat USA just around the corner, we'll find out soon.

Using SecureX sign-on to streamline access to the Cisco Stack at Black Hat by Adi Sankar

From five years ago to now, Cisco has tremendously expanded our presence at Black Hat to include a multitude of products. Of course, sign-on was simple when it was just one product (Secure Malware Analytics) and one user to log in. When it came time to add a new technology to the stack it was added separately as a standalone product with its own method of logging in. As the number of products increased, so did the number of Cisco staff at the conference to support these products. This means sharing usernames and passwords became tedious and not to mention insecure, especially with 15 Cisco staff, plus partners, accessing the platforms.

The Cisco Secure stack at Black Hat includes SecureX, Umbrella, Malware Analytics, Secure Endpoint (iOS clarity), and Meraki. All of these technologies support using SAML SSO natively with SecureX sign-on. This means that each of our Cisco staff members can have an individual SecureX sign-on account to log into the various consoles. This results in better role-based access control, better audit logging and an overall better login experience. With SecureX sign-on we can log into all the products only having to type a password one time and approve one Cisco DUO Multi-Factor Authentication (MFA) push.

How does this magic work behind the scenes? It's actually rather simple to configure SSO for each of the Cisco technologies, since they all support SecureX sign-on natively. First and foremost, you must set up a new SecureX org by creating a SecureX sign-on account, creating a new organization and integrating at least one Cisco technology. In this case I created a new SecureX organization for Black Hat and added the Secure Endpoint module, Umbrella Module, Meraki Systems Manager module and the Secure Malware Analytics module. Then from Administration a Users in SecureX, I sent an invite to the Cisco staffers that would be attending the conference, which contained a link to create their account and join the Blackhat SecureX organization. Next let's take a look at the individual product configurations.

Meraki:

In the Meraki organization settings enable SecureX sign-on. Then under Organization a Administrators add a new user and specify SecureX sign-on as the authentication method. Meraki even lets you limit users to particular networks and set permission levels for those networks. Accepting the email invitation is easy since the user should already be logged into their SecureX sign-on account. Now, logging into Meraki only requires an email address and no password or additional DUO push.

Umbrella:

Under Admin a Authentication configure SecureX sign-on which requires a test login to ensure you can still login before using SSO for authentication to Umbrella. There is no need to configure MFA in Umbrella since SecureX sign-on comes with built in DUO MFA. Existing users and any new users added in Umbrella under Admin a Accounts will now be using SecureX sign-on to login to Umbrella. Logging into Umbrella is now a seamless launch from the SecureX dashboard or from the SecureX ribbon in any of the other consoles.

Secure Malware Analytics:

A Secure Malware Analytics organization admin can create new users in their Threat Grid tenant. This username is unique to Malware Analytics, but it can be connected to a SecureX sign-on account to take advantage of the seamless login flow. From the email invitation the user will create a password for their Malware Analytics user and accept the EULA. Then in the top right under My Malware Analytics Account, the user has an option to connect their SecureX sign-on account which is a one click process if already signed in with SecureX sign-on. Now when a user navigates to Malware Analytics login page, simply clicking "Login with SecureX Sign-On" will grant them access to the console.

Secure Endpoint:

The Secure Endpoint deployment at Blackhat is limited to IOS clarity through Meraki Systems Manager for the conference IOS devices. Most of the asset information we need about the iPhones/iPads is brought in through the SecureX Device Insights inventory. However, for initial configuration and to view device trajectory it is required to log into Secure Endpoint. A new Secure Endpoint account can be created under Accounts a Users and an invite is sent to corresponding email address. Accepting the invite is a smooth process since the user is already signed in with SecureX sign-on. Privileges for the user in the Endpoint console can be granted from within the user account.

Conclusion:

To sum it all up, SecureX sign-on is the standard for the Cisco stack moving forward. With a new SecureX organization instantiated using SecureX sign-on any new users to the Cisco stack at Black Hat will be using SecureX sign-on. SecureX sign-on has helped our user management be much more secure as we have expanded our presence at Black Hat. SecureX sign-on provides a unified login mechanism for all the products and modernized our login experience at the conference.

Malware Threat Intelligence made easy and available, with Cisco Secure Malware Analytics and SecureX by Ben Greenbaum

I'd gotten used to people's reactions upon seeing SecureX in use for the first time. A few times at Black Hat, a small audience gathered just to watch us effortlessly correlate data from multiple threat intelligence repositories and several security sensor networks in just a few clicks in a single interface for rapid sequencing of events and an intuitive understanding of security events, situations, causes, and consequences. You've already read about a few of these instances above. Here is just one example of SecureX automatically putting together a chronological history of observed network events detected by products from two vendors (Cisco Umbrella and NetWitness) . The participation of NetWitness in this and all of our other investigations was made possible by our open architecture, available APIs and API specifications, and the creation of the NetWitness module described above.

In addition to the traffic and online activities of hundreds of user devices on the network, we were responsible for monitoring a handful of Black Hat-owned devices as well. Secure X Device Insights made it easy to access information about these assets, either en masse or as required during an ongoing investigation. iOS Clarity for Secure Endpoint and Meraki System Manager both contributed to this useful tool which adds business intelligence and asset context to SecureX's native event and threat intelligence, for more complete and more actionable security intelligence overall.

SecureX is made possible by dozens of integrations, each bringing their own unique information and capabilities. This time though, for me, the star of the SecureX show was our malware analysis engine, Cisco Secure Malware Analytics (CSMA). Shortly before Black Hat Asia, the CSMA team released a new version of their SecureX module. SecureX can now query CSMA's database of malware behavior and activity, including all relevant indicators and observables, as an automated part of the regular process of any investigation performed in SecureX Threat Response.

This capability is most useful in two scenarios:

1: determining if suspicious domains, IPs and files reported by any other technology had been observed in the analysis of any of the millions of publicly submitted file samples, or our own.
2: rapidly gathering additional context about files submitted to the analysis engine by the integrated products in the Black Hat NOC.

The first was a significant time saver in several investigations. In the example below, we received an alert about connections to a suspicious domain. In that scenario, our first course of action is to investigate the domain and any other observables reported with it (typically the internal and public IPs included in the alert). Due to the new CSMA module, we immediately discovered that the domain had a history of being contacted by a variety of malware samples, from multiple families, and that information, corroborated by automatically gathered reputation information from multiple sources about each of those files, gave us an immediate next direction to investigate as we hunted for evidence of those files being present in network traffic or of any traffic to other C&C resources known to be used by those families. From the first alert to having a robust, data-driven set of related signals to look for, took only minutes, including from SecureX partnerRecorded Future, who donated a full threat intelligence license for the Black Hat NOC.

The other scenario, investigating files submitted for analysis, came up less frequently but when it did, the CSMA/SecureX integration was equally impressive. We could rapidly, nearly immediately, look for evidence of any of our analyzed samples in the environment across all other deployed SecureX-compatible technologies. That evidence was no longer limited to searching for the hash itself, but included any of the network resources or dropped payloads associated with the sample as well, easily identifying local targets who had not perhaps seen the exact variant submitted, but who had nonetheless been in contact with that sample's Command and Control infrastructure or other related artifacts.

And of course, thanks to the presence of the ribbon in the CSMA UI, we could be even more efficient and do this with multiple samples at once.

SecureX greatly increased the efficiency of our small volunteer team, and certainly made it possible for us to investigate more alerts and events, and hunt for more threats, all more thoroughly, than we would have been able to without it. SecureX truly took this team to the next level, by augmenting and operationalizing the tools and the staff that we had at our disposal.

We look forward to seeing you at Black Hat USA in Las Vegas, 6-11 August 2022!

Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team: Aditya Sankar, Aldous Yeung, Alejo Calaoagan, Ben Greenbaum, Christian Clasen, Felix H Y Lam, George Dorsey, Humphrey Cheung, Ian Redden, Jeffrey Chua, Jeffry Handal, Jonny Noble, Matt Vander Horst, Paul Fidler and Steven Fan.

Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially James Holland), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Grifter 'Neil Wyler', Bart Stump, James Pope, Steve Fink and Steve Oldenbourg).

About Black Hat

For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.